Information and Entropy

Shannon and Weaver (S&W) presented an engineering solution to an information problem with something that would eventually be known as information theory. Though very mathematically, its philosophical roots are intrinsically linked to a physical property of matter known as entropy, or the amount of thermal energy, or kinetic energy of the electrons in a closed system. In another definition, entropy also measures a degree of disorder.

S&W equate uncertainty, or information that is not known, to entropy, or degree of disorder in the closed system, by assuming that the probabilistic amount of order in nature with respect to information, or that the natural state of organized information in a closed system is constant and equal to 100%. Thus the goal of their method is to totally reduce, minimize, and eliminate uncertainty.

Let’s consider the next examples. It is also important to know that S&W addressed the problem of noise reduction in electronic transmissions. In one case, a message is correctly transmitted and received. There is no loss of information and uncertainty is equal to 0. In another case, the message is garbled and the information is totally lost, so that uncertainty is equal to 1. Probability is bound by 0 and 1.

The usefulness of S&W appears in all its majestic importance when the message is partially garbled. Is in those cases when the importance of the assumptions used to calculate probabilities emerged. Faster computer processing and inexpensive data storage enabled better corrective solutions. S&W provided the theoretical approach and computer technologies allowed its implementation.

But in general, with respect to computer-based information processing, and in particular with respect to information retrieval, there is great temptation to use the theory without realizing that the environment for which it was developed resulted in a closed system. Language utilization for communication is an open system.

In other words, information theory has limitations. It behooves the researcher and student of information processing to understand that S&W bound this framework and environment in a very particular way. The enticing opportunity of information theory should be balanced with the reality of the results.

For these reasons, one should be asking why is it that systems work at all when using this and other unsuitable theories rather than trying to gain efficiency on processes that are theoretical flawed.