Information Theory

I

Introduction

Information Theory, application of mathematical principles to the problems of transmitting and storing information. Information theory stems originally from the work of American mathematician and electrical engineer Claude E. Shannon and, in particular, his classic paper “A Mathematical Theory of Communication,” published in 1948 in the Bell System Technical Journal. Shannon rewrote the paper and, with Warren Weaver, published it in book form as The Mathematical Theory of Communication in 1949. Shannon was interested in the problems of communicating information accurately from one place to another. He decided to approach communication from a mathematical point of view.

Information theory focuses on the problems inherent in sending and receiving messages and information. The theory is based on the idea that communication involves uncertain processes, both in the selection of the message to be transmitted and in the transmission of the message itself. Information theory provides a way to measure this uncertainty precisely.

In information theory, the actual meaning of the message is unimportant. Instead, the important qualities of communication are the amount of information that the message contains, the accuracy of the transmission, and the quality of the reception. All of these values are represented mathematically, so different messages and different communication systems can be compared, studied, and improved.

Information theory measures the amount of information in a message by using units called bits, short for binary digits, which use only the numbers 0 and 1 (see Number Systems). Information theory is useful because it provides a way to find the minimum number of bits required to communicate a given message. Information theory can also determine the maximum rate, in bits per second, at which a given communication channel can transmit reliable information.

---

---

Information theory is primarily a theoretical study. However, it has had a profound impact on the design of practical data communication and storage systems, such as telephones and computers. The theory can be applied to both the transmission and the storage of messages, because storage is nothing more than transmission in time. For example, both making a telephone call to a friend in another city and tape recording a message for a friend to play later in the day involve the same issues of sending and receiving messages. In information theory, no fundamental distinction is made between these two types of problems.

II

Parts of a Communication System

Any time a message is sent from a sender to a receiver, the different parts of the communication system can be represented by the accompanying schematic figure titled 'Elements of a Communication System,' adapted from Shannon’s work on information theory. The model he devised to represent a communication system always consists of five major parts: the information source, the transmitter, the channel, the receiver, and the destination.

The information source produces (or selects) the message or the sequence of messages to be transmitted to the destination. For example, the information source could be a distant spacecraft and the message could be an image of a planet, or the information source could be a rock-and-roll band and the message could be a new song.

The transmitter converts the message into a signal suitable for transmission over the channel. For example, the transmitter could be the spacecraft telecommunication system that converts a photograph of Jupiter into a television signal. Another example would be the recording studio’s audio equipment, which converts the rock-and-roll song into a sequence of tiny pits on the mirrorlike surface of a compact disc (CD).

The channel is the medium that is used to transmit the signal. The channel is often noisy, in the sense that when the signal arrives at the receiver, it may contain noise or static, or it may be slightly garbled. For example, the channel could be the millions of kilometers of empty space between Jupiter and Earth, with noise arising because the received signal is so weak. Or it could be the surface of a CD, with noise occurring because of fingerprints, dust, or scratches on the surface.

The receiver is a device that reconstructs (either exactly or approximately) the message from the received signal. It could be a large dish antenna or the electronics in a CD player.

The destination is the person (or thing) for which the message is intended. For example, the destination could be a teenager interested in planetary science or an astronomer interested in rock and roll.

Information theory is the mathematical study of these five components, individually and in combination. Existing communication systems can be studied this way, and new systems can be designed based on the knowledge gained. For example, information theory can provide a way to measure the amount of information produced by a source or to measure the ability of a noisy channel to transmit information reliably. In addition, the theory provides the theoretical basis for data compression, which is a way to squeeze more information into a message by eliminating redundancy, or parts of the message that do not contain any important information. Information theory also offers guidelines for the engineering design of transmitters and receivers.

III

Measuring Information: The Bit

In any communication system the message produced by the source is one of several possible messages. The receiver will know what these possibilities are but will not know which one has been selected. Shannon observed that any such message can be represented by a sequence of fundamental units called bits, consisting of either 0s or 1s. The number of bits required for a message depends on the number of possible messages: the more possible messages (and hence the more initial uncertainty at the receiver), the more bits required.

As a simple example, suppose a coin is flipped and the outcome (heads or tails) is to be communicated to a person in the next room. The outcome of the flip of a coin can be represented using one bit of information: 0 for heads and 1 for tails. Similarly, the outcome of a football game might also be represented with one bit: 0 if the home team loses and 1 if the home team wins. These examples emphasize one of the limitations of information theory—it cannot measure (and does not attempt to measure) the meaning or the importance of a message. It requires the same amount of information to distinguish heads from tails as it does to distinguish a win from a loss: one bit.

For an example with more than two outcomes, more bits are required. Suppose a playing card is chosen at random from a 52-card deck, and the suit chosen (hearts, spades, clubs, or diamonds) is to be communicated. Communicating the chosen suit (one of four possible messages) requires two bits of information, using the following simple scheme:

IV

Information Content and Entropy

A fundamental problem in information theory is to find the minimum average number of bits needed to represent a particular message selected from a set of possible messages. Shannon solved this problem by using the notion of entropy. The word entropy is borrowed from physics, in which entropy is a measure of the disorder of a group of particles. In information theory disorder implies uncertainty and, therefore, information content, so in information theory, entropy describes the amount of information in a given message. Entropy also describes the average information content of all the potential messages of a source. This value is useful when, as is often the case, some messages from a source are more likely to be transmitted than are others.

	More from Encarta
	•  Presidential Myths Quiz •  Coffee break: Recharge your brain