Use our Shannon Entropy Calculator to measure the average amount of information or uncertainty in a set of events or a probability distribution. This essential tool in information theory helps analyze the predictability and randomness of data, from communication systems and signal processing to data science and machine learning applications. Simply input event frequencies or probabilities to get instant results.
Formula:
The Shannon Entropy, denoted as H(X), is a fundamental concept in information theory that quantifies the expected value of the information contained in a message or a probability distribution. It is calculated using the formula:
H(X) = - Σ p(x) ⋅ log₂(p(x))
Where:
- p(x): Represents the probability of a specific outcome or event 'x' occurring from the set of all possible outcomes.
- log₂(p(x)): Is the base-2 logarithm of the probability p(x). Using base-2 logarithm means the entropy is measured in 'bits' (binary digits).
- Σ: Denotes the summation over all possible discrete outcomes (x) of the random variable X.
The formula essentially sums the information content of each outcome, weighted by its probability. A higher entropy value indicates greater uncertainty or randomness in the system, while lower entropy suggests more predictability.