To install click the Add extension button. That's it.

The source code for the WIKI 2 extension is being checked by specialists of the Mozilla Foundation, Google, and Apple. You could also do it yourself at any point in time.

4,5
Kelly Slayton
Congratulations on this excellent venture… what a great idea!
Alexander Grigorievskiy
I use WIKI 2 every day and almost forgot how the original Wikipedia looks like.
Live Statistics
English Articles
Improved in 24 Hours
Languages
Recent
Show all languages
What we do. Every page goes through several hundred of perfecting techniques; in live mode. Quite the same Wikipedia. Just better.
.
Leo
Newton
Brights
Milds

# Multinomial distribution

Parameters ${\displaystyle n>0}$ number of trials (integer)${\displaystyle p_{1},\ldots ,p_{k}}$ event probabilities (${\displaystyle \Sigma p_{i}=1}$) ${\displaystyle x_{i}\in \{0,\dots ,n\},\,\,\,\,i\in \{1,\dots ,k\}}$${\displaystyle \Sigma x_{i}=n\!}$ ${\displaystyle {\frac {n!}{x_{1}!\cdots x_{k}!}}p_{1}^{x_{1}}\cdots p_{k}^{x_{k}}}$ ${\displaystyle \operatorname {E} (X_{i})=np_{i}}$ ${\displaystyle \operatorname {Var} (X_{i})=np_{i}(1-p_{i})}$${\displaystyle \operatorname {Cov} (X_{i},X_{j})=-np_{i}p_{j}~~(i\neq j)}$ ${\displaystyle -\log(n!)-n\sum _{i=1}^{k}p_{i}\log(p_{i})+\sum _{i=1}^{k}\sum _{x_{i}=0}^{n}{\binom {n}{x_{i}}}p_{i}^{x_{i}}(1-p_{i})^{n-x_{i}}\log(x_{i}!)}$ ${\displaystyle {\biggl (}\sum _{i=1}^{k}p_{i}e^{t_{i}}{\biggr )}^{n}}$ ${\displaystyle \left(\sum _{j=1}^{k}p_{j}e^{it_{j}}\right)^{n}}$ where ${\displaystyle i^{2}=-1}$ ${\displaystyle {\biggl (}\sum _{i=1}^{k}p_{i}z_{i}{\biggr )}^{n}{\text{ for }}(z_{1},\ldots ,z_{k})\in \mathbb {C} ^{k}}$

In probability theory, the multinomial distribution is a generalization of the binomial distribution. For example, it models the probability of counts of each side for rolling a k-sided dice n times. For n independent trials each of which leads to a success for exactly one of k categories, with each category having a given fixed success probability, the multinomial distribution gives the probability of any particular combination of numbers of successes for the various categories.

When k is 2 and n is 1, the multinomial distribution is the Bernoulli distribution. When k is 2 and n is bigger than 1, it is the binomial distribution. When k is bigger than 2 and n is 1, it is the categorical distribution.

The Bernoulli distribution models the outcome of a single Bernoulli trial. In other words, it models whether flipping a (possibly biased) coin one time will result in either a success (obtaining a head) or failure (obtaining a tail). The binomial distribution generalizes this to the number of heads from performing n independent flips (Bernoulli trials) of the same coin. The multinomial distribution models the outcome of n experiments, where the outcome of each trial has a categorical distribution, such as rolling a k-sided dice n times.

Let k be a fixed finite number. Mathematically, we have k possible mutually exclusive outcomes, with corresponding probabilities p1, ..., pk, and n independent trials. Since the k outcomes are mutually exclusive and one must occur we have pi ≥ 0 for i = 1, ..., k and ${\displaystyle \sum _{i=1}^{k}p_{i}=1}$. Then if the random variables Xi indicate the number of times outcome number i is observed over the n trials, the vector X = (X1, ..., Xk) follows a multinomial distribution with parameters n and p, where p = (p1, ..., pk). While the trials are independent, their outcomes X are dependent because they must be summed to n.

In some fields such as natural language processing, categorical and multinomial distributions are synonymous and it is common to speak of a multinomial distribution when a categorical distribution is actually meant. This stems from the fact that it is sometimes convenient to express the outcome of a categorical distribution as a "1-of-K" vector (a vector with one element containing a 1 and all other elements containing a 0) rather than as an integer in the range ${\displaystyle 1\dots K}$; in this form, a categorical distribution is equivalent to a multinomial distribution over a single trial.

• 1/5
Views:
87 188
13 379
6 334
3 741
5 959
• ✪ Introduction to the Multinomial Distribution
• ✪ Multinomial Distributions: Lesson (Basic Probability and Statistics Concepts)
• ✪ Multinomial Distributions: Examples (Basic Probability and Statistics Concepts)
• ✪ The Multinomial Distribution Part 1
• ✪ statistics - multinomial distribution - finding probability - example 1

#### Transcription

Let's continue our look at some discrete probability distributions with an introduction to the multinomial distribution. The multinomial distribution is a generalization of the binomial distribution. In the binomial distribution there are only two possible outcomes on any one individual trial, and we labeled those success and failure. In the multinomial distribution, the number of possible outcomes on any one given trial is allowed to be greater than 2. Let's take a look at an example. This is approximately the distribution of blood types in the United States. And suppose we wanted to know the answer to this question: In a random sample of 10 Americans what is the probability 6 have blood type O, 2 have type A, 1 has type B, and 1 has type AB? When any one individual person is sampled, they're going to have one of these four blood types, according to these probabilities. And we're going to be able to answer this question using the multinomial distribution. Suppose we have n independent trials, and each trial results in 1 of k mutually exclusive outcomes, and these k outcomes are exhaustive, so one of them is going to occur. On any single trial these k outcomes comes occur with probabilities p_1 through p_k. And since these outcomes are mutually exclusive and exhaustive, then they must sum to 1. We also need these probabilities to stay constant from trial to trial. We're going to let the random variable X_i represent the number of occurrences of outcome of i. And i is going to take on the values 1 through k, representing those k possible outcomes on any one individual trial. So we're going to have k random variables, representing a count for each of those possible outcomes. Then the probability the random variable X_1 takes on the value little x_1 and all the way up through the random variable X_k taking on the value little x_k is equal to what we have here. Over here on this side we have p_1, the probability of outcome 1 on any one individual trial, raised to the number of times that outcome 1 happens, and all the way up to here, which is the probability of outcome k occurring on any one individual trial, raised to the number of times we need outcome k to occur. And so what we have here is the probability of any one specific ordering of x_1 occurrences of outcome 1 and x_2 occurrences of outcome 2, all the way up through x_k occurrences of outcome k, and what we have over here is the number of possible orderings that give us x_1 occurrences of outcome 1, all the way up through x_k occurrences of outcome k. And so these multiplied together give us the probability of this happening. We really should list out the possible values of X here. The random variable X_1 can take on the possible values 0,1, 2, all the way up through n. And the same is true of X_2 and all the way up through X_k. So this is true for x_i equalling 0,1, all the way up though n. But we know that n things must happen in total, so the sum of all those individual occurrences, or i equalling 1 through k must equal n. And if we think about this a little bit, any one of these random variables, when viewed individually, it will have a binomial distribution. And if you remember our mean and variance for the binomial distribution, we can say that the expectation of X_i is going to be equal to n times p_i. And the variance of the random variable X_i is going to be equal to np_i times (1-p_i). And you might remember that from our discussion of the mean and variance the binomial distribution. Let's return to our example. In a random sample of 10 Americans, what is the probability 6 have blood type O, 2 have type A, 1 has type B, and 1 has type AB? Well, we've got a random sample here, so knowing one person's blood type is going to tell us nothing about the next person's blood type. So that independence assumption is pretty reasonable here. On any one individual trial, we are going to get one of these 4 blood types. and these probabilities are staying constant from trial to trial. So the multinomial distribution is reasonable here. And we want to find out the probability that the random variable X_1, which is representing the number of people in our sample that have blood type O, the probability that takes on the value 6, and our random variable X_2, representing the number with type A, that takes on the value 2, and X_3 takes on the value 1, and X_4 takes on the value 1. And this is going to be equal to n factorial, we've got a sample size 10, so 10! over x_1 factorial, that's just the number with type O and that's 6 factorial, times 2 factorial times, 1 factorial, times 1 factorial, and now it's time for these probabilities. the probability blood type O happens on any one individual person is 0.44. And we need that to happen 6 times, so we're going to raise that to the sixth power. And then we're going to multiply that by the probability of blood type A, 0.42, squared, because we needed that to happen twice, And then we are multiplying that by 0.10, blood type B, raised to the first power, we need that happen once, and then multiplying that by 0.04 raised to the first power. And if we calculate that, we would see that that is 0.01290, when rounded to five decimal places. Let's look at another example here. An urn contains eight red balls, three yellow balls, and nine white balls. Six balls are randomly selected with replacement. What is the probability 2 are red, 1 is yellow, and 3 are white? This "with replacement' is an important notion here. If we are putting the ball back in and shaking it all up and then randomly selecting again, then the individual trials are independent, and the probability of getting a red ball, or a yellow ball, or white ball, those probabilities are staying constant through the different trials. And so the conditions for our multinomial distribution are satisfied here. And we're interested here in the probability that the random variable X_1, which I'm going to let represent the number red balls, the probability the random variable X_1 takes on the value 2, And X_2, the number of yellow balls, takes on the value 1, and X_3, the number of white balls, takes on the value 3. And this is going to be equal to n factorial, so six factorial, over 2 factorial, the number red balls, times 1 factorial, the number of yellow balls, times 3 factorial, the number of white balls. And now to the probabilities. Well we have 8 red balls, and there is 8+3+9, 20 balls in total. So the probability of getting 1 red on any one individual trial is going to be the eight red balls out of the 20 total, and we need that to happen twice, so we're gonna square that. And then we're going to multiply that by the probability of getting a yellow ball on any one individual trial, which is 3 out of 20, raised to the first power, because we need that to happen once. And then times 9/20, getting a white ball on an individual trial, cubed, because that's got to happen three times. and that works out to 0.13122, to 5 decimal places. Had the sampling been done without replacement, then the trials would no longer be independent, and the conditions of the multinomial distribution would no longer be satisfied. We would have to use something called the multivariate hypergeometric distribution to calculate the probability in this case. And, what the heck, let's run through a quick example of that. Here we've got the same problem we just looked at, except I've changed with replacement to without replacement. And there's a fundamental difference there. When I say without replacement, what that means is, if we pull out a red ball, we're putting it aside and it doesn't go back in, and then we randomly select another ball. So if we draw a red ball out on the first trial, say, it's going to be less likely to get a red ball on the second trial, So those trials are no longer independent. We've still got our random variables X_1, X_2 and X_3. And we still want to know the probability that the random variable X_1, the number of red balls, takes on the value 2. And X_2, the number of yellow balls, takes on the value 1, and X_3, the white balls, takes on the value 3. And we're going to do this through the multivariate hypergeometric distribution. On the bottom, we're going to have the number of possible samples. And we are drawing six balls from a total of 20, there are 20 balls altogether. and from those 20 we are choosing 6, so the bottom is going to be the total number of possible samples. In the top, we're going to have the total number of samples that get us what we want. The total number of samples that get 2 red and 1 yellow and 3 white. And for that we're going to say, well we need to pick from those 8 red balls we need to choose 2, so 8 choose 2. And from the 3 yellow balls we need to pick 1, so times 3 choose 1. And from the 9 white balls, we need to pick 3. 8 choose 2, times 3 choose 1, times 9 choose 3, all divided by 20 choose 6. This is the number of ways of getting what we want, over the total number of possible samples of size 6 that can be chosen from 20. And this, if we work this out, to 5 decimal places is 0.18204. Note that's a little bit different from the probability we calculated when it was with replacement

## Specification

### Probability mass function

Suppose one does an experiment of extracting n balls of k different colors from a bag, replacing the extracted ball after each draw. Balls from the same color are equivalent. Denote the variable which is the number of extracted balls of color i (i = 1, ..., k) as Xi, and denote as pi the probability that a given extraction will be in color i. The probability mass function of this multinomial distribution is:

{\displaystyle {\begin{aligned}f(x_{1},\ldots ,x_{k};n,p_{1},\ldots ,p_{k})&{}=\Pr(X_{1}=x_{1}{\text{ and }}\dots {\text{ and }}X_{k}=x_{k})\\&{}={\begin{cases}{\displaystyle {n! \over x_{1}!\cdots x_{k}!}p_{1}^{x_{1}}\times \cdots \times p_{k}^{x_{k}}},\quad &{\text{when }}\sum _{i=1}^{k}x_{i}=n\\\\0&{\text{otherwise,}}\end{cases}}\end{aligned}}}

for non-negative integers x1, ..., xk.

The probability mass function can be expressed using the gamma function as:

${\displaystyle f(x_{1},\dots ,x_{k};p_{1},\ldots ,p_{k})={\frac {\Gamma (\sum _{i}x_{i}+1)}{\prod _{i}\Gamma (x_{i}+1)}}\prod _{i=1}^{k}p_{i}^{x_{i}}.}$

This form shows its resemblance to the Dirichlet distribution, which is its conjugate prior.

## Visualization

### As slices of generalized Pascal's triangle

Just like one can interpret the binomial distribution as (normalized) one-dimensional (1D) slices of Pascal's triangle, so too can one interpret the multinomial distribution as 2D (triangular) slices of Pascal's pyramid, or 3D/4D/+ (pyramid-shaped) slices of higher-dimensional analogs of Pascal's triangle. This reveals an interpretation of the range of the distribution: discretized equilaterial "pyramids" in arbitrary dimension—i.e. a simplex with a grid.

### As polynomial coefficients

Similarly, just like one can interpret the binomial distribution as the polynomial coefficients of ${\displaystyle (p+(1-p))^{n}}$ when expanded, one can interpret the multinomial distribution as the coefficients of ${\displaystyle (p_{1}+p_{2}+p_{3}+\cdots +p_{k})^{n}}$ when expanded. (Note that just like the binomial distribution, the coefficients must sum to 1.) This is the origin of the name "multinomial distribution".

## Properties

The expected number of times the outcome i was observed over n trials is

${\displaystyle \operatorname {E} (X_{i})=np_{i}.\,}$

The covariance matrix is as follows. Each diagonal entry is the variance of a binomially distributed random variable, and is therefore

${\displaystyle \operatorname {Var} (X_{i})=np_{i}(1-p_{i}).\,}$

The off-diagonal entries are the covariances:

${\displaystyle \operatorname {Cov} (X_{i},X_{j})=-np_{i}p_{j}\,}$

for i, j distinct.

All covariances are negative because for fixed n, an increase in one component of a multinomial vector requires a decrease in another component.

When these expressions are combined into a matrix with i, j element ${\displaystyle \operatorname {cov} (X_{i},X_{j}),}$ the result is a k × k positive-semidefinite covariance matrix of rank k − 1. In the special case where k = n and where the pi are all equal, the covariance matrix is the centering matrix.

The entries of the corresponding correlation matrix are

${\displaystyle \rho (X_{i},X_{i})=1.}$
${\displaystyle \rho (X_{i},X_{j})={\frac {\operatorname {Cov} (X_{i},X_{j})}{\sqrt {\operatorname {Var} (X_{i})\operatorname {Var} (X_{j})}}}={\frac {-p_{i}p_{j}}{\sqrt {p_{i}(1-p_{i})p_{j}(1-p_{j})}}}=-{\sqrt {\frac {p_{i}p_{j}}{(1-p_{i})(1-p_{j})}}}.}$

Note that the sample size drops out of this expression.

Each of the k components separately has a binomial distribution with parameters n and pi, for the appropriate value of the subscript i.

The support of the multinomial distribution is the set

${\displaystyle \{(n_{1},\dots ,n_{k})\in \mathbb {N} ^{k}\mid n_{1}+\cdots +n_{k}=n\}.\,}$

Its number of elements is

${\displaystyle {n+k-1 \choose k-1}.}$

### Matrix notation

In matrix notation,

${\displaystyle \operatorname {E} (\mathbf {X} )=n\mathbf {p} ,\,}$

and

${\displaystyle \operatorname {Var} (\mathbf {X} )=n\lbrace \operatorname {diag} (\mathbf {p} )-\mathbf {p} \mathbf {p} ^{\rm {T}}\rbrace ,\,}$

with pT = the row vector transpose of the column vector p.

## Example

Suppose that in a three-way election for a large country, candidate A received 20% of the votes, candidate B received 30% of the votes, and candidate C received 50% of the votes. If six voters are selected randomly, what is the probability that there will be exactly one supporter for candidate A, two supporters for candidate B and three supporters for candidate C in the sample?

Note: Since we’re assuming that the voting population is large, it is reasonable and permissible to think of the probabilities as unchanging once a voter is selected for the sample. Technically speaking this is sampling without replacement, so the correct distribution is the multivariate hypergeometric distribution, but the distributions converge as the population grows large.

${\displaystyle \Pr(A=1,B=2,C=3)={\frac {6!}{1!2!3!}}(0.2^{1})(0.3^{2})(0.5^{3})=0.135}$

## Sampling from a multinomial distribution

First, reorder the parameters ${\displaystyle p_{1},\ldots ,p_{k}}$ such that they are sorted in descending order (this is only to speed up computation and not strictly necessary). Now, for each trial, draw an auxiliary variable X from a uniform (0, 1) distribution. The resulting outcome is the component

${\displaystyle j=\min \left\{j'\in \{1,\dots ,k\}:\left(\sum _{i=1}^{j'}p_{i}\right)-X\geq 0\right\}.}$

{Xj = 1, Xk = 0 for k ≠ j } is one observation from the multinomial distribution with ${\displaystyle p_{1},\ldots ,p_{k}}$ and n = 1. A sum of independent repetitions of this experiment is an observation from a multinomial distribution with n equal to the number of such repetitions.

## To simulate from a multinomial distribution

Various methods may be used to simulate from a multinomial distribution. A very simple solution is to use a uniform pseudo-random number generator on (0,1). First, we divide the (0,1) interval in k subintervals equal in length to the probabilities of the k categories. Then, we generate n independent pseudo-random numbers to determine in which of the k intervals they occur and count the number of occurrences in each interval.

Example

If we have:

 Categories 1 2 3 4 5 6 Probabilities 0.15 0.2 0.3 0.16 0.12 0.07 Superior limits of subintervals 0.15 0.35 0.65 0.81 0.93 1

Then, with a software like Excel, we may use the following recipe:

 Cells : Ai Bi Ci ... Gi Formulae : Rand() =If($Ai<0.15;1;0) =If(And($Ai>=0.15;$Ai<0.35);1;0) ... =If($Ai>=0.93;1;0)

After that, we will use functions such as SumIf to accumulate the observed results by category and to calculate the estimated covariance matrix for each simulated sample.

Another way is to use a discrete random number generator. In that case, the categories must be labeled or relabeled with numeric values.

In the two cases, the result is a multinomial distribution with k categories. This is equivalent, with a continuous random distribution, to simulate k independent standardized normal distributions, or a multinormal distribution N(0,I) having k components identically distributed and statistically independent.

Since the counts of all categories have to sum to the number of trials, the counts of the categories are always negatively correlated.[1]

## References

### Citations

1. ^ "1.7 - The Multinomial Distribution | STAT 504". onlinecourses.science.psu.edu. Retrieved 2016-09-11.