Probability

PMF

PMF or probability mass function or probability law or probability discribuion of discrete random variable is a function that for given number give probability of that value.

To denote PMF used notations:

PMF(X = x) = P(X = x) = p_X(x) = P(ω ∈ Ω : X(ω) = x)

PMF(a ≤ X ≤ b) = P(a ≤ X ≤ b) = ∑_{a ≤ x ≤ b} P(X = x)

p_X(x) ≥ 0

∑_x p_X(x) = 1

where X is a random variable on space Ω of outcomes which mapped to real number via X(ω).

Expected value

Expected value of PMF is:

E[X] = Σ_{ω ∈ Ω}Χ(x)*p(ω) = Σ_xx*p_X(x)

We write a ≤ X ≤ b for ∀ω ∈ Ωa ≤ X(ω) ≤ b.

If X ≥ 0 then E[X] ≥ 0.

if a ≤ X ≤ b then a ≤ E[X] ≤ b.

If Y = g(X) (∀ω ∈ ΩY(ω) = g(X(ω))) then:

E[Y] = Σ_xg(x)*p_X(x)

Proof TODO:

E[Y] = Σ_yy*p_Y(y)

= Σ_{y ∈ ℝ}y*Σ_{ω ∈ Ω : Y(ω) = y}p(ω)

= Σ_{y ∈ ℝ}y*Σ_{ω ∈ Ω : g(X(ω)) = y}p(ω)

= Σ_{y ∈ ℝ}y*Σ_{x ∈ ℝ : g(x) = y}Σ_{ω ∈ Ω : X(ω) = x}p(ω)

= Σ_{y ∈ ℝ}y*Σ_{x ∈ ℝ : g(x) = y}p_X(x)

= Σ_{y ∈ ℝ}Σ_{x ∈ ℝ : g(x) = y}y*p_X(x)

= Σ_{x ∈ ℝ}Σ_{y ∈ ℝ : g(x) = y}y*p_X(x)

= Σ_xg(x)*p_X(x)

E[a*X + b] = a*E[X] + b

Variance

Variance is a:

var[X] = E[(X − E[X])²] = E[X²] − (E[X])²

Standard deviation is a:

σ_Χ = sqrt(var[X])

Property:

var(a*X + b) = a²·var[X]

Total probability theorem

Let A_i∩A_j = ∅ for i ≠ j and ∑_i A_i = Ω:

p_X(x) = Σ_iP(A_i)·p_{X|A_i}(x)

https://en.wikipedia.org/wiki/Law_of_total_probability

Conditional PMF on event

Conditional PMF on event is:

p_X|A(x) = P(X = x|A)

E[X|A] = ∑_x x·p_X|A(x)

Total expectation theorem

E[X] = Σ_i P(A_i)·E[X|A_i]

To prove theorem just multiply total probability theorem by x.

Joint PMF

Joint PMF of random variables X₁, ..., X_n is:

p_{X₁, ..., X_n}(x₁, ..., x_n) = P(AND_{x₁, ..., x_n} : X_i = x_i)

Properties:

E[X + Y] = E[X] + E[Y]

Conditional joint PMF

Conditional joint PMF is:

p_X|Y(x|y) = P(X = x|Y = y) = P(X = x&Y = y) ⁄ P(Y = y)

So:

p_X, Y(x, y) = p_Y(y)·p_X|Y(x|y) = p_X(x)·p_Y|X(y|x)

p_X, Y, Z(x, y, z) = p_Y(y)·p_Z|Y(z|y)·p_X|Y, Z(x|y, z)

∑_x, y p_X, Y|Z(x, y|z) = 1

Conditional expectation of joint PMF

Conditional expectation of joint PMF is:

E[X|Y = y] = ∑_x x·p_X|Y(x|y)

E[g(X)|Y = y] = ∑_x g(x)·p_X|Y(x|y)

Total probability theorem for joint PMF

p_X(x) = ∑_y p_Y(y)·p_X|Y(x|y)

Total expectation theorem for joint PMF

E[X] = ∑_y p_Y(y)·E[X|Y = y]

Proof:

∑_y p_Y(y)·E[X|Y = y] = ∑_y p_Y(y)·∑_x x·p_X|Y(x|y)

= ∑_y ∑_x p_Y(y)·x·p_X|Y(x|y) = ∑_x ∑_y x·p_Y(y)·p_X|Y(x|y)

= ∑_x x·∑_y p_Y(y)·p_X|Y(x|y) = ∑_x x·p_X(x) = E[X]

Conditional expectation of joint PMF

Conditional expectation of joint PMF is random variable E[X|Y] defined as:

E[X|Y](y) = E[X|Y = y]

Property:

E[g(Y)·X|Y] = g(Y)·E[X|Y]

For invertible funtion h:

E[X|h(Y)] = E[X|Y]

Proof:

E[X|Y = y] = E[X|h(Y) = h(y)]

Law of Iterated Expectations

E[E[X|Y]] = E[X]

Proof (using total expectation theorem):

E[E[X|Y]] = ∑_y E[X|Y](y) = ∑_y E[X|Y = y] = E[X]

Generalisation of Law of Iterated Expectations:

E[E[X|Y, Z]|Y] = E[X|Y]

Proof, for each y ∈ Y:

E[X|Y = y] = ∑_x x·p_X|Y(x|Y = y) = ∑_x x·p_X, Y(x, y) ⁄ p_Y(y)

= ∑_x x·∑_z p_X, Y, Z(x, y, z) ⁄ p_Y(y)

= ∑_x x·∑_z p_X|Y, Z(x|Y = y, Z = z)·p_Y, Z(y, z) ⁄ p_Y(y)

= ∑_x x·∑_z p_X|Y, Z(x|Y = y, Z = z)·p_Z|Y(z|Y = y)

= ∑_x ∑_z x·p_X|Y, Z(x|Y = y, Z = z)·p_Z|Y(z|Y = y)

= ∑_z ∑_x x·p_X|Y, Z(x|Y = y, Z = z)·p_Z|Y(z|Y = y)

= ∑_z p_Z|Y(z|Y = y)·∑_x x·p_X|Y, Z(x|Y = y, Z = z)

= ∑_z p_Z|Y(z|Y = y)·E[X|Y, Z] = E[E[X|Y, Z]|Y = y]

Conditional variance

Conditional variance of X on Y is r.v.:

var(X|Y)(y) = var(X|Y = y) = E[(X − E[X|Y = y])²|Y = y]

or in another notation:

var(X|Y) = E[X²|Y] − (E[X|Y])²

Law of total variance

By applying expected value by Y on both sides:

E[var(X|Y)] = E[E[X²|Y]] − E[(E[X|Y])²] = E[X²] − E[(E[X|Y])²]

on another hand:

var(E[X|Y]) = E[(E[X|Y])²] − (E[E[X|Y]])² = E[(E[X|Y])²] − (E[X])²

By adding last two expression:

E[var(X|Y)] + var(E[X|Y]) = E[X²] − (E[X])² = var(X)

So:

var(X) = E[var(X|Y)] + var(E[X|Y])

Independence of r.v.

r.v. X and Y is independent if:

∀_x, y : p_X, Y(x, y) = p_X(x)·p_Y(y)

So if two r.v. are independent:

E[X·Y] = E[X]·E[Y]

var(X + Y) = var(X) + var(Y)

Convolution formula

If Z = X + Y and X and Y is independent r.v. then:

p_Z(z) = ∑_x p_X(x)·p_Y(z − x)

Proof:

p_Z(z) = ∑_{x, y : x + y = z} p_Z(z) = ∑_{x, y : x + y = z} P(X = x, Y = z − x)

= ∑_{x, y : x + y = z} P(X = x)·P(Y = z − x) = ∑_x p_X(x)·p_Y(z − x)

Sum of a random number of r.v

Let X_i is independent equally distributed r.v. and let Y = ∑_i = 1..N X_i, where N is r.v. Then:

E[Y|N = n] = n·E[X]

E[Y|N] = N·E[X]

Proof:

E[Y|N = n] = E[∑_i = 1..N X_i|N = n] = E[∑_i = 1..n X_i] = ∑_i = 1..n E[X_i] = n·E[X]

Variance of sum of a random number independent r.v.:

var(∑_i = 1..N X_i|N) = E[N]·var(X) + (E[X])²·var(N)

Proof:

var(Y|N = n) = var[∑_i = 1..N X_i|N = n] = var[∑_i = 1..n X_i] = ∑_i = 1..n var[X_i] = n·var(X)

var(Y) = E[var(Y|N)] + var(E[Y|N]) = E[N]·var(X) + (E[X])²·var(N)

Well known discrete r.v.

Bernoulli random variable

Bernoulli random variable with parameter p is a random variable that have 2 outcomes denoted as 0 and 1 with probabilities:

p_X(0) = 1 − p

p_X(1) = p

This random variable models a trial of experiment that result in success or failure.

Indicator of r.v. event A is function:

I_A = 1 iff A occurs, else 0

P_{I_A} = p(I_A = 1) = p(A)

I_A*I_B = I_A∩B

E[bernoulli(p)] = 0*(1 − p) + 1*p = p

var[bernoulli(p)] = E[bernoulli(p) − E[bernoulli(p)]]

= (0 − p)²·(1 − p) + (1 − p)²·p = p²·(1 − p) + (1 − 2p + p²)·p

= p² − p³ + p − 2·p² + p³ = p·(1 − p)

Discret uniform random variable

Discret uniform random variable is a variable with parameters a and b in sample space x : a ≤ x ≤ b x ∈ ℕ with equal probability of each possible outcome:

p_unif(a, b)(x) = 1 ⁄ (b − a + 1)

E[unif(a, b)] = Σ_{a ≤ x ≤ b}x*1 ⁄ (b − a + 1) = 1 ⁄ (b − a + 1)*Σ_{a ≤ x ≤ b}x

= 1 ⁄ (b − a + 1)*(Σ_{a ≤ x ≤ b}a + Σ_{0 ≤ x ≤ b − a}x)

= 1 ⁄ (b − a + 1)*((b − a + 1)*a + (b − a)*(b − a + 1) ⁄ 2)

= a + (b − a) ⁄ 2 = (b + a) ⁄ 2

var[unif(a, b)] = E[unif²(a, b)] − E²[unif(a, b)]

= ∑_{a ≤ x ≤ b}x² ⁄ (b − a + 1) − (b + a)² ⁄ 4

= 1 ⁄ (b − a + 1)·(∑_{0 ≤ x ≤ b}x² − ∑_{0 ≤ x ≤ a − 1}x²) − (b + a)² ⁄ 4

= 1 ⁄ (b − a + 1)·(b + 3·b² + 2·b³ − (a − 1) + 3·(a − 1)² + 2·(a − 1)³) ⁄ 6 − (b + a)² ⁄ 4

= (2·b² + 2·a·b + b + 2·a² − a) ⁄ 6 − (b + a)² ⁄ 4

= (b − a)·(b − a + 2) ⁄ 12

Note

From Maxima:

sum(i^2,i,0,n), simpsum=true;

         2      3
  n + 3 n  + 2 n
  ---------------
        6

factor(b+3*b^2+2*b^3 - (a-1)-3*(a-1)^2-2*(a-1)^3);

                  2                  2
  (b - a + 1) (2 b  + 2 a b + b + 2 a  - a)

factor((2*b^2 + 2*a*b + b + 2*a^2 - a)/6 - (b+a)^2/4), simp=true;

  (b - a) (2 - a + b)
  -------------------
          12

Binomial random variable

Binomialrandomvariable is a r.v. with parameters n (positive integer) and p from interval (0, 1) and sample space of positive integers from inclusive region [0, n]:

p_{binom(n, p)}(x) = n! ⁄ (x!*(n − x)!)p^xp^n − x

Binomial random variable models a number of success of n independent trails of Bernoulli experimants.

E[binom(n, p)] = E[∑_{1 ≤ x ≤ n}bernoulli(p)] = ∑_{1 ≤ x ≤ n}E[bernoulli(p)] = n·p

var[binom(n, p)] = var[∑_{1 ≤ x ≤ n}bernoulli(p)] = ∑_{1 ≤ x ≤ n}var[bernoulli(p)] = n·p·(1 − p)

Geometric random variable

Geometric random variable is a r.v. with parameter p from half open interval (0, 1], sample space is all positive numbers:

p_geom(p)(x) = p(1 − p)⁽x − 1)

This random variable models number of tosses of biased coin until first success.

E[geom(p)] = ∑_{x = 1..∞}x·p·(1 − p)⁽x − 1)

= p·∑_{x = 1..∞}x·(1 − p)⁽x − 1)

= p ⁄ (1 − p)·∑_{x = 0..∞}x·(1 − p)^x

= p ⁄ (1 − p)·(1 − p) ⁄ (1 − p − 1)² = p ⁄ p² = 1 ⁄ p

Note

Maxima calculation:

load("simplify_sum");
simplify_sum(sum(k * x^k, k, 0, inf));
  Is abs(x) - 1 positive, negative or zero?
  negative;
  Is x positive, negative or zero?
  positive;
  Is x - 1 positive, negative or zero?
  negative;
       x
  ------------
   2
  x  - 2 x + 1

E[(geom(p))²] = ∑_{x = 1..∞}x²·p·(1 − p)⁽x − 1)

= p·∑_{x = 1..∞}x²·(1 − p)⁽x − 1)

= p ⁄ (1 − p)·∑_{x = 0..∞}x²·(1 − p)^x

= p ⁄ (1 − p)·(1 − p)·(1 − p + 1) ⁄ (1 − (1 − p))³ = p·(2 − p) ⁄ p³ = (2 − p) ⁄ p²

Note

Maxima calculation:

load("simplify_sum");
(%i3) assume(x>0);
(%o3)                               [x > 0]
(%i4) assume(x<1);
(%o4)                               [x < 1]

(%i8) simplify_sum(sum(k^2 * x^k, k, 0, inf));
                                          2
                                     x + x
(%o8)                        - -------------------
                                3      2
                               x  - 3 x  + 3 x - 1

So:

var(geom(p)) = E[(geom(p))²] − E[geom(p)]² = (2 − p) ⁄ p² − 1 ⁄ p² = (1 − p) ⁄ p²