Conditioning (probability): Difference between revisions
imported>Boris Tsirelson |
imported>Boris Tsirelson |
||
(2 intermediate revisions by the same user not shown) | |||
Line 156: | Line 156: | ||
In the latter two examples the law of total probability is irrelevant, since only a single event (the condition) is given. In contrast, in the example [[#DPI5|above]] the law of total probability [[#DPC8|applies]], since the event <math>X=0.5</math> is included into a family of events <math>X=x</math> where <math>x</math> runs over <math>(-1,1),</math> and these events are a partition of the probability space. | In the latter two examples the law of total probability is irrelevant, since only a single event (the condition) is given. In contrast, in the example [[#DPI5|above]] the law of total probability [[#DPC8|applies]], since the event <math>X=0.5</math> is included into a family of events <math>X=x</math> where <math>x</math> runs over <math>(-1,1),</math> and these events are a partition of the probability space. | ||
In order to avoid paradoxes (such as the [[Borel's paradox]]), the following important distinction should be taken into account. If a given event is of nonzero probability then conditioning on it is well-defined (irrespective of any other events), as was noted [[#EP8|above]]. In contrast, if the given event is of zero probability then conditioning on it is ill-defined unless some additional input is provided. Wrong choice of this additional input leads to wrong conditional probabilities (expectations, distributions). In this sense, "''the concept of a conditional probability with regard to an isolated hypothesis whose probability equals 0 is inadmissible.''" ([[Andrey Kolmogorov|Kolmogorov]]; quoted in <ref name="Pollard-5.5-122" | In order to avoid paradoxes (such as the [[Borel's paradox]]), the following important distinction should be taken into account. If a given event is of nonzero probability then conditioning on it is well-defined (irrespective of any other events), as was noted [[#EP8|above]]. In contrast, if the given event is of zero probability then conditioning on it is ill-defined unless some additional input is provided. Wrong choice of this additional input leads to wrong conditional probabilities (expectations, distributions). In this sense, "''the concept of a conditional probability with regard to an isolated hypothesis whose probability equals 0 is inadmissible.''" ([[Andrey Kolmogorov|Kolmogorov]]; quoted in <ref name="Pollard-5.5-122" />). | ||
The additional input may be (a) a symmetry (invariance group); (b) a sequence of events <math>B_n</math> such that <math>B_n\downarrow B,</math> <math>P(B_n)>0;</math> (c) a partition containing the given event. Measure-theoretic conditioning (below) investigates Case (c), discloses its relation to (b) in general and to (a) when applicable. | The additional input may be (a) a symmetry (invariance group); (b) a sequence of events <math>B_n</math> such that <math>B_n\downarrow B,</math> <math>P(B_n)>0;</math> (c) a partition containing the given event. Measure-theoretic conditioning (below) investigates Case (c), discloses its relation to (b) in general and to (a) when applicable. | ||
Some events of zero probability are beyond the reach of conditioning. An example: let <math>X_n</math> be independent random variables distributed uniformly on <math>(0,1),</math> and <math>B</math> the event "<math>X_n\to0</math> as <math>n\to\infty</math>"; what about <math>P(X_n<0.5|B)?</math> Does it tend to 1, or not? Another example: let <math>X</math> be a random variable distributed uniformly on <math>(0,1),</math> and <math>B</math> the event "<math>X</math> is a rational number"; what about <math>P(X=1/n|B)?</math> | Some events of zero probability are beyond the reach of conditioning. An example: let <math>X_n</math> be independent random variables distributed uniformly on <math>(0,1),</math> and <math>B</math> the event "<math>X_n\to0</math> as <math>n\to\infty</math>"; what about <math>P(X_n<0.5|B)?</math> Does it tend to 1, or not? Another example: let <math>X</math> be a random variable distributed uniformly on <math>(0,1),</math> and <math>B</math> the event "<math>X</math> is a rational number"; what about <math>P(X=1/n|B)?</math> | ||
The only answer is that, once again, "the concept of a conditional probability with regard to an isolated hypothesis whose probability equals 0 is inadmissible." (Kolmogorov, quoted in <ref name="Pollard-5.5-122" | The only answer is that, once again, "the concept of a conditional probability with regard to an isolated hypothesis whose probability equals 0 is inadmissible." (Kolmogorov, quoted in <ref name="Pollard-5.5-122" />). | ||
==Conditioning on the level of measure theory== | ==Conditioning on the level of measure theory== | ||
Line 213: | Line 213: | ||
: <math> \int_0^1 g_2 (f_2(y)) \, \mathrm{d}y = \frac13 \, . </math> | : <math> \int_0^1 g_2 (f_2(y)) \, \mathrm{d}y = \frac13 \, . </math> | ||
However, the Hilbert space approach treats <math>g_2</math> as an equivalence class of functions rather than an individual function. Measurability of <math>g_2</math> is ensured, but continuity (or even [[Riemann integrability]]) is not. The value <math>g_2(0.5)</math> is determined uniquely, since the point 0.5 is an atom of the distribution of <math>X</math>. Other values <math>x</math> are not atoms, thus, corresponding values <math>g_2(x)</math> are not determined uniquely. Once again, "''the concept of a conditional probability with regard to an isolated hypothesis whose probability equals 0 is inadmissible.''" ( | However, the Hilbert space approach treats <math>g_2</math> as an equivalence class of functions rather than an individual function. Measurability of <math>g_2</math> is ensured, but continuity (or even [[Riemann integrability]]) is not. The value <math>g_2(0.5)</math> is determined uniquely, since the point 0.5 is an atom of the distribution of <math>X</math>. Other values <math>x</math> are not atoms, thus, corresponding values <math>g_2(x)</math> are not determined uniquely. Once again, "''the concept of a conditional probability with regard to an isolated hypothesis whose probability equals 0 is inadmissible.''" (Kolmogorov, quoted in <ref name="Pollard-5.5-122" />). | ||
Alternatively, the same function <math>g</math> (be it <math>g_1</math> or <math>g_2</math>) may be defined as the [[Radon-Nikodym derivative]] | Alternatively, the same function <math>g</math> (be it <math>g_1</math> or <math>g_2</math>) may be defined as the [[Radon-Nikodym derivative]] |
Latest revision as of 15:48, 29 June 2009
![](http://s9.addthis.com/button1-share.gif)
Beliefs depend on the available information. This idea is formalized in probability theory by conditioning. Conditional probabilities, conditional expectations and conditional distributions are treated on three levels: discrete probabilities, probability density functions, and measure theory. Conditioning leads to a non-random result if the condition is completely specified; otherwise, if the condition is left random, the result of conditioning is also random.
This article concentrates on interrelations between various kinds of conditioning, shown mostly by examples.
Conditioning on the discrete level
Example. A fair coin is tossed 10 times; the random variable is the number of heads in these 10 tosses, and — the number of heads in the first 3 tosses. In spite of the fact that emerges before it may happen that someone knows but not .
Conditional probability
Given that the conditional probability of the event is More generally,
for otherwise (for ), One may also treat the conditional probability as a random variable, — a function of the random variable , namely,
The expectation of this random variable is equal to the (unconditional) probability,
namely,
which is an instance of the law of total probability
Thus, may be treated as the value of the random variable corresponding to On the other hand, is well-defined irrespective of other possible values of .
Conditional expectation
Given that the conditional expectation of the random variable is More generally,
for (In this example it appears to be a linear function, but in general it is nonlinear.) One may also treat the conditional expectation as a random variable, — a function of the random variable , namely,
The expectation of this random variable is equal to the (unconditional) expectation of ,
namely,
- or simply
which is an instance of the law of total expectation
The random variable is the best predictor of given . That is, it minimizes the mean square error on the class of all random variables of the form This class of random variables remains intact if is replaced, say, with Thus, It does not mean that rather, In particular, More generally, for every function that is one-to-one on the set of all possible values of . The values of are irrelevant; what matters is the partition (denote it α)
of the sample space into disjoint sets (Here are all possible values of .) Given an arbitrary partition of , one may define the random variable Still,
Conditional probability may be treated as a special case of conditional expectation. Namely, if is the indicator of . Therefore the conditional probability also depends on the partition generated by rather than on itself;
On the other hand, conditioning on an event is well-defined, provided that irrespective of any partition that may contain as one of several parts.
Conditional distribution
Given the conditional distribution of is
for It is the hypergeometric distribution or equivalently, The corresponding expectation obtained from the general formula for is nothing but the conditional expectation
Treating as a random distribution (a random vector in the four-dimensional space of all measures on one may take its expectation, getting the unconditional distribution of , — the binomial distribution This fact amounts to the equality
for just the law of total probability.
Conditioning on the level of densities
Example. A point of the sphere is chosen at random according to the uniform distribution on the sphere. The random variables , , are the coordinates of the random point. The joint density of , , does not exist (since the sphere is of zero volume), but the joint density of , exists,
(The density is non-constant because of a non-constant angle between the sphere and the plane.) The density of may be calculated by integration,
surprisingly, the result does not depend on in (-1,1),
which means that is distributed uniformly on The same holds for and (and in fact, for whenever
Conditional probability
Calculation
Given that the conditional probability of the event is the integral of the conditional density,
More generally,
for all and such that (otherwise the denominator vanishes) and (otherwise the conditional probability degenerates to 0 or 1). One may also treat the conditional probability as a random variable, — a function of the random variable , namely,
The expectation of this random variable is equal to the (unconditional) probability,
which is an instance of the law of total probability
Interpretation
The conditional probability cannot be interpreted as since the latter gives 0/0. Accordingly, cannot be interpreted via empirical frequencies, since the exact value has no chance to appear at random, not even once during an infinite sequence of independent trials.
The conditional probability can be interpreted as a limit,
Conditional expectation
The conditional expectation is of little interest; it vanishes just by symmetry. It is more interesting to calculate treating || as a function of , :
More generally,
for One may also treat the conditional expectation as a random variable, — a function of the random variable X, namely,
The expectation of this random variable is equal to the (unconditional) expectation of
namely,
which is an instance of the law of total expectation
The random variable is the best predictor of given . That is, it minimizes the mean square error on the class of all random variables of the form Similarly to the discrete case, for every measurable function that is one-to-one on
Conditional distribution
Given the conditional distribution of , given by the density is the (rescaled) arcsin distribution; its cumulative distribution function is
for all and such that The corresponding expectation of is nothing but the conditional expectation The mixture of these conditional distributions, taken for all (according to the distribution of ) is the unconditional distribution of . This fact amounts to the equalities
the latter being the instance of the law of total probability mentioned above.
What conditioning is not
On the discrete level conditioning is possible only if the condition is of nonzero probability (one cannot divide by zero). On the level of densities, conditioning on is possible even though This success may create the illusion that conditioning is always possible. Regretfully, it is not, for several reasons presented below.
Geometric intuition: caution
The result mentioned above, is geometrically evident in the following sense. The points of the sphere satisfying the condition are a circle of radius on the plane The inequality holds on an arc. The length of the arc is 5/6 of the length of the circle, which is why the conditional probability is equal to 5/6.
This successful geometric explanation may create the illusion that the following question is trivial.
- A point of a given sphere is chosen at random (uniformly). Given that the point lies on a given plane, what is its conditional distribution?
It may seem evident that the conditional distribution must be uniform on the given circle (the intersection of the given sphere and the given plane). Sometimes it really is, but in general it is not. Especially, is distributed uniformly on and independent of the ratio thus, On the other hand, the inequality holds on an arc of the circle (for any given ). The length of the arc is 2/3 of the length of the circle. However, the conditional probability is 3/4, not 2/3. This is a manifestation of the classical Borel paradox[1] [2].
"Appeals to symmetry can be misleading if not formalized as invariance arguments." Pollard[3]
Another example. A random rotation of the three-dimensional space is a rotation by a random angle around a random axis. Geometric intuition suggests that the angle is independent of the axis and distributed uniformly. However, the latter is wrong; small values of the angle are less probable.
The limiting procedure
Given an event of zero probability, the formula is useless, however, one can try for an appropriate sequence of events of nonzero probability such that (that is, and ). One example is given above. Two more examples are Brownian bridge and Brownian excursion.
In the latter two examples the law of total probability is irrelevant, since only a single event (the condition) is given. In contrast, in the example above the law of total probability applies, since the event is included into a family of events where runs over and these events are a partition of the probability space.
In order to avoid paradoxes (such as the Borel's paradox), the following important distinction should be taken into account. If a given event is of nonzero probability then conditioning on it is well-defined (irrespective of any other events), as was noted above. In contrast, if the given event is of zero probability then conditioning on it is ill-defined unless some additional input is provided. Wrong choice of this additional input leads to wrong conditional probabilities (expectations, distributions). In this sense, "the concept of a conditional probability with regard to an isolated hypothesis whose probability equals 0 is inadmissible." (Kolmogorov; quoted in [3]).
The additional input may be (a) a symmetry (invariance group); (b) a sequence of events such that (c) a partition containing the given event. Measure-theoretic conditioning (below) investigates Case (c), discloses its relation to (b) in general and to (a) when applicable.
Some events of zero probability are beyond the reach of conditioning. An example: let be independent random variables distributed uniformly on and the event " as "; what about Does it tend to 1, or not? Another example: let be a random variable distributed uniformly on and the event " is a rational number"; what about The only answer is that, once again, "the concept of a conditional probability with regard to an isolated hypothesis whose probability equals 0 is inadmissible." (Kolmogorov, quoted in [3]).
Conditioning on the level of measure theory
Example. Let be a random variable distributed uniformly on and where is a given function. Two cases are treated below: and where is the continuous piecewise-linear function
and is the everywhere continuous but nowhere differentiable Weierstrass function.
Geometric intuition: caution
In the case given two values of are possible, 0.25 and 0.5. It may seem evident that both values are of conditional probability 0.5 just because one point is congruent to another point. However, this is an illusion; see below.
Conditional probability
The conditional probability may be defined as the best predictor of the indicator
given X. That is, it minimizes the mean square error on the class of all random variables of the form
In the case the corresponding function may be calculated explicitly,[4]
Alternatively, the limiting procedure may be used,
giving the same result.
Thus, The expectation of this random variable is equal to the (unconditional) probability, namely,
which is an instance of the law of total probability
In the case the corresponding function probably cannot be calculated explicitly. Nevertheless it exists, and can be computed numerically. Indeed, the space of all square integrable random variables is a Hilbert space; the indicator is a vector of this space; and random variables of the form are a (closed, linear) subspace. The orthogonal projection of this vector to this subspace is well-defined. It can be computed numerically, using finite-dimensional approximations to the infinite-dimensional Hilbert space.
Once again, the expectation of the random variable is equal to the (unconditional) probability, namely,
However, the Hilbert space approach treats as an equivalence class of functions rather than an individual function. Measurability of is ensured, but continuity (or even Riemann integrability) is not. The value is determined uniquely, since the point 0.5 is an atom of the distribution of . Other values are not atoms, thus, corresponding values are not determined uniquely. Once again, "the concept of a conditional probability with regard to an isolated hypothesis whose probability equals 0 is inadmissible." (Kolmogorov, quoted in [3]).
Alternatively, the same function (be it or ) may be defined as the Radon-Nikodym derivative
where measures μ, ν are defined by
for all Borel sets That is, μ is the (unconditional) distribution of , while ν is one third of its conditional distribution,
Both approaches (via the Hilbert space, and via the Radon-Nikodym derivative) treat as an equivalence class of functions; two functions and are treated as equivalent, if almost surely. Accordingly, the conditional probability is treated as an equivalence class of random variables; as usual, two random variables are treated as equivalent if they are equal almost surely.
Conditional expectation
The conditional expectation may be defined as the best predictor of given . That is, it minimizes the mean square error on the class of all random variables of the form
In the case the corresponding function may be calculated explicitly,[5]
Alternatively, the limiting procedure may be used,
giving the same result.
Thus, The expectation of this random variable is equal to the (unconditional) expectation, namely,
which is an instance of the law of total expectation
In the case the corresponding function probably cannot be calculated explicitly. Nevertheless it exists, and can be computed numerically in the same way as above, — as the orthogonal projection in the Hilbert space. The law of total expectation holds, since the projection cannot change the scalar product by the constant function 1 belonging to the subspace.
Alternatively, the same function (be it or ) may be defined as the Radon-Nikodym derivative
where measures are defined by
for all Borel sets Here is the restricted expectation, not to be confused with the conditional expectation
Conditional distribution
In the case the conditional cumulative distribution function may be calculated explicitly, similarly to The limiting procedure gives
which cannot be correct, since a cumulative distribution function must be right-continuous!
This paradoxical result is explained by measure theory as follows. For a given the corresponding is well-defined (via the Hilbert space or the Radon-Nikodym derivative) as an equivalence class of functions (of ). Treated as a function of for a given it is ill-defined unless some additional input is provided. Namely, a function (of ) must be chosen within every (or at least almost every) equivalence class. Wrong choice leads to wrong conditional cumulative distribution functions.
A right choice can be made as follows. First, is considered for rational numbers only. (Any other dense countable set may be used equally well.) Thus, only a countable set of equivalence classes is used; all choices of functions within these classes are mutually equivalent, and the corresponding function of rational is well-defined (for almost every ). Second, the function is extended from rational numbers to real numbers by right continuity.
In general the conditional distribution is defined for almost all (according to the distribution of ), but sometimes the result is continuous in , in which case individual values are acceptable. In the considered example this is the case; the correct result for
shows that the conditional distribution of given consists of two atoms, at 0.25 and 0.5, of probabilities 1/3 and 2/3 respectively.
Similarly, the conditional distribution may be calculated for all in or
The value is an atom of the distribution of , thus, the corresponding conditional distribution is well-defined and may be calculated by elementary means (the denominator does not vanish); the conditional distribution of given is uniform on Measure theory leads to the same result.
The mixture of all conditional distributions is the (unconditional) distribution of .
The conditional expectation is nothing but the expectation with respect to the conditional distribution.
In the case the corresponding probably cannot be calculated explicitly. For a given it is well-defined (via the Hilbert space or the Radon-Nikodym derivative) as an equivalence class of functions (of ). The right choice of functions within these equivalence classes may be made as above; it leads to correct conditional cumulative distribution functions, thus, conditional distributions. In general, conditional distributions need not be atomic or absolutely continuous (nor mixtures of both types). Probably, in the considered example they are singular (like the Cantor distribution).
Once again, the mixture of all conditional distributions is the (unconditional) distribution, and the conditional expectation is the expectation with respect to the conditional distribution.
Notes
- ↑ Pollard 2002, Sect. 5.5, Example 17 on page 122
- ↑ Durrett 1996, Sect. 4.1(a), Example 1.6 on page 224
- ↑ 3.0 3.1 3.2 3.3 Pollard 2002, Sect. 5.5, page 122
- ↑
Proof:
- ↑ Proof: it remains to note that is minimal at and is minimal at
References
- Durrett, Richard (1996), Probability: theory and examples (Second ed.)
- Pollard, David (2002), A user's guide to measure theoretic probability, Cambridge University Press