I took another detour into the world of binary neural networks. In the interests of evaluating dot-product-like functions of two binary vectors, I wanted to assess how the distributions of multiplied bits (equivalent to ANDed bits) relate to the distributions of the individual bits. This post captures the results of this side-trip.
Products of independent Bernoullis
The simplest case is in the light of independence, when $$X \sim Bern(\pi_{X})$$
$$Y \sim Bern(\pi_{Y})$$ and $$P(X,Y) = P(X)P(Y)$$
Let $Z = XY$ be the product of the two bits, equivalent to $Z=X \land Y$. As $Z \in \{0, 1\}$ we can model $Z \sim Bern(\pi_Z)$ for some $\pi_Z$.
The truth table for the four possible outcomes guides us here:
| $X$ | $Y$ | $Z$ | $P(X)$ | $P(Y)$ |
| 0 | 0 | 0 | $1-\pi_X$ | $1-\pi_Y$ |
| 0 | 1 | 0 | $1-\pi_X$ | $\pi_Y$ |
| 1 | 0 | 0 | $\pi_X$ | $1-\pi_Y$ |
| 1 | 1 | 1 | $\pi_X$ | $\pi_Y$ |
$P(Z=1) = P(X=1)P(Y=1) = \pi_X \pi_Y$; a bit of arithmetic also shows that $P(Z=0) = 1 – \pi_X \pi_Y$. With $\pi_X \in [0,1] \land \pi_Y \in [0,1] \implies \pi_X \pi_Y \in [0, 1]$, we see that $$Z \sim Bern(\pi_X \pi_Y)$$
Products of Non-Independent Bernoullis
Suppose that $X \sim Bern(\pi_X)$. A second Bernoulli random variable will be dependent on $X$ only if its single parameter is a function of $X$, so let $Y|X \sim Bern(g(X))$ where $g : \{0, 1\} \to [0,1]$ generates the parameter of $Y$’s distribution.
As there are only two possible outcomes from such a function $g$ we denote them individually as: $$\pi_{Y|X=0} = g(0)$$ and $$\pi_{Y|X=1} = g(1)$$
We again are interested in the distribution of a given $Z = XY = X \land Y$, and again the truth table leads us to a straightforward solution:
| $X$ | $Y$ | $Z$ | $P(X)$ | $P(Y|X)$ |
| 0 | 0 | 0 | $1-\pi_X$ | $1-\pi_{Y|X=0}$ |
| 0 | 1 | 0 | $1-\pi_X$ | $\pi_{Y|X=0}$ |
| 1 | 0 | 0 | $\pi_X$ | $1-\pi_{Y|X=1}$ |
| 1 | 1 | 1 | $\pi_X$ | $\pi_{Y|X=1}$ |
$$P(Z=1) = P(X=1) P(Y=1 | X =1) = \pi_{X} \pi_{Y | X = 1} = \pi_{X} g(1)$$
A bit of arithmetic shows that $$P(Z=0) = 1 – \pi_{X} \pi_{Y|X=1} = 1 – \pi_{X} g(1)$$
With $\pi_{X} \in [0,1] \land \pi_{Y|X=1} \in [0,1] \implies \pi_{X} \pi_{Y | X = 1} \in [0,1]$ we conclude that $$Z \sim Bern(\pi_{X} \pi_{Y | X = 1})$$
The parameter $\pi_{Y|X=0} = g(0)$ for the case where $X=0$ drops out; it is redundant as $Y=1 | X = 0$ is simply another way for $Z$ to equal zero.
In other words, the distribution of $Z$ depends only on the distributions of $X$ and of $Y | X = 1$, not on $Y | X = 0$. The constraint that all events must result in either one or the other of two outcomes allows the behavior of the combined system to be fully characterized with a constant number of parameters, and the distribution of the product / Boolean AND even of dependent Bernoullis is simply yet another Bernoulli.
Leave a Reply