3D Galton board, visualizing PDFs, Pascal’s triangle, adding/multiplying/convolving Gaussians, why pi is in the Gaussian, Fourier/convolutions, random walks, Kernel Density Estimation, AND a (less popular) proof of the CLT!
Extending the Galton Board
Have you seen those boards with falling balls and a bunch of pegs that create a defined distribution? That each ball follows an unpredictable path, but they have a predictable overall distribution? That the Gaussian is the sum of a bunch of small errors?
This is one of those statistical facts that is just straight up beautiful. Today I endeavor to extend this idea.
I have been working a lot recently in stats. In math and stats, there are a lot of formulas you just have to take as a given. You may not be able to get intuition for them, you just gotta roll through the calculations. But a good visualization once in a while really helps!
In this light, recently, I have had to use the fact that adding or multiplying two Gaussians gives another Gaussian. I’ve seen this a lot, and just had to roll with it. Along the way, a lot will be encountered.
I’ve already extended the Galton board to visualize complex stats facts. I like to use these boards to visualize facts a lot, like how Steve Mould uses 2D mechanisms. This previous post about why entropy is maximized in the Gaussian, I would say, is a good read for before this article!
A Galton board is a simple device. There are a bunch of pegs in a triangular pattern. When you drop a lot of balls in the middle, it creates a distribution.
It looks like a bell curve — and it is, but with an asterix. To see why, look at a single peg. Without loss of generality, each peg can bounce either left or right with some probability. It is 1/2 probability in reality, but that does not matter.
Since we have a Bernoulli trial — a coin flip — we have a binomial distribution. No worries though — the Central Limit Theorem tells us that this will approximate the normal distribution when we have enough balls and rows.
Pretty cool! If you widen the rows you can increase the variance of the distribution (it’s spread). We’ve already touched on some important and common stats ideas.
One last important idea — Pascal’s triangle.
You’ve seen this before — each number is the sum of the two above it. The rows approach a Gaussian! The reason why is very easy to see from the Galton board. Firstly — since Pascal’s triangle defines binomial expansion, binomial distribution will of course pop up. Secondly — imagine the path of a single ball bouncing down. Look at the first 2. Only the two pegs above it can touch it.
Now we know how to generate the normal probability density function (PDF). Great! How do we show that adding them results in another Gaussian??
Well, let’s think of our initial simple normal distribution. We could set that top peak to be whatever initial value we want (the mean). Then each bounce left will decrease the current value stored in the ball bouncing and each bounce right will increase it. At the end, we can analyze the distribution of all the values the balls took on.
So this is a Gaussian random variable. When you measure from this random variable, you are basically taking the path of one ball. (This is an entropic sounding explanation, and I already covered it briefly).
So if you take another (INDEPENDENT) Gaussian random variable and add it to another Gaussian random variable, it seems natural that the value it most “wants to be” is the sum of the means of your two variables. So we will continue just presuming that the center is the means added.
Now we are ready for our Galton board. The center is the sum of means. On top is the first random variable, and the bins directly drop onto another Galton board for the second variable.
From this, we can see why variances add as well. For the board, variance = number of rows / 4, for a 50/50 bounce. So it is just number of rows adding!
Now we need to add another dimension. We need this for multiplication!
Now, the 3D Galton board has been explored by this physics grad student. Just image a Pyramid instead of a triangle for the 3D Galton board.
These are images of Pascal’s pyramid. This is the structure the pegs would take for a 3D Galton board. [Analogous to Pascal’s triangle, the bottom layer is a 2D Gaussian]
Your intuition might say this would generate a cone shape. Indeed,
This is the 2D Gaussian. The YouTuber, radius cubed, did computational simulations of this 3D board and found the 2D Gaussian shape. With the regular Galton Board, each ball can go either left or right. With the 3D Galton board, the ball could go in any angle.
If the ball can bounce in any angle, and any angle is just as likely as any other, it is not hard to see the circular nature of the Gaussian (and why pi may appear).
Now, one can find the PDF of the Gaussian from Pascal’s Triangle by using Stirling’s approximation of the factorial on the binomial expansion. And that would also explain the occurrence of pi in the distribution. But hey, since we’re on the topic of Galton boards…
Imagine, for each point on Pascal’s pyramid:
Imagine a circle of radius 1 (WLOG, since the random walk is what matters) that each ball dropped could bounce onto the radius of. Now, if you overlay all the circles onto each other, simple Kernel Density Estimation (KDE) could give you your Gaussian! (Imagine all the points laid down, with a circular kernel). In this regard, the number of circles over an area gives the density at a given point in space.
Get Gio Bean’s stories in your inbox
Join Medium for free to get updates from this writer.
Each circle can be represented as e^(ix). Now, each bounce can be represented as a random vector. And the path of a ball is a vector sum. From this, we can see most paths will be cancelled out and end up near the center. You know what time it is.
Convolutions come up a lot in stats. Usually, when they do, they come up alongside the Fourier transform. This is because it turns convolutions into multiplications. KDE, which matches our intuition, is a convolution, so this is a natural step. This will lead us now into a more niche proof of the central limit theorem — the characteristic function proof. Essentially, we will use the Fourier transform to prove this Galton board does lead to the 2D Gaussian.
[The KDE is a convolution of a series of data points with a chosen kernel function. I recommend this hands-on exploration of KDE to get a visual for this.]
First, we have to know our distribution of points. Let’s quickly look back at 2D. An individual path, S_n, would follow
Because each bounce can go left or right. This is a random walk in one dimension. Yes, this leads to the CLT and is another way to see the Galton board’s normal distribution. See this Wolfram article. But think of the two points — L/R — as the equivalent of a circle on a line.
Equivalently, for our 3D Galton Board:
The sum of random unit vectors. A random walk in 2D (for the board at least. This is the continuous version).
SIDENOTE: This actually gives some intuition for why a random walk in 2D and 1D always returns to the origin, but a random walk in 3D doesn’t. Specifically — it is trivial to pick a random point on a circle, or either left or right for the 2D Galton Board. A 4D Galton Board would use a sphere then, like how the 3D one uses a circle and the 2D one uses a degenerate circle slice. But you can’t just pick random latitude and longitude points for points a sphere! This makes the points cluster up at the poles. Remember from Calc 3 — the spherical coordinates Jacobian? — sin(Phi)*dTheta*dPhi. This will make the random walk biased enough to not (necessarily) return to the origin.
That makes a lot more sense than “the extra dimension gives too many options to pick from,” which is the explanation I’ve always heard. Perhaps I may need to look more into the 4D Galton board in the future..
Now we are ready to see how the Gaussian emerges. The characteristic function phi(k) for a unit step in a random direction is the Bessel function, J_0(|k|). Technically, already, we can use the CF proof of the CLT and call it quits. That’s no fun though.
So, after n steps, the distribution phi_n(k) = J_0(|k|)^n.
Expanding, using the Taylor Series,
Raising that to the n gives, in limit, e^(-nk²/4), which is the Fourier transform of the 2D Gaussian. Inverse Fourier transform will give the 2D Gaussian then. Thus, we have the Gaussian from these bounces in 3D!
Now, to see this for 2D — remember our distribution is
And the definition of characteristic function is
So we just have 1/2 * e^(ik) + 1/2 * e^(-ik) which equals cos(k). This is one left/right decision. So a bunch of L/R decisions is cos(k)^n. Again, using Taylor series, cos(k) equals approx. 1 — k²/2. Raised to the n — we get approx e^(-nk²/2), which is the Fourier transform of the regular Gaussian. Really wish Medium had LaTeX support.
It seems we have lost the plot. This isn’t at all helping visualize facts about distributions! Characteristic functions, Fourier transforms… The KDE helped me see, this doesn’t really. I mean, both give great reasons why pi would be in the distribution. But still.
For this I would say — first, the KDE of the point distribution with circular Kernel wouldn’t technically give the Gaussian. That is why we looked at this. Visually, the problem is that, if the ball can bounce in any angle, it may not hit a peg, and therefore will not create a Gaussian. Formally, the Fourier transform of a circle will not give a Gaussian’s Fourier transform. Thus, we have to look at a scenario where we can ignore that error.
It also helps us see something interesting.
This told us each bounce is like picking a point on a 1 dimensional version of a Circle. And for 3D, it is picking a point on a regular 2D circle.
If you’ve seen those 4D visualizations of imagining slices (ie slice of cube is a square, and of a square is a line), the same can be done for spheres. This is hinting at an interesting fact…
Each face of Pascal’s pyramid is Pascal’s triangle.
This is how we can visualize multiplying two (independent) Gaussians. The 3D Galton board. Multiplying two dependent Gaussians is a topic for another day.
As a bonus, it also tells us why convolving two Gaussians gives a gaussian. For one, we saw the Fourier transform of a Gaussian is another Gaussian. But also — the 3D Galton board, which is a convolution in a loose sense because of KDE, generates a 2D Gaussian. This also makes it easy to see why the formula for convolution of two Gaussians is the same formula as adding two Gaussians.
Today we have gotten an explanation of adding and multiplying Gaussians, a 3D version of the Gaussian, an exploration of random walks in this frame (also providing reason why 3D random walk need not return to the origin). We saw combinatorics, probability theory, stats, and Fourier analysis.
Galton boards are pretty cool. They take Statistics and force you to look at a single trial.
Next time, we will see why multiplying two normals gives the formula it does and look at what happens with dependence.
