False Claim about Lootbox — A Simple Example of Bayesian Statistics
Background: Lootbox Probability
Let’s summarise this very old news in Taiwan that happened in 2023: a PC online game company Gamania claimed that the probability of getting a high tier item from lootbox is 10%. However, a well known Taiwanese streamer named Dinter got only 11 from 475 trials which is much lower than the expectation. Players suspected that the game company made a false claim about the lootbox probability.
Most of people prefer hypothesis testing for the analysis of the case. Now I would like to provide different analysis using Bayesian Statistics.
A Common Mistake in the discussion
There is a common misconception which states that since getting the item from lootbox is an independent event, no matter the previous result, the probability of next result faiL is always 90%. Thus, it is very possible (with probability 1–0.1 = 0.9) that the player keep fail to get the item. These people intentionally or unintentionally confused two probability: the above statement said the next single result is not affected by the previous. However, what players concern is the overall result which is how unreasonable the event it is. We shall illustrate the difference by mathematical notation:
Concern about Independent trial:
Concern about overall result:
Verdict from Hypothesis Testing (Fast-Forward)
Normally, mathematicians perform a hypothesis test of proportion to determine whether the game company lie. The steps (over-simplified) is the following:
The result suggests that the data collected is hugely incompatible with the hypothesis (which said p_naught is 0.1). Hence, it is very statistically significant to reject the null hypothesis. So, we can infer that the company lie about the probability of getting the item from lootbox.
Verdict from Bayesian Statistics
Though our case satisfies all assumptions of hypothesis testing of proportion such as random sampling and independent events etc., the result seems less informative because it only tells us how unlikely the event it is, given that the claimed probability is true. Nevertheless, the true agenda is how unlikely is the claim, given that such event happened. According to Bayes Theorem, we should care about L.H.S. of the formula:
where theta is the hypothetical value we concern (the probability of getting the item from lootbox) and D represents the experiment conducted by player. We called p(theta | D) posterior probability, p(D|theta) likelihood, p(theta) prior probability and the denominator p(D) evidence.
Get Wai Ting 13 (Specter)’s stories in your inbox
Join Medium for free to get updates from this writer.
In Hypothesis testing, theta (the hypothetical value concern) is fixed quantity (=0.1). In Bayesian Statistic, theta is a random variable follows some probability distribution. In other word, instead of fixing the value of theta, we prefer to infer the value of theta from posteriori. So, these 4 terms are probability distributions.
Likelihood
The first item p(D | theta) is the most familiar term in this equation. Given the value of theta, what is the probability of the observation. In our case, each trial was independent and only subjected to theta. Thus, we get our good-old binomial distribution but now theta is a random variable, not parameter.
Conjugate Prior
The second term p(theta) may be a bit confusing. If we already know the probability, why do we still need to make inference? However in Bayesian Statistics, we will update our judgement once we have new information. Therefore, this represents judgement about theta before observation or experiment. The judgement may affected by opinion from experts and past experience.
If our prior information prefer none of the value of theta, the prior distribution will be uniform distribution. Once we have some preference on theta, we must consider the form of prior distribution. Since it will later be multiplied with the likelihood to calculate the Posteriori, we hope to find the formula that have a very similar form of the likelihood. We call this a conjugate prior. Luckily, Beta Distribution have the similar form:
In this equation, theta is the random variable of the distribution, B(alpha, beta) is the constant, alpha and beta are parameters which affects the shape of the prior distribution. In our inference, alpha and beta are determined by ourselves, not inference. We call them hyperparameters.
The most controversial part in our calculations will be the value assignment of the hyperparameters since it is very subjective. To be fair, I would like to give more “credibility” on company’s claim but remain certain possibility of other values. That means the beta distribution have a narrow peak around 0.1 but other values are still possible. We will set alpha to be 10 and beta to be 90, so that the expected value of the distribution is 0.1. The shape of the distribution thus looks like this:
Evidence — The denominator
The denominator can be omitted because it can be derived from the integral over numerator with respect to theta. This term only guarantees that the normalisation of posteriori (i.e. sum to 1). It is not useful in our inference.
Posteriori
The multiplication of two terms produces a new beta distribution with updated parameters. There are several ways to make inferences from posteriori, one of them is Maximum a Posteriori (MAP).
Like classical Maximum Likelihood Estimation, we can derive the value of theta that maximise the posteriori. The mode of the distribution is the desired value. In general, finding mode of distribution involve complex computation. Nevertheless in beta distribution, we just need to apply the formula:
According to MAP, the most probable value of theta is 3.4904% which is much lower than the company’s claim.
We can also compute credible interval with the help of AI chatBot. For instance, 99.9999% credible interval covers 99.9999% area of posteriori distribution which means 99.9999% of the most possible outcomes. According to result from Gemini and verified by myself in GeoGebra, 99.9999% Credible Interval is (0.9883%, 8.7226%) which still doesn’t include the company’s claimed value 10%.
Conclusion
Even though we give more creditability on the company’s claim, the evidence is much overwhelming so that the posteriori favours our claim. It means the company was very likely to make a false claim about the probability of getting the item from lootbox.
Epilogue
I am not going to join the debate of frequentist vs Bayesian since it is out of my scope of capability. To me, Bayesian Statistics is much less subjective to misinterpretation. To many people Bayesian Statistics is very difficult. Nevertheless, the above case show that it is not as hard as these people thought. Also, I hope this case study will be a textbook example in the future.
