Member-only story
Survival Analysis for Data Scientists
Part 1 of 6: Time to Event, Censoring and the Kaplan-Meier Estimator. Why a customer table that is still being written gives the wrong retention numbers, how the Kaplan-Meier estimator uses the customers who have not churned yet, and what that changes in an acquisition budget.
Diogo Ribeiro is a mathematician, researcher and lecturer at the Faculdade de Media Artes e Design, Universidade Técnica do Porto.
A large share of the questions data scientists are asked are about time. How long until a subscriber cancels? How long until a loan defaults, a machine fails, an employee leaves, a patient is readmitted? The data that should answer them is almost always a snapshot: a table extracted on some date, in which many of the subscribers, loans, machines and employees are still active. For them, the event has not happened yet. We know only that it had not happened by the day the table was extracted.
Those unfinished records are called censored, and handling them is what survival analysis is for. Two common shortcuts, dropping them or counting them as successes, produce numbers that are wrong in a predictable direction, and the error does not shrink as the table grows. A third, turning the question into a classification at a fixed date, gets the right answer from a fraction of the data.
