You have spent two weeks on a churn model for 5,000 subscribers, and it scores 85.8% accuracy. A colleague asks a simple question: what would a model that always says "will not cancel" score? You run it. 94%. Higher than yours. Nothing was miscalculated. Only 6% of subscribers cancel, so the answer with no intelligence in it is right almost every time, and your two weeks of work had no reference point until now.

This post builds that reference point in a few lines, then uses it to decide whether the real model is worth shipping. The example is one SaaS product with 5,000 subscribers and two questions: will this subscriber cancel in the next 30 days, and how many support tickets will they open next month. The data below comes from a short generator script, not a real company.

A baseline is the score you get from a rule that uses none of the real signal in your data, just its shape. It is the number a model has to clear before it has proven it learned anything. Call it the Floor: a model's score only means something once you know where the Floor sits under it.

For a label like "churned," the dumbest possible rule is the majority class baseline: predict whichever value shows up most often, for every row, regardless of the actual features. If 94% of subscribers do not cancel, predicting "will not cancel" for all 5,000 of them is right 94% of the time with zero information about any individual subscriber. That is class imbalance: one outcome so much rarer than the other that guessing the common one scores well by accident.

For a number like "tickets next month," the equivalent is the mean baseline: predict the average value for every row. Score it with MAE, mean absolute error, the average size of the miss in tickets, ignoring whether the guess ran high or low. A mean baseline minimizes squared error; when the metric is MAE, as it is here, a median baseline usually scores at least as well, worth checking on your own data.

Neither baseline touches a single feature. That is the point: they set the Floor a model with no real information would already reach, so you can tell how much of a model's score is signal and how much was free.

Here is the generator, in full, so the numbers below are reproducible. It builds 5,000 subscriber rows, fixes the churn rate at about 6% by construction, and adds a ticket count for the second target.

risk mixes three real signals, tenure, recent logins, last month's ticket count, with random noise, because real churn is never fully explained by the features you happen to have. Taking the top 6% of that risk score as "churned" fixes the churn rate by design, so the baseline below is not an estimate. It is exact.

None of this involved tuning a single hyperparameter. In practice, building both baselines is closer to a ten-minute job than the two weeks the tuned model took, which is most of the argument for building it first.

The two Dummy baselines above have no real counterpart to compare against yet. Here is that missing piece, reusing the exact splits from the baseline code:

A first attempt with a default RandomForestClassifier looks like a wash: accuracy comes in at 93.3%, actually a hair below the Floor's 94.0%, and recall is 5%, barely off the Floor's 0%. Nothing in a default fit tells the trees to pay extra attention to the 6% of subscribers who churn, so mostly they don't. class_weight="balanced" is what changes that: it reweights training so the model treats the rare churners as seriously as everyone else, and once it stops ignoring them it starts predicting "will churn" far more often than either the Floor or the default fit did. Every one of those extra churn predictions that turns out wrong costs a point of accuracy, on a target where 94% of subscribers were never going to churn anyway. That's why churn accuracy moves from 94.0% down to 85.8% once the reweighting kicks in, a drop of more than eight points. That looks like a step backward, and a leaderboard glance would call it one. Don't stop there: that trade is exactly what recall was built to measure.

Recall is the share of real churners the model finds: true positives divided by all actual positives. The majority-class baseline predicts "no churn" for every subscriber, so its recall on churners is exactly 0%. It catches none of them, no matter how good 94% sounds next to it. The default fit barely moves that, to 5%: it keeps almost all of the Floor's accuracy and almost none of its usefulness. Only the tuned model, trading accuracy for reweighting, lifts recall to 46.7%, finding nearly half of the subscribers who were about to cancel, for an eight-point drop in accuracy from the Floor. That is where the two weeks went.

The trade is not free: precision on those same predictions is 20.3%, so of every five subscribers the model flags as a churn risk, only one actually cancels. Whether the trade is worth it depends on what a false alarm costs the team acting on it, a retention email is cheap to waste on the other four; a steep discount offer is not.

So what has to be true before this model is worth shipping? Three things.

A change that does not clear the Floor by more than fold-to-fold noise, in the metric that matters, by a margin that pays for its own upkeep, is not a decision yet.

Figures above are from one executed run of the code above (an 80/20 split, random_state=7). Baseline accuracy (94.0%) and baseline recall/precision (0%) are exact by construction: the generator fixes churn at 6% by quantile cutoff, and a majority-class rule never predicts churn.

The churn target just showed that pattern once: a default fit that barely earns its keep, and a tuned fit that trades one metric for another. Run the same procedure on the second target, tickets next month, and the first attempt looks like the baseline already wins. The mean baseline scores an MAE of about 0.83 tickets: on average, guessing "about 1 ticket" for every subscriber misses by under a ticket. A default RandomForestRegressor lands at about 0.83 tickets too, a gain of essentially zero.

Before writing the target off, tune the way you would for any target: constrain how deep and how greedy the trees are allowed to get (max_depth=5, min_samples_leaf=10, min_samples_split=5) instead of only adding more trees. That tuned model gets an MAE of about 0.79, a real 0.04-ticket improvement over the baseline. The default model wasn't proof the target lacks signal; it was an untuned model that hadn't found the signal yet.

Put that 0.04 through the same three checks as the churn model. Does it beat the spread you'd see from one cross-validation fold to the next on this target? A margin this small needs that check more than a large one does. Does it pay for what the model costs? A support team acting on a prediction that's off by 0.79 tickets instead of 0.83 is unlikely to change staffing on its own. The honest read isn't "the Floor won" or "the model won": tuning showed exactly how much room there was, and that room was small.

This is the case most tutorials skip: not every target has a gain worth the cost of chasing it, and the only way to know how large the gain really is is to tune before comparing, then measure the tuned model's margin over the Floor, not the first attempt's.