Two teammates, two winners, same models
You and a colleague each fit the same four regression models on the same housing data and pick a winner. You call it Gradient Boosting. They call it Random Forest. Neither of you made a mistake: you scored under MAE, they scored under RMSE, and a handful of unusually priced houses were enough to flip which of the two tree ensembles comes out on top.
That's not a fluke of this one dataset. It's what MAE and RMSE do by design. They both measure "how wrong were the predictions," but they don't weigh a bad miss the same way, and a model that's good at avoiding many small misses isn't automatically good at avoiding a few huge ones.
What MAE and RMSE actually punish
Mean absolute error (MAE) takes every prediction's miss, in dollars, and averages the size of it. Miss by $10,000 on one house and $10,000 on another, and MAE treats those two misses as equally bad.
Root mean squared error (RMSE) squares each miss before averaging, then takes the square root to bring the units back to dollars. Squaring punishes a $40,000 miss far more than four $10,000 misses combined (40,000² is 16 times 10,000², not 4 times). A model can have a great RMSE by being consistently close, or a terrible one by being close on almost everything and wildly wrong on a couple of rows.
Call that second failure mode the outlier tax: RMSE effectively charges a model extra for every prediction it gets badly wrong, no matter how good the rest of its predictions are. MAE doesn't charge that tax. That single difference is the entire mechanism behind everything that follows.
The setup: four models, one dataset, fit once
The dataset is Kaggle's "House Prices: Advanced Regression Techniques", about 1,460 homes with a SalePrice target, split 80/20 into train and test with random_state=42. What decides the story below isn't a documented outlier from the dataset's write-up: it's whichever homes land in this particular 292-row test set and turn out hardest for these four models to price.
Four models get fit once, on the same train/test split, and never refit: plain LinearRegression, RandomForestRegressor, GradientBoostingRegressor, and HuberRegressor. Huber is the odd one out on purpose: it minimizes Huber loss, which behaves like squared error for small misses and like absolute error for large ones, a training-time defense against exactly the kind of outlier tax RMSE imposes at test time. Whether that defense actually pays off in the RMSE column is one of the things the leaderboard below settles.
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, HuberRegressor
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
from sklearn.metrics import mean_absolute_error, mean_squared_error, mean_absolute_percentage_error
df = pd.read_csv("train.csv") # Kaggle "House Prices - Advanced Regression Techniques"
# Keep this simple: numeric columns only, median-fill the gaps.
numeric = df.select_dtypes(include="number").drop(columns=["Id", "SalePrice"])
X = numeric.fillna(numeric.median())
y = df["SalePrice"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
models = {
"Linear Regression": LinearRegression(),
"Random Forest": RandomForestRegressor(n_estimators=300, random_state=42),
"Gradient Boosting": GradientBoostingRegressor(random_state=42),
"Huber": HuberRegressor(max_iter=500),
}
rows = []
for name, model in models.items():
model.fit(X_train, y_train)
pred = model.predict(X_test)
rows.append({
"model": name,
"mae": mean_absolute_error(y_test, pred),
"rmse": mean_squared_error(y_test, pred) ** 0.5,
"mape": mean_absolute_percentage_error(y_test, pred),
})
pd.DataFrame(rows)
This is deliberately simple: numeric columns only, median-filled, no encoding of the dataset's many categorical columns. It's enough to fit four real models and generate four real sets of predictions to score, which is all the comparison needs.
The leaderboard under MAE
Sorted by MAE, smallest miss on average wins:
Model | MAE | RMSE | MAPE |
|---|---|---|---|
Gradient Boosting | $17,681 | $29,844 | 10.3% |
Random Forest | $18,009 | $29,037 | 10.8% |
Linear Regression | $22,976 | $36,837 | 13.5% |
Huber | $23,722 | $40,766 | 13.5% |
Gradient Boosting and Random Forest take the top two spots, separated by less than $400 in MAE. Both fit the bulk of ordinary homes tightly: a tree ensemble can carve the feature space into narrow buckets and assign each one a price close to what similar homes actually sold for, so most of its predictions land within a few thousand dollars of the truth. Linear Regression and Huber, stuck with a single global relationship between size and price, both trail by $5,000 or more.
The same leaderboard under RMSE
Sort the same four rows by RMSE instead, and the top two swap places:
Model | RMSE | MAE |
|---|---|---|
Random Forest | $29,037 | $18,009 |
Gradient Boosting | $29,844 | $17,681 |
Linear Regression | $36,837 | $22,976 |
Huber | $40,766 | $23,722 |
Random Forest, which trailed Gradient Boosting by about $330 in MAE, takes the RMSE lead by about $800. Nothing about the predictions changed between the two tables: same models, same rows. What changed is which mistakes the metric decided to punish, and by how much. Linear Regression and Huber don't move: they're third and fourth under both metrics. MAE and RMSE agree on who the two worst models are here; they only disagree on the tiebreak between the two tree ensembles.
Why: looking at the outlier rows model by model
The RMSE gap traces back to the tail of the price distribution in this test set. Pull the ten priciest homes and score each model's squared error against the total:
top10 = X_test.copy()
top10["Actual price"] = y_test
for name in ["Random Forest", "Gradient Boosting", "Huber", "Linear Regression"]:
pred = models[name].predict(X_test)
top10[f"{name} pred"] = pred
top10[f"{name} sq_err"] = (y_test - pred) ** 2
top10 = top10.sort_values("Actual price", ascending=False).head(10)
for name in ["Random Forest", "Gradient Boosting"]:
tail_sq_err = top10[f"{name} sq_err"].sum()
full_sq_err = ((y_test - models[name].predict(X_test)) ** 2).sum()
print(f"{name}: top-10 rows are {tail_sq_err / full_sq_err:.1%} of total squared error")
GrLivArea | Actual price | Random Forest | Gradient Boosting | Huber | Linear Regression |
|---|---|---|---|---|---|
4,316 | $755,000 | $584,465 | $596,785 | $470,668 | $464,989 |
2,364 | $611,657 | $443,266 | $366,664 | $345,782 | $384,280 |
2,868 | $556,581 | $415,073 | $437,469 | $325,117 | $342,360 |
2,076 | $465,000 | $376,482 | $389,547 | $300,111 | $338,557 |
2,296 | $451,950 | $393,620 | $425,763 | $316,429 | $360,210 |
2,945 | $438,780 | $382,910 | $389,852 | $331,549 | $385,427 |
2,794 | $403,000 | $335,832 | $397,426 | $331,113 | $306,061 |
1,973 | $395,000 | $315,108 | $303,159 | $282,831 | $288,238 |
2,028 | $369,900 | $359,545 | $368,872 | $293,309 | $317,485 |
1,702 | $367,294 | $376,832 | $425,090 | $290,667 | $320,661 |
import matplotlib.pyplot as plt
flip_model = models["Gradient Boosting"] # wins on MAE, loses the RMSE tiebreak to Random Forest
pred_test = flip_model.predict(X_test)
outlier_mask = X_test.index.isin(top10.index) # the 10 priciest homes from the table above
fig, ax = plt.subplots()
ax.scatter(y_test, pred_test, alpha=0.4, label="Test homes")
ax.scatter(y_test[outlier_mask], pred_test[outlier_mask], color="red", label="10 priciest homes")
ax.plot([y_test.min(), y_test.max()], [y_test.min(), y_test.max()], "k--", linewidth=1)
ax.set_xlabel("Actual price")
ax.set_ylabel("Predicted price")
ax.legend()
plt.savefig("rmse-outliers.png", dpi=150)
Every model under-shoots almost every row here, and Linear Regression and Huber miss worse than the ensembles on nearly all of them, the same pattern behind their much higher overall RMSE. The number that matters most: these 10 homes, about 3.4% of the test set, account for 41.8% of Random Forest's total squared error and 46.1% of Gradient Boosting's. RMSE for the top two models is decided almost entirely by this handful of rows, not by the other 282.
One row decides most of the tiebreak between them: the $611,657 home, where Random Forest misses by $168,391 and Gradient Boosting misses by $244,993. Squared, that $76,602 gap outweighs the several rows further down the table where Gradient Boosting actually does better. Net, Gradient Boosting's tail error runs about 16% higher than Random Forest's, enough by itself to flip their RMSE order even though Gradient Boosting has the better MAE.
That's the real mechanism, not "tree ensembles vs. linear models" in the abstract: when two models are this close, RMSE isn't grading overall skill, it's asking a tiny slice of expensive, hard-to-predict homes to cast the deciding vote. A different train/test split could easily hand that vote to the other ensemble.
Which metric matches what the business cares about
Neither metric is the "correct" one in general. The right choice depends on how a mistake actually costs you.
If the cost of being wrong scales with the size of the miss, RMSE is the honest metric. A lender sizing risk on a portfolio of home-equity loans cares a lot more about the one loan that's off by $250,000 than about two hundred loans that are each off by a few thousand dollars; a few catastrophic misses can sink the portfolio in a way that many small ones never would. RMSE's outlier tax matches that cost structure directly.
If every miss costs about the same regardless of size, MAE is the honest metric. A pricing tool that just needs to be "roughly right" on the typical listing, where being off by $15,000 on a $60,000 starter home and $15,000 on a $600,000 estate cause similar-sized headaches, has no reason to let a few extreme rows dominate the score.
MAPE (Mean Absolute Percentage Error) agrees with MAE and RMSE on the top two spots here, but it flips the bottom two: Huber narrowly beats Linear Regression on MAPE (13.45% vs. 13.46%) even though Linear Regression has the better MAE and the better RMSE. That's MAPE's own distortion: it grades a dollar miss on a cheap house more harshly than the same miss on an expensive one, so missing the cheapest test homes by a larger percentage can make a model look worse on MAPE than its MAE and RMSE numbers suggest.
That's the question worth asking before you trust any leaderboard: what does a mistake here actually cost?
Takeaway
MAE and RMSE don't always reorder a leaderboard end to end. Here they agree on the bottom two (Linear Regression, then Huber) and disagree only on which of the two close tree ensembles takes first.
A model that wins on MAE by a slim margin can still lose the RMSE tiebreak if its worst misses are a bit larger, even when the two models are otherwise close: Gradient Boosting beat Random Forest by about $330 in MAE but lost to it by about $800 in RMSE. The tail is why: the 10 priciest homes, 3.4% of the test set, account for 42–46% of each ensemble's total squared error, and Gradient Boosting's tail error runs about 16% higher than Random Forest's.
HuberRegressor's loss function is designed to downweight large residuals during fitting, but that's a training-time property, not a guarantee it wins any particular test-time metric: here it scored worst of the four on both MAE and RMSE.Before you trust a leaderboard, ask which metric produced it, then ask whether that metric's idea of a bad mistake matches your actual cost of being wrong.
If a few catastrophic misses would hurt you far more than many small ones, score under RMSE. If every miss costs about the same, score under MAE, and treat MAPE as a sanity check on the cheapest cases, not the tie-breaker.
These numbers come from one train/test split (
random_state=42). A different split could move the exact margins, but it's unlikely to hand Linear Regression or Huber a top-two finish on this dataset.
