A fraud model scores 99.5% accuracy on 10,000 test orders, and the team puts that number on the dashboard. Then someone tries a model that flags nothing at all. It also scores 99.5%, because only 50 of those orders were fraud.

Nothing was miscalculated. The metric was answering a different question from the one the business was asking. This post is about how to tell which question a metric answers before you trust it.

All numbers below are illustrative. We will use one imaginary food-delivery app with three models: one flags fraudulent orders, one estimates delivery time, and one ranks restaurants in search.

Here is the fraud test set: 10,000 orders, 50 of them fraudulent. Two candidates.

Model A flags nothing. It gets 9,950 right and 50 wrong, and scores 99.5%.

Model B flags 80 orders. Of those, 40 really are fraud, and it misses the other 10. It is right on 9,910 legitimate orders plus the 40 frauds it caught, so it is right on 9,950 orders and scores 99.5%.

Same score. Model A stops zero fraud. Model B stops 40 of 50 and annoys 40 honest customers with a flagged order. Which you prefer depends on what fraud costs and what an annoyed customer costs, and the accuracy number cannot tell you, because it is identical.

Accuracy counts every mistake as equal. Here the mistakes are not equal, and 99.5% of the orders are the easy kind.

A metric is a stand-in. You cannot measure "is this model good enough for the problem it solves" directly, so you pick a number that approximates it, then optimize and report that number. When the number drifts from what you care about, teams keep improving it while the product gets no better. So choose the stand-in on purpose. Ask three questions first:

Once you can answer the second question, the metric mostly picks itself. The common ones follow, grouped by the kind of output the model produces.

A classifier outputs a label: fraud or not fraud. It can be wrong in two ways. A false positive is a legitimate order flagged as fraud. A false negative is a fraudulent order let through. The confusion matrix counts all four outcomes, and every classification metric is a way of combining those four counts.

Each standard metric answers one question:

F1 has two limits. It weights precision and recall equally, so it is the wrong stand-in if a miss costs ten times a false alarm. And it ignores true negatives, so it would not move if Model B correctly passed 10,000 more legitimate orders, even though accuracy would rise. So do not treat it as a default. Report precision and recall separately, to see which mistake drags the score down (Model B: 50% and 80%, not just 0.62), and add the raw count of false alarms, which a rate can hide (Model B blocked 40 honest customers).

Model A has recall of 0% and, since it flags nothing, an undefined precision (0 / 0), which scikit-learn reports as 0 by default with a warning unless you set it.

Most classifiers output a score, such as 0.83 fraud, not a label. The threshold is the score above which you flag an order. Lower it and recall rises while precision falls; raise it and the trade reverses. Precision, recall and F1 all describe one threshold, so a change in threshold is a change in the metric.

ROC AUC (area under the ROC curve) scores the model at every threshold at once. Pick one random fraud order and one random legitimate order, and ROC AUC is the chance the model scored the fraud order higher: 0.5 is a coin flip, 1.0 is perfect separation. That measures ranking, not the decisions you make at your threshold. With rare fraud, legitimate orders are so numerous that a small share of them scoring high can outnumber the real catches, so ROC AUC can look strong while precision is poor.

For rare fraud, look at PR-AUC (area under the precision-recall curve, which scikit-learn reports as average precision). It falls when false alarms swamp the real catches. Its random-guess baseline is the fraud rate, 0.5% here, not the 0.5 of ROC AUC, so read it against that. Use ROC AUC to compare rankers on roughly balanced classes, and PR-AUC when the positive class is rare.

Next is regression, where the metrics measure distance instead of counting mistakes.

The delivery-time model outputs minutes until the food arrives. A mistake is now a distance, so the metrics measure distance.

MAE (mean absolute error) averages how many minutes off the estimates are. RMSE (root mean squared error) squares each error, averages, then takes the square root, so big errors count for much more.

Take five orders, with illustrative errors in minutes:

Both models are off by 5 minutes on average. Model Y is nearly perfect on four orders and 21 minutes wrong on one. MAE cannot tell them apart. RMSE can.

Which is worse is the cost question again. If a 20-minute miss makes a customer cancel and file a complaint while a 2-minute miss does nothing, big errors cost more than proportionally, and RMSE is the better stand-in. If every minute of error costs about the same, MAE says exactly that and is easier to explain: "we are off by 5 minutes on average."

Search results are not a classification problem. When a customer types "ramen," the app cares less about each restaurant's score than about which ones appear first, because people mostly act on the top few.

Precision at k asks: of the top k results (say the top 3), how many are relevant? NDCG (normalized discounted cumulative gain) adds up the relevance of each result but discounts it the further down the list it sits, then divides by the score of the best possible ordering, so 1.0 means a perfect list.

Say two of five results are good ramen places.

Precision at 5 calls them equal. But List 2 buries the good places at the bottom, and a customer who looks at the top three sees nothing useful. Precision at 3 catches it, and so would NDCG, which scores List 1 at 1.0 and List 2 at roughly half that. Choose k to match how far your users actually look: if they rarely scroll past the first screen, a good result below it is worth little.

Here is the same thinking as a five-step checklist you can run before you adopt a metric:

The same questions work in reverse when you review someone else's choice. When someone tells you their model scores 0.94 on something, ask: What is the cost of each mistake in this product? Which mistake does this metric count? What score would a model that does nothing useful get? If they cannot answer the third, the metric has not been tested against the problem.