Testing a model on the same data it learned from is like grading your own homework — you need a fair test.
Complete lesson & earn 250 PX
EXERCISE
1Testing a model on the same data it learned from is like grading your own exam — you will always get 100%, but you have learned nothing.
EXERCISE
2Train the model on 80% of data, test on 20%, and compare R² scores — if both are similar, your model generalises well.
EXERCISE
3Only after your model passes the train/test check should you trust it to make real predictions.
Imagine a student who memorises every answer in the textbook but cannot solve a single new problem. That student overfit the textbook. In ML, a model that scores perfectly on but fails on new data has the exact same problem.
The solution: split your data.
import numpy
numpy.random.seed(2)
x = numpy.random.normal(3, 1, 100)
y = numpy.random.normal(150, 40, 100) / x
# 80% for training, 20% for testing
train_x = x[:80]
train_y = y[:80]
test_x = x[80:]
test_y = y[80:]
print(f"Training samples: {len(train_x)}")
# Output: Training samples: 80
print(f"Testing samples: {len(test_x)}")
# Output: Testing samples: 20
Why 80/20?
| Split | Training | Testing | Use Case |
|---|---|---|---|
| 80/20 | 80% | 20% | Most common, good default |
| 70/30 | 70% | 30% | Small datasets (more test = safer) |
| 90/10 | 90% | 10% | Very large datasets (millions of rows) |
Pro Tip: The test set must be data the model has NEVER seen during training. If even one test point leaks into training, your accuracy score is inflated and meaningless. This concept — keeping train and test strictly separate — is the foundation of honest ML evaluation.
Think of training R² as your practice exam score and testing R² as your real exam score. If practice is 95% but the real exam is 40%, you memorised instead of learning.
The complete train/test workflow:
import numpy
from sklearn.metrics import r2_score
numpy.random.seed(2)
x = numpy.random.normal(3, 1, 100)
y = numpy.random.normal(150, 40, 100) / x
# Split
train_x = x[:80]
train_y = y[:80]
test_x = x[80:]
test_y = y[80:]
# Train a polynomial model (degree 4)
mymodel = numpy.poly1d(numpy.polyfit(train_x, train_y, 4))
# R² on TRAINING data
train_r2 = r2_score(train_y, mymodel(train_x))
print(f"Train R²: {round(train_r2, 4)}")
# Output: Train R²: 0.7988
# R² on TESTING data
test_r2 = r2_score(test_y, mymodel(test_x))
print(f"Test R²: {round(test_r2, 4)}")
# Output: Test R²: 0.8086
Interpreting the results:
| Train R² | Test R² | Verdict |
|---|---|---|
| 0.80 | 0.81 | ✅ Great — model generalises well |
| 0.99 | 0.30 | ❌ Overfitting — memorised training data |
| 0.40 | 0.38 | ⚠Underfitting — model is too simple |
Pro Tip: The ideal scenario is when train R² and test R² are CLOSE to each other AND both are high. If train is much higher than test, your model is overfitting. If both are low, your model is underfitting. This gap between train and test performance is the essence of the bias-variance tradeoff — the most important concept in all of ML.
A from an unvalidated model is like advice from an unqualified doctor — it might be right, but you have no reason to trust it. Validation first, predictions second.
Complete validated prediction workflow:
import numpy
import matplotlib.pyplot as plt
from sklearn.metrics import r2_score
numpy.random.seed(2)
x = numpy.random.normal(3, 1, 100)
y = numpy.random.normal(150, 40, 100) / x
# Split
train_x, test_x = x[:80], x[80:]
train_y, test_y = y[:80], y[80:]
# Train
mymodel = numpy.poly1d(numpy.polyfit(train_x, train_y, 4))
# Validate
train_r2 = r2_score(train_y, mymodel(train_x))
test_r2 = r2_score(test_y, mymodel(test_x))
print(f"Train R²: {round(train_r2, 3)}, Test R²: {round(test_r2, 3)}")
# Output: Train R²: 0.799, Test R²: 0.809
# Both ≈ 0.80 → model is validated ✅
# Now predict with confidence
print(f"5 minutes → ${round(mymodel(5), 2)} spent")
# Output: 5 minutes → $22.88 spent
print(f"3 minutes → ${round(mymodel(3), 2)} spent")
# Output: 3 minutes → $47.75 spent
The prediction reliability scale:
# When to trust your prediction:
# ✅ Train R² ≈ Test R² AND both > 0.7 → Trust it
# ⚠Train R² ≈ Test R² but both < 0.5 → Weak trust
# ❌ Train R² >> Test R² → Do NOT trust
# ❌ Model not validated at all → Unknown risk
Pro Tip: Notice the model predicts a customer spending 5 minutes will spend $22.88. But be careful — the model also implies 6 minutes = $200 (overfitting at the edges). Always sanity-check predictions against common sense. A model is a tool, not an oracle.
Split your data 80/20 into training and testing sets. Train on 80%, test on 20%, and compare R² scores to check for .