Validation You Can Trust: Random, Blocked and Leave-One-Well-Out Splits

Chapter 10, Generalization, Bias, and Variance

Learning objectives

  • Build three splits of the nine wells: random over the samples, blocked by depth, and leave one well out, and check that no well sits in two folds of the last
  • Measure the leak: the share of held-out samples whose neighbour, read through most of the same rock, trained the model
  • Show that a random split flatters kNN and a tree at every capacity, and that it chooses a different model
  • Hold out whole wells in Python with scikit-learn's LeaveOneGroupOut

A Score Is Only as Honest as Its Split

This is the book's most important lesson: a held-out score means what its split means, and a random split of well-log samples does not mean a new well. Split the nine wells' 10809 samples at random into nine folds and 0.988 of the held-out samples have their neighbour, half a foot away and read through 80 % of the same rock by the gamma-ray tool, among the training rows. kNN at k = 15 on the porosity logs and log10 RT then scores 0.990; holding out one whole well at a time, the same model scores 0.976. The random split flatters by 0.014, at every k, and it picks k = 21 where the honest split picks k = 15. Hold out what the field will hold out: a whole well.

Validation You Can TrustInteractive figure, enable JavaScript to interact.

Three Ways to Cut Nine Wells

The data are the book's own learning set, all synthetic: Ogbon-1, the Petrophysics book's well, and eight hypothetical appraisal wells of the book's field, 1201 samples each every 0.5 ft, logged by Ogbon-1's own build and the book's own tool model; pay by the generator's truth. Each split cuts the samples into nine folds, and each fold is called by a model fitted on the other eight, its logs standardized on those rows alone:

  • Random: the samples shuffled (seeded) and cut into nine slices. Each fold holds 1201 samples scattered through all nine wells.
  • Blocked by depth: the samples of all nine wells sorted by depth and cut into nine runs. A fold is the same depths in every well at once.
  • Leave one well out: each fold is one whole well. No well sits in two folds, and the model never sees a sample of the well it calls.

The scores come from the book's cross-validation table, which holds all three splits for kNN and for a tree, on the porosity logs and log10 RT with and without depth; the folds in plate (a) are built in your browser from the table's own settings.

The Leak

A tool reads a window of rock, not a point: the book's gamma-ray tool averages 5 samples, so two neighbouring samples share 80 % of the rock they read, and a random split leaves the neighbour of 0.988 of its held-out samples in training. A model marked on such a sample is marked on rock it has as good as seen. The split by depth leaks only at the edges of its runs, 0.010; leaving one well out leaks nothing, 0.000, because every neighbour of a held-out sample is in its own well and so in its own fold.

What the Leak Costs

On the porosity logs and log10 RT, the random split's score is above the honest one at every k, by between 0.014 and 0.015. For a tree at depth 6 it scores 0.987 against 0.975 with one well held out, 0.012 of flattery. Worse than the inflation is the choice: tuned on the random split, kNN would take k = 21 and report 0.990; held out one well at a time, that model scores 0.975, and the honest split's own choice, k = 15, scores 0.976. A leaky split chooses the model and then reports the inflated number for the model it chose.

The split by depth asks a different question. It holds out a slice of the section in every well at once, and for the tree at depth 6 it scores 0.961, below one well held out: rock at those depths is missing from training in every well together. A field does not ask what lies between two depths in all its wells; it asks what the next well holds. Chapter 9 takes up depth itself as a feature.

In Python

The set is one CSV, the book's own, with a comment header naming every column's unit.

import numpy as np
import pandas as pd

URL = "https://ogbonlab.com/mlgeo-core/data/ogbon-learning-set.csv"
df = pd.read_csv(URL, comment="#")
df["LRT"] = np.log10(df["RT_ohmm"])
B = ["GR_API", "RHOB_gcc", "NPHI_vv", "DT_usft", "LRT"]
X = df[B].to_numpy()
y = df["pay"].to_numpy()
wells = df["well"].to_numpy()

A pipeline refits the scaler inside every fold, on the training rows alone. A shuffled KFold is the random split; LeaveOneGroupOut with the well as the group holds out whole wells.

from sklearn.model_selection import (
    KFold, LeaveOneGroupOut, cross_val_score)
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

model = make_pipeline(StandardScaler(),
                      KNeighborsClassifier(15))
rnd = KFold(9, shuffle=True, random_state=7)
s_rnd = cross_val_score(model, X, y, cv=rnd)
s_well = cross_val_score(
    model, X, y, groups=wells,
    cv=LeaveOneGroupOut())
print(s_rnd.mean(), s_well.mean())

The book's kernel gives 0.990 for its random split and 0.976 with one well held out. Every fold holds 1201 samples, so the mean of the folds' scores is the pooled score. scikit-learn's shuffle is not the kernel's, and it breaks ties among equally near neighbours its own way, so its third decimal may differ.

The leak can be counted the same way: give each sample its fold, then ask whether the sample above or below it in the same well is in another one.

fold = np.empty(len(df), dtype=int)
for f, (_, te) in enumerate(rnd.split(X)):
    fold[te] = f
same = wells[1:] == wells[:-1]
cut = same & (fold[1:] != fold[:-1])
leak = np.zeros(len(df), dtype=bool)
leak[1:] |= cut
leak[:-1] |= cut
print(leak.mean())

The book's kernel gives 0.988 with its own random folds; with LeaveOneGroupOut's folds the same count gives 0.000.

What These Scores Are

Leaving one well out is the fairest split this set allows, and it is still kind: every well carries Ogbon-1's beds, moved down the structure, and one tool model logs them all. Real wells differ more, in their rocks, their tools and their boreholes, and a model scored on them should be scored on wells it never saw.

This page is prerendered for SEO and accessibility. The interactive widgets above hydrate on JavaScript load.