FAU Erlangen-Nürnberg
Institute of Micro- and Nanostructure Research
Week 4: a model, a loss, an honest split. Today: where the table comes from, and the model that is the default for tables. One real data set all the way.
1 934 × 11

four images become one table: 1 934 objects, 11 descriptors each
→ Acts 1 + 2 · Objects, Descriptors
94 → 69 %

accuracy of the same random forest: random folds versus a new image
→ Act 5 · Validate
84 %

on a new image with six descriptors that do not know the image
→ Acts 2 + 6 · Descriptors, Explain
Four HAADF-STEM images of one cross-section: an oxide film on a LaAlO₃ substrate, with an amorphous layer on top. Different sessions, magnifications and doses.
The question for every intensity maximum: is it a substrate column, a film column, or no column at all?
Act 1 · Objects
Where are the objects in the image?
Image B, a strip across the interface. From left to right: the raw image; smoothed, with the slow background removed; the local maxima; the fitted positions, coloured by the label.
Three steps, three hyperparameters: the smoothing width, the minimum distance, the threshold. Each of them changes the table.
1 Model of one column: \(I(\mathbf r)=A\,\exp\!\big(-\lvert\mathbf r-\mathbf r_0\rvert^2/2\sigma^2\big)+b\) in a small window around the maximum.
2 Least squares: \(\min_{A,\mathbf r_0,\sigma,b}\sum_{\text{pixels}}\big(I_\text{obs}-I(\mathbf r)\big)^2\). Non-linear in \(\mathbf r_0\) and \(\sigma\): Gauss–Newton, Week 4.
3 Output for each column: position \(\mathbf r_0\), peak height \(A\), width \(\sigma\), background \(b\), residual. The first descriptors.
4 Precision: \(\sigma_{r_0}\approx\sigma/\sqrt{N}\) for \(N\) detected electrons in the column. The pixel size does not appear.
Left: the repeatability of a column position, measured on 219 substrate columns from two independent sums of image D. Right: a cut through one column, for one frame and for the mean of ten frames (shown with 2 × 2 binning, 20 pm), with the Gaussian fits.
Pixel size 10 pm. Precision with one frame: 12 pm. With five frames: 5.6 pm. Factor \(\sqrt5\): the precision follows \(1/\sqrt{\text{dose}}\).
A synthetic particle image processed with scikit-image: smooth, Otsu threshold, binary mask, watershed to split touching particles, and one table row per particle (regionprops).
| Object | Detector | Hyperparameters | Typical descriptors |
|---|---|---|---|
| atom column | smooth → maxima → Gaussian fit | smoothing width, minimum distance, threshold | position, height, width, neighbours |
| particle, grain, pore | smooth → threshold → watershed | smoothing width, threshold rule, marker distance | area, equivalent diameter, eccentricity, solidity |
Two minutes, in pairs. Say what changes in the table.
Act 2 · Descriptors
Which numbers describe one object?
| Descriptor | Meaning | Unit | Kind |
|---|---|---|---|
amp, bg |
peak height and background of the fit | counts | raw intensity |
sigma |
width of the fit | nm | shape |
resid |
RMS residual of the fit / peak height | – | shape |
contrast |
peak height / background | – | ratio |
I_rel |
peak height / median peak height of the 4 nearest | – | ratio |
d1, d4 |
distance to the nearest; mean distance to the 4 nearest | nm | geometry |
d_spread |
std / mean of the 4 distances | – | geometry |
n_near |
number of candidates within 0.45 nm | – | geometry |
psi4 |
four-fold order \(\big\lvert\tfrac14\sum_{j=1}^{4}e^{4i\theta_j}\big\rvert\) of the 4 nearest | – | geometry |
1 934 rows × 11 columns. From here on the image is gone: the model sees only this table.
Top: the peak height amp of the substrate columns, for each of the four images. Bottom: the relative intensity I_rel of the same columns.
The median peak height differs by a factor of 4.3 between the images. A model that splits on amp learns which image, not which column.
| The acquisition changes … | The descriptor must be … | How to get it |
|---|---|---|
| the field of view (shift) | translation invariant | use distances to neighbours, not positions |
| the scan direction (rotation) | rotation invariant | use distances and angles between neighbours |
| the order of the neighbours | permutation invariant | sum or average over the neighbours |
| gain, offset, dwell time | intensity invariant | use ratios to the local neighbours |
| the pixel size | scale aware | measure in nm, not in pixels |
And two more: continuous (a small shift of an atom gives a small change) and sensitive (different physics gives different numbers).
Image B. Left: each candidate coloured by the four-fold order psi4. Right: coloured by the distance to the nearest neighbour, d1.
Radial symmetry functions (ACSF, \(G^2\)) Behler, Jörg et al., (2007), doi:10.1103/PhysRevLett.98.146401
\[G_i(r_s)=\sum_{j\ne i} e^{-(r_{ij}-r_s)^2/2w^2}\,f_c(r_{ij})\]

Two minutes, in pairs. Which invariance does it break, and how do you repair it?
Act 3 · Tree
How does a tree ask questions?
A tree of depth 3 on the six invariant descriptors, trained on all 1 934 labelled candidates. Each node shows the question, the number of rows and the class counts.
1 Impurity of a node with class fractions \(p_k\): Gini \(I=1-\sum_k p_k^2\), entropy \(I=-\sum_k p_k\log_2 p_k\).
2 A split (feature \(j\), threshold \(t\)) sends \(N_L\) rows left and \(N_R\) rows right.
3 Impurity decrease: \(\Delta = I(\text{parent})-\frac{N_L}{N}I(L)-\frac{N_R}{N}I(R)\)
4 Choose the pair \((j,t)\) with the largest \(\Delta\). Regression: \(I\) = variance of \(y\) in the node.
Two classes: the impurity against the fraction \(p\) of one class.
The algorithm
min_samples_leaf rows.What follows from it
Accuracy against the depth of one tree. Left: three raw features (amp, bg, sigma). Right: six invariant descriptors. Training rows, random 5-fold, and a new image.
Raw features, full tree: 100 % training, 93 % random folds, 64 % new image. The random folds do not show the overfitting.
Two trees of depth 3 on two descriptors, trained on two bootstrap samples of the same rows. The shading shows the predicted class.
Same data, same algorithm, a different tree. High variance is the weakness of one tree, and the opportunity for many.
Two minutes, in pairs.
d1 by \(\log\) d1. What changes in the tree? → Nothing but the threshold values. Only the order matters.min_samples_leaf), and check the features for image identity.Act 4 · Ensemble
Why are many trees better than one?
1 Bootstrap: draw \(N\) rows with replacement, \(B\) times. Train one deep tree on each sample.
2 Aggregate: average the \(B\) predictions (regression) or take the majority vote (classification).
3 Independent trees, each with variance \(\sigma^2\): \(\;\mathrm{Var}\big(\tfrac1B\sum_b f_b\big)=\sigma^2/B\). The bias does not change.
4 Correlated trees, pairwise correlation \(\rho\): \(\;\mathrm{Var}=\rho\,\sigma^2+\dfrac{1-\rho}{B}\,\sigma^2\;\xrightarrow{B\to\infty}\;\rho\,\sigma^2\)
More trees remove the second term only. The floor \(\rho\sigma^2\) stays: to go lower, make the trees less alike.
Random forest Breiman, Leo, (2001) = bagging + one more random choice:
max_features, default \(\sqrt F\) for classification)
1 Start with a constant: \(F_0(\mathbf x)=\bar y\). Loss \(\mathcal L=\sum_i \ell\big(y_i,F(\mathbf x_i)\big)\).
2 Pseudo-residuals: \(r_i=-\,\partial \ell/\partial F(\mathbf x_i)\). For the squared error: \(r_i=y_i-F(\mathbf x_i)\), the residual.
3 Fit a small tree \(h_m\) to the \(r_i\) (depth 1 to 6).
4 Update: \(F_m=F_{m-1}+\eta\,h_m\). Repeat steps 2 to 4.
Gradient descent of Week 4, \(\mathbf w\leftarrow\mathbf w-\eta\nabla\mathcal L\), but in function space: each tree is one gradient step.
Boosting with trees of depth 2 on the six invariant descriptors. Log-loss on a new image (solid) and on the training images (dashed) against the number of trees, for four learning rates.
η = 1: best at 5 trees, then worse than the start. η = 0.1: best at 98 trees. The training loss goes to zero in all cases.
| Random forest (bagging) | Gradient boosting | |
|---|---|---|
| Trees | deep, low bias, high variance | small, high bias, low variance |
| Training | parallel, independent | in sequence, each on the last error |
| Lowers | the variance | the bias |
| More trees | never hurt | overfit: use early stopping |
| Main hyperparameters | max_features, min_samples_leaf |
\(\eta\), number of trees, tree size |
| Tuning effort | small | larger; tune inside grouped CV |
| On our task, new image | 84 % | 82 % |
Tree ensembles are the default first model for a table: mixed units, thresholds, useless columns. The default is a hypothesis: Act 5 tests it.
Two minutes, in pairs.
Act 5 · Validate
Does it work on a new image?
Six models, two feature sets, two splits. Light bars: random 5-fold over the candidates. Red bars: leave one image out.
Raw features: 94 % on random folds, 69 % on a new image. Invariant descriptors: 87 % and 84 %. The gap measures the leak.
Random forest, leave-one-image-out, all four images. Rows: truth. Columns: prediction. Large numbers: share of the row (recall on the diagonal).
| New image, accuracy | all 11 | invariant 6 | ACSF 8 |
|---|---|---|---|
| majority class | 49 % | 49 % | 49 % |
| logistic regression | 80 % | 75 % | 82 % |
| tree, depth 3 | 57 % | 75 % | 74 % |
| random forest | 74 % | 84 % | 84 % |
| boosting | 67 % | 82 % | 84 % |
The rules
With all 11 features, logistic regression beats the random forest by 6 points. Image A alone: forest 49 %, logistic 63 %.
Two minutes, in pairs. What is the group?
max_depth on the same folds that give your final score. → Optimistic. Tune inside the training folds (nested CV), or keep one image out of everything.Act 6 · Explain
Which descriptors does the model use?
Random forest on all 11 features plus one column of pure random numbers (“noise”). Left: impurity importance. Middle: permutation importance on the training images. Right: permutation importance on a new image (bars: mean of four images; lines: spread).
Left: permutation importance on a new image for the six invariant descriptors, and again after adding one or two exact copies of d4 to the table. Right: accuracy when the feature is removed and the model is trained again.
d4 falls from 11 points to 2.5 to 0 when copies exist. Remove d1 or d4: −1 point. Remove both: −6 points.
1 One prediction, one sum: \(f(\mathbf x)=\phi_0+\sum_j\phi_j\)
2 \(\phi_0\) = the mean prediction over the background data.
3 \(\phi_j\) = the mean change of the prediction when feature \(j\) joins, averaged over all orders of the features (Shapley value).
4 TreeSHAP: exact for trees in polynomial time. No sampling.
Each point: one candidate of image B. Colour: the value of the descriptor.
What SHAP and importance tell you
bg, amp): a leak detectorWhat they do not tell you
A model can be right for the wrong reason. Only an experiment or a physical model tests a cause.
Two minutes, in pairs. True or false, and why?
bg has the highest permutation importance on the training images, so I must keep it.” → False. On a new image it is zero: it identifies the image.d1 has a low permutation importance, so I can delete it.” → Not safe. d4 holds the same information. Test the group.d1 causes a film column.” → False. It shows what the model uses. Cause needs an experiment.1 Classical ML needs a feature table: rows = objects, columns = descriptors. Detection errors propagate into it.
2 Column finding: smooth → local maxima → 2-D Gaussian fit. Precision ∝ 1/√dose; Poisson likelihood at low dose.
3 Good descriptors are invariant (translation, rotation, permutation, gain): ratios like \(I_\text{rel}\), distances, ACSF/SOAP.
4 Tree: greedy split with the largest \(\Delta = I(\text{parent})-\frac{N_L}{N}I(L)-\frac{N_R}{N}I(R)\). No scaling, no extrapolation, high variance.
5 Average of \(B\) trees: variance \(\rho\sigma^2+\frac{1-\rho}{B}\sigma^2\). A random forest lowers \(\rho\) with random feature subsets.
6 Boosting = gradient descent in function space: fit a small tree to \(-\partial\mathcal L/\partial F\), add \(\eta h\). Early stopping.
7 Baseline ladder on identical grouped folds: majority → linear → forest/boosting. Good descriptors help the linear model.
8 Cross-validate by image, session or specimen. Features that identify the image inflate random-split scores.
9 MDI is biased (noise gets credit). Permutation on held-out groups; permute correlated features together.
10 SHAP: \(f(\mathbf x)=\phi_0+\sum_j\phi_j\), exact for trees. It explains the model, not the cause; it depends on the background.
Notebook (molab): notebooks/unit05_features_trees.py: “From an image to a classifier you can explain, on real HAADF-STEM data”
I_rel, d1 and psi4 yourself and test their invariance.Ten exercises with automatic checks; CPU only, NumPy and Matplotlib. Sign in to molab → Fork → your own copy.
Reading: Breiman, Random forests Breiman, Leo, (2001); Grinsztajn et al. on trees and tabular data Grinsztajn, Léo et al., (2022); Lundberg et al. on TreeSHAP Lundberg, Scott M. et al., (2020).
Today we designed the descriptors by hand. Next week the model learns its own features, and it must beat today’s ladder on the same grouped folds.
Material outside the 90-minute path, for questions and self-study.
| Library | Idea | When to use |
|---|---|---|
HistGradientBoosting* (scikit-learn) |
histogram bins, native missing values | default; no extra dependency |
| XGBoost Chen, Tianqi et al., (2016) | second-order steps, L1/L2 on leaf values | large tables, GPU |
| LightGBM | leaf-wise growth, very fast | very large tables |
| CatBoost Prokhorenkova, Liudmila et al., (2018) | ordered boosting, categorical features | many categorical columns |
Tuning recipe: (1) η = 0.05 to 0.1; (2) the number of trees by early stopping on grouped validation data; (3) tree size (max_leaf_nodes 8 to 64); (4) min_samples_leaf, L2; (5) subsampling of rows and columns.
A tree splits along the axes of the table. A boundary that is diagonal in the feature space needs many steps.
Out-of-bag (OOB)
Regression trees
| Failure | Sign | Cure |
|---|---|---|
| threshold too low | maxima in the vacuum and in amorphous regions | raise the threshold; classify the candidates (today) |
| σ too large | close columns merge (dumbbells) | smaller σ; fit two Gaussians |
| σ too small | several maxima for one column at low dose | larger σ; a minimum distance |
| image border | cut neighbourhoods, wrong descriptors | discard a margin of one neighbour distance |
| scan distortion, drift | distances change along the slow scan axis | average fast frames; correct the drift first |
| thickness gradient | intensities change across the image | local normalisation (I_rel) |

©Philipp Pelz - FAU Erlangen-Nürnberg - Data Science for Electron Microscopy