nodeology.ai
MRS Spring 2026, Part II of III

Foundation Models Adaptation II: evaluation is part of the method

A model result is only useful if the evaluation matches the scientific question: splits that avoid leakage, the right F1 for the label balance, and lighter checks for tasks that are not classification.

01

Train, validation, test - and why it matters

Use training for fitting, validation for choices, and test once for reporting. Split by physical object when related images could leak information.

80%
Train

fit parameters

10%
Validation

choose thresholds and settings

10%
Test

report once

Split by physical object or experiment when related images may share information.
Loss curve intuition
earlybest checkpointlatetrain lossvalidation loss

Training loss can keep improving while validation loss turns upward. Report the held-out test result after model choices are finished.

02

Confusion matrices and F1

Precision, recall, and F1 all come from the same four counts. Choose the metric that matches the scientific cost of mistakes.

Model scores vs. decision threshold
threshold 0.50

Drag the threshold. Coral dots are mistakes: positives that fall below it (FN) and negatives that clear it (FP). Every metric on the right is just a different way of counting them.

TP19

predicted +, actually +

FP9

predicted +, actually -

FN5

predicted -, actually +

TN27

predicted -, actually -

precision
0.68
recall
0.79
F1
0.73
Figure 3 One threshold, four counts. Move the decision threshold and watch TP, FP, FN and TN redistribute, dragging precision, recall and F1 with them.
03

Multi-label metrics: micro-F1 and macro-F1

Micro-F1 pools all decisions. Macro-F1 makes rare labels visible.

Example image
ground truth
ice ring, halo
prediction
ice ring
exact match?
no, but not necessarily scientifically useless
Per-label F1 on a synthetic test set
ice ring40 instances0.90
halo30 instances0.85
loop scattering6 instances0.44

The two common labels stay fixed. Only the rare label degrades, and the two averages tell very different stories about it.

micro-F1
0.85

Pools all decisions first. Frequent labels dominate, so it barely notices the rare label failing.

macro-F1
0.73

Averages per-label scores. The rare label carries a full third of the weight, so its failure is visible.

Thresholds are part of the method. Lower thresholds catch more anomalies but create more false alarms; higher thresholds are cleaner but miss subtle cases. Tune on validation, not test.
Figure 4 Micro vs macro F1. Slide the number of missed rare anomalies; pooled micro-F1 barely moves while macro-F1 falls, because rare labels only count when averaged per class.
04

Non-classification metrics

Use the lightest evaluator that can catch the failure: schema checks, runnable tests, reference answers, grounding checks, or judges.

First question
What makes this output wrong?
Format fixed?

Use schema checks and parsers.

Code must run?

Run unit tests, examples, and integration tests.

Known answer?

Compare against reference outputs.

Must cite evidence?

Check grounding, retrieval, and source coverage.

Open-ended quality?

Use rubrics, human review, or LLM judges as a last layer.

Pick the lightest reliable evaluator before asking another model to judge the answer.