The public dataset
The RefleX dataset has 6,311 diffraction images, seven anomaly labels, official splits, and many images with more than one anomaly.
The task: multi-label XRD anomaly detection
Input one diffraction image. Return every anomaly label that applies.
Preparing scientific images for foundation models
Dynamic range choices matter. The training and inference preprocessing should be the same.
clip, log-scale, normalize
use the same transform for training and inference
- Detector counts span orders of magnitude
- Raw values can exceed 10,000
- Natural-image encoders expect different image statistics
- Convert 16-bit detector images before modeling
- Use a fixed range for comparability
- Percentile normalization can reduce outlier effects
- Use log10(x + epsilon) when needed
- Compress bright peaks
- Expose weak features humans inspect
Notebook 1: few-shot prompting
Send diffraction images and text examples in the prompt. The model weights never change.
- Prompt design
- Output parsing
- Effect of adding support examples
- Set OPENAI_API_KEY before running
Random support examples can miss important labels. A stronger version would choose examples by similarity and label balance.
Notebook 2: frozen features + light classifier
Encode the images once with CLIP, then fit kNN or a linear classifier on the embeddings.
- Frozen CLIP ViT-L/14-336 encoder
- Mean-max pooled patch-token embeddings
- Heads: kNN, one-vs-rest logistic regression, low-shot logistic regression
- The encoder is reused rather than retrained
0.819 macro-F1 on full RefleX test set
0.789 macro-F1 with validation-tuned threshold
Notebook 3: LoRA fine-tuning of Gemma 3 4B
Fine-tune Gemma 3 4B with QLoRA, PEFT, and TRL. In this run, extra few-shot examples hurt after SFT.
- LoRA r=16, alpha=16, dropout 0.05
- 1 epoch, lr 2e-4, gradient accumulation 4, MAX_SOFT_TOKENS=280
- Accept the model terms before downloading
- A free-tier T4 GPU is not enough for this run
After SFT, extra in-context examples made this run worse. Measure the adapted model both with and without examples.
Simpler methods can be better
On this benchmark, frozen CLIP plus kNN is simpler and stronger than the more complex alternatives tested here.
SFT and few-shot may not help each other
Prompting and fine-tuning do not automatically combine well. Test the combined setup before relying on it.
Glossary
Key terms from the tutorial, related ptychography references, and acknowledgements.
Glossary
A quick reference for the main terms used above.
Ask the model directly, no worked examples.
Include a handful of worked examples in the prompt.
A numeric summary of an image or text.
The model part that turns input into a representation.
Predict from the most similar labeled examples.
A simple rule drawn in feature space.
Updating model weights on your task.
A small trainable add-on instead of full retraining.
Lower-precision storage to save memory.
Related samples in train and test that inflate scores beyond real-world performance.
Pool-then-score (micro) vs. score-then-average across classes (macro).
Supervised fine-tuning on labeled input–output pairs.
LoRA on a quantized base model, often small enough for a single GPU.
Out-of-distribution: inputs unlike what the model trained on.
For ptychography examples beyond this XRD tutorial, see arXiv:2511.02503 and arXiv:2410.09034, which show how LLMs and VLMs can be adapted to ptychography tasks and how agentic workflows can be built around tested model recipes.
Acknowledgement: this research used resources of the Advanced Photon Source, a U.S. DOE Office of Science User Facility operated under Contract DE-AC02-06CH11357, with funding support from Argonne LDRD 2023-0049. Thanks to APS MIC/CAI/SDM groups, USC Shao Group, Rice Han Group, and Cornell Muller Group.