Where foundation models fit in
Foundation models are deep-learning models trained broadly once, then adapted many times for specific tasks.
What foundation models can do
A pretrained model can help with routine analysis, natural-language interfaces, and discovery workflows when the task is well defined.
Automate routine tasks
Summarize, classify, extract, write code, and check repeated analysis steps.
Create natural interfaces
Let scientists ask for analysis, metadata, and visual explanations in normal language.
Accelerate discovery
Combine search, simulation, and experiment loops around measured outputs.
Pretraining is done once. Adaptation happens in the lab.
Model providers usually handle broad pretraining. Scientific users decide how the model is adapted, tested, and used.
model learns broad patterns from large datasets
choose prompts, examples, features, or adapters
use only after behavior has been measured
- Domain mismatch
- Hallucinations
- Hidden failure modes
- Benchmark first
- Adapt if needed
- Scaffold with workflows
The adaptation landscape
This map compares strategies by how much they change the input, how much they change the model, and how much work they require.
Zero-shot
jump to sectionDescribe the task and ask the model. Model weights do not change. Useful as a baseline even when you expect it to fail.
- labels
- 0
- compute
- API or local CPU
- iteration
- seconds
- fast triage
- format testing
- early prototyping
- unstable formats
- hallucinated labels
- weaker domain grounding
Click a strategy to compare its costs. More expensive is not automatically better: move to heavier methods only when simpler ones stop making real progress.
Compute and memory trade-offs
Labels, compute, and iteration time change quickly as you move from prompting to adapters and full fine-tuning.
Strategy 0: zero-shot prompting
A good prompt states the task, goal, context, and constraints clearly. Start simple before adding fragile prompt tricks.
Given this diffraction image, identify every anomaly that applies.
What should the model do?
What counts as a useful answer?
What domain facts or label definitions matter?
What constraints, units, or format should be respected?
- Fast triage
- Format testing
- Early prototyping
- Unstable formats
- Hallucinated labels
- Weaker domain grounding
Ask for structured outputs
For automation, the answer needs to be easy to parse and check. A schema helps downstream code trust what it receives.
response_format={"type": "json_object"}
{
"wavelength_A": 0.9763,
"detector_distance_mm": 250.0,
"beam_center_px": [1024, 1024],
"exposure_s": 0.5
}Simple and low-boilerplate, but you still need to validate the fields.
class XRDMetadata(BaseModel):
wavelength_A: float
detector_distance_mm: float
beam_center_px: tuple[int, int]
exposure_s: float | None = None
response_format=XRDMetadataMore setup, but the response is easier to check and use in Python.
Few-shot prompting
Few-shot prompting puts worked examples in the prompt. The model weights do not change.
- Allowed label set
- Expected answer format
- Multi-label edge cases
- A model that cannot read the pattern
- Data leakage
- An evaluation split that does not match the task
Frozen features
A scientific image becomes an embedding vector. A small classifier then works on those saved vectors.
- Numeric summary of an image
- Similar images should be nearby
- The classifier learns from vectors instead of raw pixels
- Run the pretrained encoder once
- Save one vector per image
- Train a small head on vectors + labels
- Evaluate with the same split
Example clustering of embeddings
If the encoder is useful, visually similar diffraction patterns should sit near one another in embedding space.
kNN vs. linear probe
Both methods use the same saved embeddings. kNN predicts from nearby examples; a linear probe learns a simple boundary.
Both methods read the same embedding. kNN votes with the 5 nearest labeled points; the linear probe only checks which side of its boundary you are on. Try the pocket of anomalous points at the bottom left: kNN follows the local labels, the linear probe cannot.
Move toward the boundary, or into the bottom-left pocket, to find where the two rules split.
- Sensitive to noisy labels
- Needs similarity search at inference
- Does not give calibrated probabilities by default
- Needs enough examples per class
- Must be re-fit when labels change
- Does not show similar examples by itself
LoRA fine-tuning
LoRA keeps most model weights fixed and learns a small low-rank update for the task.
- Full fine-tune: update all weights
- LoRA: train adapter matrices
- QLoRA: quantized base plus LoRA adapters
- Rank controls adapter size
- Typical r = 8-32
- Larger r adds flexibility and overfit risk
- Swap adapters without replacing the base model
- Each task can have its own adapter
- QLoRA stores the base in 4-bit and trains adapters in float16
Fine-tuning tools
There are many post-training methods beyond this tutorial. Open-source tools make them practical enough to try.
These tools make experiments easier to run, but they do not replace clean splits, metrics, and ablations.