AI Bits #1: What a scientific image-classification score leaves out

Correction, 9 September 2026: this issue previously described a pretrained model as doing science with almost no training data and overstated what generated explanations establish. The original January publication date remains. This revision separates the source finding from wider claims.

What the paper reports

Stoppa and colleagues’ 2025 paper studies real-versus-bogus optical transient classification across Pan-STARRS, MeerLICHT and ATLAS datasets. Its abstract reports 93% average accuracy using Gemini with 15 examples and concise instructions, alongside textual descriptions. It also describes a second model assessing output coherence.

Those are the authors’ findings for that task, not an MLAI replication. The 15 examples are not the foundation model’s entire training history. A coherence check is not an independent ground-truth label. This revision checked the abstract and publication record; it does not verify every experiment or determine performance on a new dataset.

Separate three claims before repeating them

A 95% score that misses every target

Fictional arithmetic—not the paper’s data: imagine 1,000 labelled candidates: 50 genuine targets and 950 non-targets. A classifier labels every candidate a non-target.

It gets 950 of 1,000 labels right: 95% accuracy. But it finds zero of the 50 targets: target recall is 0%. Precision for target predictions has a zero denominator because there are no positive predictions; do not present it as a meaningful measured percentage without stating the convention used.

This does not show that the published study made this error. It shows why a headline score is insufficient by itself. Ask for class counts, the confusion matrix and the consequences of false positives and false negatives. A higher overall score need not answer the question you care about.

Exercise: turn an explanation into checkable observations

Fictional output: “This is a real target because the bright feature is compact and appears consistently.” Before accepting that statement, separate its parts:

• What input supports “compact”, and how is that property defined?
• Were multiple observations supplied to support “consistently”, or did the explanation invent them?
• Does the reference label agree, and who or what established it?
• What would count as contradictory evidence?

If the required input or expertise is missing, record that limitation. Do not generate a second confident paragraph to fill the gap. This is a reading exercise; it does not teach astronomical classification or certify a model explanation.

Copy a classification-result reading record

Paper/version and exact task:
Model and supplied examples versus pretraining:
Dataset and reference-label method:
Class counts and evaluation split:
Metric definition and aggregation:
False positives, false negatives and missing results:
Explanation claim and supporting input:
Independent checks and their limitations:
What cannot be inferred for another dataset:
Question to investigate next:

Use “not reported in the material I checked” rather than inventing a value. Keep training examples separate from evaluation cases, and record whether the full methods or only the abstract were reviewed. No account, model run or download is needed for this exercise.

Bring a better question to an MLAI discussion

Try: “Which errors are hidden by this average, and what evidence supports the explanation?” Bring the source and one uncertainty. Choose an event whose topic and experience level fit, then check its location or online format. Attending does not provide scientific validation or deployment approval.

Source and editorial scope

The linked abstract and publication record were checked on 9 September 2026. The confusion matrix, explanation example and record are editorial teaching material, not a new experiment. Previous product recommendations and unrelated general advice were removed to focus on interpreting this scientific result.

Frequently Asked Questions