How to evaluate an AI answer: facts, completeness and style | AI Bits #2

Editorial correction, 9 September 2026: this issue previously described model choice as a mirror of personality and generalised about engineers and creators. Those claims went beyond the evidence. The original January publication date is retained; this revision focuses on comparing answers for an actual task.

What the small study found

Yunusov and colleagues’ August 2025 paper reports 32 participants, evenly distributed across four Keirsey types, using GPT-4 and Claude 3.5 for four collaborative tasks. Its abstract reports different preferences within those groups despite similar aggregate helpfulness ratings.

That is a result about this sample, task set and model comparison. It does not establish a rule for assigning people a chatbot, prove the validity of personality categories, or rank current models. The 29 October 2026 check also inspected the experimental setup: 32 university students recruited through university mailing lists and direct outreach; eight selected per Keirsey group. Sixteen used each model, giving four people per personality/model cell. Each person used one assigned model for four tasks in random order through an interface hiding the model name. Helpfulness ratings from this small student sample do not establish population-wide preferences. The underlying data were not independently analysed.

Try this without a model subscription

Fictional exercise; all event details and answers are invented. Read the source card first:

Harbour AI Reading Circle is an online discussion on 14 October at 6 pm Sydney time. It lasts 45 minutes. Registration is required. The notice does not say whether a recording will be available.

Task: write a brief reply explaining when and how to attend, and whether there will be a recording. Do not invent information.

Answer A

Join the online discussion on 14 October at 6 pm Sydney time. It lasts 45 minutes, and you need to register. The notice does not confirm a recording.

Answer B

We’d love to see you at the Harbour AI Reading Circle! Drop in online at 6 pm Sydney time on 14 October—no registration needed. Don’t worry if you miss it: a recording will be shared afterwards.

These are editorial examples, not generated model outputs. Decide which wording you prefer, then check each against the source before reading the comparison below.

You can prefer a warmer opening while rejecting Answer B as written. A useful revision would keep that opening, restore the registration requirement and say that recording availability is unconfirmed. Do not turn a preference for warmth into a personality label or a conclusion that the answer is more accurate.

A corrected warm answer

We’d love to see you at the Harbour AI Reading Circle online on 14 October at 6 pm Sydney time. The discussion lasts 45 minutes and registration is required. Recording availability has not been confirmed.

This editorial revision keeps the tone while restoring the facts. Compare it against every requirement in the source card. For numerical evaluation, the accuracy and recall example shows another way a single attractive score can conceal failure.

Use the distinction in a real comparison

1. Choose a permitted, low-stakes task with source material and clear requirements. Do not upload private material merely to compare tools.
2. Record the model/version, settings, date, prompt and relevant conversation context. Different histories are different conditions.
3. Set factual and task checks before reading the outputs. Record failures separately from preference ratings.
4. If willing participants compare outputs, hide model names where practical and vary presentation order. This is a bias-reduction step, not proof of an unbiased study.
5. Ask why an answer was preferred and what needs changing. Keep disagreement and uncertain judgments; do not manufacture a consensus.

One task cannot establish an overall winner. If you edit the prompt between runs, note the change rather than attributing every difference to the model. Keep failed outputs in the record instead of selecting only favourable examples.

Record preference without losing correctness

Task, source material and required facts:
Models/versions/settings/date (or editorial examples):
Prompt and conversation conditions:
Required checks defined before comparison:
Contradictions, omissions and unsupported additions:
Preferred presentation and reason:
What must change before this answer is usable:
Disagreement, uncertainty and comparison limits:
Permission to retain or share feedback:

No personality questionnaire is needed for this exercise. Ask what the person needs from the answer rather than collecting sensitive or unnecessary personal information.

Discuss one disagreement at an MLAI event

Bring a permitted example and ask: “Do we disagree about whether this is correct, or about how it is written?” Choose an event whose topic and level fit, and check online or in-person details. The fictional event above is not an MLAI listing or a registration invitation.

Source and limits

The abstract was checked on 9 September and the recruitment, model assignment and procedure sections on 29 October 2026. The answer pair, checklist and record are editorial teaching material, not a new experiment or evidence about named current products. Prior tool and book endorsements were removed to keep the issue focused.

Frequently Asked Questions