ChatGPT should verify factual and procedural answers instead of confidently filling gaps

Summary

I am raising what I believe is a systemic reliability problem with ChatGPT, rather than a complaint about one particular incorrect answer.

Over a number of conversations, I have repeatedly received answers that were presented as factual instructions or calculations, but which turned out to be incorrect, outdated, or based on assumptions that had not been verified.

The most concerning aspect is not that ChatGPT makes mistakes. Humans and software both make mistakes.

The concerning aspect is that ChatGPT can fill gaps in its knowledge with plausible information and present the result with the confidence and specificity of verified information.

This creates a situation where the user may have no obvious indication that the answer contains invented or assumed details.

I believe this is particularly problematic for procedural questions, current information, calculations, navigation, software instructions and other matters where the answer can be objectively verified.

The distinction I think ChatGPT needs to make

There is a fundamental difference between:

“What is your opinion?”

and:

“How do I do this?”

For an opinion, reasoning from available knowledge is appropriate. I am perfectly comfortable with an answer such as “I think X because…”

But when I ask:

“How do I enable this setting?”

the answer contains factual claims about the existence, location and behaviour of a particular setting.

Likewise:

  • “What route should I drive?”
  • “What does this airline programme currently allow?”
  • “What formula should I use in this spreadsheet?”
  • “Does this IPTV playlist work?”
  • “Will it snow in Oberon?”
  • “How do I configure this Android device?”

These are not primarily opinion questions. They are questions where research, current data, calculation, testing or verification are appropriate.

I should not have to know in advance that ChatGPT’s answer needs verification and explicitly instruct it to verify something that is objectively verifiable.

Examples from my actual interactions

1. VLC and IPTV

I recently asked whether there was a VLC setting that could turn VLC into a live-TV player.

ChatGPT initially explained that VLC could play IPTV playlists and provided IPTV-org playlist URLs.

When I reported that the Australian playlist only produced four channels, ChatGPT suggested another playlist. When that produced only two channels, it continued explaining possible reasons and eventually recommended a different IPTV application.

The problem was that the initial answers were given as though the behaviour of the playlists and VLC had been established.

Only after I challenged the answers did ChatGPT acknowledge that the information should have been verified against the current VLC Android implementation and the actual playlist.

This is a good example of the problem because the relevant information is objectively testable.

The system could have:

  • inspected the current IPTV-org playlist;
  • established what channels it actually contains;
  • checked current VLC Android playlist behaviour;
  • distinguished between an M3U playlist containing multiple streams and VLC’s ability to present those streams as a usable channel list.

Instead, it constructed a plausible explanation from general knowledge.

2. Road-trip planning and map routing

I have previously asked ChatGPT to plan road trips in Thailand.

On one occasion, the proposed driving times and distances were so inaccurate that I specifically challenged ChatGPT and asked it to check that the proposed routes were actually using real roads.

This is particularly frustrating because road routing is an objectively verifiable problem.

A proposed route should not be estimated from geographical intuition when current mapping information is available.

The appropriate process should be:

establish the actual route → calculate the actual road distance → establish realistic driving time → present the result.

Instead, the system produced a plausible-looking itinerary that contained incorrect distances/times.

I should not have to discover that by independently checking the route.

3. Android/Samsung instructions

I have also encountered this repeatedly when asking for instructions concerning Samsung/Android devices.

The problem is particularly obvious with Android because menus and settings vary according to:

  • Android version;
  • Samsung One UI version;
  • device model;
  • application version.

ChatGPT can nevertheless produce very specific instructions for a setting that either does not exist on the relevant version or is located somewhere else.

The instructions sound authoritative because they are presented as a sequence such as:

Settings → X → Y → Z

But if the model has not established that those menus exist on the user’s particular version, the specificity is misleading.

Again, this is something that should trigger verification rather than confident reconstruction from memory.

4. Spreadsheet formulas and calculations

I have also used ChatGPT for spreadsheet and Australian tax-related calculations.

There have been occasions where formulas or calculations required me to independently check the result and identify problems.

This is another category where the system should be able to do better.

If ChatGPT proposes a formula, it should be possible to test the formula against the relevant values before presenting it as correct.

If a tax calculation depends on current Australian tax rules, the current ATO rules should be checked.

A formula that “looks right” is not sufficient.

5. Current airline programme information

I have asked questions about Thai Airways’ Royal Orchid Plus programme, including changes to qualification and redemption rules.

These are inherently time-sensitive questions.

The correct behaviour should be to consult the airline’s current information before stating what the current rules are.

An answer based on information learned previously can easily be outdated while still sounding completely authoritative.

The common failure pattern

Across these examples, the problem is not random factual error.

The common pattern is:

  1. The model recognises the general subject.
  2. It has partial or outdated knowledge.
  3. There are gaps in the information.
  4. It constructs a plausible answer that fills those gaps.
  5. The answer is expressed fluently and confidently.
  6. The user has little indication that some of the information was inferred rather than verified.
  7. Only external checking reveals the problem.

This is particularly dangerous because plausibility is not the same thing as truth.

A response can be internally coherent and still be factually wrong.

Why I think this is a product/reliability issue

I don’t believe the solution should simply be:

“Users should ask ChatGPT to verify the answer.”

That places the burden on the user to recognise when ChatGPT might be hallucinating.

But the user often cannot recognise that.

If I know enough about VLC, road routing, Samsung settings or spreadsheets to recognise an incorrect answer, I can correct ChatGPT.

But someone who doesn’t have that knowledge may simply accept the answer.

That creates a particularly dangerous asymmetry:

The people most able to detect an AI error are often the least dependent on the AI’s answer.

The people who most need accurate instructions may be the least able to recognise fabricated instructions.

What I think should change

I am not suggesting that ChatGPT should browse the internet for every question.

That would be unnecessary and could introduce other problems.

Instead, I think the system needs to classify the nature of the question before deciding how to answer.

For example:

Opinion / subjective judgement

“Which car is better?”

Reasoning and opinion are appropriate.

Stable factual knowledge

“What is the capital of Thailand?”

A direct answer is normally appropriate.

Current factual information

“Who is the current CEO?”

External verification should normally occur.

Procedural/instructional information

“How do I enable this setting on my Samsung?”

Verification should be strongly preferred where the procedure may depend on software/device version.

Quantitative questions

“Calculate my tax.”

The calculation should actually be performed, and the underlying rules/data should be checked where they are time-sensitive.

Navigation/routing

“Plan a route from A to B.”

Actual road/network data should be used rather than estimated geographical distances.

High-consequence information

Legal, medical, financial and similar questions should have an especially high verification threshold.

I would like to see a stronger verification architecture

Ideally, the process would be something like:

Question → classify the question → determine whether verification is required → retrieve evidence/use appropriate tools → formulate answer → check answer against evidence → communicate uncertainty.

The system should also distinguish between:

  • Verified fact
  • Reasonable inference
  • Unverified information
  • Conflicting information
  • Unknown

That would allow ChatGPT to say:

“I believe this is the case, but I haven’t verified it.”

instead of:

“Yes, this is how it works.”

when the first statement is actually true.

The system should also test its own output where possible

This could prevent many errors.

For example:

  • A spreadsheet formula can be executed against sample data.
  • A calculation can be independently calculated.
  • A URL can be opened.
  • A current software procedure can be checked against documentation.
  • A route can be checked against actual mapping data.
  • An IPTV playlist can be inspected.
  • A legal case citation can be checked to establish that the case actually exists.

The system should ask itself:

“Can I cheaply test the thing I am about to tell the user?”

If the answer is yes, I believe it should generally do so.

The importance of uncertainty

I don’t expect ChatGPT to know everything.

In fact, I would much rather receive:

“I can’t verify that.”

than receive a detailed but fabricated answer.

An AI saying “I don’t know” is not a failure.

An AI confidently inventing an answer because it feels like an answer is the failure.

I think model evaluation should therefore measure not only whether the final answer is correct, but whether the model correctly recognises when it has insufficient evidence to answer.

In other words, epistemic calibration should be treated as a core reliability metric.

Why this matters beyond my own experience

I initially considered simply deleting ChatGPT from my devices and starting again with another account.

But that would not address the underlying problem.

Nor do I think this is unique to me.

Other users must encounter the same behaviour. Some will recognise it and become frustrated. Others may simply conclude that “AI is stupid” and stop using it.

But there is a more serious possibility: users may not realise that the answer is wrong.

That is particularly concerning when AI is used for:

  • legal information;
  • financial decisions;
  • medical information;
  • technical configuration;
  • education;
  • employment;
  • navigation;
  • research;
  • business decisions.

The more convincing the language becomes, the harder it can be for an inexperienced user to distinguish knowledge from fabrication.

This is not an argument against AI

Quite the opposite.

I believe AI could make a significant positive contribution to the information environment if it becomes a tool that helps people distinguish:

what is known → what is supported by evidence → what is inferred → what is uncertain.

That would be substantially more valuable than simply generating the most plausible answer to every question.

The internet already has enormous amounts of misinformation.

AI has the potential either to amplify that problem by generating plausible misinformation at enormous scale, or to help counter it by making reliable information easier to find, understand and verify.

I would strongly prefer the latter.

The question I would like OpenAI to investigate

I am therefore not asking for an individual response to my incorrect VLC answer.

I would like this treated as a systemic model/product reliability issue:

Why does ChatGPT sometimes provide highly specific procedural or factual answers from incomplete knowledge when the information is objectively verifiable and appropriate tools for verification are available?

And:

Can the system be improved so that verification is automatically triggered for appropriate classes of questions, rather than relying on the user to know when to ask for verification?

Finally:

Can evaluation of future models include practical tests of whether instructions actually work, rather than simply whether the generated answer appears plausible?

I would be very interested to know whether OpenAI is already investigating this particular failure mode, how it is being measured, and what mechanisms are being developed to reduce it.

I am not expecting perfection.

I am asking for something more fundamental:

When ChatGPT does not know something, it should be much better at recognising that it does not know it. And when the answer can be verified, it should make substantially greater use of verification before presenting an answer as fact.