HRH Projects: SLM with LLM Fallback

Recently I combined my earlier Intent Router project with a smaller local-first conversational model I have been experimenting with.

Here is a video showcasing the project:

The basic idea is:

Let the smaller, inspectable system handle requests when it is confident enough, and only fall back to a larger LLM when it actually needs one.

Instead of sending every message directly to something like gpt-oss:20b or gpt-oss:120b, the chatbot first attempts to understand, retrieve, plan, and answer the request locally.

For clarity, when I use SLM here I mean the smaller local model/system. It is not a miniature transformer or LLM. The local model is mostly a classical/hybrid conversational system built from retrieval algorithms, symbolic language processing, small trained statistical models, memory, planning, and deterministic grounding checks.

The local system

The local chatbot uses a mixture of traditional/classical techniques, including:

  • TF-IDF retrieval

  • BM25

  • character-based similarity

  • Bayesian/category evidence

  • symbolic language parsing

  • request decomposition and planning

  • grounding checks

  • session memory and entity tracking

  • ambiguity handling

  • example-trained repeat/revision detection

  • a small learned social-state transition model

  • a separate learned response-policy model

The retrieval system does not simply check:

if user_prompt == "What is Mars?"
    return "Mars is..."

Training records contain multiple example phrasings, answers, categories, and tags. A new question is converted into features and compared against the indexed records.

The current retrieval ranking combines several different signals rather than depending on an exact phrase match. The bundled configuration, for example, gives weight to TF-IDF, character similarity, BM25 and Bayesian evidence.

So these can all point toward the same knowledge:

What is Mars?
Describe Mars.
Tell me about planet Mars.

without requiring three separate hand-written if statements.

The bundled Mars record itself stores several input formulations connected to the same grounded answer.

What happens to a request?

A simplified version of the local processing path is:

User message
    ↓
Language / structural parsing
    ↓
Entity + reference resolution
    ↓
Request analysis
    ↓
Retrieval ranking
    ↓
Information-need planning
    ↓
Reference selection
    ↓
Grounded composition
    ↓
Adequacy + grounding checks
    ↓
Local answer OR LLM fallback

The language layer can recognise things such as:

  • emphasis

  • negation

  • exclusions

  • restrictions

  • corrections

  • comparisons

  • ordered requests

  • quoted text

  • hypothetical statements

For example:

Tell me about Mars, especially its *moons*.

does not just become a bag of words. The emphasized concept is retained as a focus signal and influences retrieval and planning.

Likewise:

Mars, not Venus.

can be treated as a correction rather than as two unrelated planet names.

Planning rather than simply retrieving one answer

For a simple request such as:

What is Mars?

one strong grounded record may be sufficient.

For:

Compare Mars and Earth and explain why Mars is red.

the planner can create several information needs:

1. Information about Mars
2. Information about Earth
3. Why Mars appears red

Those needs are retrieved separately and then combined into a final answer.

The planning system also tracks whether each need was actually resolved. A request is therefore not considered complete merely because something vaguely relevant was found.

The project currently supports Fast, Standard and Deep planning profiles. These change the available planning budget, number of query variations, retrieval passes, candidate inspection, synthesis and consistency checking rather than simply making the final answer longer.

Ambiguity

The system can also keep more than one interpretation of a question alive.

For example:

What is the largest country?

could mean:

Instead of pretending that one interpretation is definitely what the user meant, the planner can branch into different grounded interpretations and rank them separately.

A response can therefore give the most likely interpretation first and mention another credible interpretation:

By area, Russia is the largest country. If you mean population, India is the largest.

The uncertainty here is attached to what the user meant, rather than pretending the factual answer itself is uncertain.

Memory and entities

The local system also maintains a small symbolic entity graph.

For example:

My name is Alex.
My cat is called Bob.

should produce something conceptually like:

USER
 ├── name → Alex
 └── owns/associated-with → CAT
                           └── name → Bob

rather than the rather unfortunate interpretation:

USER.name = Bob
USER.type = cat

Properties are attached to resolved entities, allowing later questions such as:

What is my cat’s name?

to be answered from session memory.

The entity system is deliberately kept separate from general world knowledge, so simply mentioning “Mars” does not turn Mars into a personal session-memory object and override the normal science database.

Repeated requests

Another part of the experiment is repeat/revision intent.

If the user asks:

Tell me 1 fact about Mars.

and then sends exactly the same request again, the system can treat the repetition as evidence that the user wants another grounded answer rather than merely returning the same sentence forever.

The repeat classifier uses features such as:

These features feed a small trained model whose possible outputs include:

  • alternate answer

  • innocent repeat

  • correction

  • refinement

  • new request

  • accidental duplicate

The implementation trains this classifier from readable prompt-pair examples rather than defining the final classification with a chain of hard-coded behavioural rules.

For example, the training data distinguishes an exact post-answer repeat from a rapid duplicate submission, a spelling correction, a refinement, and a completely new request.

A small learned conversational state

There is also an experimental learned interaction-state system.

It tracks eight simulated variables:

  • frustration

  • patience

  • engagement

  • confusion

  • curiosity

  • satisfaction

  • surprise

  • rapport

These are not supposed to represent actual feelings.

Instead, the previous state plus generic interaction measurements are fed into a trained transition model. The resulting state is then fed into a separately trained response-policy model.

The idea is:

interaction features
        +
previous state
        ↓
trained transition model
        ↓
updated conversational state
        ↓
trained response policy
        ↓
presentation behaviour

So the architecture is hard-coded, but the actual mapping from interaction patterns to state changes is learned from editable examples.

I have tried to keep the distinction roughly as:

Hard-code the mechanism; train the behaviour.

Knowing when not to answer

One of the more important changes was separating:

“I found something similar”

from:

“I can actually satisfy the request.”

If the user asks:

Write some p5.js code for movement with a blue circle.

the local database might contain information about p5.js.

A basic retrieval system could find:

p5.js is a JavaScript library for creative coding…

and confidently return it.

That is relevant, but it does not satisfy the requested task.

The local system therefore calculates things such as:

These are combined into a separate local adequacy/certainty value. The current fallback system checks this broader certainty rather than relying only on retrieval similarity.

So:

"Write p5.js code"

requires something compatible with:

action = generate
output = code

A retrieved:

action = define
output = prose

record may be related to p5.js, but it does not count as an adequate local answer.

LLM fallback

This is where the earlier Intent Router is now used.

When the local model decides it cannot adequately handle a request, it asks Pattern Processor to determine what kind of model should handle it.

The router estimates:

It then compares these requirements with compatible models in the registry.

A relatively easy request might therefore be routed to gpt-oss:20b or a smaller Qwen model, while a more demanding reasoning or coding task may justify gpt-oss:120b.

The largest model is no longer rewarded simply for being the largest model. Apparently even AI infrastructure eventually discovers budgeting.

Routing is visible before execution

Fallback is deliberately not automatic.

If the local chatbot needs another model, it displays a route panel containing:

The user can also select another compatible model from the panel.

Nothing is sent until Submit is pressed.

The routing and execution stages are intentionally separate. In the current implementation, Hybrid Lab prepares the custom route first and only executes it after explicit submission.

This makes it possible to inspect the system’s reasoning before committing to the model request.

Uncertainty and non-examples

The router also supports non-examples.

For instance:

Make the zibble do the other thing in the usual way.

should not confidently become “code generation” simply because some nearby training examples contain words such as make or thing.

Non-examples help describe the boundary of an intent rather than only showing the router what positive examples look like.

If the router cannot determine what a request means reliably, it can abstain and ask for clarification instead of selecting an arbitrary external model.

This gives the system two separate uncertainty checks:

Local model:
"Can I answer this adequately?"

Intent Router:
"If not, do I understand what kind of model should handle it?"

If both are uncertain, the result is clarification rather than random delegation.

Models currently in the registry

At the moment I have models such as:

  • GPT-OSS-20B

  • GPT-OSS-120B

  • Qwen3-Coder 30B

  • DeepSeek-R1 32B

  • Qwen3-Coder-Next Latest

with more planned as I continue developing the project.

For example:

The aim is eventually for the router to select not just between different-sized text models, but between different types of model entirely.

Why combine the two systems?

The part I find interesting is that the LLM is no longer automatically the first system in control.

The rough architecture is:

User
  ↓
Local conversational system
  ↓
language understanding
retrieval
planning
memory
grounding
adequacy check
  ↓
Confident?
  ├── Yes → local answer
  │
  └── No
       ↓
   Intent Router
       ↓
   Suitable model?
   ├── Yes → user approves route → LLM/specialised model
   └── No  → clarification

So the smaller system acts as the conversational controller, while larger generative models become specialised fallback resources rather than the default answer to everything.

The local model itself is also considerably more than a scripted:

if prompt == X:
    return Y

chatbot.

The knowledge is still explicitly trained and inspectable, but the path from a new input to a response can involve fuzzy retrieval across several algorithms, linguistic parsing, interpretation branching, entity resolution, learned repeat classification, information-need planning, candidate ranking, grounding audits, adequacy calculation, and finally optional external routing.

That inspectability is one of the main reasons I have enjoyed experimenting with it. When something goes wrong, I can usually inspect which stage went wrong, rather than only seeing a final generated answer.

It is still very experimental, but merging the two projects has made it a much more interesting test of local-first conversational processing with selective LLM delegation.

I would be interested in thoughts on the certainty calculation, fallback boundary, model ranking, training/non-example design, evaluation, or cases where a smaller local system should attempt a task before involving a larger model.

Or, really, any feedback on the architecture.