I’m working on an AI-powered mobile keyboard application and want to add an on-device grammar correction feature using a small open-source LLM.

The expected behavior is straightforward: the user types a sentence or paragraph, and the model should correct grammar, spelling, punctuation, and minor sentence-structure issues while preserving the original meaning and writing style as much as possible.

For example:

Input:
“I have went to office yesterday but i didn’t completed the work.”

Expected output:
“I went to the office yesterday, but I didn’t complete the work.”

I’m trying to understand the best way to fine-tune a small LLM specifically for this task.

I would appreciate advice on:

• What should the fine-tuning dataset look like? Should I use pairs of incorrect_text → corrected_text, or an instruction/chat format?
• Roughly how many training examples would be needed before seeing meaningful improvement?
• Is SFT/LoRA/QLoRA appropriate for a task this focused?
• Should the dataset contain artificially generated grammatical errors, real-world typing mistakes, or a mixture of both?
• How should I prevent the model from unnecessarily rewriting sentences that are already correct?
• What evaluation metrics would be useful for measuring grammar correction quality while also measuring meaning preservation?
• For a mobile keyboard use case, would it make more sense to fine-tune a very small LLM, or use a smaller encoder/encoder-decoder model specifically trained for grammatical error correction?

The eventual goal is to run the model locally on mobile devices, so model size, inference latency, memory usage, and battery/CPU consumption are important constraints.

I’m particularly interested in hearing from anyone who has fine-tuned a small model for grammar correction, text rewriting, or another on-device NLP use case.

What training approach, dataset strategy, and model architecture would you recommend?

The smaller you go model-wise, the lower the performance will generally be, somewhat unavoidable, especially when it requires specialized topical knowledge to rewrite.

Language comprehension took terabytes of training data to impart and will generally be saturated, so there is not much to improve on in terms of “grammatical errors” by any fine-tuning training you can do - except for the exact form you want output to take without needing to prompt or lead-up about it.

Fine tuning device-sized models is beyond the scope of any OpenAI offering or their developer community, and OpenAI’s own API for fine-tuning their proprietary models is being shut down. Current AI, having been post-trained on instruction-following, can perform well with prompting.

The minimum side of small models from OpenAI ends at 20B with their open-source release last year:

Try to start here with a prompted task into a small local mobile model, and pay 0 compute for fine-tuning if unnecessary after evals:

That is - if your users can tolerate gigabytes of download for an AI keyboard app.