Training a small model on our data: from 21 to 90 out of 98
By Karim Fagri, founder of Digifinance · · 4 min read
In our previous article, Luna graded our assistant's answers perfectly: 98 out of 98 cases. But Luna is a paid API, hosted outside our servers.
A simple question: can we teach this job to Laya, a small free model that runs on our own servers?
What we did
The job is always the same. The model gets an answer and a fact. It must say whether the answer states the fact, contradicts it, or doesn't mention it.
We prepared two sources of examples. First our own: 2,674 cases taken from our documents. Then a public dataset: 30,000 Wikipedia sentences, each one checked.
Training took 10 minutes on a free cloud GPU. We then tested each version on the same 98 cases as Luna.
The results
The longer the bar, the better. In red, serious mistakes: a fact the answer doesn't state, credited anyway.
Serious mistake: a fact the answer doesn't state, credited anyway. Two identical trainings vary by about 3 cases.
Our own examples make all the difference: Laya goes from 21 to 87 cases passed. Public data alone makes it careful, but it doubts too often.
The best result combines both: first the public data, then ours. 90 cases passed out of 98.
Not a replacement yet
Laya still makes one serious mistake: it credits a negative sentence as if it said the opposite. For a grader, that is exactly the mistake to avoid.
Another limit: two identical trainings give results about 3 cases apart. So the gap between 87 and 90 is not proven yet. Next step: repeat each training several times, and add negation examples.
What it costs
Once trained, Laya runs on an ordinary server, with no graphics card. It always gives the same result.
Luna already costs less than half a cent per full test. So the real gain is not money: it is keeping your data at home and depending on no provider.
What we take away
Three lessons that apply beyond our case:
- A small free model can learn a precise task, if you give it your own examples.
- Public data helps, but doesn't replace yours.
- Always test the model on cases it has never seen, and count the serious mistakes, not just the score.
Method and limits
A small test, to read as a trend.
- Model: Laya, "typed-decisions" variant, trained with a light adapter (LoRA), October 12, 2026.
- Our examples: 2,674 cases. 2,245 were written and labelled by an AI (Claude) from our documents; 429 are real answers from our assistant, labelled by Luna. None of the 98 test cases was used for training.
- Public data: 30,000 pairs from the VitaminC dataset (Wikipedia sentences, CC BY-SA 3.0 licence).
- Test: the 98 checks from our first article. A fact is credited only above a probability of 0.8.
- Response time: about 2.6 s on an ordinary processor (6 threads), 1.2 s on a laptop's small GPU.
Frequently asked questions
Do you need a GPU to train a model?
For training, yes, but a few minutes are enough, and a free cloud GPU will do. To use it afterwards, an ordinary server is enough.
How many examples do you need?
In our case, about 2,700 targeted examples were enough to go from 21 to 87 cases passed out of 98. What matters most is that they look like your real data.
Want a model trained on your data?
We bring AI into the tools of companies in Morocco, and we test it on your data before it goes into production.