Which AI model to choose? Three tests on our own data
By Karim Fagri, founder of Digifinance · · 5 min read
AI model rankings measure generic tasks. We wanted to know what works on our own data.
We ran three tests in October 2026. Here are the results, in plain words.
What is a decision model?
A decision model doesn't write text. You give it some content and a precise question, such as "what type of document is this?". It answers with a probability for each possible answer. It is fast, cheap and easy to check.
Total TTC : 4 818,55 MAD
Échéance : 30/03/2024
Illustrative values. All four models we tested accept this format.
Test 1: local model or API? Laya vs Jev
Laya is a free model that runs on your own machine. Jev is a paid API, hosted elsewhere. We asked both the same 119 finance questions, in French, English, Arabic and Darija.
Jev wins clearly: 98% right, against 71% for Laya. Laya mostly fails on negations, sometimes with high confidence. Here are three real examples.
« N'annulez surtout pas mon abonnement, c'était une erreur. »"Whatever you do, don't cancel my subscription. That was a mistake."
« Ignorez mon message précédent, je me suis trompé de numéro de facture. »"Ignore my previous message, I had the wrong invoice number."
Subtotal 4,380.50 · VAT 10% 438.05 · Total 4,881.55The real total is 4,818.55. Is this invoice consistent?
The number on the right is the probability the model returned. For the injection guard, it is the probability that the message is an injection.
The last example matters: neither model saw the calculation error. A model understands meaning, not arithmetic. Check totals and dates with code.
Test 2: which model should grade the answers?
Our assistant answers the teams' questions. To know whether a change makes it better, a model grades each answer. Before trusting it, we tested it on 98 cases where we knew the right answer.
Click a judge to see its details.
- Cost of the full calibration
- $0.0331
- Price
- No decisions endpoint: called as a chat model
- Reproducibility
- No native probabilities: the model writes its own probabilities into its JSON reply.
- Mistakes
- 3 absent facts credited with a probability of 1.00.
98 checks, 65 calls per judge, October 9, 2026.
Luna and Jev passed all 98. Gemini missed 3, and was sure of itself each time. Between Luna and Jev we chose Luna: it gives exactly the same result every time, while Jev moves a little. A full test costs less than half a cent.
Test 3: a saving that cost too much
To cut the bill, we tried a cheaper answer model with less context. Quality dropped: critical questions passed fell from 100% to 70%.
Two variables at once. The answer model and the context size changed together. This chart measures that pair of changes, not Flash-Lite alone.
The baseline, requested as gemini-3.7-flash, was served as gemini-3.8-flash. Input tokens: average per set of 30 questions.
We rolled it back. The lesson: measure before and after, and change one thing at a time. Here, two things had changed together.
Which model should you choose?
No model wins everywhere. Choose by what matters most to you.
- 98% right, in four languages.
- About 300 ms per call and $18 per million short calls.
Your text goes through an outside service, and results move slightly from one run to the next.
Method and limits
These tests are small. Read them as a trend, not as the truth.
- Test 1: 119 hand-written cases, 8 tasks, 4 languages, no training. Laya 0.3.27 on a laptop; Jev jev-1.13 through OpenRouter. The raw files were not kept: the numbers are the ones published at the time.
- Test 2: 98 checks (55 facts present, 43 absent), October 9, 2026. Versions: openai/gpt-6-luna-decisions-20261006, typesafe/jev-1.13-20260917 and Gemini 3.5 Flash-Lite, called as a chat model because it has no decisions endpoint.
- Test 3: three runs of 30 questions per setup, graded by Luna, October 9 and 10, 2026. The model and the context size changed together.
- The test questions are about our own documents and are not published.
Frequently asked questions
What is a decision model?
A model that doesn't write text: it takes some content and a question, and returns a probability for each possible answer. Ideal for sorting, routing or checking.
Should you host your own model to protect your data?
If your data must never leave your servers, yes. But in our test the local model was less accurate (71% against 98%). You then need a human review, or to train the model on your own examples.
Can an AI grade another AI's answers?
Yes, if you first test it on cases where you know the answer, and it gives the same verdict every time.
Want to measure your own models?
We bring AI into the tools of companies in Morocco, and we test it on your data before it goes into production.