Post-training an open-source classifier for Indian banking

The decision a bank makes a million times

Every message that reaches an Indian retail bank's contact centre has to be filed before anyone can act on it. A card that has not arrived goes to dispatch, who can pull up a courier docket. Cash that left the account without leaving the ATM goes to reconciliation, who can raise a dispute against the switch. A card the customer thinks was stolen goes to fraud, and it goes there in seconds, because every minute the card stays live is a minute someone else can spend on it.

Banking77 encodes exactly this decision. It is a public set of 13,083 real customer messages to a digital bank, each tagged with one of 77 reasons for writing in. The reasons are the queues. PolyAI released it in 2020 under CC-BY 4.0 and it is still the standard test for this kind of routing. The dataset is on Hugging Face.

The model's whole job is to pick one of the 77. It writes no reply, looks nothing up, and never speaks to the customer.

Four rows from the held-out split, in the original English and in the Hinglish twin we generated, with the label each one carries and what the queue behind that label actually does.

What comes back

A label and a number. Nothing else.

{
  "answers": {
    "intent": {
      "choice": "pending_cash_withdrawal",
      "confidence": 0.94,
      "probabilities": {
        "pending_cash_withdrawal": 0.94,
        "declined_cash_withdrawal": 0.03,
        "wrong_amount_of_cash_received": 0.01
      }
    }
  }
}

The number matters as much as the label. A bank routing on this needs to know when the model is guessing, so it can hold anything under a threshold for a person to read. A model that says 0.94 and is right 94% of the time is usable. A model that says 0.94 and is right 70% of the time quietly fills the wrong queues, and nobody finds out until complaint volumes double.

Customers do not write in English

The English version of this problem is close to solved. Sentence embeddings with a logistic regression head will get you into the low nineties, and have been able to for years. English on its own is the wrong test for India.

Hinglish is Hindi written in English letters with English words mixed through it, and it is how a large share of Indian customers type into a banking app or a WhatsApp thread. Mera naya card abhi tak nahi aaya is one way to say the card never arrived. Card abhi tak nahi aaya, order ek week pehle kiya tha is another. Neither is formal Hindi and neither is English, and a bank does not get to choose which one shows up.

Banking77 is English only, so we built the Hinglish half. Every message has a twin carrying the same request and the same label, typed the way a customer would type it. Running the four arms against both halves shows where the problem actually is.

The two arms that have only ever seen English fall apart. Kev drops from 81.5% to 59.7%, and the embedding classifier, which is the strongest arm in the whole test on English, drops from 93.4% to 65.4%. Jev gives up under two points and Sonnet gives up almost nothing, and both of them sit near 81% whichever language you hand them.

Small models that have only read English cannot read Hinglish. The two that can are wrong on one message in five, in either language.

Kev, and the closed model it copies

Kev is an open model published by Jared Palmer in September 2026. It is built on a frozen Qwen2.5-0.5B backbone, 494 million parameters, with a LoRA adapter and a small pointer head bolted on top. 9.3 million parameters are trainable, which is 1.9% of the backbone. The adapter and the head are Apache-2.0. The weights are public.

It does not generate text. You hand it a document and a set of typed questions, it runs one forward pass, and it reads a probability for every option directly off the head. Its API calls this a Choice question. A 77-way routing decision is a Choice question with 77 options.

Kev exists because of Jev. Jev 1.13 is TypeSafe's closed decision model, served over a public HTTP endpoint at /v1/systemone, and it is the model people in this corner of the industry are currently watching. Archer Hume worked out what Jev's architecture probably is. Palmer built that architecture in the open and made it answer the same API contract. Kev and Jev accept the same request body and return the same response shape, so either one sits behind the same line of application code.

That is what made this a clean comparison. The closed model everyone is watching, and the open reproduction of its architecture, on identical inputs, with one difference between them: we trained ours on the job.

Why a Choice model is cheap

Nothing gets written. A prompted model receives all 77 labels and their descriptions on every call, reasons over them, and emits an answer, so you pay for the label card once per message and for the tokens it writes back. Kev prefills the message and the option list and reads the answer off the head. That is where 65 milliseconds and ₹1.30 per thousand come from, and no amount of prompt engineering closes that gap for a model that has to decode.

The data

We used the official split: 10,003 messages to train on, 3,080 held back to test. Out of the training rows we carved a stratified 1,001-row validation set with seed 42, leaving 9,002. The 3,080 test messages were never trained on, never validated against, and never consulted while picking hyperparameters.

Every message got a Hinglish twin under the same row id and the same label. Grok 4.6 wrote the 3,080 test twins. GPT-4o wrote the 9,002 training twins and the 1,001 validation twins, because Grok 4.6 cannot turn thinking off and ten thousand rewrites with thinking on was not worth the bill. Each row records which model wrote it, and no row in the test file was touched by a second generator.

The rewrite protocol was written down before generation and enforced after it.

• Spoken Hinglish, roman Hindi mixed with English, the way someone types into a bank app or a WhatsApp thread. Not formal Hindi and not a word-for-word translation.
• Leave UPI, ATM, PAN, FD, NEFT, IMPS, RTGS, OTP, CKYC, KYC, PIN, CVV, IFSC and Aadhaar exactly as they appear. Nobody writes these out in Hindi.
• Throw out any rewrite that came back as English with a few Hindi words dropped in.
• Throw out Devanagari. Customers type in Latin script.
• Throw out anything where the request changed. If the rewrite of a refund complaint reads like a card complaint, the label is now wrong and the row will teach the model something false.

200 rows from each split were read against those rules before the split was promoted.

The ATM refused the card, the money never came out, and the app still shows the withdrawal as pending. Both rows carry pending_cash_withdrawal.

The 77 label strings are untouched, including the two that look like typing mistakes. Refund_not_showing_up has a capital R and reverted_card_payment? ends in a question mark. Tidying them up would make our numbers incomparable with every other published Banking77 result.

This is a proxy for a bank's inbox. The Hinglish is machine-written from English originals, and real traffic arrives with typos, half sentences, account numbers pasted mid-message and three complaints in one paragraph. Anyone taking this to production should rebuild the twin set out of their own tickets. What the dataset does support is the question we were asking, which is what a short fine-tune does to a small open model.

The fine-tune

Kev takes labelled Choice records, so there is no chat formatting, no instruction tuning, and no completions to grade. A training row is the message, the 77 option keys, and the right answer.

We started from the published kev-0.5b weights rather than from bare Qwen, and carried on training the same LoRA and pointer head. Two epochs over 18,004 rows: 9,002 English messages and their 9,002 Hinglish twins. Batch size 4 with gradient accumulation 4, bf16, cross-entropy loss and nothing else. No permutation KL, no ordinal weighting, no none-of-the-above option, no injected distractors. One NVIDIA L4.

The option list Kev sees is the bare 77 keys with no descriptions, because 77 options with one-line glosses does not fit inside the 1,024-token budget Kev allows per question branch. The prompted Sonnet arm does get the descriptions, along with a frozen system prompt telling it that it is the intent router for a large Indian retail bank contact centre and that the customer may write in English or in spoken Hinglish. The small model went in with less help than the large one.

What training bought

Two epochs moved Hinglish accuracy from 59.7% to 88.1%, a gain of 28.4 points, and English accuracy from 81.5% to 92.7%.

English holding is the result that makes this usable. A bank cannot run one model for customers who write in English and a second for customers who write in Hinglish, because it does not know which it is getting until the message lands. Training on both languages at once cost nothing on either.

Two other numbers moved in the same direction.

The confidence number started telling the truth

Expected calibration error on Hinglish fell from 0.122 to 0.044. The published model was confidently wrong: it would report high probability on labels it was getting wrong more than a tenth of the time. After training, the number it reports is close enough to its actual hit rate that a bank can set a threshold on it and send whatever falls below to a person. Prompted Sonnet sits at 0.111 on the same measure, which is roughly where untrained Kev started.

It learned the rare queues too

Macro-F1 on Hinglish went from 58.5% to 88.2%. Accuracy on its own can hide a model that has learned the twenty queues carrying most of the traffic and given up on the other fifty-seven. Macro-F1 weighs all 77 labels equally, and here it moved almost as far as accuracy did.

Sonnet's two numbers pull apart instead. Its Hinglish macro-F1 is 79.6% against 81.2% accuracy, and its English macro-F1 is 72.5% against 81.4% accuracy. That gap is the signature of a model doing well on the common intents and worse down the tail, which is where the awkward queues live.

The control

We ran a boring baseline to check that the finding is about the data rather than about Kev. BGE-small-en-v1.5 embeddings with a logistic regression head is the ordinary answer to a 77-way text classification problem, and on English it is the strongest arm in the whole test at 93.4%. On Hinglish the same head drops to 65.4%. Retrain it on both languages and it recovers to 87.4% while English holds at 93.2%.

The shape of the result does not depend on Kev's architecture. Any model that has only seen English falls over on Hinglish, and training on Hinglish fixes it. Kev is the arm worth shipping because it is the one that also answers the API contract and returns in 65 milliseconds.

The bill

Put accuracy against what the run actually cost and the four arms separate cleanly.

Sonnet and Jev are billed usage, read off OpenRouter and converted at ₹83.50 to the dollar. Kev is GPU time: measured prefill on one L4 at $0.85 an hour, which is close to what India capacity costs, divided across the messages it handled. Both Kev arms cost the same to run, because training changed the weights and left the size and the shape of the model alone.

At a million messages a month, which is an ordinary volume for a retail bank contact centre, the routing line reads:

A million messages a month at the measured per-thousand rate. Rounded to the nearest hundred rupees.

Latency runs along the same lines. 65 milliseconds median for the fine-tuned model, 191 for Jev, 2,099 for Sonnet. Routing in 65 milliseconds finishes before the ticket does. Routing in two seconds is a queue the rest of the system has to be designed around.

These are working numbers from a measured run. GPU rates, batch sizes and utilisation all move them, and none of it is a price quote.

What this does not show

Sonnet did not answer every row

Sonnet covered 3,026 of the 3,080 Hinglish messages and 2,746 of the 3,080 English ones. The prepaid key hit its $20 cap partway through. We left the prompt frozen rather than trimming the label card to squeeze the rest in, because a shorter prompt would have turned the remaining rows into a different test. Every other arm ran on all 3,080.

We prompted Sonnet rather than training it

A frontier model fine-tuned on Banking77 would score higher than a prompted one. It would also move the bill above ₹83 per thousand rather than below it, so the quality comparison gets harder and the cost comparison gets easier.

The English half may be contaminated

Banking77 has been public since 2020. Sonnet, Jev and the Qwen backbone under Kev may all have seen those messages in pre-training, and we have no way to check. The Hinglish twins were generated in September 2026 for this work and have never been published anywhere, so the Hinglish column carries the least contamination risk. It is also the column with the widest gap between the fine-tune and everything else.

Two models we did not run

Claude Fable and GPT-5.6 are absent. Each would need a full run against the frozen prompt to be comparable, and publishing four arms we measured is better than publishing six where two are estimates.

77 queues is small

A real bank runs more queues than this, and Banking77 covers one bank's product line rather than yours. A production version starts from your queue list and your tickets, and the first honest number comes off a held-out set built out of them.

Doing this on your own work

Everything about this was small. One public dataset, a language twin we generated, 9.3 million trainable parameters, two epochs, one L4. The parts that took judgement were deciding what exactly the model had to decide, writing the Hinglish protocol tightly enough that the rewrites stayed honest, and holding the test set still while everything else moved.

Most enterprises are already running a decision like this somewhere. A queue, a form field, a review step, a risk flag. It fires thousands of times a day, the answer comes off a list somebody wrote down years ago, and today it is either being paid for at frontier rates or done by hand.

If you have one, write to us with the decision and roughly how often it happens. We will tell you what it scores today before anybody talks about training anything.