Internal benchmark on synthetic data · run run-20261022172936-test · 2026-10-22 · 3 models, same instruction
A10 · Account Aggregator statement · Jul to Sep 2026 · 52 lines
Suresh Joshi, Lucknow · employer Orbit Logistics India · declared ₹31,000 a month · asks ₹30,000 over 15 months at 22%
| Date | Narration | Debit | Credit | Model label |
|---|---|---|---|---|
| 07-03 | IMPS-695334554952-ORBIT LOGISTICS INDIA-SAL JUL | 31,000 | Salary | |
| 07-05 | ACH D- KSHITIJ NBFC LTD-EMI4997 | 2,400 | EMI to another lender | |
| 07-06 | IMPS-698092729807-SHWETA NAIK-SMPL-XXXXXXX5646-RENT | 5,600 | Rent | |
| 07-08 | UPI-SANTOSH SINGH-santosh@okhdfc-670695855445-SPLIT | 1,672 | To or from a person | |
| 07-10 | NACH DR MOOLYA CAPITAL 315883 | 1,600 | EMI to another lender | |
| 07-11 | ATW-465692XXXXXX3294-S1ANHB51-MYSURU | 2,000 | Cash withdrawal | |
| 07-11 | UPI-SHOPMANDI-shopmandi@ybl-687965123704-PAYMENT | 2,178 | Spend | |
| 07-12 | UPI-METRO MEDICALS-metromedicals@ybl-613058227440-PAYMENT | 1,389 | Spend | |
| 07-12 | IMPS-686813110206-SURESH JOSHI-OWN A/C TRF | 2,254 | Own account | |
| 07-14 | UPI-CITY WATER BOARD-billpay@ybl-694401550316 | 1,655 | Utility bill | |
| 07-14 | UPI-MOBILEPE RECHARGE-mobileperecharge@ybl-674743939912-PAYMENT | 1,382 | Spend | |
| 07-15 | POS 497828XXXXXX3479 BIGCART SUPERMARKET | 1,874 | Spend | |
| 07-18 | UPI-CITYCAB RIDES-citycabrides@ybl-679885019804-PAYMENT | 1,665 | Spend | |
| 07-24 | UPI-FRESHKART-freshkart@ybl-612097193999-PAYMENT | 888 | Spend | |
| 07-24 | UPI-QUICKBITE FOODS-quickbitefoods@ybl-644878673809-PAYMENT | 1,683 | Spend | |
| 07-26 | UPI-CHAI POINT 24-chaipoint24@ybl-625075677619-PAYMENT | 1,771 | Spend | |
| 07-27 | UPI-CITYCAB RIDES-citycabrides@ybl-637998128519-PAYMENT | 1,254 | Spend | |
| 08-01 | UPI-BIGCART SUPERMARKET-bigcartsupermarket@ybl-635964878299-PAYMENT | 2,312 | Spend | |
| 08-01 | UPI-CHAI POINT 24-chaipoint24@ybl-675858582463-PAYMENT | 1,462 | Spend | |
| 08-02 | UPI-SHOPMANDI-shopmandi@ybl-678992540901-PAYMENT | 760 | Spend | |
| 08-03 | BY SALARY ORBIT LOGISTICS INDIA AUG2026 | 31,000 | Salary | |
| 08-03 | UPI-QUICKBITE FOODS-quickbitefoods@ybl-686580143310-PAYMENT | 1,144 | Spend | |
| 08-04 | IMPS-623702812264-SHWETA NAIK-SMPL-XXXXXXX5985-RENT | 5,600 | Rent | |
| 08-05 | ACH D- KSHITIJ NBFC LTD-EMI7245 | 2,400 | EMI to another lender | |
| 08-09 | UPI REV-FRESHKART-678016222198-REFUND | 496 | Refund | |
| 08-10 | ACH D- MOOLYA CAPITAL-EMI7739 | 1,600 | EMI to another lender | |
| 08-10 | POS 402623XXXXXX7103 METRO MEDICALS | 1,831 | Spend | |
| 08-13 | POS 406346XXXXXX4270 CHAI POINT 24 | 2,013 | Spend | |
| 08-15 | UPI-INDANE GAS-billpay@ybl-606188711221 | 1,674 | Utility bill | |
| 08-16 | UPI-BIGCART SUPERMARKET-bigcartsupermarket@ybl-606872788025-PAYMENT | 295 | Spend | |
| 08-16 | IMPS-687916376278-SURESH JOSHI-OWN A/C TRF | 5,418 | Own account | |
| 08-18 | ATW-486250XXXXXX1755-S1ANMY87-BELAGAVI | 5,000 | Cash withdrawal | |
| 08-24 | UPI-FRESHKART-freshkart@ybl-642197797726-PAYMENT | 2,144 | Spend | |
| 08-24 | UPI-SANTOSH SINGH-santosh@okhdfc-635604474530-PAYMENT FROM PHONE | 765 | To or from a person | |
| 09-03 | IMPS-697219532169-ORBIT LOGISTICS INDIA-SAL SEP | 31,000 | Salary | |
| 09-04 | UPI-BIGCART SUPERMARKET-bigcartsupermarket@ybl-612396215647-PAYMENT | 1,812 | Spend | |
| 09-05 | NACH DR KSHITIJ NBFC LTD 168505 | 2,400 | EMI to another lender | |
| 09-06 | UPI-SHWETA NAIK-shweta@okaxis-RENT SEP | 5,600 | Rent | |
| 09-07 | UPI-QUICKBITE FOODS-quickbitefoods@ybl-654951627063-PAYMENT | 766 | Spend | |
| 09-08 | UPI-RAILYATRI TICKETS-railyatritickets@ybl-694459799374-PAYMENT | 1,383 | Spend | |
| 09-08 | IMPS-689339924696-SURESH JOSHI-OWN A/C TRF | 2,064 | Own account | |
| 09-10 | ACH D- MOOLYA CAPITAL-EMI3440 | 1,600 | EMI to another lender | |
| 09-11 | NEFT CR-SMPL0009876-MOOLYA CAPITAL-LN DISB 546255 | 40,000 | Loan from another lender | |
| 09-12 | UPI-CITY WATER BOARD-billpay@icici-634260726091 | 2,144 | Utility bill | |
| 09-12 | UPI-QUICKBITE FOODS-quickbitefoods@ybl-637257667165-PAYMENT | 466 | Spend | |
| 09-13 | UPI-MOBILEPE RECHARGE-mobileperecharge@ybl-660299021238-PAYMENT | 1,742 | Spend | |
| 09-16 | ATW-414417XXXXXX9049-S1ANNG11-JAIPUR | 5,000 | Cash withdrawal | |
| 09-18 | UPI-SANTOSH SINGH-santosh@okhdfc-639305629953-PAYMENT FROM PHONE | 2,467 | To or from a person | |
| 09-27 | POS 486268XXXXXX6956 FRESHKART | 1,002 | Spend | |
| 09-27 | POS 499751XXXXXX4174 BIGCART SUPERMARKET | 1,201 | Spend | |
| 09-27 | POS 420644XXXXXX2828 RAILYATRI TICKETS | 1,841 | Spend | |
| 09-30 | INT.PD:01-07-2026 TO 30-09-2026 | 60 | Interest |
Memo from Dvar small model's labels
refer- Salary a month
- ₹31,000
- Months with salary
- 3 of 3
- EMIs to other lenders
- ₹4,000
- This loan's EMI
- ₹2,306
- FOIR
- 20%
- Bounce charges
- 0
- Betting share of debits
- 0.0%
- New loans from others
- 1
Why: 1 new loan from another lender.
Memo from the analyst's labels
refer- Salary a month
- ₹31,000
- Months with salary
- 3 of 3
- EMIs to other lenders
- ₹4,000
- This loan's EMI
- ₹2,306
- FOIR
- 20%
- Bounce charges
- 0
- Betting share of debits
- 0.0%
- New loans from others
- 1
Why: 1 new loan from another lender.
What if the ₹40,000 loan credit from another lender were labelled salary?
Behind the answer
- Model
- dvar:glm-5.3-flash
- Time
- 8.69 s
- Cost per statement
- $0.0016
- Scorer
- 52/52 lines, same decision
Run run-20261022172936-test, 2026-10-22. Same instruction for every model, temperature 0. Cost is from tokens and list prices.
The sample policy, applied in code
- Decline: 3 or more bounce charges in 3 months, betting over 8% of debits, or salary under ₹15,000 a month.
- Refer to an analyst: FOIR over 55%, salary missing in any month, a new loan from another lender, or any bounce.
- Approve: everything else.
The model never sees this policy or computes a number. It labels lines; code adds them up and decides. A wrong label shows up as a wrong number in the memo, and the scorer catches it.
All 21 test statements, every model
| Model | Lines right | Same decision as the analyst | Statements with a critical failure | p50 | Cost per 1,000 statements |
|---|---|---|---|---|---|
| Dvar small modeldvar:glm-5.3-flash | 100.0% (933/933) | 21/21 | 0 | 6.20 s | $1.07 |
| Low-cost frontieropenai/gpt-6-luna | 100.0% (933/933) | 21/21 | 0 | 617 ms | $0.45 |
| Mid-tier frontieranthropic/claude-sonnet-5.5 | 100.0% (933/933) | 21/21 | 0 | 551 ms | $9.60 |
Run run-20261022172936-test.