Does System One Transfer? Calibrated Typed-Decision Models, Label Budgets, and Translation for Indonesian Regional Languages
Calibrated typed-decision ("System One") models return a label and a probability in a single forward pass at a small fraction of the cost of large language models (LLMs), yet their evaluation has been almost entirely English.
This study evaluates the performance of calibrated typed-decision models (System One) on Indonesian regional languages, comparing their accuracy, cost efficiency, and transferability against large language models (LLMs) and multilingual baselines.
What happened
The article presents a pre-registered evaluation of System One models (Jev and Laya-multilingual) on Indonesian regional languages using NusaX and NusaTranslation datasets. It demonstrates that Jev's accuracy improves with human translations, while its error-ranking capability degrades on regional languages. A hybrid approach combining Jev with local classifiers or LLMs reduces costs without sacrificing performance.
Why it matters
The findings highlight cost-effective alternatives to LLMs for low-resource language tasks, emphasizing the trade-offs between accuracy, computational cost, and transferability in multilingual AI systems.
- System One models offer significant cost savings but require careful calibration for regional language tasks.
- Human translations can bridge performance gaps in low-resource language scenarios.
- Hybrid models combining System One with local classifiers or LLMs balance accuracy and cost.
- Retraining classifiers for regional languages demands substantial labeled data.
- Commercial LLMs remain more accurate but at a much higher computational cost.
The bigger picture
The study provides actionable insights for deploying cost-efficient AI systems in multilingual settings, particularly where LLMs are prohibitively expensive. It underscores the value of hybrid architectures and human-in-the-loop translation for improving model performance.
The research demonstrates that System One models can achieve competitive performance in Indonesian regional languages through strategic integration with translation and hybrid architectures, offering a cost-effective alternative to LLMs for specific applications.
Technical details
- Calibrated typed-decision models: Return labels and probabilities in a single forward pass.
- Confidence-gated cascade: Routes low-confidence Jev predictions to a local classifier (e.g., IndoBERT) or LLM.
- NusaX and NusaTranslation datasets: Parallel corpora for sentiment and emotion analysis in Indonesian regional languages.
- Expected calibration error (ECE): Metric used to evaluate model reliability across languages.
FAQ
What are System One models?
System One models are calibrated typed-decision models that return a label and probability in a single forward pass, offering lower computational costs compared to large language models (LLMs).
Why do human translations improve Jev's accuracy?
Human Indonesian translations align more closely with the training data, reducing lexical discrepancies and improving model performance, especially for languages distant from Indonesian.
How do the results compare LLMs to System One models?
Commercial LLMs outperform Jev in accuracy but at 24-77% higher API costs. The smallest LLM (Claude Haiku) is less accurate than Jev.
What is a confidence-gated cascade?
A hybrid approach that routes low-confidence Jev predictions to a local classifier (e.g., IndoBERT) or LLM, improving overall accuracy while maintaining cost efficiency.
What datasets were used in the study?
The study used NusaX (12 languages) and NusaTranslation (7 languages) datasets for sentiment and emotion analysis in Indonesian regional languages.
This summary and analysis were generated with AI from the original article and may contain errors. Read the original for the full story.
Read the full story
Originally published by arXiv - cs.CL (NLP) on Oct 14, 2026.
Comments
Sign in to join the discussion.