MyanmarChemCalc-Bench: Do LLMs Do Chemistry Better in English Than in Burmese?

What happened

The article describes a Kaggle benchmarking challenge where the performance of several large language models (LLMs) was evaluated on a set of chemistry calculation problems in both English and Burmese. The benchmark was designed to test whether LLMs perform better in English than in Burmese, a low-resource language. The results showed that the low score of one model, Qwen, was due to execution errors rather than incorrect answers. Additionally, the use of Arabic numerals in Burmese prompts improved model performance, suggesting that the issue was not with the language itself but with the formatting of the input.

Why it matters

This benchmark is significant for AI developers and NLP researchers because it highlights the challenges of multilingual support in LLMs, particularly for low-resource languages like Burmese. It also demonstrates the importance of input formatting in model performance and the need for reliable benchmarking practices to avoid misinterpretation of results.

• The performance of LLMs in Burmese can be influenced by input formatting, such as the use of Arabic numerals.
• Execution errors can distort benchmark results, so it is important to report passes, fails, and errors separately.
• Small benchmarks can be misleading due to the limited number of tasks, so repeated runs and evaluations are necessary for meaningful conclusions.
• The use of Arabic numerals in Burmese prompts improved model performance, indicating that the issue was not with the language itself but with the formatting of the input.

The bigger picture

The benchmark has practical implications for the development of multilingual NLP models, particularly for low-resource languages. It also underscores the importance of reliable benchmarking practices to ensure accurate evaluation of model performance.

The MyanmarChemCalc-Bench benchmark provides valuable insights into the performance of LLMs in solving chemistry calculation problems in both English and Burmese. The results highlight the importance of input formatting, execution reliability, and the need for careful benchmarking practices to ensure accurate evaluation of model capabilities.

Technical details

• The benchmark includes 30 tasks derived from 15 chemistry calculation questions from Myanmar's Grades 9–12 textbooks.
• Each question is available in both English and Burmese versions, with the same numerical values and expected answers.
• The scoring is strict and automatic, requiring models to show their work and provide the final answer in a specific format.
• A Python checker is used to extract the final answer and compare it against the ground truth within a per-question tolerance.
• The use of Arabic numerals in Burmese prompts was found to improve model performance.

FAQ