CL-SN55 941 —
GPT-61S 890 —
GPT-56T 861 —
GPT-6S 837 -1.2%
CL-OP55X 831 -1.4%
MUSE-SPK 804 —
QWEN-38X 795 —
GPT-6A 782 —
CL-OP5H 724 -1.6%
CL-OP5X 722 -1.6%
KIMI-K3X 718 -0.1%
CL-OP46H 707 -1.7%
CL-FAB5H 705 -7.4%
CL-OP55H 705 -1.7%
CL-OP47H 696 -1.7%
GEM-37FH 661 -9.7%
GEM-38FH 656 -4.8%
GPT-56S 578 -0.5%
CL-OP47 543 -0.5%
INKL 531 —
GEM-3P 494 -0.2%
CL-OP46 490 -0.2%
GEM-31P 485 -0.4%
CL-OP48 483 -0.2%
GPT-6L 436 —
CL-SN55 941 —
GPT-61S 890 —
GPT-56T 861 —
GPT-6S 837 -1.2%
CL-OP55X 831 -1.4%
MUSE-SPK 804 —
QWEN-38X 795 —
GPT-6A 782 —
CL-OP5H 724 -1.6%
CL-OP5X 722 -1.6%
KIMI-K3X 718 -0.1%
CL-OP46H 707 -1.7%
CL-FAB5H 705 -7.4%
CL-OP55H 705 -1.7%
CL-OP47H 696 -1.7%
GEM-37FH 661 -9.7%
GEM-38FH 656 -4.8%
GPT-56S 578 -0.5%
CL-OP47 543 -0.5%
INKL 531 —
GEM-3P 494 -0.2%
CL-OP46 490 -0.2%
GEM-31P 485 -0.4%
CL-OP48 483 -0.2%
GPT-6L 436 —
← Back to feed

LILT Launches AURORA: First AI Leaderboard to Score Frontier Models on Non-English Agentic Tasks

LILT launched AURORA on September 30, a leaderboard that scores frontier AI models on agentic, multimodal, and socio-cultural tasks delivered in languages other than English. Nearly every published measure of frontier model progress has relied on English-centric or machine-translated benchmarks. AURORA uses tasks designed and verified by native-language domain experts.

Four Evaluation Tracks

  • Multilingual Terminal-bench: Coding tasks in software written for native-language users
  • Multilingual τ³-bench: Multi-turn customer support across airline, telecom, retail, and banking in target languages
  • Multilingual MultiChallenge: Long-context instruction-following, memory, and self-coherence
  • Multilingual GAIA-v2-LILT: Agentic reasoning and tool use

Rankings Split by Language

LILT’s initial analysis shows the rankings are not stable across languages. In coding tasks specifically: GPT-5.5 leads in Spanish, Claude Opus 5.5 wins in Japanese, and Muse Spark 1.3 tops in Serbian. Model quality can differ substantially between languages: the model that leads on an English leaderboard may not lead in the target language an enterprise ships to.

The Commercial Argument

Enterprise agent deployments are no longer English-only experiments. Agents now take real actions in customer-facing workflows across dozens of languages, and wrong decisions surface in user complaints, not lab benchmarks. LILT’s CEO Spence Green framed it directly: “The industry is choosing models on a scoreboard that stops at English. That gap used to cost you an awkward translation. Now that agents are taking real actions in the real world, it costs you a wrong decision, and you find out from your users.”

AURORA is developed by LILT’s Applied AI practice, a PhD-led team with ten-plus years in multilingual AI. LILT’s enterprise client list includes NVIDIA, Intel, and L’Oréal. The leaderboard is live at aurora.lilt.com. Private custom benchmarks are available to enterprise teams on request.