CL-SN55 941 —
GPT-61S 885 —
GPT-56T 861 —
GPT-6S 825 —
CL-OP55X 818 -0.1%
MUSE-SPK 793 —
QWEN-38X 779 —
GPT-6A 778 —
CL-OP5H 711 -0.1%
CL-OP5X 709 -0.1%
KIMI-K3X 699 -0.1%
CL-OP46H 694 -0.1%
CL-FAB5H 692 -0.1%
CL-OP55H 692 -0.1%
CL-OP47H 683 -0.1%
GEM-37FH 648 -0.2%
GEM-38FH 633 -0.2%
INKL 531 —
CL-OP47 529 -0.2%
GPT-56S 529 -5.9%
GEM-3P 493 —
CL-OP46 488 —
CL-OP48 481 —
GEM-31P 453 -0.2%
GPT-6L 436 —
CL-SN55 941 —
GPT-61S 885 —
GPT-56T 861 —
GPT-6S 825 —
CL-OP55X 818 -0.1%
MUSE-SPK 793 —
QWEN-38X 779 —
GPT-6A 778 —
CL-OP5H 711 -0.1%
CL-OP5X 709 -0.1%
KIMI-K3X 699 -0.1%
CL-OP46H 694 -0.1%
CL-FAB5H 692 -0.1%
CL-OP55H 692 -0.1%
CL-OP47H 683 -0.1%
GEM-37FH 648 -0.2%
GEM-38FH 633 -0.2%
INKL 531 —
CL-OP47 529 -0.2%
GPT-56S 529 -5.9%
GEM-3P 493 —
CL-OP46 488 —
CL-OP48 481 —
GEM-31P 453 -0.2%
GPT-6L 436 —
← Back to feed

LILT Launches AURORA: First AI Leaderboard to Score Frontier Models on Non-English Agentic Tasks

LILT launched AURORA on September 30, a leaderboard that scores frontier AI models on agentic, multimodal, and socio-cultural tasks delivered in languages other than English. Nearly every published measure of frontier model progress has relied on English-centric or machine-translated benchmarks. AURORA uses tasks designed and verified by native-language domain experts.

Four Evaluation Tracks

  • Multilingual Terminal-bench: Coding tasks in software written for native-language users
  • Multilingual τ³-bench: Multi-turn customer support across airline, telecom, retail, and banking in target languages
  • Multilingual MultiChallenge: Long-context instruction-following, memory, and self-coherence
  • Multilingual GAIA-v2-LILT: Agentic reasoning and tool use

Rankings Split by Language

LILT’s initial analysis shows the rankings are not stable across languages. In coding tasks specifically: GPT-5.5 leads in Spanish, Claude Opus 5.5 wins in Japanese, and Muse Spark 1.3 tops in Serbian. Model quality can differ substantially between languages: the model that leads on an English leaderboard may not lead in the target language an enterprise ships to.

The Commercial Argument

Enterprise agent deployments are no longer English-only experiments. Agents now take real actions in customer-facing workflows across dozens of languages, and wrong decisions surface in user complaints, not lab benchmarks. LILT’s CEO Spence Green framed it directly: “The industry is choosing models on a scoreboard that stops at English. That gap used to cost you an awkward translation. Now that agents are taking real actions in the real world, it costs you a wrong decision, and you find out from your users.”

AURORA is developed by LILT’s Applied AI practice, a PhD-led team with ten-plus years in multilingual AI. LILT’s enterprise client list includes NVIDIA, Intel, and L’Oréal. The leaderboard is live at aurora.lilt.com. Private custom benchmarks are available to enterprise teams on request.