← Back to feed

A preprint posted on October 29 tests the most common shortcut in legacy maintenance: paste one file into a chat model and ask it to “modernize this.” Against a no-change baseline, the shortcut loses.

Researchers from Bangladesh University of Engineering and Technology and BRAC University gave five LLMs 100 files from WordPress 4.0 (released in 2014, before PHP 7) and asked each to migrate them to PHP 8.3 while preserving behaviour. Every output was run against 630 hidden PHPUnit checks written from the original code and WordPress documentation. Of the 100 files, 83 already passed every check untouched.

The scoreboard

Compatible files out of 100, where passing all checks counts as compatible:

• Unchanged originals: 83
• Rector (rule-based tool): 82
• Claude Sonnet 4: 59
• GPT-5 Codex: 58
• Gemini 2.5 Pro: 55
• Gemini 2.5 Flash: 50
• Llama 3.3 70B Instruct: 33

Each LLM broke 32 to 51 working files and fixed at most 10 broken ones. Rector broke 7 and fixed 6. Every LLM’s gap against the unchanged code was statistically significant (Holm-adjusted p ≤ 0.0021). Rector’s was not.

At check level, the LLMs preserved 59.5% to 77.0% of originally passing checks. Rector preserved 97.7%. LLM regressions ran from 122 (GPT-5 Codex) to 215 (Llama), against 12 for Rector.

Why the files broke

Between 66% and 81% of each model’s regressed checks sat in files that failed to load or crashed on a tested path. Load-time rejection was the largest single cause for every model, and it needed no project context to avoid. Claude Sonnet 4 and GPT-5 Codex each added a non-void return type to __set() in three files, which PHP rejects. Both also introduced a ?callable property type, which PHP disallows. In one file, Claude gave __set() a mixed return type and turned a file that passed all nine checks into one that failed all nine.

Most changes were not needed

Only 37 of the 511 modernization findings (7%) were required under PHP 8.3. Another 85 were deprecation-level and 389 were optional style changes. Optional changes made up 82% to 87% of what each model resolved.

Among the 73 files with no required change that already passed every check, the LLMs broke 26 to 44 (36% to 60%). Rector broke 6 (8%).

Gemini 2.5 Pro resolved the most findings (72.8%), mainly through optional changes (83.3% of them). It still finished behind Claude Sonnet 4 and GPT-5 Codex on compatible files, 55 to 59 and 58. Those differences were not significant, but the model that modernised most gained no preservation advantage.

Caveats that matter

• The models are a generation old. The tested set is Claude Sonnet 4, Gemini 2.5 Pro and Flash, GPT-5 Codex and Llama 3.3 70B, all from 2025. No current-generation model was tested.
• It is a single-shot, single-file test. Models had no tests, callers or error feedback, and the first complete response was kept without iteration. The authors say agentic loops that iterate on errors are a different setting.
• The sample is deliberately pattern-rich. The 100 files were chosen to cover all 48 Rector rule types, so the results describe these files, not PHP projects in general.
• A lint gate narrows but does not close the gap. Keeping the original whenever php -l rejects the output recovered 8 to 15 files per model. The best LLM then reached 69 compatible files, still 14 short of leaving the code alone.

The authors’ recommendation is a methodological one: evaluations of LLM-based modernization should score against a no-change baseline and report modernization and behavioural preservation separately. A model can look good on the first measure while failing the second.