← Back to feed

A paper accepted at COLM 2026 finds that whether a language model adds disclaimers like “I’m just an AI” is controlled primarily by the chat template format applied at inference time — not by the underlying model weights. The implication: self-reports used in AI safety debates may be artifacts of deployment configuration, not signals about what a model “knows” or “believes” about itself.

The Finding

The researchers tested 8 popular open-source instruct models up to 9B parameters (arXiv: 2609.25021). When a standard chat template is applied, models consistently produce disclaimer-heavy responses (“As a language model, I cannot…”). Remove the chat template and pass the same query as raw text — the disclaimers largely disappear and models produce more “experiential” language (“I feel…”, “In my experience…”).

The switch is binary enough that the authors describe the chat template as a toggle: present, the disclaimer voice is up and experiential voice is down; absent, the pattern reverses.

Activation Steering Reproduces the Effect

To confirm this isn’t just surface-level prompting, the team probed the internal activations of 3 of the models. They found a single geometric direction in activation space that mediates the behavior. Subtracting this direction from an instruct model’s activations reduces disclaimers without the chat template being removed. Adding it back to an instruct model running without a template restores disclaimer rates to near-template levels.

A randomly chosen direction of the same magnitude has little effect. The direction is specific to the disclaimer-voice mechanism.

Implications for Alignment Research

This creates a methodological problem for a body of alignment work that uses model self-reports as data. Studies asking models whether they experience uncertainty, preferences, or distress often rely on whether the model says things like “I don’t have feelings” or “I’m uncertain.” If those responses are partially set by how the model is deployed, not by its weights, then:

• Comparisons between models are confounded by template differences
• Results from base models (no template) and instruct models are measuring different things
• Fine-tuning interventions aimed at changing self-reports may be operating on the template effect rather than the underlying representation

The paper is explicit: “What models say about themselves is not a fact about them. What they say doesn’t come only from weights, but it is partially set by the chat template.”

Scope and Limits

The study uses models up to 9B parameters. Frontier models (GPT-6, Claude Opus 5.5, Gemini 3 Pro) operate at a different scale and with more elaborate RLHF alignment. Whether the same activation direction exists and dominates at frontier scale is an open question. The chat template effect may be larger, smaller, or overridden by alignment training on larger models.

The researchers do provide an activation direction that other teams can apply to steer disclaimer voice — a useful tool for alignment researchers running controlled comparisons regardless of scale.

The paper has been accepted at the COLM 2026 Workshop on Efficient Reasoning and KONVENS 2026’s Eval4SD workshop.