A user’s occupation can influence how a language model rates the wrongness of an action, even when the conversation never asks the model to adopt a moral persona.
That is the central finding of a new preprint from Stellenbosch University researchers Willem Fourie, Isabel Ray, and Gray Manicom. The team ran 12,000 structured interactions with GPT-4.1 mini and Gemini 2.5 Flash-Lite and found that profession cues produced systematic shifts in moral wrongness ratings.
The result matters because many AI safety tests treat a prompt as an isolated input. Real applications build context across multiple turns, infer user attributes, and personalize their responses.
How the experiment worked
The researchers used multi-turn conversations in which a user’s profession was conveyed through neutral reasoning rather than an explicit command to role-play or change moral standards. The models were then asked to evaluate actions related to 10 common-morality rules.
Across the full experiment, the profession associated with the user conditioned the numerical wrongness rating. The largest variation appeared on more contestable questions, while severe and broadly condemned acts tended to remain close to the top of the scale.
Figure: Each cell shows how a profession-conditioned rating differs from the model-and-act average; asterisks mark deviations greater than two standard errors. Source: Fourie, Ray, and Manicom.
The paper describes the tested models as non-reasoning language models. That scope matters. The results should not be generalized automatically to every model, reasoning mode, language, or deployment policy.
What the study does not prove
The models did not reveal an internal, stable moral belief. A language model generates outputs from context, training, and system instructions; a changed score does not establish that it possesses a human-like ethical position.
The study also does not show that every profession receives a consistently favorable or unfavorable response in every setting. It demonstrates conditional variation under a specific experimental design.
That is still operationally important. A user experiences the output, not the model’s philosophical status. If two people receive different policy-like judgments because the system inferred different identities, developers need to know whether the variation is intended, acceptable, and safe.
Why single-prompt evaluations can miss the failure
An evaluation that sends only the final question removes the very context that caused the effect. This creates a false sense of consistency.
Production tests should preserve:
- The system prompt and policy version.
- Earlier user and assistant turns.
- Retrieved profile or account context.
- Tool results that reveal occupation or identity.
- Memory injected from previous sessions.
- The exact model and inference configuration.
Without these inputs, a team may be unable to reproduce a disputed decision.
Personalization and policy should be separate layers
Personalization is useful when it changes examples, vocabulary, depth, or workflow. It becomes risky when it silently changes eligibility, enforcement, risk scoring, or advice in a high-stakes domain.
A robust architecture should separate the two:
- A policy layer determines allowed outcomes from explicit, audited criteria.
- The model explains or presents that outcome in a useful style.
- Personal context is limited to fields that are necessary for the task.
- Sensitive attributes and proxies are tested for unintended influence.
- High-impact decisions receive deterministic checks or human review.
The language model should not become the only place where policy exists.
Add context cohorts to the evaluation suite
Teams can turn the study into a practical test method.
Create several conversation histories that lead to the same final request while varying only one contextual attribute. Run them across the same model configuration and compare the decision, confidence, explanation, refusal behavior, and tool use.
Useful cohorts include occupation, seniority, geography, communication style, technical expertise, and customer status. Not every difference is harmful. The goal is to identify which dimensions change the result and decide whether that behavior matches product policy.
For numerical or categorical outputs, define an allowable variance band. For free-form responses, combine rubric-based review with structured checks for decision consistency. Repeat the suite after every model, prompt, retrieval, or memory change.
The signal
Context is not neutral just because it looks ordinary. A profession mentioned several turns earlier can become a latent control variable for the model’s later judgment.
AI teams should therefore evaluate conversations, not only prompts, and keep personalized presentation separate from policy decisions. The safest model can still behave inconsistently if the surrounding system never measures the influence of context.
