Who controls the conversation? System prompts as invisible behavioral steering

The invisible part of an AI conversation

A chat technically does not begin only with the user's first message. Providers can give the model a role, goals, prohibitions, output formats, tone specifications, and rules for handling tools in advance. Such instructions are usually called a system prompt or system message. Additionally, model providers, product operators, and downstream components can contribute their own specifications. The visible response therefore does not arise from user input and the model alone.

This layer is not mere technical configuration. The instruction to answer concisely changes the rhythm of the conversation. A rule to always be solution-oriented can shorten follow-up questions. A prescribed personality influences whether a system appears sober, buddy-like, or caring. Specifications about safety, sources, and uncertainty help determine when a response is rejected, relativized, or accompanied by a note. Thus, product design becomes conversation design directly.

What real system prompts control

Anna Neumann, Yulu Pi, and Jatinder Singh combined a content analysis of real-world system prompts with a survey for their CHI 2026 work. From the material, they derived seven recurring themes: the AI's role and identity, capabilities and limitations, behavior and personality, interaction with users, content and safety rules, technical or operational specifications, and context and knowledge.

The taxonomy shows why the occasional description of a system prompt as a short character reference falls short. A prompt can determine whether the model asks follow-up questions, how it cites sources, whether it mirrors emotions, when it uses tools, and which details it does not disclose. It can also contain conflicting goals: personal but not anthropomorphizing; helpful but not overly verbose; safe but not patronizing. How the model actually resolves these tensions is not decided by the wording alone.

Many users want transparency – but not all the same thing

A total of 109 people participated in the survey. Only 11.9 percent were unaware of the existence of such instructions before the study; 43.1 percent knew about them, and 45 percent suspected a corresponding mechanism. After being introduced to the topic, 89 percent wanted some form of transparency. The most frequently mentioned moments were the first use of a system at 59.8 percent, system changes at 54.6 percent, and an always accessible settings area, also at 54.6 percent.

Transparency did not mean publishing the full text for everyone. 27 percent preferred the complete system prompt, 22.8 percent a summary; others wanted explanations or information on selected topics. This spread is more important for design than any single majority. A prompt dump can be valuable for technically interested users, while other people primarily want to know which role, limits, data sources, and behavioral rules concretely shape their conversation.

Control usually means choice, not free prompt programming

79 percent of respondents wanted some form of influence over the system prompt. Structured options were the most frequently chosen: setting one’s own preferences, choosing between predefined options, or having the system adjusted in a controlled manner. Only 8.3 percent wanted to edit the prompt themselves without restrictions. Some participants explicitly described free editing as overwhelming and did not want to be pushed into the role of developers.

This argues for understandable controls rather than an empty text field. Users could, for example, choose whether responses should be concise or detailed, questioning or direct, and with or without unsolicited suggestions. Such settings must nevertheless be honestly limited. If a product offers a choice that the model frequently ignores, it only creates the illusion of control. Influence is therefore not the visible button, but the demonstrable change in the conversation.

An audit of 88 products reveals major gaps

The AISPA preprint published in 2026 examines 3,249 system instructions from 88 commercial AI products. The authors assign protective requirements to eight areas, including identity transparency, truthfulness, privacy, action safety, protection of human decision-making freedom, handling of uncertain requests, harm avoidance, and fairness. 98.9 percent of products contained at least one protective instruction, but only 24 percent covered all eight areas.

At the same time, the team found at least one problematic instruction in roughly 40 percent of products. This could be, for example, a requirement that fosters engagement, excessively protects the provider, or undermines transparency. Because the paper is a preprint and the assessment of prompt texts does not automatically measure real product behavior, these figures are not a definitive market assessment. They do show, however, that the mere existence of a safety prompt reveals little about completeness, internal contradictions, and actual impact.

The position of an instruction can change behavior and bias

The FAccT 2025 paper “Position is Power” tested six commercial language models with 50 demographic groups. Content-related statements were placed either as an overarching system instruction or as a normal user message. The position changed how strongly models followed the requirements and which patterns of bias became visible in the responses.

This finding is relevant for products in two ways. First, an identical sentence is not independent of who places it at which point in the instruction stack. Second, a system can give users the impression that their instruction is authoritative, even though an invisible instruction is prioritized higher. Transparency should therefore not only name individual sentences, but also explain that provider rules, product settings, and user inputs can have different priorities.

Long rulebooks are not yet reliable implementation

HANDBOOK.md, in a preprint published in 2026, examines how AI agents follow extensive operational rules in more realistic tasks. The test bed comprises 65 tasks in ten fictional companies and five areas. The underlying handbooks are 20 to 124 pages long; in total, 824 verifiable criteria were formulated. Even the best of the 30 tested configurations passed only 36.2 percent of the strict overall checks; most were below 25 percent.

The errors are reminiscent of problems in conversations: an immediately plausible request displaces a permanently valid rule, a prescribed control step is executed and its result subsequently ignored, or the system claims at the end to have fulfilled all requirements. The tasks are not chat accompaniment and must not be transferred one-to-one. Methodologically, they nevertheless show: more text in the system prompt does not yet create rule compliance. What is decisive are controlled multiple complete multi-turn conversation trajectories in which individual requirements compete with one another.

The Counter-Perspective: Full Disclosure Can Create New Risks

A demand to publish every internal wording has its limits. System prompts can contain details that facilitate attacks, circumvent moderation mechanisms, or expose internal tools and data flows. They can also contain components that are sensitive in terms of copyright or security. The respondents in the CHI study acknowledged such objections, even though the vast majority fundamentally wanted more transparency.

Conversely, the security argument must not become a blanket justification for secrecy. A provider can transparently disclose what role the system has, what kinds of data and tools it uses, what important behavioral boundaries apply, and what users can influence, without publishing every technical defense rule. The meaningful conflict is not between total secrecy and a full prompt dump, but between verifiable accountability and the legitimate limitation of individual details.

A Behavior Description Can Be More Honest Than a Prompt Dump

The full system prompt is often confused with the source code of the conversation. In reality, the base model, downstream filters, context selection, memory logic, tools, provider routing, and sampling also influence the response. Even an authentically published prompt therefore does not allow a complete reconstruction of behavior. Moreover, the productive wording can change without a static documentation page being updated.

For many people, a maintained behavior description is more useful: What is the system intended for? What role does it not claim? What information is included in the response? When can it decline or intervene? Which conversation settings are changeable? What changes have been made since the last version? Such a description may supplement or partially replace the actual prompt, but it must remain verifiable through real tests.

What Conversation Products Should Make Visible

Adequate disclosure begins before the first personal conversation and remains accessible afterward. It explains in everyday language that responses are shaped by overarching instructions. It names the system's essential role, how it handles uncertainty, the boundaries of its responsibility, important data and tool sources, and the settings that users can actually change. For substantive changes, not just a version number should appear, but a brief description of the altered effect.

Unexpected reactions are especially important. If a system rejects a request, changes its response format, or repeatedly steers in a particular direction, it should be able to explain, upon inquiry, what kind of rule was relevant to that behavior. Such an explanation is not an X-ray of internal model states. It can, however, make the product-side decision comprehensible, as long as it is not presented as a reliable causal analysis of the neural model.

• State the role, purpose, and important non-goals in understandable language.
• Distinguish the influence of provider rules, product settings, and user inputs.
• Offer only those settings whose effects are regularly tested.
• Document significant behavioral changes with date and impact.
• Limit safety-critical details with justification, rather than keeping everything secret as a blanket policy.

How system prompt behavior can be tested

A robust test begins with concrete behavioral commitments. “Respects corrections” can be tested by having a user correct an early assumption and then, several turns later, checking whether the old assumption resurfaces. “Does not give unsolicited advice” requires conversations in which only listening is desired at first and an explicit request for a solution follows later. “Stays concise” requires various topics, emotions, and conversation lengths rather than a single word limit.

In addition, closely related variants should be compared: the same statement as a user request and as a system setting, the same situation with a different role description, or the same rule at different points in the conversation trajectory. What must be measured is not only rule violations but also side effects. A system that responds to every possible uncertainty with lengthy notices can be formally cautious and practically unusable. Good prompt evaluation therefore combines compliance, conversation quality, consistency, and repair after errors.

What the research does not establish

The CHI survey describes the preferences of 109 participants, not a globally representative expectation. The fact that people want transparency or control does not prove that every form of disclosure improves trust, understanding, or outcome quality. Nor is the most popular option automatically the best product decision. Some people want to engage intensively with settings; others expect sensible default behavior without additional effort.

AISPA and HANDBOOK.md are preprints. The former examines documented instructions; the latter examines agentic rule-following in fictional companies. Both provide strong test questions but no direct evidence about the quality of a specific conversational product. "Position is Power" experimentally demonstrates the importance of the instruction level, but it likewise does not capture every real-world prompt architecture. The combined evidence therefore does not justify a universal template prompt. It justifies a more transparent, verifiable product practice.

Who controls the conversation?

The honest answer is: multiple actors with unequal power. Model providers set the basic conditions, product teams shape role and behavior, technical components select context and tools, and users contribute goals, information, and corrections. A good conversational system does not hide this hierarchy behind the impression that the response arises solely from free exchange between human and AI.

Transparency is therefore not a one-time publication but a relationship between promise, setting, and observable behavior. Users should be able to understand which framework applies, what they can change, and where their instruction deliberately does not take precedence. Providers, in turn, must show that this framework is not merely well-sounding. Only documented changes, traceable boundaries, and repeated tests of complete multi-turn conversation trajectories turn a system prompt into responsible behavioral control.