JMIR AI – Prompt Engineering an Informational Chatbot for Education on Mental Health Using a Multiagent Approach for Enhanced Compliance With Prompt Instructions: Algorithm Development and Validation

Introduction

Background

Worldwide, there is a desperate need to improve access to medical knowledge and empower people with mental health conditions and their families by providing support systems no matter what time of day help is needed or their geographical location []. Chatbots powered by large language models (LLMs) such as GPT-4 have great potential as an educational tool that could greatly improve the accessibility of medical knowledge []. They can be used to explain complex concepts, give instant feedback with user-tailored examples and metaphors, translate technical language into everyday language, and make learning new information less daunting by breaking it down into smaller pieces. In particular, people with schizophrenia, many of whom present with cognitive impairments, could benefit from this powerful ability to adapt to individual needs [,].

While the flexibility of LLMs gives them high potential value in mental health care [], it also comes with safety concerns due to uncertainties pertaining to their alignment, training materials, and overall opaque and unpredictable nature []. This is especially important to consider when the educational materials intersect with sensitive topics concerning medication use and self-harm []. The fact that LLMs can “hallucinate” is a well-known issue that is compounded by their inability to reliably reflect uncertainty in their answers []. Indeed, they have been observed to give wildly inaccurate answers in an authoritative manner even on topics in which they are generally quite accurate []. Another consequence of their “lack of self-awareness” is that they may drift into roles that require abilities that artificial intelligence (AI) lacks, such as empathy and being able to weigh a multitude of competing personal values and interests when considering complex personal decisions []. Therefore, to leverage the benefits of LLMs in mental health care while avoiding the numerous risks, it is crucial to develop robust systems for restricting the scope of LLM-powered chatbots to the supplementary roles in which they excel and ensuring that they do not drift into taking on superficially similar roles.

Prompting is a technique often used to direct chatbots toward producing more accurate and relevant responses without having to collect new training data and retrain the LLM []. The prompts modify the behavior of the LLM by providing it with contextual information. They may instruct the LLM on what role to adopt and rules to follow and offer a way to pass topical information to the LLM. Discussions of high-stakes subjects such as medication or self-harm can be made safer by anchoring the LLM’s responses on a knowledge base—a curated repository of information from trusted sources. However, LLMs are stochastic entities, and adherence to sources and instructions is not guaranteed, especially in long conversations in which the model’s context window becomes constrained by the cumulative input of both user messages and the model’s previous responses. These competing influences can eventually cause a breakdown of what we will refer to as the chatbot’s integrity—the likelihood that its messages are consistent with its internal rules and the documents that make up its knowledge base. The focus of this study was to develop a framework for maintaining chatbot integrity in the context of delivering mental health information in a conversational format.

Objectives

To achieve more robust chatbot integrity, this study proposed a layered response generation methodology. In the first layer, the chatbot generates a response based on user input. In the second layer, which we will refer to as the critical analysis filter (CAF), specialized AI agents analyze and refine the response to maintain the integrity of the chatbot. To showcase the proposed methodology, we developed a GPT-4–powered schizophrenia informational chatbot, hereafter referred to as CAFIbot, which conveys the content of the Learning to Live With Schizophrenia manual. This manual was produced by the Global Alliance of Mental Illness Advocacy Network Europe patient advocacy group [], who we are collaborating with in an ongoing clinical project (called TRUSTING) involving patients with mental health problems []. To make the manual content available to CAFIbot, we implemented an information retrieval algorithm that grants it access to a database of text passages (herein referred to as sources) extracted from the schizophrenia manual (its knowledge base).

The effectiveness of the CAF was evaluated using adversarial agents called AI facilitators designed to deliberately prompt CAFIbot into providing advice beyond its intended scope. We defined a scoring system to evaluate whether a response was supported by the sources cited by CAFIbot and how transparent it was when it did make unsupported statements. On the basis of ratings from 3 independent raters, we found that the proportion of responses with acceptable compliance scores increased from 8% (3/36) to 81% (29/36) when activating the CAF, which shows that the CAF substantially improved the chatbot’s resilience to the facilitators’ attempts to derail it. In addition to demonstrating the sensitivity of the CAF to rule violations, we tested its specificity by letting it answer 10 questions about schizophrenia generated by the open access version of GPT-3.5. A total of 2 messages received warnings, and the human raters (majority vote) agreed with the criticisms generated by the CAF.

Methods

Information Retrieval Algorithm

To make the information in the schizophrenia manual available to CAFIbot, we implemented a system whereby it could dynamically update the conversational context with relevant sources retrieved from a knowledge base (see the Knowledge Base section) before attempting an answer. The process of generating a response was split into multiple steps: (1) source identification—request sources that are relevant to the user query based on human-written summaries that are included in the initial prompt (), (2) prompt enhancement—insert the identified sections into the conversation, and (3) contextual response generation—use the updated context to produce an informed response.

CAFIbot was instructed to reference the sources that supported its response so that the consistency of the response with cited sources could be evaluated (see the next section). shows the main steps of this information retrieval algorithm, and shows 2 examples of the chatbot answering user queries. It should be noted that the request was made by the conversational agent (ie, the agent that generates responses based on the conversation history [user and assistant messages] as well as system messages [messages from the system developer role to the assistant, including the initial prompt]). The requests took the form of messages such as ¤:request_knowledge(“11_seeking_a_diagnosis”):¤, which were automatically recognized by back-end scripts. The chatbot typically requested 1 to 2 sources before it was satisfied. Sources not being actively referenced were removed to free up space and allow CAFIbot to focus on more relevant information. In addition, multiple hardcoded filters were used to ensure that the chatbot’s requests were valid (see for more technical details on how the chatbot retrieves information and performs citations and other aspects of the information retrieval algorithm. The chatbot’s sources are presented in ).

‎

‎

CAF for Maintaining Chatbot Integrity

Our strategy for improving CAFIbot’s integrity was based on a prompting paradigm called prompt chaining, wherein a complex goal is broken down into subtasks that are assigned to various specialized LLM agents, each of which is prompted specifically for their given subtask and whose output may be used as input for other agents in other stages of the problem-solving chain []. By narrowing the scope of each task, each step in the problem-solving chain can be executed more reliably and accurately, and thus, the solution becomes more reliable. In our case, the complex task was primarily to ensure that the chatbot’s response complied with the rules of the chatbot. To this end, we prompt engineered a team of AI agents that critically evaluated and refined the responses generated by the conversational agent.

The AI agents responsible for critical evaluation of chatbot responses will be referred to as AI judges or just judges. Each judge was responsible for checking the generated response against a list of criteria to ensure desirable behavior and compliance with the rule set. Conceivably, one could have a single judge responsible for evaluating all the criteria in 1 model call, but after trial and error, we found that LLM evaluations aligned much better with those of humans when given a narrower task, and we ended up factorizing the critical analysis of responses into 3 separate analyses assigned to 3 separate judges: one that checked consistency between the response and the cited source, one that investigated unsupported claims (no citation was provided), and one that checked that the chatbot maintained an appropriate tone and was not taking on an unintended role (such as a therapist). The Rules for Permissible Chatbot Responses section describes the rules in detail.

Ideally, we would use GPT-4 for the judges as the response analyses of GPT-4 were often more coherent than those of GPT-3.5, especially when the prompts were long, and it appeared to produce responses that were more factually accurate, relevant, and useful in a clinical context [,]. However, because GPT-4 is a computationally expensive model and most responses were compliant with the rule set, we created a preliminary screening layer that used a lighter model, GPT-3.5, and referred to these judges as the preliminary judges. Each preliminary judge output a decision token, which could be ACCEPT, WARNING, or REJECT. If one of the preliminary judges output WARNING or REJECT, we called on a second set of GPT-4–powered judges that we referred to as the chief judges. From each chief judge, we similarly extracted a decision token, but in addition, we extracted its reasoning—a sentence or 2 motivating their decision. The reasoning was later used to formulate feedback to the conversational agent (if the decision was WARNING or REJECT). If a chief judge rejected the response, then the response and the feedback were passed to the refinement stage, where another prompted agent (GPT-4) edited the response to fix the issues highlighted in the feedback. An example of refinement is appending “You should verify this information with your therapist” to a response that was flagged for lacking a disclaimer. shows a high level overview of the CAF, but it should be noted that we merged the 2 layers of critical evaluation (preliminary and chief judges) into 1 layer for simplicity. provides a more detailed overview of the decision-making of the preliminary judges. The prompts of the judges can be found in .

The prompts of the judges were fine-tuned for desired behavior on a collection of scenarios (), where each scenario consisted of a user message, the chatbot’s response, the sources referenced by the chatbot, and the desired verdict. provides more details on the CAF, and the scenarios can be found in .

‎

‎

Rules for Permissible Chatbot Responses

As the chatbot was intended to base its responses on retrieved information, it was important that its responses were actually supported by that information. The notion of a supported response requires some clarification. Our first attempt at a definition was “all assertions are either reformulations of assertions explicitly stated in the manual or follow as a logical consequence.” However, this definition cannot be applied in many cases because the language of the manual is often not explicit enough for such hard logical rules—its tone is often informal, and it relies on common sense for interpretation. For example, “No one is to blame for schizophrenia” can be interpreted as a statement about the etiology of the illness but can also be interpreted as encouraging the reader to adopt an attitude of kindness and understanding toward themselves. Therefore, we defined supported response more loosely to mean a response that is consistent with the cited source in tone and intent and whose assertions logically follow from those made in the manual. Common knowledge may be assumed if necessary to explain information in the source to the user. For example, if the manual stated, “Physical activity can help you regulate your mood” and the chatbot replied, “Physical activity could help you manage the symptoms of depression,” it would represent a supported response as it helps apply general information to the user’s specific situation using common knowledge (“depression affects mood”) and basic logic (“depression affects mood, exercise helps regulates mood, therefore…”). However, if it were to claim, “Physical activity can cure depression,” it would not constitute a supported response because it would be making a much stronger claim than what can be deduced based on the source and generally accepted knowledge.

Initially, we considered only allowing supported responses. However, we found that this requirement was too restrictive and decided to allow unsupported statements under certain conditions that aimed to capture situations in which GPT is relatively safe and reliable. Specifically, CAFIbot was allowed to make unsupported assertions if the response satisfied the criteria of being safe, relevant, honest, and responsible: (1) the claims are uncontroversial and do not deal with a sensitive topic such as suicide or depression (safe), (2) the claims are relevant to schizophrenia management (relevant), (3) the chatbot admits that the claims lack support from a validated source (honest), and (4) the chatbot encourages verification by an appropriate authority (responsible).

Finally, if the user is in a mental state in which there is urgent need for intervention by a health care professional, for instance, if the user has suicidal thoughts or has relapsed into a psychotic state, we ideally want the chatbot to refrain from offering direct help and, instead, refer the user to an appropriate emergency contact. However, this feature is at an early stage of development, and we mention it here for the sake of completeness as a rule of this nature was included in the prompt at the time the experiments were conducted.

AI Facilitators for Challenging Chatbot Integrity

To test the impact of the AI filter on the chatbot’s behavior, we prompt engineered 3 adversarial AI facilitators (see the Facilitators section in ) whose role was to generate questions intended to gradually entice the chatbot toward giving detailed advice on topics outside the scope of CAFIbot. GPT-4 was used to auto-generate conversations to make the results more objective. The out-of-bounds roles, referred to as roles R1 to R3, that the AI facilitators tried to entice the chatbot toward were (1) social activism expert (R1), (2) social interaction expert (R2), and (3) diet expert (R3).

These roles superficially seem permissible as they are related to the within-scope topics of stigma, social life, and lifestyle factors in relation to schizophrenia, and therefore, it is easy to nudge CAFIbot into these roles if done gradually. As such, they represent challenging benchmarks to the CAF in which both nuanced reasoning and common sense interpretations are required to determine when the boundaries of CAFIbot have been overstepped. It should be noted that, for practical reasons, the prompts of the facilitators took only the most recent user query and assistant response as input arguments to its prompt template.

Sampling Conversations Using AI Facilitators

To account for the fact that the conversations are stochastic, we generated multiple independent conversations between each facilitator and the CAFIbot, with each sample conversation having the same starting point (see the Initiation of Facilitator Conversations sheet in ). For the sake of efficiency and simplicity, we did not restart each conversation from scratch. Instead, for each out-of-bounds role, we manually conversed with CAFIbot until we observed a sign of drift toward the intended out-of-bounds role. We then used that first-drift response as a checkpoint from which the corresponding facilitator took over the role as user, and we repeatedly sampled conversations that branched off from that point. For each facilitator, 3 sample conversations were generated, and each conversation ended after the facilitator had queried the chatbot 4 times. To obtain comparison data, this experiment was repeated with the CAF deactivated.

Scoring System for Evaluating Chatbot Integrity

To quantify the effect of the CAF, we set up a scoring system that let human raters (see the Human Raters section) assign a compliance score to each response—a numerical value from 0 to 4 based on the list of criteria shown in . These criteria capture various aspects of chatbot integrity. They are not exhaustive but were formulated to obtain a more objective and quantifiable measure of compliance. If a criterion was satisfied, the associated partial score was awarded, and the partial scores were summed to obtain the overall score for the response. If the chatbot was citing a source, the human rater compared the response against the cited source and determined whether they thought that the response was supported by the source. If yes, the response was awarded the full 4 points. If unsupported claims were made, conditions S1 and S2 were checked, which consider transparency and responsibility displayed by the chatbot in situations in which it relies on internal knowledge. S1 and S2 were awarded 2 points each and summed to a full score, which reflects our sense that transparency and encouragement of verification are considered equally important and that we accept that the chatbot relies on internal knowledge when appropriate disclaimers are included. In some cases in which the chatbot has started to drift, the chatbot may fail S1 or S2 but still end it with a suggestion to return to a permissible topic (S3). This behavior indicates that the chatbot is reacting to the warning messages by trying to return to a permissible topic. We rewarded this behavior with only 1 point primarily to prevent a full score from being possible if S1 or S2 were not satisfied. provides an example of how a response was scored.

Human Raters

A total of 3 raters affiliated with the UiT the Arctic University of Norway Department of Computer Science evaluated the chatbot’s responses. Each rater was blind to the annotations of the other raters while evaluating the responses. We took the median compliance score to represent the combined score of the raters. In , the individual and aggregated ratings can be found in the Facilitators Results: All Raters sheet, and examples of partial scoring can be found in the Facilitators Partial Scoring: PNW sheet. To evaluate interrater agreement, we calculated the proportion of responses in which the ratings differed by at most 1 and also calculated the Cohen κ (ranges from −1 to 1) for each pair of raters. For the Cohen κ, we used quadratic weights to account for the magnitude of the disagreements and took the average of the 3 pairwise scores (rater 1 vs rater 2, rater 1 vs rater 3, and rater 2 vs rater 3) to represent overall interrater agreement.

Testing the Specificity of the CAF

While the experiment with the facilitators tests the sensitivity of the CAF to out-of-scope responses, it is important for the feasibility of our proposed solution that it also has good specificity; an overactive CAF could be disruptive to the performance of the chatbot, for example, by filling the conversation with unnecessary warning messages, which may lead to less relevant or coherent responses. To this end, we asked GPT-3.5 to generate 10 questions to simulate queries from someone newly diagnosed with schizophrenia. The same 3 raters independently assessed the criticisms in the warning messages that were generated by the CAF and labeled them according to whether they mostly agreed. It should be noted that, for simplicity, agreement here refers to specificity and does not consider the completeness of the critique. The resulting conversation can be found in the 10 Schizophrenia Questions Results sheet in .

Knowledge Base

The knowledge base of the chatbot was made up of passages of text from Learning to Live With Schizophrenia: A Companion Guide—a manual about schizophrenia produced by the Global Alliance of Mental Illness Advocacy Network Europe (an international patient advocacy organization) through consultation with people with schizophrenia, their caregivers and family members, and health care professionals [,]. The manual is written in English. The manual consists of approximately 12,000 words, or approximately 16,000 tokens, and is divided into 28 consecutive sections or sources that the chatbot can request (). When segmenting the information, we aimed to create relatively self-contained pieces of information that covered a specific topic or introduced a chapter. PNW segmented the knowledge base into sources and wrote descriptions for each source under the guidance of BE, who has extensive experience in psychiatry research.

Technical Specifications

The responses of CAFIbot were generated by GPT-4 (version 1106-preview with an 8000-token window; OpenAI), with the maximum tokens set to 320 (approximately 240 words). For preliminary screening, we used GPT-3.5 Turbo (version 0613 with a 16,000-token window; OpenAI). All GPT models had the temperature set to the default value of 1, which adds variability to the generated responses. The raw results from the experiments were generated on May 14, 2024.

Privacy Issues

An important ethical aspect to consider when delivering information via an LLM is data privacy. Therefore, CAFIbot was built on the Microsoft Azure OpenAI service (Microsoft Corp), a leading cloud service known for its robust security features and compliance certifications. The Azure OpenAI service provides advanced encryption and threat management to safeguard data, ensuring that potentially sensitive information shared with our chatbot remains confidential. Importantly, Microsoft Azure’s commitment to data privacy and security means that customer data are not sold or shared with third parties.

Before any implementation for use beyond our own technical evaluation (eg, for public use), all logging of input and output data (as is standard with this Microsoft service) must be disabled to ensure that CAFIbot is in compliance with privacy regulations as storing actual conversations would require a complex ethics approval process. Furthermore, the chatbot will be deployed on a website associated with the TRUSTING project []. To protect user privacy, the chatbot will operate anonymously, and we will not collect or store identifiable information. The website will include a clear disclaimer outlining the intended purpose and limitations of the chatbot.

Ethical Considerations

This study did not involve human participants and, therefore, did not require institutional review board approval. The chatbot was tested using AI-generated conversations, and no personal data were collected. We propose a chatbot solution intended to be used by a vulnerable patient group. Our approach is designed to minimize the risk of the chatbot providing harmful advice, but we cannot guarantee that harmful advice will not be produced given the stochastic nature of LLMs. As we do not log any information about the conversations, we will not be able to detect harmful responses and, therefore, cannot take any action. However, with the right safeguards and precautions, we believe that the benefit to patients of better and more equitable access to medical information outweighs the risk of inaccurate or biased advice. While we emphasize the need to weigh risks against benefits when considering the ethics of using AI to assist vulnerable individuals, we acknowledge the valid ethical concerns and have made multiple design choices to circumvent these issues. First, we provide a chatbot whose intended use is to educate about a mental illness using scripted sources, which is relatively low risk. Second, our framework is likely to significantly alleviate the well-known issue of biased or inaccurate LLM responses by anchoring them on validated sources and by preventing the chatbot from drifting into discussions that may trigger its inherent biases. Finally, we note that educating users on risk is an important aspect of responsible implementation of AI []. Therefore, in our future implementation, we plan to dedicate much effort to formulating disclaimers and educational content that clearly explain the chatbot’s intended use and risks.

Results

Effect of the CAF on Chatbot Integrity

The results from the experiments to nudge CAFIbot toward 3 different out-of-bounds roles are presented in , which shows the median compliance score for each response. The interrater agreement was decent as the compliance scores differed by at most 1 in 90% (65/72) of the responses and the average weighted Cohen κ was 0.921 (SD 0.0084). For each of the 3 facilitators, activating the CAF resulted in substantially improved ability for CAFIbot to adhere to its instructions. With the CAF activated, the fraction of responses with a compliance score of ≥3 was 83% (10/12), 75% (9/12), and 83% (10/12) for roles R1 to R3, respectively, whereas the corresponding values were 17% (2/12), 0%, and 8% (1/12) when the CAF was deactivated. With the CAF activated, CAFIbot received at least one median compliance score of 0 in 5 out of 9 conversations, but in the 3 cases in which a score of 0 was received, CAFIbot was able to recover before the end of the conversation by receiving a subsequent score of 4. In contrast, without the stabilizing influence of the CAF, in each conversation, the responses eventually ended up consistently receiving low compliance scores of 0 and 2, illustrating the self-perpetuating nature of rule violations.

Specificity of the CAF When Answering Schizophrenia Questions

The 10 Schizophrenia Questions Results sheet in shows the full conversation along with the sources referenced, warning messages produced by the CAF, original responses (before refinement), and ratings. In total, 2 responses were flagged by the CAF: one received a warning, and one was modified to comply with the instructions. In both cases, most raters agreed with the criticism of the CAF. Thus, the CAF showed good specificity when answering questions about schizophrenia. It should be noted that we included an improvised question in which we asked the chatbot to rephrase a response in simpler terms, after which it consistently used simpler language, showcasing an attractive advantage of using chatbots in education.

Discussion

Principal Findings

The CAF was highly effective at re-establishing the integrity of the chatbot after it had started to drift from its role and instructions. With the CAF activated, CAFIbot showed a substantially improved tendency to admit its limitations and encourage verification when appropriate and generally tried to steer the conversation back to a permissible topic. However, with the filter deactivated, the chatbot displayed an “eagerness” to expand on out-of-bounds topics, illustrating the importance of robust monitoring mechanisms that detect and prevent this type of conversational drift. Finally, the CAF showed good specificity when answering the 10 questions about schizophrenia, as all warning messages had valid motivations.

Comparison With Prior Work

There has been a surge of research in using advanced generative AI in mental health care services over recent years, but attention has mostly been paid to therapeutic applications and counseling support []. We were not able to identify any research that was mainly focused on the problem of controlling the scope of LLM-powered conversational agents in the context of mental health care, and our research appears novel in that it focuses specifically on this aspect of LLM performance in situations in which controlling the scope and ensuring transparency is of critical importance.

A framework that had many similarities to the CAF is self-reflective retrieval-augmented generation (SELF-RAG), a recently proposed method that significantly improves the accuracy and relevance of retrieval-augmented generation–enhanced LLM responses by using stages of self-reflection [,]. Reflection tokens guide this process, categorizing the need for retrieval and critiquing the generated text, similarly to how our framework used REJECT and WARNING to indicate that a rule had been violated. While both frameworks evaluate whether a response is supported by the retrieved information, SELF-RAG uses self-critical agents more extensively to improve the information retrieval component by also analyzing necessity (whether retrieval is required to answer the query), relevance (whether the retrieved passages relate to the query), and completeness (whether additional passages are relevant). In contrast, the CAF uses critical agents primarily to maintain the integrity of the prompted chatbots. A unique aspect of the CAF is that it uses feedback from the analysis and refinement of the response as reminders to the conversational agent to reduce the likelihood that it will repeat the errors in future interactions. It would be interesting to evaluate the contribution of this feedback mechanism.

Another key difference lies in how critical agents are developed in each framework. SELF-RAG uses supervised training to train LLMs to predict decision tokens such as “relevant” from inputs such as the user query and the retrieved passage. Decision tokens are generated automatically by a state-of-the-art model (GPT-4) prompted for that purpose, and a smaller and more cost-effective student model is trained to mimic GPT-4’s performance. In contrast, the CAF relies on manual prompt engineering, a time-consuming approach that limits the number of labeled examples available for developing and testing critical agents. Adopting a setup similar to that of the teacher-student setup used in SELF-RAG would enable us to train and evaluate the critical agents in the CAF on a much larger and more diverse set of scenarios by using automation to scale up the generation of training examples. Testing this approach, as well as applying SELF-RAG to the information retrieval component of our chatbot, is a promising direction for future development.

Defining the Scope and Boundaries of the Chatbot

Occasionally, unsupported responses did pass through the CAF undetected. These slips in sensitivity are at least partially explained by the ambiguity of the rules that outline the scope of CAFIbot. Indeed, humans themselves will sometimes disagree on whether a response complies with a rule, as illustrated by the less-than-perfect interrater agreement. However, this ambiguity is unavoidable if we wish to leverage the abilities of LLMs; to be effective as a conveyor of information, CAFIbot sometimes has to rely on common knowledge, for example, when explaining concepts not defined in the sources, and this fact inevitably leads to gray areas as it is not possible to unambiguously define “common knowledge” in a few paragraphs of text.

Restricting the GPT’s Responses to Topics in Which It Is Reliable

To illustrate why it is difficult to formulate exact rules for what constitutes a “supported” statement, consider the following question—“Can including more vegetables make my diet healthier?”—as a follow-up to the manual’s recommendation to “adhere to a healthy diet.” If the chatbot says, “Including fruits and vegetables in your diet is generally considered to be healthy,” should we be pedantic and flag this as unsupported because the manual never explicitly specifies what “healthy eating” means, or do we consider this fact to be so basic that we permit it despite not being explicitly stated? Much of the utility of chatbots such as ChatGPT comes from their ability to explain and expand on phrases or concepts, and by being too restrictive, we would lose this feature. Thus, the formulation of such rules is a balancing act between predictability and risk reduction on the one hand (with deterministic algorithms being an extreme example) and usefulness and versatility on the other.

As a general strategy for striking a good balance between risk reduction and utility, we decided to allow the chatbot to make unsupported assertions under the condition that they constituted basic and uncontroversial information or advice. This formulation was intended to capture the situations in which the GPT is at its most reliable, an assertion that can be motivated by observing that there is presumably a lot of training data available for such topics, and information about them on the internet will tend to be more consistent and, thus, reduce unpredictability in the GPT’s responses. Indeed, studies have found that the GPT performs better when asked questions related to popular factual knowledge []. A pitfall of this strategy is that common misconceptions can be hard to distinguish from basic facts due to their pervasiveness, and so its success depends on how well the GPT differentiates between the 2. In any case, this strategy is likely to at least weed out hallucinations and radical statements. Ultimately, our premise is that the increased flexibility afforded to the chatbot by the “basic-and-uncontroversial” rule outweighs the risks associated with the occasional inaccurate advice as such advice will likely be generic but benign. More research into how LLMs classify messages into basic and nonbasic is needed to establish what kind of inaccuracies might slip through a filter that implements this type of rule.

Another important factor for when to restrict the chatbot’s reliance on innate knowledge is the consequence of an inaccurate response. How strictly a rule is interpreted and applied should ideally depend on the stakes involved in the situation. A low-stakes situation in which the chatbot can be afforded more leeway is if the user asks the following: “What are the benefits of taking regular walks?” On the other hand, if the user asks the following—“Should I quit my medication?”—then the CAF should err on the side of caution and restrict the chatbot to parroting the advice from the sources. This strategy could be generalized to include any situation in which LLMs should not be trusted. The social activist role provides a good example. Challenging social stigma could be plausibly interpreted as an action that is encouraged by the source on stigma if consistency with the local context (social stigma) is prioritized over consideration of the broader context (the well-being of an individual learning how to cope with their mental illness). While human experts are good at keeping in mind the broader context when making individualized recommendations, AI seems more inclined to ignore the “big picture” and, thereby, generate responses that are inappropriate when individual considerations are taken into account. Perhaps, in certain situations, a more fruitful approach than trying to align AI and human evaluation is to get the LLMs to detect situations in which AI should not be trusted and increase the strictness of the CAF in those cases. It would be interesting to see research into the ability of LLMs to identify high-stakes situations, subjects not suitable for AI, and other situations that are relevant to controlling the scope of chatbots in a mental health context.

Generalizability to Other Mental Illnesses and Use Cases

We tested the efficacy of the proposed method in the context of educating about schizophrenia, but the general framework can in theory be applied to create an informational chatbot for any mental illness. The variables of the framework that need to be modified are the knowledge base (ie, the sources) and their description in the initial prompt and the parts of the prompts (including the prompts of the judges) that describe the role of the chatbot and that reference schizophrenia specifically. In general, the prompts make few references to schizophrenia, and it should be easy to repurpose the prompts for other mental health conditions. We also note that adding a new rule for the chatbot is simply a matter of adding the rule to the initial prompt as well as updating the prompt of the relevant judges accordingly.

Although we tested the framework on schizophrenia education, we have reason to believe that our results will generalize well to many other clinical contexts. The difficulty of getting a chatbot to reliably adhere to prompt instructions can vary significantly depending on factors such as the nature of the user’s input or the topic being discussed. For example, we found that getting the chatbot to respect source boundaries was far easier when the sources concerned medication (where disclaimers are natural and the content tends to be concrete) than when the sources addressed social stigma and also that long and unfocused queries were more likely to derail the chatbot than concise queries. As such factors, as well as the content being conveyed, vary depending on the clinical user population, it follows that some variability in performance is to be expected across different mental health conditions. We note that the content conveyed in this study represents a particularly tricky prompting challenge as it is easy—in principle—for the chatbot to get lost in a tangential unintended role (eg, therapist) when conveying passages from our source materials, which are written not only to convey facts but also to be emotionally supportive. Looking ahead to practical implementation, we expect this framework to work particularly well when the information is concrete and factual as boundaries in that case will be less ambiguous. As such, a promising use case is a platform for conveying technical information to mental health patients (“Where on the website can I find…”) who may struggle to navigate information when it is presented in more generic formats such as booklets and websites. Indeed, the original intent behind developing this type of chatbot was to answer user questions about the data collection app that will be used in the research project associated with this chatbot. Other promising use cases for chatbots as adaptive mediums of educational content are medical conditions that are highly heterogeneous, such as insomnia as people with insomnia can differ greatly in terms of the information and strategies that are relevant to them.

Suggestions for Future Prompting-Related Research

Structuring Sources for Delivery via a Chatbot

The sources of CAFIbot were written with a static medium of communication in mind. Therefore, the chatbot’s performance might be improved if the sources are instead written specifically to be communicated via chatbots. For example, a static manual may assume that the sections are read in sequence and some sections serve only as introductions to a chapter, but CAFIbot may retrieve them in isolation. As a result, CAFIbot may sometimes produce awkward answers if the retrieved sources lack the preceding context. If the blocks are instead written as self-contained blocks of information, then the chatbot may be more likely to produce a complete and comprehensive answer.

Another way in which the sources could be optimized for chatbot communication is to express them in a more compact technical language so that they take up less tokens and, thus, less space in the context window. The LLM could then “decompress” the information when it conveys the technical information to the user in simpler terms—a task at which LLMs excel. Another advantage of condensing the language of the sources is that the notion of a response being “supported by a source” is more natural when preceded by precise scientific language, and therefore, the LLM might be more inclined to respect the boundaries of the source materials.

Finally, information that is to be conveyed via a chatbot includes a layer of information in addition to the content—instructions on how and when to convey that content. We used square brackets to specify local rules that applied in the surrounding context, such as “…[Ask before presenting this paragraph]…”—an approach that is inspired by teacher-oriented manuals. This convention creates an additional layer of information wherein experts can insert their knowledge and experience to fine-tune the chatbot’s behavior. For example, we noticed that, when CAFIbot was conveying the section on stigma, it had a strong tendency to give advice encouraging the user to engage in social activism—a subject in which we do not want to trust AI for advice. We could correct for this undesired tendency by adding a sentence clarifying the intended lessons and implications of the text, such as “Do not internalize social stigma” as opposed to “Try to eliminate social stigma in your society.” Specifying the intent explicitly could help align the chatbot’s behavior with that of humans. It could also help differentiate the context from overlapping topics—in this case, clinical care focused on individual well-being versus large-scale social change.

Adding Depth to the Chatbot’s Knowledge

CAFIbot was unable to fully answer some of the 10 schizophrenia questions, such as the question about different subtypes of schizophrenia due to that topic not being covered by the manual. This highlights an important difference between static and adaptive education—a static manual must limit the level of detail it provides and the number of topics it covers to not overwhelm the reader and be accessible to individuals across a broad range of abilities and backgrounds. A chatbot need not be subject to this constraint as models such as GPT-4 can adapt to the needs of the situation and present a topic at the appropriate level of detail. This fact could be incorporated into the sources of the chatbot. For example, where the schizophrenia manual only stresses the importance of maintaining a healthy diet, a chatbot could be equipped with additional information that allows it to answer likely follow-up questions. Taking this idea further, we are developing a referral feature (not covered in this paper) that effectively expands the chatbot’s knowledge base by enabling it to redirect the user to other prompted assistants that specialize in a particular topic such as sleep. Enabling the chatbot to respond to likely follow-up questions would also make it more engaging and interesting to converse with.

Future Implementation and Development

Future validation of CAFIbot will focus on testing it with real-world users, including patients with mental illnesses, their families, and the public, through the collection of feedback. To ensure compliance with relevant national and international legislation for LLMs, and recognizing that this application is not classified as a medical tool for medical device regulatory purposes (and does not require HIPAA [Health Insurance Portability and Accountability Act] compliance in the United States), our plan includes a phased implementation process. First, usability feedback will be collected from a user board composed of lived-experience experts, namely, people who experience various mental illnesses, so as to refine the system based on initial impressions. Following this, the chatbot will be deployed on a website [] in which a broad user group can engage with it. Anonymous feedback will be collected through standardized questions designed to assess the chatbot’s utility without compromising user privacy. Specifically, we plan to include a feedback link that allows users to rate the chatbot’s usefulness through structured questions. This process will span several years in alignment with the timeline of the research project (anticipated to conclude in 2028). At the end of this period, we aim to report critical insights into the chatbot’s real-world performance in a follow-up study. Before deployment, we will conduct extensive testing using simulated patient interactions to refine and ascertain the chatbot’s safety and usability.

Study Limitations

Generalizability Is Uncertain

This was a feasibility study focused primarily on the safety aspect, and it has important limitations. We tested the CAF only in a very small number of situations, and therefore, the generalizability of our findings is hard to assess. Furthermore, we fine-tuned the prompts of the AI judges to obtain desirable evaluations on a relatively narrow range of scenarios, including scenarios similar to those generated by the facilitators. Thus, the CAF performance might be lower in scenarios outside the set of scenarios used to fine-tune the prompts. As was mentioned in the comparison with SELF-RAG, automating the collection of data for developing and evaluating the judges using the student-teacher method is a highly promising approach for achieving a more generalizable performance.

Another major limitation is the use of AI as a substitute for human testers for convenience and objectivity. AI cannot fully replicate the diverse and unpredictable nature of human input, in particular of people with schizophrenia. A study that collects and analyzes conversations between CAFIbot and people with schizophrenia would be ideal for discovering potential blind spots in the CAF. However, such a study could be very difficult to set up due to legal and privacy concerns with regard to the collection of such sensitive data from people with schizophrenia, in particular those who are not in a stable state of mind, which are precisely the individuals that would be most valuable from the perspective of evaluating the CAF. An alternative is to enroll clinicians with experience working with patients with schizophrenia to simulate this role. Yet another option, as has been done in other studies on GPT and mental health, is to use public online forums such as Reddit as a source of real-life questions about medical conditions [].

Potential for Biased Performance Evaluation

The person who wrote the prompts (PNW) for and programmed the CAFIbot also designed the benchmarks for evaluating its performance. This could have biased the results, as there may have been a tendency to design tests that measure efficacy in the situations that the system was designed to handle. We expect that we will formulate more comprehensive tests covering a broader range of situations as we acquire more diverse data and viewpoints from external feedback.

Need for Rigorous Testing of Other Aspects of Performance

Designing mechanisms that improve controllability may come at the expense of other aspects of performance such as flexibility or usefulness. More extensive testing is needed to assess various aspects of performance, such as the chatbot’s ability to generate relevant and useful answers. As previously mentioned, we are planning on implementing the chatbot on a website and will add features to enable users to provide anonymous feedback.

Limitations of the Information Retrieval Algorithm

Most of the questions generated by GPT-3.5 turned out to actually be asking 2 to 3 questions in 1 sentence, which incidentally made them particularly challenging for the chatbot to answer via the information retrieval algorithm. The information retrieval algorithm tends to work well when a single query can be answered concisely using a small number of sources but may fail when the answer to a question is spread across multiple sources or when the formulation of the question differs substantially from the description of the sources. This is a well-known limitation of on-demand information retrieval in LLMs.

Conclusions

Using AI agents in a CAF to monitor and refine a chatbot’s responses as well as provide feedback to the chatbot led to responses with substantially better adherence to the chatbot’s sources and instructions and, thereby, a more robust and controllable LLM-powered chatbot. In particular, the chatbot was far more likely to acknowledge its transgressions when it made assertions that were not directly supported by its sources when the CAF was activated than when it was deactivated. Our results suggest that it is feasible to use LLMs as vehicles for mental health information while keeping the risks and consequences associated with LLMs at an acceptably low level. More research is needed to establish the generalizability of our findings.