Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI npj Digital Medicine

Recent findings have shown that current deidentification practices for accelerometer-measured physical activity data are insufficient to ensure privacy64 and health data breaches have increased over the past decade65. Masking individual data points by grouping data points with similar characteristics is no longer a reliable approach in modern data landscapes where data is easily linked across multiple sources. Additionally, any data that can be combined with other information is considered personal identifiable information, making it difficult to distinguish between data categories. To safeguard personal records against revealing individual identities, more advanced techniques are necessary beyond simply categorizing data as personal identifiable information or not. These findings have important implications for the use of chatbots for improving lifestyle behaviours, highlighting the need for robust data privacy measures to protect user privacy.

This process helps the chatbot determine the urgency of the patient’s condition and guide them to the most suitable course of action, whether it’s self-care advice, scheduling an appointment, or directing them to emergency services. In healthcare, AI-powered chatbots are the first point of contact for patients seeking medical assistance. Patients can interact with these chatbots through various messaging platforms, web interfaces, or mobile applications. The chatbot initiates a conversation by asking patients about their symptoms, medical history, and other relevant information. Longhurst, one of the study authors, doesn’t see those findings as showing that chatbots are better than doctors at answering patient questions. His takeaway is that doctors under tight time constraints — such as when flooded with patient portal messages — write short and just-the-facts responses, while chatbots generate longer answers within seconds.

Some teachers are experimenting with chatbots to create course plans, assignments, and exam questions more quickly than they have traditionally done on their own. “One educator thought it did a reasonably good job, although it needed some tweaking,” says Willies-Jacobo at Kaiser. Students can use chatbots as starting points for research, writing, and studying — essentially as sophisticated search engines to aggregate and summarize information (while making sure to verify it).

MEDICAL IMAGING

Verify critical health information with your health care professional to ensure its accuracy. This is when the chatbot will confidently present incorrect or misleading information as fact. To combat this issue, DUOS uses the RAG technique to personalize the AI’s knowledge base to your health plan’s benefits and individual information rather than a broad knowledge base.

Resistance is a natural behavioral response to innovative technology, as its adoption may change existing habits and disrupt routines (Ram and Sheth, 1989). The delayed transmission of innovation in the early stages of its growth is primarily attributed to people’s resistance behavior (Bao, 2009). Previous research has primarily focused on the impact of rational calculations underlying individuals’ technology adoption/resistance intentions on behavioral decisions. For example, the technology acceptance model (TAM) proposes that individuals’ desire to accept a certain technology is determined by the degree to which it improves work performance and its ease of use (Davis, 1989; Tian et al., 2024). Furthermore, according to the equity implementation model (EIM), people’s concerns about the ratio of technical inputs to benefits and comparisons with the advantages obtained by others in society have a substantial influence on their adoption behavior (Joshi, 1991).

The R2 values for resistance intention, resistance willingness, and resistance behavioral tendency were 0.377, 0.391, and 0.429, respectively, each of which was higher than 0.19 (Purwanto, 2021). The f2 value was used to estimate whether the latent variables had substantial effects on the endogenous variables. Table 5 indicates that, f2 values range from 0.015 to 0.646; this clarifies that in the model, two paths have weak effects and four paths exceed the medium effect (higher than 0.15; Cohen, 1988).

Participants assigned to the control group were not given access to COVID-19 vaccine-related chatbots. After the intervention period, participants from both groups were asked to complete the same questionnaire about COVID-19 vaccines; chatbot users were asked to evaluate their usage experience. Further research on the effects of prototypical properties by Blanton et al. (2001) suggests that negative prototypical perceptions are more likely to lead to personal behavioral changes. In an investigation of teenage smoking resistance, it was observed that negative prototype perceptions were more likely to profoundly influence behavioral decisions than positive perceptions (Piko et al., 2007).

For the four content analyses (completeness and conformity for both Chat-GPT versions), the interrater agreement was calculated via Cohen’s kappa. The clear wording, official publication date and special focus of the guidelines every four years make it methodologically feasible to determine whether the current guideline text and the underlying primary literature were part of the training text of the AI chatbots. Get the latest information about companies, products, careers, and funding in the technology industry across emerging markets globally.

Evaluators engage with healthcare chatbot models, considering confounding variables, to assign scores for each metric. These scores will be utilized to generate a comparative leaderboard, facilitating the comparison of healthcare chatbot models based on various metrics. Black patients were less likely than White patients to trust the technology, while Native American patients were more likely.

For example, adjusting the beam search parameter66 can impact the safety level of the chatbot’s answers, and similar effects apply to other model parameters like temperature67, which can influence specific metric scores. The Privacy metric is devised to assess whether the model utilizes users’ sensitive information for either model fine-tuning or general usage42. First, users may share sensitive information with a chatbot to obtain more accurate results, ChatGPT but this information should remain confined to the context of the specific chat session and not be used when answering queries from other users43. Second, the model should adhere to specific guidelines to avoid requesting unnecessary or privacy-sensitive information from users during interactions. Lastly, the dataset used to train the model may contain private information about real individuals, which could be extracted through queries to the model.

GPT generates the next text in response without really “understanding” which may be a daunting concern in mental healthcare. Thus, the “intelligence” of the bots which is currently limited to simulated empathy and conversational style is questionable to address complex mental health and to demonstrate effective care. Thus, digital tools need to be used as a part of the “spectrum of care” rather than just as a sole measure of healthcare. AI plays a crucial role in dose optimization and adverse drug event prediction, offering significant benefits in enhancing patient safety and improving treatment outcomes [53]. By leveraging AI algorithms, healthcare providers can optimize medication dosages tailored to individual patients and predict potential adverse drug events, thereby reducing risks and improving patient care. Personalized treatment, also known as precision medicine or personalized medicine, is an approach that tailors medical care to individual patients based on their unique characteristics, such as genetics, environment, lifestyle, and biomarkers [47].

Chatbots can also be useful after an initial diagnosis, and can be used to help individuals manage their long-term health (Bates, 2019). You can foun additiona information about ai customer service and artificial intelligence and NLP. It is predicted that in the future, patients will have the ability to share their healthcare records and information with medical chatbots, to help further improve their application and accuracy (Bates, 2019). AI has the potential to revolutionize mental health support by providing personalized and accessible care to individuals [87, 88].

Artificial intelligence chatbots for the nutrition management of diabetes and the metabolic syndrome

The insurgence of ChatGPT has led to questions about how the tool can help remove some secure direct messaging duties from provider workloads. For the study, which was published in JMIR mhealth and uhealth, researchers conducted an exploratory observational study of ten mental healthcare apps with a built-in chatbot feature. They qualitatively analyzed 3,621 consumer reviews from the Google Play Store and 2,624 consumer reviews from the Apple App Store. Additionally, Northwell Health launched a chatbot at the beginning of the year in an effort to lower morbidity and mortality rates among pregnant people. Called Northwell Health Pregnancy Chats, the chatbot provides patient education, identifies urgent concerns, and directs patients to an ED when necessary.

However, RPM technologies present significant opportunities to enhance patient well-being and improve care by allowing providers and researchers to take advantage of additional patient-generated data. Genomics has sparked a wealth of excitement across the healthcare ChatGPT App and life sciences industries. Genetic data allows researchers and clinicians to gain a better understanding of what drives patient outcomes, potentially improving care. The findings from AI chatbots have limited generalizability because of their dynamic nature.

of Americans Would Be Uncomfortable With Provider Relying on AI in Their Own Health Care

This was generally consistent across disease severity, which researchers tested by prompting patients to imagine they had leukemia versus having sleep apnea. BS has full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. We planned to include sedentary behaviour as an outcome of interest, however there was insufficient data in the included studies to perform a meta-analysis on this outcome. Furthermore, following registration of our protocol, we decided to add subgroup analyses for output type and AI or NLP use, as it would be a valuable addition, and may help guide future research on chatbot development. Epic’s sepsis algorithm, as an example, wasn’t technically approved as a medical device and was only used with electronic health records as a predictive or screening tool, without making any health claims that would be required to go through the FDA at the time. In 2022, the FDA released updated guidance that these tools should now be regulated as medical devices, but will not pursue any actions against similar algorithms currently in use.

From a Saudi perspective, Sehaa, a big data analytics tool in Saudi Arabia, uses Twitter data to detect diseases, and it found that dermal diseases, heart diseases, hypertension, cancer, and diabetes are the top five diseases in the country [67]. Riyadh has the highest awareness-to-afflicted ratio for six of the fourteen diseases detected, while Taif is the healthiest city with the lowest number of disease cases and a high number of awareness activities. These findings highlight the potential of predictive analytics in population health management and the need for targeted interventions to prevent and treat chronic diseases in Saudi Arabia [67]. AI can optimize health care by improving the accuracy and efficiency of predictive models and automating certain tasks in population health management [62].

AI is advancing the state of healthcare

However, AI chatbots present unique and varied challenges when it comes to protecting patient privacy and complying with HIPAA. Two recent viewpoints published in JAMA explored the health privacy and compliance risks of AI chatbots, each offering thoughts on how providers can navigate HIPAA compliance and honor their duty to protect patient data as these tools gain prominence. To ensure objectivity and reduce human bias, providing precise guidelines for assigning scores to different metric categories is indispensable. This fosters consistency in scoring ranges and promotes standardized evaluation practices. Utilizing predefined questions for evaluators to assess generated answers has proven effective in improving the evaluation process.

AI-powered chatbots help reduce the workload on healthcare providers, allowing them to focus on more complicated cases that require their expertise. Thus, it would be valuable for future studies to incorporate in-depth interviews as well as qualitative research methodologies such as grounded theory to obtain more comprehensive results on the impact of health chatbots on individuals. Third, the measurement of resistance behavioral tendency in this study may not strictly represent actual benefits of chatbots in healthcare resistance behavior. It is recommended that future research adopt more direct methods to measure people’s actual resistance behavior toward health chatbots. Fourth, consistent with prior research, the current study investigated resistance psychology and behavior primarily from the perspective of individual perceptions, attitudes, and behaviors. However, the factors influencing individual resistance to innovative technologies are diverse (Talwar et al., 2020; Dhir et al., 2021).

Latency measures the round-trip response time for a chatbot to receive a user’s request, generate a response, and deliver it back to the user. Low latency ensures prompt and efficient communication, enabling users to obtain timely responses. It is important to note that performance metrics may remain invariant concerning the three confounding variables (user type, domain type, and task type).

• This suggests that people’s heuristic perceptions of the negative images and risk beliefs concerning health chatbots are important determinants of their resistance behavioral tendency.
• In the future, the authors predicted that AI chatbot developers will work directly with healthcare providers to develop HIPAA-compliant chat functionalities.
• The questionnaires were standardised across countries and contexts to compare outcome variables of interest.

According to Prof. Gilbert, LLM-chatbots developed today do not meet key principles for AI in healthcare, such as bias control, explainability, systems of oversight, validation and transparency. Some of the main takeaways from these systems center on the importance of human engagement even when chatbots are doing much of the communicating. Aside from keeping doctors in the loop on messaging and clinical assessment, health systems need to supplement electronic interactions with personal attention. It’s much easier for a human to demonstrate those qualities when checking symptoms than a chatbot, the researchers said. But if chatbots can demonstrate that they are effective, they are permissible to patients. Those figures may prompt some doubt in online symptom checkers, and patients are catching on.

The closed-source AI, Chat-GPT 3.5, was accessed as free to use version on July 12, 2023. The complimentary Bing version of Chat-GPT 4 was subsequently accessed on September 5, 2023. All prompts pertaining to the relevant chapters were posed within a single session in the sequence in which they appeared in the guidelines.

Ilara Health has developed a chatbot that serves as a first point of analysis for symptoms. It aids in identifying possible health issues that need further investigation by a doctor. The chatbot’s AI compares reported symptoms with a vast database of medical information to suggest potential diagnoses, streamlining the initial stages of the healthcare delivery process. Navigating regulatory landscapes can present significant hurdles for AI chatbots in healthcare (30).

That means healthcare organizations, facing labor shortages, can dedicate their resources to high-acuity issues. The current percentage of the population with access to the internet is 64.4% (Statista, 2023). One out of 10, a rough average, have access to conventional mental healthcare, which means that not everyone has access to mental healthcare, especially in the LMICs (Singh, 2019). Improving accessibility is another important factor to consider when determining whether ChatGPT is ready to change mental healthcare. ChatGPT should be designed to be accessible to those who may not have access to the internet.

The road that lies ahead, despite being replete with opportunities, calls for a strategy that is methodical, ethically grounded, and technologically fortified to properly actualize the enormous potential that the combination of AI chatbots and healthcare heralds. One of the most pernicious difficulties, however, is the bias inherent in AI models that are largely trained on Western data. These biases can lead to erroneous suggestions, reducing confidence and trust in AI solutions even more. To address this, there is an urgent need for statistics embedded in African contexts, as well as collaborations that bring local healthcare professionals, ethicists, and community leaders to the development table. Such collaborative efforts will not only increase the relevance of AI chatbots but will also develop trust and acceptance among the communities they want to serve.

Intelligent triage algorithms enable chatbots to accurately assess the severity of a patient’s condition and determine whether a hospital visit is warranted. When self-care or visiting a primary care physician would suffice, chatbots provide relevant guidance and redirect patients to the appropriate care setting. This reduces waiting times for patients who require immediate hospital attention and alleviates the burden on emergency departments, allowing healthcare professionals to focus on critical cases. AI-powered chatbots collect information and actively provide personalized recommendations and guidance to patients. They analyze the data gathered during the conversation and compare it with vast medical knowledge bases to offer tailored advice on symptom management, self-care measures, and when to seek further medical attention. This proactive approach empowers patients to make informed decisions about their health, reducing unnecessary waiting times and ensuring they receive appropriate care.

ChatGPT AI Chatbot Proves Effective for Patient Queries, Health Literacy

AI algorithms can analyze patient data to assist with triaging patients based on urgency; this helps prioritize high-risk cases, reducing waiting times and improving patient flow [31]. Introducing a reliable symptom assessment tool can rule out other causes of illness to reduce the number of unnecessary visits to the ED. A series of AI-enabled machines can directly question the patient, and a sufficient explanation is provided at the end to ensure appropriate assessment and plan. Developed by a team of medical professionals and AI experts, Medibot focuses on delivering personalized medical advice and triage services. This chatbot uses natural language processing (NLP) algorithms to understand patient symptoms and provide appropriate recommendations, ranging from at-home remedies to suggesting when to consult a healthcare professional. Medibot’s versatility has made it a valuable resource in providing reliable and accessible healthcare advice to a wide range of individuals.

By using AI instead of seeking help from caregivers, those living alone may experience more feelings of loneliness. While technology can provide valuable support to find information about health benefits, it cannot replace human interaction. With an increased use of technology, maintaining regular contact with friends and family is important to prevent isolation. New technologies are helping aging adults navigate complicated health care benefits, but it does come with some challenges. The success of NHS 111 Online demonstrates the potential of AI-powered chatbots to streamline patient triage and improve access to care on a large scale. Implementing NHS 111 Online has significantly benefited patients and the healthcare system.

When combined with automatic evaluation methods like ROUGE and BLEU, these benchmarks enable scoring of introduced extrinsic metrics. Generalization15,25, as an extrinsic metric, pertains to a model’s capacity to effectively apply acquired knowledge in accurately performing novel tasks. In the context of healthcare, the significance of the generalization metric becomes pronounced due to the scarcity of data and information across various medical domains and categories. A chatbot’s ability to generalize enhances its validity in effectively addressing a wide range of medical scenarios.

Despite their advantages, AI chatbots have notable limitations, particularly their ability to provide nuanced emotional support. Mental health issues are complex and deeply personal, often requiring a level of empathy and understanding that AI currently cannot replicate. While chatbots can offer essential support and information, they lack the emotional intelligence to fully grasp the subtleties of a user’s emotions and experiences. This can result in responses that may seem generic or inappropriate, failing to effectively meet the user’s needs (Miner, Milstein, and Hancock, 2017).

In the ensuing sections, we expound on these components and discuss the challenges that necessitate careful consideration and resolution. The Interpretability metric assesses the chatbot’s responses in terms of user-centered aspects, measuring the transparency, clarity, and comprehensibility of its decision-making process45. This evaluation allows users and healthcare professionals to understand the reasoning behind the chatbot’s recommendations or actions. Hence, by interpretability metric, we can also evaluate the reasoning ability of chatbots which involves assessing how well a model’s decision-making process can be understood and explained. Interpretability ensures that the chatbot’s behavior can be traced back to specific rules, algorithms, or data sources46. Findings from this systematic review and meta-analysis indicate that chatbot interventions are effective for increasing physical activity, fruit and vegetable consumption, sleep duration and sleep quality.

• Additionally, the leaderboard allows users to filter results based on confounding variables, facilitating the identification of the most relevant chatbot models for their research study.
• In healthcare, AI-powered chatbots are the first point of contact for patients seeking medical assistance.
• By investigating the role of irrational factors in health chatbot resistance, this study expands the scope of the IRT to explain the psychological mechanisms underlying individuals’ resistance to health chatbots.
• However, few studies have examined the influence of irrational motivations and psychological mechanisms on health chatbot resistance behaviors.
• Scepticism regarding the accuracy and effectiveness of healthcare chatbots may be a significant barrier to widespread adoption.

Health chatbot providers, in particular, should utilize influential media channels to continuously disseminate information regarding health chatbots’ scientific utility to address asymmetric perceptions and promote an objective understanding of this technology. First, our sample sizes across all three regions were smaller than our target sample sizes. Although we have found some statistically significant findings, our small sample sizes might lack sufficient power to detect the effect of the intervention in its entirety, e.g., dose-dependent effect (Supplementary Table 14).

In the second viewpoint article published in JAMA surrounding AI chatbots and health data privacy, the authors posited that AI chatbots simply cannot comply with HIPAA in any meaningful way, even with industry assurances. Generative AI in healthcare offers promise for tasks such as clinical documentation, but clear regulations and standards are needed to maximize benefits and minimize risks. However, a notable concern arises when employing existing benchmarks (see Table 2) to automatically evaluate relevant metrics. These benchmarks may lack comprehensive assessments of the chatbot model’s robustness concerning confounding variables specific to the target user type, domain type, and task type. Ensuring a thorough evaluation of robustness requires diverse benchmarks that cover various aspects of the confounding variables.