The Hidden Gaps (and Unexpected Opportunities) Of AI-Powered User Experience In Healthcare

Meaning, Symptoms, Causes, and Treatments

Why Are You Still Gaining Weight While Working Out? Understanding The Mystery Behind Workout And Weight Gain

#घुटने बदलने पर जरूरी h ये physiotherapy exercise #kneepain #TKR #rehabilitation #physiotherapy

Pocketworks is a growth-focused mobile partner, developing apps that make a difference to people and planet. In this article, Director of User Experience, Anna Scandella, discusses the drawbacks and opportunities of using AI for User Experience in Healthcare projects.

Artificial intelligence is transforming healthcare, from diagnostics to treatment recommendations, promising efficiency, scalability, and improved patient experiences. However, when it comes to user experience (UX) in healthcare, at Pocketworks, we’re increasingly seeing AI treated as an infallible solution, which is worrying.

UX in healthcare is complex because patients, caregivers, and healthcare professionals interact unpredictably with digital tools.

To ensure we are designing the right products and services that effectively meet the needs of all audiences, we have to gather robust data that gathers a wide range of opinions and experiences.

The challenge is not just understanding what users do, but why they do it. Despite its sophistication, AI struggles with this critical aspect.

The Assumption vs The Reality

Take the example of one of our clients, Carbs & Cals, a diabetes management app.

As part of our ongoing product and roadmap development, we’ve recently been exploring new features that will encourage users to log and track their food intake.

We wanted to test the difference between AI-generated predictions and researching with actual users. To do this, we built and trained a custom GPT on our key target audience profile.

We asked the AI model to predict what percentage of users would actively engage with logging food they have consumed. The model predicted that 29% would, when we know from our product analytics that only 18% of this core audience do.

This gap was significant enough to reveal how AI models can often misread behavioural intent.

Why AI Falls Short in Healthcare UX

AI is highly effective at recognising patterns in large datasets, making it invaluable for tasks such as processing medical records, or predicting disease progression. However, its limitations become more apparent when it comes to human behaviour.

One of the core issues is the training data. Many AI models are built using historical datasets that fail to represent diverse populations. For example, women have been significantly underrepresented in clinical trials for decades. Research shows that only 33% of participants in cardiovascular trials are female, despite heart disease being a leading cause of death among women.

When AI models misinterpret user behaviour, they reinforce biases and drive flawed product decisions that could have serious consequences for patient care.

Gartner predicts that by 2026, AI models from organisations that operationalise AI transparency, trust, and security will achieve a 50% improvement in terms of adoption, business goals, and user acceptance.

AI’s Role in User Research

This doesn’t mean AI has no place in UX research. It can be a valuable tool for drafting surveys, identifying common themes, and analysing large datasets. It speeds up the research process and can highlight patterns that might otherwise go unnoticed. However, AI should never replace direct user research.

In healthcare, user behaviour is often influenced by deeply personal factors that AI cannot predict. A person may avoid logging food because of guilt or anxiety around eating habits, not because they dislike the idea. AI is unable to interpret these emotions accurately.

AI often struggles with sentiment analysis, frequently misreading sarcasm. A user might say, “This app is great at suggesting the wrong foods”, and an AI might register that as positive feedback (this actually happened to us when evaluating the sentiment of an app review). A human researcher would instantly recognise the sarcasm.

The Ethical and Sustainable Cost of AI Bias

Flawed AI models also present ethical and sustainability concerns. A product built on incorrect assumptions will eventually require rework. That means collecting new data, retraining models, and revising entire product strategies. The financial and environmental impact of this is concerning.

Training large AI models consumes enormous amounts of energy. Some AI systems require computational power equivalent to the lifetime emissions of five cars. Rebuilding models due to bias or flaws increases costs, making AI inefficiencies both a sustainability and ethical issue.

Responsible AI development minimises waste and ensures effective technology from the start.

A Hybrid Approach: AI and Human Expertise

Effective healthcare UX requires AI and human insight to work together, with AI as a tool, not a decision-maker. Key principles for AI-supported UX research:

Diversify Training Data: Ensure datasets represent diverse demographics.
Validate with Real Users: Test AI insights against actual behavior before major decisions.
Use AI for Patterns, Not Interpretation: AI detects trends; humans analyse causes.
Monitor Bias Continuously: Regularly review AI to prevent bias.
Prioritise Ethical Development: Consider AI’s sustainability impact to minimise waste.