This article first appeared in DOTmed HealthCare Business News.
The integration of artificial intelligence (AI) into clinical decision support (CDS) systems holds immense promise for clinical practice. It can help clinicians analyze vast amounts of information, reduce manual work, and deliver key insights at the point of care. As promising as these advancements are, they come with potential risks in clinical workflows.
Algorithm vigilance, also known by the tongue-twisting name algorithmovigilance, refers to the continuous monitoring and evaluation of algorithms to ensure they maintain reliability, safety, and effectiveness over time. Much like the practice of pharmacovigilance—which monitors drugs for adverse effects after they hit the market—algorithm vigilance ensures that AI tools perform as intended in real-world clinical environments. Ongoing attention and oversight matters as performance – specifically accuracy – can wane or drift over time. Probabilistic and non-deterministic models are especially vulnerable to these issues, but they are also present in the algorithms that underpin large language models and agents becoming widely adopted today.
AI has proven its ability to transform healthcare by improving efficiency, reducing cognitive load, and enhancing patient outcomes. Predictive models can assist in early diagnosis and resource allocation. AI can streamline clinical documentation, and decision-support tools can help reduce medication errors.
However, as with any innovation in medicine, we must also recognize the risks. One significant risk is bias. AI can be tainted by systemic biases reflected in healthcare data, which may lead to disparities in patient care. For example, algorithms trained on datasets without appropriate representation might deliver less accurate predictive outcomes for minority groups, people with rare clinical conditions, and other under-researched populations. Securing access to quality data with appropriate representation is no easy task.
Another risk is the lack of trust many clinicians have when it comes to AI. Some healthcare providers are hesitant to rely on AI tools, because they want to be sure they are using evidence-based, accurate information to diagnose and treat patients. Effective education, transparent design, proactive governance, and assurance of continuing evaluation can mitigate these concerns.
Monitoring the performance of all AI agents and tools is important for the duration they are used in clinical practice. AI performance may change over time and as populations, treatments, and standards of care evolve. This phenomenon, known as algorithmic drift, further emphasizes the need for ongoing oversight.
Step 1: Engage stakeholders early and often
Collaboration is key. Clinical and IT leaders, as well as key interprofessional clinician stakeholders, must work closely with their AI vendors. Together, they can ensure that AI tools address real-world clinical challenges. They should start with a clear use case, design and continue to collaborate through deployment and beyond.
Step 2: Establish a governance framework
An interdisciplinary team should share responsibility for approval and oversight of new use cases, recommending updates, and establishing metrics. Clinical, technical, executive, and even patient voices are important in this effort. Before launching any AI tool, this team should define clear processes for addressing accuracy, quality, or safety concerns.
Step 3: Prioritize transparency and accountability
AI tools must provide interpretable outputs. For example, AI-powered search in clinical decision support should share its underlying reasoning, flag high-risk questions, and cite trusted sources. Reinforce the idea that AI is a tool to support—not replace—critical thinking or clinical judgment. It’s important that users understand they are accountable for their decisions, and AI is one tool they can use in their decision-making process.
Step 4: Validate algorithm performance in real-world conditions
Teams should start small with pilot programs to help uncover unintended consequences or limitations of an AI tool before broader deployment. They should also consistently assess the impact of AI tools on efficiency metrics, clinician satisfaction, and ultimately patient outcomes.
Step 5: Monitor metrics that matter
Drift and bias are inevitable because medical knowledge is always evolving. Minimize the impacts by regularly auditing AI tools to detect changes in performance and any emerging disparities in predictions across demographic groups. Additionally, monitor clinician feedback to pinpoint any issues with the AI’s comprehension, integration, clarity, or interpretability.
Building trust with algorithm vigilance is imperative to have robust systems for monitoring and improving AI tools to achieve the full potential of these groundbreaking technologies.
Read more about AI adoption in clinical practice.