Is Artificial intelligence Stupid? – Misconceptions to a Legal Albatross

Is Artificial intelligence Stupid? – Misconceptions to a Legal Albatross

My engagement with artificial intelligence (AI) began in 1998, when I was a young physician exploring medical informatics. My first formal exposure came through Enrico Coiera’s 1997 textbook, Guide to Medical Informatics, the Internet and Telemedicine, which I keep in my library. Two chapters in particular, “Intelligent Systems” and “Intelligent Monitoring and Control”, introduced concepts that remain central to AI today.

Then, as now, AI depended on access to data, sufficient computational capacity, machine learning methods, and algorithms capable of generating projections from data. For many years, limitations in each of these areas slowed AI’s progress. Over roughly the past five years, however, public awareness and adoption of AI have accelerated dramatically.

There has been influx of different levels of expertise in AI as a subject matter with so much confusion and mainly from people who just came out of few months of attending courses and without a depth of understanding of AI limitations, ethical and legal issues that may be involved.

This article is divided into two parts and explores all these from my perspective as a physician, an attorney, a clinical informatician and ethicist. This part of the article looks mainly at misconceptions while the legal issues will be discussed in a separate article.

Misconceptions That Prompted This Article

Several misconceptions about AI became especially apparent to me during a recent alumni retreat. Much of the discussion centered on data availability and access, particularly in low- and middle-income countries (LMICs), where some participants assumed that useful data in healthcare clinical decision will be limited by access and storage.

Other assumptions were more clinically focused. Some suggested that AI could provide instant answers to nearly every clinical question, potentially enabling an inexperienced physician to manage complex cases with ease. There was also a belief that medical negligence would be minimized, or even avoided, whenever a physician used AI as part of clinical decision-making. One discussion leader went further, describing AI tools as medical devices that might reduce liability. This is a widely held view in some AI- healthcare communities especially in the UK.

These assumptions deserve scrutiny. Drawing on my background in clinical informatics and my legal training as an attorney, I will also address the legal implications of relying on AI in clinical settings, as stated in a second and separate article.

This article aims to challenge those assumptions and show how misconceptions about AI, particularly in clinical decision support, can create burdens for clinicians, patients, and health systems, including potential sources of litigation.

Databases, Data Collection, and Retrieval

There is a general misconception that Artificial Intelligence (AI) looks things up in a giant medical database. But the reality in Clinical Decision Support (CDS) lies in access to training data versus operational data, computation of a prediction which is not a mere search and the quality of the data along with its representativeness to the situation.

1.        Training data vs operational data: The training data tend to exist as data collected from historical Electronical Health Records (EHR), imaging archives, registries, clinical trial data, etc. Often, these are used to learn patterns and towards machine learning. There could be bias in the selection or acquisition of these data that may be unique to the population. The operation data are more likely live patient’s data from laboratory results, vitals, medical notes/records, images, etc that are then fed into the model at run time to produce a result towards clinical decision. These operational data may interact with the training data and may also become part of the pool of the training data later.

2.        Not simple retrieval: AI-CDS should generate predictions rather than simply search medical references such as UpToDate or Cochrane. These predictions are derived from patterns learned across thousands to millions of prior cases.

3.        Data quality and representativeness: This could be based on the adequacy of the data collected and the under-representation of training data for certain populations like ethnic minorities or in rare conditions, which can create a model which will systematically underperform for those groups. This creates a core source of bias and potential harm

Consider an AI cardiac-event alert model trained primarily on data from a single tertiary care center. In that original setting, the model may appear effective because it has learned patterns that reflect that hospital’s patient population, workflows, documentation practices, and clinical environment.

However, when the same model is used in a community hospital with a different patient mix and workflow, it may perform differently. It could generate excessive alerts in one setting while missing or misclassifying risks in another.

For the clinician, the output may appear as a simple “alert” for a possible cardiac event. The alert is not a database lookup; it is a statistical prediction shaped by the data environment in which the model was trained. If that environment does not reflect the current patient’s context, the alert may be clinically irrelevant or misleading.

The consequences can be significant. An inappropriate alert may prompt unnecessary testing, treatment, or intervention, increase costs and expose the patient to avoidable iatrogenic harm.

Computational Power

The general misconception is that more computational power leads to a better clinical decision support, but the reality is far from this.

It is stated that compute enables complexity but not wisdom. It means that high computational power allows training of large models, for example deep neural networks for imaging, but it does not guarantee clinical validity, safety or fairness.

There is also an aspect of din=minishing returns with more computational power. Beyond a point, more parameters and more Graphics Processing Units (GPUs) may improve benchmark metrics slightly but add opacity, cost, and energy use without clear clinical benefit.

There are also regulatory and operational constraints. In hospitals, latency, integration with Electronic Health Records (EHR), and reliability often matter more than raw model size.

An applicable clinical scenario may occur for example in a smaller, well-validated logistic regression model for Deep Venous Thromboembolism (DVT) risk, integrated cleanly into the EHR, may be safer and more usable than a massive deep-learning model that is opaque, slow, and poorly integrated, even if the latter has marginally better Area Under the Receiver Operating Characteristic (AUROC) in a paper. The AUROC is often used in statistics and machine learning, measuring the ability of a model to distinguish between two classes which in this scenario will be absence or presence of DVT, and with the discriminative power of the probability of a random positive result is higher than that of a negative result from that model.

This can lead to a bogus clinical decision system.

Algorithm

A major misconception Is that algorithm is objective and neutral.

In reality, algorithms encode assumptions. The choice of features, loss functions, thresholds, and evaluation metrics all embed value judgments with example that could be for the prioritization of sensitivity versus specificity, or short-term versus long-term outcomes.

The saying of “Garbage In, Garbage Out” (GIGO)” or “Rubbish In, Rubbish Out” (RIRO) is more subtle but relevant because even with clean data, if the algorithm optimizes the wrong target like predicting who gets admitted instead of who really needs admission, it can reproduce and entrench systemic biases.

In addition, context dependence is relevant in the deployment of any algorithm which may be tuned for one setting like US/UK/EU versus Low-Middle Income Countries (LMIC), or Intensive Care Unit (ICU) versus outpatient, may be unsafe in another.

Such can happen in a clinical scenario where a triage algorithm trained to predict “likelihood of ICU admission” might systematically under-prioritize patients from groups historically less likely to be admitted (due to structural bias), even if their physiological risk is high. This makes it not neutrality but a codified history.

Machine Learning (ML)

The general misconception is that ML learns like a human clinician and that it keeps getting smarter on its own and faster.

But the reality is that ML is more about pattern recognition that the understanding of data or issues under question. ML models learn statistical associations, not pathophysiology or clinical reasoning. Basically, it lacks the ability to know or appreciate why a pattern exists.

The two major ML models are the static and adaptive models.

Many of the deployed models are static in which once trained, remain frozen. It means that they do not automatically improve unless retrained. The retraining means using extra resources that may include new set of data, update on algorithm, etc with associated cost that smaller institutions may not be able to afford.

On the other hand, the adaptive/continual learning systems raise extra regulatory and safety issues because performance can drift over time.

Data drift and performance decay can occur with changes in practice patterns, coding, or population, for example new treatment options, pandemics, which can then degrade model performances if not monitored and recalibrated. These can also come with additional resources that many small institutions may not be able to afford.

An illustration of this will be an example of a pneumonia mortality prediction model trained pre-COVID which may later misclassify COVID-related pneumonia if it is not updated and leading to underestimation of risk, inappropriate triage and treatment options.

Outputs and their role in clinical decision making

As was with “alerts” in EHR, there is a misconception that the AI gives the answer and the clinician just follows it. This part also brings to the fore the ethical and legal consequences of such misconception.

The reality is that most AI platforms used in healthcare are not supposed or created to make the clinician follow an “order”.

AI-CDS is usually advisory like the “alerts” in EHR with most with most system output a risk score, classification, or ranked list like stating if a high risk for pulmonary embolism, a suspicious nodule or not consider Acute Kidney Injury, all not a binding order.

Human-AI teaming is essential, and the best practice is to treat AI as a second reader or an early warning system, with the clinician retaining responsibility to interpret, contextualize and sometime override. This responsibility create legal liability for the clinician with no legal basis for a “medical device” error or malfunction as has been alluded to by some people.

The key is the need for explainability and transparency with outputs that show key features, confidence levels, or example cases can improve trust and appropriate use, bur=t many models remain black boxes.

A situation can arise in an instance where an AI flags a lung nodule as highly suspicious for malignancy. The radiologist reviews the image, considers prior scans, patient history, and may agree or disagree. The AI is a prompt, not a verdict.

Another example is that a CDS tool may recommend a treatment plan for a patient with atrial fibrillation based on evidence-based guidelines and an algorithm that supports prescribing an anticoagulant. However, the tool might overlook a recent minor decline in the patient’s creatinine clearance that should have prompted a dose adjustment because the information appeared only in a prior office note as free-text narrative, making it difficult for the AI model to extract accurately. This illustrates why human oversight remains essential and should be required, rather than relying entirely on AI output, particularly when the clinician has limited experience.

These issues can create legal albatross and legal landmines that require awareness for effective navigation.

In the second part of this article to be published separately, I hope to explore issues of AI-CDS liability landscape exploring US versus UK/EU regulations, defining AI-CDS as a medical device, clinician liability under medical negligence based on standard of care and reasonableness, product liability looking at developer/vendor liability, looking at this if it could be “shared liability among developers, vendors, clinicians, regulators, etc, practical steps in risk management, while also exploring some ethical issues that may affect the use of AI-CDS.

In conclusion, there is a need to further define the essence of AI-CDS in clinical care, along with greater education to get rid of some of the common misconceptions that have been pointed out earlier. There is a need for a greater oversight in the use of AI-CDS to also include the education of clinicians on AI-CDS as a tool that will still require adequate application of clinical skills with AI-CDS.

--------------------------------

Dr. Segun Dawodu is the owner and chief webmaster of Dawodu.com. He has a medical degree from University of Ibadan, MBA from Johns Hopkins University Carey Business School, MSc in Global Health Leadership from University of Oxford, LL.B from University of London, MS in Clinical Informatics from Northwestern University, Evanston, IL, LL.M in Medical ethics/Intellectual property law from University of London, LL.M in International Corporate and Commercial Law from Kings’ College, London, LL.M in US law from George Mason University, Postgraduate Diploma in Economics of Law/Competition Law/Antitrust Law from King’s College, London, Creative Writing Diploma from University of Pennsylvania and Associate of Kings College ( a Bachelor’s degree equivalent in Theology, Philosophy and Ethics). He is currently undergoing Ph.D (law) research in Telemedicine, AI and cross-border policy and regulations at University of London, Institute of Advanced Legal Studies.