There's been little research on whether the 320,000 medical appsnow in use actually improve health, according to a report on AIpublished Dec. 17 by the National Academy of Medicine. (Photo:Shutterstock)

Health products powered by artificialintelligence, or AI, are streaming into our lives, from virtualdoctorapps to wearable sensors and drugstorechatbots.

IBMboasted that its AI could "outthink cancer." Others say computer systems that readX-rays will make radiologists obsolete.

"There's nothing that I've seen in my 30-plus years studyingmedicine that could be as impactful and transformative" as AI, saidDr. Eric Topol, a cardiologist and executive vice president ofScripps Research in La Jolla, Calif. AI can help doctors interpretMRIs of the heart,CT scans of the head and photographsof the back of the eye,and could potentially take over many mundane medical chores,freeing doctors to spend more time talking to patients, Topolsaid.

Even the Food and Drug Administration ― which has approved morethan 40 AI products in the past five years ― says "the potential ofdigital health is nothing short of revolutionary."

Yet many health industry experts fear AI-based products won't beable tomatch the hype. Many doctors and consumer advocates fear that the tech industry, which lives bythe mantra "failfast and fix it later," is putting patients at risk ― and that regulators aren't doingenough to keep consumers safe.

Early experiments in AI provide a reason for caution, saidMildred Cho, a professor of pediatrics at Stanford's Center forBiomedical Ethics.

Systems developed in one hospital often flop when deployed in adifferent facility, Cho said. Software used in the care of millions ofAmericans has been shown to discriminate against minorities.And AI systems sometimes learn to make predictions based on factorsthat have less to do with disease than the brand of MRI machine used, the time a blood test is taken or whether a patient was visited by a chaplain. In one case, AI software incorrectlyconcluded that people with pneumonia were less likely to dieif they had asthma ― an error that could have led doctors todeprive asthma patients of the extra care they need.

"It's only a matter of time before something like this leads toa serious health problem," said Dr. Steven Nissen, chairman ofcardiology at the Cleveland Clinic.

Medical AI, which pulled in $1.6 billion in venture capitalfunding in the third quarter alone, is "nearly at the peak ofinflated expectations," concluded a July report from the researchcompany Gartner. "As the reality gets tested, there will likely be arough slide into the trough of disillusionment."

That reality check could come in the form of disappointingresults when AI products are ushered into the real world. EvenTopol, the author of "Deep Medicine: How Artificial IntelligenceCan Make Healthcare Human Again," acknowledges that many AIproducts are little more than hot air. "It's a mixed bag," hesaid.

(Lynne Shallcross/KHN Illustration; Getty Images)

Experts such as Dr. Bob Kocher, a partner at the venture capitalfirm Venrock, are blunter. "Most AI products have littleevidence to support them," Kocher said. Some risks won't becomeapparent until an AI system has been used by large numbers ofpatients. "We're going to keep discovering a whole bunch of risksand unintended consequences of using AI on medical data," Kochersaid.

None of the AI products sold in the U.S. have been tested inrandomized clinical trials, the strongest source of medicalevidence, Topol said. The first and only randomized trial of an AIsystem ― which found that colonoscopy with computer-aided diagnosisfound more small polyps than standard colonoscopy ― was publishedonline in October.

Few tech startups publish their research in peer-reviewedjournals, which allow other scientists to scrutinize their work,according to a Januaryarticle in the European Journal of Clinical Investigation. Such"stealth research" ― described only in press releases orpromotional events ― often overstates a company'saccomplishments.

And although software developers may boast about the accuracy oftheir AI devices, experts note that AI models are mostly tested oncomputers, not in hospitals or other medical facilities. Using unproven software"may make patients into unwitting guinea pigs," said Dr. Ron Li,medical informatics director for AI clinical integration atStanford Health Care.

AI systems that learn to recognize patterns in data are oftendescribed as "black boxes" because even their developers don't know how theyhave reached their conclusions. Given that AI is so new ― and manyof its risks unknown ― the field needs careful oversight, said Pilar Ossorio, a professor of law andbioethics at the University of Wisconsin-Madison.

Yet the majority of AI devices don't require FDA approval.

"None of the companies that I have invested in are covered bythe FDA regulations," Kocher said.

Legislation passed by Congress in 2016 ― and championed by thetech industry ― exempts many types of medicalsoftware from federal review, including certain fitness apps,electronic health records and tools that help doctors make medicaldecisions.

There's been little research on whether the 320,000 medical apps now in useactually improve health, according to a report on AI published Dec.17 by the National Academy of Medicine.

"Almost none of the [AI] stuff marketed to patients reallyworks," said Dr.Ezekiel Emanuel, professor of medical ethics and health policyin the Perelman School of Medicine at the University ofPennsylvania.

The FDA has long focused its attention on devices that pose thegreatest threat to patients. And consumer advocates acknowledgethat some devices ― such as ones that help people count their dailysteps ― need less scrutiny than ones that diagnose or treatdisease.

Some software developers don't bother to apply for FDA clearanceor authorization, even when legally required, according to a 2018 study in Annals of Internal Medicine.

Industry analysts say that AI developers have little interest inconducting expensive and time-consuming trials. "It's not the mainconcern of these firms to submit themselves to rigorous evaluationthat would be published in a peer-reviewed journal," said JoachimRoski, a principal at Booz Allen Hamilton, a technology consultingfirm, and co-author of the National Academy's report. "That's not how theU.S. economy works."

But Oren Etzioni, chief executive officer at the Allen Institutefor AI in Seattle, said AI developers have a financial incentive tomake sure their medical products are safe.

"If failing fast means a whole bunch of people will die, I don'tthink we want to fail fast," Etzioni said. "Nobody is going to behappy, including investors, if people die or are severelyhurt."

Relaxing standards at the FDA

The FDA has come under fire in recent years for allowing thesale of dangerous medical devices, which have been linked by theInternational Consortium ofInvestigative Journalists to 80,000 deaths and 1.7 million injuries over the pastdecade.

Many of these devices were cleared for use through acontroversial process called the 510(k) pathway, which allows companies to market"moderate-risk" products with no clinical testing as long asthey're deemed similar to existing devices.

In 2011, a committee of the National Academy of Medicine concluded the 510(k) process is sofundamentally flawed that the FDA should throw it out and startover.

Instead, the FDA is using the process to greenlight AIdevices.

The FDA, headquartered just outside Washington, D.C., has longfocused its attention on devices that pose the greatest threat topatients.

Of the 14 AI products authorized by the FDA in 2017 and 2018, 11were cleared through the 510(k) process, according to a November article inJAMA. None of these appear to have had new clinical testing,the study said. The FDA cleared an AI device designed to help diagnose liver and lung cancer in2018 based on its similarity to imaging software approved 20 years earlier. That software haditself been cleared because it was deemed "substantiallyequivalent" to productsmarketed before 1976.

AI products cleared by the FDA today are largely "locked," sothat their calculations and results will not change after theyenter the market, said Bakul Patel, director for digital health atthe FDA's Center for Devices and Radiological Health. The FDA hasnot yet authorized "unlocked" AI devices, whose results could varyfrom month to month in ways that developers cannot predict.

To deal with the flood of AI products, the FDA is testing aradically different approach to digital device regulation, focusingon evaluating companies, not products.

The FDA's pilot "pre-certification"program, launched in 2017, is designed to "reduce the time andcost of market entry for software developers," imposing the "leastburdensome" system possible. FDA officials say they want to keeppace with AI software developers, who update their products much more frequently than makers oftraditional devices, such as X-ray machines.

Scott Gottlieb said in 2017 while he was FDA commissioner thatgovernment regulators need to make sure its approach to innovativeproducts "is efficient and that it fosters, not impedes,innovation."

Under the plan, the FDA would pre-certify companies that"demonstrate a culture of quality and organizational excellence,"which would allow them to provide less upfront data about devices.

Pre-certified companies could then release devices with a"streamlined" review ― or no FDA review at all. Once products areon the market, companies will be responsible for monitoring their own products' safety and reporting back to theFDA. Nine companies have been selected for the pilot: Apple, FitBit,Samsung, Johnson & Johnson, Pear Therapeutics, Phosphorus,Roche, Tidepool and Verily Life Sciences.

High-risk products, such as software used in pacemakers, will still get a comprehensive FDAevaluation. "We definitely don't want patients to be hurt," saidPatel, who noted that devices cleared through pre-certification canbe recalled if needed. "There are a lot of guardrails still inplace."

But research shows that even low- andmoderate-risk devices have been recalled due to serious risksto patients, said Diana Zuckerman, president of the National Centerfor Health Research. "People could be harmed because somethingwasn't required to be proven accurate or safe before it is widelyused."

Johnson & Johnson, for example, has recalled hip implants and surgical mesh.

In a series of letters to the FDA, the American Medical Associationand others have questioned the wisdom of allowing companies tomonitor their own performance and product safety.

"The honor system is not a regulatory regime," said Dr. JesseEhrenfeld, who chairs the physician group's board of trustees.

In an October letter to the FDA, Sens. Elizabeth Warren (D-Mass.),Tina Smith (D-Minn.) and Patty Murray (D-Wash.) questioned theagency's ability to ensure company safety reports are "accurate,timely and based on all available information."

Scott Gottlieb said in 2017 while he was FDA commissioner thatgovernment regulators need to make sure its approach to innovativeproducts "is efficient and that it fosters, not impedes,innovation."

When good algorithms go bad

Some AI devices are more carefully tested than others.

An AI-powered screening tool for diabetic eye disease was studiedin 900 patients at 10 primary care offices before being approved in2018. The manufacturer, IDx Technologies, worked with the FDA foreight years to get the product right, said Dr. Michael Abramoff,the company's founder and executive chairman.

The test, sold as IDx-DR, screens patients for diabeticretinopathy, a leading cause of blindness, and refers high-riskpatients to eye specialists, who make a definitive diagnosis.

IDx-DR is the first "autonomous" AI product ― one that can makea screening decision without a doctor. The company is nowinstalling it in primary care clinics and grocery stores, where itcan be operated by employees with a high school diploma. Abramoff'scompany has taken the unusual step of buying liability insurance tocover any patient injuries.

Yet some AI-based innovations intended to improve care have hadthe opposite effect.

A Canadian company, for example, developed AI softwareto predict a person's risk of Alzheimer's based on their speech.Predictions were more accurate for some patients than others."Difficulty finding the right word may be due to unfamiliarity with English, rather than to cognitiveimpairment," said co-author Frank Rudzicz, an associate professorof computer science at the University of Toronto.

Doctors at New York's Mount Sinai Hospital hoped AI could help them usechest X-rays to predict which patients were at high risk ofpneumonia. Although the system made accurate predictions fromX-rays shot at Mount Sinai, the technology flopped when tested onimages taken at other hospitals. Eventually, researchers realizedthe computer had merely learned to tell the difference between thathospital's portablechest X-rays ― taken at a patient's bedside ― with those takenin the radiology department. Doctors tend to use portable chestX-rays for patients too sick to leave their room, so it's notsurprising that these patients had a greater risk of lunginfection.

DeepMind, a company owned by Google, has created an AI-basedmobile app that can predict which hospitalized patients willdevelop acute kidney failure up to 48 hours in advance. A blog post onthe DeepMind website described the system, used at a Londonhospital, as a "game changer." But the AI system also produced two falsealarms for every correct result, according to a July study inNature. That may explain why patients' kidney function didn'timprove, said Dr. Saurabh Jha, associate professor of radiology at theHospital of the University of Pennsylvania. Any benefit from earlydetection of serious kidney problems may have been diluted by ahigh rate of "overdiagnosis," in which the AI system flaggedborderline kidney issues that didn't need treatment, Jha said.Google had no comment in response to Jha's conclusions.

False positives can harm patients by prompting doctors to orderunnecessary tests or withhold recommended treatments, Jha said. Forexample, a doctor worried about a patient's kidneys might stopprescribing ibuprofen ― a generally safe pain reliever that poses asmall risk to kidney function ― in favor of an opioid, whichcarries a serious risk of addiction.

As these studies show, software with impressive results in acomputer lab can founder when tested in real time, Stanford's Chosaid. That's because diseases are more complex ― and the healthcare system far more dysfunctional ― than many computer scientistsanticipate.

Many AI developers cull electronichealth records because they hold huge amounts of detailed data,Cho said. But those developers often aren't aware that they'rebuilding atop a deeply broken system. Electronic health recordswere developed for billing, not patient care, and are filled with mistakes or missing data.

A KHNinvestigation published in March found sometimeslife-threatening errors in patients' medication lists, lab testsand allergies.

In view of the risks involved, doctors need to step in toprotect their patients' interests, said Dr. Vikas Saini, acardiologist and president of the nonprofit Lown Institute, whichadvocates for wider access to health care.

"While it is the job of entrepreneurs to think big and takerisks," Saini said, "it is the job of doctors to protect theirpatients."

Kaiser Health News isa nonprofit news service covering health issues. It is aneditorially independent program of the Kaiser Family Foundation,which is not affiliated with Kaiser Permanente.