In early 2024, the US Food and Drug Administration (FDA) authorized an artificial intelligence/machine learning (AI/ML)鈥揺nabled medical device for dementia prognosis.¹ This device was granted a de novo request, signifying the first FDA authorization for prognostic assessment software of mild cognitive impairment. It is indicated to aid in the clinical assessment of how likely patients 鈥渨ho are diagnosed with amnestic mild cognitive impairment are to progress to clinical Alzheimer鈥檚 disease-dementia within 5 years of assessment.鈥�¹

During the authorization process, the FDA identified risks to the health of patients with device failure or incorrect analysis. They considered falsely high outputs resulting in inappropriate patient treatment and nondiagnosis of other conditions and falsely low outputs causing delayed diagnosis and treatment. To mitigate these risks, the FDA incorporated a special control that applies in the postauthorization setting: the manufacturer must demonstrate that the device performs as intended based on data from postmarket surveillance. More specifically, such data must demonstrate the accuracy, precision, and reproducibility of the device for predicting progression of patients with mild cognitive impairment to more severe forms of cognitive impairment or dementia. The mechanism for this special control is that the device has received authorization, but failure to comply would be considered adulteration or misbranding of the device. This special control will also apply to future substantially equivalent devices that receive FDA clearance via the 510(k) pathway, although the FDA may determine that this surveillance is not required for a given device based on the totality of the information provided for premarket review.

This strategy is part of a broader move by the FDA to encompass a life cycle approach to regulation. A recent publication from the FDA² describes 4 areas in AI in which FDA centers are collaborating, including supporting research related to the evaluation and monitoring of AI performance. It explains that demonstration projects will be used to ensure adherence to standards and maintain performance and reliability throughout the life cycle. It also separately describes the FDA鈥檚 intention to advance the development of regulatory approaches that support innovation. A focus on postmarket surveillance may facilitate innovation through allowing regulatory requirements to be met once a device can be more easily deployed and used. However, the current publicly available information for this device does not indicate the extent of the reduction, if any, of the premarket testing requirements.

While the conduct of postauthorization testing has not been a traditional regulatory step for AI/ML-enabled medical devices, it may increasingly be used to allay concerns regarding their accuracy and reproducibility in real-world settings. In the pharmaceutical setting, postmarket studies commonly identify adverse events. However, an analysis³ of the FDA accelerated approval program showed that only 55% of hematology and oncology drugs had fulfilled their postmarketing requirements and verified their clinical benefit. To ensure patient access to safe and effective medical devices, it is crucial that the FDA enforce robust and timely completion of postmarket surveillance requirements for AI/ML-enabled medical devices.

For the dementia prognosis device, the FDA specifies timelines for patient enrollment and periodic reporting of postmarket testing, including that the first participant is enrolled within 6 months and all participants within 24 months of the postmarket study protocol approval.¹ The FDA also addresses the concern as to whether this device will have generalizability across the intended patient population. They state that the postmarket testing must be conducted in patients of diverse race, socioeconomic status, and levels of education; who are from community-based care settings; and who have comorbidities that may impact the effectiveness of the device. They require at least 50% of patients from community-based health care settings and at least 50% of patients who are not White.

The FDA authorization of the first prognostic assessment software of mild cognitive impairment raises not only regulatory questions but also new challenges in clinical practice. A prognostic device differs from the many AI/ML-enabled diagnostic devices in that it is predicting an outcome that may only occur years after the prediction is made. It may therefore not be possible to verify its accuracy in real time. In addition, the FDA has established this type of device as combining information from multiple clinical data sources. While the multimodal nature of an AI/ML-enabled device can better reflect a health care professional鈥檚 approach to care,⁴ its complexity can decrease explainability and verifiability. This initial device uses brain MRI scans, demographic information, and cognitive assessment scores.

It will be imperative for clinicians and patients to be cautious about how much weight to place on a prediction while the accuracy of these devices is confirmed. Indeed, the FDA states that such devices are intended for adjunctive use and not as stand-alone diagnostic or prognostic tools. At the same time, both patients and clinicians may be attracted to, and place emphasis on, an algorithm that outputs a simple yet powerful metric like a percentage.

The fast-changing nature of Alzheimer disease treatment presents an additional challenge for the ongoing relevance of such a device. The treatment criteria for antiamyloid therapies have required evidence of amyloid pathology through amyloid positron emission tomography or cerebrospinal fluid profiling⁵; neither are inputs for this device, which was submitted to the FDA in September 2022. A question also arises as to how specifically this device predicts Alzheimer disease compared with dementia more broadly, especially given the inherent uncertainty in clinically diagnosing Alzheimer disease.⁶ Interestingly, the longitudinal data from the trials of antiamyloid therapies could provide a data source to further assess such a device. These data may even facilitate showing that a device enables better patient selection for these medications, although the risks and benefits compared to the current standard of care would need to be carefully considered before clinical use.

The implementation of prognostic assessment software may span different subspecialties within neurology in the upcoming years. The FDA has identified this device type as nonspecific for the type of dementia, enabling devices for other cognitive conditions to follow this first device via the 510(k) pathway. The FDA also allows flexibility for the prognostication window of future devices by stating the device type must produce an output indicative of the likelihood of progression 鈥渨ithin a specified period of time鈥�¹ but not providing any limits. Hypothetical uses for a prognostic device in other areas of neurology include determining a strategy for disease-modifying therapy in multiple sclerosis or establishing the risk of hemorrhagic transformation from thrombolytic treatment in acute ischemic stroke. The mathematical nature of the algorithms underlying these devices means that their outputs may be a probability rather than a binary result. A key task for neurologists will be to explain the implications of these probabilities to their patients; they will then need to assist patients in translating these numbers into clinical decisions with only a small number of discrete options.

To date, the FDA has authorized approximately 1000 AI/ML-enabled medical devices.⁷ It is likely that many more AI/ML-enabled prognostic devices will soon follow. The FDA鈥檚 life cycle regulation, including through enforced and robust postmarket surveillance and testing, will be critical in continuing to balance the fostering of innovation with patient access to safe and effective AI/ML-enabled medical devices.

Corresponding Author: James M. Hillis, MBBS, DPhil, Department of Neurology, Massachusetts General Hospital, 55 Fruit St, Boston, MA 02114 (james.hillis@mgh.harvard.edu).

Published Online: November 18, 2024. doi:10.1001/jamaneurol.2024.3835

Conflict of Interest Disclosures: Dr. Hillis reported institutional grants from Annalise.ai, Cerebriu, GE Healthcare, Heuron and Viz.ai; in addition, Dr Hillis had a patent for computed tomography angiography large vessel occlusion model (US11751832B2) issued. Dr Cliff reported grants from Arnold Ventures. Dr Vokinger reported grants from Swiss National Science Foundation. No other disclosures were reported.