Key Points

QuestionÌý Is a small mean treatment effect in the Kansas City Cardiomyopathy Questionnaire (KCCQ) score (<5 points) clinically important in patients with heart failure?

FindingsÌý In this patient-level meta-analysis of 11 randomized clinical trials including 9977 patients, a small difference in the mean KCCQ score between 2 treatment groups was associated with substantial differences in the proportions of patients experiencing clinically important changes in their health status.

MeaningÌý Results of this study suggest that while statistically significant mean treatment differences in the KCCQ may be small, clinical significance should be based on the distributions of patients having clinically important benefits from treatment.

ImportanceÌý The Kansas City Cardiomyopathy Questionnaire (KCCQ) is a commonly used outcome in heart failure trials. While comparing means between treatment groups improves statistical power, mean treatment effects do not necessarily reflect the clinical benefit experienced by individual patients.

ObjectiveÌý To evaluate the association between mean KCCQ treatment effects and the proportions of patients experiencing clinically important improvements across a range of clinical trials and heart failure etiologies.

Design, Setting, and ParticipantsÌý A patient-level analysis of 11 randomized clinical trials, including 9977 patients, was performed to examine the association between mean treatment effects and the KCCQ Overall Summary Score (OSS) and the absolute differences in the proportions of patients experiencing clinically important (≥5 points) and moderate to large (≥10 points) improvements. There was no target date range, and included studies were those for which patient-level data were available. Validation was performed in 7 additional trials. The data were analyzed between July 1 and September 15, 2023.

Main Outcomes and MeasuresÌý Proportion of patients experiencing an improvement of 5 or more and 10 or more points in their KCCQ score (with each domain transformed to a range of 0 to 100 points, where higher scores represent better health status).

ResultsÌý Group mean KCCQ-OSS differences were strongly correlated with absolute differences in clinically important changes (Spearman correlations 0.76-0.92). For example, a mean KCCQ-OSS treatment effect of 2.5 points (half of a minimally important difference for an individual patient) was associated with an absolute difference of 6.0% (95% prediction interval [PI], 4.0%-8.1%) in the proportion of patients improving 5 or more points and 5.0% (95% PI, 3.1%-7.0%) in the proportion improving 10 or more points, corresponding to a number needed to treat of 17 (95% PI, 12-25) and 20 (95% PI, 14-33), respectively.

Conclusions and RelevanceÌý Inferences about clinical impacts based on population-level mean treatment effects may be misleading, since even small between-group differences may reflect clinically important treatment benefits for individual patients. Results of this study suggest that clinical trials should explicitly describe the distributions of KCCQ change at the patient level within treatment groups to support the clinical interpretation of their results.

Clinical trials increasingly use patient-reported outcomes (PROs) as key end points.^1,2 A challenge with PROs, however, is that their scales are not intuitively interpretable. To address this challenge, qualitative and quantitative steps are required to define what changes in a PRO represent clinically important improvements in patients’ health status. In heart failure, the Kansas City Cardiomyopathy Questionnaire (KCCQ) is a psychometrically robust, disease-specific PRO that has been qualified by the US Food and Drug Administration (FDA) for drug and device evaluations.^3-5 Despite its acceptance by regulatory agencies, discerning the clinical importance of population-level mean treatment effects remains a challenge.⁶

Numerous studies have consistently shown that a 5-point change in the KCCQ is a small but clinically important change from both patient and clinician perspectives, although some have suggested even smaller changes as clinically important.^7-12 Moreover, a 5-point change in the KCCQ has been shown to be associated with cardiovascular death and hospitalization across a range of heart failure etiologies, reinforcing its clinical relevance.^7,8,13 However, these clinical thresholds apply to changes within individual patients rather than mean treatment differences between groups of patients. Failure to appreciate this distinction has led to conclusions that small mean treatment effects are not clinically important.^14-16

Clarifying the distinction between mean treatment effects and the underlying clinical impacts of treatment in individual patients’ health status, which is best described by examining the distributions of clinical change,¹⁷ represents an important gap in knowledge. To explore the association between mean treatment effects and the differences in the proportions of patients experiencing clinically important changes, we conducted a patient-level meta-analysis of a variety of randomized trials to examine the association between mean KCCQ treatment effects and the absolute differences in the proportion of patients experiencing clinically meaningful improvement. By establishing this association, we sought to highlight the importance of presenting both the mean treatment effect and the distribution of clinically important within-patient changes in clinical trials to support the interpretation of their findings.

We included 11 multicenter randomized parallel-arm trials^18-28 using the KCCQ for which we had access to patient-level data. There was no target date range, and included studies were those for which patient-level data were available. The data were analyzed between July 1 and September 15, 2023. Because patients experience the symptoms and impacts of their heart failure but not their anatomy or physiology, we included a variety of studies representing different types of heart failure (heart failure with reduced ejection fraction [HFrEF], heart failure with preserved ejection fraction [HFpEF], valvular heart disease, and obstructive hypertrophic cardiomyopathy) and treatments (drugs and devices) to support the generalizability of the analyses. As all data were deidentified before analyses, the Saint Luke’s Hospital (Kansas City, Missouri) Institutional Review Board granted a waiver of informed consent, although each patient participating in each trial signed informed consent before randomization.

The KCCQ is a 23-item self-administered disease-specific PRO that quantifies patients’ symptoms, physical and social function, and quality of life due to heart failure over the past 2 weeks. Each domain is transformed to a range of 0 to 100 points, where higher scores represent better health status. The Clinical Summary Score (CSS) combines the Total Symptom (symptom burden and frequency) and Physical Limitation scores to mirror the New York Heart Association class from the patient’s perspective. The Overall Summary Score (OSS) further includes the Social Limitation and Quality of Life domains to capture broader impacts of heart failure in patients’ health status. A small but clinically important change for an individual patient is 5 points for either improvement or deterioration, whereas a moderate to large change is 10 or more points, thresholds that have been observed in multiple heart failure etiologies, including HFrEF, HFpEF, valvular heart disease, and hypertrophic cardiomyopathy.^10,12,29,30 We focused our analyses on the OSS, which is the most comprehensive measure of impacts of heart failure in patients’ health status. However, we also provide results for the KCCQ-CSS in Supplement 1 because it is the most comprehensive description of patients’ health status qualified by the FDA Center for Drug Evaluation and Research.⁴

Our primary objective was to describe the association between mean KCCQ treatment effects and outcomes in the proportion of patients experiencing clinically important changes. To accomplish this, we conducted patient-level meta-regression analyses of 11 trials using KCCQ-OSS response definitions of improvements of 5 or more points and 10 or more points, reflecting any clinically important patient improvement and a moderate or larger improvement, respectively. We also conducted a similar analysis examining the association of mean treatment effects with differences in the proportion of patients deteriorating by at least 5 and 10 points. Among the 10 trials that contained multiple follow-up KCCQ assessments, these assessments were considered as repeated measures within trials, with each assessment serving as a data point at which the effect of treatment could be estimated, resulting in a total of 32 follow-up assessments across the 11 trials.^18-28 No formal comparisons were performed for which statistical significance would be relevant. We provided the point estimates and 95% prediction intervals (PIs) to describe the association between population-level mean treatment effects and the underlying distribution of previously well-established categories of patient-level clinically important change.

We first estimated the treatment difference in mean KCCQ-OSS scores and the absolute difference in response rates for each trial and each follow-up assessment. Each trial was analyzed separately, with generalized estimating equations using robust covariance estimation with an autoregressive working correlation structure to account for deviations from the theoretical error distribution and intrastudy correlations among repeated follow-up assessments. The models included adjustment for baseline KCCQ score to improve precision, using restricted cubic splines to account for potential nonlinearity, and treatment-by-time and baseline-by-time interactions.

All trials contained missing KCCQ scores, either due to death or missed assessments. The frequency of missing data for each trial is summarized in eTable 1 in Supplement 1. We assumed missing scores for survivors were missing at random, conditional on treatment group and observed scores at other time points, for which we used multiple imputation to estimate missing scores. We imputed data separately for each of the 11 trials. For each trial, we generated 80 randomly imputed datasets using predictive mean matching, regressing each KCCQ assessment on all other available KCCQ assessments and treatment assignment. We did not impute missing scores due to death; all treatment effect estimates should be interpreted as conditional on survival to that time point. Following others’ suggestions for handling missing data in patient-level meta-analyses,^31,32 we used the imputed datasets for estimating and pooling the treatment effects and then performed the meta-analysis on the pooled estimates (rather than performing 80 meta-analyses and then pooling).

We then fit a mixed-effects meta-regression model to the pooled treatment effects and corresponding covariance matrices obtained from each trial using the R package metafor, regressing the absolute differences in response rates on the differences in mean scores.³³ We used restricted cubic splines to allow for potential nonlinearity. We also calculated the overall strength of correlation using weighted Spearman correlation, with weights corresponding to the inverse of the variances of the estimated absolute differences in the response rates. Finally, we used the meta-regression model to translate hypothetical differences in mean KCCQ scores ranging from 1 to 5 points into a predicted number needed to treat (NNT) by dividing 1 by the predicted absolute difference in the proportion of responders.³⁴ Uncertainty in estimates is described with 95% PIs, which reflect the expected range of possible values for a future clinical trial (as opposed to CIs, which show the uncertainty in the average over all trials). Finally, to assess whether these findings varied depending on baseline health status, we augmented the meta-regression models with an interaction term by trial mean baseline score. All analyses were performed with SAS version 9.4 (SAS Institute Inc) and R version 4.3.1 (R Core Team).

To validate the observed associations, we identified studies not included in the meta-analysis that reported both the mean KCCQ treatment effects and the distribution of patient-level response. Because patient-level data were not available for these studies, risk differences were based on the published summary statistics for KCCQ responder rates, and 95% CIs were calculated using the standard formula for a difference in proportions between 2 independent samples.³⁵

The 11 trials,^18-28 including 9977 patients (eFigure 1 in Supplement 1), had a range of baseline mean (SD) KCCQ-OSSs from 31.8 (18.9) to 67.6 (21.3). The number of included patients per trial varied from 62 to 2129 (Table). Follow-up assessments were conducted at different time intervals ranging from 1 week to 24 months (eTable 1 in Supplement 1).

Mean treatment KCCQ-OSS effects were highly correlated with the proportions of patients improving by clinically significant amounts (Spearman correlations 0.76-0.92) (eTable 2 in Supplement 1). The association between mean treatment KCCQ-OSS effects and the differences in the proportions of patients reporting clinically important improvement (≥5 points) (Figure 1A), moderate to very large improvement (≥10 points) (Figure 1B), clinically important deterioration (≤5 points) (Figure 1C), or moderate to very large deterioration (≤10 points) (Figure 1D) demonstrated substantial effects in patient-level changes, even with small mean treatment effects. For example, with a mean treatment effect of 2.5 points between groups (ie, half of what is considered a meaningful within-patient change), the absolute difference in the proportion of patients who improved by 5 or more points was 6.0% (95% prediction interval [PI], 4.0%-8.1%), and 5.0% (95% PI, 3.1%-7.0%) for an improvement of 10 or more points. These absolute differences correspond to NNTs of 17 (95% PI, 12-25) and 20 (95% PI, 14-33), respectively (Figure 2; eTable 2 in Supplement 1). A similar association was observed for the KCCQ-CSS (eFigure 2 in Supplement 1). A mean treatment effect of 2.5 points on the KCCQ-CSS corresponded to a 6.7% (95% PI, 4.6%-8.7%) absolute difference in the proportion of patients who improved by 5 or more points and a 5.5% difference (95% PI, 3.5%-7.5%) for improvement by 10 or more points, corresponding to NNTs of 15 and 18, respectively (eTable 2 in Supplement 1). A similar association between mean treatment KCCQ effects and the differences in the proportions of patients experiencing clinically important deterioration in their OSS (Figure 1C and D) and CSS (eFigure 2C and D in Supplement 1) was observed.

To investigate whether these associations varied by the severity of participants’ baseline health status impairment, interactions with trial mean baseline KCCQ scores were tested. The P values for all were >0.1 (eTables 2 and 3 in Supplement 1), indicating insufficient evidence that overall population severity affects the observed associations.

In 7 additional clinical trials with publishing mean KCCQ data and responder analyses (eTables 4 and 5 in Supplement 1), the estimated mean treatment effects and differences in response rates were similar to those included in the meta-analyses (Figure 3; eFigure 3 in Supplement 1). These results support the robustness of the primary analyses.

As PROs are increasingly used as outcomes in clinical trials, their accurate interpretation requires translating them into a clinical framework so that patients, clinicians, payers, and regulators can better understand the clinical benefits of treatment. Using 11 clinical trials, representing a broad range of populations with heart failure, etiologies, and treatments, a strong correlation between mean treatment effects and the proportion of patients experiencing clinically important health status changes was found. These correlations highlight the problem with applying within-patient thresholds of change (eg, ≥5 points) to mean treatment effects for a population-level mean treatment difference. Even a 2.5-point mean difference in KCCQ-OSS between treatment groups, half of what is considered a clinically important patient-level change, translates to an expected NNT of 17 for 1 treated patient to have at least a small but clinically important benefit from therapy and an NNT of 20 for 1 patient to experience a moderate or larger health status benefit from treatment. These findings highlight that applying a patient-level threshold of clinical importance to mean population-level treatment effects leads to an erroneous interpretation of there being no important treatment benefit despite significantly more treated patients experiencing a clinically important improvement in their health status. Similar patterns were also observed when treatment results in worse health status outcomes. To support the clinical interpretation of clinical trials, both the mean KCCQ treatment effects and responder analyses, which describe the distributions of clinical change across treatment arms, should be reported.

This association was observed not only across various types, severities, and etiologies of heart failure but also extends to the Eplerenone Post–Acute Myocardial Infarction Heart Failure Efficacy and Survival Study (EPHESUS), which included a subset of patients with stage B rather than stage C heart failure. Conversely, the inclusion of studies like Cardiovascular Outcomes Assessment of the MitraClip Percutaneous Therapy for Heart Failure Patients With Functional Mitral Regurgitation (COAPT) and the Placement of Aortic Transcatheter Valves-1B (PARTNER-1B), where baseline health status was particularly poor, further demonstrated the consistency of this association across different causes and severities of heart failure. Including these studies likely increased the power to detect any differences in this association based on baseline health status, yet no such differences were observed.

The findings of this study underscore the need to provide a better clinical framework for reporting the health status outcomes of heart failure trials. Historically, clinical trials, such as the Treatment of Preserved Cardiac Function Heart Failure With an Aldosterone Antagonist (TOPCAT), Effects of Ivabradine on Cardiovascular Events in Patients With Moderate to Severe Chronic Heart Failure and Left Ventricular Systolic Dysfunction (SHIFT), and Multicenter Automatic Defibrillator Implantation Trial With Cardiac Resynchronization Therapy (MADIT-CRT) trials, only reported the mean differences between treatment arms in their primary publications (mean [SD] differences in the KCCQ-OSS of 1.36 [0.44], 2.4 [1.47], and 1.5 [0.95], respectively).^36-38 While these mean treatment effects were statistically significant, understanding their clinical importance is challenging. In contrast, more recent trials, such as the Dapagliflozin and Prevention of Adverse Outcomes in Heart Failure (DAPA-HF) and Dapagliflozin Effects on Biomarkers, Symptoms and Functional Status in Patients With HF With Reduced Ejection Fraction (DEFINE-HF) trials augmented their reports of mean KCCQ treatment effects with responder analyses showing the differences in the proportions of patients experiencing large health status improvements with treatment.^18,39

An additional advantage of comparing the proportions of patients who attain a certain magnitude of benefit is the ability to describe the NNT for how many patients would need to be treated for 1 to attain a specific health status benefit. While NNTs are most often used for clinical events, defining the NNTs for clinically meaningful health status improvements may be equally or more important for helping patients understand the likely benefits of treatment.⁴⁰ Multiple studies have clarified that patients often prioritize health status (symptoms, function, and quality of life) outcomes as much or more than clinical events.^41,42 Unlike some clinical events, such as hospitalizations for heart failure, where there is great variability in admitting patients across hospitals and physicians,^43-45 the reproducibility of the KCCQ⁴⁶ may be even more generalizable. Moreover, the NNTs for health status treatment benefits with lower mean treatment effects (eg, an NNT of 17 for a 2.5-point difference between groups for change in the KCCQ-OSS), is comparable with the NNTs of other guideline-recommended cardiovascular treatments. For instance, the Angiotensin-Neprilysin Inhibition vs Enalapril in Heart Failure (PARADIGM-HF) trial found an NNT for the composite outcome of cardiovascular death or heart failure hospitalization to be 25 for 3 years of treatment with sacubitril/valsartan compared with enalapril.⁴⁷ We believe that being able to communicate the health status benefits of a treatment to patients may be more informative than the complex composite end points that clinicians currently consider when recommending treatment.

The underlying premise of our study hinges on 2 major assumptions. First, we believe that the etiology of patients’ heart failure is often opaque to patients as they don’t feel their ejection fraction, their valvular dysfunction, or their dynamic left ventricular outflow gradient. Rather, what they experience are symptoms of dyspnea or fatigue and the impacts these have in their function (physical and social) and quality of life. The consistency of the observed associations across trials of different heart failure etiologies supports this assumption. Second, we conducted our analysis on all trials for which we had individual patient-level data but assumed that these associations between mean population differences and the underlying distributions of patient experience would be consistent. Supporting this assumption was the similar trial-level associations in studies not included in our meta-analyses.

Despite showing an association between mean population-level treatment effects and the clinical benefits for individual patients, the described associations should not be used to estimate the effects of treatment on the distribution of patients’ improvement. Our intent was to highlight that small mean differences, below standard within-patient thresholds of clinically important change, can represent important patient-level treatment benefits. While studies are powered on estimated mean treatment differences to maximize power and minimize cost, the distributions of response across clinical categories of no (<5 points), small clinically important changes (≥5 to ≤10 points), moderate to large (≥10 to <15 or 20 points), and very large (≥15 or 20 points) changes should be reported to help interpret the mean differences between groups.

The findings from this study should be interpreted in the context of the following potential limitations. While we were able to include 11 diverse clinical trials, there may be other manifestations of heart failure, or differences in treatment effects, that might deviate from these observed associations. Nevertheless, the broad range of heart failure etiologies and interventions, along with the comparability of findings from studies not included in the meta-analyses, supports the generalizability of these associations. Second, these findings focused on the KCCQ, for which extensive preexisting evidence supports the clinical significance of different changes in scores. Other PROs may have different associations and need to be independently tested. Finally, while the figures show the regression line averaging the associations across all studies, there was variability within and across studies, further supporting the recommendation that future studies report the distribution of clinically important changes in each treatment group.

Because of the growing recognition that patients highly value health status outcomes, clinical trials are increasingly collecting and reporting the consequences of treatment in patients’ symptoms, function, and quality of life. It is critically important, however, to avoid misinterpreting population mean differences between groups when judging the clinical benefits of therapy. Rather, the proportion of patients in each treatment group meeting clinically meaningful thresholds of health status change should be routinely reported to assist patients, clinicians, and regulatory authorities in valuing the potential benefits of treatment on patients’ health status.

Accepted for Publication: October 11, 2024.

Published Online: November 15, 2024. doi:10.1001/jamacardio.2024.4470

Corresponding Author: John Spertus, MD, MPH, Saint Luke’s Mid America Heart Institute, 4401 Wornall Rd, Kansas City, MO 64111 (spertusj@umkc.edu).

Author Contributions: Dr Spertus and Mr Jones had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Abdel Jawad, Spertus, Sherrod.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Abdel Jawad, Spertus, Sherrod, Ikemura.

Critical review of the manuscript for important intellectual content: Abdel Jawad, Jones, Arnold, Cohen, Sherrod, Khan, Chan.

Statistical analysis: Jones, Sherrod.

Administrative, technical, or material support: Abdel Jawad, Spertus, Arnold, Sherrod, Khan, Ikemura.

Supervision: Spertus, Arnold, Cohen.

Conflict of Interest Disclosures: Dr Abdel Jawad reported being supported by the National Heart, Lung, and Blood Institute (NHLBI) under award number T32H110837. Dr Cohen reported receiving grants from Edwards Lifesciences, personal fees from Edwards Lifesciences, grants from Abbott, personal fees from Abbott, grants from Medtronic, personal fees from Medtronic, grants from Boston Scientific, and personal fees from Boston Scientific outside the submitted work. Dr Khan reported receiving grants from NHLBI award number T32HL110837 during the conduct of the study. Dr Ikemura reported receiving grants from Bristol-Myers Squibb (BMS) during the conduct of the study. Dr Chan reported receiving grants from NHLBI outside the submitted work. Dr Spertus reported receiving grants from BMS, personal fees from BMS, and personal fees from Cytokinetics during the conduct of the study; in addition, Dr Spertus reported having a patent for copyright to the Kansas City Cardiomyopathy Questionnaire with royalties paid; and Dr Spertus discloses providing consultative services on patient-reported outcomes and evidence evaluation to Alnylam, AstraZeneca, Bayer, Janssen, BMS, Terumo, Cytokinetics, and Imbria. He holds research grants from the National Institutes of Health, the Patient-Centered Outcomes Research Institute, the American College of Cardiology Foundation, BMS, Cytokinetics, Imbria, and Janssen. He owns the copyright to the Seattle Angina Questionnaire, Kansas City Cardiomyopathy Questionnaire, and Peripheral Artery Questionnaire and serves on the Board of Directors for Blue Cross Blue Shield of Kansas City. No other disclosures were reported.

Funding/Support: Drs Abdel Jawad, Sherrod, and Khan are currently supported by the National Heart, Lung, and Blood Institute under Award Number T32H110837. Dr Chan is supported by the National Heart, Lung, and Blood Institute of Health under Award Number R01HL160734.

Role of the Funder/Sponsor: The funding organizations had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The contents of this project are solely the responsibility of the authors and do not necessarily represent the official views of the National Center for Advancing Translational Sciences, the National Heart, Lung, and Blood Institute, the National Institutes of Health, or the Department of Health and Human Services.

Meeting Presentation: This paper was presented at the AHA Scientific Sessions 2024; November 15, 2024; Chicago, Illinois.

Data Sharing Statement: See Supplement 2.