Opinion: Why PHTI’s Recent Assessment of Digital Diabetes Management Tools is Inadequate

In late March, PHTI (Peterson Health Technology Institute) raised the bar for all of us working to digitize healthcare when they issued their first evidence assessment reporting the evaluation of digital diabetes management tools that support improved glycemic control in people with type 2 diabetes—or claiming to. 

The head-scratcher here is that despite claiming to assess a technology – and despite using an assessment framework purpose-built to evaluate digital health technologies – the report did not evaluate any technology products. Rather, it evaluated eight companies providing well-established care pathways virtually (i.e. healthcare services). 

Now, all these companies met the inclusion criteria of “connecting to a noncontinuous glucose monitor.” However, these tools – some connected to EHR and some not – are also pervasive as part of routine diabetes care provided in the clinic. As far back as 2006, 87% of adults with diabetes treated with insulin checked their blood glucose at least daily, and 63% of all adults with diabetes did the same. 

So, what exactly was being evaluated?

Honestly, I’m not sure. Except I’m certain it was not a digital technology.  

The decision to apply a product evaluation framework to digitally enabled healthcare services represents a larger issue plaguing the digitization of healthcare that must be addressed as a priority because quality patient care is at stake. 

Critical considerations for evidence evaluation in the digital era of healthcare:

1. We must distinguish between digital health products and digitally enabled care services

As Steve Steinhubl and Eric Topol said, digital medicine is on its way to being just plain medicine. And just like every other big industry that has gone before us, every aspect of healthcare will be digitized: From back office administration to direct patient care, quality improvement to clinical research, and population health to precision medicine. 

In the same way that we don’t apply the evaluation frameworks for the ROI of diagnostic assays, molecular products, or traditional medical devices to one another *or* to healthcare services, we must take a more nuanced and fit-for-purpose approach to the evaluation of digital health products as distinct from digitally enabled healthcare services. 

The PHTI assessment concluded that most of the services they evaluated are bad technologies, akin to saying an apple is a bad-tasting orange. However, this mischaracterization of digital products and services in healthcare is not a problem unique to PHTI’s assessment framework but an industry-wide issue. 

I cannot tell you how many times I’ve been asked about how the FDA regulates virtual-fist care; they don’t. The FDA regulates medical products, not healthcare. I’ve also watched digitally enabled care providers get passed around payer organizations, pushed towards reimbursement pathways for vendors with software-as-a-service models when they are, in fact, using providers, as defined in federal law, to provide healthcare as defined in federal regulations.

Until well-intended work such as PHTI’s rectifies this mischaracterization of digitally enabled services, its output will only add to confusion and friction in the progress we’ve seen from digitally enabled patient care.

2. Value and the patient must be our North Star

It is not by accident that the digitization of healthcare – characterized by high-resolution, liquid data and rapidly advancing computing capabilities – is being accompanied by a resurgence of value-based care initiatives. We need flows of high-quality data to support the evaluation of care outcomes and insights into the processes of care that drive those outcomes (and those that do not) to make the business case for assuming risk. 

To successfully advance value-based care in an increasingly digitized industry, we must develop appropriate evaluation frameworks for digital health products and digitally enabled healthcare services. 

This requires that we differentiate between the evaluation of whether a digital health product or a digital component of a digitally enabled care service is fit for purpose – an evaluation that should include evidence that supports any performance claim it may make as well as consideration of privacy, security, accessibility, usability, and equity – and whether this solution is delivering value. 

This also requires that we define value more holistically than in the recent PHTI assessment, in which short-term clinical effectiveness and cost-effectiveness were the primary drivers. 

Let’s start with the patient. While PHTI did engage patient users of the digitally enabled services in their report, their perspectives were not systematically included in the value or throughout the evaluation process. This feels rather performative and consistent with ongoing critiques of product evaluation frameworks issued by the Institute of Clinical and Economic Research (ICER), PHTI’s partners in developing the ICER-PHTI Assessment Framework for Digital Health Technologies.

We should also include equity and access as essential measures of value alongside effectiveness and cost-effectiveness. , as our team did when we partnered with the Veteran’s Health Administration to develop a value-based innovation framework that applies to all digital health innovations, whether products or digitally enabled services. 

I was alarmed by PHTI’s conclusion that there was no evidence that diabetes management programs evaluated advanced health equity. Their study design—particularly the selection of companies whose business models are contingent on serving patients covered by self-insured employers—rendered a data set that could never be used to answer the question, “Is it used in diverse settings and by groups that need it most?” 

Their evidence assessment also failed to recognize that a comparator arm of ‘usual care’ absent considerations of access to this care is a flawed framework. For example, Omada Health –  one of the eight companies evaluated in the PHTI report – provides diabetes management solutions to tens of thousands of rural Alaskans who otherwise would not have access to any support to manage their condition. To every one of those patients accessing care previously out of reach, achieving statistically equivalent clinical outcomes is not a failure of the digitally enabled program. It is an enormous success. 

3. Altitude is everything

The PHTI assessment reports ROI at the ‘category level’, breaking digital diabetes management solutions (ahem, services) into three different categories: Remote patient monitoring, behavioral and lifestyle modification, and nutritional ketosis. 

Setting a discussion of these categories aside, if we want to evaluate ROI at the ‘category level’ – and we should – we need a more inclusive approach to solutions in scope.

PHTI evaluated eight different start-ups providing three different categories of digitally enabled diabetes management solutions (not technologies) and conducted a systematic review of the scientific literature to draw ‘category level’ conclusions about the digitization of a $23Bn (2022) market. And they did so without contemplating all of the other digital solution providers in this space, the patient perspective, or equity and access to care. 

In addition, this assessment—summarized by a dashboard of big red dots and warnings not to adopt—uncovered critically important findings that digitally enabled behavioral and lifestyle modifications can have outsized positive impacts on particular subpopulations, particularly individuals with high HbA1c – a measure from a blood test of how much sugar is in your blood and used to diagnose diabetes – starting insulin for the first time. But it buried them in a roll-up of the findings to the ‘category level.’ 

These issues highlight the risks of extrapolating and translating findings – either from the scientific literature to a single solution or a small handful of solutions to a ‘category level’ in our complex, $4.3Tn industry – without sufficient contextual information. 

In the digital era of healthcare, we have the ability to interrogate comprehensive datasets when we want to draw industry-wide ‘category level’ conclusions. We also have the capacity to conduct targeted subanalysis to identify the greatest opportunities to benefit every one of the patients our industry exists to serve. Methodological mashups that jump altitudes between ‘category level’ and solution studies and report population-level conclusions without championing sub-population variance are simply not appropriate, given the proliferation of data available. 

New eras are characterized by reinvention

PHTI is truly a pioneer in digital health, holding us accountable for delivering value in this digital era of healthcare. Their first assessment has started conversations across the field about our responsibility to evaluate the return on the substantial investment required to digitize our enormous and complex domestic healthcare industry. That is an impressive achievement.  

Their report’s intent to deliver data supporting an evidence-based approach to investing in healthcare innovation in our resource-constrained environment is spot on. But first, we must establish a shared and fundamental understanding of the different components of digital health.

And this is a need that stretches far beyond the impact of the PHTIs assessment framework.

When most people in our industry cannot differentiate between traditional and generative AI, how can we appropriately evaluate these models and maintain their security?

When accreditation bodies are actively building certifications for virtual care, how will they recognize that it’s all just healthcare in the digital era and reflect these updates in their certification programs for traditional care?

When the failure of a single digitally enabled provider results in cries of the ‘collapse of telehealth’ but the closure of dozens of rural hospitals is ignored, how on earth do we plan to care for every person our industry exists to serve?

Definitions are important. A common unifying language is important. And fit-for-purpose evidence-based evaluation frameworks are important. 

As Taylor Swift taught all of us, a new era is characterized by successful reinvention. As we enter the digital era of healthcare at scale, we must redefine how we care for people… and how we evaluate that care. And this begins with a deep understanding of the new products, care pathways, and the difference between the two.

Photo credit: Venimo, Getty Images