TherAppX Review: or how do we manage to go from 350,000+ health apps to 1,000+ in our app library?

Overcoming Digital Health Tools reviewing challenges

As recently highlighted by the European mHealth Hub, who published a knowledge tool on the different health apps assessment frameworks in Europe, it is very challenging to operationalize DHT assessments. This report also highlights the challenges of keeping up with the rapid pace to which new DHT come in the market and the importance of timely assessments:

“During the COVID-19 outbreak, multiple apps have been developed in a very short period of time, and given the situation, it may not be possible for governments or other agencies to conduct an in-depth assessment. This constitutes clearly a challenge for health apps quality in the context of public health emergencies.”

As DHT come and go, or are frequently updated, what’s the perfect balance between thorough evidence-based recommendations and what’s achievable to overcome the challenges of continuous evaluation?

At TherAppX in the past two years, we’ve invested in technologies and processes that allow us to reach that balance by 1) highlighting 58+ data points on an app in 24-48 hours; 2) reviewing updates on thousands of DHT each day; 3) screening for new DHT that come in the market every week.

Here’s how we achieved that.

Step 1 - Discoverability and prioritization

To be informed of new DHT that come in the market and correspond to specific patient needs, we programmed an algorithm. Every week, it automatically sends us lists of newly launched DHTs. We also use this algorithm to request lists of DHT that fit specific criteria when we need them.

This allows us to cover both the needs of identifying newly launched DHT in the market and perform a quick market screening or systematic reviews as we call them. Using that algorithm, it becomes feasible to find all the DHT that empower patients living with diabetes through self-management as an example.

This algorithm allows us to skim rapidly through all DHT by using specific criteria (e.g. excluding those that had no update in the past six months, have relatively low ratings, are not available in the official languages of the country of our users, etc.). We can even skim DHT by analyzing descriptions provided by their manufacturers. When a given app is considered relevant through those multiple digital layers of filters, an app analyst adds it to our human-led reviewing funnel.

There is also another way we find and prioritize DHT to review. Paying customers of TherAppX Core have the power to steer our reviewing funnel in the direction they find the most useful by requesting reviews (based on their usage plan). We complete those requests in 24 to 48 hours if the tool is publically available or ask the developer to give us access if not. As soon as our team completes the review, our customer receives an email notification.

In terms of prioritization, we start with the review requests from our customers; then, we proceed with the ones who passed through our digital filter, starting with the highest number of downloads.

At this point, the DHT monograph contains the information from our algorithm:

Step 2 - Acceptability & Feasibility

When starting a review, the app analysts assess relevant criteria on six key domains (usability, privacy, functionality, interoperability, regulatory, and economic criteria). This assessment aims to highlight if a tool contains what we refer to as no-go criteria. In simple words, would this DHT expose patients to specific (privacy) risks? Can patients and providers make an informed decision regarding how their data is managed and used? Are there bugs or usability issues?

These no-go criteria are part of a 30+ more data points analysis, which will be completed by the app analyst for every app included in TherAppX Core:

Usability (user experience)

Those data points include assessments of compatible connected devices and Healthkit/GoogleFit connectivity, touch controls, design and bugs. Since our digital filter is efficient, only 5% of the assessed DHT have minor usability issues such as bugs or lack of user-friendliness.

Privacy & security

We first look at whether the manufacturer announces compliance with privacy policies (e.g. PIPEDA, GDPR, HIPAA, etc.) or security standards (ISO27001, IEC, SOC2, etc.). Then, we assess the developer’s transparency, mainly in its privacy policy and data rights management. We display relevant information (e.g. collection of Personal Health Information, data hosting countries, personal data recipients, etc.) in TherAppX Core.

Functionality & interoperability

Our app analysts identify the tool’s key features and the health conditions which may benefit from a given app, as announced by the developers. Among the features, we’ll assess an app's capacity to exchange data with a healthcare provider (e.g. simple sharing by email vs synchronization with an Electronic Health Record).

Regulatory status

We identify whether the tool has received endorsement from regulatory agencies such as the FDA, the European Commission and Health Canada.

Evidence

Each app is searched in Pubmed and Google scholar as we screen for published articles about the app. Our app analysts will categorize each paper according to its study type, whether it’s a clinical trial or featured in a systematic review, or any of the four other kinds of studies we track in TherAppX Core.

Economical Feasibility

Our algorithm can collect pricing information, but we still might not have the real portrait of an app’s business model. Is it available on the App Store for wellness purposes, or is it a Digital therapeutics tool available only through clinicians? To further analyze an app, app analysts will use the developer website, among other sources of information, to determine how a patient may access the DHT. We match this information with economic studies to highlight DHT that were the subject of at least one cost-effectiveness analysis.

| Quick note: CORE acts as a dictionary on Digital Health Tools. We include ALL DHT found relevant by our advanced software & researchers, whether they’re good or not. It allows users to see objective criteria on each of them. Users may discriminate between DHTs by browsing, using filters, or taking advantage of the DHT comparison functionality (coming soon).

At this point, the DHT monograph contains information from our algorithm and our app analyst review:

Step 3 - Clinical Assessment

This step is where TherAppX stands out, as we are the only organization in the world that achieved to review 700+ DHTs by real humans involved in patient care on a day-to-day basis (i.e. clinicians).

Our clinicians review all DHT which do not contain no-go criteria. To predict clinical relevance, they use a process that is very similar to the app analysts’. They assess different data points (20) and write an app description displayed in TherAppX CORE for providers. Then, Core’s users that pay for the Prescribe functionality may use and personalize (if needed) our posologies prewritten by our clinicians during their assessments, when they send an app recommendation to a patient (coming soon)

Some data in the clinician review process point to the clinical objectives of the reviewed tool. Each patient has different needs for using DHTs. Providing this specific information to providers allows them to make informed decisions, confident that the data comes from a peer and not the DHT manufacturer itself.

Other data points provided by the clinicians and featured in TherAppX CORE are on the tool’s behaviour change components (as defined by ABACUS). This information makes it possible to determine if the DHT has the minimal requirements to drive behaviour change for patients requiring such interventions.

Finally, while a DHT may have great features, this doesn’t mean these are relevant to all health conditions mentioned by an app maker in its description. For instance, we've often seen DHT targeting anxiety and depression and providing features applicable only to anxiety situations. Therefore, our clinicians confirm the DHT’s appropriateness for each claimed health condition and provide our users with this information in TherAppX CORE.

To conclude, data displayed in TherAppX Core achieved an “almost perfect” inter-reviewer variability rate, based on Cohen’s Kappa statistical analysis (we’ll cover this aspect in another article coming soon).

At this point, the DHT monograph is complete:

Step 4 - Evolution Assessment

As stated above,  DHTs update and change frequently.

To overcome this challenge, we make sure to catch every previously reviewed DHTs update. Then, we analyze changelog information from those DHT. Should an update contain sufficient modifications to any of the data points we previously explored, we change it in the TherAppX CORE library.

Most of the time, updates are minor. Once in a while, manufacturers roll out significant changes of their DHT (refactoring, for example), which may impact how patients react to it from a clinical perspective. In such a case, both researchers and clinicians update their past reviews to account for the new information they have in hand.

Moreover, using our advanced algorithm, we can detect if an app is no longer available on a given marketplace (e.g. Apple App Store).

Thereby, users will have the most accurate and up-to-date information about each app in our CORE  library at any moment!

About our relations with DHT manufacturers

At TherAppX, we keep distances between our review processes (our products’ foundations) and DHT manufacturers. For our products to display the best information for our users, we don’t engage ourselves in business relationships with vendors or manufacturers.

That being said, we allow manufacturers to feed information to our app analysts if they are aware of missing data in TherAppX CORE, or if they want to fast-track the evaluation of their DHT by our team. They can do so by using this form.

Our users never see information provided by manufacturers and vendors. They are available only to app analysts. This self-assessment helps them zero-in on the information in the first two steps of our process. From time to time, TherAppX’s app analysts may also send manufacturers and vendors information requests using the previous form.

Last but not least, we offer manufacturers the possibility to access our comprehensive assessments for improvement purposes (including data points not currently available in Core). It is our way to contribute to enhancing the overall quality of health DHT offered to patients. If you’re interested in owning a piece of your DHT comprehensive assessment, please contact our sales team at support@therappx.com.

Conclusion

We hope our series of blog articles help you understand our processes. In a previous article, we’ve highlighted our vision when it comes to health app evaluation. We’ve focused on explaining how a mix of quality criteria and Real-World validation ensures patient-centric DHT recommendations that translate into value. In this one, we highlight our review process and the different data points we gather, which you may also find in Core.