Open challenges for the automatic synthesis of clinical trials

Abstract

Objective

An important criterion for selecting clinical trials to be compared in systematic reviews and meta-analyses is that they measure the same outcomes. However, this represents a challenge as there is a wide variety of outcomes, and it is difficult to standardize them for comparing clinical trials containing them. To address this challenge, we utilized our annotated dataset, which includes 211 abstracts of clinical trials related to glaucoma and type 2 diabetes mellitus. We then developed a tool that provides an overview of the annotated clinical trial information and enables users to group them by outcomes.

Results

Using our visualization tool, we formed groups of outcomes and their respective clinical trials. We were able to determine the most common outcomes in clinical trials for these diseases. As a case study on diabetes, we compared our outcomes with those consented by diabetes stakeholders and found that many of the grouped outcomes are aligned with the consented ones. This demonstrates that tools such as the one presented can help standardize clinical outcomes, which in turn help in the synthesis of clinical trials. Finally, we also offer some recommendations that could help in the automation of clinical trials based on outcome standardization.

Introduction

Systematic reviews (SR) and meta-analyses (MA) are regarded as the gold standard in healthcare decision-making because they offer strong evidence for addressing clinical research questions by synthesizing clinical trial (CT) information. Therefore, they are valuable resources for clinicians, government policymakers, and pharmaceutical companies as they help determine which treatments are the best for a specific disease and population in terms of efficiency and safety. This can also help in deciding on the costs and availability of treatments. However, conducting these studies can be time-consuming and arduous, due to the many steps involved such as literature search, removal of duplicate citations, study selection, data extraction, quality assessment, statistical analysis, data visualization and report writing [1, 2]. While tools are available to assist with this process, many still require vast human intervention [2,3,4]. In this paper, we mainly focus on an important part of the selection stage, which consists of deciding which CT publications are eligible for a SR or MA. An important criterion is to select those CTs that have comparable outcomes that can be synthesized. However, due to the heterogeneity of outcomes and their measurements, it can be difficult to correctly compare and combine studies of the same disease. An effort to mitigate the heterogeneity problem is made by the Core Outcome Measures in Effectiveness Trials (COMET)¹ initiative [5]. COMET is working on creating standard outcomes for each specific disease called core outcome sets (COS) [6]. This is achieved through consensus among relevant stakeholders, including patients and healthcare professionals, who agree on the minimum set of standard outcomes for a particular disease. The expectation is that researchers use COSs in all studies for a particular disease, adding other outcomes if necessary. This would facilitate the automation of SRs and improve the quality of their assessments.

In this paper, we explore the challenges that may arise when automatically synthesizing CT information. To this end, we analyze the information contained in annotated CT abstracts [7] using a web tool developed for this purpose. We assume that our analysis applies to cases where the information is already available in a format that can be automatically processed. The following sections describe the methods used for our study, a case study on T2DM, a discussion on the challenges for the automation of CT synthesis and ways towards it, and a discussion of the limitations of the present study and final remarks.

Methods

In this section, we describe the dataset used and the developed web-based tool for our study.

Dataset

We use an annotated corpus that consists of 107 abstracts on glaucoma and 104 abstracts on type 2 diabetes mellitus (T2DM). These abstracts were previously utilized in another study for the synthesis of CTs [8]. In this earlier work, glaucoma was selected due to its relatively well-defined outcomes, while T2DM was chosen due to the expertise of one of our medical collaborators in this area, despite it being more complex than glaucoma. The abstracts were annotated following the C-TrO ontology schema [9] that covers the PICO elements (P: population/problem, I: intervention, C: comparison, and O: outcome), which are relevant components of a CT. C-TrO considers intervention groups (or arms) to measure the outcomes/endpoints² of the studied interventions/treatments, as well as other relevant elements of published clinical studies, such as baseline values, changes from baseline values caused by interventions, and the corresponding statistical information. Supplementary Fig. 1 shows the C-TrO schema, and the annotation dataset can be accessed via the link provided in “Availability of data and materials”.

The clinical trial visualization tool (CTV tool)

Figure 1 depicts the overview interface of the web-based CTV tool we have developed for the purpose of this study.³ The upper left area contains a list of CT outcomes. To facilitate the assignment process, the outcomes are initially assigned to outcome groups. The initial assignment is done by calculating the similarity between an outcome and the names of the available outcome groups using the Levenshtein distance [10] which measures the similarity between two strings. The group name with the highest similarity score is then chosen to represent the outcome. Users can remove an outcome from a group by selecting “No group” in the group drop-down list. Or they can move it to another group by selecting a different group from this list. The bottom left area displays a summary of the formed groups, showing the number of outcomes in each group. Clicking on a group name displays the list of CTs containing the outcomes belonging to that group on the right side area, where detailed information about a selected CT is shown by clicking on it.

Fig. 1

Overview interface of the CTV tool. In the example, the group “fasting plasma/blood glucose” is selected. It contains 53 CTs which are listed on the right side

Case study on T2DM

The goal of this case study was to see to what extent we could categorize the outcomes in our T2DM dataset according to a consented set of outcomes (i.e., COS) for T2DM. We chose to conduct the study on T2DM since, at the time of writing this paper, there was only one suitable SR identifying COS for T2DM, yet none for glaucoma in the COMET database [11]. We then chose the available SR in COMET called SCORE-IT (Selecting Core Outcomes for Randomised Effectiveness Trials in Type 2 Diabetes) [12] as the baseline for comparison. In SCORE-IT, the identification of core outcomes entailed first extracting potential core outcomes from registered trials of therapies for T2DM. Subsequently, they were categorized according to the COMET outcome domains. Discrepancies in their categorization were resolved through consensus among SCORE-IT authors. The majority of analyzed trials (88%) in SCORE-IT contain outcomes in the “metabolism and nutrition” domain. The path followed by these core outcomes in the COMET taxonomy is: “Physiological/clinical” core area $\rightarrow$ “Metabolism and nutrition” core domain $\rightarrow$ T2DM core outcomes. The identified core outcomes are related to lipids and lipoproteins (21%), HbA1c (18%), hypoglycaemia (14%), fasting plasma/blood glucose (11%), glycaemic variability (8%), postprandial response (8%), and insulin sensitivity (5%). The remaining outcomes are diverse.

We used the CTV tool to visualize and assign outcomes to the SCORE-IT COS (or SCORE-IT groups). When there were not any suitable SCORE-IT groups to categorize two or more similar outcomes, we created groups into which these outcomes could be categorized (i.e., potential COS) and added to the CTV tool (e.g. body weight). If there was only one outcome without a category, it was assigned to the general category “Other” (e.g. bile acid synthesis). In this way, we categorized all outcomes. Table 1 shows the outcome groups for T2DM. The first column contains the SCORE-IT COS and the second column has the outcome groups created and that are not included in SCORE-IT.

Table 1 Outcome groups for T2DM. (*Outcomes identified in this study and not included in SCORE-IT)

After grouping the outcomes of the CTs in our dataset, we found that similar to SCORE-IT, the domain “metabolism and nutrition” was the most frequent, with the following core outcomes related to HbA1c (14%), HbA1c with a given level target (30%), postprandial response (14%), fasting plasma/blood glucose (9%), insulin sensitivity (7%), glycemic variability (4%), and lipids (4%)⁴. The remaining outcomes (18%) had low percentages and were assigned to the groups in the second column of Table 1. It is important to note that in SCORE-IT the number of CTs containing outcomes is counted, whereas we consider all outcomes reported in the CT abstracts in our counts. This includes outcome subgroups and time point outcomes. Nevertheless, our counting provides a good indication of which outcomes are most commonly found in T2DM trials.

We could categorize 82% of the outcomes into the SCORE-IT groups and the rest (18%) into the groups we created. This shows that we can use SCORE-IT to a large extent and also identify new potential core outcomes with the CTV tool.

Challenges for the automatic synthesis of clinical trials

In this section, we discuss the challenges encountered during our endeavor to automate outcome-based CT synthesis. Our analysis utilizes both glaucoma and T2DM datasets.

Authors may describe the same outcome in different ways. For example, Table 2 shows different descriptions for the same outcome “Diurnal IOP” across different CTs. The naming of outcomes may be ambiguous, at different levels of detail, or overly descriptive, making it uncertain whether a set of outcomes can be considered similar and therefore comparable. Furthermore, the naming of outcomes may include, for example, a target measurement (e.g., HbA(1c) < or = 7%) or any other characteristic that describes their function in a CT rather than what they intrinsically are. This lack of unique outcomes that can group outcomes that refer to the same concept in the same clinical context but with different wording can lead to varied and seemingly different outcomes across CTs [13], complicating CT synthesis.

Table 2 Different outcome descriptions for “Diurnal IOP”

Authors often report their methods and results without following recommendations for writing abstracts that despite being short, accurately describe the relevant CT information. This often results in missing relevant CT information from abstracts. For example, the Consolidated Standards of Reporting Clinical Trials (CONSORT) [14] outlines recommendations for reporting clinical trial abstracts and outcomes in trial reports [15, 16]. However, in many cases, publishers and CT registries do not require authors to use standard results and follow reporting guidelines such as CONSORT.

Developing a COS can be a slow and hard process since it requires reaching a consensus among relevant stakeholders. Besides the ambiguity and inconsistency in how outcomes are described across different studies, a major obstacle, as noted by Saldanha et al. [17], can be matching the scope of the COS with the scope of the SR. For example, the COS target population may be too narrow/broad compared to the population in the SR, or the COS target intervention may be too narrow/broad relative to the SR intervention [17]. Thus, it is not a general practice for researchers to use a COS in CT protocols and SRs.

Grouping CTs by duration or time points can be confusing, as the same treatments across CTs may have different measurement values depending on when the measurements were taken.

The heterogeneity of measurements has to be considered when synthesizing CTs or pooling them for secondary studies. For example, clinicians should consider converting the measurement values to homogenized outcome information for SRs when similar outcomes across CTs are reported in different units. However, pooling is only feasible if there is highly detailed information about the original studies, and collaboration with the original authors of the CTs, who can retain the ownership of their data. Other challenges in combining data are the quality and completeness of the data being shared, different ways of reporting events, different designs, and insufficient description of the trial settings. These factors can lead to misinterpretations or bias. In this respect, Wilkinson et al. [18] observed that if the protocols of different trials are not identical, but their baseline populations are similar enough, then their data can be integrated and combined.

The analysis of the CT information and its grouping based on core outcomes shows that several aspects of automatic CT synthesis still need to be addressed. In the following section, we offer some suggestions that we believe could contribute to the automation of CT synthesis.

The way towards the automation of clinical trials synthesis

To advance in the automation of CT synthesis, we propose the following:

• 1. Journals and CT registries should require the use of standard outcomes, like COSs and the COMET taxonomy, when reporting CTs. If standard outcomes are not used, authors should explain their reasons, providing valuable feedback to initiatives that develop such standard outcomes.

• 2. Journals and CT authors should adhere to guidelines such as CONSORT when writing abstracts.

• 3. Standard outcome sets should be defined by identifying common outcomes across trials studying the same disease and creating a consented list of them. This can be done by experts in the specific medical area, supported by tools such as the one presented in this work.

• 4. Standardized outcomes should be transformed into DOI-type objects, which have persistent identifiers. This object could contain information such as a full description, provenance, synonyms of the outcome name in a vocabulary, etc. In this way, such “digital outcome identifiers” could be linked to other clinical information available on the web through knowledge graphs and specialized vocabularies.

• 5. Tools should be used to enable the input of CT information into a structured template, following an ontology or knowledge graph structure, to ensure that the information is machine-readable (e.g., CTrO-Editor [19]).

Adopting these suggestions would improve and facilitate searching, synthesis of CTs, and reuse of CTs in secondary studies. These measures are aligned with the FAIR principles [20] to make CT data machine-readable, enabling computer systems to find, access, interoperate, and reuse CT data. This approach would also help address the growing volume of CT information and the ambiguity surrounding outcome descriptions.

Limitations and final remarks

Currently, there are very few SRs of core outcomes for most medical conditions in the COMET database or other sources. This makes it challenging to compare outcomes from trials of different conditions using consented core outcomes. Since our dataset only covers two health conditions, it also limits our ability to compare diseases for which core outcomes are available. Some CT information may be missing in our dataset because CT abstracts may not contain all relevant information or due to missed/wrong assigned annotations. Despite these limitations, it has been demonstrated that tools such as the CTV tool can help to identify and categorize common outcomes, which may contribute to future efforts in developing core outcomes for diseases which do not have them yet. This can potentially enhance the automation of CT synthesis, as well as their search and reuse.