Introduction
Under the Universal Health Care (UHC) Act of the Philippines, all health technologies should undergo health technology assessment (HTA). In this law, HTA is defined as a systematic evaluation of the social, economic, organizational, and ethical impact of health-related technologies and interventions, using a multidisciplinary process, to improve health outcomes (1). This process ensures the rational utilization of various health technologies that will be funded by the government.
The current Philippines’ HTA Methods Guide provides broad guidelines in the conduct of systematic reviews and economic evaluations for the production of standard HTA reports for healthcare decision makers and other target audiences in the health system (2). It does not provide guidance specific to the use of real-world evidence (RWE) for the clinical evaluation of health technologies.
There are several guidance papers on the use of RWE to support regulatory decision-making developed in Canada, the United Kingdom, the United States, Europe, and some Asian countries (3–Reference Lin, Ahn and Bayani8). This manuscript details the process of the development of the guidance document for the use of real-world evidence in the clinical evaluation of health technologies in the Philippines.
Methodology
Study design
This study consisted of two phases. Phase 1 was a comprehensive, systematic review of all available HTA methods guides and literature related to the use of RWE in the clinical evaluation of health technologies. Based on the results of the review, a draft HTA methods guide on the use of RWE was created. Phase 2 was a validation study by expert consultation through key informant interviews (KIIs), and pilot assessment of the methods guide.
Phase 1
Search strategy
We performed a comprehensive and systematic search of electronic medical databases, including MEDLINE, Cochrane Library, Google Scholar, Scopus, and EMBASE. We used free text search or medical subject headings (MeSH) of the following: ‘real world data’, “real world studies,” “real world evidence,” and “health technology assessment” or “biomedical technology assessment.” We also searched HTA organization websites, including The International Network of Agencies for Health Technology Assessment (https://www.inahta.org/), and regional and national HTA organizations as listed in Supplementary Material 1. The systematic search was independently performed by CTL and MIA.
Manual search of references was also done. Preprints and grey literature were included. Unpublished literature was searched by writing to content experts in the field and surveying conference proceedings and books of abstracts. All HTA methods guides related to the use of RWE to evaluate clinical effectiveness and safety of health technologies, and articles related to the definition of RWE and real-world data (RWD), use of RWE for clinical evaluation of health technologies, search and screening, critical appraisal, data extraction, or data analysis and synthesis of RWE were included.
Data extraction
All HTA method guides were retrieved. Critical appraisal of the included studies was done independently by two authors, namely MIA and the assigned chapter writer (NEI for Chapter 1, IGC for Chapter 2, AMJ for Chapter 3, HGB for Chapter 4, ADB for Chapter 5, and KLC for Chapter 6) using the appropriate critical appraisal tool following the JBI recommendations (9). Data extraction was independently done by the same two authors. The following data were extracted:
-
a. Definition of RWD and RWE.
-
b. Guidance on when it is appropriate to use RWE for clinical evaluation of health technologies.
-
c. Methods for the search and selection of RWE for clinical evaluation of health technologies.
-
d. Methods for the critical appraisal of RWE.
-
e. Methods for data extraction of RWE for clinical evaluation of health technologies.
-
f. Methods for data analysis and synthesis of RWE for clinical evaluation of health technologies.
Data analysis was performed by the same two authors, whereas CTL and MMA performed cross-validation of the analysis.
Data presentation and preparation of draft HTA methods guide
The results of the systematic search were reported following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guideline for systematic reviews. If there were variations in methods from the different sources included in the review, the differences were to be described. The choice of which method was ultimately recommended in the HTA methods guide, and the justification for this choice, were to be described.
All data extracted were organized into an outline. The writers developed the first draft of the HTA methods guide. Two editors reviewed, edited, and collated all sections to develop the complete first draft of the HTA methods guide.
Phase 2
Study procedure
We validated the drafted HTA methods guide from Phase 1 through expert consultation using KIIs and pilot assessment.
Study participants for KIIs
The selection criteria for participants of the key informant interviews were:
-
1. With previous experience in the creation of a methods guide on the use of RWE for HTA or regulatory decision making; OR
-
2. With expertise in using RWE for clinical evaluation of health technologies.
We used purposive sampling to choose three methodological experts for the KIIs. The author/contributor list of all HTA methods guides retrieved in Phase 1 was reviewed. We contacted authors with available e-mail addresses or contact numbers to participate in the study. We also inquired with the Philippine HTA Council if they have contact details of potential participants for the KIIs.
KII questions
The questions for the KII were generated after completion of the draft HTA methods guide from Phase 1. The questions centered on the validity and completeness of the proposed guidance document on the use of RWE for the evaluation of health technologies. Questions were pretested with two Philippine experts who had experience in the creation of the first edition of the Philippine HTA methods guide. The questions were revised based on the results of the pretesting.
Conduct of the KII
The KII was conducted by the authors through an online platform. We asked for consent from the KII participants to record the interview. The responses of the KII participants were transcribed and summarized. The HTA methods guide was revised based on the feedback of the KIIs.
Pilot assessment of revised HTA methods guide
Pilot assessment of the revised methods guide was then conducted using various health technology topics. Five potential users of the methods guide were purposively selected from the list of previous evidence review experts with varying experience in preparing evidence summaries for the clinical evaluation of health technologies. The pilot assessors evaluated the following health technologies: 1) Pazopanib, Sunitinib, and Pembrolizumab + Axitinib for renal cell carcinoma; Trastuzumab emtansine for breast cancer; 2) Palbociclib and Ribociclib for breast cancer, 3) Ceftaroline fosamil for community acquired pneumonia, 4) Biphasic Insulin Aspart 30 for type 2 diabetes mellitus, and 5) Atezolizumab and Pembrolizumab for nonsmall cell lung cancer. The specific steps where the guidance document was applied by the pilot assessors are summarized in Supplementary Material 2. Written feedback on the appropriateness, applicability, and ease of use of the guide was then obtained. The results of the pilot assessment were summarized. Revisions to the guidance document were made based on the results of the pilot assessment.
Review of the Philippine HTA Council and Division
The revised guidance document was reviewed by all members of the Philippine HTA Council and HTA Division. No specific criteria were used by the HTA Council and Division members in evaluation, but a comprehensive review encompassing the content, applicability, and consistency of the guidance document to the published Philippines’ HTA Methods Guide was conducted. Their feedback was incorporated into the final version of the guidance document.
Results
Literature search
After deduplication, initial screening of titles and abstracts, and full-text review, a total of seventy-nine journal articles and nine guidance papers from different countries were included in the literature review. The PRISMA flow diagram is shown in Figure 1.
Figure 1. PRISMA flow diagram. This figure illustrates the search process and results, where ultimately, seventy-nine journal articles and nine guidance papers were included.
Data extraction and synthesis
The information obtained from the articles and guidance papers were collated and organized into six sections, namely, section one: operational definitions of RWD and RWE; section two: when it is appropriate to use RWE; section three: methods for the scoping and selection of RWE; section four: critical appraisal of RWE; section five: methods for data extraction; and section six: methods for the synthesis and statistical analysis. There were twenty-nine journal articles containing relevant information for section one; thirty-five articles containing relevant data for section two; fourteen articles for section three; twenty-five articles for section four; six articles for section five; and thirteen articles for section six. Of the nine guidance papers, three provided comprehensive information for all six sections. The characteristics of the included papers and critical appraisal are in Supplementary Materials 3 and 4. The first version of the guidance document was drafted based on this information.
Expert consultation through key informant interviews
Three experts from Singapore, Thailand, and Australia reviewed the first version of the guidance document. The experts commended the guidance document for its clarity, comprehensiveness, ease of understanding, and practical approach to providing guidance on the use of RWE for clinical assessment of health technologies.
The major points for revision of the guidance document raised by the experts included the following: (1) providing further discussion on the appraisal of RWE from secondary data sources, (2) clarifying the role of qualitative research in RWE, (3) clarifying whether N-of-1 trials were considered RWE, (4) refining the indications on when to use RWE for clinical evaluations, (5) consideration of target trial emulation (TTE) as RWE, and (6) the necessity of following reporting guidelines to ensure consistency and quality in the crafted data extraction form.
One expert recommended that the importance of transparency and reproducibility for secondary data sources be highlighted in the guidance document, because the initial draft seemed to focus more on primary data sources. This recommendation was incorporated into the revised draft, with an emphasis on the need for accuracy and completeness of data obtained from registries, databases, electronic medical records, and health insurance and administrative records.
The expert also suggested including a discussion on the role of qualitative studies to provide a more comprehensive perspective on health professionals’ experiences and consumers’ perspectives regarding the health technology in the guidance document. This comment was incorporated in the guidance document by discussing that qualitative studies also provide RWE for patients and caregivers’ experiences. We clarified, however, that because the focus of the guidance document is on clinical evaluation, the document will not focus on qualitative evidence synthesis.
One expert suggested specifying whether N-of-1 trials are considered RWE. The expert said that there is currently no consensus as to whether N-of-1 trials can be considered RWE. She opines that N-of-1 trials may be considered RWE if the study was conducted without using stringent inclusion and exclusion criteria. However, if N-of-1 trials were conducted with stringent inclusion and exclusion criteria, then they are more akin to clinical trials. Based on our literature review, N-of-1 trials utilize randomization to introduce the sequence of interventions to the study participants. We introduced the concept of N-of-1 trials in this guidance document. We did not consider N-of-1 trials as a type of RWE; rather, we explained that RWE may complement the results of N-of-1 trials in conditions where it is not feasible or ethical to perform RCTs. The expert agreed with our revision.
Another major point raised by an expert was ensuring that RCTs are not replaced by RWE when RCTs are feasible to conduct. The discussion emphasized that health technologies where the only available evidence is underpowered RCTs or RCTs appraised as high risk of bias should not automatically lead to the use of RWE. It was cautioned that RWE may not be appropriate in cases where underpowered RCTs arise from poor study design or execution. The context of rare diseases was highlighted as a scenario where the criteria for accepting RWE may differ. The discussion on when to accept RWE underscored the need for well-defined criteria, with caution against accepting RWE as a substitute for poorly done RCTs. It was emphasized that RWE may only be acceptable when RCTs are not feasible, such as in rare diseases. RWEs may also be considered when RCTs cannot capture long-term endpoints.
The concept of target trial emulation was also discussed. TTE is a novel approach wherein principles of RCTs are applied to existing data sources to emulate an open-label RCT. The novel approach of TTE and the current lack of standard appraisal tools were discussed. The expert acknowledged that TTE may be discussed further in future versions of the guidance document.
One expert recommended linking available reporting guidelines to the data extraction form to ensure consistency, avoid duplication of efforts, and maintain transparency. This linkage helped ensure that the extracted data follows the established reporting guidelines. For example, disclosing funding sources in research adds to its credibility. Including this information in the data extraction form helps the HTA council to evaluate the RWE more objectively.
Other suggested revisions included clarifying the comparator in RWE (e.g., standard care), mentioning the hierarchy of evidence among observational studies, further detailing the checklist of items for reporting RWE from observational sources (i.e., background, rationale, study design and setting of observational studies using routinely collected health data), updating the name of the Canadian HTA unit, and updating citations and references. The second version of the guidance document was drafted following the expert consultation using KIIs.
Pilot assessment
Five pilot assessors evaluated the second version of the guidance document based on its purpose and target audience, clarity and readability, accuracy and completeness, and consistency and formatting. All assessors agreed that the manual is organized in a logical and easy-to-follow manner. The majority of the assessors believed that the manual meets the needs of its target audience, the language is clear and easy to understand, the technical concepts are adequately explained, and the procedures and instructions are logical and comprehensible.
The main points for revision following the pilot assessment included concerns about the phrase “noninterventional setting” in the definition of RWD, because it may be interpreted as interventions received during routine medical care. In response to this comment, we revised “noninterventional setting” to “nonexperimental setting” in the definition of RWD. We also clarified the difference between RWD and data obtained from RCTs, provided examples of surrogate outcomes and selection bias, explained the reason for the underutilization of standard reporting guidelines for RWE, and provided further explanation why adjusted effect estimates and partial regression coefficients are preferred over raw data in the data extraction and analysis. We also briefly described the concept of network meta-analysis. Lastly, we clarified why the rating for certainty of evidence for observational studies typically starts at a low level of certainty, due to the lack of randomization of these studies. Minor revisions were also made for consistency in headings, text formatting, citations, abbreviations, and grammar.
Review of HTA council and HTA division
The guidance document was then reviewed by members of the Philippine HTA Council and the HTA Division. The main points for revision were 1) clarifying that pragmatic clinical trials, although conducted in an experimental setting, are still considered RWE; 2) the addition of pandemics as emergency situations during which RWE may be used to guide urgently needed decision-making whereas results of RCTs are not yet available. However, immediate re-assessment should be done once results of RCTs are available, because RCTs are still the most appropriate study design for clinical evaluation of health technologies; and 3) addition of chapter summaries for each chapter. The guidance document was then finalized following these revision points, and can be accessed as Supplementary Material 5.
Discussion
The Philippines’ guidance document incorporates the available literature as of March 2024 on the use of RWE for clinical evaluation of health technologies. We produced a comprehensive guidance document that covered all steps related to the use of RWE in the clinical evaluation of health technologies, including definitions of RWE and RWD, guidance on when to use RWE for clinical evaluation, searching and selecting RWE, critical appraisal, data extraction, and data analysis and synthesis of RWE. Through expert consultation, pilot assessment, and review of the HTA Council and HTA Division, we ensured that the guidance document was accurate, updated, and complete, but also succinct, practical and easy to understand for our potential users.
This guideline contains similar methodological principles to other existing RWE guidelines. All frameworks characterized RWD as data obtained from routine clinical practice and provided the basic principles for generating quality RWE. However, the priorities and intended applications of the guidelines vary based on local health system requirements. The frameworks from the US and France generally focused on regulatory decisions, including postmarketing surveillance and label expansion, whereas the UK guidelines discussed the use of RWE throughout the life cycle of a health technology and included evaluation of service delivery (3;5;7). The guidelines from Canada (CADTH and INESSS) focused on RWE to guide reimbursement decisions in public health programs and pre and postdrug marketing evaluations (4;6). In contrast, the REALISE guideline, which applies to health systems in Asia, with mostly low- to middle-income countries, focused on practical ways in the collection and analysis of RWD or RWE when resources are limited (Reference Lin, Ahn and Bayani8). This Philippine guideline focuses on the application of RWE in HTA assessment to inform decision-making of regulatory agencies in the procurement and reimbursement of health technologies, which is a pressing need given the limited resources and fragmented health system of the country.
The definition, methodological designs, and applications of RWE are continuously evolving. We will update this guidance document as necessary to incorporate future reports and guidelines relevant to the use of RWE in clinical evaluation of health technologies. The primary target audience for this guidance document is researchers aiming to produce HTA reports using RWE. Technical concepts were presented and explained concisely, with the intention that these concepts be understood clearly without overwhelming the reader with too much technical details. Reviewers who wish to delve deeper into the technical components of these concepts are encouraged to review the latest relevant literature.
The development of this guidance document has some limitations. First, due to the financial and logistical constraints, only three methodological experts were chosen as key informants. Given the continuous evolution in the definition, methodological designs, and applications of RWE, and heterogeneity in the perspectives on using RWE, the inclusion of more key informants would have greatly enriched the guidance document. Second, the review by the stakeholders, including the HTA Council and HTA Division, yielded comments on the use of RWE for HTA in general. Specific comments regarding its application to particular health technologies were not included, because guidance documents for specific health technologies (e.g., orphan drugs) are being developed.
Conclusion
The use of RWE to inform healthcare decision-making has been gaining increased salience. This document provides guidance for defining, utilizing, searching, appraising, and analyzing RWE in conducting clinical evaluation of health technologies as part of HTA in the Philippines. RCTs remain the most appropriate evidence for the clinical evaluation of health technologies. It is crucial to exercise caution when using RWE for HTA due to inherent biases and the potential for generating misleading conclusions that may under- or overestimate the effectiveness of health technologies. However, RWE may be used in the clinical evaluation of health technologies when RCTs are insufficient to make a decision due to failure to capture long-term clinical outcomes, and when RCTs are impossible to conduct due to feasibility or ethical issues.
Recommendations
We aimed to produce a guidance document that was complete, accurate, but also easy to use by the target audience. Close coordination and discussion with key stakeholders were essential. Throughout the development of the guidance document, the team closely worked with the Philippine HTA Council and Division to ensure applicability and acceptability of the proposed methodology and outline. Assessment of the guidance document by methodological experts and potential end-users, and provision of a pilot-tested assessment tool also contributed greatly to the process. We encountered difficulty in identifying and securing the availability of foreign methodological experts. This challenge may be overcome through improving social networks of methodological experts on an international scale, allocating more time to identify and invite methodological experts through official institutional affiliations, and allotting adequate financial resources to facilitate the consultation process.
Supplementary material
The supplementary material for this article can be found at http://doi.org/10.1017/S0266462325100512.
Acknowledgements
We would like to thank the following individuals for their participation and support in creating the methods guide: Wanrudee Isaranuwatchai, Lucylynn Lizarondo, Wee Hwee-Lin, Kerwyn Jim C. Chan, Kimberly Mae C. Ong, Zaira Nina D. Duque, Frangelo Conrad Tampus, Cary Amiel G. Villanueva, Maria Vanessa Sulit, Adovich Rivera, and the members of the HTA Council and HTA Division.
Funding statement
This project was supported by the Department of Science and Technology Philippines - Health Technology Assessment Division.
Competing interests
The authors declare none.
Open access
