Introduction
Dissemination and Implementation (D&I) science seeks to address the well-documented gap between research and practice by advancing understanding of the strategies and methods for promoting the widespread and routine uptake of evidence-based interventions in clinical and community settings [Reference Brownson, Colditz and Proctor1]. D&I science represents a cross-cutting domain in the spectrum of clinical and translational science, especially relevant to T3 and T4 research [Reference Leppin, Mahoney and Stevens2]. Clinical & Translational Science Awards (CTSAs) are encouraged to provide opportunities for education and capacity building to support D&I science (PAR-21–293), in addition to having experienced D&I scientists provide consultation, mentorship, and collaboration to faculty and trainees [Reference Huebschmann, Johnston and Davis3]. Yet there are challenges ensuring adequate numbers of D&I experts willing and available to serve as mentors and teachers in D&I training programs [Reference Chambers, Proctor, Brownson and Straus4,Reference Brownson, Jacob and Carothers5]. As a result, D&I science capacity building efforts in CTSAs has become a common topic of exploration within the field.
There have been several recent efforts to track D&I capacity-building efforts. For example, Viglione et al [Reference Viglione, Stadnick and Birenbaum6] cataloged 165 D&I science capacity-building programs worldwide, with 68% based in the United States. Huebschmann and colleagues [Reference Huebschmann, Johnston and Davis3] identified approaches to providing training in D&I science, including academic graduate degree-granting programs and certificate programs, mentored fellowships such as K12 training programs, postdoctoral fellowships, and national and university-based training, certificate, and degree programs. Additionally, Mehta and colleagues [Reference Mehta, Mahoney and Leppin7] described opportunities for enhancing the capacity and impact of D&I science at CTSAs. To support D&I capacity-building efforts, a proliferation of online resources and interactive tools have also been developed to aid in the selection and application of D&I principles and methods [Reference Ford, Rabin, Morrato and Glasgow8]. To organize these resources, D&I experts in CTSAs and other academic institutions have developed catalogs of resources to help both new and established D&I researchers access and navigate these trainings and tools. For instance, there are catalogs of D&I theories, models and frameworks [Reference Tabak R.G., Chambers and Brownson9] and measures [Reference Rabin, Lewis and Norton10]. However, the multitude of resources and associated catalogs can overwhelm investigators seeking clear guidance on where to find the most up to date and comprehensive information about D&I science and practice methods. As such, there is value in creating a centralized, curated list of these various cataloging efforts.
The Clinical & Translational Science Awards (CTSA) Program’s “Advancing Dissemination and Implementation Sciences in CTSAs” Working Group (WG) convened in 2020 to provide practical tools and insights to help CTSA hubs integrate D&I science. A stated goal of the working group was to address previously identified need for D&I expertise capacity building across CTSA hubs [Reference Shelton, Dolor and Tobin11,Reference Dolor, Proctor, Stevens, Boone, Meissner and Baldwin12]. Indeed, CTSA hubs that provide mentoring and collaborative team science support for investigators integrating D&I science into their research and career development activities reported increased demand that had begun to outstrip existing capacity. As part of this work and in recognition of the CTSA consortium’s emphasis on D&I capacity building, the WG prioritized the need for an aim focused on developing practical approaches to guiding and growing the D&I workforce, including the development of a centralized resource hub for cataloging toolkits, websites, videos and other resources already offered by CTSA hubs.
This paper presents the WG’s development and initial implementation of the CTSA Compendium of D&I Resource Catalogs - a curated and dynamically-updated list of resource catalogs relevant to the conduct of D&I science.
Materials and methods
Working group initial planning
As an initial step, a sub-working group consisting of four regular members was formed following a solicitation of interest to the full working group, identifying collaborators from three CTSA hub institutions (Colorado Clinical & Translational Sciences Institute, Columbia University’s Irving Institute for Clinical and Translational Research, North Carolina Translational and Clinical Sciences Institute). An informal environmental scan (i.e., internet search, discussions with other D&I colleagues) revealed that several efforts to catalog D&I resources already existed, such as the Dissemination and Implementation Models in Health [13] website and Washington University in St Louis’ D&I Research Toolkits website [14]. Rather than duplicating these efforts, the WG elected to create a central resource guiding investigators (especially those new to D&I) to existing resource catalogs. The WG then began to develop a set of inclusion criteria for the resource catalogs, and explore options for hosting, updating, and disseminating the Compendium, as described below.
Development of inclusion criteria
To identify what types of resource catalogs should be included and create a framework for categorizing catalogs, the sub-working group first reviewed a list of example catalogs suggested by WG members. Using this list of exemplar catalogs, with the goal of enhancing access to rigorously-developed and relevant catalogs, the sub-working group developed a working list of inclusion criteria, then sought input and clarification from the WG on these criteria. According to the final set of Compendium catalog inclusion criteria, catalogs needed to be (a) publicly available or available via free membership, (b) curated/sponsored by an academic or non-profit organization with expertise in D&I, (c) appropriate for an academic research audience, and (d) actively maintained and updated; catalogs also needed to (e) contain multiple individual resources that (f) appeared to have been systematically gathered and organized according to a specific methodology (e.g., organized by themes or topic areas to allow for sorting by a user, rather than simply creating a list). Catalogs were not required to be specific to the United States (US) or hosted by a US institution to be eligible for inclusion. Catalogs sponsored by for-profit entities were not included, to avoid concerns about endorsement of specific for-profit enterprises.
Initial catalog solicitation, screening, and categorization
After establishing the initial set of criteria, the sub-working group asked during a WG meeting (with a follow-up email) that the WG members suggest relevant resource catalogs to the WG, resulting in an initial screening list of potential resource catalogs. After screening for the inclusion criteria above, 35 resource catalogs remained for inclusion in the Compendium. The sub-working group then used an adapted content analysis [Reference Prior15] approach in which sub-working group members assessed and described topics covered and purpose of each catalog and then collectively discussed common topic groupings to develop an initial list of categories to organize the resource catalogs contained in the Compendium. After reviewing and refining these categories with the WG, the final list of categories included “Comprehensive Resources,” “Funding Opportunities and Resources,” “Methods and Measures,” “Theories, Models and Frameworks,” “Practice Resources,” Training and Workforce Development Resources,” “Health Equity and DEI,” and “CTSA Infrastructure and Processes.” These categories are not mutually exclusive; a resource catalog may be tagged with multiple categories.
To make initial engagement with the Compendium simpler for users, the team selected 13 comprehensive and/or beginner-friendly resource catalogs to display as “featured.” Featured catalogs (up to three within each category) are expected to change over time to avoid bias towards any particular institution’s resources.
Technology, hosting, and suggestion form
To maximize its use and sustainability, the sub-working group identified several desired features for the platform that would be used to host the Compendium: open access, no hosting costs, ability to filter by category or tags, ability to identify featured resource catalogs, ability to add, edit or remove resource catalogs as needed, and ability to be either linked and embedded depending on CTSA preference. Several hosting options were evaluated, including hosting the Compendium on a central website hosted by the Center for Leading Innovation and Collaboration (CLIC), the former coordinating center for the CTSA Program. Ultimately the sub-working group chose a combined approach in which the Compendium was created and hosted using the Airtable™ platform, and a page was also created for the Compendium CTSAs within the CLIC resource library. Airtable™ is a free online collaboration tool which uses a spreadsheet/database format to store information, and offers gallery-style “public views” that can be searched, filtered, sorted, and easily embedded on websites. CTSAs were provided with both the embeddable Airtable™ gallery and the link to the CLIC resource library page, and invited to choose their preferred implementation option.
Additionally, a resource catalog suggestion form was created (see Supplemental File 1) to allow others within the translational and D&I communities to submit additional resource catalogs for possible inclusion. To submit a catalog, the submitter provides contact information, a description of the catalog, suggested categories for the catalog, and affirms that the catalog meets each of the criteria for inclusion. Form responses are periodically reviewed by two members of the sub-working group, who then respond to the submitter to confirm that the suggested resource catalog has been added to the Compendium, explain why the suggested resource catalog doesn’t meet the inclusion criteria, or request more information as needed to make a decision.
Implementation guidance and dissemination
The sub-working group created a short website implementation guidance document (see Supplemental File 2) to assist CTSAs in embedding the Compendium on their institutional websites. The guidance document included background information about the Compendium, technical instructions for embedding the Airtable™ gallery, and suggested language to accompany the Airtable™ gallery on CTSA institutional websites, which included a link to the resource suggestion form. The guidance document also included an example of the Airtable™ gallery integrated into a CTSA website and contact information for additional questions or support.
Results
Compendium characteristics
The CTSA Compendium of D&I Resource Catalogs (Figure 1) is a searchable, filterable database of (at launch) 35 catalogs that are central to advancing the science and impact of D&I research and practice, across eight (non-exclusive) domains: comprehensive resources (n = 9); theories, models, and frameworks (n = 10); methods and measures (n = 18); funding opportunities and resources (n = 6); practice resources (n = 6); training and workforce development resources (n = 12); CTSA infrastructure and processes (n = 1); and health equity and diversity, equity and inclusion (DEI) (4). Most included catalogs are open-access and sponsored by academic institutions. To facilitate navigation, catalogs can be filtered by category, interface type (interactive tool, static website, etc), audience, open access, and designation as a “featured resource.” Table 1 summarizes the characteristics of included catalogs; each catalog entry also includes information about the hosting institution and a description of the purpose and scope of the catalog (either provided on the resource suggestion form, or collected directly from each catalog’s website during the initial population of the Compendium).
Figure 1. Compendium resources at initial launch (January 2022).
Table 1. Characteristics of included catalogs
* Diversity, Equity and Inclusion.
† Clinical and Translational Science Awards.
‡ Portable Document Format.
Compendium initial implementation
The Compendium was initially launched to the members of the WG in July 2022; in October 2022 volunteers were solicited from the WG to serve as “early adopters” of the Compendium. These early adopters (n = 8) agreed to implement the Compendium on their institutional website(s) either by embedding the Airtable™ gallery directly or by linking to it, and to share their experiences with the implementation process and guidance document with the sub-working group. Early adopters received a follow up email providing the implementation guidance document (see Supplemental File 2) and contact information for support.
In July 2022, the Compendium was added to the “Research & Operational Tools” resource library maintained by the Center for Leading Innovation and Collaboration (CLIC), the previous coordinating center for the CTSA Program.
In January 2023, the sub-working group conducted an internet search to identify institutions who had implemented the Compendium, either by embedding the Airtable™ gallery directly or by linking to a version of the Compendium hosted elsewhere. Nine institutions were identified by this web search, including five institutions from the volunteer “early adopter” group, the three host institutions for the sub-working group, and one additional institution. This represents 14% of the overall CTSA D&I working group, where 64 CTSA institutions are represented. A follow-up search in March 2024 identified a total of 16 institutions implementing the Compendium, representing 25% of the overall D&I working group. Table 2 provides further implementation details. See Figure 2 for examples of website implementation of the Compendium.
Figure 2. Compendium web implementation examples from (a) Irving Institute for Clinical and Translational Research, (b) Colorado Clinical and Translational Sciences Institute, and (c) North Carolina Translational and Clinical Sciences Institute.
Table 2. Compendium implementation characteristics
* These details unknown for one site due to firewall restrictions.
Early adopter survey
In December 2022, the sub-working group conducted a survey (see Supplemental File 3) of the Compendium’s “early adopters” to learn more about their experiences and perception of the implementation process and the value of the resource to their institutions. The survey included items asking if the Compendium had been implemented, any barriers to implementation, where it had been implemented (including whether it was a CTSA website), how easy or hard it was to implement, perceived value of the centralized process for dynamically updating the Compendium, whether the Compendium is worth the effort to implement, and open-ended feedback. All CTSA institutions participating in the national D&I WG were invited in October 2022 to participate as “early adopters;” eight WG members accepted this invitation and agreed to implement the Compendium of their institutional website(s), and received the “early adopter” experience survey. Four respondents (50%) had implemented the Compendium on at least one institutional website; all of these respondents described the implementation process as “easy.” All respondents (50%) who had not yet implemented the Compendium indicated that they planned to implement it, citing web development capacity concerns (75%) and competing CTSA institutional priorities (25%) as reasons for the delay. (A follow-up internet search in March 2024 confirmed that these institutions did ultimately implement the Compendium.)
All respondents said the dynamically-updated aspect of the Compendium was “valuable” or “very valuable,” and 75% of respondents indicated that this resource was worth the effort needed to implement it on their institutional websites (with 25% of respondents indicating they weren’t sure yet about this – these respondents also had not yet implemented the resource).
Catalog submission review
The resource suggestion form was launched in May 2022 to the full D&I working group; it became publicly available at the launch of the Compendium in July 2022. Between May 2022 and March 2024, a total of nine new resources were submitted for possible inclusion in the Compendium. Of these resources, four were added to the Compendium after review by the sub-working group, four were determined not to meet all criteria set out in the suggestion form, and one was placed on hold pending a publicly available link.
Discussion
To make D&I science and practice methods more accessible and usable, a wide range of resources and tools have been created; however, despite this growth in information available, it can be challenging and overwhelming for researchers to navigate or be aware of all available resources, or to know which are most up to date or actively maintained with the latest guidance. For example, D&I frameworks such as the Consolidated Framework for Implementation Research (CFIR) [16], RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) [17], Exploration, Preparation, Implementation, Sustainment (EPIS) [18] and the Translational Science Benefits Model (TSBM) [19] have their own websites with extensive guidance on their application. To aid researchers in selecting among the many theories, models, and frameworks, tools such as the Dissemination and Implementation Models in Health [13] webtool and the Theory, Model, and Framework Comparison and Selection Tool (T-CaST) [Reference Birken, Rohweder and Powell20] have been developed. D&I measures have been cataloged by the Society for Implementation Research Consortium [21]; engagement and co-design methods have been cataloged in the Engagement Methods Navigator webtool [22,Reference Kwan, Ytell and Coors23]. The Center for Implementation (TCI) Toolbox [24] includes a number of resources related to adaptations, equity, implementation support practitioners, and engagement and co-creation. Implementation strategies have been cataloged and systematically organized by the Expert Recommendations in Implementing Change project team [Reference Powell, Waltz and Chinman25]. These are just a handful of the resource catalogs represented in the Compendium.
The sources of Compendium catalogs ranged from individual academic institutions (often D&I units associated with CTSA hubs), national funders, collaborating academic networks, professional societies, and non-profit organizations. Host institutions of included catalogs were largely US-based, reflecting the solicitation of resource catalogs primarily thru CTSA consortium channels; however, included catalogs did feature a variety of individual international resources (including knowledge translation resources from Canada and the United Kingdom). While most catalogs were in the form of websites, some were white papers or peer-reviewed journal publications. While most catalogs represent D&I science methods resources, some are oriented toward sharing tools and resources for implementing evidence-based practice. For instance, the Rural Health Information Hub’s Tools for Success includes a robust set of 25 toolkits for implementation of evidence-based practice in rural health settings [26]. Table 1 summarizes characteristics of included resource catalogs.
Several categories in the Compendium had very few resource catalogs. There are only 3 catalogs included in the health equity category, of which two are annotated bibliographies and one is a blog post discussing D&I frameworks useful for addressing health equity and links to a series of readings. There was only one catalog related to D&I CTSA Infrastructure and Processes [Reference Shelton, Dolor and Tobin11]. While more extensive catalogs of resources in these domains may exist, they have not yet surfaced for inclusion in the Compendium. There are potentially resources available within these categories but they have not yet been cataloged as a unique set; some health equity resources are included in the more “comprehensive D&I” resource catalogs. It was purposefully outside the scope of the Compendium to include individual resources that could not reasonably be conceptualized as a catalog, in the interests of creating a short, easy to navigate tool to explore the many existing catalogs (and avoiding duplicating the work of the many catalogs that already exist). At times we balanced inclusiveness with strict adherence to the catalog definition. Throughout ongoing WG activities, the Compendium will be continuously refined to prioritize catalogs that meet the initial inclusion criteria. In parallel, regular (annual or semi-annual) reviews by the Compendium manager (a D&I program manager employed by the primary hosting CTSA), who manually checks that the catalogs are still actively maintained and links are still active. To maintain low costs, the Compendium’s Airtable™ platform is administratively managed by one CTSA program, and is reviewed and updated in partnership with the current CTSA D&I-focused working group where necessary; this takes only a few hours per year. The curated nature of the Compendium is an important feature of its intended usability.
Early indications are that CTSAs are willing and able to implement the Compendium either by directly embedding it within institutional websites, or by linking to an externally-hosted version; more evaluation is needed to assess the functionality, usefulness, and impact on D&I science capacity. The Airtable™ platform selected for the Compendium has pros and cons. While it is free, very easy to update as a basic spreadsheet on the backend, and simple to embed in a website with only a few lines of HTML code, it has limited options for customizable user interfaces. As a collaborative effort of the national CTSA consortium, with few dedicated resources beyond a small portion of personnel time for CTSA-supported working group members and NCATS’ encouragement to collaborate across the consortium, the Compendium is largely a volunteer effort with no budget. Thus we rely upon the broader D&I and clinical and translational science community to develop and submit new and updated catalogs to the Compendium. Should the Compendium become a consistently valued resource, there may be opportunities for fundraising for a more sophisticated platform.
Limitations
Because the Compendium is structured as an embeddable web resource, no overall web traffic metrics are available, limiting the ability to centrally assess either number of users/visitors, or user behavior while using the resource. Additionally, the limited customization ability of the free platform used to develop the Compendium constrains options to create a more user-friendly interface or invite user feedback via web surveys. As noted above, the Compendium is currently a volunteer effort with no budget, but in the event that funds become available to support a paid hosting solution, user metrics will be a key measure of the success of the Compendium, as well as end user feedback through website survey invitations.
We did not conduct a formal systematic review of all potential D&I resource catalogs; collaboration with library scientists may identify more catalogs that could be included.
Future directions
The CTSA “Advancing Dissemination and Implementation Sciences in CTSAs” Working Group (WG) reconvened in 2023, including a sub-working group focusing on continuing to expand and disseminate the Compendium. This sub-working group aims to expand the Compendium with an emphasis on equity, practice-based research networks and practitioner resources, improve the usability of the Compendium interface using user testing methods and by developing additional practical guidance for end users, and enhance adoption of the Compendium across the CTSA consortium. CTSAs and other institutions interested in D&I capacity building are encouraged to incorporate the Compendium into their institutional websites and promote its use in D&I mentoring, consults, and collaboration. It may be used by D&I leaders and educators to guide creation of local D&I resource lists that are most relevant to the needs of their learners, mentees, and partners and most aligned with their theoretical or practical approaches to D&I. This centralized tool is meant to enhance efficiencies and reduce the need for local D&I leaders to do their own searches for resources or create their own catalogs, enabling them to spend their time supporting their teams and advancing the field of D&I science and practice.
Conclusions
Understanding more about the development, usage and perceived value of D&I science cataloging efforts can facilitate more strategic resource development by CTSAs striving to meet the rising demand for D&I expertise. To be successful, centralized efforts to catalog and organize the myriad D&I resources available online should consider the local barriers and perceived benefits to implementing and navigating CTSA network resources.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/cts.2025.10087.
Acknowledgements
The authors wish to acknowledge the contributions of the Clinical & Translational Science Awards (CTSA) Program’s “Advancing Dissemination and Implementation Sciences in CTSAs” Working Group in providing expert validation throughout the process of developing the Compendium, and the valuable input received from volunteer “early adopters” within this group.
Author contributions
Lindsay Lennox: Conceptualization, Data curation, Formal analysis, Methodology, Project administration, Resources, Writing-original draft, Writing-review & editing; Rachel Shelton: Conceptualization, Data curation, Formal analysis, Methodology, Writing-original draft, Writing-review & editing; Catherine Rohweder: Conceptualization, Data curation, Methodology, Writing-original draft; Bethany Kwan: Conceptualization, Data curation, Formal analysis, Methodology, Project administration, Resources, Supervision, Writing-original draft, Writing-review & editing.
Funding statement
This work was supported by NIH Grants UL1 TR002535, UM1 TR004399, UL1TR001873 and UM1TR004406. Additionally, this work was supported in part by the University of Rochester Center for Leading Innovation and Collaboration (CLIC), under Grant U24TR002260. Contents are the authors’ sole responsibility and do not necessarily represent official NIH views.
Competing interests
The authors have no conflicts of interest to declare.
Open access
