BACKGROUND

Nearly one-third of people with HIV (PWH) in New York City (NYC) are not retained in HIV care and are at increased risk for morbidity, mortality, and onward HIV transmission.1,2 “Data to Care” projects use demographic, clinical, and administrative data from different sources to facilitate health care re-engagement efforts by outreach specialists, including care coordination teams, community-based organizations (CBOs), and public health field workers.3 Increasingly, Data to Care information is sourced from state and national health information exchanges (HIEs). Data to Care projects using HIEs have been applied across a diverse range of health conditions, including those supporting PWH.4–7 In February 2019, several diverse NYC HIV service organizations and evaluators collaborated to implement the Bottom Up Data to Care Project (Bottom Up or the Project). A bottom-up design integrates disparate systems to meet local needs or policy goals including conducting community outreach.8 The Bottom Up Project uses HIE tools, such as real-time care encounter alerts, to locate and stimulate peer support outreach by a CBO to assist PWH to return to medical care.

Bottom Up incurred multiple challenges in its pilot phase from February 2019 to June 2021, which are common in implementation9,10 although COVID-19 was unique in magnitude and impact.11 Despite the challenges, the Project responded and maintained activities, adapted, and matured over the period. We used qualitative thematic and process analysis, informed by implementation science and a novel analytic methodology to apply concepts from resilience scholarship to explore implementation during a time of health care system stress. In this paper, we seek to, first, describe our novel analytic conceptual approach and methodology using the Consolidated Framework for Implementation Research (CFIR) and resilience. We then share findings related to how various factors and resilience processes including adaptive and ongoing implementation led to a matured pilot project and continued collaboration across partner organizations. We also consider whether the findings provide insights into how implementation actually works in complex, emergent HIV and related interventions more broadly.

CONCEPTUAL APPROACH

Our conceptual approach combines the CFIR and a novel resilience framework (RF) within qualitative methodology that amalgamates resilience concepts found in the literature12 (Fig. 1).

FIGURE 1.:

Bottom Up conceptual approach to incorporate CFIR and RF.

CFIR is a widely used9,13 framework of multiple factors shown to positively or negatively influence implementation across different settings. Factors are structured into domains: (1) intervention characteristics, (2) outer setting, (3) inner setting, (4) characteristics of individuals, and (5) process. We chose CFIR for its broad taxonomy across different social–ecological levels to characterize barriers or challenges to Bottom Up implementation and facilitating or responsive efforts to implement. Going forward in this article, we will use “challenges” and “responses” as preferred terms, the former due to the varied issues the Project faced and the latter to connote its proactive nature. Although helpful to pinpoint specific and multidimensional factors, CFIR does not take account of adaptations to interventions or participating organizations. CFIR is also limited in not offering guidance on how to assess the relative influence of different challenges and how practices change as a result of varying influences.14

Scholarship in the field of resilience explores how individuals, organizations, and systems dynamically respond to adverse events or challenges. There are 3 dimensions to the proposed RF. The first categorizes challenges or adversities as either crisis or chronic. Crisis challenges are unexpected, rapid in onset, and shocking, such as public health emergencies like COVID-19, rapid escalations of violence, and natural disasters causing disruption due to their overwhelming nature. Chronic challenges are persistent, repeated, somewhat predictable, longer term, and stressful such as funding shortfalls, staffing changes, and chronic data management problems.15 Chronic challenges seem as more common and benign, but disruption over a longer term may be significant.

A second RF dimension characterizes responses as absorbing, adapting, and transforming. Absorptive capacity is the ability to react by coping or weathering a challenge and, ultimately, resuming stable activity. A metaphor for this might be powering through a rainstorm without protection. Adaptive capacities entail making active, ideally effective adjustments. Here, an umbrella protects. Transformative capabilities introduce a deep and potentially lasting change in structural ways,16 such as reading a forecast to avoid rain. Challenges (chronic/crisis) and responses (absorbing/adapting/transforming) interact and show how projects reckon with complexity, uncertainty, and emergence.17 For example, under certain circumstances, as challenges increase in intensity or are weathered for longer periods of time, techniques such as absorbing and adapting may become insufficient; understanding these contexts versus ones where strategies can work over the longer term reveal the qualities and circumstances driving an initiative's potential.

A third RF dimension identifies organizational factors influencing resilience. Barasa et al15 define 9 factors, from access to resources to human capital, preparedness, planning, information management, redundancy/finding alternative strategies, governance (improvisation, deliberation, and inclusion), leadership, organizational culture, and collaborative networks. Characteristics highly overlap with CFIR, although CFIR brings in a wider set of outer factors and resilience emphasizes organizational improvisation and flexibility.

Bringing CFIR and RF into 1 analytic frame contributes to implementation science and fills a gap in the resilience literature.18 This novel analytic methodology describes what projects experience, including on-going unpredictable and volatile conditions, and how they respond to them. This approach draws upon and contributes to advancing the design and conduct of process evaluations19 used in implementation science to examine factors and processes influencing implementation at formative and later stages. The proposed addition of resilience to CFIR may be especially helpful to studying implementation processes before formal evaluations with stable and measurable processes and outcomes. Furthermore, understanding implementation as a complex of actions and reactions that can be specifically characterized (ie, absorbing, adapting, and transforming), can also support identifying levers to drive a more stable and thus measurable program over time, and contribute to the aspiration of implementation science to close the research to practice gap in a timely fashion.

METHODS Project Settings

Bottom Up involves the collaboration of 6 organizations: (1) an HIV Clinic in NYC providing primary and supportive services to more than 2400 PWH within a large academic medical center; (2) a CBO serving 6000+ clients providing peer support and navigation for medical and mental health care, harm reduction, housing, treatment adherence support, and other services; (3) a HIE based in NYC and Long Island (NY HIE), connecting 8000+ NY area participating hospitals, behavioral health organizations, and others with information for more than 20 million individuals; (4) a supportive housing agency providing assistance and benefits coordination to 38,000 public assistance–eligible PWH in NYC (representing 44% of PWH in NYC)20; (5) an Academic Partner as project evaluators aiming to translate research into sustainable, cost-effective population-level interventions, strategies and policies; and (6) a Technical Assistance Partner supporting Bottom Up as one of several NY-based pilot projects.

Bottom Up Project Description

Bottom Up implementation pilot phase from February 2019 to June 2021 consists of 6 steps, and a preceding HIE consent activity that includes, in brief, creating and sharing a “lost to follow-up” (LTFU) of HIV clinic clients, activating HIE alerts for clients, and coordinating with a CBO to find the clients and re-engage them in medical care (Fig. 2 and see Supplement 1, Supplemental Digital Content, https://links.lww.com/QAI/C147).

FIGURE 2.:

Bottom Up Project description steps.

Data Sources and Collection Procedures

One Academic Partner member conducted semistructured interviews in May 2020 with leadership or frontline implementation staff from participating organizations to understand project components and implementation (N = 6: 2 HIV Clinic, 1 HIE, 2 CBOs). Interviews, audio-recorded and conducted remotely through video using a topic guide, lasted approximately 30–60 minutes. Interview topics, drawn from CFIR-related constructs, included information about the organization and role in the Project; start up and training; implementation processes, barriers, and facilitators; and perceived outcomes or benefits. Interviews were transcribed by a third party company. We conducted documents review of Bottom Up planning minutes and project protocols (N = 35) from February 2019 to June 2021, the conclusion of the evaluative pilot funding period. Planning meetings over video mostly occurred bi-weekly and typically lasted 1 hour. On average, 7 persons representing 1–2 members per organization attended. Technical Assistance Partner or Academic Partner members generated minutes. The study protocol was reviewed and received exemption by the Institutional Review Board at City University of New York Graduate School for Public Health and Health Policy.

Analysis Procedures

Two Academic Partner members initially coded challenges and organizational responses from each data source (interviews and documents) by Bottom Up step and placed extractions into a summary template created in Excel. The template was structured using a priori CFIR and RF constructs to characterize challenges and responses, and included dates to track changes over time (see Supplement 2, Supplemental Digital Content, https://links.lww.com/QAI/C148). Authors ABL and DA reviewed, discussed, and agreed upon CFIR and RF construct applications. We reviewed for patterns based on frequencies of constructs and used analytic memos to identify and comparatively summarize how Bottom Up responded to challenges and adapted over time by confronting the mix of stressors. The larger team that included implementation-oriented team members reviewed the coding and interpretation for member-checking, insight generation, and write-up. Our analysis draws from rapid research by using summary templates and regular team feedback to inform program development.21

RESULTS

The results are presented in 2 sections. Using CFIR and RF, the first describes the challenges at each project step and the second presents the Bottom Up response and factors facilitating them.

Bottom Up Challenges by Step

The Project experienced a total of 20 major challenges, with a range of 2–5 challenges at each prestep/step. Most arose in 2020, 1 year after the launch. At the prestep, obtaining consent from clients for information sharing among implementers was difficult because of the COVID-19 pandemic and switches in electronic medical record systems. At Step 1, the generation of the LTFU list was time-consuming and funding losses reduced staff dedicated to its generation. At Steps 2 and 3, HIE alerts were not sent to project staff or were incompatible with workflows (eg, came in at night). At Step 4, client engagement was difficult due to funding reductions and COVID-19, which halted in-person outreach and staff availability. At Step 5, re-engagement with the medical system was significantly hampered by COVID-19; with reduced hospital capacity, the HIV clinic was less able to schedule rapid appointments and not all clients accessed telemedicine. At Step 6, data management challenges included a rigid data entry system and limited staff training. A few challenges resolved and then presented as challenges again (Fig. 3).

FIGURE 3.:

Bottom Up Project challenges, responses, resolutions, and resilience characteristics.

Challenges Described Through CFIR and RF Constructs CFIR Challenges

Relevant CFIR domain constructs are presented as Bottom Up challenges in Table 1. The most common was the intervention's complexity and associated execution, evident in the involvement of several organizations, each of which and in their coordination relied upon technology, data management, and active client engagement. The complexity of creating the initial LTFU list was vividly described by an HIV Clinic Project Manager:

I would say one of the main challenges was kind of navigating through the different databases to find all this information…client specific information, like housing information, the client's address, sat in one table and then all the client's visit [information] sat in another table, a client's lab information sits in another table. You have all these different tables on the back end of the database…Even diagnosis codes, you would think it would be simple and all the diagnosis codes would exist in one table, but it's not the case.

TABLE 1. - CFIR Barriers and Facilitators by Bottom Up Project Step

Available resources within each of the participating organizations (CFIR's inner setting) limited human capital to perform at several Bottom Up steps (LTFU list generation and outreach to clients), again underscoring complexity. Available resources also included existing organizational data management systems, which proved limited, a “major challenge” explained by one Bottom Up lead, given the “changing needs of the organization.”

External factors such as policy changes created challenges arising at 4 Bottom Up steps. The invalidation of HIE consents resulted from changes in interorganizational systems and legal issues. COVID-19 policies at state and health system levels also increased appointment scheduling barriers and halted typical outreach efforts, as described by the CBO lead:

..the model [Bottom Up] is based on a very high touch, in-person connection from the community-based side. COVID required us, as a community organization, to transition very rapidly from in-person to remote work. We had very little time to prepare for it and we did not have all of the technology that we needed in order to do it successfully.

RF Chronic and Crisis Types

Both chronic and crisis challenges affected Bottom Up implementation (Fig. 3). Chronic challenges arose at each step as evidenced by the persistence and recurrence of technology and information management issues, barriers to client engagement, and the impact of limited resources such as reduced staffing on activity execution. Crisis challenges, denoted by outer disruptions, also limited Bottom Up activities. They arose at some but not every step (1, 2, 4, and 5) and included COVID-19 restrictions, vulnerability of staff to COVID-19, limited medical system access, and rapid funding losses. Unexpected funding shortfalls had the potential to exacerbate chronic staffing pressures, as a CBO lead described:

So [like] any intervention, we had some peers that had some challenges or barriers, so we had a couple of turnovers with our peers. Then we had something happen with our funding and we lost one of the full-time staff that was on the project.

Bottom Up Project Responses

Across Bottom Up's 20 challenges came 18 problem-solving responses aimed to address the issues in meaningful if not resolvable ways (Fig. 3). Six responses led to documented resolutions by June 2021, including IT fixes, securing funding to redress previous reductions and expand staffing, and developing effective outreach strategies during the COVID-19 pandemic to protect staff vulnerable to severe illness by substituting leadership to do outreach. The challenges without a response were associated with chronic issues inherent in the more difficult aspects of the Project, such as client engagement and effectively leveraging real-time HIE alerts.

Response to Challenges by CFIR and RF Constructs CFIR Facilitators

Constructs from each CFIR domain present as facilitation factors supporting responses to challenges (Table 1). The most prominent that helped address implementation barriers are leadership engagement (inner setting), project adaptability (intervention characteristics), and cosmopolitanism, defined in CFIR as “the degree to which an organization is networked with other external organizations” (outer setting). Leaders from participating organizations consistently presented at meetings to discuss problems and develop solutions. The meeting format effectively addressed IT issues. Adaptability characterized various changes to components. For example, 1 staff member said of leaders who took up outreach efforts as peers stepped back due to COVID-19 vulnerability:

… our supervisors roll up their sleeves and get in the streets if need be, and help us to locate clients. So, we have a lot of support at the Outreach CBO. Nobody is a big I and a little U, which I really, truly appreciate.

Shifting data tracking and limiting the LTFU list and distribution based on capacity were other alternative methods of performing project components in a limited but effective manner. Repeatedly using their knowledge network to address problems demonstrated cosmopolitanism. Partners used their decades of experience within health care and community fields to offer guidance to one another, such as sharing consenting practices. They shared information as equal partners with an appreciation for the respective organizational and leadership strengths.

RF Response Types

Project staff demonstrated all 3 types of resilience responses (Fig. 3). Absorbing came up 5 times in the prestep and 3 Bottom Up steps (1, 3, and 4) as responses that did not directly address the challenge but coped with them by passively not responding (eg, “no documented response”) or actively weathering (eg, “suspended practice until COVID-19 abated”). As a result, no responses of this type led to a documented solution. Those occurring during the prestep indicate the limited power of Bottom Up to address legal and regulatory challenges. Other challenges that were weathered rather than actively addressed (waiting for funding, contending with a high burden of outreach) resulted in some productivity loss such as fewer clients outreached.

Adaptations, the most common of resilience types defined as adjustments or attempted fixes occurring in response to 14 challenges, arose at every step in reaction to both chronic and crisis challenges. They include sharing “best practices” between organizations to improve consent, lowering LTFU client numbers and adjusting schedules to list-make, brainstorming how to improve alerts, developing protocols to respond to COVID-19 challenges, and adjusting data trackers. Adaptations variably resulted in resolution; effective strategies included setting up additional calls to fix technical problems and modifying efforts to complete tasks. Adaptations that did not result in resolution included issues with consent procedures and some technical difficulties (eg, incorrect alert firing) from data and HIE systems.

Factors influencing adaptive resilience required finding “redundant or collateral pathways” to execute the same tasks in different ways. For example, implementers made an alternative “flat file” at Step 2 to transfer CBO consents to HIE. Organizational leadership presented another adaptation trait. For example, when list-making took too much of the HIV clinic leader's time, he shifted his schedule and said in an interview, “I became smarter about scheduling time at the end of every month specifically to do the LTFU list curation and pushed other things out of the way, and I prioritized that list review and curation process.” Finally, adaptation responses drew upon leadership practices and the collaborative network to share insights and develop strategies. As another team member said:

This was a very unique project in my opinion because all of the major stakeholders participated in calls to see how they could help, how they could improve, and what they thought best practices was. Because I've been on a lot of other projects, but I've never been on a project that brought everybody who served play a part together so consistently. Not quarterly, not annually, but literally every week or every other week for months.

Transformation, the least frequent response occurring 3 times, is characterized as larger structural or organizational changes to Bottom Up and the involved organizations. The first transformation pertained to the securing of new funding to improve staffing capacity. This important development allowed the program to maintain or resume activities. The second transformation introduced a new information management system to track outreach, engagement, and alignment at step 6 by the CBO, thereby shifting its ability to manage data. The CBO leader said the new system was “an investment and [also indicated] a challenge.” Such profound system-level changes signal organizational leadership and culture, preparation and planning, and evolving appreciation for information management to advance the Project.

DISCUSSION

Overall, Bottom Up contended with challenges of chronic and crisis types, associated with the project's intervention complexity, constrained organizational resources, and the external policy environment. The Project used different resilience mechanisms to respond. Absorptive strategies demonstrate adjusting to challenges, such as receiving but not acting on the rapid firing of real-time alerts. Adaptations illustrate creativity and tactical know-how, to offer quick fixes such as sharing a “flat file,” as well as more radical intervention such as leaders taking over and delivering services for vulnerable staff. Transformative strategies positively disrupted the challenges by offering new opportunities for the Project and also the involved organizations. CFIR and resilience both point to the project's ability to draw upon specific organizational resources and organizational leadership, and a project design capable of strategic adaptations in an ever-evolving process.

Bottom Up responses also reveal the effects of chronic versus crisis challenges. COVID-19 and funding losses were significantly disruptive crisis-oriented challenges. Rather than paralyze the Project, it tethered them to specific and concrete activities and, in some cases, led to resolutions. At each step, the Project deployed resilience responses (absorbing and adapting) to lessen the challenge intensity. Thus, even the most crisis-like challenges led to practical and effective action.22 While action-oriented, some crisis-associated challenges extended implementation and the time to stabilize the program, illustrated by the impact of COVID-19 on the consent and accompanying nonresolution. If the implementation timeline had been stricter, COVID-19 may have been more disabling.

Chronic, prolonged, and persistent challenges elicited all 3 resilience responses. The literature15 suggests that chronic stressors can diminish a project or organization over time. The combination of chronic stressors and absorptive mechanisms, such as ongoing staff fatigue as a result of under-funding, might under certain conditions irreversibly exhaust a project. Bottom Up offers a different case, in which chronic challenges seemed to reveal the nonessential activities within each step; the Project either contended with (ie, absorb), adapted, or jettisoned activities. It was flexible to down- or upshift its activities as conditions allowed. For example, reducing the LTFU list to be manageable downshifted the rapidity of finding clients, but it fit organizational capacities. Similar to Greenhalgh et al description of hard cores and soft peripheries,23 Bottom Up distinguished between and pursued “must do” (list-making) from “nice to do” (the number of clients on the list). Bottom Up revealed its ability to experiment with scale and to assess priorities, conducting an efficient trade-off between must do and nice to do given its complexity. Moreover, the process also tolerated a high degree of nonresolution of challenges (a total of 13) and used a technique whereby the team selected which challenges to mitigate. Of those it weathered rather than addressed, there were consequences such as reaching fewer clients in an expedited fashion.

In addition to downshifting and pragmatically trading off program ideals to fit context, the central role of leadership commitment supported experimentation and knowing how to optimize the learning network (in CFIR terminology, “cosmopolitanism”). Leadership is generally important to implementation as noted in Barasa and CFIR frameworks, but this analysis shows how; leaders used savvy to strategically absorb or adapt, particularly to navigate in a regulated yet fluctuating environment. Their commitments and decision-making status, combined with a deep understanding of the operational conditions and the Project's functional components, supported tailoring and effective strategies to move forward.

Transformative responses further suggest that by confronting challenges, projects grow by altering the involved organizations. There were 2 notable transformations within the responses: the acquisition of a sizable resource to build up activity and acquiring a more robust data system to produce information to monitor the project. Factors influenced adaptations and transformations alike (ie, leadership, organizational culture and access to material resource, and project adaptability/collateral elements). Transformations took time to come to fruition; trial-and-error and adaptive strategies came first such as trying to track data or get by with fewer staff and later the CBO and HIV Clinic arrived at transformative opportunities. Importantly, transformation responses went beyond the Project to change organization conduct with implications for broader management. They signaled a “proactive” stance, which has been identified as harder for organizations to do but with more potential to become not only a learning initiative, but a learning organization.24

Combining resilience and CFIR brought needed nuance to understand the “how” of implementation in dynamic and “real-world” situations. Resilience surfaced the heterogeneity of responses over time, and the dynamic situations giving rise to them. Reflecting on the broader implementation literature, the active responses may be characterized by some of the well-known strategies defined as methods or techniques to improve implementation, including program funding, sharing knowledge, “network weaving,” and conducting ongoing facilitation-oriented problem-solving.25 These strategies have also been used in implementation projects in the HIV field to address known and emergent barriers and facilitators.26 Our analysis goes further to emphasize the dynamics surrounding their use and potential discontinuation, as well as in some cases, offer greater detail about the strategies themselves. For example, the “downshifting” response illuminates how scaling as an implementation strategy can be nonlinear and bidirectional, expanding, and contracting in response to the conditions. Organizational transformation responses also went beyond the identified strategies, suggesting the need to consider how interventions have environmental/organizational effects. These insights would not have emerged through CFIR alone, nor by only identifying the presence of recognized implementation strategies, although such characteristics and broad and multilevel contexts help explain factors enabling or restricting the Project. Organizations implementing projects and initiatives should consider typologizing their efforts using both RF and CFIR and consider whether responses link to known implementation tactics to explore project journeys, planned and emergent responses that best pace the process of reaching project stabilization.

Our study has limitations. First, it contained only 2 data sources: 1 time interviews and documents. The timing of semistructured interviews, occurring in the middle of the pilot period, may not have provided opportunity to fully reflect on emerging challenges and responses that came up later in the implementation found in the documents. Meeting minutes may also have captured more challenges than responses. The case study format also limits cross-case learning; methods should be expanded to other cases and then tested across geographies and intervention settings. Finally, clients were not interviewed about their implementation perspectives, another useful information source that should be expanded upon in future studies.

CONCLUSIONS

Bottom Up, particularly adept at scaling to the situation, matured by consistently involving leadership in a shared learning process of project design review and responsive and adaptive implementation to contend with and overcome challenges. Such insights are useful in understanding the Project's processes of change and growth. This learning should be applied to build a broader understanding of project implementation to better understand how project growth and maturation occur (or do not) and can be harnessed toward greater insights and success. Tracking early project implementation also supports being able to arrive at the point of conducting rigorous and thorough mixed methods process and outcomes evaluations.

ACKNOWLEDGMENTS

The authors extend their gratitude to Bottom Up Project clients, the Linkage to Care staff and Peer Specialists at Alliance for Positive Change, Terriell Peters, Fulvia Alvelo, Maaz Siddiqui, Tom Moore, Naitik Patel, and Kathy Mills. The authors thank the Einstein-Rockefeller-CUNY Center for AIDS Research, the New York City Council, and the CUNY Institute for Implementation Science in Population Health (ISPH) for providing funding support and staff time.

REFERENCES 1. ETE Dashboard. HIV care continuum—people living with HIV: New York city. Available at: https://etedashboardny.org/data/prevalence-and-care/hiv-care-cascades-nyc/. Accessed January 26, 2023. 2. Cohen MS, Chen YQ, McCauley M, et al. Prevention of HIV-1 infection with early antiretroviral therapy. New Engl J Med. 2011;365:493–505. 3. Irvine MK, Chamberlin SA, Robbins RS, et al. Improvements in HIV care engagement and viral load suppression following enrollment in a comprehensive HIV care coordination program. Clin Infect Dis. 2015;60:298–310. 4. Sharp J, Angert CD, McConnell T, et al. Health information exchange: a novel re-linkage intervention in an urban health system. Open Forum Infect Dis. 2019;6:ofz402. 5. Magnus M, Herwehe J, Gruber D, et al. Improved HIV-related outcomes associated with implementation of a novel public health information exchange. Int J Med Inform. 2012;81:e30–e38. 6. Bove JM, Golden MR, Dhanireddy S, et al. Outcomes of a clinic-based surveillance-informed intervention to relink patients to HIV care. J Acquir Immune Defic Syndr. 2015;70:262–268. 7. Sweeney P, DiNenno EA, Flores SA, et al. HIV data to care—using public health data to improve HIV care and prevention. J Acquir Immune Defic Syndr. 2019;82(suppl 1):S1–S5. 8. Coiera E. Building a national health IT system from the middle out. J Am Med Inform Assoc. 2009;16:271–273. 9. Kirk MA, Kelley C, Yankey N, et al. A systematic review of the use of the consolidated framework for implementation research. Implement Sci. 2016;11:72. 10. Addison D, Baim-Lance A, Suchman L, et al. Factors influencing the successful implementation of HIV linkage and retention interventions in healthcare agencies across New York state. AIDS Behav. 2019;23(suppl 1):105–114. 11. Brazier E, Ajeh R, Maruri F, et al. Service delivery challenges in HIV care during the first year of the COVID-19 pandemic: results from a site assessment survey across the global IeDEA consortium. J Int AIDS Soc. 2022;25:e26036. 12. Ware NC. Qualitative contributions to implementation research on HIV prevention and treatment. J Acquir Immune Defic Syndr. 2019;82(suppl 3):S217–S221. 13. Damschroder LJ, Reardon CM, Widerquist MAO, et al. The updated consolidated framework for implementation research based on user feedback. Implement Sci. 2022;17:75. 14. Varsi C, Ekstedt M, Gammon D, et al. Using the consolidated framework for implementation research to identify barriers and facilitators for the implementation of an internet-based patient-provider communication service in five settings: a qualitative study. J Med Internet Res. 2015;17:e262. 15. Barasa E, Mbau R, Gilson L. What is resilience and how can it Be nurtured? A systematic review of empirical literature on organizational resilience. Int J Health Policy Manag. 2018;7:491–503. 16. Blanchet K, Nam SL, Ramalingam B, et al. Governance and capacity to manage resilience of health systems: towards a new conceptual framework. Int J Health Policy Manag. 2017;6:431–435. 17. Braithwaite J, Churruca K, Long JC, et al. When complexity science meets implementation science: a theoretical and empirical analysis of systems change. BMC Med. 2018;16:63. 18. Gilson L, Barasa E, Nxumalo N, et al. Everyday resilience in district health systems: emerging insights from the front lines in Kenya and South Africa. BMJ Glob Health. 2017;2:e000224. 19. Moore GF, Audrey S, Barker M, et al. Process evaluation of complex interventions: medical research council guidance. BMJ. 2015;350:h1258. 20. New York City Department of Health and Mental Hygiene Epidemiology Program. HIV Surveillance Annual Report, 2021. HIV Epidemiology Program; http://www.nyc.gov/assets/doh/downloads/pdf/dires/hiv-surveillance-annualreport-2021.pdf (2022). 2022. 21. Vindrola-Padros C, Johnson GA. Rapid techniques in qualitative research: a critical review of the literature. Qual Health Res. 2020;30:1596–1604. 22. Beaunoyer E, Dupéré S, Guitton MJ. COVID-19 and digital inequalities: reciprocal impacts and mitigation strategies. Comput Hum Behav. 2020;111:106424. 23. Greenhalgh T, Robert G, Macfarlane F, et al. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82:581–629. 24. Moore JE, Bumbarger BK, Cooper BR. Examining adaptations of evidence-based programs in natural contexts. J Prim Prev. 2013;34:147–161. 25. Powell BJ, Waltz TJ, Chinman MJ, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10:21. 26. Zulkiewicz BA, Burrus O, Harshbarger C, et al. Identifying implementation strategies that address barriers and facilitate implementation of digital interventions in HIV primary care settings: results from the pilot implementation of positive health check. AIDS Behav. 2021;25:154–166.