3.1 Participant Characteristics

Table 1 summarises the socio-demographic characteristics of the participants. Twenty researchers participated in this study, 17 in the three-step test interviews and three in the additional user interviews. Participants were mainly based in the Netherlands (n = 17, 85%), with three from Thailand (15%). Ages ranged from 24 to 57 years (median = 31). Eighteen participants identified as female (90%), and two as male (10%). The three-step test interviews lasted between 42 and 111 min (median = 61), whilst the additional interviews lasted between 14 and 27 min (median = 18).

Table 1 Participant characteristics

Amongst the three-step test interview participants, most were early-career researchers with a master’s degree (n = 12, 71%) and limited methodological experience. All participants had either previously participated in or were preparing to participate in economic evaluations. At the time of the interview, the majority reported being beginners or unfamiliar with R (82%) and economic evaluations (88%). The three additional interviewees were early- to mid-career researchers from the Netherlands. Their familiarity with R ranged from beginner to expert (one beginner and two advanced/expert users), and their familiarity with economic evaluations was advanced or expert.

3.2 Qualitative Synthesis

The thematic analysis generated three themes describing the researchers’ experiences of using the methodological tutorial, either during the interview or in their own research: (1) ‘Getting in and getting it to run’, (2) ‘Understanding and making sense of what tutorials teach’ and (3) ‘Adapting and applying tutorials in practice’. Each theme, along with its subthemes, is described in detail below. Supporting illustrative quotations are presented in Table 2. Figure 1 shows an overview of the identified barriers and facilitators across the three themes.

Table 2 Supporting illustrative quotesFig. 1

Overview of factors influencing the implementation of methodological tutorials

3.3 Theme 1: Getting in and Getting it to Run

This theme describes the practical and technical challenges participants faced before beginning the tutorial exercise.

3.3.1 Installation, Setup and Access

Participants described installation and setup as one of the most challenging aspects of the tutorial. Many found installing the required software (R and RStudio) time-consuming and relied on internet searches for guidance. Accessing materials also proved difficult: participants encountered issues unzipping files, identifying the correct project file and opening it in RStudio. Some participants missed instructions and opened individual files instead of the R project as intended, leading to errors. Even after accessing materials, several felt ‘lost’ navigating the R interface, and basic actions such as installing or checking packages were unclear to many.

To overcome these challenges, participants used various strategies. Some relied on generative artificial intelligence (AI) tools (e.g. ChatGPT) and prior course materials to remember basic commands. Others suggested tutorials should include links to introductory resources or describe minimal software basics (e.g. in an appendix). Prior coding experience helped some participants get started, whilst complete beginners found the environment intimidating.

3.3.2 Running the Code and Handling Issues

Participants expressed uncertainty when running tutorial code, as they were unsure if code chunks executed correctly, whether the script was still running or how long execution should take. Beginners found long runtimes confusing or frustrating and wanted guidance on checking code execution, whilst experienced users viewed long runtimes as normal, depending on computer performance, analysis complexity and dataset size.

Managing errors and warnings was another recurring challenge. Participants reported that error and warning messages were difficult to interpret. In several cases, participants continued running the code without addressing the errors or warnings. Other participants, drawing on experience with similar software, viewed such messages as routine. Several participants described relying on external support to resolve issues, including colleagues with statistical expertise or generative AI tools, which they found useful for diagnosing and solving error messages. A ‘frequently asked questions’ section was also suggested to help users troubleshoot common issues.

3.3.3 Structural Clarity, Consistency and Accuracy of Tutorial Materials

Participants emphasised the importance of clear, consistent and well organised tutorial materials. Several reported difficulties locating the appendix or understanding the role of the supplementary materials and some were even unsure whether different sources (article, appendix, online repository) provided identical information or complementary content. Minor discrepancies between the code and manuscript, such as typographical errors, further contributed to uncertainty and reduced confidence in the material.

At the same time, participants appreciated the strong alignment between the published paper and code. Numbered code chunks and a structure that closely mirrored the manuscript’s flow enabled users to follow both sources simultaneously and better understand how each step contributed to the overall analysis. When reflecting on how structural clarity could be improved, some participants suggested that visual aids, such as screenshots of the software environment, would have been helpful, whereas others preferred concise written instructions. Participants also raised concerns about the long-term accuracy of tutorial materials. They noted that updates to software or packages could cause the code to stop working, underscoring the need to host content that is likely to change over time in an online repository that can be updated.

3.4 Theme 2: Understanding and Making Sense of What Tutorials Teach

This theme addresses challenges participants faced in understanding tutorial content, both in language, methodological explanations and interpreting outputs.

3.4.1 Clarity and Comprehensibility of Tutorial Content

Participants emphasised the need for clear language and accessible explanations. Most found the terminology appropriate for those experienced in economic evaluation and/or statistics, but some struggled with jargon, unclear abbreviations or technical instructions (e.g. utility score, Quality-Adjusted Life Years, bootstrapping). They searched for definitions and suggested including a glossary of technical terms or references to background materials.

Participants also wanted clearer explanations of what the code was doing and why. They appreciated the justification of methodological choices, but some were unsure about the purpose of certain code chunks or functions. They preferred ‘base’ code over package functions because it made operations explicit rather than relying on ‘black box’ functions. Suggested improvements included adding comments within the code and placing detailed methodological explanations, such as statistical procedures and formulas, in an appendix.

3.4.2 Trusting the Process

Participants generally trusted the tutorial materials. Many felt comfortable proceeding if the code ran without errors, viewing this as an indication that the code had executed correctly. Others relied on trust in the software or authors when unable to verify each step.

Visual comparison was particularly reassuring: matching outputs to the figures or values presented in the tutorial increased confidence. When applying the code to their own data, participants indicated they would cross-check results using familiar software (e.g. STATA), consult statisticians or colleagues or use generative AI tools. Seeing that others had similar issues would also be reassuring.

3.4.3 Finding and Interpreting Outputs

Participants had mixed experiences locating and interpreting tutorial outputs. Some could not find relevant results in the console; others struggled to identify where plots or statistics were shown. Clearer guidance on output locations or saving results would have been appreciated.

Confidence in interpreting results varied. Participants with experience in economic evaluation and/or statistics felt comfortable, whilst others needed more clarification. Several noted that detailed instructions on interpretation may fall outside a single tutorial’s scope and would best be learned through coursework or external resources. When uncertain, participants used internet searches or generative AI tools to understand or interpret outputs.

3.5 Theme 3: Adapting and Applying Tutorials in Practice

This theme covers participants’ views on applying tutorial materials in practice, adapting code to their own data and considering for whom tutorials are suitable.

3.5.1 Adapting the Code to One’s Own Dataset and Needs

Participants expressed mixed confidence in adapting tutorial code to their data. Some felt equipped to attempt adaptation, whilst others would doubt the trustworthiness of outputs after modification. Some were prepared to adapt selected parts of the script; others would prefer adjusting their dataset to avoid changing the code.

Adapting the code was seen as more demanding than completing the tutorial. Modification required basic coding skills, time and the ability to verify correctness, all factors that could be challenging for beginners. The use of package functions was also viewed as a barrier because they are harder to modify or inspect.

Participants identified support systems that could facilitate adaptation: assistance from statisticians or colleagues, prior statistical programming experience and generative AI tools. They suggested providing both function-based and explicit ‘base’ code versions, plus examples of how to extend the workflow to different types of datasets.

3.5.2 Who Tutorials Are—and Are Not—For

Participants considered who is best suited to use and adapt methodological tutorials. Many felt that any motivated researcher could follow them, even without formal training in economic evaluations and/or statistics, if willing to invest time in self-study. Complete beginners may struggle without support, as adaptation requires some coding and statistical skills. Participants noted that researchers with prior experience in epidemiology and/or biostatistics would find the tutorial more accessible, whereas applied clinical researchers may risk making errors if working entirely independently. Overall, motivation, available time and willingness to engage with background material were seen as key factors to using methodological tutorials effectively.

3.5.3 Role of Tutorials in Learning and Applied Research Practice

Participants described methodological tutorials in general as practical and efficient learning tools, especially as statistical methods become more complex. They valued tutorials for providing clear, standardised workflows and code structures adaptable to their research and as a useful starting point for learning. Participants reported learning new methods or concepts, noting that deeper understanding required further reading or practice, as expected when using tutorials. Tutorials were also viewed as beneficial for both supervisors and students. Those who had applied the tutorial in their own work stressed that tutorials represent substantial work and should be properly cited, as they meaningfully contribute to research practice.