Participants and Setting

This study was conducted with 10 participants at a university-based autism center in a rural-serving southern region in the United States. The study was approved by the university’s Human Research Protection Program review board. The autism center provided grant-funded ABA services across a variety of settings, including in-home (via telehealth), in community and employment settings, in public schools and college campuses, and in the center’s on-campus clinic. All participants were behavior technicians, instructors, or BCBAs who were actively engaged in ABA practice across the various service lines within the organization.

Table 1 summarizes participant demographics. Most of the participants had between two and four years of experience practicing ABA, although two participants had less than or equal to one year of experience. Four participants were full-time BCBAs at the center. Of these, two were enrolled in doctoral coursework in special education and one held a doctoral degree in special education. Two of the four BCBAs had been employed at the autism center for less than three months and had prior supervised research experience in ABA outside of the center. The remaining two had been employed at the autism center for over two years; one had minimal prior research engagement, whereas the other had previous mentored research experience at the center. One participant was an uncredentialed, full-time instructor in the employment services program with a doctoral degree related to curriculum design and minimal research engagement at the center.

Table 1 Participant demographics

Five participants were part-time registered behavior technicians (RBTs) who had worked at the center for less than three months with no research engagement. One RBT was an undergraduate student, while the remaining four RBTs were enrolled in master’s-level ABA graduate coursework and were actively accruing BCBA Supervision Fieldwork hours. All graduate-level participants were part-time Research Assistants who allocated 10 h per week to service delivery and 10 h per week to research-related tasks as directed by their faculty supervisor. This project was initiated, in part, because so few clinical staff had research experience, and research engagement was required as part of their roles. Because attendance at group supervision was a required component of training at the autism center (see “Group Supervision”), this study qualified as research conducted in an established educational setting. Participants were not informed of the study.

Three faculty supervisors, two female and one male, coordinated and facilitated the group supervision meetings. Faculty supervisors held ranks of assistant, associate, and full professor in the university’s Department of Special Education and autism center. Their university-level teaching experience was 1.5, 14, and 22 years, respectively. Their experience conducting practice-based research ranged from 7 to 22 years across clinical, school, and/or community settings. Collectively, they had 91 peer-reviewed publications. At least two faculty supervisors attended each meeting. The first and second author alternated leading meeting discussions and creating supplemental materials (e.g., handouts, slideshow), and all supervisors in attendance participated in the group discussion. Outside of group supervision, each graduate student trainee (doctoral and master’s) met individually with their designated faculty supervisor(s) at least twice per month for 30 min to one hour to receive feedback and guidance on research products (e.g., manuscripts, posters). Faculty supervisor assignments for graduate students were determined at the time of hiring, prior to the study, based on service location (e.g., clinic or school service supervisor). Students were encouraged to lead and develop one research project that was relevant to, or could be conducted within, their service line in the organization.

Group supervision meetings were conducted in a large meeting room within the autism center containing 10 tables with two chairs each, one large television screen, and one mobile whiteboard on wheels. Participants were seated in self-selected pairs, and tables were arranged in a U shape with the television screen in the opening between tables to better facilitate discussion between participants. A total of six, 90-min meetings were conducted over nine weeks. Meetings were held no more than once per week. Some weeks were skipped due to holidays, with no more than three weeks between meetings.

Materials

Each participant was provided with a copy of Applied Behavior Analysis Research Made Easy: A Handbook for Practitioners Conducting Research Post-Certification (Valentino, 2022; hereafter, handbook). Participants were encouraged to write or make notes in their personal copies. Faculty supervisors created and disseminated a weekly schedule of topics and assigned readings to all participants.

During group meetings, faculty supervisors prepared a PowerPoint slideshow with accompanying text and images that aligned with the assigned handbook chapter and discussion points. Participants were asked to bring their handbooks and writing utensils to the meetings. Participants were permitted, but not required, to use their laptops for note taking or reviewing the presentation slides on their personal devices. The whiteboard and marker were used for some activities (Table 2). No additional handouts or readings were provided during the nine weeks.

Table 2 Supervision schedule and examples, questions, and activities per weekProcedure

Two faculty supervisors reviewed the handbook to identify the reading assignments for each week. We did not deviate from the order in which chapters were presented in the handbook, and one to two chapters were assigned for each meeting. Each week, faculty supervisors asked participants to read assigned chapters and complete the interactive pages of the handbook to the best of their ability prior to the group meeting. We did not conduct formal checks of handbook pages for completion.

Pre- and Post-assessment

In an introductory meeting, the faculty supervisors met with participants to review the group supervision expectations and weekly schedule. At the end of the meeting, participants were asked to complete the pre-assessment survey (for details of the assessment, see the section under “Data Collection and Analysis, Likert-Scale Assessment”; see Table 3 for the list of assessment questions) using a link that was disseminated via Quick Response (QR) code and email. One email reminder to complete the pre-assessment was sent three days prior to the first group meeting.

Table 3 All survey questions with follow-up questionsGroup Supervision

The supervision curriculum was structured to align with the handbook (Valentino, 2022), which provided the framework to guide weekly individual group supervision meeting topics, activities, and discussions. The group supervision meeting followed a consistent format of: ABA faculty supervisors presented a brief 10 to 15-min review of the topic(s) with embedded discussion followed then by 30 to 45 min of activities. The remaining meeting time was devoted to client-specific discussion and feedback. Activities included small and large group discussions, demonstrations of target skills, research interest and skill mapping, and individual reflections with prompts (see Table 2 for a list of all discussion prompts, examples, and activities by week).

Most discussion prompts were delivered in a think-pair-share (TPS) model. TPS is a large-group instructional approach that (a) gives participants a discussion prompt, (b) provides time (e.g., 1–5 min) for participants to think about their response individually, (c) pairs participants up to share what they thought about with one other participant, and (d) asks participants to return to the large group to share their thoughts and perspectives. Prior research on TPS in educational settings shows that participants like having time to think about their response to prompts prior to group discussion (e.g., Guenther & Abbott, 2024). In addition, participants may feel more comfortable participating in group discussion once they have had the opportunity to share their perspectives with one other person.

Research skill demonstrations were selected on the basis of their alignment with the reading content and included how to (a) use the university library website to search for literature, (b) use a citation management tool (e.g., Zotero.com), and (c) use a publication outlet locator (e.g., Springer Nature, n.d.). Two group activities were designed to help participants identify possible research collaborators and/or mentors within the organization. The first involved individuals writing research topics they were interested in on a whiteboard (e.g., parent training) and asking other participants to write their name next to topics that they were also interested in. The second activity involved presenting a list of research-specific tasks (e.g., reading, material organization, data collection, graphing, writing, oral presentations) and asking participants to write their names next to skills for which they thought they could be a mentor. Participants were provided with a photo of each finished activity for reference.

Prior to the study, the same group met for group supervision with the same frequency and similar format—discussion followed by clinical data review—except with different assigned readings determined by group input on relevant clinical topics and without TPS or activities. Group supervision meetings, including those in this study occurred during paid work hours, and attendance was mandatory for all participants prior to and during the study. The center did not provide clinical services one day per week so that employees could prioritize group supervision attendance, professional development, research, and administrative tasks.

Experimental Design

We used a mixed-methods design to evaluate how the six-week group supervision experience influenced participants’ self-reported understanding and capability in conducting practice-based ABA research. A one-group pre-assessment–post-assessment design was used to analyze Likert-scale responses. Participants completed a pre-assessment prior to the group meetings, and a post-assessment following the completion of six-week supervision experience. Depending on the participant’s answer, some Likert-scale ratings had associated follow-up questions where participants were asked to elaborate on their answer in an optional, short-answer response. This design allowed for the examination of changes in participant scores from pre- to post-assessment. Additionally, we analyzed their short-answer responses using two approaches: accuracy check (quantitative) and thematic analysis (qualitative) for both pre- and post-assessments.

Data Collection and AnalysisQuantitative Analysis Likert-Scale Assessment

The primary dependent variable was participants’ Likert-scale ratings of their research understanding and capabilities on the self-reported assessment. The self-reported assessment was administered via Qualtrics, an online survey tool, and e-mailed to each participant one week prior to the first group meeting and the afternoon following the final group meeting. The assessment contained 17 Likert-type questions developed by the faculty supervisors using Valentino (2022; Table 3). The questions were designed after reading the entire handbook and selecting areas of foundational understanding and capabilities that may have improved by reading and participating in the group supervision meetings. The response options for all questions included: (1) strongly disagree, (2) disagree, (3) neither agree nor disagree, (4) agree, and (5) strongly agree.

Accuracy Check

If participants selected “agree” or “strongly agree” for some assessment items, the survey automatically presented a follow-up question that was a free-text response (see Table 3 for Likert and follow-up questions). Participant responses to follow-up questions 1, 3, and 10 were checked for accuracy to evaluate whether participants could generate a free-text response that supported their Likert rating. To complete the accuracy check, responses to the follow-up questions for questions 1, 3, and 10 were scored as either correct or incorrect. See Table 4 for definitions of correct and incorrect responses by question.

Table 4 Definitions of correct and incorrect responses for accuracy checks

A faculty supervisor read and scored each response for accuracy using the operational definitions of correct and incorrect. A second, independent faculty supervisor scored 100% of the responses. We compared primary and secondary scoring results for exact agreement. An agreement was counted if both independent scorers marked the participant’s response as either correct or incorrect. Mismatches in scoring were counted as disagreements. Exact agreement for correct and incorrect responses was 94%.

Research Products

The third dependent variable was the cumulative number of research products. We counted the cumulative number of conference research posters or manuscript submissions to peer-reviewed journals for all participants during and three months following the completion of the supervision experience. This included any submission that was rejected or pending a decision at the time of this writing. A research product was counted if one or more participants were included as authors in any position. That is, we did not double-count research products if more than one participant was listed as an author. The same project could be counted twice if it resulted in more than one product (e.g., manuscript and a poster).

Qualitative Analysis Thematic Analysis

If participants selected “agree” or “strongly agree” for some assessment items, the survey automatically presented a follow-up question that was a free-text response asking them to elaborate on their answer (see Table 3 for Likert and follow-up questions). Participant responses to follow-up questions on items 5–10, 13, and 15 were thematically analyzed to evaluate how participants’ responses changed pre- and post-group supervision. Different from the accuracy check, these follow-up questions were aimed at understanding participant-specific interests and/or capabilities.

We conducted the thematic analysis using Naeem et al. (2023) and Nowell et al. (2017) as frameworks for reviewing text responses, searching for and developing codes, transforming codes into defined themes, and analyzing changes in response themes on pre- and post-assessments. One faculty investigator and research assistant read all short answer responses. The faculty investigator identified keywords for common ideas identified throughout the responses; similar keywords were inductively categorized and defined to develop codes (see Tables 7–12 in Supplemental Materials). Each open-ended response was coded using the theme(s) in the structured codebook. Subsequently, a research assistant with expertise in thematic analysis examined patterns across these code families to identify higher-order, cross-cutting themes, rather than restricting themes to individual questions, to reflect patterned meaning in participants’ overall practice-based research orientation. We grouped codes related to feasibility constraints, concrete research steps, and accomplishments into one broader theme (research capacity and progress). Codes related to research benefits and, in what ways, were also grouped into a second broader theme (impact orientation: who benefits and how). Finally, we grouped codes that describe research-related competencies and mentorship activities into a final broader theme (mentorship readiness and support needs; see “Results”). We used pre- and post-assessment comparisons to descriptively examine changes in code frequency in our dataset, rather than to deduce stable shifts in participants’ underlying perspectives. This cross-question thematic approach was selected because many of the responses were brief and some items were unanswered.

One faculty investigator independently read and coded 100% of participant short-answer responses as related to one or more coded themes. A second faculty investigator coded 88% of participant responses. Interrater agreement was calculated using exact agreement by code occurrence across pre- and post-test responses. Exact agreement was calculated by dividing the number of code applications on which both coders agreed by the total number of code applications scored by at least one coder and multiplying by 100. Exact agreement was 96.8% for the pre-test and 91.2% for the post-test.

Content Review of Research Products

We conducted a content review of research products to report on the breadth of research topics across participant submissions. To do this, participants, submitted the titles, abstracts, and key words of any research manuscripts or posters, submitted during or in the three months following the group supervision experience. One faculty investigator reviewed the titles, abstracts, and key words of participant research products to develop themes. Topics were categorized into themes, and each research poster and manuscript was coded with one or more of those themes.

Social Validity

We evaluated the social validity of the handbook and supervision experience through open-ended questions embedded into the post-assessment survey. Participants were asked to rate their overall satisfaction from zero (highly unsatisfied) to five (highly satisfied) and answer three open-ended questions (see Table 6 for open-ended questions).