Prior to beginning the study, we submitted a determination application, which outlined our proposed study, to an institutional review board (IRB). After reviewing the application, the IRB concluded that the study did not require IRB human subjects review.Footnote 5 The data collectors were four undergraduate psychology students and two graduate students in a Master’s ABA program. The first author trained each data collector via Zoom using behavioral skills training until each data collector could independently display all of the search steps as outlined in Fig. 1. Thereafter, the first author met weekly with data collectors and provided booster training sessions as needed. The current study evaluated publications by keynote speakers and invited speakers near or prior to the speaker’s presentation date. For example, if an individual presented in 2021 and data collectors identified a topic match in a peer-reviewed article published in 2022, data collectors counted the article as a match with the presentation topic for that presenter. This approach allowed for studies that may have been under peer review when speakers presented.

Flowchart of review process for keynote speakers and invited speakers

For the initial phase of data collection, the six data collectors acquired publicly available information about keynote and other invited presentations at state ABA conferences. Thereafter, they gathered information about the keynote and other invited speakers’ publication history and, when necessary (due to inadequate publication records), information about their alternative experiences, such as professional and practice history. Data collection began in April 2023 for all ABAI-affiliated state chapter conferences in 2021, 2022, and 2023, and the study concluded in April 2024. The official ABAI website listed 54Footnote 6 affiliated regional and state chapters. The state chapters were selected for this study. The profile contained a link to the individual websites of each chapter. Because we opted to evaluate conferences at the individual state level, we did not evaluate several regional conferences (i.e., Berkshire Association for Behavior Analysis and Therapy, Four Corners ABA, Southeastern ABA, Mid-American ABA, and Hoosier ABA). We evaluated keynote and invited speakers for conferences that occurred during the years 2021, 2022, and 2023, with the exception of when the organization did not list both the titles of presentations and the speaker’s names online. In addition, we attempted to evaluate speakers at Behavior Association of Michigan, Delaware ABA, District of Colombia ABA, Lone Star ABA, Oklahoma ABA, Philadelphia Metropolitan ABA, and West Virginia Behavior Analysis Association, but these chapters did not host conferences during the study period. After applying the aforementioned criteria, the analyses included keynote speakers and invited speakers from 42 organizations.

Similar to the methods used by Richling et al. (2014), data collectors extracted information about the keynote and invited presenters from the conference- and organization-related newsletters, web pages, social media pages and posts, and virtual programs. The information needed to complete our evaluation included the (a) names of the keynote and invited speakers, (b) availability of CEUs for the keynote and invited presentations, (c) keynote and invited speaker certification level (i.e., BCBA or BCBA-D), and (d) respective presentation titles. Each ABAI-affiliated chapter website or social media account typically contained information on the conference speakers, keywords to search for more information about their annual conference, or both. When information about conference speakers was not publicly available, data collectors emailed board members who were listed on the respective conference websites for the conference programs. Data collectors also used the BACB Certificant Registry, which publicly lists current and expired certifications, to determine (or otherwise verify) (a) whether each speaker was a BCBA or BCBA-D and (b) the year each speaker was first certified.

If speakers gave multiple presentations at a given conference, data collectors counted each CEU opportunity separately. It was possible for a given keynote or invited presenter to be an expert on one presentation topic and have no expertise on another topic at the same conference. In the event that one presentation featured two keynote or invited presenters, data collectors included only the speaker with the most expertise (described below). We excluded invited speaker panels, workshops, and invited presentations that did not provide CEUs. In five cases, the conference program contained language that implied (e.g., by referring to “featured” speakers) but did not clearly specify whether their speakers were, in fact, invited presenters. We opted to include these speakers as invited presenters in our evaluation.

Figure 1 outlined the general process data collectors followed for evaluating keynote and invited presentations. The primary dependent measures were each speaker’s number of publications on their presentation topic and each speaker’s overall publication count. Dixon et al. (2015a) evaluated faculty members’ research productivity by counting their publications in behavior-analytic journals using Google Scholar. For the current study, data collectors used Google Scholar to collect data for publications in any peer-reviewed outlet. When necessary, data collectors used ResearchGate to obtain additional information on the speaker (e.g., their affiliation) to confirm that they had evaluated the correct individual from the corresponding conferences.

Each keynote and invited speaker’s presentation title and abstract typically provided enough information to determine keywords regarding the main topic of their CEU presentation. If data collectors could not find the description, they searched Google for the title of the presentation. In most cases, they found an abstract if the speaker (a) provided online CEU content, (b) repeated the presentation at another conference, or both. As a hypothetical example, if they collected data on a presentation titled “Assessment and Treatment of Adolescent Problem Behavior Maintained by Escape,” some keywords could have been “escape” and “problem behavior.” If the abstract mentioned “functional analyses” and “antecedent manipulations,” data collectors would have included these as keywords to refine the search in future steps. In order to provide a liberal evaluation, the publications could have differed from the conference presentation in areas such as title, behavior, design, and procedure. For example, if the speaker presented on feeding disorders, all peer-reviewed publications detailing feeding in the abstract or title counted as “a publication on the topic.”

To determine the number of publications on the presentation topic, data collectors read each title and abstract of the peer-reviewed articles listed on Google Scholar and then manually searched for matches with the presentation topic. If one or more keywords matched a publication and the presenter was the author or a co-author of the article, the data collectors counted that publication toward the speaker’s expertise for that presentation. Data collectors discontinued this search process after they found more than 10 publications on the topic by the speaker. Data collectors then used Google Scholar to determine the total number of publications by the speaker. Conversely, when Google Scholar yielded no articles authored or co-authored by the presenter, data collectors coded the presenter as having (a) no topic-specific published articles and (b) no overall publications.

If the initial search process did not yield any topic-specific published articles for a speaker, data collectors searched for alternative experiences on LinkedIn™ and official university websites. We collected data on this broad variable to understand why organizations selected speakers without scientific expertise, as evidenced through publishing, as keynote or invited speakers. If the presenter worked at a university, the university website typically contained their curriculum vitae. Data collectors scored alternative experiences that pertained to the presentation topic, such as (a) authoring books/chapters, newsletters, podcasts, and blogs; (b) clinical and personal experiences (e.g., founder of a company, personal diagnoses, parent of child with diagnoses, volunteer work listed on LinkedIn); and (c) practicing in a related discipline (e.g., speech pathologist, clinical psychologist, lawyer). Specifically, data collectors scored the personal experience category when a presenter self-reported being autistic (in the conference program), and the diagnosis was pertinent to the topic of the keynote or invited presentation. For the other experiences, data collectors used the same liberal criteria described above for matching presentations to publications.

For all keynote and invited speakers, data collectors assigned the topic of their presentation to one of the 13 ABAI conference presentation categories (see also Kangas & Vaidya, 2007). As shown in Fig. 1 (bottom), researchers combined ABAI’s broad topics to make eight categories for the current study: Autism/Developmental Disabilities (AUT/DDA), Applied Animal Behavior (AAB), Experimental Analysis of Behavior (EAB), Community, Social, and Sustainability Issues/Philosophical, Conceptual, and Historical Issues (CSS/PCH), Teaching Behavior Analysis/ Organizational Behavior Management (TBA/OBM), Clinical, Family, and Behavioral Medicine/Behavior Pharmacology and Neuroscience (CBM/BPN), Behavioral Development/Verbal Behavior (DEV/VRB), and Education (EDC). Researchers combined ABAI topic categories on the basis of overlapping content of categories, a low count of presentations in two similar categories, or both.

We assessed intercoder agreement using procedures similar to those described by Richling et al. (2014). Two data collectors independently coded 34% of the variables on the keynote and invited speakers. Researchers calculated intercoder agreement scores using the average agreementFootnote 7 across presenters for topic-specific publications and total publication count, separately. For each presenter, we divided the smaller topic-specific publication count by the larger topic-specific publication count. We then summed up the values across presenters, divided that value by 250 (the number of observations with a secondary observer), and multiplied the quotient by 100%. We repeated this process for the total publication count with each presenter. Researchers used an exact agreement method for CEU availability, speaker affiliation, presentation title, ABAI topic categorization, and the number of alternative sources. Between the two data collectors, each variable had to contain point-to-point correspondence. Researchers calculated exact agreement scores for each dependent measure by dividing the number of agreements by the number of disagreements plus agreements and multiplying the quotient by 100%. The average intercoder agreement scores for topic-specific publications and total publication count were 83% (range, 43–100%) and 93% (20–100%), respectively. The intercoder agreement scores for affiliation, title, topic categorization, and number of alternative sources were 100%, 100%, 90% (75–100%), and 87% (35–100%), respectively.

After data collectors completed the process described above, we assigned the invited speakers a “level of expertise” rating based on the number of publications that were specific to their presentation topic. We categorized speakers with 0 topic-specific publications as having no scientific expertise on that topic. In addition, we categorized speakers with 1–3, 4–6, and 7–9 topic-specific publications as having low, moderate, and high-moderate expertise, respectively. Notably, if a speaker had only one topic-specific publication, we speculated (but did not verify) that it was likely to be a study they conducted as their thesis or dissertation. To this end, it is likely that the speaker demonstrated their knowledge of their topic to their respective committee members. Moreover, Kranak and Onofrio (2025) noted that multiple organizations used the same keynote and invited speakers within and across conferences and argued that this practice could impede the professional development of junior faculty. For these reasons, we did not view having only one topic-relevant presentation as an inherent problem for a keynote or invited speaker. Finally, we categorized speakers with 10 or more topic-specific publications as having strong expertise on their presentation topic.