The instrument development process described below follows the activities related to ECD and infuses CBPR principles at each step. For example, during domain analysis, we developed a list of constructs and made construct maps in partnership with the Kenyan research team, our community advisory board, a group of village elders, teachers, and caregivers of children with autism from Eldoret, Kenya. Similarly, we collected evidence of what beliefs, attitudes, opinions, or behaviors are associated with the constructs of interest by having Kenyan team members brainstorm with members of the advisory board, all with relatively little involvement from Western researchers. The survey items emerged from that brainstorming work. Following the survey item development, our Kenyan partners conducted cognitive interviews with a subset of the advisory panel to ensure the items elicited the information they were intended to capture. We detail these steps in the ECD process that blends CBPR principles in the next sections, with a primary focus on the first two stages of ECD (domain analysis and tool design/testing). A visual for this process is provided in Fig. 1.

Fig. 1The alternative text for this image may have been generated using AI.

Instrument development process

Step 1. Construct Development (Domain Analysis)Initial Construct Identification

Building from our team’s prior work and knowledge of the literature in autism education in Kenya and in LMICs, our international team met to identify initial constructs to be measured with this survey. By jointly developing initial constructs to be measured, we focused on CBPR principles of recognizing the local community as a unit and on identifying and prioritizing community-driven issues (Minkler et al., 2003). In developing the Autism in the Context of Education—Kenya Survey (ACE-KS), we sought to explore knowledge and beliefs about autism from the perspectives of caregivers and educators. Initial constructs identified by the team included (a) knowledge of causes of autism, (b) beliefs about educational and life outcomes for autistic people, (c) knowledge of access to behavioral and educational services, and (d) beliefs regarding how autistic children should be taught and cared for. These initial constructs were then refined through a review of the literature, a review of existing measures, and feedback from the community advisory panel. We detail that literature review in the supplemental materials. However, we found very few instruments related to autism/NDD developed specifically for sub-Saharan Africa, and none developed specifically for Kenya (Bakare et al. 2008, 2009).

Initial Team Interviews/Discussion

Following the review of existing measures, the first and second authors revised the constructs and shared those with the team for feedback and revision during weekly videoconference meetings held via Zoom. This CBPR process of iterative refinement of constructs supported participation and bidirectional learning from all team members and allowed us to focus more clearly on the community-defined constructs of interest. During our weekly meetings, each construct was presented, and team members provided feedback and refinements to the construct with a focus on cultural relevance and appropriateness, and potential interpretation by Kenyan respondents. Team members offered suggestions on refining constructs, expanding constructs, and distinguishing potential overlapping constructs. Weekly team discussion continued for three weeks until the team reached consensus on the following constructs: (a) knowledge of autism and causes, (b) knowledge of the diagnostic process, (c) beliefs related to optimal outcomes for children with autism, (d) beliefs about how autistic children should be taught, (e) beliefs about stigma and attitudes toward people with autism, and (f) how caregivers should care for autistic children.

Construct Mapping

Next, the team created a construct map for each construct of interest. A construct map is a theory-driven and thoroughly researched ordering of qualitatively different levels of a given construct, focusing on one characteristic at a time (Wilson, 2009). At each level, the researcher might identify the attitudes, perceptions, beliefs, or knowledge of respondents that are unique to that level. In simple terms, a construct map defines what is to be measured in terms that are general enough to be interpretable to the public (including those taking the survey), but specific enough to guide the development of the instrument. While the idea of a construct map stems from the field of achievement testing, it is also employed in survey design (e.g., Maul et al. 2008; Nuansri et al., 2016).

Creating a construct map yields many benefits in the instrument design process, detailed by Wilson (2009). First, the map can help ensure the construct is defined succinctly, including in terms of a developmental progression on that construct. Second, and related, a construct map can provide insights into dimensionality. Concepts that are too complex to be captured straightforwardly in a single map can be broken apart into separate constructs and, therefore, maps. Third, the map can be used to streamline the item-writing process. Specifically, once attitudes and behaviors at each qualitative level are articulated, items can be written to match those attitudes and behaviors. Fourth, one can ensure the items adequately cover the range of the construct by ensuring items correspond to every level of the map. (See Supplemental Materials for two of our construct maps). Finally, once data are collected, one can check that the rank ordering of items along the map matches the rank ordering of items based on the proportion of actual respondents endorsing each item. Example construct maps for two of our constructs can be found in the supplemental materials.

The initial construct map was shared by the first author with the team and refined over the course of two Zoom meetings with Kenyan collaborators. Revisions focused on cultural adaptation of language. For example, the word “construct” was viewed by the team as being too abstract for future survey respondents and was replaced with the term “ideas.” Other adaptations to language included reducing jargon and technical terms. For example, the construct of “optimal outcomes for children with autism” was changed to “best possible life for children with autism.” The construct map was further culturally adapted along the dimensions of metaphors and content (Bernal et al., 1995) to align the constructs and behaviors with cultural values, customs, and practices. Specifically, the construct of causes of autism underwent significant revision to not only encompass the wide range of local beliefs about disability etiology, but also to ensure respondents would not feel judged for selecting behaviors at the extreme ends of the continuum. The revised construct map was then shared with the Kenyan community advisory board to finalize construct development and construct mapping. The construct maps provided a blueprint for the CBPR process of brainstorming beliefs, attitudes, and behaviors associated with levels of each construct without constraining our community member participants in terms of the constructs being measured or what defines them.

Step 2. Instrument Design (Tool Design and Testing)Community Advisory Board Convening

The collaboration with a community advisory board was essential for our approach to blending CBPR into the ECD process. The advisory board helped ensure Kenyan research partners and community members were involved in all stages of the project, from construct development to data collection, to data interpretation, and finally to results dissemination. By gathering community members, we focused on community autonomy and respect for their needs, values, strengths, and culture (Mikesell, Bromley, and Khodyakov 2013).

We convened a community advisory board consisting of caregivers of children with autism, education leaders with backgrounds in special education, and community leaders from sub-counties within a large county in western Kenya. Kenyan team members led the nomination of individuals to serve on the community advisory board. A key element of CBPR is long-term commitment to partnership goals from all partners. The initial list was discussed and expanded with additional nominations of community members who had previously engaged with the team through community outreach and previous research activities, demonstrating a commitment to the goals of understanding and supporting people with autism and their families. The community advisory board consisted of 13 individuals, including four village chiefs and elders, three caregivers of autistic children, one autistic self-advocate, two head teachers/school directors of special schools serving children with disabilities, one member of the Ministry of Education, one pediatrician, and one program director of a teacher training program on autism.

The community advisory board met with the research team for a two-day meeting. The meeting was conducted in English, which is an official language in Kenya. During small group activities, some board members spoke briefly in Swahili, also an official language of Kenya. While the whole research team welcomed and encouraged the use of both languages, board members often prompted each other to speak in English so that their thoughts could be understood by researchers who did not understand Swahili. The board meeting was structured based on Kenyan cultural practices to align with CBPR principles of recognizing community autonomy and building on community strengths and resources. Cultural practices included an opening prayer, individual and extensive self-introductions of persons in the meeting in order of position/hierarchy in the community, a morning tea break, and an orientation to the project. During the introduction, the US and Kenyan principal investigators jointly presented their history of collaboration and the research and outreach already undertaken in the county, the purpose of the project, and the goals for the community advisory board meeting. The first author then presented an overview of activities and processes to be undertaken during the meeting with respect to construct development and brainstorming beliefs, attitudes, and behaviors aligned with the construct maps.

The structure of the community advisory board meeting was culturally adapted according to the Ecological Validity Framework construct of concepts and context (Bernal et al., 1995). Specifically, to engage the community advisory board in construct development and construct mapping, community advisory board members were organized into homogeneous groups based on their role as community leaders, education leaders, or family leaders, at the guidance of our Kenyan colleagues. This homogeneous grouping was essential for ensuring all voices were heard and that different perspectives could be shared and recognized and aligned with the CBPR features of leveraging community strengths and promoting bidirectional learning among community advisory board members and between the board members and the research team.

Kenyan facilitators were present at each small group to facilitate discussion, record notes, and maintain focus on the topic at hand. Facilitators used a script developed collectively by the whole research team and rehearsed the script with the team prior to the advisory board meeting. (See Supplemental Materials.) A single prompt was provided to the small groups, who then reflected on this prompt through dialogue and note-taking on chart paper. These small groups then reported to the larger group. The first prompt requested board members to reflect on whether the initial constructs were comprehensive, distinct, and appropriate. In general, the group suggested only minor revisions to these construct definitions and maps (e.g., using synonyms for particular words that might be better understood in the community). A consensus across groups was quickly reached with approval of these initial constructs.

The community board members then worked in the homogeneous small groups to brainstorm all the possible attitudes, behaviors, and levels of knowledge associated with each construct. We began by having community board members reflect on their own attitudes, behaviors, and knowledge, then asked them to broaden the brainstorming to include the same for members of their communities. For example, how might someone who sees little, if any, stigma attached to autism behave toward someone with autism, and how might someone who has very negative attitudes and beliefs about autism behave toward someone with autism?

After each small group session, we reconvened as a whole group to share out the lists of beliefs/attitudes/behaviors/knowledge and to add any new beliefs/attitudes/behaviors/knowledge identified by the groups. With this more comprehensive list of beliefs, attitudes, behaviors, and knowledge, the whole group was asked to rank each entry on each construct from most to least common in the community. The intent of this process was to help situate these behaviors on the construct maps while prioritizing community autonomy. Examples of two of these construct maps are available in the Supplemental Materials.

In keeping with the CBPR framework, our team also wanted to ensure research benefits were fairly distributed across the community. Thus, at the conclusion of day two, the whole group was again convened, where community board members reflected on the board meeting, offered suggestions for next steps, and shared ideas for how this project could be of mutual benefit to their communities, families, and to research. The community advisory board shared the importance of partnering with trusted community leaders for participant recruitment. They also shared the need for immediate community benefit while the survey was being developed and distributed. This discussion led to the creation of a virtual community for autism information and resource sharing on the platform WhatsApp.

Community advisory board members were essential in reviewing the constructs and brainstorming attitudes, knowledge, and behaviors associated with the constructs. Though this process was time-consuming, it improved the likelihood that the measures developed would elicit what they were intended to (Farrell et al., 2024). Furthermore, when collaboratively generated by those with in-depth understanding of autism’s role in the community (including in families and schools), the process constitutes a form of validity evidence regarding how constructs are represented in the survey. In short, the process helps ensure validity by design, in line with ECD principles.

Item Development

Item writing was developed iteratively over several stages. First, in the days immediately after the stakeholder convening, we gathered as a research team to identify the behaviors, attitudes, and knowledge identified by the stakeholders that seemed essential to include in the survey. This process was driven by Kenyan team members, who had a better sense of the authenticity of the behaviors, attitudes, and knowledge being considered. Part of the conversation was ensuring that the behaviors, attitudes, and knowledge selected covered the range of the construct map for a given construct.

After that initial culling, subsets of the research team drafted items without response categories. The focus was on ensuring the item stems for each block of items were consistent and clear. Following multiple rounds of whole-team discussion about the items, especially their wording and clarity, response options were added for each block of items. Grounded in a CBPR iterative development process, we conducted several additional rounds of whole-team discussion about those items with response categories. This process yielded an initial draft of the survey scales for piloting.

Piloting

Cognitive Interviews. We initially piloted the survey using a cognitive interview framework. The purpose of cognitive interviews is to gather information from respondents about their cognitive processes as they view the survey (Willis, 2005), and to then change the instrument as needed (Karabenick et al., 2007). Following the community advisory board meeting, two Kenyan research assistants trained in cognitive interviewing conducted two rounds of cognitive interviews using an interview protocol developed by the team (see Supplemental Materials) and informed by guidance in cognitive interviews for instrument development by Knafl et al. (2007). Because the survey was intended to be administered to caregivers and educators, in each round of cognitive interviews, we interviewed a caregiver of a child with autism and a special needs educator or educational leader. The first round of interviews included two members of the community advisory board. The second round included two community members who had not engaged with our team in previous research or outreach activities. The aim of these interviews was to determine if potential survey respondents interpreted the survey items as intended. Cognitive interviews were administered individually by a Kenyan research assistant. Respondents were asked to “think aloud” as they reviewed the survey wording, response option format, range of response options for each item, and whether most teachers or caregivers would be able to answer each item accurately. Interviews were audio-recorded and transcribed. Transcriptions were analyzed to identify themes related to clarity and cultural appropriateness of items and response options.

The survey was then revised based on these themes. Revisions pertained to the ordering of items, wording of individual items, and clarification and adjustment of response options. With respect to ordering, the demographics section was moved to the end of the survey, and a rationale for why personal information was requested in this section was provided. Item wording changes were suggested to reduce potential confusion or to select a word that was more commonly used in the community. For example, “downtime” was replaced with “free time,” “evil spirits” was replaced with “witchcraft,” and “trade skills” was replaced with “vocational skills.” Respondents expressed concern with items that had a large number of response options and with differentiation between these options. For this reason, some response options were reduced from five to three or from four to two. The final survey included 84 items and six scales with response options ranging from binary to a four-point Likert.

Translation into Kiswahili

The process of translating the survey from English to Kiswahili involved a forward- and backward-translation approach to ensure accuracy and cultural relevance. First, the survey was translated from English to Kiswahili using Google Translate, after which native Kenyan Kiswahili-speaking members of the research team reviewed and refined the translation to ensure clarity, appropriate terminology, and alignment with local linguistic nuances. Once finalized, the Kiswahili version was then back-translated into English using Google Translate to assess whether the original meaning was preserved. The research team compared the back-translated version with the original survey, identifying any discrepancies and making necessary adjustments to the Kiswahili translation to improve accuracy. This iterative process helped ensure that the final Kiswahili survey maintained the intended meaning while being understandable and contextually appropriate for caregivers and educators in Kenya.

Initial Psychometric Analyses and Results

After completing cognitive interviews and making further refinements, and then translating the survey, we piloted the instrument with approximately 30 participants. With this early data collection, we conducted basic psychometric analyses and, to the extent appropriate, refined the survey once again before final administration for this project.

Psychometric Analyses. While small sample sizes precluded us from engaging in comprehensive latent variable modeling to help establish the validity of the measure for its intended uses, we nonetheless conducted descriptive analyses of the item responses. Many (though not all) of these analyses are either rooted in classical test theory or are rough approximations of the parameters estimated in factor analytic models. We provide details on some of the results in the supplemental materials and Figs. 2 and 3.

Fig. 2The alternative text for this image may have been generated using AI.

Item–total correlations by survey scale

Fig. 3The alternative text for this image may have been generated using AI.

Inter–item correlations across survey scales

One potential concern when designing a survey is that items responses will vary only minimally. For example, if nearly everyone selects “Strongly Agree” for an item, then the item responses will have low variability. To examine variability in the item responses, we plotted histograms of responses for each item and examined summary statistics such as standard deviations, skewness, and kurtosis. Items with particularly leptokurtic distributions (i.e., items that did not elicit much variability among responses) were considered for removal at this stage. We also collapsed response categories if they were not used (e.g., no one selected “strongly disagree”).

Next, we examined inter-item correlations, particularly among items from the same scale (i.e., measuring the same construct), which should be strongly correlated with each other. Similarly, correlations should be lower across constructs, which also speaks to the factor structure of the constructs being measured. We further examined item-total score correlations, separately within each scale and over the entire set of items. Item-total correlations quantify the association between an item and the total score (without that item) for all items in a scale. Higher item-total score correlations indicate that a given item is a more reliable indicator of the latent variable of interest. Items with negative or near-zero inter-item correlations within a construct and item-total score correlations less than 0.30 were considered for removal. Finally, we computed Cronbach’s alpha for each scale to quantify each scale’s internal consistency. Given the small sample size, no statistical result was used alone as grounds to remove an item; rather, we used these results as a mechanism for reviewing the items one last time with our Kenyan research team.

Psychometric Results. In general, the results were very promising. Except for one survey scale, all Cronbach’s alpha values were above 0.70. Furthermore, only two items produced item-total correlations that were below 0.30, which generated conversation about refinement or removal. As for factor structure/dimensionality, inter-item correlations were indeed higher for items within the same scale than across scales.

Given our very small sample size, we did not remove items based on their psychometric properties alone. However, items flagged during the analyses were brought before the research team, and we discussed why the item might be performing at a suboptimal level. For some items, we decided to leave them unchanged under the assumption that small sample sizes could have been a primary culprit. For other items, we made additional (and final) changes based on those conversations.

Perhaps the most substantive change resulted from examining the variability of the item responses. We found that, for the scale measuring caregiver priorities in caring for an autistic child, almost all respondents agreed that each task was important. This likely occurred because most of the priorities involved tasks of daily living (toilet training, providing shelter, taking the child to school, etc.). In response, we removed three items with very low variability in the item responses and replaced them with items related to slightly more “controversial” priorities. These new items were derived from data collected during the advisory board convening, and involved actions like helping the child (upon reaching adulthood) find employment and meet a potential spouse. The final version of the ACE-KS is available in Appendix A.