On informativeness and reducibility in competence models

Knowledge structure theory (KST; Doignon and Falmagne, 1985, Doignon and Falmagne, 1999, Falmagne and Doignon, 2011) provides a valuable mathematical framework for the assessment of knowledge and learning. One of the central concepts in the theory is knowledge state. It is inferred from responses to items and is defined as the set K of all items that an individual can solve in a finite knowledge domain Q. The knowledge state is a latent construct inferred from observable performance. Building on KST, competence-based extension of knowledge structure theory (CbKST; Doignon, 1994, Düntsch and Gediga, 1995, Falmagne et al., 1990, Gediga and Düntsch, 2002, Heller et al., 2013, Korossy, 1997, Korossy, 1999) takes into account the set of skills an individual has available, which is referred to competence state. CbKST overcomes the limitation of focusing only on observable performance by explicitly incorporating the competence state, which is linked to the sets of skills required to solve the items. Within CbKST, these two constructs are characterized by two key mappings: the skill function and the problem function. The skill function links items to skills by assigning to each item one or more subsets of required skills. There are two common types of skill functions, the so-called conjunctive (all skills associated with an item are necessary for solving it) and disjunctive (having available any of the skills associated with an item is sufficient for solving it) skill functions. The problem function, goes in the opposite direction, maps every set of skills to the set of items that can be solved using those skills, as defined by the skill function. Together, these concepts provide a comprehensive framework for understanding the relationship between an individual’s latent skills (the competence state) and the set of items they are capable of solving (the knowledge state), which is in turn inferred from their observable performance.

Abundant results have been achieved in CbKST (Doignon, 1994, Heller and Stefanutti, 2024, Heller et al., 2015, Heller et al., 2013, Korossy, 1997, Korossy, 1999, Stefanutti and de Chiusole, 2017, Stefanutti et al., 2023). One of its primary aims is to infer the latent competence state of an individual from the knowledge state derived from their responses to test items. The role of the problem function is critical. When the problem function is injective, then there is a unique competence state underlying the knowledge state. That is, there is a one-to-one correspondence between knowledge states and competence states. Heller et al. (2017) introduced a condition called the identifiability condition, which is both necessary and sufficient for uniquely identifying the competence state corresponding to a given knowledge state. Additionally, other conditions, referred to as the witness condition and the generalized witness condition (Heller et al., 2017, Heller et al., 2015, Heller et al., 2016), are sufficient but not necessary for ensuring the injectivity of the problem function under a conjunctive skill function.

However, as Heller et al. (2015) have demonstrated, the problem function need not always be injective. In such cases, the same knowledge state may correspond to multiple competence states. To address this challenge, the approach is to work with the equivalence classes induced by the problem function. Each such class encompasses all competence states that delineate the same knowledge state.

On the one hand, under the assumption of conjunctiveness, the equivalence class is closed under intersection and thus contains its greatest lower bound, termed the floor by Stefanutti and de Chiusole (2017). The floor represents the collection of all minimally sufficient skills required to delineate the knowledge state induced by the problem function when applied to any member of the corresponding equivalence class. The concept of the floor becomes particularly significant when knowledge states and competence states do not exhibit a one-to-one correspondence. For every knowledge state, there exists exactly one floor that delineates it. In other words, a knowledge state provides unambiguous information about the minimal subset of skills an individual has mastered, as represented by the corresponding floor.

On the other hand, the lack of a one-to-one correspondence makes the equivalence classes of competence states induced by the problem function worthy of study in their own right. For a collection of competence states, the greater the number of equivalence classes derived from the problem function, the lower the informational uncertainty regarding the competence state underlying the knowledge state, this equates to greater informativeness about an individual’s competence state. Competence-based test development (CbTD; Anselmi et al., 2022, Anselmi et al., 2024) is a recent and innovative approach to designing tests that are as informative as possible about individuals’ competence state. Rooted in the theoretical framework of CbKST, Anselmi et al. (2022) outlined procedures for test development in three key contexts: the construction of a test from scratch, the improvement of an existing test, and the shortening of an existing test. Each of these procedures contributes to maximize informativeness about individuals’ competence states. The comparison of informativeness between different models forms the foundation of this article. We establish a connection between this comparison and the concept of floors introduced earlier. Under specific restrictions on the collection of competence states, the informativeness of two conjunctive models can be represented through the set inclusion of their corresponding floors. Additionally, due to the one-to-one relationship between a knowledge state and the floor that delineates it, we reformulate this relationship through the application of ordered sets. This allows difference in informativeness to be expressed through order-preserving maps with distinct characteristics, offering a novel approach to comparing different models. Building on informativeness, reducibility (Anselmi et al., 2022) pertains to the possibility of removing items without making the model less informative. A reducible model allows for a proper subset of the domain while preserving its informativeness. In contrast, irreducibility refers to the inability to remove items without diminishing the model’s informativeness. Irreducibility is particularly significant in practical applications, such as educational assessment, where the goal is to efficiently measure students’ competencies by selecting and administering the minimal number of questions necessary, while avoiding redundant ones. In this paper, we address these challenges and provide solutions to optimize test design.

This paper is organized as follows. Section 2 introduces the foundational concepts. Section 3 explores the informativeness of conjunctive competence models through two representation methods: one based on the set inclusion of floors, and the other utilizing mappings between two specially ordered sets. The disjunctive competence models and general competence models are also examined. Section 4 focuses on the reducibility of conjunctive competence models, providing characterizations for both reducibility and irreducibility. Finally, Section 5 concludes the paper.

Comments (0)

No login
gif