Supporting Collaborative Exam Development

Matthew Anderson, Professor Kevin Webb

This summer, I took part in a project with Professor Kevin Webb to develop Conceptum, a website designed to facilitate collaborative exam development, especially in Computer Science. The website’s primary focus was on concept inventories, which are exams with think-aloud, interview-style questions given at the start and end of a course that are used to identify systematic misconceptions based on students’ responses. Educators can then use the results to make more objective comparisons of teaching strategies and curricula across professors and institutions, helping them implement more robust and effective teaching strategies. Concept inventories are commonly seen in several STEM disciplines, namely physics, and have played key roles in developing the pedagogy of some fields. Though only one Computer Science concept inventory existed before our project, we believe that they could have a similar impact in further evolving Computer Science education. Thus, our goals with the Conceptum project were twofold: to create a tool to facilitate and streamline concept inventory development and, by extension, encourage the advancement of Computer Science concept inventories.

Facilitating concept inventory development is so important because of how costly and intensive the process currently is. Educators, often across multiple institutions, must conduct open interviews with hundreds of students, record everything they can, then evaluate the results. Concept inventory developers must hold to a principle called exam validation, which means that if too many students misinterpret a question, that question is revised, and the entire process is repeated. This means tracking hundreds of questions, many with different versions, and hundreds of interviews, and as no serviceable tool currently exists to do this, Conceptum should serve as a robust infrastructure to store, query, and filter this data when building a final concept inventory.

After the work done this summer, Conceptum is nearly complete. All the functionality we set out to implement is done, and aside from some cosmetic touch-ups, the user interface and display is also complete. There is still room to expand upon the existing features, but Conceptum is more than functional in its current state, and should be ready for use soon.