Choosing the Right Methods of Data Collection
What kinds of data can I (and should I) collect?
by Colleen M. Kuusinen, Assistant Director for SoTL
If the first question you're asking yourself is, "Should I use qualitative or quantitative methods?" you may be skipping some important questions. The first is, of course, your research question. As a reminder, there are often two types of questions faculty are investigating in SoTL studies: "What works for what instructional goal?" and "What is happening?" studies. Check out our "Getting Started" page to review these types of research questions.
With a clear "What works for what instructional goal?" research question, you first need to ask yourself "What constitutes acceptable evidence, or data, that I've reached my instructional goal(s)?" The answer to this question is informed by your own expertise in teaching your area, disciplinary standards for assessment, and acknowledgement of realistic limitations on our time and resources.
You likely first considered final, or summative, assessments (assignments) in your course: exams, papers, presentations or other final products that students typically produce towards the end of your course. These are fine sources of data--keep in mind, however, that these assignments may represent the synthesis of different skills and forms of knowledge beyond the scope of your SoTL study. In addition, it's best to have multiple sources of data that help to make student learning visible. Think about these other formative assessments that you can use to collect data on how and what students are learning as your study progresses:
- Selected questions from an exam or standardized assessment measure
- Classroom Assessment Techniques (CATs): Minute papers, clicker data, muddiest points
- Evidence of student thinking: Observation of students (recorded as field notes), audio recordings of student group work, think-alouds
- Instructor reflections: Your own written reflections on student learning and how that relates to your instruction
- Student self-report of learning: Interviews or focus groups with students (current or past), surveys
Any number of these sources of data could be used for "What is happening?' SoTL studies, though because of the nature of the question, qualitative measures of student thinking, case studies, interviews, and focus groups tend to be more useful for these studies.
- This brief yet informative guide from the Vanderbilt University Center for Teaching talks a little about direct vs. indirect evidence of learning and breaks down qualitative and quantitative sources of data.
- In addition, this 2005 article from Dan Bernstein and Randy Bass discusses the limitations of just using graded work, especially for uncovering the "intermediate processes" of student learning.
Once you've identified some methods you might want to try, remember to search the literature for other SoTL scholars who've used those methods. In addition, here are some resources when you're ready to dive deeper:
- Social Science Research Methods Knowledge Base
- Social Science Research: Principles, Methods, and Practices (free, open access textbook)
- Finding Pre-existing Questionnaires and Instruments (always recommended above creating your own)
- UGA offers all faculty and graduate students a free account with Qualtrics, an online survey development tool
- Guidance for Creating Likert Scales
- AAC&U's VALUE (Valid Assessment of Learning in Undergraduate Education) rubrics for assessing written work