Skip to content

Validity, Self-efficacy, TAs, and CLASS

October 29, 2012

Maxwell (1992), Understanding and validity in qualitative research, Harvard Ed. Review 62(3), 279.

This paper was recommended to me by Rachel. Maxwell argues that validity is necessarily tied to the type of understanding one hopes to gain from the research. Validity applies not to data or methodology but to accounts based on said data and methodologies. As qualitative researchers everything one observes is subject to interpretation by the observer. Validity is thus not a matter of comparing an account to “the truth” (researchers do not have access to “the truth”) but is instead a matter of comparing an account to something external to the account. Maxwell lays out five categories of validity for qualitative research:

  1. Descriptive validity – Does the account accurately represent what really happened with all of the important occurrences included? No interpretation, no debatable theoretical framework, just a question of whether the account accurately reflects the physical occurrences.
  2. Interpretive validity – Does the account accurately represent what various occurrences meant to the participants involved? Deals with participant’s intention, affect, beliefs, etc. No debatable theoretical framework but could debate whether throwing an eraser represented anger or frustration for the participant.
  3. Theoretical validity – Does the account accurate represent a theoretical interpretation. Deals with whether an event described as co-constructing knowledge truly represents co-constructing knowledge according to that theory.
  4. Generalizability – Can the account of this specific situation be extended to other persons, times, and settings? In qualitative research generalizability is usually achieved through appeal to a theoretical framework (rather than by statistics). The qualitative study contributes to understanding the limits and applicability of a theory. The theory then describes what might be expected in other circumstances.
  5. Evaluative validity – Does the account describe certain actions by the participants as correct or incorrect? Ex. Was the student justified in throwing the eraser? Maxwell mainly cites other authors who have discussed this form of validity.

……………………………

Sawtelle, Brewe, Goertzen, & Kramer (2012), Identifying events that impact self-efficacy in physics learning, PRST-PER 8, 020111.

Many studies of self-efficacy involve having subjects reflect on prior experiences and talk about how those events affected their confidence in a given area. Sawtelle et. al look for verbal and non-verbal markers that are associated with events that impact self-efficacy. Based on a literature review, the authors identify three sorts of experience that have the potential to impact self-efficacy:

  • Mastery experiences – Experiences of personal attainment or failure.
  • Vicarious learning – Observing others perform a task and comparing your own performance to theirs.
  • Social persuasion – Receiving direct or indirect, verbal or non-verbal evaluative feedback about your performance. This could come from peers or from a source of authority.

The authors video tape three students from an intro modeling physics course as they engage in group problem solving outside of class. The authors analyze the video and identify segments approximately 30 seconds in length that they believe contain one or more of the above experiences. The authors then review the video with the students (individually) a year later and ask the students to talk about their experiences and how those experiences made them feel. In this way the authors validate that segments they identify as vicarious learning or social persuasion are in fact experience the students state as impacting their perceived ability. The authors note that in their methodology VL and SP experiences play a more prominent role than mastery experiences (which previous, reflection studies have identified as being a very significant part of self-efficacy).

This is an interesting paper with nice examples of video analysis and lots of good references on self-efficacy. It makes me curious about momentary self-efficacy and long term self-efficacy. Many of the events they identify look to me like they could impact a student’s self-efficacy in that moment or in that problem solving session but may not, individually, have an impact on the student’s self-efficacy at the end of the chapter or the end of the class. I would be interested to see a study comparing students’ perception of certain events right at the end of class and then their perception of those same events at the end of the semester. I would also be interested to see an analysis that looks at these sorts of events both individually and as part of an longitudinal network of interactions that potentially determine the student’s self-efficacy at the end of the semester.

……………………………

Spike & Finkelstein (2012), Preparing tutorial and recitation instructors: A pedagogical approach to focusing attention on content and student reasoning, AJP 80, 1020.

I heard Ben talk about this work at AAPT so I was excited to see the paper with more details. The paper details the authors’ attempts to slightly modify weekly TA/LA meetings to focus TAs and LAs on student thinking (as opposed to just physics concepts) and to better align TA and LA expectations with the focus of a given recitation activity (in this case conceptual understanding as emphasized in the UW tutorials).

The weekly meetings were preceded by an online survey asking about expected student difficulties and a tutorial pre-test. The meetings started by working through the tutorials as students followed by examples of previous student responses. At the end of the meeting TAs and LAs were again surveyed about expected student difficulties. TAs and LAs then taught their classes and reported back about observed student difficulties. The authors found that prior to the meetings the expected difficulties would be broad and focused on mathematical complexities while after the weekly meeting the expected difficulties would be more targeted and more focused on the conceptual aspects emphasized in the actual tutorial.

Somehow the distinction between “expected difficulties with Newton’s 2nd law” and “expected difficulties in this activity about Newton’s 2nd law” was not something I had thought much about prior to reading this paper. This is an important distinction as they could potentially involve different skill sets and different resources on the part of the students. The authors note that they specifically asked TAs and LAs to list the three most common difficulties associated with a given lesson rather than the three most important difficulties as they did not want to introduce confusion about how to judge relative importance. I’m curious if through this experience the TAs and LAs had any shifts in the importance or difficulty of conceptual understanding versus mathematical understanding. My experiences in recent years has certainly given me an increased respect for the importance and difficulty of conceptual questions. I’m curious if the TAs and LAs started the semester thinking the math was more difficult than the concepts and ended the semester thinking the opposite. (Or, more likely, a less extreme version of this shift.)

……………………………

Adams, Perkins, Podolefsky, Dubson, Finkelstein, & Wieman (2006), New instrument for measuring student beliefs about physics and learning physics: The Colorado Learning Attitudes about Science Survey, PRST-PER 2, 010101.

This paper describes the development and validation of the CLASS survey. This is a very useful reference for Mark and I. (Especially useful are the time frame over which various validation tests and modifications are carried out as well as the numbers of interviews, surveys, classes, etc. used in the study.) The authors describe interviews with experts and students to check question interpretations and establish the expert consensus. They also discuss their use of both exploratory factor analysis and subject grouping of survey statements to create an initial set of potential categories (or factors) which are then fine tuned individually through the use of principal component analysis (same as reduced basis factor analysis?). This is a process we plan to explore with our survey as well.

The authors note that when factor analysis is performed on other surveys with predetermined categories (for example the MPEX) they find little correlation between students responses on statements within the same category. This suggests that novices may not (unconsciously) group statements the same way experts do. Alternatively, it could be that novices’ beliefs are more fragile and fractured (in pieces) than experts’ beliefs. In combining both exploratory factor analysis and predetermined categories, the authors arrive at categories that are not all independent and thus have non-zero correlations and in some cases the same statement is listed in multiple categories.

Advertisements

From → Uncategorized

Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: