knowledge of classical test theory, factor analysis, and item response theory
The Term Paper. As noted, to demonstrate your knowledge of classical test theory, factor analysis, and item response theory, you will do a project that should occupy you for most of the quarter. The goal is to have you select “items” to measure a continuous trait or attribute of people in a sample from some population. An item is a stimulus (often, a short phrase) to which persons must respond with a response on a structured binary (e.g., “yes” vs “no”) or ordinal (e.g., “disagree”, “neither agree or disagree”, “agree”) response scale. [Note that, although we could, we do not allow strictly continuous responses.] By “trait” or “attribute” I mean any “quality that makes one person or thing different from another” (Merriam-Webster definition). By “ordinal scale” I mean a response that is scored with ordered integers having 2-9 possible responses (e.g., 1, 2, 3, 4, or 5 for a 5-point response). By “continuous” I mean that these differences among persons are in principle quantitative and could be aligned along a single continuum. By sample and population, I mean these concepts from Intro Stat – the population might be a reasonably well-defined group of people, such as “college students,” or “depressed male inpatients.”
An example of an educational trait might be “knowledge of calculus,” for which an item would be a calculus question and a binary response to the item would be correct or incorrect, perhaps scored “1” or “0”. A few examples of traits or attributes in other fields are given on the web handouts titled “Personality Traits from Wikipedia Trait Theory”, “Illustrative Political and Social Attitudes,” and “Illustrative Health Behaviors and Health Status.”
It does not matter what trait you choose to work with. However, you must choose a trait for which you can locate, download, and work with an empirical dataset (or, data set) that some investigator has already gathered and made publicly available. By a “dataset” I mean an N x p matrix of numbers (numerical values representing the structured responses such as yes or no) representing the responses of N respondents (subjects, clients, participants) on p items. I would like 300 < N < 10000, and 12 < p < 30. In such a matrix, each row represents one respondent’s data, and each column corresponds to an item. Note that an item response is just a variable as defined in statistics. If the data is incomplete (i.e., the matrix has missing entries), you will need to reduce the data set to one that has no missing data (not the best practice, but necessary due to brevity of the course). You may need to recode or reverse-score variables so that their responses are ordinally meaningful. It is harder work to work with a larger number of items, say, p > 20 to begin with; although you could start with 12 items, this is not as challenging. The difficulty of the challenge you take on will be noted in grading (see below).
Part 1:
Length: 1-2 pages. This report identifies your data source (with reference) and describes the trait to be measured. This report should convince the reader that you have a very good grasp on: the meaning of the trait you are measuring; which specific items might be the best indicators of that trait; which items need rekeying; the specific programs and procedures you will use next to analyze your data. Note: Do not report on any data analyses in this report.
Part 2:
Length 2-4 pages. Start with a short summary on the trait you are measuring, and provide the data source for this trait. This report describes summary descriptive statistics on the data that you have selected, downloaded, and analyzed by some available statistical computer program. Computer printout is ok to back up discussion. Provide means, SDs, and a correlation matrix. Report on any issues (both good and bad) you notice with these data regarding its use for scale construction. Explain how you handled missing data, how you recoded for meaning, etc. Verify that no variable correlates 1.0 with any other variable, that no variable correlates zero with all other variables, and that no variable has any correlation outside the -1 to +1 range.
recode each of your items so that all of the items are binary coded (e.g., 0/1 or 1/2).
Part 3:
Length 2-4 pages. Again, start with a short summary on the trait you are measuring, and provide the data source for this trait. This report summarizes one or more exploratory factor analyses (FA) that you did using polychoric correlations (with EQSIRT or another program). Do these analyses only with the binary items you feel passed your previous scrutiny. Your task is to use FA to develop a shorter (say, 10 items from 20 to start with) unidimensional scale, i.e., a scale that others in principle could use to measure the attribute you studied. Summarize what analyses you did, and what you concluded from those. Computer printout is ok.
Part 4:
Length 2-4 pages. Computer printout is ok. This report summarizes one or more item response theory (IRT) analyses that you did with the EQSIRT or equivalent program on your binary item data. As before, make use of previous knowledge from earlier analyses. Your task is the same as before, to develop a shorter unidimensional scale from a larger set of items, but in this report use only IRT theory to do so.
This term paper is to be written like a scientific report having a standard publication format such as: Introduction, Methods, Results, Discussion, and References. Place your tables and figures at the end after References, each on a separate page. Note that because you may do several proce- dures in sequence, both Method and Results may have subheadings, where each subheading deals with a specific procedure. If possible, use the APA (American Psychological Association) publication formatting rules; but any consistent and logical style is ok with me.