Publication Type: Conference Paper/Unpublished Manuscript
Review Method: Peer Reviewed
Abstract: Survey authors commonly have respondents rate or rank a series of items along some dimension of judgment. Alwin and Krosnick (1985) indicated that the different tasks leads to significantly different latent structures among the variables.

Method

Respondents: 1882 respondents participated, randomly drawn from the Harris Poll Online panel.

Design:

Target assigned (more often visited or less often visited grocery store)
Number of elements to evaluate (5 versus 10)
Evaluative task:
Absolute rating of quality
Comparative rating of quality
Importance rating
Likelihood of influence
Quality Ranking
Importance Ranking

Procedure:
1. Asked about grocery stores they visited.
2. Assigned 1 store to rate
3. Rated familiarity with store and each element
4. Rated criteria (e.g. overall evaluation)
5. Evaluated store on elements (e.g. store’s price of products).
6. Asked 2 questions on task difficulty and accuracy.

Results:

Importance ratings paralleled importance rankings in terms of order. Rating means did not change between the 5 and 10 element conditions, but there were significant shifts for the ranking means.

The Absolute Rating of Quality and Comparative Rating of Quality groups had significantly higher average correlations with the criteria (.48 and .45, respectively) than with the Absolute Ranking of Quality group (the highest average correlation for a single element with was .13).

Respondents with 10 elements perceived that the task was more difficult and they were less accurate than those with 5 elements. Respondents assigned rating tasks perceived the task as easier and they felt more accurate doing it than those assigned the ranking task.

Discussion:

Ranks obtained seemed to be dependent on the presence or absence of other elements, so the selection of the other elements seems a critical, yet understudied, area.

Publication Type: Conference Paper/Unpublished Manuscript
Review Method: Peer Reviewed
Abstract: College ranks are a highly charged
feature of the American educational landscape. The most influential
ranks are those published by the national periodical US News &
World Report, and universities regularly trumpet their standings in the
USN&WR ranks whenever they are able to do so. Although
organizations that publish college ranks claim that their figures can
distinguish minute gradations in quality across educational
institutions, we show using scaling techniques that the information
content in USN&WR ranks is much lower than the USN&WR
organization might care to admit. Namely, our results show that
USN&WR ranks cannot distinguish between the topmost educational
institutions in the country. More generally, the results show that
latent variables, like quality levels of academic institutions, should
always be accompanied by measures of uncertainty so that users of the
levels are not lulled into a false sense security in assuming that the
levels are much more precise than they actually are.

Publication Type: Conference Paper/Unpublished Manuscript
Review Method: Peer Reviewed
Abstract: Measurement is an intervention. It is not inert, it never merely represents. Just how, and how much, measurement intervenes on its object depends on its location, the constituencies for its products, and the extent to which measurement products get built into institutions and cause people to think and act differently. This paper investigates the reflexive effects of one powerful measure, the rankings of U.S. law schools that are published annually by the U.S. News and World Report [USN], a weekly news magazine. Through a series of commensurative practices that turn qualities into quantities, rankings formalize "reputation." They construct a precise, hierarchical and very public relationship among all accredited law schools. They impose on law schools an implicit and universalized definition of “excellence.” To understand the reflexivity of rankings-- their capacity to transform that which they purport to measure-- we focus on four broad kinds of effects that rankings induce: the redistribution of resources, including time, money, and attention; the re-ordering of work; the emergence of classificatory schemes which reflect ranking criteria and which redefine terms of relevance and accountability; and people's changing perceptions of power, vulnerability and competition in their relations with each other and with other law schools. We argue that to appreciate the reflexivity of rankings, it is crucial to understand their constitutive power, the capacity of rankings to create new objects (e.g. a “third tier school”) and new relations among objects (e.g. "peer schools" defined by rank); it is also crucial to appreciate their cognitive power which includes understanding how rankings are used by people to make sense of situations.

Publication Type: Conference Paper/Unpublished Manuscript

Publication Type: Conference Paper/Unpublished Manuscript
Abstract: This study examined commitment in the context of the online social networking site, MySpace. Survey results indicated that constructs of commitment are covariates of ranking, and constraint commitment is a positive predictor of ranking in the Top 8. Additionally, a clear pattern emerged: one’s commitment, whether personal dedication or constraint commitment, is not an indicator of the ranking one will receive from their romantic partner. Relationship type and ranking in the Top 8 were also investigated, and the prevalence of other strong ties such as family and close friends were visible within the Top 8.