Publication Type: Conference Paper/Unpublished Manuscript
Review Method: Peer Reviewed
Abstract: Event data are one of the primary sources of data used in conflict studies; e.g., COPDAB and WEIS data, and more recently, ACLED and PKOLED. In most cases, however, little attention has been paid to issue of coder reliability (Rothman 2007, Mitchell

Publication Type: Conference Paper/Unpublished Manuscript
Abstract: The long time series of estimated party policy positions generated by
the Comparative Manifesto Project (CMP) is the only such time series
available to the profession and has been extensively used in a wide
variety of applications. Recent work (e.g. Benoit, Laver, and
Mikhaylov 2007; Klingemann et. al. 2006, chs. 4--5) focuses on
non-systematic sources of error in these estimates that arise from the
text generation process. Our concern here, by contrast, is with error
that arises during the text coding process, since nearly all
manifestos are coded only once by a single coder. First, we discuss
reliability and misclassification in the context of hand-coded content
analysis methods. Second, we report results of a coding experiment
that used trained human coders to code sample manifestos provided by
the CMP, allowing us to estimate the reliability of both coders and
coding categories. Third, we compare our test codings to the
published CMP "gold standard" codings of the test documents to assess
accuracy, and produce empirical estimates of a misclassification
matrix for each coding category. Finally, we demonstrate the effect
of coding misclassification on the CMP's most widely used index, its
left-right scale, and draw conclusions for future use and design of
the CMP data.