Cohen, A., Doveh, E., and Nahum-Shani, I. (2009). Test agreement for multi-item scales with rwg (j) and ADm (j) indices. Organ. Res. Methoden 12, 148-164. doi: 10.1177/1094428107300365 Dunlap, W. P., Burke, M. J., and Smith-Crowe, K. (2003). Accurate tests of statistical significance for average difference indices to interrater agree. J.
Appl. Psychol. 88, 356-362. doi: 10.1037/0021-9010.88.356 Burke, M. J., and Dunlap, W. P. (2002). Estimate of the Interrater agreement with the average variation index: a user manual. Organ. Res.
Methods 5, 159-172. doi: 10.1177/109442810205002 Meade, A. W., and Eby, L. T. (2007). Use group agreement cues in validating construction in several stages. Organ. Res. Methods 10, 75-96. doi: 10.1177/1094428106289390 where Sx2 is the average value of points differences in judges` assessments. Figure 2 shows that rwg (j) has favourable ownership of linearity, which means that it is not influenced by the increase in scale elements. Lindell et al.
(1999) suggested that interpretation could be supported by maintaining the range of values permitted by the values of James et al. (1984) and rwg (j) (i.e. 0-1.0). Lindell et al. (1999) indicated that this could be done by determining the expected accidental variance, e2, on the maximum possible divergence, known as maximum dissent. The maximum dissent (`mv2`) is: Brown, R. D., and Hauenstein, N.M. A. (2005). The Interrater agreement is envisaged: an alternative to rwg indices. Organ.
Res. Methods 8, 165-184. doi: 10.1177/1094428105275376 Harvey, R. J., and Hollander, E. (2004, April). “Benchmarking rWG interrater agreement indices: let`s drop the.70 rule-of-thumb,” in Paper Presented at the Meeting of the Society for Industrial Organizational Psychology (Chicago, IL). Multi-level leadership researchers applied direct consensus composition or reference-layer-consensus models for data aggregation at the individual level at a higher level of analysis. The composition of the consensus assumes that there is sufficient consensus within the group regarding the management structure of interest; in the absence of agreement, the entire governance structure is unsustainable. At the same time, guidelines to help leadership researchers make decisions about data aggregation have received little attention.
In particular, a discussion of how data aggregation decisions can improve or conceal the theoretical contribution of a study – a central priority of this article – has not been addressed in depth. Recognizing that empirical generalization depends on the accuracy with which aggregation decisions are applied, we examine the often overlooked assumptions associated with the most common consensus statistic used to justify data aggregation – rWG and rWG (J) (James, Demaree and Wolf, 1984). Based on a dataset published as part of a “Leadership Quarterly Special Issue” (Bliese, Halverson, Schriesheim, 2002), we highlight the potential abuse of rWG and rWG (J) as the only statistic justifying aggregation at a higher level of analysis. We conclude with prescriptive implications for promoting consistency in the way leadership research is conducted and reported at multiple levels. If this value – the share of error in judges` assessments – is subtracted by 1, the remaining variance can be interpreted as a pro-rata agreement. Therefore, the IRA may be for individual scales: Lindell, M.