No announcement yet.


This topic is closed.
  • Filter
  • Time
  • Show
Clear All
new posts

  • Reliability


    I am posting this on behalf of some students of mine. They have
    conducted a reliability study in which three raters tested and
    re-tested a group of twelve subjects on two different days.
    Currently, they are attempting to calculate an intraclass correlation
    coefficient (ICC) as described by Shrout and Fleiss (1979) and
    Portney and Watkins (Book called 'Foundations of Clinical Research).

    Portney and Watkins state that you can use any variable in the

    'The specifice facets included in the demoninator will vary,
    depending on whether rater, occasions, or some other facetis the
    variable of interest in the reliability study. For example, if we
    include rater as a facet, then the total observed variance, which, of
    course, does not include direct estimates of true variabce (as this is
    unknown). Theoretically, however, we can estimate true score variance
    by looking at the difference between observed variance among subjects
    and error variance. These estimates can be derived from an analysis
    of varaince.'

    The example given in the text has four raters evaluating six subjects
    on a single day. In calculating the ICC, the between subjects mean
    square, error mean square and between raters mean square are taken
    from a repeated measures ANOVA. We can follow this and reproduce
    it quite easily by doing a repeated measures ANOVA with a single
    effect of, RATER. Now in my students study, there are main effects
    of both RATER and SESSION.. We can't decide which mean square terms
    to use because there are also interactions involved.

    We could simplify it by calculating the ICCs separately for test 1
    and test 2 but then lose the effect of session. Perhaps the study
    isn't suited for ICCs.

    Anyone have any advice on how to approach this, or references that I
    can point them to?

    Thank you,

    David M. Hooper, Ph.D.
    Department of Rehabilitation Sciences
    University of East London
    Romford Road
    London E15 4LZ
    Phone 0181-590-7000 (4025)

    To unsubscribe send UNSUBSCRIBE BIOMCH-L to
    For information and archives: