Tweet
Hello,
I am posting this on behalf of some students of mine. They have
conducted a reliability study in which three raters tested and
re-tested a group of twelve subjects on two different days.
Currently, they are attempting to calculate an intraclass correlation
coefficient (ICC) as described by Shrout and Fleiss (1979) and
Portney and Watkins (Book called 'Foundations of Clinical Research).
Portney and Watkins state that you can use any variable in the
analysis.
'The specifice facets included in the demoninator will vary,
depending on whether rater, occasions, or some other facetis the
variable of interest in the reliability study. For example, if we
include rater as a facet, then the total observed variance, which, of
course, does not include direct estimates of true variabce (as this is
unknown). Theoretically, however, we can estimate true score variance
by looking at the difference between observed variance among subjects
and error variance. These estimates can be derived from an analysis
of varaince.'
The example given in the text has four raters evaluating six subjects
on a single day. In calculating the ICC, the between subjects mean
square, error mean square and between raters mean square are taken
from a repeated measures ANOVA. We can follow this and reproduce
it quite easily by doing a repeated measures ANOVA with a single
effect of, RATER. Now in my students study, there are main effects
of both RATER and SESSION.. We can't decide which mean square terms
to use because there are also interactions involved.
We could simplify it by calculating the ICCs separately for test 1
and test 2 but then lose the effect of session. Perhaps the study
isn't suited for ICCs.
Anyone have any advice on how to approach this, or references that I
can point them to?
Thank you,
David
David M. Hooper, Ph.D.
Department of Rehabilitation Sciences
University of East London
Romford Road
London E15 4LZ
Phone 0181-590-7000 (4025)
d.m.hooper@uel.ac.uk
-------------------------------------------------------------------
To unsubscribe send UNSUBSCRIBE BIOMCH-L to LISTSERV@nic.surfnet.nl
For information and archives: http://www.lri.ccf.org/isb/biomch-l
-------------------------------------------------------------------
I am posting this on behalf of some students of mine. They have
conducted a reliability study in which three raters tested and
re-tested a group of twelve subjects on two different days.
Currently, they are attempting to calculate an intraclass correlation
coefficient (ICC) as described by Shrout and Fleiss (1979) and
Portney and Watkins (Book called 'Foundations of Clinical Research).
Portney and Watkins state that you can use any variable in the
analysis.
'The specifice facets included in the demoninator will vary,
depending on whether rater, occasions, or some other facetis the
variable of interest in the reliability study. For example, if we
include rater as a facet, then the total observed variance, which, of
course, does not include direct estimates of true variabce (as this is
unknown). Theoretically, however, we can estimate true score variance
by looking at the difference between observed variance among subjects
and error variance. These estimates can be derived from an analysis
of varaince.'
The example given in the text has four raters evaluating six subjects
on a single day. In calculating the ICC, the between subjects mean
square, error mean square and between raters mean square are taken
from a repeated measures ANOVA. We can follow this and reproduce
it quite easily by doing a repeated measures ANOVA with a single
effect of, RATER. Now in my students study, there are main effects
of both RATER and SESSION.. We can't decide which mean square terms
to use because there are also interactions involved.
We could simplify it by calculating the ICCs separately for test 1
and test 2 but then lose the effect of session. Perhaps the study
isn't suited for ICCs.
Anyone have any advice on how to approach this, or references that I
can point them to?
Thank you,
David
David M. Hooper, Ph.D.
Department of Rehabilitation Sciences
University of East London
Romford Road
London E15 4LZ
Phone 0181-590-7000 (4025)
d.m.hooper@uel.ac.uk
-------------------------------------------------------------------
To unsubscribe send UNSUBSCRIBE BIOMCH-L to LISTSERV@nic.surfnet.nl
For information and archives: http://www.lri.ccf.org/isb/biomch-l
-------------------------------------------------------------------