Measure of reliability for dependent variable

[Brian Horsak]

Dear all,

I have a (probably very simple) question regarding the assessment of reliability for quantitative measures.

* We want to determine mean muscle activation for a specific task (squat) under two conditions (stable-unstable).
* Three squat trials for each participant will be averaged to calculated the parameter mean activity - these were assessed in a single session
* So lets say I have two conditons (stable unstable) and one muscle to assess, this gives two parameters which will be used to analyse if there are any differences between both conditions

If I would like to test reliability of my depend variable (this should be the mean activity), is using the cronbach´s Alpha (for the three trials for each condition) a valid method? In this example I would have two Cronbach Alpha values (one for the stable and one for the unstable condition).

Thank you,
Brian

[Gannon White]

Re: Measure of reliability for dependent variable

I'm curious because unless you move the electrode and all recording issues are held constant why are you concerned with the mean activity reliability? You are talking about reliability of the 3 trials of the same condition (reliability for 3 stable squats)? What's going to change from trial to trial for a given movement would be the individual MUs/fibers recruited and how that's reflected in the EMG signal.

Regards,
Gannon

[Brian Horsak]

Re: Measure of reliability for dependent variable

Hi Gannon,

well one session included six trials out of which three were used for processing. Start and End of each squat were assessed manualy - we did not have any elektr. goniometer or other measurement equipment (this of course is a major drawback). So the mean is dependent on the time interval selected. However, we used a standard DV video synchronized to the signals as well as the signal traces to identify start and end of each shquat (I know using a video is also questionable for determining start and end in trials for emg). Therefore, we want to check if data were stable over all three trials.

Thanks,
Brian

[Dan Robbins]

Re: Measure of reliability for dependent variable

Hi Brian,

I am curious why you woulld like to use Cronbach's Alpha, as I understand it this is typically used for more observational style studies than EMG style studies.

Why not simply look at intergrated EMG over normalised squat durations? That would provide you with an indication of the volume of muscle activity during each squat condition. You could also try cross-correlations of the signals? This would provide you with an idea of the similartity between the activation patterns. Depending on the movement style you have selected you could potentially look at signal frequency changes, or for a basic direct comparision you could simply normalise the unstable ampltiude to the stable squat amplitude (provided you trust your video syncronisation of course).

Not sure if that helps, something to think about at least.
Dan

[Brian Horsak]

Re: Measure of reliability for dependent variable

Why not simply look at intergrated EMG over normalised squat durations?

I think that time normalizing of squat cycles does presume correct identified start and end points of each cycle. Therefore, for me normalization makes no sense. Please correct me if I´m wrong ; )

I`m just trying to get a measure of how reliable my determination of time intervals (and therefore the mean activity parameter) was.

Thanks,
Brian

[Matthew Tenan]

Re: Measure of reliability for dependent variable

Hi Brian,

It sounds to me like you are interested in the "agreement" of of data within your two groups so as to justify averaging them. Is that correct? I would suggest two methods:

1. Intraclass correlation analysis (this is the method I prefer, typically) - The effectiveness of this may depend upon your total amount of data, but if the ICC is high (meaning high agreement between the data), it would indicate good internal consistency.

2. Plotting the mean values of each trial with the 95% confidence intervals to visually determine similarity.

Hope any of that is helpful.

Matt

[Christian Maiwald]

Re: Measure of reliability for dependent variable

Hi Brian,

It sounds to me like you are interested in the "agreement" of of data within your two groups so as to justify averaging them. Is that correct? I would suggest two methods:

1. Intraclass correlation analysis (this is the method I prefer, typically) - The effectiveness of this may depend upon your total amount of data, but if the ICC is high (meaning high agreement between the data), it would indicate good internal consistency.

2. Plotting the mean values of each trial with the 95% confidence intervals to visually determine similarity.

Hope any of that is helpful.

Matt

Hi Brian

I'll try to make a brief description of my approach to reliability here: there's 2 different ways, one is based on agreement, the other on consistency. They're not the same, and - although popular - the consistency approach has several disadvantages. Some helpful literature about all this can be found from Martin Bland and Douglas Altman, published in the British Medical Journal, Lancet and others, dealing with method comparison studies in clinical research. Keyword here might be "Limits of Agreement". The approach is easily adapted to what you're trying to do. We have also published a methodological paper on this topic which can be found here:
Maiwald, C.; Axmann, D. & Grau, S. Measurement error in footwear research biomechanics. Footwear Science, 2011, Vol 3, p117-124. It basically contains an adaptation of the Bland & Altman philosophy to biomechanical data.

As Matt wrote, you are probably interested in AGREEMENT of repeated measures of your data. Then you should use a method, that actually computes agreement in your data.
I suggest to avoid any correlation method (Cronbach's alpha, Intraclass Correlations, Cross-Correlation), since only some of them actually assess agreement - and most of them don't. Some of the most popular statistical software packages do not even tell you what exact type of coefficient is being computed - which is cruical for assessing agreement with ICCs.

Briefly explained, you should calculate a measure that is based on differences between your repeated data recordings. If there are only two measurements per subject, this is easy and can be accomplished using the so called "Bland & Altman" Plot and Limits of Agreement (LoA). It's a method that results in an interval, in which you can expect 95% of your differences between the repeated measures to lie. You can then rate the interval as being acceptably small, or inacceptably wide (unreliable measurements).
If there are more than two measurements per subject, the so called "Root Mean Square Error" does the trick, and is interpreted the same way as LoA, easily calculated via ANOVA.
All these approaches are based on discrete measures, by the way, not an entire timeseries. Reliability of timesieries is an even more complex topic and would require something like prediction bands, I'm not going into detail here. Literature for this can be found here: Sutherland, D.; Kaufman, K. R.; Campbell, K.; Ambrosini, D. & Wyatt, M. Clinical use of prediction regions for motion analysis. Dev Med Child Neurol, 1996, Vol 38, p773-781.

This is a wide field of discussion, which cannot be put into this post. But all the agreement methods are far simpler to compute than any ICC, and way easier to interpret. A comprehensive overview on reliability methods can be found here: Atkinson, G. & Nevill, A. M. Statistical methods for assessing measurement error (reliability) in variables relevant to sports medicine. Sports Med, 1998, Vol 26, p217-238.
Hope this information helps a little.

Cheers
Chris

[Gannon White]

Re: Measure of reliability for dependent variable

Hi Gannon,

well one session included six trials out of which three were used for processing. Start and End of each squat were assessed manualy - we did not have any elektr. goniometer or other measurement equipment (this of course is a major drawback). So the mean is dependent on the time interval selected. However, we used a standard DV video synchronized to the signals as well as the signal traces to identify start and end of each shquat (I know using a video is also questionable for determining start and end in trials for emg). Therefore, we want to check if data were stable over all three trials.

Thanks,
Brian

Gotcha. If the video is synced and you have some criteria (e.g. crossing a certain angle at the hip or knee) to visually identify start and finish I think it is just fine.

Good Luck!
Gannon