Tweet
Great Question! I wanted to respond to this earlier but got caught up in other things.
I have a couple of comments/thoughts regarding this:
First, the simple answer is in the equation for determining sample size.
The inputs required to determine sample size are alpha, the population standard deviation (estimated of course), and finally a desired measurment interval (sometimes expressed as the confidence interval, X+/-B).
Consequently, if you require the confidence interval to be less than a certain value, the sample size must change accordingly.
Second, the nature of the test used matters. You mentioned the significant difference was identified using a paired t-test. This changes that nature of the question.
An independent t-test would be solidly connected to the measurement accuracy. For example, you are asking if I take 30 random people for A and 30 random people for B, do I get the same or different answer (mean score). Random measurement errors just add noise to the results increasing the standard deviation of the sample output.
The paired t-test is asking a slightly different question. It recognizes an influence or connection between A and B in the individual sample. i.e the score Bob receieved on A is related to the score Bob receieved on B. The statistical question here is not will I get the same mean but what are the odds A will always be more or less than B.
In the example below, each rating was a whole number. The difference between the means is below one; however, a paired test still shows a sig. difference. In the case of this paired test, the statistics is focused on the fact that over 85% of the subjects had a higher paired score for A and no one had a higher score for B.
Sample A B A-B
Bob 9 9 0
Rob 8 7 1
Tom 7 6 1
Lil 6 5 1
Sam 5 4 1
Jon 4 3 1
Jen 3 2 1
Avg 6 5.14 0.86
Std 2.16 2.41 0.38
pvalue(Ind 2 tailed t-test) 0.497039
pvale (Paired 2 tailed t-test) 0.000965
Any inaccuracy in equiment should be random, therefore if A and B truly were equal a certain number of ratings should swing in favor of B. In this case it didn't and the paired p-value shows that.
As to your original question "How to document significant digits?" I did like your response:
>
> Fleisig's Four Steps for Significant Digits
>
>
>
> 1. In general, do not present individual values, mean values,
> standard deviations, or other calculated values in smaller units than
> the accuracy of your equipment.
>
>
>
> 2. If you find a statistical significance with a magnitude less
> than the accuracy of the measurement, then question whether strict
> enough statistics were used. For example, if you find a small
> difference to be significant with a t-test, then perhaps the alpha-level
> was set too high.
>
> 3. If you decide the statistical test was appropriate, then report
> the values with the added decimal place.
>
> 4. If you report such a statistical difference, then explain
> whether or not you believe the difference has practical significance.
> In most cases, the researcher will decide the small difference has no
> practical significance. If you feel the difference is important, then
> you probably should have used more accurate equipment.
>
-Scott A. Ziolek
Manger Comfort Engineering and Human Factors
Dymos Technical Center
---------------------------------------------------------------------------------
Hi everyone,
This is general question relevant to a lot of our research. It should
be simple, but I don't recall the answer.
How many digits should be used in reporting data?
Here is an example. A biomechanist is trying to show a difference
between Technique A and Technique B. He/she recruits 36 subjects and
records each one of them performing both techniques, using equipment
that measures each person to the nearest 0.1 unit. How many decimal
places should be used in reporting the data? I can make a "common
sense" determination, but is there some accepted procedure or guideline?
Example data:
TECHNIQUE A
Subject 1: 13.3 units
Subject 2: 35.0 units
Subject 3: 22.2 units
Subject 36: 18.4 units
TECHNIQUE B
Subject 1: 12.9 units
Subject 2: 35.0 units
Subject 3: 21.9 units
Subject 36: 18.5 units
Even though the mean within-subject difference is less than 0.1 units, a
paired t-test reveals a statistically significant difference.
Technique A mean: 25.33333 units
Technique B mean: 25.36170 units
Average difference : 0.02837 units
Paired t-test p-value: 0.035
How many decimal places would you use in reporting these mean values and
difference in your paper? Why?
Thanks in advance,
- Glenn S. Fleisig, Ph.D.
Glenn S. Fleisig, Ph.D., Smith & Nephew Chair of Research
American Sports Medicine Institute
833 St. Vincent's Drive, Suite 100
Birmingham, AL 35205
(email) glennf@asmi.org
(tel) 205-918-2139
www.asmi.org
I have a couple of comments/thoughts regarding this:
First, the simple answer is in the equation for determining sample size.
The inputs required to determine sample size are alpha, the population standard deviation (estimated of course), and finally a desired measurment interval (sometimes expressed as the confidence interval, X+/-B).
Consequently, if you require the confidence interval to be less than a certain value, the sample size must change accordingly.
Second, the nature of the test used matters. You mentioned the significant difference was identified using a paired t-test. This changes that nature of the question.
An independent t-test would be solidly connected to the measurement accuracy. For example, you are asking if I take 30 random people for A and 30 random people for B, do I get the same or different answer (mean score). Random measurement errors just add noise to the results increasing the standard deviation of the sample output.
The paired t-test is asking a slightly different question. It recognizes an influence or connection between A and B in the individual sample. i.e the score Bob receieved on A is related to the score Bob receieved on B. The statistical question here is not will I get the same mean but what are the odds A will always be more or less than B.
In the example below, each rating was a whole number. The difference between the means is below one; however, a paired test still shows a sig. difference. In the case of this paired test, the statistics is focused on the fact that over 85% of the subjects had a higher paired score for A and no one had a higher score for B.
Sample A B A-B
Bob 9 9 0
Rob 8 7 1
Tom 7 6 1
Lil 6 5 1
Sam 5 4 1
Jon 4 3 1
Jen 3 2 1
Avg 6 5.14 0.86
Std 2.16 2.41 0.38
pvalue(Ind 2 tailed t-test) 0.497039
pvale (Paired 2 tailed t-test) 0.000965
Any inaccuracy in equiment should be random, therefore if A and B truly were equal a certain number of ratings should swing in favor of B. In this case it didn't and the paired p-value shows that.
As to your original question "How to document significant digits?" I did like your response:
>
> Fleisig's Four Steps for Significant Digits
>
>
>
> 1. In general, do not present individual values, mean values,
> standard deviations, or other calculated values in smaller units than
> the accuracy of your equipment.
>
>
>
> 2. If you find a statistical significance with a magnitude less
> than the accuracy of the measurement, then question whether strict
> enough statistics were used. For example, if you find a small
> difference to be significant with a t-test, then perhaps the alpha-level
> was set too high.
>
> 3. If you decide the statistical test was appropriate, then report
> the values with the added decimal place.
>
> 4. If you report such a statistical difference, then explain
> whether or not you believe the difference has practical significance.
> In most cases, the researcher will decide the small difference has no
> practical significance. If you feel the difference is important, then
> you probably should have used more accurate equipment.
>
-Scott A. Ziolek
Manger Comfort Engineering and Human Factors
Dymos Technical Center
---------------------------------------------------------------------------------
Hi everyone,
This is general question relevant to a lot of our research. It should
be simple, but I don't recall the answer.
How many digits should be used in reporting data?
Here is an example. A biomechanist is trying to show a difference
between Technique A and Technique B. He/she recruits 36 subjects and
records each one of them performing both techniques, using equipment
that measures each person to the nearest 0.1 unit. How many decimal
places should be used in reporting the data? I can make a "common
sense" determination, but is there some accepted procedure or guideline?
Example data:
TECHNIQUE A
Subject 1: 13.3 units
Subject 2: 35.0 units
Subject 3: 22.2 units
Subject 36: 18.4 units
TECHNIQUE B
Subject 1: 12.9 units
Subject 2: 35.0 units
Subject 3: 21.9 units
Subject 36: 18.5 units
Even though the mean within-subject difference is less than 0.1 units, a
paired t-test reveals a statistically significant difference.
Technique A mean: 25.33333 units
Technique B mean: 25.36170 units
Average difference : 0.02837 units
Paired t-test p-value: 0.035
How many decimal places would you use in reporting these mean values and
difference in your paper? Why?
Thanks in advance,
- Glenn S. Fleisig, Ph.D.
Glenn S. Fleisig, Ph.D., Smith & Nephew Chair of Research
American Sports Medicine Institute
833 St. Vincent's Drive, Suite 100
Birmingham, AL 35205
(email) glennf@asmi.org
(tel) 205-918-2139
www.asmi.org