Scott Ziolek Lear

05-24-2007, 01:32 AM

Great Question! I wanted to respond to this earlier but got caught up in other things.

I have a couple of comments/thoughts regarding this:

First, the simple answer is in the equation for determining sample size.

The inputs required to determine sample size are alpha, the population standard deviation (estimated of course), and finally a desired measurment interval (sometimes expressed as the confidence interval, X+/-B).

Consequently, if you require the confidence interval to be less than a certain value, the sample size must change accordingly.

Second, the nature of the test used matters. You mentioned the significant difference was identified using a paired t-test. This changes that nature of the question.

An independent t-test would be solidly connected to the measurement accuracy. For example, you are asking if I take 30 random people for A and 30 random people for B, do I get the same or different answer (mean score). Random measurement errors just add noise to the results increasing the standard deviation of the sample output.

The paired t-test is asking a slightly different question. It recognizes an influence or connection between A and B in the individual sample. i.e the score Bob receieved on A is related to the score Bob receieved on B. The statistical question here is not will I get the same mean but what are the odds A will always be more or less than B.

In the example below, each rating was a whole number. The difference between the means is below one; however, a paired test still shows a sig. difference. In the case of this paired test, the statistics is focused on the fact that over 85% of the subjects had a higher paired score for A and no one had a higher score for B.

Sample A B A-B

Bob 9 9 0

Rob 8 7 1

Tom 7 6 1

Lil 6 5 1

Sam 5 4 1

Jon 4 3 1

Jen 3 2 1

Avg 6 5.14 0.86

Std 2.16 2.41 0.38

pvalue(Ind 2 tailed t-test) 0.497039

pvale (Paired 2 tailed t-test) 0.000965

Any inaccuracy in equiment should be random, therefore if A and B truly were equal a certain number of ratings should swing in favor of B. In this case it didn't and the paired p-value shows that.

As to your original question "How to document significant digits?" I did like your response:

>

> Fleisig's Four Steps for Significant Digits

>

>

>

> 1. In general, do not present individual values, mean values,

> standard deviations, or other calculated values in smaller units than

> the accuracy of your equipment.

>

>

>

> 2. If you find a statistical significance with a magnitude less

> than the accuracy of the measurement, then question whether strict

> enough statistics were used. For example, if you find a small

> difference to be significant with a t-test, then perhaps the alpha-level

> was set too high.

>

> 3. If you decide the statistical test was appropriate, then report

> the values with the added decimal place.

>

> 4. If you report such a statistical difference, then explain

> whether or not you believe the difference has practical significance.

> In most cases, the researcher will decide the small difference has no

> practical significance. If you feel the difference is important, then

> you probably should have used more accurate equipment.

>

-Scott A. Ziolek

Manger Comfort Engineering and Human Factors

Dymos Technical Center

---------------------------------------------------------------------------------

Hi everyone,

This is general question relevant to a lot of our research. It should

be simple, but I don't recall the answer.

How many digits should be used in reporting data?

Here is an example. A biomechanist is trying to show a difference

between Technique A and Technique B. He/she recruits 36 subjects and

records each one of them performing both techniques, using equipment

that measures each person to the nearest 0.1 unit. How many decimal

places should be used in reporting the data? I can make a "common

sense" determination, but is there some accepted procedure or guideline?

Example data:

TECHNIQUE A

Subject 1: 13.3 units

Subject 2: 35.0 units

Subject 3: 22.2 units

Subject 36: 18.4 units

TECHNIQUE B

Subject 1: 12.9 units

Subject 2: 35.0 units

Subject 3: 21.9 units

Subject 36: 18.5 units

Even though the mean within-subject difference is less than 0.1 units, a

paired t-test reveals a statistically significant difference.

Technique A mean: 25.33333 units

Technique B mean: 25.36170 units

Average difference : 0.02837 units

Paired t-test p-value: 0.035

How many decimal places would you use in reporting these mean values and

difference in your paper? Why?

Thanks in advance,

- Glenn S. Fleisig, Ph.D.

Glenn S. Fleisig, Ph.D., Smith & Nephew Chair of Research

American Sports Medicine Institute

833 St. Vincent's Drive, Suite 100

Birmingham, AL 35205

(email) glennf@asmi.org

(tel) 205-918-2139

www.asmi.org

I have a couple of comments/thoughts regarding this:

First, the simple answer is in the equation for determining sample size.

The inputs required to determine sample size are alpha, the population standard deviation (estimated of course), and finally a desired measurment interval (sometimes expressed as the confidence interval, X+/-B).

Consequently, if you require the confidence interval to be less than a certain value, the sample size must change accordingly.

Second, the nature of the test used matters. You mentioned the significant difference was identified using a paired t-test. This changes that nature of the question.

An independent t-test would be solidly connected to the measurement accuracy. For example, you are asking if I take 30 random people for A and 30 random people for B, do I get the same or different answer (mean score). Random measurement errors just add noise to the results increasing the standard deviation of the sample output.

The paired t-test is asking a slightly different question. It recognizes an influence or connection between A and B in the individual sample. i.e the score Bob receieved on A is related to the score Bob receieved on B. The statistical question here is not will I get the same mean but what are the odds A will always be more or less than B.

In the example below, each rating was a whole number. The difference between the means is below one; however, a paired test still shows a sig. difference. In the case of this paired test, the statistics is focused on the fact that over 85% of the subjects had a higher paired score for A and no one had a higher score for B.

Sample A B A-B

Bob 9 9 0

Rob 8 7 1

Tom 7 6 1

Lil 6 5 1

Sam 5 4 1

Jon 4 3 1

Jen 3 2 1

Avg 6 5.14 0.86

Std 2.16 2.41 0.38

pvalue(Ind 2 tailed t-test) 0.497039

pvale (Paired 2 tailed t-test) 0.000965

Any inaccuracy in equiment should be random, therefore if A and B truly were equal a certain number of ratings should swing in favor of B. In this case it didn't and the paired p-value shows that.

As to your original question "How to document significant digits?" I did like your response:

>

> Fleisig's Four Steps for Significant Digits

>

>

>

> 1. In general, do not present individual values, mean values,

> standard deviations, or other calculated values in smaller units than

> the accuracy of your equipment.

>

>

>

> 2. If you find a statistical significance with a magnitude less

> than the accuracy of the measurement, then question whether strict

> enough statistics were used. For example, if you find a small

> difference to be significant with a t-test, then perhaps the alpha-level

> was set too high.

>

> 3. If you decide the statistical test was appropriate, then report

> the values with the added decimal place.

>

> 4. If you report such a statistical difference, then explain

> whether or not you believe the difference has practical significance.

> In most cases, the researcher will decide the small difference has no

> practical significance. If you feel the difference is important, then

> you probably should have used more accurate equipment.

>

-Scott A. Ziolek

Manger Comfort Engineering and Human Factors

Dymos Technical Center

---------------------------------------------------------------------------------

Hi everyone,

This is general question relevant to a lot of our research. It should

be simple, but I don't recall the answer.

How many digits should be used in reporting data?

Here is an example. A biomechanist is trying to show a difference

between Technique A and Technique B. He/she recruits 36 subjects and

records each one of them performing both techniques, using equipment

that measures each person to the nearest 0.1 unit. How many decimal

places should be used in reporting the data? I can make a "common

sense" determination, but is there some accepted procedure or guideline?

Example data:

TECHNIQUE A

Subject 1: 13.3 units

Subject 2: 35.0 units

Subject 3: 22.2 units

Subject 36: 18.4 units

TECHNIQUE B

Subject 1: 12.9 units

Subject 2: 35.0 units

Subject 3: 21.9 units

Subject 36: 18.5 units

Even though the mean within-subject difference is less than 0.1 units, a

paired t-test reveals a statistically significant difference.

Technique A mean: 25.33333 units

Technique B mean: 25.36170 units

Average difference : 0.02837 units

Paired t-test p-value: 0.035

How many decimal places would you use in reporting these mean values and

difference in your paper? Why?

Thanks in advance,

- Glenn S. Fleisig, Ph.D.

Glenn S. Fleisig, Ph.D., Smith & Nephew Chair of Research

American Sports Medicine Institute

833 St. Vincent's Drive, Suite 100

Birmingham, AL 35205

(email) glennf@asmi.org

(tel) 205-918-2139

www.asmi.org