Who's familiar with P value?

Hi, ori. I haven't been around much, but it's good to see you still posting.

The p-value is a statistical term that means the likelihood of that observation occurring by chance if there are no actual differences. The assessment against "no actual difference" is called the null hypothesis and scientists usually hope to reject that hypothesis. Most studies are set to reject the null hypothesis if the p-value is less than 0.05. In this case the p-value of <0.001 means that the null hypothesis is rejected and significant differences have been found.

What your statement doesn't go into is the concept of the power of the study. Generally, the higher the number of participants in the sample, the higher the power of the analysis and the narrower the gap between the differences is to find significance. In this case 56.6% was found to be strongly significantly different than 52.3% (p-value much lower than 0.05). In another study with a smaller sample size and less power the p-value may have been closer to 0.05, or even above it, resulting in no significant differences observed. The absolute numbers don't tell you much. Keep in mind that these are samples of a population, not the entire population, and the gap between differences become significant based on more than just the absolute numbers.

Welcome back, JPB.

Does "the power of the study" mean "the size and significance of the study"?

Sort of. The power of a study is the likelihood of finding a significant difference where none truly exists. It's a balancing act between having a large enough sample size to find a significant difference where one does exist and not having so much power that you find a significant difference when there isn't one.

It comes down to getting the best sample size to assess a statistically significant difference vs an actual significant difference. I'll give you an example... if I use 1 million data points in each of two samples in an analysis then I'm going to call a minute observed difference statistically significantly different. Is a mean cholesterol value of 199.5 in one group clinically different than a cholesterol value of 200.5 in another group? If I have 1 million data points in each sample (male vs female, under 40 years old vs over 40 years old, etc) and compare them to cardiac experience then I will find 199.5 statistically significantly different than 200.5, but are they clinically significantly different? No. My analysis had too much power.