What kind of T-test/ANOVA

What's your hypothesis?

What you have is repeated measures on three algorithms. Depending on your hypothesis (what question are you asking about the relationship of the three algorithms?), you'll need to find someone to help you with a repeated measures analysis on discrete (binomial) data. This isn't easy or straight forward for someone who isn't experienced in statistical analysis.

I don't have any idea who told you you should be doing either a t-test (multiple t-tests in multiple pairwise comparisons in this case), or an ANOVA, because both of those tests are for continuous random variables and you have binomial (correct/incorrect) data.

google repeated measures analysis of binomial data for some ideas.

My objective is to find out the best algorithm among 3 of them. In order to perform this, I want to set my null hypothesis as there is no significant differences in accuracy in 3 algorithms.

The important result of this analysis would be the "Accuracy" which is a numerical value (e.g., 87%). So, that's why I think we can use T-test or ANOVA.

I would like to ask you to help me more. Thank you.

You definitely don't want to use multiple t-tests or an ANOVA on the means of the percentages. You can't take an average of your percentages and do an ANOVA -- you'll lose all of the effect of your samples sizes. You have binomial data (correct/incorrect) and it should be analyzed as such. The appropriate way to analyze binomial data is with a chi-square test. Your study isn't quite so straightforward, however.

You have a complicated data model (three algorithms, binomial outcomes, repeated three times on different sample sets). As long as the data are uncorrelated (you didn't use the same articles in the three corpora) then you don't need to go the repeated measures route, but you're looking at a three-way extension to the Cochran–Mantel–Haenszel test. The only thing I can think of would be to compare algorithm A to B, then B to C, and A to C but you'll have to control for the experiment-wise error.

Here's a discussion on the CMH test for comparing two methods (algorithms). Perhaps you could post your question there and ask for help in extending to three algorithms.

http://www.talkstats.com/showthread.php/14430-Chi-squared-test-for-multiple-samples

Thank you so much for your reply.

I think more explanation might help to understand the issue. So, I mention the Number of articles in each Corpus:

Corpus X = 10 articles
Corpus Y = 1800 articles
Corpus Z = 1400 articles
TOTAL = 3210 articles

If we consider the total number of correct/incorrect answers in all corpora (instead of the accuracy in each corpus), I think we would have more accurate analysis. For example, we have 2998 correct answers (out of 3210 articles) for "Algorithm A" which gives 93% accuracy. But if we only consider the accuracies without considering the number of articles in each corpus, we would have (80%+93%+94%)/3 = 89% accuracy. (That's the effect of our samples sizes, that you mentioned)

I have to remind that the important measure is to find out the Best Algorithm. So, we can consider all corpora as one corpus with 3210 articles.

I am willing to try chi-square, but I really do not know what can be the expected value for each algorithm.

If you need more explanation, please let me know.
Thanks a lot.

You can do that, but you have to be careful combining the data across the corpora. Is there anything unique or different between the three corpora? How were the articles selected? Did anything change in the algorithms between the three passes (X, Y, and Z)?

Assuming you could have just as easily chosen 3210 articles at once, then you're safe in combining the corpora. If theres a reason that they were separate then you should account for corpora in your model. You're going to have a problem with a chi-square with n=10 vs 1800 or 1400 if you keep them separate. Chances are good that one or more of your cells will contain n<5 and you'll be looking at an Exact test.

You don't need to calculate your expected values by hand. There are numerous online calculators.

Keep in mind that you'll only drill down and look at the algorithms 2 at a time if you find a significant difference at the complete 3-by-2 level.

Try combining all of your outcomes from the three corpora into a 3-by-2 model and see how it looks.

Here's a calculator that will allow you to create a 3 (row) by 2 (column) table and calculate a chi-square.

http://www.vassarstats.net/newcs.html