try/tries

The correct verb is "try," since the sentence takes the subjunctive mood.

It is important that he try/tries to study often.

The subjunctive mood can be a rather difficult concept. Very generally speaking, it is used in relative clauses that express formal commands, requests, or desires:

"It is imperative that you be there tomorrow."
(Formal command)

"I ask that you be on time tomorrow."
(Request)



These sentences are in the simple present subjunctive, but the subjunctive mood can be used for any of the other standard (indicative) verb tenses.

You can find a basic overview of the subjunctive here; it explains things much more thoroughly than I can.

I'd say they're both correct, but have different shades of meaning.

"It is important that he try to study often" emphasizes the value of study, and the attempt to study. It says nothing about whether or not he actually tries to study--it just says that it's important that he should study.

"It is important that he tries to study often" is not a subjunctive because the implication is that he actually is acting--it is a report of his real behavior--he is trying to study often, and that is important (because it means he's more likely to get into Harvard, maybe).