Could any native speakers show the difference between three sentences?

Number Two begins with the anterior past. To assure agreement, it should be written: Before he had entered the institute, Jim had served in the army. The sentence, thus amended, would be perfectly reasonable in certain contexts.

I don't like sentence 2; "had" put there there looks awkward to me. You'd need more context to make that work.

Sentence 2 means that Jim's army service came immediately before the time he entered the institute.
Sentence 1 means that his army service COULD have been at another time in his past.
Sentence 3 means the same a. sentence 1.

Look at this sentence:
"He knocked on the door and came in."
The events are in the correct sequence.
For Jim, the sequence of events is:
first, he served in the army
and second, he entered the institute.

However, in (1), that sequence is reversed: we have the second action/event of 'entered' coming first in the sentence, and the first action/event of 'served' coming second.
To show that the time-line is reversed, we use the Past Perfect Tense: "…Jim had served in the army."

In conversation, or informal writing, the use of the conjunctions 'before' and 'after' make it clear which event came first, and came second. We therefore omit the Past Perfect verb form.
Hence, (3) is the informal way of speaking/writing (1), and the meaning is still quite clear: before the institute, he was in the army.

There are ways in which the Past Perfect can be the first verb form in a sentence, but not in (2), and I don't want to complicate this by mentioning those. Let me just say that (2) is incorrect. The Past Perfect is indicating that 'entered the institute' came before some other event…and the conjunction 'before' is indicating that 'served in army' came before institute…so what is this event that 'entered the institute' came before, and did event X come before or after the 'served in army' event?!!!! Too puzzling!

Sentence 2 means that Jim's army service came immediately before the time he entered the institute.

I don't like sentence 2; "had" put there there looks awkward to me. You'd need more context to make that work.

Sentence 2 means that Jim's army service came immediately before the time he entered the institute.

There are ways in which the Past Perfect can be the first verb form in a sentence, but not in (2), and I don't want to complicate this by mentioning those. Let me just say that (2) is incorrect. The Past Perfect is indicating that 'entered the institute' came before some other event…and the conjunction 'before' is indicating that 'served in army' came before institute…so what is this event that 'entered the institute' came before, and did event X come before or after the 'served in army' event?!!!! Too puzzling!

Sentence 2 means that Jim's army service came immediately before the time he entered the institute.