didn't vs wasn't

As Setanta aptly noted, the verb is be.

No. The verb is 'used'.

Sounds like a case of idiom misery

This rings odd to my ear, as well, because "wasn't he?" questions another sentence, "he was taller."

The statement, "he used to," is questioned by, "didn't he?"

"He used to, wasn't he?" doesn't sound grammatical.

Like what you said, "used to" is an idiom.

You're right. Formal writing would be more understandable as "He was taller. . . "

Are the above correct?

Its all based in context on the story surrounding the sentence. Many options can be acceptable depending on how the story is written.

The third is incorrect.

Laughs out loud whilst simultaneously roaring at the risibility.