Is it wrong to use " the time was late, I had to go home." ?

No, that's a fairly natural expression in writing. I think it's more common to say, "It was late, so I had to go home."

Also, if you write your original sentence, you need a semicolon between the clauses:

The time was late; I had to go home.

Or make two sentences (notice the capitalization and punctuation):

The time was late. I had to go home.

It is a bit awkward, redundant and not something you'd be likely to hear. I'd say " It was late and I had to go home."

No need to Refer to time when you already wrote that it was late.