Comma after the adjective here?

I'd say no.

Don't worry about such a small issue. Both versions mean the same, anyway.

This is not a complete sentence. If you mean "I am sorry that I couldn't reply to you promptly" then that sentence would not have a comma. If you are using the conversational "sorry" e.g.

Q. Is this the road to London?
A. Sorry, I don't know.

then yes a comma is required.

I agree with Contrex. Sorry, but that's my answer.

In this case, "sorry" is not essential to the answer, so it could be considered an 'aside', which is usually parenthetical, ie., comma-ed, as in, "The girl, who was six foot tall, ate a sandwich."

Same as:
Happy to have helped you.
Sorry to have been late.

Thanks everybody for your answers.