I don't think you are responsible. You agreed to a set time ahead of both of your flights. You looked after her for the four days. You then dropped her off as scheduled and agreed upon. You had plans with someone else. She is not your child or your responsibility. She is the mother and she is responsible.
What you did does sound a little heartless and maybe morally questionable, but ultimately, you are not responsible for her.
What if you hadn't answered your phone?
What eventually happened? I'm curious.