That'd be a good legal question for your lawyer or the court to answer. Perhaps the fine print of your divorce and custodial rights section of your agreement can yield an answer.
However, the path of least resistance is my path of choice. Do you have a working relationship of some sort? Hopefully, this is not some sort of getting back at your ex for some bad blood between you.
I'd call my ex up or contact him and find out what's going on. Unless it really screws up your plans in a major way, I'd grant him the visitation with the agreement that he won't let this happen again in the future.