Is my husband having an affair?

My husband (32) and I (30) have been together almost ten years, and married for two. Just about two weeks ago, we were getting ready to go away on vacation and the day before, he was taking our dogs to a family member... which was about an hour away. He had my sister's vehicle. Anyways, my father (who I haven't speaked to in almost two years) and my stepmother are saying they saw him that day driving, smoking a cigarette, one of our dogs in the window, and a blonde girl in the car who was leaning in towards him and they both were laughing. They had called my sister about it because they thought the girl was me (I'm blonde), and they were worried that our 4 month old nephew (my sister's son) was in the vehicle with him smoking (because I was watching him that day.) Needless to say though, I was not in the vehicle.

When I confronted my husband about it, he insisted that there was absolutely no one else in the vehicle besides him and the dogs. He looked me straight in the eye several times and told me this, then kept trying to go over time frames with me to reassure me. I asked him if he would be willing to take a lie detector test and again he looked me straight in the eye and said, "I absolutely would. And then when you see I am not lying, you can thank your father for trying to ruin our lives."

Anyways, I have analyzed everything. He never comes home late from work, barely ever goes anywhere without me, and there are no other signs that I can see or have come across to make this accusation out to be true. I thought back on that day to the time frame from when he left and to when he got back, and honestly, I am not seeing where there was time for him to be hanging out with anyone. For the few hours he was gone, I actually was on the phone with him a couple times... one time being for almost a half hour. So really I am just baffled. My father can be a troublemaker when he wants to be so if it were just him saying this, I probably would have brushed it off. But what's getting to me is my stepmother. Part of me just does not see her lying if it weren't true. She doesn't always agree with my father and will say if he's exaggerating. But with this, she swearing on people's lives that this is what they both saw and she said she has absolutely no reason to lie to me.

So, at this point, I don't know what to think or do. My husband and I have been trying to conceive and we were also looking into buying our first home within the near future. Now I feel we are set back and I don't know where to go with this. I don't want to throw our marriage away if he is telling me the truth. If he isn't, then we have obviously have a lot to figure out.

Any advice or opinions would be very much appreciated. Thanks.