Short answer, yes! You should have sex with him. In middle age, casual relationships are normal. Go for it.
You could ask him directly. I would say yes if you are reasonably attractive and I weren't dating someone else, but it would be awkward and seem contrived.
Why not just have a normal fling? You flirt with him, make sure he knows you are interested. Have dinner, and then invite him to your place or ask him to invite you.
Sex should be fun. We all understand casual relationships, and we are all playing the game. Just go for it.