Oh! That information changes everything.
You two once had an "open" relationship, now partner has decided that's not for them and wants you to stop, too.
Open relationships within a marriage (or other relationship) takes a mutual OK. It's now not OK.
You don't say how adamant this partner is with this "rule" placed on you. Is it a deal-breaker for your partner? Have you invited your partner into the relationship, for a 3-some?
See if you can live with what you think is a restriction. If not, separate.. It's not fair to either of you.
Also, consider if this side attraction is a "forbidden fruit" that you are not allowed to have, thus it gets more attention than it really warrants. You may lose something really valuable just for a fleeting fantasy, just based on sex.