I have no problem with people smoking pot or taking any other drug, providing they are able to keep it within the bounds of their personal life and commitments.
What you are describing, sad to say, is someone who is addicted.
Either that or, forgive my bluntness, he doesn't love you.
I'm going to assume the former, not the latter, is true, and there is no reason why you should not also.
I'm sure he likes to smoke pot.
I'm sure he likes to eat cake or watch TV, or take a nap at 6:00pm.
The question is, if his wife, for reasons that are entirely not frivolous, asks him to not smoke pot, why won't he or can't he?
Frankly, I doubt he feels good about the situation, but addicts never do. They are also going to strongly resist admitting they have "a problem."
What to do?
I think you should discuss this with his brother or other members of his family.
He needs additional perspectives as I'm pretty certain he has rationalized away yours.
At the risk of seeming cliché, he's going to need a wake-up call.
You might be able to deliver him one if you left him, but I would suggest you first try involving his family.
Be prepared though.
Addicts don't appreciate efforts to break their addiction. Chances are pretty good that he will, at first at least, react negatively to your involving his family.
Someone (perhaps his brother) needs to break through his addictive defenses and make him realize that your needs are more important that his own.
Ultimately, your current needs are more important than any he can manufacture and rationalize.
I'm not sure that your are actually in peril from his smoking but it is enough that you think you are. You deserve his attention and sacrifice, and if he can't give it to you should consider the value of the relationship.
I'm going to be brutally frank here: You are not describing a healthy, loving relationship, and I think you know it.
By all means make the effort to set things straight, but if it doesn't work out, you can have a life without him.
God bless, heal and sustain you.