No, it's not selfish of you to want it back or to ask for it.
However, it seems to be a sensitive issue for him. Or something he is not willing to even discuss and will avoid by starting a fight. Or he justs like to take and not give 'cause he is that kind of guy.
Back Off. Stop making it an issue about yourself (Oh, am I selfish, is it my hygiene, is it me me me??).
Stop questioning yourself. Instead, start looking to see what is going on with you hun-hun there.
......Can you see any possibility of <insert graphic image> soon?.....
p.s. If this is the worse of your worries, man, you've got it sweet!