It IS difficult to figure out. Men don’t touch each other very often except in sports like wrestling, football , or some other sports. They touch each other after the sports are played with hugs and slugs and grabbing guys around the waist or pushing and shoving or hitting each other. Men hug each other but not for very long. Little boys will hang on their father and hug and kiss them but that doesn’t seem to last very long. Unfortunately there’s a stigma associated with men showing affection to each other.
Now girls are different. Girls hang on each other and touch each other all the time, so do women. They get very personal with each other, touching each other‘s clothes and hair and back size and nobody thinks about it. They sleep together and cuddle often. But when men do these acts ( personal touching, cuddling), it may be considered perplexing.
I don’t have any advice for you except that if you feel this may be moving into a sexual area with your friend and it bothers you, then stop setting yourself up in the situation where it might happen. Wrestling or some other sport may offer you opportunity to touch each other. But feel free to hug or put your arm around his shoulder, but putting your hand on his private parts takes this into another area. (He may have other things in mind, however.)
Your wife may think it’s funny now, but I doubt if she would like it if it moved into an affair ( or she may have other things in mind)