You do realize that he only gave you his phone for a limited purpose and you overstepped that, right? And do you also realize that he has a right to privacy? And that he might have fantasies which - shock, horror, don't include you, but he does nothing beyond looking and thinking?
If he is demanding all sorts of conditions on your use of your phone, then I suppose that's one thing. But if that is not the case, he's kinda not the person in the wrong here.