He said he was ok with casual. but didn't try anything sexual with me

This guy and I dated during the summer, but we weren't getting along. So we stopped dating and were friends with benefits for a month. he said it was over between us and that he did not want to be intimate anymore. It ended in January because, again, we were not getting along.

Recently, he and the guy had been texting. He told me he had a "spicy" dream about me. And I asked him if he wanted to have casual sex with me. He said, "That's a complicated question," and he asked me out for a casual lunch. But he said he was ok with casual as long as it's just casual.

We went to lunch yesterday he didn't mention anything about sex. He did glance down at my breasts a few times. But probably because my bra was showing as the dress I wore was too big (I've been losing weight). After lunch, he told me it was nice to see me and hugged me. I hugged him back but didn't say anything. He then texted me later, saying it was nice seeing me today. I did reply that time via text. I'm surprised he didn't mention anything sexual. He told me via text (before we met) that he was ok with casual as long as it was just casual. During lunch, he asked me how I've been doing, etc. I mentioned that I needed to find an outdoor sport to do during the summer. He also mentioned pickleball (which is a sport that he enjoys and likes to play). He knows I have never played it before, so I'm not sure why he mentioned it. Anyway, it was just odd. I was expecting him to say something sexual.
I don't get it he said he was ok with casual, talked about how he had a sex dream about me. But did not mention anything sexual when we met for lunch.