is it worth the pain

Have been dating for almost a year. Its a good relationship in most aspects. We do alot together , enjoy our time. My kids love him he loves them. We dont live together, not a big deal. Im a single mom and he is disabled. We both live with our parents. We dont have sex. I can deal with that. But sometimes i would like to be held while i sleep, be able to wake up and kiss him good morning. When i ask to stay over, or ask that he stay with me, he usually says maybe, then stays over late... later than i would like to stayup, then announces he is going home. Or if i have asked to stay over he says no flat out. Its starting to have a very negative affect on my self esteem. I understand his disability makes sex not possible at times and i am willing to forgo that part for him. But not even wanting to share a bed with me makes me feel i am wasting my time. And that i wont have a future with him. I want to and he says the same but i cant get past this. I have talked with him about it and said it makes me feel unattractive and bad and rejected, he apologizes, and then it just happens again i dont know what to do im feeling resentful and wanting to push him away , and feeling like i am giving a hell of alot up for very little. What should i do