@ajohn,
Yes taking care of a new born can be a challenge - you are tired a lot. Bringing to grandparents house and working and taking care of house chores can be tiring.
Can you also have grandparents watch the new born for a few hours on the weekend and you and your spouse go out to a movie, dinner, or just a walk by yourselves?
Here is another thought if you can't - can you go out to dinner or a movie with your newborn? How cranky is the baby? We used to go out to dinner with our baby all the time - she was pretty easy going so we would just bring her where ever we wanted to go -- you need some play time. A new born sleeps - so if you have an easy baby you simply go out; keep the baby in the carseat while you enjoy yourself. Other ways - put the baby in a snuggly and go for a hike.
To be honest with our first, our playtime didn't change much. Yes we were tired during the week with work and taking care of the baby, but the weekends we just packed her up and brought her where ever we went.