I realize you know the things I'm about to say, but maybe hearing them from a 3rd party.....
Even if you decided for sure you'd like more children, that's no guarantee you'll even get pregnant. Sure, you may have had no problem in the past, but you may have been lucky. What if you did have 1 child, and a 2nd pregnancy just never happens? What if one pregnancy never happens?
Also, there's nothing magical about turning 35, like all of a sudden Boom!, you're high risk. Sure there might be a steep decline in fertility, or increase in problems, but one can't say being 36 is somehow innately dangerous.
I've always been skeptical about statistics and the way they are phrased to make one either complacent, or terrified. For instance, I just looked up the incidence of Downs at 35, and 40. It sounds very scary to say at 30 you have a 1 in 939 chance, at 35 a 1 in 353 chance, and at 40 a 1 in 85 chance. But, if one were to say a 40 year old woman has a 2.5% chance of having a Downs baby, it just doesn't sound so bad.
There's actual risk, and relative risk.
What if you don't become pregnant, or have a miscarrage, or any other number of things? Would it end your relationship? If you didn't have more children, would you have a happy life? Would the 2 of you? Would having more children make you a happier person?
There's always adoption you know, if you don't want to bring any more people into the world. Not saying this about you, have no idea, but I find it funny when people believe adoption is a great thing.....for someone else.
One thing I don't get is your partner saying "it's not fair" re having a kid with him. It's not fair? Would he love his own fruit of his seed more? Would it be, even slightly, a "mine, and them"? He really has been great for you, you have a good life. If you two want another child or children, and can care for them, I hope you have them.