@Loh Jane,
Basically it's all down to context, 'That is to say that the men are handsome,' could stand on its own. It merely states that the men are handsome, but, if it's on its own it would sound better with a few less words, 'The men are handsome.'
Having said that both sentences really depend on those that precede them. 'That is to say that the men are handsome,' suggests something positive has been said about the men, and the sentence reinforces that positive statement. As in the example below.
The men are lean, suntanned and muscular. That is to say that the men are handsome.
However, with the sentence, 'That is not to say that the men are not handsome,' the implication is that something negative has been said about the men, but regardless of this they're still good looking. As in the example below.
The men are rude, loud and obnoxious. That is not to say that the men are not handsome.