Anyone know author or title of this book?

I read a book years ago wanted to recommend it to someone but could not remember the title or author. It was set in a small southern town.
The main character was a young woman. She was engaged to be married to an idealistic young architect (from Atlanta I think) who wanted to design affordable low income housing. He was consumed by his work and had neglected her romantically. The town had many small town type characters. The young lady was attractive and kind and a young, retarded boy had become infatuated with her and followed her around constantly somewhat to her chagrin though she was aware he only desired to look out for her and protect her if necessary. Her grandfather was wealthy and powerful and ran the town. He, for reasons I don't recall, did not want her to marry the architect and secretly hatched a plot to break up the relationship. He hired a young, handsome "cad" a con-man type to attempt to seduce her away from the architect.
If you know the title or author of this novel please post it.
It would have been published at least 20-30 years ago.
Thanks