The Keepers of the House by Shirley Ann Grau
Entrenched on the same land since the early 1800s, the Howlands have, for seven generations, been pillars of their Southern community. Extraordinary family lore has been passed down to Abigail Howland, but not all of it. When shocking facts come to light about her late grandfather William's relationship with Margaret Carmichael, a black housekeeper, the community is outraged, and quickly gathers to vent its fury on Abigail. Alone in the house the Howlands built, she is at once shaken by those who have betrayed her, and determined to punish the town that has persecuted her and her kin.