Incidentally, this is the same reason that West Virginia separated from the rest of Virginia to become its own state. It wasn't because the economically poor White people in the Appalachians were so opposed to slavery. They weren't prosperous enough to own slaves, and they didn't believe the rich White plantation owners in the lowlands (who ran Virginia) cared about their interests as small holders and subsistence farmers.

West Virginia's didn't want any part in a war fought to maintain the rule of plantation owners. Ironically, the war came to them anyway. Also ironically, West Virginia remained mired in systemic racism anyway.