Another excerpt from Howe’s What Hath God Wrought:
But in his enthusiasm for American institutions, the incoming president got carried away. “And if we look to the condition of individuals what a proud spectacle does it exhibit! On whom has oppression fallen in any quarter of the Union? Who has been deprived of any right of person or property?” Monroe took it for granted that the answer to these rhetorical questions was negative. If someone had responded by pointing to 1.5 million persons held in chattel slavery, or to white women firmly deprived of rights of person and property, or to expropriated Native Americans, the president would have been startled, then irritated by the irrelevancy. To him and most of those in his audience, such people did not count. (p.92)
The Founders took for granted the idea that the United States was a “white man’s country.” It was not specifically legislated into the Constitution because it was uncontested at the time.