The terms "people of color," "person of color," etc., have a rich history:

From The American Gazetteer by Jedediah Morse (Boston: S. Hall [et al.], 1797):

In 1786 the numbers were 16,167 whites; 838 free people of color, and 62,115 negroes.
...
The small provision farms called Conacos, fall generally to the lot of the poorer colonists, or most commonly people of color, or freed people.
...
Before the late revolution, there were in these parishes about 42,000 white people, 44,000 free people of color, and 600,000 slaves.

From A History of Methodism in Alabama by Anson West (Nashville: Anson West, 1803), page 598:

In the list of the appointments of the Tennessee Conference made November, 1831, and made then for the next year, is found the following:

"Thomas M. King, Missionary to the people of color in Madison and Limestone Counties, North Alabama.
Gilbert D. Taylor, Missionary to the people of color in Franklin and Lawrence Counties, North Alabama."