I don't know if I'm doing it quite right, but in my own writing I try to say Black people when I mean Black people, and to use "people of color" when I mean (or also mean) other marginalized ethnic groups, usually taking care to specify what I'm talking about.

Several times I've found myself writing "Black people and other people of color." I try to avoid using BIPOC, which would be a useful acronym, but which I fear is not widely understood outside communities interested in social justice activism.