Exactly, Tejasvi. And in Mexico, it’s very common for people to refer to one another by the color of their skin. “Negro” meaning “black” in Spanish, you often hear people of Mexican heritage in the United States use the word as a nickname.
A friend of mine used to call his son Negro all the time.
My Mexican American boyfriend called me güero all the time, affectionately referring to my light skin color.
That kind of reference to skin color would be at least tone deaf, and probably worse, in general American culture.
So, yes! Cultural connotations matter a lot.