Did you use “mentally challenged” as a way to call someone stupid or were you talking about someone with a learning difficulty or developmental disability?
Most of the time if you cause offensive with your words, it’s not the specific words as much as the tone, context and implication.
I grew up in australia which has a very different understanding of offensive/acceptable language. Before i say something that might be touchy, i mentally double check - is this information actually relevant? Am i saying this in good faith?
With a humble tone and a clear reason the worst response i’ve had was a polite “oh i think X is a better word, but yeah”. Speak in good faith, people will respond with grace.
Given the history of racism and colorism in the US, reducing a person of color to just their skin color can sometimes cause upset. Using a non-skin-tone related descriptor is generally safer, if possible.