People have said this to me and whenever it comes up I find myself a little put off by it. I'm not "offended" per se and I don't believe that I'm voluntarily making the decision to be put off by it. However, it just seems weird to me that this person automatically assumed that the brown people country I grew up in couldn't teach me how to properly communicate using one of the most important languages in the world.

I definitely agree with you that it's wrong to call it an aggression and there definitely shouldn't be any rules about saying something like that, but I do think there is something wrong with going into a conversation assuming that the other person does not possess a skill because of their ethnicity.