On a literal basis it is racist because you invoke race. On a logical basis it is racist because you ignore context and treat the use of a word based on the color of the skin of the person using it. On a moral basis it is racist because one group getting offended is not any more or less valid than some other group getting offended.
Most importantly, the fact that the word is offensive to someone does not invalidate the racist double-standard. That's what context means: that the use of a word or phrase has multiple properties. On the one hand, use of the word by a white person is offensive. On the other hand, saying a white person can not use that word is racist. Both of these things are true at the same time.
You can attempt to gloss over these facts by saying "oh, historical use" or "oh, it's offensive" and many other reasons, but that does not remove the racism inherent to the double standard which is based on the color of the skin of the speaker. If the word was not offensive, it would still be a racist double standard.
You need to check the definition of racism. It's not simply discrimination on the grounds of race, it includes that there is a belief in the superiority / inferiority of the race(s) under discussion. Maybe it's just me but I really don't see how that applies here. No one is saying you can't use the word because only black people are good enough to use it so on a logical basis, no, it's not racism, at least not by the real definition.
In terms of the double standard, it's not a double standard at all, it's a standard that exists in all conversations which is that the context and the relationships and history between the two people involved matter. I can say things to my daughter which would be massively offensive for you to say. I can insult my best friend using an unambiguously obscene word and he'll laugh at me, you do it and he'd likely swing for you. In both these cases it's entirely clear that the context is important so I don't see why these are fine but it becomes a double standard to invoke context elsewhere.
These are very, very old arguments that have been attempted time and time again but they simply don't hold water.
To be clear you're drawing a comparison between a white person getting offended by not being allowed to use an offensive word, and a black person getting offended by a word which has strong connotations to literally generations of oppression including but not limited to being bought and sold as property?
Most importantly, the fact that the word is offensive to someone does not invalidate the racist double-standard. That's what context means: that the use of a word or phrase has multiple properties. On the one hand, use of the word by a white person is offensive. On the other hand, saying a white person can not use that word is racist. Both of these things are true at the same time.
You can attempt to gloss over these facts by saying "oh, historical use" or "oh, it's offensive" and many other reasons, but that does not remove the racism inherent to the double standard which is based on the color of the skin of the speaker. If the word was not offensive, it would still be a racist double standard.