Of course it is racist!!

If the reason was anything to do with the colour of the skin, then yes. You do not have to be in the minority to be a victim of racism.