Go back with 100 years and that’s exactly what you would hear it called. It is one of those insult terms like “God particle” or “big bang” † that were created by someone to mock something and it ended up becoming the actual name. Add in the usual vowel shifting and consonant drift that occurs over time and rabbit becomes rarebit.

† apparently I only know physics examples