It's all about how many other people share your belief.

If it just you or a small handful of people, you're crazy or eccentric. If it's a few hundred you start getting into the cultist region. Pass a few thousand and you become religious.

Yes, those number were pulled out of thin air, but the sentiment remains.