Exactly. It's like this quote from Somerset Maugham.
"When people say they do not care what others think of them, for the most part they deceive themselves. Generally they mean only that they will do as they choose, in the confidence that no one will know their vagaries; and at the utmost only that they are willing to act contrary to the opinion of the majority because they are supported by the approval of their neighbours. It is not difficult to be unconventional in the eyes of the world when your unconventionality is but the convention of your set. It affords you then an inordinate amount of self-esteem. You have the self-satisfaction of courage without the inconvenience of danger."
Everyone has their limits as to how far they'll stray from
everyone they know of or talk to though how far that is varies greatly from one person to the next. At some point, if they're left unattended, our lower Maslovian needs start getting in the way of what we might otherwise do. But that's the whole point here. People can get their validation, understanding, and acceptance from places and people they'd never meet otherwise and thus find the fortitude to continue pursuing unusual or specialized interests that could easily be lost otherwise.
I'll grant that there's still the issue of making the things you come up with work for you, personally, but that would be true regardless.