Communication is really damn difficult, no matter where you stand on it. And you are completely justified in asking people to back off when you don't feel like talking. Frankly, I would rather people feel comfortable enough to tell me to leave them alone when they're not feeling social, because I tend to be oblivious and not always pick up on those cues. It's a respect issue, really.

Also, I am echoing what Juri said about how you shouldn't have to justify your love for anything.