Writing fiction is work. Being told "oh you're just writing your fantasies/neurosis/fears" is denial of said work.
We're not writing for ourselves, else we'd be journaling. We're writing to get to *you*, and that requires getting out of the self.