I listen to many books on tape while I travel for work and I can say that the change in voices often pulls me out of the story.
There is a rhythm that flows when a single reader is telling the story. I am much more likely to listen to a man read rather than a woman, but since the ratio of male to female is pretty low I don't see a ton of men jumping in to read fanfic.
All that said I would listen to a story read if I liked the story and the reader has a pleasant voice.
