I would never want to 'listen' to stereo music (ie, non-surround-sound content) in speaker 4 mode. For parties, yes, and for background music, absolutely, but for true 'listening' - never.
Why? Well firstly the sound is arriving at your ears twice. Usually, the rear speakers are closer to your ears than the front ones, so the sound from these will reach your ears first. This 'smears' everything - recording engineers go to great length to make sure the sound from each instrument isn't arriving at loads of microphones at different times in a disturbing way (bear in mind sound actually travels quite slowly!) so you are presented with the 'truest' sound when sat in front of your speakers. Playing through 4 speakers completely destroys this!
Secondly, you lose any sort of stereo-image with speaker 4 mode. Stereo sound is formed by two sound points (speakers in this case) both reaching your ears, with the sound from the left speaker reaching the left ear first, then the right ear, and visa-versa for the other speaker. Adding in a second pair of speakers (which are often positioned less than optimally) and you start to get the idea - the more speakers reproducing sound, the more 'diffused' and less 'defined' your sound becomes!
Ultimately, it's all down to the discretion of the listener. Listening to stereo through 4 speakers would be absolutely laughed at by most people in the professional/hi-fi world though - it's a very 'multimedia' thing to do, with all except the most ambient of music (think Brian Eno).