Difficult loads are NOT the ones that go to low impedance, but those who go into low impedance and have a severe phase shift, like f.i. the well known B&W 802D. In these cases you can force the output devices to sustain more than 5 times the normal dissipation, and leed them to clipping, operate in non linear way. If you have some output impedance, that will reduce the efect of the phase shift...
If an amp is linear enough, it will sound better without feedback, anytime. I gave up feedback since I studied it at university, specially when using opamps (you could have a bandwith as short as 100Hz and expect it to be linear by increasing the input (via the feedback) above the 100Hz till 100kHz. All you did was saturate the amp with the feedback signal, but a steady safe sinusoide would look great...
Feedback is a must in a poorly designed amp, but an evil in a properly designed one.
Distortion can mean a lot, but just consider this, our own ears introduce a lot of harmonic distortion (above 2%), but our brain can correct it so you listen clear. Phone signal is band limited, chopped, distorted, etc.. but you recognise most of the times who is calling just by the voice.
Our brain has the ability to overcome and compensate a lot of the sound problems, so if you don't feed your ears with problems your brain cannot compensate, distortion is not big issue (although I agree that no distotion is better than some distortion, any time)