Well then, I'd worry more about the effect of the filter on the flatness of the frequency response for the reason below. [Maybe your amp should do over-sampling to simplify the  filter design  - sorry, I couldn't resist..]

You're merely asking me to repeat myself

In a minimum phase system, if you have a flat frequency reponse, there won't be any time anomalies...

On the other hand, if I were dealing with a non-minimum phase system (and loudspeakers with multiple drivers fall into this category), I would aim for a flatter frequency response at the expense of the group delay rather than vice versa. This is because the ear is sensitive to ampitude response, but not phase relationships, so it makes sense to optimize for frequency response.