Speaker grills On or Off ?

I have a pair of KEF Reference 103s, and I read an article in a magazine a rew years ago along the lines of 'remove your grills and hear better sound.' The grills of my KEFs are held on magnetically, so I easily removed them. They are made of a sort of sponge foam, and as I removed them they crumbled to dust. Now I non-optionally listen to them sans grills.

I think the answer to the original question is to listen to speakers as they were designed to be listened to. If they were voiced with grills it seems clear to me that listening with grills is the place to start. If one subsequently comes to the conclusion that they are better without grills, then so be it.