Having just finished reading Dick Clarke's book, it's very obvious that the al Qaeda threat was ignored for many years. Clarke suggests that this is evidence that the US administation under-estimated them, but the fact is that they were largely an irrelevancy for a long time - a small group of violent fanatics, yes, but with minimal support amongst the wider Muslim world.
What's changed that? Why do al Qaeda appear as freedom fighters to large numbers of Muslims who wouldn't themselves commit acts of violence?
The answer is politics. Palestine, Afghanistan, and, now, the stupid Iraq war. If you want to inflame a holy war between Muslims and non-Muslims, you do what Bush did, and invade Iraq (a country that, as Clarke makes clear, the administration knew offered no terrorist threat to the US), instead of concentrating on stabilising Afghanistan.
Violent lunatics will always exist, in a tiny minority, whether they be Christian lunatics like the Oklahoma bombers, or Muslim lunatics like al Qaeda. The lunatics only gain genuinely popular support when the political issues that cause disquiet are left unresolved. Sort out those issues equitably, and the popular base for al Qaeda will melt away.
-- Ian