Improving intelligence collection, coordination and analysis has been a major focus for Western governments since the Sept 11, 2001 attacks and the 2003 Iraq invasion, events involving profound faults in preparedness.
Here are some of the challenges, and solutions, being addressed by intelligence service managers.
A changing threat.
Al Qaeda has been a big change for the services, created in a previous era essentially to fight bureaucracies like themselves. A lack of al Qaeda hierarchy and the brief lifecycle of sub-groups acting without central direction make these networks hard to penetrate.
The moral high ground.
If spies want to recruit good sources, the pool of potential sources must believe the agency they will help is worthy of the risks they take, former British Secret Intelligence Service head Richard Dearlove and co-author Tom Quiggin wrote. For more, click here.
Explosive growth in open source information from the internet and new media, more active covert collection and the blending of overseas and domestic intelligence streams puts great pressure on analytical capacity, according to Jim Judd, former Director of the Canadian Security Intelligence Service.
Consultant Kevin O'Brien says information deluge "creates the danger of over reliance on the technology to do the so-called thinking for you." For more, click here.
Thomas Fingar, a former U.S. Deputy Director of National Intelligence for Analysis, has praised the impulse among younger analysts to collaborate and conduct peer review. "For them it is the natural way to operate." He has said older spies have seen this impulse as "somewhere between" heretical and impossible.
Jonathan Evans of Britain's MI5 Security Service said in January the average age of his staff of over 3,000 was under 40. Fingar said in 2008 that 55 percent of U.S. intelligence community analysts had joined since 9/11, adding there was a significant lack of mid-career analysts. For more, click here.
Writing on the U.S. intelligence community, Douglas Hart and Steven Simon have said it is "saddled with large numbers of new recruits who are, on average, ill equipped to manage the complex analytical demands posed by a new, highly distributed and strongly motivated adversary operating within a framework of values, beliefs and experiences alien to the average American." For more, click here.
Culture: Cathedrals versus Pancake People
Tracking a leaderless foe which frames its cause in religious terms requires rare cultural sensitivity and language skills, say analysts. Some fret that web-surfing clogs minds with data shorn of this context. The educational ideal -- creating a "complex, dense and 'cathedral-like' structure of the highly educated and articulate personality", in the words of U.S. playwright Richard Foreman -- has given way to the reality of `pancake people', youths whose knowledge is spread wide and thin, wrote Nicolas Carr. This view is dismissed as a caricature by many in the IT community. For more, click here.
More scope for frontline agents.
According to Dearlove, frontline personnel and mid-level managers must be allowed to cultivate their own relationships and methods to track structureless groups, with best practice then shared with other teams. The brief lifecycle of some militant cells means a source will frequently be of use only during planning of one attack, when in the past sources could last several years, so recruitment must be constant and dynamic.
Online Team Working
To prod analysts into sharing information and peer reviewing their work, U.S. intelligence is developing areas of secure cyberspace where work in progress can be posted for discussion, virtual analytical teams set up, or information retrieved across different agencies. These portals include A-Space, Intellipedia and the Library of National Intelligence, variously featuring wikis, blogs, social networking, RSS feeds and content tagging.
Commenting on Intellipedia, Fingar has said: "It's not anonymous. We want people to establish a reputation. If you're really good, we want people to know you're good ... If you're an idiot, we want that known too."
More openness has been forced on spies by public inquiries, court proceedings, the media, greater use of freedom of information law and a desire by politicians to respond to public concern. Openness helps build community support, but also raises the issue as to what is legitimately secret and what is not.
Joint Training Across Agencies
The distinction between domestic and foreign intelligence is increasingly blurred, and so joint training courses are needed for intelligence, police and military personnel. "It may have been a form of heresy to state this five years ago," according to Dearlove and co-author Quiggin, writing in 2006.
Global Futures Forum
The GFF is a U.S.-backed, global intelligence networking forum that gathers government and non-government experts from more than 30 countries to discuss transnational threats. Forum members meet in small community gatherings in the United States and abroad, in larger annual forums, and on a password-protected Web site that enables online conversations around the clock.
Apart from the GFF, analysts must continuously assess where the best knowledge and expertise on a given topic is ÃƒÂ‚Ã‚Â– and it's increasingly likely it is not resident within the services, but in a company or university. Traditional services now really don't have a monopoly, or possibly even majority share over, intelligence in the way that they used to, says O'Brien.