Phil,

Let me add a couple of comments to your post.

1. Information Sharing and US Persons.

The critical part of your first quote block below is:

Under American law, United States citizens and green-card holders may not be singled out in intelligence-collection operations by the military or intelligence agencies. That protection does not extend to visa holders, but Mr. Weldon and the former intelligence official said it might have reinforced a sense of discomfort common before Sept. 11 about sharing intelligence information with a law enforcement agency.

This is just flat-out wrong. First, the "law" cited is Executive Order 12333, which defines the Intelligence Community and its authority to conduct operations. The most important provision of EO 12333 is its rules for when the IC can collect information on "US Persons." This is a specific definition and applies to a) US citizens, b) Permanent Resident Aliens (green-card holders), c) un-incorporated organizations composed by a majority of a) or b), or d) US corporations not owned by a foreign government.

The general rule for EO 12333 is basically, "thou shalt not collect information (that is, spy on) US Persons, except . . ." The "except" portion is critical. There are 13 exceptions under which an intelligence agency can collect information on US Persons. These include for personnel security investigations, for administrative purposes, when the subject gives consent to collect. The two most important categories are for Foreign Intelligence purposes (that is, collecting information on US Persons who are agents of a foreign power) and for Counterintelligence purposes. The Counterintelligence exception also includes collection for counternarcotics and international counterterrorism purposes. It also allows for collection of not just individuals reasonably believed to be engaged in international terrorism activities, but also collection of information on people associated with individuals reasonably believed to engaged in international terrorism activities (for the purpose of determining the relationship — if no significant relationship, then the info is destroyed).

All of this falls under the rubric of "intelligence oversight", which is a well-ingrained program within the military intelligence community ensuring that collection (especially HUMINT) activities did not retain information on US Persons without authority.

So when someone says that the military couldn't share information because of rules against collection of US Person information, that is not an accurate statement. Even if Mohammed Atta was a Permanent Resident Alien, the Intelligence Community was free to spy on him, collect the information, database it, and use it in intelligence reporting community-wide. The excuse doesn't make sense logically, either: How could have Able Danger conducted intelligence collection, using Army intelligence resources at LIWA have built the briefing to begin with? Once the information has been properly collected, it can be shared (theoretically). The reason the information was not forwarded probably had more to do with the infamous "wall" created by Justice Department's misreading of the Foreign Intelligence Surveillance Act, fixed by the Foreign Intelligence Surveillance Court of Appeals and the Patriot Act.

2. Datamining.

When one talks of datamining, they need to be clear on what exactly they mean. All datamining is is a set of tools used to dip into databases which can sort through the data and provide the answer to the query. Think of Westlaw or Lexis/Nexis on speed. The controversy in TIA was created over the types of databases they were going to search. DARPA was talking about applying the analytical toolkits to civilian business databases such as credit cards, hotel bookings, rental cars, etc. Even then, there were rules in place (using EO 12333 as the foundation) to ensure that the databases were not being queried without a legitimate purpose.

However, TIA was killed in a spate of misinformation. Before its death, these toolkits were already being used and improved by the Information Dominance Center at Fort Belvoir, VA. The significant difference between TIA's goals and the IDC's actual practice is that IDC is only analyzing databases of Intel Community information already collected.

Here's how IDC (at this discussion level) works: if Army intelligence has a reasonable belief that I am either engaging in or supporting international terrorist activities (or had a relationship with those engaged in the same) based on information that I had attended a mosque with a known terrorist and/or had made pro-terrorist statements, the first thing Army intel is going to do is to "tip" its own databases and see what has already been collected on me. Low and behold, they find a report written about a Gitmo detainee, explaining the pocket litter when he was captured. In his pocket litter was my business card. Now analysts can build linkages and determine relationships between me and others in the database to see if there may be other linkages out there that look innocuous at first, but when compared with other pieces of information show patterns and indications of terrorist activity. And the best thing is that the database check is multi-INT, tapping into SIGINT, HUMINT, and other sources of information. What happens with this information? Do the links mean anything? Maybe, maybe not. It's usually not enough to create "actionable intelligence". But what it does is become used to determine whether further investigation is required or whether the linkages are just coincidental. It is also used to build the bigger intel picture of the hydra that is international jihadist terrorism.

If I could actually explain how the datamining toolkits work, you would be amazed at their speed and accuracy in pointing out links and indicators (think Minority Report without the precogs). Of course, with great power comes great responsibility, but after the TIA debacle, the owners of IDC have ensured that its operations receive the highest levels of oversight and scrutiny. In the IDC operations I was involved in, each US Person database query had to have legal counsel approval.

The defense contractors involved in developing the datamining software are interested in taking it civilian, so you may see variations of the technology on the market soon. For instance, it would very helpful in complex litigation, such as tobacco or asbestos litigation, where discovery leads to thousands of documents and reports.

There's a longer discussion about whether information already databased by the intel community is already "collected" for purposes of intelligence oversight and therefore can be queried using analytical toolkits without worrying about collection exceptions. This will be for another day.

UPDATE:

One other issue that doesn't pass the smell test is the reluctance on the part of the military to pass information on terrorist threats w/in the borders over to the FBI. There is a 1979 document called the Delimitations Agreement, between the DOJ and SECDEF which outlines investigatory authorities for counterintelligence (and by extension counterterrorism) cases between the Bureau and military CI agencies. The bottom line is that DOD does not have jurisdiction to investigate civilians in the United States without FBI coordination. They can investigate civilians overseas who may post a threat to DOD personnel/installations, but once bad guys pass over into the States, FBI retains investigatory jurisdiction.

Able Danger appears to be the codename to the forerunner of the Information Dominance Center, or at least IDC-type tools were being applied to the Able Danger mission, but analysis presumes collection, or investigation. If Able Danger was conducting an investigation in to Atta et. al., it would have no jurisdiction to conduct collection or investigative activities in Brooklyn — that would fall under FBI.

To say that the "wall" prevented DOD from talking to FBI doesn't make sense. The "wall" existed between FBI-Criminal Division and FBI-Intelligence Division and was managed by the DOJ Office of Intelligence Policy and Review (OIPR). DOD was free, in fact required, to discuss counterintelligence and counterterrorism investigations taking place in the United States (or even taking place outside the United States if they involved US persons or potential prosecutions) with FBI-Intelligence officials.

It sounds more and more like the NYT and possibly Weldon were fed a stinker.

UPDATE 2:

Some have suggested that DOD may not have shared Able Danger information with the FBI because it was a JCS Special Access Program (SAP) with a limited number of people being able to get read on*, or that DOD lawyers felt that FBI didn't have the clearance or compartments required to receive the information. That doesn't wash. SAPs don't have any magic exemption from the ban on conducting intelligence collection inside the United States w/o FBI coordination (Posse Comitatus, anyone?) and you would think that this information would be important enough (individuals in the US, preparing for some sort of terrorist activity) to read on at least the Chief of the Intel Division or Louis Freeh. If it was a mere policy decision based on sensitivities or appearances, then someone in the Pentagon screwed up in a big way.

*fun piece of interesting trivia — the list of who gets access is sometimes called a "bigot list" and people who are on the list have been "bigoted". Sometimes its good to be a bigot.