Years ago, I worked on a closed loop monitoring system, and one of the parts of the design was an analysis of the data that was considered not relevant, looking for data that is mistakenly discarded.
A friend sent me CIA publication that discusses catching the outlier ideas, the stuff that gets discarded.
Hunting for Foxes
Capturing the Potential of Outlier Ideas in the
Intelligence Community
Clint Watts and John E. Brennan
Outlier:
—A data point far outside the
norm for a variable or population;
—An observation that “deviates so much from other
observations as to arouse
suspicions that it was generated by a different mechanism”;
—A value that is “dubious in
the eyes of the researcher”;
—A contaminant.
Source: J. Osborne, “The Power of outliers
In war you will generally
find that the enemy has at
any time three courses of
action open to him. Of
those three, he will invariably choose the fourth.
—Helmuth Von Moltke
Here is the main point made.
Of all the examinations of
intelligence surprise and failure, Richards Heuer provides
perhaps the most succinct characterization of the problem:
Major intelligence failures are usually caused
by failures of analysis, not
failures of collection. Relevant information is
discounted, misinterpreted, ignored, rejected,
or overlooked because it
fails to fit a prevailing
mental model or mind-set.
The trouble about filters is you filter out good data with the bad data.
Which brings up the issue of Outliers.
Outlier:
—A data point far outside the
norm for a variable or population;
—An observation that “deviates so much from other
observations as to arouse
suspicions that it was generated by a different mechanism”;
—A value that is “dubious in
the eyes of the researcher”;
—A contaminant.
