Why Hunting For Terrorism With A Data Panopticon Doesn’t Work

data miningFrom seven years ago, Bruce Schneier explains why collecting massive amounts of data won’t allow us to find terrorist patterns hidden within:

Many believe that data mining is the crystal ball that will enable us to uncover future terrorist plots. But we’re not going to find terrorist plots through systems like this. We’re not trading privacy for security; we’re giving up privacy and getting no security in return.

Data mining works best when there’s a well-defined profile you’re searching for, a reasonable number of attacks per year, and a low cost of false alarms. Credit card fraud is one of data mining’s success stories: all credit card companies data mine their transaction databases, looking for spending patterns that indicate a stolen card. Many credit card thieves share a pattern — purchase expensive luxury goods, purchase things that can be easily fenced, etc. — and data mining systems can minimize the losses in many cases by shutting down the card. In addition, the cost of false alarms is only a phone call to the cardholder asking him to verify a couple of purchases.

Terrorist plots are different. There is no well-defined profile, and attacks are very rare. Taken together, these facts mean that data mining systems won’t uncover any terrorist plots until they are very accurate, and that even very accurate systems will be so flooded with false alarms that they will be useless.

There are trillions of connections between people and events — things that the data mining system will have to “look at” — and very few plots. This rarity makes even accurate identification systems useless. Terrorist attacks are rare, so any “test” is going to result in an endless stream of false alarms.

, , , , , ,

  • Liam_McGonagle

    “Data mining works best when there’s a well-defined profile you’re searching for, a reasonable number of attacks per year . . . ”

    How sad that supposedly grown-up Americans need to be told this explicitly. Yet I suppose that is the way of people gripped in pants-wetting terror.

    I remember expressing the author’s sentiment to a coworker once a number of years ago and was met with the following response:

    “That’s okay for you to say. But I have kids and family responsibilities. If you died I’m sure nobody would even notice you were gone.”

    Takeaway: You can’t reason with terrified people, so don’t even bother to try. You’ll only demean yourself.

  • Ittabena

    The premise of the article seems to be that since they say they are data mining to stop terrorists, we believe them. Most of us already know that is not why data mining is in place. Preaching to the choir…

  • BuzzCoastin

    the “data mining” of Uncle Homeland
    is primarily designed to funnel tax dollars to the Carlye Group
    and it’s subsidiaries
    then get financial “intelligence” for use in the stock markets
    next comes fencing the sheeple with data mines for easier fleecing

    • gustave courbet

      Good point, but don’t forget about the potential for blackmail as well. If you know who politicians are porking on the side, you own them.

  • rtb61

    Actually the real danger is straight up laziness. As those responsible for the security of the nation really more and more on computers to do their work for them so they will fall further and further behind on normal investigatory skills.
    As the opposition switches tactics and start to rely on manual operations more and more, so they will become invisible to agents glued to their monitors.
    The age of digital misinformation will grow, with carefully groomed online lives being created and maintained, lives that the lazy agents will carefully follow and believe, believing of course because the computers say it’s true.

  • marvin nubwaxer

    it’s a government program that costs too much and returns very little.