All entries for Tuesday 11 June 2013

June 11, 2013

Needles in the Mega–Haystack: NSA versus KGB

Writing about web page http://www.bbc.co.uk/news/technology-22811580

Widespread concerns about mass surveillance in Western societies have been triggered by two revelations in The Guardian: a court order of the U.S. Foreign Intelligence Surveillance Court giving the FBI unlimited access to the call logs of the Verizon telephone network; and details of the Prism program that gives the U.S. National Security Agency – and maybe others, such as our own GCHQ, access to servers through which foreign communications pass.

Natural questions arise. Are our liberties at risk, along with our privacy? Are we moving in the wrong direction along the spectrum that runs from a free and democratic society to a totalitarian police state?

To help answer such questions, it would seem only sensible to ask how surveillance works in real totalitarian police states. The answer might give us a reality check. That comparison is what I’m going to offer. I’m going to point out some important similarities between what the U.S National Security Agency (and others) are up to and the functions of the secret police under communist rule. I’m also going to show some differences. My conclusion is going to be that we are a long, long way from mass surveillance in the style of the Soviet KGB or China’s Public Security Bureau. But that should not be completely reassuring.

Here are the similarities that look important to me:

Mass surveillance

American counter-intelligence is in the business of mass surveillance. They’re looking at everyone. Jeremy Bash, chief of staff to former CIA director and defense secretary Leon Panetta, is quoted in the New York Times as saying:

If you’re looking for a needle in the haystack, you need a haystack.

That haystack is the millions and billions of bits of our data that are being gathered. Mass surveillance was also the business of the KGB, as it is the business of the secret police under any dictator. In fact, counter-intelligence everywhere has an unquenchable thirst for personal facts. Every secret policeman knows that the most dangerous enemy is the one you don’t have on file. You can keep tabs on the ones already in the Rolodex – but what about the sleepers, the new recruits, the ones that are out there and completely invisible to you? It’s what you don’t know that can kill you. So, in the interests of staying alive you can never know enough.

Detection relies on big data

How do you find the enemy you don’t know? By using data and looking for patterns in the data. This is what the KGB did. They looked for several kinds of patterns. They were pioneers of profiling, for example. They figured that many disloyal people had markers in common, although exactly what mattered changed from one period to another. In one period it was your social origins – upper class (which meant the regime had taken your property) or poor. In other periods it was whether you had family members that had fled abroad, or you spoke a foreign language, or you had stayed behind when the war came and tried to live quietly under German occupation. So, the KGB looked for people with those markers. Another thing the KGB looked for was who knew whom or was related to whom. When they put a person under surveillance, they obsessively tracked friends and family members, telephone callers, letter writers, and so on. A third thing was just to look for unusual patterns of activity in the street and at work. To know what was unusual, they had first to know what was usual, and this in itself required data collection on a massive scale. The abnormal would stand out only against the normal. Qualitatively, this isn’t different from what the FBI or the NSA are doing. They too are mainly just looking for anomalies, or patterns of interest in the data.

The goal is prevention

The ultimate goal of surveillance is prevention. Exactly what is being prevented may vary. Most western intelligence agencies today are trying to prevent another 9/11 or its London equivalent, another 7/7. They are also trying to prevent the public from finding out exactly how they are doing this, because that knowledge might help their targets to pass under the radar. China’s Public Security Bureau has a wider set of goals: to prevent public disorder, to prevent open criticism of China’s leaders and political order, and to prevent everyone from getting the idea that open opposition could ever be normal and go unpunished. The KGB’s goals were pretty similar. To do any of these things you have to be ready to react instantly to signals that something is up. Sometimes you receive a signal, and you can wait and see how it develops. Sometimes you have to react and nip it in the bud even before you know what it is that “it” might be. To prevent the bad stuff you have to review all situations that look as if they have a potential for going bad, and consider all people that look as if they have a potential to become enemies. Identifying the potential enemies is always and everywhere a judgement call.

Risk of type I errors.

So much in this line of work is a judgement call that errors are inevitable. Some are what statisticians would call Type I errors and some are of the opposite type – Type II. You make a Type I error when you see a pattern in randomness, so for example a person has a random resemblance to a terrorist by having the wrong appearance and being in the wrong place at the wrong time, and suddenly you’ve got them on a plane to Guantanamo Bay. And then a Type II error is when you miss a pattern, or overlook a real spy or terrorist. To explain this another way, when you’re looking for a needle in a haystack, and it’s important to avoid missing it, it’s inevitable that you will turn up lots of things that might be needles because they look quite like needles and in fact you might have even stuck one in the pin cushion before you realized that it’s just a shiny thorn … and now you can’t be bothered to retrieve it. Yes, and that means that where there is scope for error there is also scope for abuse, because secret policemen are not all dedicated professionals; among them will be those that are too lazy, or too ambitious, or too much in love with power to correct a mistake. In most situations Western societies show a preference for Type II errors over Type I errors; we’d rather leave a criminal at liberty than imprison an innocent person. That’s not so hard when we’re talking about shoplifting; it’s harder by orders of magnitude when the criminal at liberty has the potential to behead a bystander or fly a passenger jet into a shopping mall.

Those are the ways in which western counter-intelligence looks very much the same as counter-intelligence under totalitarian rule. But there are also some key differences. Here they are:

Law governed and openly contested

Most obvious is the existence of a legal framework. It was not always like this but in both Britain and America the intelligence services now operate within the law, subject to both legislative and judicial oversight. The law permits some things and not others. The NSA can find out that X sent an email to Y, but it can’t read your email without a court order that names you and convinces a judge of probable cause. This framework may well look unsatisfactory, and may indeed be unsatisfactory; I’m not a lawyer and don’t pretend to know. At the same time, we also have a free press and intrepid journalists that have strong incentives to find scoops and dig out scandals. As a result, the scope of secrecy and surveillance is law-governed (although imperfectly), open to free discussion (to the extent that we know of it), and contested (vigorously and continually). If you don’t like the law you can take the contest to the polls, and do the hopey-changey thing of tossing out the law makers. Or you can take a personal stand, break the law, and answer for it in the courts like Bradley Manning (although this does not seem to be the path preferred by Julian Assange or Edward Snowden). The contrast with the situation in countries under communist rule could not be more stark. There the KGB responded only to the instructions of the ruling party (and the same no doubt holds in China, Cuba, North Korea, and North Vietnam); there was and is no answerability to the parliament, the courts, or the press. What is more, the merest mention of secrecy and surveillance was completely suppressed; the existence of secrets was a well policed secret.

A much bigger haystack

America’s haystack is of unimaginably vast dimensions. It’s so big that, according to Edward Luce in the Financial Times, it employs a data-intelligence complex with a staff of nearly a million and a budget of $80 billion. The KGB’s haystack was pretty large in its time. It was put together from many individual straws: agent reports of gossip from canteen queues and student dormitories, surveillance reports, information gathered from microphones, phone taps, opening the mail, and so forth. In 40 years the archive of KGB counter-intelligence in Soviet Lithuania (a country of around 3 million people) accumulated at least a million pages of documents. On that basis, the total paperwork of the entire Soviet KGB archive (for 70 years and a country of 200 million people and more) ought to exceed that of Soviet Lithuania by at least two orders of magnitude. And this was in a society with one landline system and one mail service, without networked computers or mobile phones, where no one even had free access to a photocopier. When even intercity phone calls had to be booked through an operator in a city exchange, it was relatively easy for the KGB to monitor anyone’s personal network. So the size of America’s haystack must be thousands of times larger than this, and probably tens or hundreds of times larger than even China’s haystack. This observation, at first alarming, is testimony to the fact that we live in a free society in which communication is unfettered and of negligible cost by historical standards. We, the citizens, are the ones that make the haystack so large by our abundant use of the freedom to communicate.

Many fewer needles

The problem of finding needles in this vast haystack is magnified by the fact that western societies do not appear systematically to produce needles – certainly not on the scale of more repressive societies. As the sociologists Inkeles and Bauer (in The Soviet Citizen, 1959) reported from the first wave of the Harvard Interview Project, the Soviet system of repression was apparently based on the assumption that everyone had a reason to hold a grudge against the communist rulers somewhere in their past. A parent had lost property, a brother had been arrested, a husband shot, a cousin’s family resettled in the remote interior. As time passed the salience of such historical events might recede, yet for some reason each new generation of Soviet-educated citizens kept on throwing up new kinds of nonconformity and outright disloyalty that had to be monitored and checked. In contrast western societies are not governed by dictators that have systematically expropriated property and penalized wide social classes and ethnic groups; they also provide multiple channels for citizens to express discontent and resentment and organize for social and political change. Despite this, there are still needles: enemies of openness and tolerance. But they are far fewer in number than the hostile forces that repressive regimes cannot help but produce and reproduce continually.

More type I errors.

You put a much bigger haystack together with far fewer needles and the implication is unmistakeable. When the haystack is small and needles are many, the chances of making Type I errors are reduced. Under communist rule, if it pricked like a needle and it looked like a needle, there was at least a good chance that it was a needle. Any western intelligence agency trying to find those few needles in today’s mega-haystack has a much reduced chance of coming up with real needles compared with their communist counterpart, and a correspondingly heightened chance of false positives. The fact that so many people are looking for the few needles, that the number of big data analysts must exceed the probable number of real terrorists by a factor of one hundred or even ten thousand, just makes it much, much worse. So you want to make a career as an analyst. How can you distinguish yourself if you never identify a threat? How can you fend off boredom if you never reach the point of saying: “This is someone we should look at more closely”? So you do it, and you make a mistake. Well, it was worth looking into. And that is most unfortunate, because as a society we want to live in safety but we also hate Type I errors. We intensely dislike the idea that an incidental bystander might get investigated, or even detained, because of an intelligence error. So intelligence errors sow cynicism and mistrust.

Now I’ll summarize. NSA versus KGB: Is there good or bad news in the comparison? To me the news looks mostly good. Compared with the KGB, the NSA looks quite benign. But there is also a warning. The warning flows from the observation that there is no limit on what our guardians would like to know about us. The more they know, the better informed they are. But the more resources they have, the greater is the scope for over-ambition, the abuse of power, and the false positives that we rightly fear. How much is enough? The purpose of national security is not to suffocate us with cotton wool. It is to enable us to be the people we would like be and to protect the rule of law that we would like to have. In a free, open society the limits of security are something we, the citizens, should always debate, contest, and, if necessary, push back.

About me: I've spent much of the past five years working with archives of the KGB of Soviet Lithuania held at the Hoover Institution Archive. This work is in a paper I have coming out soon in the Journal of Economic History and in other work in progress or under review.