Bulk Data Collection

Massive government data collection programs with intimidating names such as “Prism,” “Magic Lantern,” “Stormbrew,” “Bullrun,” and “Carnivore” may be the stuff of recent headlines, but their history stretches back to before the beginning of electrical forms of communication. There was a time when governments routinely collected postal and telegraphic communications; after all, governments provided those services. Telegraph service in most countries (with the exception of the United States, where private telegraph companies operated) was provided by government offices of posts and telegraphs from the 1870s onwards. Initially, the privacy of electrical forms of communication was not protected by the same laws that protected letters sent through the mail. For example, the Indian Telegraph Act of 1885 gave any officer of the British government total powers to intercept or detain any message relating to any subject from any person in the interest of public safety. Also, although letters had to be steamed open, or their wax seals painstakingly lifted, in order to be read, contents of telegrams were voluntarily revealed by the sender to the telegrapher in order to be sent. The telegraph providers kept copies of messages in case of garbles or other situations requiring resending or comparison, thus the contents of most of the world’s telegraphic communications were already in the offices of government, and those in the possession of private telegraph companies could be collected— relatively easily—by an investigating authority.

Electrical forms of communication had made data collection by governments much easier than in postal form. An example from Hapsburg Vienna—the efficient Geheime Kabinets-Kanzlei—shows how paper correspondence was monitored in the 1770s. The GKK went to great lengths to avoid delaying the intercepted mail and thus tipping off the recipients that their correspondence was being monitored. Letters to addressees of interest (primarily foreign diplomats) were diverted to the GKK , where the seals were lifted (this was a painstaking and not always successful operation; the “hot knife” method required a deft hand and getting the temperature of the knife right), the contents were copied, and the letters resealed and returned to the post office for delivery. The telegraph bypassed much of that effort.

In the months prior to the American Civil War, the U.S. military gradually increased its control of the private telegraph lines “to stop all messages of a disloyal character” especially purchases of supplies that might aid the secessionists in the South.

During the impeachment proceedings against President Andrew Johnson (March to May 1868), a member of the Impeachment Committee —Massachusetts Representative Benjamin Butler—engaged in what in today’s terminology would probably have been referred to as “bulk data collection.” Private detectives in Butler’s pay collected copies of thousands of telegrams transiting the offices of telegraph companies in Washington, DC and Baltimore, MD in his quest for evidence.

Telegraph customers—perhaps like today’s social networking users—had little expectations of privacy. Commercial customers in particular developed their own codes, not only to protect the contents of their messages from competitors, but also to compress their messages and save money on word counts. The telegraph companies and post and telegraph services themselves developed commercial codebooks as services to their customers to consult at the offices when writing up the telegraph forms. During World War I, copies of many of these commercial code books had to be filed with the General Post Office, and eventually the United States and Great Britain forbade the use of codes in telegrams. Without stretching the historical parallel too far, one could see this regulation as the 1900s precursor of the government-mandated “backdoor,” or of the Clipper chip — the device that the National Security Agency developed for use in all communications devices to allow the NSA to unlock encrypted communications.

Radio made data collection even easier; any government or person with the right receiver could listen to any transmission. Encryption was vital in this new world of electrical communication.

World War I brought governments, especially the government of Britain, into the data collection profession on a grand scale. On 5 August 1914, one day after Germany’s invasion of Belgium, and of Great Britain’s declaration of war on Germany, the British hauled the German transatlantic cables running across the bottom of the North Sea to the surface, and cut them. This meant that Germany had either to send its transatlantic traffic—including communications to its embassies in the United States and Mexico—via the cables of neutral nations (e.g. Sweden and the United States), or via radio, where anyone with the right receiver could intercept them.

Those neutral cables ran through Great Britain. The Prism of its time was in place. When, on 16 January 1916, the encrypted text of what was to become known as the Zimmerman Telegram was delivered— by the Germans—to the U.S. Embassy in Berlin to be sent over U.S. cables to the German embassy in Washington for onward transmission to Mexico, the British were able to intercept and decode it. In order to disguise the facts that a) the British could break the German ciphers, and b) the British were data-collecting off of American telegraph cables, British Intelligence stole a copy of the Western Union telegram from the Mexico City telegraph offices to make it look as if the data breach had occurred there instead. That is often the way with secrecy: having collected data in one location, a second data collection was needed to disguise the first.

Between the World Wars, the American Cipher Bureau (known as the “Black Chamber”) under Herbert Yardley obtained the coded telegrams of foreign governments through the cooperation of the presidents of the Western Union Telegraph Company and the Postal Telegraph Company. This was “warrantless data collection” and it occurred during peacetime. In many cases, Yardley simply asked the presidents of the telegraph companies to allow intelligence officers to copy the messages in the telegraph offices. Sometimes the Cipher Bureau obtained copies by bribing telegraph company employees.

In addition to the telegrams supplied by the telegraph companies, the Postal Censorship Office supplied the Black Chamber with intercepted communications. Mail and cable censorship stations existed at Balboa-Cristobal in Panama, Brownsville, Chicago, El Paso, Honolulu, Laredo, Los Angeles, Miami, New York, New Orleans, Nogales, San Antonio, San Francisco, San Juan, Seattle, (postal mail), and Akron, Baltimore, Honolulu, Miami, New Orleans, New York City, Panama, San Antonio, San Juan, Seattle, Tucson, and Reykjavik, Iceland (cable).

At about the same time in Britain, a 23 December 1920 act of parliament required owners or controllers of telegraphic cables to produce—upon warrant—any telegram that a government official requested.

How large were these efforts? Or, in today’s parlance, how much of the communications spectrum did these agencies “touch”? MI-8, the American military intelligence unit established in April 1917 —the forerunner of the Cipher Bureau— read 10,735 foreign messages in its first eighteen months. This compares to the 250 million internet communications per year that PRISM has been estimated to have been collecting. The Black Chamber cost $329,212 ($4,410,000 in today’s value) for ten years, or one-hundredth of one percent of the State and War Departments’ combined budgets.

Data collection and communication interception has been around for centuries; but the scale and the cost have changed.

Bulk Data Collection

Further Reading