Massive US military social media spying archive left wide open in AWS S3 buckets • The Register
Date: 2017-11-18T02:36:30Z

archive.is webpage capture | Saved from |
history |
| 18 Nov 2017 02:36:30 UTC
All snapshots | from host www.theregister.co.uk
Webpage Screenshot
share download .zip report error or abuse | short link long link markdown html code <a href="http://archive.fo/yv2T2"> <img style="width:300px;height:200px;background-color:white" src="https://archive.fo/yv2T2/d4c8901383046b0b7c813057824324baf24dbe87/scr.png"><br> Massive US military social media spying archive left wide open in AWS…<br> archived 18 Nov 2017 02:36:30 UTC </a> wiki code {{cite web | title = Massive US military social media spying archive left wide open in AWS… | url = https://www.theregister.co.uk/2017/11/17/us_military_spying_archive_exposed/ | date = 2017-11-18 | archiveurl = http://archive.fo/yv2T2 | archivedate = 2017-11-18 }}

# This page was last archived 14 hours ago

If this snapshot looks obsolete you can again
The Register uses cookies. Find out more.
Log in Sign up Forums

- M³
- CLL
- Events
- Whitepapers
- The Next Platform

- Servers
- HPC
- Cloud
- Storage
- Networks
- Virtualization
- BOFH

##### Big Cable's pillow talk with FCC to forbid US states from writing own net neutrality rules

##### Speedy roadster for filers hits the road. Don't worry Elon, it's a storage vehicle

##### So what does EE's 5G test really signal?

##### A challenger appears: Specs for Samsung's potential Optane killer

##### Windows Update borks elderly printers in typical Patch Tuesday style

##### Microsoft can't give away beta cert exams, so starts charging

##### WordPress 4.9: This one's for you, developers!

##### Fear not, driverless car devs, UK.gov won't force you to write Trolley Problem solutions

##### Massive US military social media spying archive left wide open in AWS S3 buckets

##### Shamed TLS/SSL cert authority StartCom to shut up shop

##### For goodness sake, stop the plod using facial recog, London mayor told

##### Lloyds' Avios Reward credit cardholders report fraudulent activity

##### HMRC scoops up Microsoft veep for CIO role

##### So you're already in the cloud but need to come back down to Earth

##### A big ask for any nerd, but going outside (your usual data sets) can be good for you

##### So much data, so little time: How to not flip your wig processing it

##### 'Do the DevOps?' No thanks! Not until a 'blameless post-mortem' really is one

##### Splunk prototyping DevOps efficiency measurement tool

##### Silverlight extinguished while Angular wins fans among developers

##### Samsung shows off Linux desktops on Galaxy smartmobes

- Policy
- The Channel

##### Then there were four: Another draft US law on 'foreign' (aka domestic) mass spying emerges

##### True Telecom busted by Ofcom for 'slamming', misselling and more

##### Car tax evasion has soared since paper discs scrapped

##### UK private sector joins public in... Escape from DXC Max

##### OnePlus 5 x T + five short months = Some p*ssed off fanboys

##### Anonymized location-tracking data proves anything but: Apps squeal on you like crazy

##### What is DJI Terra? Mystery builds over Chinese biz's trademark

##### Google aims disrupto-tronic ray at intercoms. Yes, intercoms

- Geek's Guide

##### 'It's back to the drawing board...' Innocent axions found not guilty of dark matter crimes

##### Rocky Ross 128 b might harbour ALIENS – and it's headed right for us

##### 576-megapixel 'Zwicky Transient Facility' telescope sees first light

##### Boffins on alert: Brace yourselves for huge gravitational wave coming within a decade

- Artificial Intelligence
- Internet of Things

##### Crewless dinghy signs to UK Ship Register for Middle East mission

##### Why Boston Dynamics' backflipping borg shouldn't scare you

##### Connected and self-driving cars are being sent to Coventry

##### It's artificial! It's intelligent! It's in my home! And it's gone bonkers!

- Verity Stob

##### Dick move: Navy flyboy flings firmament phallus for flabbergasted folk

##### New UK aircraft carrier to be commissioned on Pearl Harbor anniversary

##### The Reg parts ways with imagineer and thought pathfinder Steve Bong

##### Yes, I took Putin's roubles to undermine Western democracy. This is my story

#### Security

# Massive US military social media spying archive left wide open in AWS S3 buckets

## Dozens of terabytes exposed, your tax dollars at work

By Iain Thomson in San Francisco 17 Nov 2017 at 20:08
19 SHARE ▼
Three misconfigured AWS S3 buckets have been discovered wide open on the public internet containing "dozens of terabytes" of social media posts and similar pages – all scraped from around the world by the US military to identify and profile persons of interest.
The archives were found by veteran security breach hunter UpGuard's Chris Vickery during a routine scan of open Amazon-hosted data silos, and these ones weren't exactly hidden. The buckets were named centcom-backup, centcom-archive, and pacom-archive.
CENTCOM is the common abbreviation for the US Central Command, which controls army operations in the Middle East, North Africa and Central Asia. PACOM is the name for US Pacific Command, covering the rest of southern Asia, China and Australasia.
Vickery told The Register today he stumbled upon them by accident while running a scan for the word "COM" in publicly accessible S3 buckets. After refining his search, the CENTCOM archive popped up, and at first he thought it was related to Chinese multinational Tencent, but quickly realized it was a US military archive of astounding size.
"For the research I downloaded 400GB of samples but there were many terabytes of data up there," he said. "It's mainly compressed text files that can expand out by a factor of ten so there's dozens and dozens of terabytes out there and that's a conservative estimate."

## Leaky S3 bucket sloshes deets of thousands with US security clearance

READ MORE
Just one of the buckets contained 1.8 billion social media posts automatically fetched over the past eight years up to today. It mainly contains postings made in central Asia, however Vickery noted that some of the material is taken from comments made by American citizens.
The databases also reveal some interesting clues as to what this information is being used for. Documents make reference to the fact that the archive was collected as part of the US government's Outpost program, which is a social media monitoring and influencing campaign designed to target overseas youths and steer them away from terrorism.
Vickery found the Outpost development configuration files in the archive, as well as Apache Lucene indexes of keywords designed to be used with the open-source search engine Elasticsearch. Another file refers to Coral, which may well be a reference to the US military's Coral Reef data-mining program.
"Coral Reef is a way to analyze a major data source to provide the analyst the ability to mine significant amounts of data and provide suggestive associations between individuals to build out that social network," Mark Kitz, technical director for the Army Distributed Common Ground System – Army, told the Armed Forces Communications and Electronics Association magazine Signal back in 2012.
"Previously, we would mine through those intelligence reports or whatever data would be available, and that would be very manual-intensive."
Before you start scrabbling for your tinfoil hats, the army hasn't made a secret of Coral Reef, even broadcasting the awards the software has won. And social media monitoring isn't anything new, either.
However, it is disturbing quite how easily this material was to find, how poorly configured it was, and that the archives weren't even given innocuous names. If America's enemies – or to be honest, anyone at all – are looking for intelligence, these buckets were a free source of information to mine.
After years of security cockups like this in the public and private sectors, Amazon has tried to help its customers avoid configuring their S3 buckets as publicly accessible stores, by adding full folder encryption, yellow warning lights when buckets aren't locked down, and tighter access controls.
"This was found before these new Amazon controls were added," Vickery said. "So we have yet to see how effective that yellow button will be."
Vickery said he notified the American military about the screwup, and the buckets have now been locked down and hidden. Unusually, the military contact thanked him for bringing the matter to their attention – usually talking to the armed forces is a "one-way street," Vickery said.
The Register asked the army for comment, and for more details on Outpost and Coral Reef, but wheels turn slowly in the Green Machine. We'll update the story as soon as more information is known. ®
Tips and corrections
19 Comments

## Sign up to our Newsletter-Get IT in your inbox daily

- MORE
- Us Military
- Amazon s3

### Most read

1. #### Massive US military social media spying archive left wide open in AWS S3 buckets

2. #### Tesla launches electric truck it guarantees won't break for a million miles

3. #### Drone maker DJI left its private SSL, firmware keys open to world+dog on GitHub FOR YEARS

4. #### Inside Internet Archive: 10PB+ of storage in a church... oh, and a little fight to preserve truth

5. #### Amazon Key door-entry flaw: No easy fix to stop rogue couriers burgling your place unseen

### More from The Register

#### Yet more British military drones crash, this time into the Irish Sea

Not looking good for the Thales Watchkeeper fleet

#### How about that time Russian military used a video game pic as proof of US aiding ISIS?

This week in fact

#### US military gets authority to shoot down citizens' small drones

Meanwhile, DJI says it had no idea army was using its kit

#### Oz military megahack: When crappy defence contractor cybersecurity 'isn't uncommon', surely alarm bells ring?

30GB of data nicked in 'Alf's Mystery Happy Fun Time' attack

#### US military spies: We'll capture enemy malware, tweak it, lob it right back at our adversaries

Collateral damage in 3, 2, 1…

#### HMS Queen Elizabeth is delayed, Ministry of Defence confesses

Haughty minister tries brushing it off as pedantry

#### French senator demands public inquiry into Microsoft military deal

C'est mauvais, le framework de beaucoup d'argent pour le Redmond

#### First the Rise of the Machines, now this: UK military's Exercise Information Warrior

Comment From drones to smart warships. Well, sort of

### Whitepapers

#### Mobile Resource Strategy in Midsize Firms

The importance of infrastructure upgrades on the mobile worker experience.

#### Scaling Customer Communications

Do CCM offerings need to change in response to the new generation of consumers?

#### Get the Most out of Salesforce by Extending ECM

How to build stronger relationships and boost sales by extending Enterprise Content Management and Salesforce.

#### Preparing for the General Data Protection Regulation

For most organizations, the implications of GDPR are both significant and far-reaching.

### Sponsored links

- Get The Register's Headlines in your inbox daily - quick signup!

#### About us

- Who we are
- Under the hood
- Contact us
- Advertise with us

#### More content

- Week’s headlines
- Top 20 stories
- Alerts
- Whitepapers

#### Situation Publishing

- The Next Platform
- Continuous Lifecycle London
- M-cubed
- Webinars

The Register - Independent news and views for the tech community. Part of Situation Publishing

#### Sign up to our Newsletters

Join our daily or weekly newsletters, subscribe to a specific section or set News alerts
Subscribe
.
.
Biting the hand that feeds IT © 1998–2017
Cookies Privacy Ts&Cs
SECURITY WARNING: Please treat the URL above as you would your password and do not share it with anyone. See the Facebook Help Center for more information.

0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%