Hacker Timesnew | past | comments | ask | show | jobs | submitlogin

This is (I was surprised) a pretty good article. Financial services are regulated and based on recent experience, they're concerned with systemic risk. Most industries do not have anyone responsible for worrying about this kind of thing.

It seems reasonable to start worrying about the fragility potentially introduced by these massive internet infrastructure companies.



If you wanted to blow something up to make the west suffer, an AWS datacenter would probably be a pretty good target. I wonder at what point that becomes a legitimate national security concern, and the government steps in to provide protection.


Yes, this goes to show that there are ~zero terrorists in the US. The US has an almost infinite supply of soft targets, but since 9/11 there have been no attacks of any consequence. There have been a few minor attacks like Boston bombers, but those have all been laughably amateurish.

The US has no terrorism problem. In 1979, the Irish killed the Queen's uncle in law, and in 1984, they blew up a hotel where Thatcher was staying. That's a serious terrorism problem! What the US has is nothing by comparison. We were unlucky on 9-11, and ever since we've been distorting our foreign policy out of unreasonable fear.


legitimate national security concern, and the government steps in to provide protection

There are a LOT of far softer targets that go unprotected. A terrorist attack on a sewage plant for a major city would be far more devastating than knocking out a few websites.


I feel like the point of the article we're commenting on is that AWS is far from "a few websites"


You can live without Netflix much longer than you can without a flushing toilet, is the point I'm making. Yet we don't have armed guards patrolling the sewageworks... It's a matter of priority how finite security personnel are deployed.


"Yet we don't have armed guards patrolling the sewageworks"

There's probably a moderate amount of unseen security in larger areas surrounding water/sewage plants.

Also, we've dealt with regional natural disasters before. Drought, fire, flooding, etc - we have experience mobilizing resources and redistributing as needed.

We've never dealt with an extended outage of one or more AWS data centers for days at a time. How many govt/university systems would be unable to function because of direct or indirect dependencies on AWS-related services?

S3 going down for, let's say 4 days, would cause big havoc on so many projects and systems I know of.

I'm pretty sure most people have no clue how much of their data and systems functionality is reliant on AWS-related services.


How about whatever is running on AWS GovCloud?

Water outages happen often enough - they distribute water manually with tanker trucks. When sewage is overwhelmed it overflows into a river and they tell people to not go for a swim. https://en.wikipedia.org/wiki/Sanitary_sewer_overflow


GovCloud has stuff that would be bad if it goes down, but most institutions that use it have a DR plan.


If AWS went down hard, it would be more than just "I can't watch Netflix".

Some people wouldn't be able to do their computer work, send receive emails, others might not receive their paychecks or be able to pay bills.


That still ranks lower than "sewage treatment gone". Maslow's pyramid still applies.

The sewage plant hits both physiological needs and safety needs (it's a pretty dire health threat quickly). AWS has weak tendrils to financial safety, but mostly affects layers above that.


The old historical lesson applies. People will say they're rebelling for various reasons. But at the end of the day it's for land, water, or food.


Receiving your paycheck could be worse. There are ways to deal with water outages. Worst case you let sewers overflow into rivers, as discussed in another comment. If salaries of large companies remain unpaid this would be far more severe for a lot of people.


>would be more than just "I can't watch Netflix".

And, I think even more than we can imagine. It's one thing to count the number of services that are direct customers/dependents of AWS, and we really don't know how deep that goes.

But, add to that the non-AWS based services that directly or indirectly rely on AWS-based services.


I used to work at a company that had a large cage in a local data center - one of the other cages in the same room was apparently for the local police force (we weren't supposed to know this!).

It wouldn't surprise me if those police systems have moved, or are moving, to Azure or AWS data centers.


Police would probably still work fine without data centers for a few days. As long as they can communicate and 911 works, they can do most of their job in the short run.

It's much worse if benefits or paychecks don't get paid. For many people that means that they won't be able to buy groceries as soon as a few days later. Not the standard HN crowd, but many people depend on regular paychecks and can be in trouble if it comes even one day late.


As long as they can communicate

Funny thing with the police, in the 60s there was flooding and they couldn't coordinate a response, so radio amateurs had to step in and provide comms. Those enthusiasts are still around under the codename RAYNET. The police eventually got their act together and invested in a system called TETRA which is self-contained and independent of any other infrastructure so they would never be caught with their pants down again.

Now they've ditched TETRA to save money and they just run over the 4G network like anyone else...


But if you attack a massive sewage treatment plant and completely disable it, you can effect maybe... 10 million people, if we're being generous?

If you hit AWS, just Netflix alone is going to effect 100 million people. There are definitely a few payroll providers that are going to have some reliance, so people all over are going to stop receiving their paycheques. Many businesses will be temporarily disabled.

On a national level we've got a lot more eggs concentrated in the AWS baskets than we do the sewage treatment baskets.


The thing is the dispatch system for those security guards may well be running on AWS or other cloud - once they become uberized, even more likely.


To the extent that this article is correct, financial services would be affected.


Knocking out all of AWS is very different from knocking out a single data center.


Especially because AWS regions are broken up into multiple availability zones (data centres in the same area). So taking out a single data centre won't do much if the AWS customers have correctly designed their systems for high-availability (ie having redundant instances in other AZs/regions with their data backed up elsewhere).


A single availability zone can be spread across many physical data centres.

According to this rackspace article, the largest AZ has 5 data centres.

https://blog.rackspace.com/aws-101-regions-availability-zone...


Take this for what it is - I'm a software developeer who works in the cloud, not a cloud expert.

The abstraction behind AZs is what every AZ counts for at least 1 data centre. So for every region there is at least 2 AZs, and every AZ means at least 1 data centre (or 5 in your link).

This just means it's harder to take out a whole region by destroying individual data cetnres. Since most regions consist of 2-5 AZ, and AZs consist of 5+ data centres, that means destroying dozens of data centres.


That's a big if. The S3 outage in February this year showed that there are a lot of sites that aren't designed for this.


The S3 outage was across the whole US-EAST-1 region, not just one data centre.

I know my company and a few others that are redundant within a region (ie if one AZ goes down), but not if a whole region goes down.


This all assumes that "taking out" datacenters is a physical/hardware operation.

When you widen the potential attack surface to include software vulnerabilities, unauthorized access, process flaws and other "soft" vectors, a much wider--possibly coordinated--attack that is potentially far more crippling can be imagined.


The question is how load would behave if more than 1 AZ fails for several days. If 2 out of 5 AZs in US-East are down for a week, everyone would distribute to the other 3 (and probably on all three to be sure). I'm not sure if they have enough spare capacity to handle that. The AZ model is designed for single failures and failures that are a few hours, not days.

Of course just speculation, I have no knowledge about DR plans at Amazon.


Granted it was only for a few hours but they had a huge outage in February this year (https://aws.amazon.com/message/41926/). This affected pretty much all us-east customers (Which are most AWS customers) so it should be possible to extrapolate from there. Although I think a lot of customers learned the importance of replication that day so the next outage might not be that bad.


I get what you're saying.

However, we actually have evidence that a sewage plant for a major western city can suffer a sudden catastrophic failure[1] and it won't even make it above the fold.

[1] https://projects.seattletimes.com/2017/west-point/


A great soft target is electricity grid. The timelines for manufacturing large transformers for example can be many months and they usually don't have as much redundancy as you would expect. A semi coordinated attack against the grid, using dynamite to take out a number of pylons along main HV lines, easily accessible because they're out in rural areas. Combine with a number of sticks thrown over fences at transformers. And you could massively impact the US economy for months.


The economist had a pretty good article about that. It dealt with electromagnetic impulses, as those could take out several at once:

http://www.economist.com/news/world-if/21724908-huge-potenti...


Wouldn't you have to blow up at least all the datacenters in a region to make an impact?


There are probably some people careless enough that destroying a single data centre would damage them. Most serious contenders (e.g. My employer) have fully redundant active-active setups across regions, so you'd have to engineer a massive outage to take out real shops.

Of course, if you completely remove an entire AWS region you might induce very damaging stresses on other regions as people fail over.


I think the number of applications running on AWS in a non-HA manner is very high, so a single datacenter going offline would have some impact.


All you need to blow is a few cables.


Why blow up anything or damage any cables? Hack the computer of an Amazon employee and do your damage there. The last S4 outage was because of someone fat fingering a script, imagine what someone could do that really wanted to mess stuff up.


This. Could've saved myself a comment elsewhere on this thread.

But, yeah, I'm kinda' surprised that this HN crowd in particular is so focused on hardware vectors.


And ultimately it leads you back to targeting the IT operations centre for the business where they provision equally to say both AWS and Azure for redundancy. At that point you can knock all of the biz cloud capacity off.


Well, you need to disable redundant ISP or power cables to all datacenters in a region. and that would probably be pretty easy to recover from in a day or two.

I imagine Amazon has some on site security measures as well.

It would be interesting to see how important services would cope with their main region going down for more than a few days.


Not that I think it is wise to discuss optimising a terrorist activity on a public forum, though it is interesting from a threat analysis point of view.

You don't need to severe both power and data cables. For power the datacentre should be able to cope for a while, particularly if it has access to fuel deliveries. Data cable should be both easier to severe and more difficult to recover from (not the least to identify which cable has been cut where).

Most of the infrastructure of a country in term of cables run along railway tracks, sewers, etc. This means thousands of kilometers of cable even for a small country, and it is impossible to secure everything. It is impossible to severe everything either but you don't need to severe every single cable. As long as you severe enough of the backbone, the other cables will be overloaded.

So I don't think it would take that much effort for a network of terrorists to create havoc in the communications of a country for at least a few days to a week. And the consequences for the economy can be pretty dire. We have seen with BA what happens when their datacentre goes offline. Their all fleet is grounded. I imagine the consequences of a country-wide outage could be pretty dramatic. Unlikely anyone would die but you could really dent the GDP.


But if that's your methodology, you'd need to cut several cables in different locations simultaneously. A single cable doesn't take too long to fix (a couple hours, tops) and there are probably several redundant backups to handle bandwidth in the interim.


Then all that's needed to repair is a few cables.


fight club update anyone?


Amazon's own data centers are tiny (compared to most) that taking one out would almost be a waste of time. Even one of their large colocations would probably be a waste.

If you wanted that type of destruction, and be noticed, you would need a city leveler type event in a data center heavy area.


Most people wouldn't even notice. They'd have to blow up a region and at that point, you have bigger problems.


I really hate the "too big to fail" meme and I strongly agree with Bernie in that if you are too big to fail you are too big to exist.

That should be the priority.


I don't know what Bernie's specific plans were/are but it's not unusual. At least, at first blush, this is a lot of people's reactions. Too-big-to-fail = danger = make it smaller.

Realisitically, this has not been the solution implmented (in the EU & US, at least). In the EU, it is even more crucial as the "solutions" to this problem are applied to state finances as well as financial institutions.

In terms of policies, there are two competing approaches: (1) Reduce the size of "too-big-to-fail" institutions. (2) Regulate them more heavily (or some other strategy) so that they will not fail. In the EU, this is being applied to states, not just financial institutions. Rules that (supposedly) reduce catastrophic risk.

Almost all seripous policy proposals are in the no. 2 category. Tighten regulation, reduce the risk of failure. Tighter regulation lends to stronger incumbents and larger average company size so by doing 2, you are probably doing the opposite of 1.

As I said, I don't know what Bernie's proposal is or how mature it is as a policy (as opposed to a politician statement). It would be notable if a left wing politician propsed loosening bank regulations, though definitely not impossible or unreasonable.


Paul Volcker's solution was that any time a bank is so big that we need to bail it out to avoid systemic problems, then the bank should be broken up. We can't always see all the problems in advance, but we can break them up in the aftermath.


Agreed, it seems like the software approach as well. You wouldn't want a class or a piece of code to be "too big" to fail, you'd refactor it into smaller pieces which can be overviewed more easily.


It's really a basic engineering principle. Smaller and more distributed systems are more reliable.


This sounds like the microkernel / monolith debate. Except monoliths kind of won.


It depends on what you want to optimize for. micro is good for some things, mono is good for other things. Neither is inherently "better", it just depends on your goal


> I really hate the "too big to fail" meme and I strongly agree with Bernie in that if you are too big to fail you are too big to exist.

That seems like the most reasonable response. And yet, since the great recession, our policy has been "make 'too big to fail' even bigger".

The problem is that the banks have become too powerful for anyone to challenge. A Teddy Roosevelt type of political leader can't exist today.


Corporate entities were radically more powerful in Teddy Roosevelt's day, far beyond what they are today. They were almost entirely unchained in regards to economic power, whereas today there is hyper regulation (tens of thousands of pages of it, including direct Fed control over the banking system).

Standard Oil was as powerful as the US Government in that era, as were the railroads. JP Morgan was far more powerful than the US Treasury. Cornelius Vanderbilt - pre Roosevelt - all by himself had greater financial capability at his peak than the US Government at the time.

The difference, is back then there was wide-spread and growing fear of the combinations and would-be monopolies. Today, Americans are relatively unconcerned by Microsoft (desktop), Google (search, android), Walmart & Amazon (retail), Intel (microprocessors), Facebook (social), Cisco, Boeing, etc.


Why leave it at American companies? The Dutch East India Company was far more powerful and bigger than any company seen today.


Beyond that, it was more powerful than many (maybe most) modern governments. It was an international power unto itself.


Here is an smbc comic making fun of this phenomenon. http://www.smbc-comics.com/index.php?id=3794


Why is every arrangement of characters now a "meme". That word has moved beyond devoid of meaning, at this point it's like a black hole of nothingness of a word.


> A meme (/ˈmiːm/ MEEM) is an idea, behavior, or style that spreads from person to person within a culture.

Seems like it's fitting here, does it not? Certain banks being "too big to fail" is an idea passed among persons within our culture.


kind of like the idea of a meme (/ˈmiːm/ MEEM) is a meme (/ˈmiːm/ MEEM).


> Financial services are regulated and based on recent experience, they're concerned with systemic risk. Most industries do not have anyone responsible for worrying about this kind of thing.

I'd say that most industries do not have anyone responsible for worrying about it high enough in the management chain.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: