Optimistically, I hope it filters out the people who were only interested in it for the money.
When I was in school, decades ago now, very few people went into CS compared to other majors. Everyone I knew going into it did it because they loved it. I would have done it regardless of the career opportunities because I want to build stuff.
Interviewing candidates over the years since then, my experience has been there are still very few of those passionate nerds and a lot of people who did it for other reasons, like the money or similar. There is nothing inherently wrong with this. I don’t fault people for it.
Maybe if we get very lucky, it will go back to a relatively few passionate people building stuff because it is cool?
Having an industry’s labour supplied only by those inherently passionate about it is a great way to crush wages and working conditions. Look at what companies like Blizzard get away with because their employees just want to make video games at their favourite dev studio. While they’re a pain in the ass sometimes, I welcome the devs who are only here for the cash.
This is totally leaving out the supply and demand aspect. People like the idea of making games more than working on the plumbing of some accounts payable software, so Blizzard can pay less and treat worse than NicheBoringFinanceCo.
The parent comment is describing supply and demand. If Blizzard attracts a larger supply of workers who will accept lower pay and worse conditions because they intrinsically want the job, Blizzard gains leverage. That is exactly why studios like Blizzard can get away with more than “NicheBoringFinanceCo.”
If an “industry’s labour [is] supplied only by those inherently passionate about it” the post says it would “crush wages and working conditions”.
That runs completely counter to the basics of supply and demand in a perfect competition market. It would be market with far fewer (labor) suppliers, who could therefore command a higher wage, not lower.
You are only looking at supply. Neither supply nor demand by themselves adequately describe prices (even in supply-demand 101 theory; in practice of course it gets significantly more complicated than just supply and demand). There are fields with few suppliers where supply is extremely cheap and fields with few suppliers where supply is extremely expensive.
Is the number of suppliers low because demand is also low or is the number of suppliers low because demand is high but supply is constrained?
A field that previously had a supply of labor in it "for the money" who all leave is indicative of the former scenario not the latter.
That does not lead to higher wages. That leads to low wages.
(There are a variety of reasons why this story is too simple and why I remain uncertain about developer salaries in the short term)
There is a broader question of whether having people who are in it for the money leave independently "causes" wages to go down (e.g. if you were to replace all such people with people "purely in it for the passion"). My suspicion is yes. Mainly because wage markets are somewhat inefficient, there are always mild cartel-like/cooperative effects in any market, people in it for passion tend to undersell labor and the people in it for the money are much less likely to undersell their labor and this spills over beneficially to the former.
Note that this broader question is simply unanswerable assuming perfect competition, i.e. a supply-demand 101 perspective (which is why it doesn't make sense to posit "perfect competition" for this question).
It posits durable behavioral differences among suppliers that are not determined purely by supply and demand which do not update reliably in the face of pricing. This is equivalent to market friction and hence fundamentally contradicts an assumption of perfect competition.
The only way the people who are only in it for the money leave the industry is if the money gets worse. If the money stays the same why would they leave
To use your example of someone working on the plumbing of an accounts payable system, who is passionate about that? The supply is near zero. That, like most jobs, is going to need to be done entirely by people who are just doing it for the money, and there is nothing wrong with that.
Your example runs counter to the laws of supply and demand too. You understand that wages will rise when supply is restricted, but you don't want to accept that supply will respond to the price signal in the form of more people entering that job market.
> That, like most jobs, is going to need to be done entirely by people who are just doing it for the money
why then do they all have those interview rounds where you have to talk about what really attracted you to work at this boring company and how you would love to do that kind of work? They evidently haven't gotten the memo.
I have never once pretended to be “passionate” about working. Never wrote a single line of code that I haven’t gotten paid for since I graduated from college 30 years ago. I was a hobbyist before college for 6 years.
I’ve gone through the BigTech guantlet successfully. Even then I showed I cared about doing my job well and competently.
I have purposefully thrown nuggets out during interviews letting companies know that I had a life outside of work, I’m not going to work crazy hours and in the latter half of my career, I don’t do on call.
We've banned this account. We can't have vile comments like some of the ones your account has posted in recent days, without taking any action, if we're to have any standards at all here.
If you need a lot of low quality code in a hurry, AI can definitely do that for you now. The path to making money by writing mediocre code for people who don't really care that much is going to look like managing a network of bots that constantly spit out a huge volume of code that kind of mostly works and if it sometimes doesn't then whatever. The people in it for the money can probably make a decent amount in the "high volume low quality" space.
Then there's the code that needs to actually work, or have some thought put into it. Consider the process of writing IETF RFCs. Can you get an LLM to spit out English text that conforms to their formatting? Absolutely you can. Is the RFC it emits going to be something you'll want to have the whole world trying to implement as a standard? Not likely. So the people doing that are going to be doing it something closer to the old way.
I am kind of considering the idea of changing my LinkedIn profile to one of me with a 'wild rag', checkered shirt, and broad brimmed straw hat and calling myself a robot wrangler and see if I get any takers.
As many of us in the early IT generation, I came because of I wanted to build games and program cool stuff.
Today, while I admit Games are supercomplex stunning apps, I hate it and I love to do boring finance app development :-))
If you would have told me in my 20ies that I will end up in banking & finance IT, I would have laughed at you - today I really like it and I do not play a single game anymore.
See also: public school teachers. You either need to be insanely passionate or incredibly stupid to take ~$55k/year for long hours as an educator that is also a babysitter. And insanely passionate teachers are in short supply.
I bet a lot of teachers look at what devs do and think that its also insane to sit in front of computer all day, in a no boundary job, working on something you really don't care about and is potentially really bad for civilization only to make money off and lose your sense of self.
My spouse has expressed this nearly verbatim after transitioning out of a 16 year career in middle and grade school education to medical curriculum development. It was hell on her mental health but at least there was a clear motivation and purpose for being there.
There are a lot of other benefits of being a teacher especially if it’s a secondary income in a two income family. Namely you are on the same schedule as your kids. My mom is a retired high school teacher.
Long hours? Teachers work the same hours or less than other adults per “New Measures of Teachers’ Work Hours and Implications for Wage Comparisons” by West.
“Teachers work an average of 34.5 hours per week on an annual basis (38.0 hours per week during the school year and 21.5 hours per week during the summer months).”
That’s leaving out the benefits of incredibly strong union protections, it being a state job with matched benefits, absurd job security even in the face of terrible performance, etc.
There's no way these numbers can be correct. My school was 8 am to 3 PM, that's 35 hours a week right there for full time teachers. But teachers spend many more hours outside the class preparing lessons, grading work, and following up on things. If you even spend a week teaching something you quickly realize how much extra prep work goes into it.
From the study: "Teachers work more than they are required to work by contract, but less than self reported hours of work. I find that teachers are more likely to overestimate their hours of work in the CPS than workers in other occupations, and conclude that this is likely because of an uneven work year".
Even by your own example, you're only at 35 hours a week, and that's before you subtract out the weeks of summer vacation, winter vacation, spring break, etc; where the workload is certainly far less than 40 hours a week.
The Alaskan teacher's union is ranked 15th overall in the US [1]. I'm betting they're just fine, and that any issues are more general "Alaska-problems" than anything specific to teaching, unions, etc.
And ignoring that the other four factors are: Resources and Membership, Involvement in Politics, Scope of Bargaining, and State Policies, shows that you just want something that agrees with your anecdote.
Why are teachers special to merit any "protections" that aren't afforded to all employees, public or private?
Reading the report, i see that it's from 2012. My dude, you are way off base to begin with, not to mention 15 years out of date. And things have changed significantly. regardless:
1) Resources and Membership: Membership is essentially compelled, and the resources of the union rarely support member teachers. Three of the anchorage teachers in my life say their union reps are useless and they have little agency in rectifying the problem.
2) No comment: Politics in AK is FUBAR, and as an aside I imagine less gets spent on politics because we all know the oil companies own it all.
3) "Alaska education leaders value bottom-up decision making (see sidebar);" Absolute nonsense. Decision making is almost entirely dominated by outside economic concerns and the behavior of the state and federal government from year to year. I say this as someone whose brother has participated in nearly every union negotiation for the last 20 years at ASD.
4) Irrelevant to the livelyhood of alaskan teachers, AFAICT
> Why are teachers special to merit any "protections" that aren't afforded to all employees, public or private?
Teaching in public school, like serving in the military or working in emergency services, is a career that we should maintain for the well-being of our country and citizens. If teachers cannot earn a living wage -- to have the basic dignity of owning a home and raising a family should they want to -- then we are worse off as a country over time.
To be clear, I am biased here. I started my career working for ASD, have lots of family that work for ASD in both admin and teaching, and many friends directly involved in education in Alaska. Public education in Alaska is a shit show, and seems to be on the an accelerating downswing since covid. The unions aren't helping the situation either, hence my opinions.
The only thing that report does for me is show that our metrics for what makes a good teachers union or a strong teachers union are wrong.
>Optimistically, I hope it filters out the people who were only interested in it for the money.
I hope not, because we don't need software developers to be "starving artist 2.0".
And on that note: I vividly remember people staying away from the video game development industry because it was deemed "passion industry", and that had a really negative connotation of long working hours for asymmetrical return, and more.
I don't look forward for every other software engineering branch to become like that.
Seems… improbable. There will certainly be less of us, but the fact remains that nobody wants to debug this shite vibecoded apps companies are pushing, and some simply are not able because of skill atrophy and perverse incentives to use AI at the cost of stability.
Brother, we need to eat. You don't need to go to college to learn about some topic, you can pirate textbooks. You need to go to college to get a piece of paper saying you did. If you were passionate about computer programming, you can do it in your free time while you flip burgers or do whatever you need to survive
>You need to go to college to get a piece of paper saying you did. If you were passionate about computer programming, you can do it in your free time while you flip burgers or do whatever you need to survive
This is a naive view of the average (or even above average) person's approach to learning, as well as an overly cynical read on the intellectually motivating atmosphere that comes from earnestly engaging in an academic environment.
Unless you were unfortunate enough to go during peak covid years, then that was just a skill issue. If it was truly beneath you, you could have been writing and publishing papers.
> Maybe if we get very lucky, it will go back to a relatively few passionate people building stuff because it is cool?
I think we basically lost this when software/computer/internet entered the mainstream. Now, like everything else, it has to be bland, unoffensive, and a commodity.
I really wish this entire romanticism of the good old days when people only got into computer science because they breathed ate and dreamed about computers would die.
It was never reality - I graduated in 1996 and have worked at 10 jobs everything from lifestyle companies, to startups, to boring old enterprise to BigTech and now consulting companies. To a tee everyone has treated it like a job and not some religious calling. There is absolutely nothing wrong with coming to work at 8 leaving at 6 and not thinking about computers until the next day.
You don’t need to be doing side projects and open source contributions to do your job as a software developer anymore than a surgeon needs to be performing operations at home.
No I wouldn’t have chosen a major because I enjoyed it if it didn’t make any money. I didn’t then and I still haven’t found a method to get over my addiction to food and shelter.
That's just your experience, though.
It reflects mine, before I went to elite companies, where it is quite normal for people to live-and-breath software, at almost all hours.
I've worked at FAANG, yes, as well as one of the top two frontier AI labs, quant, and now, in a similar role doing very technical research.
Do you not think it's considered "elite" to e.g. work at such companies in highly technical roles in the same way that a PhD at Stanford is considered "elite"? As a holder of the latter, I do. If not, what would you consider an "elite" team?
Maybe you think the statement was pretentious, but your response: "I hope you don’t call your average FAANG and adjacent “elite” - that's sad" is, truly, the most pretentious thing I've ever read on this site. So I'll ask: what do you consider elite?
Yes I’ve worked at FAANG and the average mid level or even senior developer is not that impressive. Anyone with time on their hands and a decent proficiency can grind enough leetCode to get in.
Not that I did personally, I came in in the internal cloud consulting division (yes a full time blue badge, RSU earning employee).
You know then while all developers have to work at scale. Most of the work is built on pre-existing scalable components.
There are 1 million developers+ possibly if you count all of the FAANG + adjacent developers. I’ve nope a few of them during interviews after I left because I knew they couldn’t handle not being coddled by BigTech and wouldn’t know what to do with ambiguous requirements , an empty AWS account (even if they worked at AWS) and empty git repo.
But back to the point, they very much treated their job as a just a way to earn money and RSUs. They would have been a fool to treat a company as toxic as Amazon as anything else.
Yes I knew what I was getting into going in. I was a 46 and it was my 8th job out of college. I made my money, made connections, put it on my resume and moved on
These days? Quant, AI employees who have 2 commas in their yearly salaries, etc.
It’s not random mid level developer at a FAANG who “grinded leetCode” or even a senior developer who memorized “Designing Data Intensive Applications”.
You really didn’t think I was some 22 year old posting on r/cscareerquestions who was mesmerized by people “working at a FAANG” did you? For me it was just my 8th job out of now 10 and just another way to exchange labor for money.
>before I went to elite companies, where it is quite normal for people to live-and-breath software, at almost all hours.
Honest question: Do they actually _want_ to live-and-breathe software, or do they work in a highly competitive and highly compensated environment where doing that is implicitly required?
Defintely a mix, though I agree with you that the majority fall under, "they work in a highly competitive and highly compensated environment where doing that is implicitly required."
I'm not saying that this is an incorrect read, but I think it's important to consider that young people might be responding to the general desperation of a tight labor market across the last generation. It used to be that you could get a degree - any degree - and that would be enough to get you in the conversation for a position somewhere. Today, a degree isn't any sort of guarantee of any sort of job - in your field, entry level, dead-end retail, anything. Tuition skyrocketed and only a few fields kept pace. So, you get the degree in the field that's a "winner." Of course, this just increases competition, robs other fields of needed competency, etc. Prisoner's dilemma?
> Maybe if we get very lucky, it will go back to a relatively few passionate people building stuff because it is cool?
This is a really narrow way to look at it and define it lucky. What you describe will absolutely be a shitstorm for everyone - passionate workers and non passionate alike. Management doesn't care about your passion, it cares about the bottom line. Lots of folks will get fired - passionate people as well, or see their salaries cut and their job security evaporate.
There's no winners in the scenario you described other than the employers.
I am working on a domain specific agent that includes the concept of skills. I only allow one to be active at a time to reduce the chances for conflicting instructions. I use a small sub-agent to select/maintain/change the active skill at the start of each turn. It uses a small fast model to match the recent conversation to a skill (or none). I tried other approaches, but for my use case this was worked well.
My model for skills is similar to this, but I extended it to have explicit use when and don’t use when examples and counter examples. This helped the small model which tended to not get the nuances of a free form text description.
A reason, at least for a period of time, was accuracy of the delivery systems. You can’t attack a hardened target with a 100kt weapon and a delivery system with a 1km CEP, for example.
We have done a similar operation routinely on databases under pretty write intensive workloads (like 10s of thousands of inserts per second). It is so routine we have automation to adjust to planned changes in volume and do so a dozen times a month or so. It has been very robust for us. Our apps are designed for it and use AWS’s JDBC wrapper.
My company has been intentionally causing attrition in the US by moving to effectively a 996 style schedule. As people quit, their positions are moved to the India office. It is not an officially communicated policy. I have just surmised this based on private conversations with the executives and what is actually happening.
Interesting day. I've been on an incident bridge since 3AM. Our systems have mostly recovered now with a few back office stragglers fighting for compute.
The biggest miss on our side is that, although we designed a multi-region capable application, we could not run the failover process because our security org migrated us to Identity Center and only put it in us-east-1, hard locking the entire company out of the AWS control plane. By the time we'd gotten the root credentials out of the vault, things were coming back up.
Good reminder that you are only as strong as your weakest link.
This reminds me of the time that Google’s Paris data center flooded and caught on fire a few years ago. We weren’t actually hosting compute there, but we were hosting compute in AWS EU datacenter nearby and it just so happened that the dns resolver for our Google services elsewhere happened to be hosted in Paris (or more accurately it routed to Paris first because it was the closest). The temp fix was pretty fun, that was the day I found out that /etc/hosts of deployments can be globally modified in Kubernetes easily AND it was compelling enough to want to do that. Normally you would never want to have an /etc/hosts entry controlling routing in kube like this but this temporary kludge shim was the perfect level of abstraction for the problem at hand.
Probably. This was years ago so the details have faded but I do recall that we did weigh about 6 different valid approaches of varying complexity in the war room before deciding this /etc/hosts hack was the right approach for our situation
I remember Facebook had a similar story when they botched their BGP update and couldn't even access the vault. If you have circular auth, you don't have anything when somebody breaks DNS.
Wasn't there an issue where they required physical access to the data center to fix the network, which meant having to tap in with a keycard to get in, which didn't work because the keycard server was down, due to the network being down?
Way back when I worked at eBay, we once had a major outage and needed datacenter access. The datacenter process normally took about 5 minutes per person to verify identity and employment, and then scan past the biometric scanners.
On that day, the VP showed up and told the security staff, "just open all the doors!". So they did. If you knew where the datacenter was, you could just walk-in in mess with eBay servers. But since we were still a small ops team, we pretty much knew everyone who was supposed to be there. So security was basically "does someone else recognize you?".
Well, you put a lot of trust in the individuals in this case.
A disgruntled employee can just let the bad guys in on purpose, saying "Yes they belong here".
That works until they run into a second person. In a big corp where people don't recognize each other you can also let the bad guys in, and once they're in nobody thinks twice about it.
way back when DC's were secure but not _that secure_ i social engineered my way close enough to our rack without ID to hit a reset button before getting thrown out.
late reply but, no, i really needed to hit the button but didn't have valid ID at the time. My driver's license was expired and i couldn't get it renewed because of a outstanding tickets iirc. I was able to talk my way in and had been there many times before so knew my way around and what words to say. I was able to do what i needed before another admin came up and told me that without valid ID they have no choice but to ask me to leave (probably like an insurance thing). I was being a bit dramatic when i said "getting thrown out" the datacenter guys were very nice and almost apologetic about asking me to leave.
There's some computer lore out there about someone tripping a fire alarm by accident or some other event that triggered a gas system used to put out fires without water but isn't exactly compatible with life. The story goes some poor sys admin had to stand there with their finger on like a pause button until the fire department showed up to disarm the system. If they released the button the gas would flood the whole DC.
My point is that while the failure rate may be low the failure method is dude burns to death in a locked server room. Even classified room protocols place safety of personnel over safety of data in an emergency.
It wasn't Equinix, but I think the vendor was acquired by them. I don't actually blame them, I appreciated their security procedures. The five minutes usually didn't matter.
I remember hearing Google early in it's history had some sort of emergency back up codes that they encased in concrete to prevent them becoming a casual part of the process and they needed a jack hammer and a couple hours when the supposedly impossible happened after only a couple years.
> To their great dismay, the engineer in Australia could not open the safe because the combination was stored in the now-offline password manager.
Classic.
In my first job I worked on ATM software, and we had a big basement room full of ATMs for test purposes. The part the money is stored in is a modified safe, usually with a traditional dial lock. On the inside of one of them I saw the instructions on how to change the combination. The final instruction was: "Write down the combination and store it safely", then printed in bold: "Not inside the safe!"
> It took an additional hour for the team to realize that the green light on the smart card reader did not, in fact, indicate that the card had been inserted correctly. When the engineers flipped the card over, the service restarted and the outage ended.
There is a video from the lock pick lawyer where he receives a padlock in the mail with so much tape that it takes him whole minutes to unpack.
Concrete is nice, other options are piles of soil or brick in front of the door. There probably is a sweet spot where enough concrete slows down an excavator and enough bricks mixed in the soil slows down the shovel. Extra points if there is no place nearby to dump the rubble.
Probably one of those lost in translation or gradual exaggeration stories.
If you just wanted recovery keys that were secure from being used in an ordinary way you can use Shamir to split the key over a couple hard copies stored in safety deposit boxes a couple different locations.
The Data center I’m familiar with uses cards and biometrics but every door also has a standard key override. Not sure who opens the safe with the keys but that’s the fallback in case the electronic locks fail.
The memory is hazy since it was 15+ years ago, but I'm fairly sure I knew someone who worked at a company whose servers were stolen this way.
The thieves had access to the office building but not the server room. They realized the server room shared a wall with a room that they did have access to, so they just used a sawzall to make an additional entrance.
my across the street neighbor had some expensive bikes stolen this way. The thieves just cut a hole in the side of their garage from the alley, security cameras were facing the driveway and with nothing on the alley side. We (the neighborhood) think they were targeted specifically for the bikes as nothing else was stolen and your average crack head isn't going to make that level of effort.
I assume they needed their own air supply because the automatic poison gas system was activating. Then they had to dodge lazers to get to the one button that would stop the nuclear missle launch.
add a bunch of other poinless scifi and evil villan lair tropes in as well...
Most datacenters are fairly boring to be honest. The most exciting thing likely to happen is some sheet metal ripping your hand open because you didn't wear gloves.
Still have my "my other datacenter is made of razorblades and hate" sticker. \o/
I had a summer job at a hospital one year in the data center when an electrician managed to trigger the halon system and we all had to evacuate and wait for the process to finish and the gas to vent. The four firetrucks and station master who shoved up was both annoyed and relieved it was not real.
Not sure if you’re joking but a relatively small datacenter I’m familiar with has reduced oxygen in it to prevent fires. If you were to break in unannounced you would faint or maybe worse (?).
Not quite - while you can reduce oxygen levels, they have to be kept within 4pp so at worst, will make you light headed. Many athletes train at the same levels though so it’s easy to overcome.
That'd make for a decent heist comedy - a bunch of former professional athletes get hired to break in to a low-oxygen data center, but the plan goes wrong and they have to use their sports skills in improbable ways to pull it off.
Halon was used back in the day for fire suppression but I thought it was only dangerous at high enough concentrations to suffocate you by displacing oxygen.
Not an active datacenter, but I did get to use a fire extinguisher to knock out a metal-mesh-reinforced window in a secure building once because no one knew where the keys were for an important room.
Management was not happy, but I didn’t get in trouble for it. And yes, it was awesome. Surprisingly easy, especially since the fire extinguisher was literally right next to it.
Nothing says ‘go ahead, destroy that shit’ like money going up in smoke if you don’t.
P.S. don’t park in front of fire hydrants, because they will have a shit eating grin on their face when they destroy your car- ahem - clear the obstacle - when they need to use it to stop a fire.
Not to speak for the other poster, but yes, they had people experiencing difficulties getting into the data centers to fix the problems.
I remember seeing a meme for a cover of "Meta Data Center Simulator 2021" where hands were holding an angle grinder with rows of server racks in the background.
"Meta Data Center Simulator 2021: As Real As It Gets (TM)"
Yes for some insane reason facebook had EVERYTHING on a single network. The door access not working when you lose BGP routes is especially bad because normal door access systems cache access rules on the local door controllers and thus still work when they lose connectivity to the central server.
Depends. Some have a paranoid mode without caching, because then a physical attacker cannot snip a cable and then use a stolen keycard as easily or something. We had an audit force us to disable caching, which promptly went south at a power outage 2 months later where the electricians couldn't get into the switch room anymore. The door was easy to overcome, however, just a little fiddling with a credit card, no heroic hydraulic press story ;)
If you aren't going to cache locally than you need redundant access to the server like LTE access and plan for needing to unlock the doors if you lose access to the server.
This sounds similar to AWS services depending on DynamoDB, which sounds like what happened here. Even if under the hood parts of AWS depend on Dynamo, it should be a walled-off instance separate from Dynamo available via us-east-1.
I was there at the time, for anyone outside of the core networking teams it was functionally a snow day. I had my manager's phone number, and basically established that everyone was in the same boat and went to the park.
Core services teams had backup communication systems in place prior to that though. IIRC it was a private IRC on separate infra specifically for that type of scenario.
I remember working for a company who insisted all teams had to usr whatever corp instant messaging/chat app but our sysadmin+network team maintained a jabber server + a bunch of core documentation synchronized on a vps in a totally different infrastructure just in case and sure enough there was that a day it came handy.
Ah, but have they verified how far down the turtles go, and has that changed since they verified it?
In the mid-2000s most of the conference call traffic started leaving copper T1s and going onto fiber and/or SIP switches managed by Level3, Global Crossing, Qwest, etc. Those companies combined over time into Century Link which was then rebranded Lumen.
That's similar to the total outage of all Rogers services in Canada back on July 7th 2022. It was compounded by the fact that the outage took out all Rogers cell phone service, making it impossible for Rogers employees to communicate with each other during the outage. A unified network means a unified failure mode.
Thankfully none of my 10 Gbps wavelengths were impacted. Oh did I appreciate my aversion to >= layer 2 services in my transport network!
That's kind of a weird ops story, since SRE 101 for oncall is to not rely on the system you're oncall for to resolve outages in it. This means if you're oncall for communications of some kind, you must have some other independent means of reaching eachother (even if it's a competitor phone network)
That is heavily contingent on the assumption that the dependencies between services are well documented and understood by the people building the systems.
Rogers is perhaps best described as a confederacy of independent acquisitions. In working with their sales team, I have had to tell them where there facilities are as the sales engineers don't always know about all of the assets that Rogers owns.
There's also the insistence that Rogers employees should use Rogers services. Paying for every Rogers employee to have Bell cell phone would not sit well with their executives.
That the risk assessments of the changes being made to the router configuration were incorrect also contributed to the outage.
There is always that point you reach where someone has to get on a plane with their hardware token and fly to another data centre to reset the thing that maintains the thing that gives keys to the thing that makes the whole world go round.
Not sure if this counts fully as 'distributed' here, but we (Authentik Security) help many companies self-host authentik multi-region or in (private cloud + on-prem) to allow for quick IAM failover and more reliability than IAMaaS.
There's also "identity orchestration" tools like Strata that let you use multiple IdPs in multiple clouds, but then your new weakest link is the orchestration platform.
Disclosure: I work for FusionAuth, a competitor of Authentik.
Curious. Is your solution active-active or active-passive? We've implemented multi-region active-passive CIAM/IAM in our hosted solution[0]. We've found that meets needs of many of our clients.
I'm only aware of one CIAM solution that seems to have active-active: Ory. And even then I think they shard the user data[1].
Ory’s setup is indeed true multi-region active-active; not just sharded or active-passive failover.
Each region runs a full stack capable of handling both read and write operations, with global data consistency and locality guarantees.
We’ll soon publish a case study with a customer that uses this setup that goes deeper into how Ory handles multi-region deployments in production (latency, data residency, and HA patterns). It’ll include some of the technical details missing from that earlier blog post you linked.
Keep an eye out!
Wow, you really *have* to exercise the region failover to know if it works, eh? And that confidence gets weaker the longer it’s been since the last failover I imagine too. Thanks for sharing what you learned.
You should assume it will not work unless you test it regularly. That's a big part of why having active/active multi-region is attractive, even though it's much more complex.
Sure it was, you just needed to login to the console via a different regional endpoint. No problems accessing systems from ap-southeast-2 for us during this entire event, just couldn’t access the management planes that are hosted exclusively in us-east-1.
It's a good reminder actually that if you don't test the failover process, you have no failover process. The CTO or VP of Engineering should be held accountable for not making sure that the failover process is tested multiple times a month and should be seamless.
Too much armor makes you immobile. Will your security org be held to task for this? This should permanently slow down all of their future initiatives because it’s clear they have been running “faster than possible” for some time.
Totally ridiculous that AWS wouldn't by default make it multi-region and warn you heavily that your multi-region service is tied to a single region for identity.
I always find it interesting how many large enterprises have all these DR guidelines but fail to ever test. Glad to hear that everything came back alright
People will continue to purchase Mutli-AZ and multi-region even though you have proved what a scam it is. If east region goes down, ALL amazon goes down, feel free to change my mind. STOP paying double rates for multi region.
DHH wrote a blog post complaining about how there are fewer "native Brits" in London, and then linked to Wikipedia's article about the number of white people in London. He also brought up a march by Tommy Robinson, but framed it as just a couple of exceeding normal guys out for a walk, and not a bunch of nationalists.
It came off as xenophobic and racist, so sponsors pulled funding while others (some quite high profile) refuse to work with DHH. There's a non-zero amount of reading between the lines, so here's the blog post so everyone can decide for themselves:
I have been a Linux desktop user for 20+ years. It is incredible how far it has come. There is nothing Microsoft can do that will drive the normies away though. Microsoft knows this and that is why we are where we are.
I have been running Linux since 2011, and so much more stuff is in the “Just Works” category, especially if you have AMD graphics. When I installed NixOS on my Thinkpad about a year ago, it was almost comical how easy it was for me; I had gotten used to having to waste an entire day messing with drivers and fixing issues in 2012-2015, so it felt kind of weird for stuff to work as expected immediately.
I am trying very hard to get my parents to use something like Linux Mint because the Windows 11 auto-update on my mom’s computer actually prevented it from booting (making me waste an entire day remotely having them flash a live USB so I could rsync over her files to me…thanks MS!), so this might be enough of a final straw for them.
I have tried switching family members over after malware incidents. The most success was setting my 80 year old grandmother up with Lubuntu. She had no issue picking it up. I don’t think she even really noticed vs Windows. Lasted a few years until she went to an iPad for accessibility reasons.
When I was in school, decades ago now, very few people went into CS compared to other majors. Everyone I knew going into it did it because they loved it. I would have done it regardless of the career opportunities because I want to build stuff.
Interviewing candidates over the years since then, my experience has been there are still very few of those passionate nerds and a lot of people who did it for other reasons, like the money or similar. There is nothing inherently wrong with this. I don’t fault people for it.
Maybe if we get very lucky, it will go back to a relatively few passionate people building stuff because it is cool?
reply