It's crazy to me that this is possible in the first place. Standard practice is to have a fleet of test vehicles that are effectively production except in an early release group.
Or, you know, having an A/B boot partition scheme with a watchdog. Things that have been around for decades at this point.
Disclaimer: Former Googler, Worked closely with Automotive.
To me it's all-too-understandable how this is possible.
Maybe they've got a test fleet, but it accepts code signed with the test build key.
Maybe they've got a watchdog timer, but it doesn't get configured until later in the boot process.
Maybe they've got A/B boot partitions, but trouble counting their boot attempts - maybe they don't have any writable storage that early in the boot process.
I wouldn't be surprised if, as a newer company, they'd made a 'Minimum Viable Product' secure boot setup & release procedure, and the auto-fallback and fat-finger-protection were waiting to get to the top of the backlog.
So, using Polestar as a reference as it's both a vehicle that I've worked on, and one that I personally drive.
> Maybe they've got a test fleet, but it accepts code signed with the test build key.
Polestar solves this by only delivering signed updates to their vehicles. The vehicle headunit will refuse to flash a partition that isn't signed by the private key held by Polestar. Pulls double duty to prevent someone from flashing a malicious update, as well as corruption detection.
> Maybe they've got a watchdog timer, but it doesn't get configured until later in the boot process.
Based on what the Rivian reports are showing (Speedometer, cameras, safety systems are working), they likely are running their infotainment as a "virtual machine" within their systems. Again, something that Polestar does.
Implementation of a watchdog with a "sub-system" like this is relatively braindead simple.
> Maybe they've got A/B boot partitions, but trouble counting their boot attempts - maybe they don't have any writable storage that early in the boot process.
Generally, A/B partitioning is part of the bootloader, the first program that executes after the reset (on many modern processors) pin is released. This also leads to reboot counters and such being stored as part of the NVRAM that is available at boot.
Opinion: Maybe I'm biased, but maybe if you can't develop something yourself, there's reason for you to get an off the shelf option that handles a lot of these things.
Disclaimer: Former Googler, Worked closely with Automotive.
To be honest, I don't think Polestar set a very high bar for software quality.
I am currently renting a Polestar 2 from Hertz, and sometimes the HUD doesn't work(it's 50/50 if it will turn on).
That means, I don't see speed, battery charge, etc, while driving.
Infotainment system is working though.
You're definitely correct in the fact that Polestar isn't the highest in software quality, but it was the examples of what they did right that I wanted to focus on.
They're garbage when it comes to their mobile app and some of the controls on the infotainment system.
> sometimes the HUD doesn't work(it's 50/50 if it will turn on).
You should definitely reach out to Hertz and ask for a car swap. Sounds like there's a bad connection between the display and the IHU. Both screens are operated by the same system, so it's unlikely to be a software failure.
Teslas occasionally need to reboot / hard reset their software too, when driving no less, and during that period all that information, and most of the controls, are unavailable (like windshield wipers, etc.)
I feel you. I've had my machine blocked for 5 hours as it was pushing an update down my throat. Luckily it gave me a 2 minute warning before the initial reboot so that I could close everything down.
As that was during a client emergency, I had a very happy client that day.
That's not true. You cannot shift gears in the Model X, S, or 3 without the touchscreen - the only way to change into or out of park, drive, or reverse is to swipe on the touchscreen. Only the Model Y has a stalk and it is being removed in the next version.
Also even on older Teslas lights are controlled exclusively via touchscreen.
I don't know what to tell you, you are wrong. I drive a Tesla, I have specifically checked all of what I mentioned when I had this (in my opinion highly problematic) issue happen to me. Have you actually tried it?
As someone who lives in a place where it rains almost every day for 9 months of the year, this reinforces my decision to never buy a Tesla. Does this kill the headlights too?
I had it happen two times this year, but the original comment is wrong, you can use the wipers (and also the headlights) while the touch screen is rebooting.
Answer: To clarify, more that a company should stick to their core competencies unless there's a drastic need or opening in the market that could be filled (and to build a new competency).
In this particular case, there's nothing particularly unique that Rivian is doing with their Infotainment system that couldn't already be handled by an incumbent in the space, (Android Automotive, QNX, etc.) especially given how modular the systems themselves are.
As State Farm says, "We know a thing or two because we've seen a thing or two".
Rivian thinks of itself as a software company. The first thing you sign when you go to buy a vehicle is a software copyright notice IIRC. The first thing in the owner's manual is a notice that the software copyrights and intellectual property belong to Rivian, etc etc.
> The vehicle headunit will refuse to flash a partition that isn't signed by the private key held by Polestar. Pulls double duty to prevent someone from flashing a malicious update, as well as corruption detection.
And of course preventing people from modifying and controlling hardware that they own, having paid 6 figures for (in the case of the Polestar 3 anyway). But that's table stakes for embedded systems in this day and age. Security for me, not for thee.
> Maybe they've got A/B boot partitions, but trouble counting their boot attempts - maybe they don't have any writable storage that early in the boot process.
You do not report a successful boot until and unless the entire system loads up successfully. You will definitely have writable storage by then.
I understand the sentiment, but think about the alternatives.
There are a few different kinds of updates that can be applied, each with their own protective layers.
Infotainment updates, like what happened to Rivian aren't that dangerous. You lose "convienience features" like maps, air con, etc, but generally nothing that could kill you or someone else.
Then there's system updates, which is where danger noodle things happen. Automotive manufacturers are significantly more risk averse to updating these components, and generally, if _anything_ within the system looks wonky, it's an immediate revert.
If I, as a Polestar owner, wanted to get an update for my vehicle, the nearest service center is 1.5h away. If I lived in Montana (United States), it would be realistically impossible for me to update my car. Thus, if we want to enable competition within the markets, we shouldn't have regulations that force a new manufacturer to have a global network just to add CarPlay to a screen.
Dad has a 1966 Oldsmobile with air conditioning. In the last 57 years General Motors has never found a need to update the switch. It still works flawlessly.
It’s stupid that we invented a way to not only remotely break an on/off switch but also a culture that rolls the dice on that until the inevitable happens.
>Infotainment updates, like what happened to Rivian aren't that dangerous. You lose "convienience features" like maps, air con, etc, but generally nothing that could kill you or someone else.
Also speedometer, which is hardly a convenience feature.
On the other hand, we update irreplaceable spacecraft billions of miles away with new software.
It should be fine to push software updates out, as long as the correct safety and fallback procedures are in place. It simply has to be designed to handle failure and procedures need to be in place to mitigate risks.
It sounds like that wasn't the case here. Also, why wouldn't you have a small initial release pool when you have such a large potential for disruption?
The art of shipping software—like on a disk, where once it’s out the door, it’s out the door and you may never get another shot—is dead or dying. Even in some embedded areas of the industry now.
> What amazes me is that any grown up person thinks it is a good idea to update vehicles as if they were telephones
What amazes me is that any grown up person thinks it is a good idea to update telephones as if they were software and not phones.
Or rather that it is a good idea to have phones that need updates? Either way, we're all one 1/2 assed push update to a fridge, vacuum, washing machine, phone or car away from a really annoying day.
As the update only affects infotainment and not critical systems, it seems like a reasonable tradeoff to me. Just because a car can fail in ways that kill people doesn't mean all parts of a car are equally critical.
This isn't true. If you look at the release notes for any of Rivian's updates they all include vehicle related firmware changes. This is not simply infotainment.
Beyond that "infotainment" includes driver critical information - like the speedometer which, for many affected, means there's no working driver screen.
Yeah... I worked on an embedded project with literally 2 engineers, and we had an A/B partitioning scheme, and a recovery partition (we fully qualified the recovery image and it was flashed to the units on day 1, it was guaranteed to boot and it would just sit and wait for the user to initiate a firmware load). The app on the device would reset a U-boot variable once it was successfully loaded, so U-boot could check the number of failed boot attempts. If it was >= 5 reboot attempts without booting successfully, it would go into the recovery partition.
There's really no excuse from Rivian on this, this is shoddy
I interviewed at Rivian. They told me about how they needed to grant users access to things like keys, AC, ignition, etc. So they built a hierarchical, recursive group checking IAM system.
That just felt like a massive product to build and maintain for what really could have been backed by AWS iam. GCP IAM if they really really needed hierarchy. I guess I'm not surprised at this outage.
Rivian does have a test fleet, and they test it for weeks before releasing.
This particular issue is because they apparently distributed the firmware signed with the wrong cert.
Not a bug in the software itself.
That is independent of testing the software, but still a distribution issue.
* "signed with the wrong cert" should mean the software package is rejected before it it is installed.
* software upgrades are tricky and there should be at least 2 versions available so that fallback to the previous is possible and automatic in case of issues.
It is great to live in an ideal world, and in fact, in most software, you can do what you are suggesting quite cheaply. But once you get past the sort of "quip on hacker-news" level of thinking about this, or trivial and cheap production testing scenarios, people have to make real tradeoffs because it's never that simple.
Talking about those is much more interesting than just asserting that everything should be a certain way, without any consideration for real world constraints, like cost of units, etc.
It would be interesting if you could develop about what you mean by real world constraints and how cost of units affect what I wrote, which I did from real world experience.
I'm not defensive, I just find your extreme position remarkably silly.
You included nothing about how costs affect anything. You simply assert that you should always test prod on pristine production units.
There are plenty of times outside of software where production units cost millions or you can only produce them so quickly, or both, and where your extreme take would result in remarkable cost or a competitor eating your lunch.
Which is precisely why its not done, and in the real world tradeoffs are made between what really needs 100% assurance and not. Spending money or losing customers for 5 9s of reliability through testing when two are needed is not a best practice, and is often explicitly called out as such.
In the case of rivian, maintaining a significant fleet of expensive, pristine, exact customer spec (ie not debuggable) cars just to try to get 100% prod ota success assurance is unlikely to provide value vs getting 98% assurance and not doing that (by rough calculation, it stands at 98% after this incident).
My position is neither silly nor extreme. It's the way it is usually done and other comments here have been along the same lines.
In fact you are trying to spin what I wrote to an extreme to make your point.
By the way, it is not about 100% success assurance but assurance that failure does not brick the unit. This is an assurance that should be, and can be, close to 100%, indeed a good number of 9s because, obviously you cannot brick 2 cars out of 100 for every software upgrade!
I've never worked in automotive but it's pretty easy to imagine how this might play out in a car, where a single update might bundle updates for several programmable devices.
It's easy to imagine a central SoC receiving the update, verifying its signature against a local key and then reprogramming some MCU over an internal interface. But then after resetting the MCU, you realize that the image you just flashed isn't compatible with the boot security keys burned into that MCU. It's not uncommon for a device performing the OTA update to not have access to the "source of truth" keys / certificates used to verify the updated image at boot time.
Not that this is a great excuse. If you add OTA updates to a product that has this design, you should really be confident in your recovery solution.
A/B partitions tends to solve that. You will only switch to the new partition when the update is 100% verified installed. If it doesn't complete in an atomic manner, your device will just boot into the previous healthy partition.
A/B gets complicated in the real world. BL1 may not support A/B for example, so to implement A/B bootloaders you may need a shim that can read/write NVM to handle that. Your HSM may not have slots for multiple keys to have different signatures, so upgrading one may trample the other if your update code doesn't check that.
Lots of ways to screw this up, especially in automotive where you're likely to be dealing with TI and their (in)secure boot.
I've solved this problem god only knows how many times now and I've rarely found an automotive board that doesn't introduce fun, new edge cases. OTA can't exceed x kilobytes of memory, the processor isn't fast enough to verify signatures and write the image in < x seconds, can't write the image to flash unless the signature is verified, but the image doesn't fit in RAM, the server delivering the update is 3+ networks away from the device receiving the update, etc.
With all due respect, that all sounds like Programmer Induced Problems(tm).
Cars are a long solved problem, being around for over a century. Telephony and computing hardware and infrastructure today are in the realm of the ludicrously good compared to even just a few decades ago, even if we consider bottom of the barrel worst case scenarios. If software somehow can't work a solved problem using ludicrously good hardware, the programmers (and their managers) are the problem.
I agree here. A partition is just a partition. It's taking one disk and abstracting it into two. This is not hard.
The rovers on Mars, the Voyager, these problems have been solved for a long time. The compute in a Tesla these days can probably run Crysis.
You can do this OTA to a Raspberry Pi running Nerves via remote SSH and it works really well. The Nerves runtime utilizes A/B partitions for OTA updates.
I once worked at a startup that sold a very expensive "enterprise" network appliance that didn't have partitioning. I had worked on network appliances before which did have this capability, so I asked the VP of engineering about it. He said it just wasn't a priority given all the other work that needed to be done. I wouldn't be surprised if Rivian had the same startup mentality that might lead to such a situation.
Let's use two of the examples I gave above. How would you go about modifying the silicon to support new features in the boot rom, and how do you get around the pigeonhole principle when that silicon vendor doesn't ship enough bits of OTP memory to store multiple keys? M-x butterfly doesn't quite get there, but maybe there's another Emacs command I could use?
You choose different silicon? You get a different vendor? You build your own boards and chips?
For a vehicle that costs $100k+ it shouldn't be hard to double or triple the budget for onboard compute considering it is vital to the operation of the entire vehicle.
Note that I never claimed the automotive world has functional people processes. Functional people processes make a lot of technical issues much easier, but they're usually off the table in traditional manufacturing. The security team insists on x requirements for security reasons. Hardware team insists on this chip because it's the only one that makes the budget work.
A shockingly large part of my job is telling both that one team won't be getting what they want and to work it out among themselves. Rinse and repeat between dozens of boards because the relevant teams don't talk to each other and none of them read the "design requirements to ensure we don't have to tell you no" doc either. One time they didn't even tell us there was a board until the end of December, when delivery was scheduled for Feb.
I live in Detroit so what you’re saying does not surprise me in the least bit. I have some friends in automotive at some of the big 3, and suppliers, and I’ve heard some really terrible stories.
It'd be pretty silly to implement an OTA scheme that didn't check signatures before installing updates. That would mean any random attacker could soft-brick the module by sending an invalid image, which a development image should be to a production vehicle.
You could get this situation if the application code accepted signatures the bootloader does not though. I can imagine that accidentally occurring.
Sure, but it could also be signed binaries inside the OTA, a variant of what you are talking about.
IE OTA is a signed package, inside package are also signed binaries.
OTA itself is properly signed, a single binary (infotainment) is signed with wrong key.
While most OTA verifiers will verify the OTA signature (which this would pass), most don't verify the individual inside-package binary signatures at install time, only at runtime.
That's not how I've ever implemented OTA, but I'll grant that it's possible if it was designed by someone with no idea what they were doing. Certainly not a good OTA process though, for this among many other reasons.
It makes sense for each component to check signatures of its code to prevent various kinds of attacks -- e.g. someone coming and reflashing just infotainment or motor controllers with something malicious.
So, OTA update comes in, containing a bundle of software for different subsystems. It's sent to different subsystems. Then, those subsystems check integrity at startup, but one subsystem's bootloader isn't happy because the firmware looks to be invalid.
You can only prevent this if the OTA knows how to do equivalent verification for every subsystem in the car that checks integrity. (And, of course, even if you do this, there's other ways you can go wrong that aren't specific to integrity checks).
OTA is generally developed by tier 1, so this is probably a bug in the tier 1's code. (Samsung, Panasonic, Sony, etc are common tier 1s in this space.)
I am still not sure why I would update software on car, a piece of hardware that, IMHO, shoupd be able to run air gapped 24/7. Exceptions: recurring bugs, GPS maps and security updates. All of which can be done either during service (preferred, if they brick it, they are liable) or by plugging in something. OTA updates just seem completely pointless...
Edit: Also, why the heck isn't the entertainment system completely air gapped from the software running the car?
Rivian consistently ships a lot of new features and improvements, you can see the changelogs here [1]. I think you can pretty fairly critique a lot of them with: They are just solving a problem that they created by making it too techy, or they are shipping stuff they should have completed before releasing the product. I do think its hard to argue that the updates aren't adding anything of value though. There's convenience stuff like pet mode or bird's eye camera view that were added after release, but there are also things like new driving modes (soft sand and snow) or improved DC charging curves and smarter battery conditioning that legitimately improve the quality of the product as a vehicle.
> Edit: Also, why the heck isn't the entertainment system completely air gapped from the software running the car?
As for this, the entertainment system can control basically every feature of the car and is often the primary or only way to accomplish certain things. Even in much much dumber cars the infotainment is still part of the CAN bus and is able to interact with the rest of the vehicle.
Funny, our 2020 MY Jaguar controls car functions from the digital screen in front of the driver, the middle console screens only control AC, entertainment, phone, navigation and other non-car related stuff. No idea how the architecture looks behind all that so. But seriously, even if on the same bus, just don't the media player, radio and connected phone access to the systems actually running the car from engine to brakes. And please, please, finish developing the embedded software running on car before shipping said car. Then it can be air gapped, if not it requires OTA and internet access, raising all kinds of security issues...
And please, please, finish developing the embedded software running on car before shipping said car.
Sorry, but things no longer work that way, and never will again. This is a good thing, as long as processes are improved to avoid situations like this one.
It's supposedly a good thing because more features can be added over time, but if they were features worth adding in the first place, the car should have shipped with them already.
That is the big difference between hardware and software engineering: Once hardware is shipped, there is nothing you can do besides repaurs and retrofits requiring a workshop. Software can updated and changed today anytime.
My biggest issue with modern cars, and it seems this is spreading to other embedded systems than cars, is treating those as software: connect them to the web and run OTA updates everytime you need to fix a bug. That requires some form of inzernet cinnection, which requires regular security updates, which require OTA update capability. All that because, bluntly, software devs cannot be bothered with just finishing software running on non-connected hardware that just works, not can they be bothered following hardware development in case hardware is the main component, as in cars. And no, no car manufacturer is a software company, nor phone maker like Apple.
The proper way of fixing software bigs in automotive used to be, again thank you Tesla for breaking something that just worked, to recall the affected cars to a workshop to conduct the software update. Honestly, the only thing on a car that should be updated by the owner is maps for the GPS unit.
You may not have a say in the matter. Most cars are on some kind of IoT private network with their own cell modems and if the manufacturer decides to push an update there isn't much you can do about it short of ripping out the cell modem. Which may well have unpredictable consequences.
Is not reducing the effective cost of a bad update by 10x or more worthwhile?
Sure, but if you are rolling out to 1% of users per hour, you detect the problem in a couple of hours and much fewer than 2% of users will have applied the update. This is a relatively small support problem.
While if you roll out to everyone at once, you'll detect the problem sooner (within an hour) but have 10x as many affected.
Or, you know, having an A/B boot partition scheme with a watchdog. Things that have been around for decades at this point.
Disclaimer: Former Googler, Worked closely with Automotive.