I built a whole remote software update mechanism for a control binary that ran on 25k+ servers across multiple data centers.
Rest assured that after the first time I messed it up (which required ssh into each box individually), I wrote a lot of unit and integration tests to make sure that it never failed to deploy again. One of the integration tests ensured that the app started up and could always go through the internal auto update process. This ran in CI and would fail the build if it didn't pass.
While I fully understand that this is hard to get right 100% of the time, a mess up of this level by a car manufacturer is pretty amazing to me.
Rivian is an embedded use case, though, which is not at all like a fleet of servers.
Having worked for companies that produce network devices - including devices that are unreachable for example for 6 months of the year - and on software installation and upgrade, I am baffled how this bricking is possible. For one thing, you generally use some kind of confirmed boot mechanism - you upgrade a standby partition, set an ephemeral boot value that causes device to boot the alternate image, and reboot - only when the image is declared "up" does that get persisted (and then the alternate is upgraded, in order to prevent rollback in the event of a media error). You use watchdogs that are tied to actual forward progress (and not just some demon that the kernel schedules and bangs on the watchdog even if the rest of the system is hung) and if they fail, the WD reboots you. (This is one of the reasons that event driven programming is somewhat preferred - actually processing events from a single dispatch thread makes it easier to reason about the system.)
On top of that, you make sure that the core system is an immutable filesystem so that you can validate the _offline_ alternate image before rebooting (write-and-read-back-uncached) and periodically scrub the alternate image (same).
Like.. this is all embedded 101, stuff people have been widely doing since the mid 1990s and I think I can find examples going back to the 70s. Sometimes you get a little more sophisticated (allow sub-packages or overlays and use a manifest to check the ensemble instead of just a single image), but it's very standard.
Assuming Rivian does know embedded 101, my guess is that the infotainment system is running Android and the watchdog reported all green once the system services all came online and that it doesn't actually check whether the application layer is really working because, as you know, that would require the watchdog to run a full regression suite before giving the okay, which isn’t practical. Since the update swapped the system to an internal dev cert, they cant push an immediate update to change the boot args because the management plane daemon won’t connect to the C&C server, or it can but the blob they push wouldn’t pass signature validation, or the TEE won’t unlock the device keys because the roots changed. Whatever the case, someone has to go blow a fuse and re-flash the thing, or at least rewrite the boot args via serial. Just a guess.
If it is the most likely “management plane TLS certs” issue, I bet the watchdog won’t confirm the new boot args until the command dispatch daemon gets a pong from the C&C server moving forward (:
Hey now, preventing SEVs doesn't lead to impact. If we all collectively let this become a raging dumpster fire we can all heroically fix it and greatly exceed expectations for the half.
Did you just use standard Yocto or similar tools to build such images? Are there standard daemons for managing hardware watchdogs (besides systemd since that's too simple as you say)? I think there's a lot of niche knowledge in the embedded space and many programmers are used to cloud systems and at most target. The most embedded experience most programmers have is likely iOS/Android development where all of the actual embedded concerns are handled for you. Even Google (soft)bricked a bunch of phones with the latest Android 14 update [1].
IMO there's not a lot of regular OSS for building embedded systems that comes with A/B partitioning, watchdogs, secure and verified boot - it's all custom at every org and tailored for individual products.
As somebody currently working at an automaker on software systems, the amazing thing to me is that a mess up of this level doesn’t happen weekly. It’s rough out here.
Thank you. At least you're honest about it, the other day someone was trying real hard to convince me that software developers at automakers are made of magic fairy dust.
Relatively crappy pay, complex toolchains, long build times, layer upon layer of (really bad) legacy code, badly specified (if they're specified) protocols between subsystems, subsystems that are completely opaque (no source code provided), homegrown OS's or older RTOS's, subset-of-C to keep it safe(r), tricky debugging environments and if you're really unlucky anemic hardware.
I hope I didn't miss anything but I wouldn't be surprised if I did.
Yeah, I think you missed something. The "software architects, heavy enterprisey tooling, and minions" approach to development where some of the architects could be good developers, but they don't develop, and the minions are often not that good and also not given any autonomy, so they are in a state of learned helplessness and just do what they're told without much thinking or initiative. It results in over-abstracted, over-complicated, slow, unreliable, and sometimes just stupid code.
Automotive varies widely between "basically modern Linux systems with proper updates" and the most janky, home-grown update systems imaginable, sometimes even within the same components and teams.
Yah, I know from friends at ford and vw that there's still vxworks and qnx, but even there, good grief, a-b with confirmed boot is about as basic as you can get.
I confess I've seen incredible sloppiness about when a confirmation is done (too early, including in the initial init stages which is way too soon) and watchdogs (spawn off a process that has a while loop stroking the wd - just absolutely pointless).
I've heard all of the above, often "stroking". I never used those because I like systems where you have a random challenge code to respond to. Then the software has to not be acting as wonky to react correctly.
Indeed, a good RTOS from 10-20 years ago works just as good now as it did back then. The only things that change are dev environments and the driver support.
> This ran in CI and would fail the build if it didn't pass.
I don't mean to be pedantic, but since we're talking about what should happen instead, this is insufficient. It works until the day you realize you made some kind of manual change to your CI infra, or that CI has some non-standard configuration that makes it work for you but not some significant fraction of the fleet.
People should do what you described in CI, but as well as that, you need phased rollout, where e.g. the build can only be rolled out to the next percentage point of randomly selected users in a specific segment (e.g. each hardware revision and country as independent segments) after meeting a ratio of successful check-ins, in the field, from the new build by production customers in that segment. That's the actual metric for proceeding with the rollout: actual customers are successfully checking in from the new version of the software.
Except, that's actually not sufficient either. What if the new build is good, but it contains an update to the updater which bricks the updater? Now you're getting successful check-ins from the new version in the field, but none of those customers will ever successfully auto-update again. So, test the new updater's ability to go forwards successfully, too.
A good way to handle the who-updates-the-updater issue is to use a triple partition updater. A updates B, and then B updates C, then C updates A. If anything about the new version prevents it from properly updating its neighbor, that neighbor won't be able to close the loop, and you'll fall back to A. This simplifies the FSBL, because it just boots the three partitions in a loop, no failure detection required. You don't need to triplicate the full application either, just the minimum system needed to perform an update, and then have the "application" in it's own partition to be called by the updater.
> It works until the day you realize you made some kind of manual change to your CI infra, or that CI has some non-standard configuration that makes it work for you but not some significant fraction of the fleet.
Nah, my CI process was solid. This was proven in the field over the course of years.
> I don't mean to be pedantic... you need phased rollout
You don't need to be pedantic, but better to ask the question rather than assume that was all that I did. =) You have to realize that what I built, worked flawlessly. It wasn't easy either, took a lot of trial and error.
I did have a CIDR based rollout. I could specify down to the individual box that it would run a specific version. Or I could write "latest" to always keep certain boxes running on the latest build. This was another part of my testing, but ended up not being fully necessary because I had enough automated testing in CI that "latest" always worked.
> but it contains an update to the updater which bricks the updater?
This happened, so I wrote a lot of test code to make sure that would never happen again. My CI would catch that since I was E2E testing that it could actually run the upgrade process.
Once I implemented all of this, I never had a single failure and would routinely, several times a day, deploy to the entire cluster, over the course of a couple years.
It was all eventually consistent as I could also control the "check for update" frequency as well.
I think there's a minor confusion here, where you think the purpose of my response involves doubting whether your system was successful. I understand it was successful. My response is to the sense in which your comment can be interpreted as advice to other people on what they should build.
I think the fact that you were able to survive with CI-only doesn't mean that we should encourage others to skip implementing a phased rollout based on verified customer successes, including testing of their new updaters before the first time they accidentally brick all the updaters, rather than afterwards. That's what I was hoping to help avoid, through my comment.
Interesting. At any point in time, I had errors from hardware, software and networking. Even the racks would be getting overwhelmed at certain times. Simply being able to ssh into every host wasn't guaranteed. I'm not sure how you did it.
+1 to this, we have a 0.1% hardware failure rate every time we do a rolling restart (40-50k nodes). Some just never come back, in the best case, but actively misbehave in the worst. If the node is unresponsive we remove it from the cluster and fix it async.
If the daemon was running, it would ping a central server on a schedule and report its status, the response from the server was if there was a new version available (with the binary in the response), or not. This combined ping/update service really cut down on the overall traffic, and failures.
If the machine had crashed, it would start up, start my daemon, and that daemon would start the ping/update process all over again.
A large portion of the machines were iPXE booted... so, just reboot was one option and it would all start from scratch again.
Yes, some of the boxes had flaky power supplies or would fail an ssd, and that would cause a technician to go out and manually fix things.
I found it was critical to think of everything as eventually consistent because my hardware was boxes with 12 GPUs and they were flaky and would crash the whole box randomly. I got used to boxes rebooting hundreds of times. My process would also auto-tune the GPU for stability too, changing clock/power settings until the individual cards would become stable and stop the crashing.
The only time I had problems was when the daemon was dead. I had a dashboard where I could see which machines hadn't reported their status. It was easy to pick those off by hand.
I used to wear your shoes in IE6/7 ages (no longer, I gave up during the "framework of the week" race and went all-backend), and it wasn't simple at all. Browser compatibility with all their rendering nuances, individual system oddities and all sort of fragile stuff.
And fortunately, no one bats an eye at a slightly broken site, but everyone hates even a slightly broken vehicle.
It's simple because we tolerate certain limitations in the web platform.
If you had a hard requirement that a page load could never take more than 100ms, regardless of network conditions, you'd have quite a challenge on your hands.
I'm not really a frontend dev any more but I was for a long time. I can assure you that the only reason you think your code works is because no one tells you it's broken. If you use an error logging or telemetry service (Sentry, Rollbar, New Relic, etc) you will be aware that errors happen in frontend code all the time. It's just that most of the time bugs don't crash the app, and the user doesn't know what to expect so they see a broken feature and think it's meant to be like that.
> While I fully understand that this is hard to get right 100% of the time, a mess up of this level by a car manufacturer is pretty amazing to me.
I feel like it's going to happen to someone that makes network devices eventually. I'm always scared to update my (several hundred) UniFi devices. Their update process isn't foolproof and they push auto-updates via the UI pretty hard.
Several years ago they caused some people's devices to disconnect from the management controller when they enabled 'https' communication. Prior to that, if you were pointing devices at 'https://example.com:8080...' they would ignore the 'https' part and do an 'http' request to port '8080'. Then they pushed their 'https' update which expected an 'https' connection and didn't fall back to the old behavior for anyone that was mistakenly using 'https' in their URL initially. Some people on their forums complained about having to manually SSH to every device to fix the issue.
It was caused by an end-user mistake, but they knew it was a potential issue. AFAIK, their attitude on it hasn't changed and a lot and at the time their response was that they knew it would break some people, but that it wouldn't be that many (lol).
IMO, the issue with those systems is that basic communication back to the update / config server is part of the total package which is too complex (ie: a full Debian install). I'd rather see something like Mender (mender.io) where the core communications / updates come from a hardened system with watchdog, recovery, rollback logic.
Think of how crazy it is to have something like pfSense doing package based updates rather than slice based updates. At least with boot environments they could add some watchdog and rollback type logic, but it'll still be part of the total system instead of something like a hardened slice based setup where the most critical logic is isolated from everything else and treated like a princess.
Do you have any insight on package vs slice based systems for updates? Did you isolate update logic from the rest of the system or am I out of touch with that opinion?
When possible, I used a fail back mechanism. If the update failed to fully come up, then the watchdog timer would catch it, the bootloader would notice the incomplete boot, and attempt to boot from the previous known working image in that case.
out of morbid curiosity.... how long did it take to ssh into and fix all of those servers? I imagine even automating a fix (if possible) would still take a good amount of time.
The way I built my app was that I could install it cleanly via a curl | bash.
So, I just had a simple shell script that iterated through the list of IP addresses (from the DHCP leases), ran curl | bash and that cleaned up the mess pretty quickly.
As a non-developer, the whole situation with a bad software update to the Voyager spacecraft really puts things into perspective as far as how bad remote updates can be.
It’s also a testament to the way that the system was designed that they were able to get it back online.
I used to work for a company that built satellite receivers that would be installed in all sorts of weird remote environments in order to pull radio or tv from satellite and rebroadcast locally.
If we pushed a broken update it might mean someone from the radio company would have to make a trip to go pull the device and send it to us physically.
Our upgrader did not run as root, but one time we had to move a file as root.. so I had to figure out a way to exploit our machine reliably from a local user, gain root, and move the file out of the way. We'd then deploy this over the satellite head end and N remote units would receive and run the upgrade autonomously. Fun stuff.
Turns out we had a separate process running that listened on a local socket and would run any command it received as root. Nobody remembered building or releasing it but it made my work quick.
No no, I figured that out afterwards, in a past development iteration someone added it on purpose and then forgot all about it - "oh yeah we needed that to <solve some mundane problem>".
So... worse than subterfuge? That being said it only listened on the local socket, so it's slightly less bad, and I don't want to get into the myriad of correct ways that original problem could have been solved, but lets just say that company doesn't exist anymore.
> Turns out we had a separate process running that listened on a local socket and would run any command it received as root. Nobody remembered building or releasing it but it made my work quick.
No offense, but what a shit show. It makes me assume no source control, and a really good chance that state actors made their way into your network/product. This almost happened at a communication startup I know, with three letter agencies helping resolve it. State actors really like infiltrating communication stuffs.
oh, yeah, this place was a total shit show. BUT we were ISO9001 certified!! So we had source control (CVS) and a Process (with a capital P) to follow. In this case that code was added in a previous development iteration because someone needed to run something as root when a user pressed a certain button on the LCD panel in front and this was the decoupled solution they wrote intentionally. Somehow I feel like that makes it worse than if it was a malicious three letter agency lol.
It's crazy to me that this is possible in the first place. Standard practice is to have a fleet of test vehicles that are effectively production except in an early release group.
Or, you know, having an A/B boot partition scheme with a watchdog. Things that have been around for decades at this point.
Disclaimer: Former Googler, Worked closely with Automotive.
To me it's all-too-understandable how this is possible.
Maybe they've got a test fleet, but it accepts code signed with the test build key.
Maybe they've got a watchdog timer, but it doesn't get configured until later in the boot process.
Maybe they've got A/B boot partitions, but trouble counting their boot attempts - maybe they don't have any writable storage that early in the boot process.
I wouldn't be surprised if, as a newer company, they'd made a 'Minimum Viable Product' secure boot setup & release procedure, and the auto-fallback and fat-finger-protection were waiting to get to the top of the backlog.
So, using Polestar as a reference as it's both a vehicle that I've worked on, and one that I personally drive.
> Maybe they've got a test fleet, but it accepts code signed with the test build key.
Polestar solves this by only delivering signed updates to their vehicles. The vehicle headunit will refuse to flash a partition that isn't signed by the private key held by Polestar. Pulls double duty to prevent someone from flashing a malicious update, as well as corruption detection.
> Maybe they've got a watchdog timer, but it doesn't get configured until later in the boot process.
Based on what the Rivian reports are showing (Speedometer, cameras, safety systems are working), they likely are running their infotainment as a "virtual machine" within their systems. Again, something that Polestar does.
Implementation of a watchdog with a "sub-system" like this is relatively braindead simple.
> Maybe they've got A/B boot partitions, but trouble counting their boot attempts - maybe they don't have any writable storage that early in the boot process.
Generally, A/B partitioning is part of the bootloader, the first program that executes after the reset (on many modern processors) pin is released. This also leads to reboot counters and such being stored as part of the NVRAM that is available at boot.
Opinion: Maybe I'm biased, but maybe if you can't develop something yourself, there's reason for you to get an off the shelf option that handles a lot of these things.
Disclaimer: Former Googler, Worked closely with Automotive.
To be honest, I don't think Polestar set a very high bar for software quality.
I am currently renting a Polestar 2 from Hertz, and sometimes the HUD doesn't work(it's 50/50 if it will turn on).
That means, I don't see speed, battery charge, etc, while driving.
Infotainment system is working though.
You're definitely correct in the fact that Polestar isn't the highest in software quality, but it was the examples of what they did right that I wanted to focus on.
They're garbage when it comes to their mobile app and some of the controls on the infotainment system.
> sometimes the HUD doesn't work(it's 50/50 if it will turn on).
You should definitely reach out to Hertz and ask for a car swap. Sounds like there's a bad connection between the display and the IHU. Both screens are operated by the same system, so it's unlikely to be a software failure.
Teslas occasionally need to reboot / hard reset their software too, when driving no less, and during that period all that information, and most of the controls, are unavailable (like windshield wipers, etc.)
I feel you. I've had my machine blocked for 5 hours as it was pushing an update down my throat. Luckily it gave me a 2 minute warning before the initial reboot so that I could close everything down.
As that was during a client emergency, I had a very happy client that day.
That's not true. You cannot shift gears in the Model X, S, or 3 without the touchscreen - the only way to change into or out of park, drive, or reverse is to swipe on the touchscreen. Only the Model Y has a stalk and it is being removed in the next version.
Also even on older Teslas lights are controlled exclusively via touchscreen.
I don't know what to tell you, you are wrong. I drive a Tesla, I have specifically checked all of what I mentioned when I had this (in my opinion highly problematic) issue happen to me. Have you actually tried it?
As someone who lives in a place where it rains almost every day for 9 months of the year, this reinforces my decision to never buy a Tesla. Does this kill the headlights too?
I had it happen two times this year, but the original comment is wrong, you can use the wipers (and also the headlights) while the touch screen is rebooting.
Answer: To clarify, more that a company should stick to their core competencies unless there's a drastic need or opening in the market that could be filled (and to build a new competency).
In this particular case, there's nothing particularly unique that Rivian is doing with their Infotainment system that couldn't already be handled by an incumbent in the space, (Android Automotive, QNX, etc.) especially given how modular the systems themselves are.
As State Farm says, "We know a thing or two because we've seen a thing or two".
Rivian thinks of itself as a software company. The first thing you sign when you go to buy a vehicle is a software copyright notice IIRC. The first thing in the owner's manual is a notice that the software copyrights and intellectual property belong to Rivian, etc etc.
> The vehicle headunit will refuse to flash a partition that isn't signed by the private key held by Polestar. Pulls double duty to prevent someone from flashing a malicious update, as well as corruption detection.
And of course preventing people from modifying and controlling hardware that they own, having paid 6 figures for (in the case of the Polestar 3 anyway). But that's table stakes for embedded systems in this day and age. Security for me, not for thee.
> Maybe they've got A/B boot partitions, but trouble counting their boot attempts - maybe they don't have any writable storage that early in the boot process.
You do not report a successful boot until and unless the entire system loads up successfully. You will definitely have writable storage by then.
I understand the sentiment, but think about the alternatives.
There are a few different kinds of updates that can be applied, each with their own protective layers.
Infotainment updates, like what happened to Rivian aren't that dangerous. You lose "convienience features" like maps, air con, etc, but generally nothing that could kill you or someone else.
Then there's system updates, which is where danger noodle things happen. Automotive manufacturers are significantly more risk averse to updating these components, and generally, if _anything_ within the system looks wonky, it's an immediate revert.
If I, as a Polestar owner, wanted to get an update for my vehicle, the nearest service center is 1.5h away. If I lived in Montana (United States), it would be realistically impossible for me to update my car. Thus, if we want to enable competition within the markets, we shouldn't have regulations that force a new manufacturer to have a global network just to add CarPlay to a screen.
Dad has a 1966 Oldsmobile with air conditioning. In the last 57 years General Motors has never found a need to update the switch. It still works flawlessly.
It’s stupid that we invented a way to not only remotely break an on/off switch but also a culture that rolls the dice on that until the inevitable happens.
>Infotainment updates, like what happened to Rivian aren't that dangerous. You lose "convienience features" like maps, air con, etc, but generally nothing that could kill you or someone else.
Also speedometer, which is hardly a convenience feature.
On the other hand, we update irreplaceable spacecraft billions of miles away with new software.
It should be fine to push software updates out, as long as the correct safety and fallback procedures are in place. It simply has to be designed to handle failure and procedures need to be in place to mitigate risks.
It sounds like that wasn't the case here. Also, why wouldn't you have a small initial release pool when you have such a large potential for disruption?
The art of shipping software—like on a disk, where once it’s out the door, it’s out the door and you may never get another shot—is dead or dying. Even in some embedded areas of the industry now.
> What amazes me is that any grown up person thinks it is a good idea to update vehicles as if they were telephones
What amazes me is that any grown up person thinks it is a good idea to update telephones as if they were software and not phones.
Or rather that it is a good idea to have phones that need updates? Either way, we're all one 1/2 assed push update to a fridge, vacuum, washing machine, phone or car away from a really annoying day.
As the update only affects infotainment and not critical systems, it seems like a reasonable tradeoff to me. Just because a car can fail in ways that kill people doesn't mean all parts of a car are equally critical.
This isn't true. If you look at the release notes for any of Rivian's updates they all include vehicle related firmware changes. This is not simply infotainment.
Beyond that "infotainment" includes driver critical information - like the speedometer which, for many affected, means there's no working driver screen.
Yeah... I worked on an embedded project with literally 2 engineers, and we had an A/B partitioning scheme, and a recovery partition (we fully qualified the recovery image and it was flashed to the units on day 1, it was guaranteed to boot and it would just sit and wait for the user to initiate a firmware load). The app on the device would reset a U-boot variable once it was successfully loaded, so U-boot could check the number of failed boot attempts. If it was >= 5 reboot attempts without booting successfully, it would go into the recovery partition.
There's really no excuse from Rivian on this, this is shoddy
I interviewed at Rivian. They told me about how they needed to grant users access to things like keys, AC, ignition, etc. So they built a hierarchical, recursive group checking IAM system.
That just felt like a massive product to build and maintain for what really could have been backed by AWS iam. GCP IAM if they really really needed hierarchy. I guess I'm not surprised at this outage.
Rivian does have a test fleet, and they test it for weeks before releasing.
This particular issue is because they apparently distributed the firmware signed with the wrong cert.
Not a bug in the software itself.
That is independent of testing the software, but still a distribution issue.
* "signed with the wrong cert" should mean the software package is rejected before it it is installed.
* software upgrades are tricky and there should be at least 2 versions available so that fallback to the previous is possible and automatic in case of issues.
It is great to live in an ideal world, and in fact, in most software, you can do what you are suggesting quite cheaply. But once you get past the sort of "quip on hacker-news" level of thinking about this, or trivial and cheap production testing scenarios, people have to make real tradeoffs because it's never that simple.
Talking about those is much more interesting than just asserting that everything should be a certain way, without any consideration for real world constraints, like cost of units, etc.
It would be interesting if you could develop about what you mean by real world constraints and how cost of units affect what I wrote, which I did from real world experience.
I'm not defensive, I just find your extreme position remarkably silly.
You included nothing about how costs affect anything. You simply assert that you should always test prod on pristine production units.
There are plenty of times outside of software where production units cost millions or you can only produce them so quickly, or both, and where your extreme take would result in remarkable cost or a competitor eating your lunch.
Which is precisely why its not done, and in the real world tradeoffs are made between what really needs 100% assurance and not. Spending money or losing customers for 5 9s of reliability through testing when two are needed is not a best practice, and is often explicitly called out as such.
In the case of rivian, maintaining a significant fleet of expensive, pristine, exact customer spec (ie not debuggable) cars just to try to get 100% prod ota success assurance is unlikely to provide value vs getting 98% assurance and not doing that (by rough calculation, it stands at 98% after this incident).
My position is neither silly nor extreme. It's the way it is usually done and other comments here have been along the same lines.
In fact you are trying to spin what I wrote to an extreme to make your point.
By the way, it is not about 100% success assurance but assurance that failure does not brick the unit. This is an assurance that should be, and can be, close to 100%, indeed a good number of 9s because, obviously you cannot brick 2 cars out of 100 for every software upgrade!
I've never worked in automotive but it's pretty easy to imagine how this might play out in a car, where a single update might bundle updates for several programmable devices.
It's easy to imagine a central SoC receiving the update, verifying its signature against a local key and then reprogramming some MCU over an internal interface. But then after resetting the MCU, you realize that the image you just flashed isn't compatible with the boot security keys burned into that MCU. It's not uncommon for a device performing the OTA update to not have access to the "source of truth" keys / certificates used to verify the updated image at boot time.
Not that this is a great excuse. If you add OTA updates to a product that has this design, you should really be confident in your recovery solution.
A/B partitions tends to solve that. You will only switch to the new partition when the update is 100% verified installed. If it doesn't complete in an atomic manner, your device will just boot into the previous healthy partition.
A/B gets complicated in the real world. BL1 may not support A/B for example, so to implement A/B bootloaders you may need a shim that can read/write NVM to handle that. Your HSM may not have slots for multiple keys to have different signatures, so upgrading one may trample the other if your update code doesn't check that.
Lots of ways to screw this up, especially in automotive where you're likely to be dealing with TI and their (in)secure boot.
I've solved this problem god only knows how many times now and I've rarely found an automotive board that doesn't introduce fun, new edge cases. OTA can't exceed x kilobytes of memory, the processor isn't fast enough to verify signatures and write the image in < x seconds, can't write the image to flash unless the signature is verified, but the image doesn't fit in RAM, the server delivering the update is 3+ networks away from the device receiving the update, etc.
With all due respect, that all sounds like Programmer Induced Problems(tm).
Cars are a long solved problem, being around for over a century. Telephony and computing hardware and infrastructure today are in the realm of the ludicrously good compared to even just a few decades ago, even if we consider bottom of the barrel worst case scenarios. If software somehow can't work a solved problem using ludicrously good hardware, the programmers (and their managers) are the problem.
I agree here. A partition is just a partition. It's taking one disk and abstracting it into two. This is not hard.
The rovers on Mars, the Voyager, these problems have been solved for a long time. The compute in a Tesla these days can probably run Crysis.
You can do this OTA to a Raspberry Pi running Nerves via remote SSH and it works really well. The Nerves runtime utilizes A/B partitions for OTA updates.
I once worked at a startup that sold a very expensive "enterprise" network appliance that didn't have partitioning. I had worked on network appliances before which did have this capability, so I asked the VP of engineering about it. He said it just wasn't a priority given all the other work that needed to be done. I wouldn't be surprised if Rivian had the same startup mentality that might lead to such a situation.
Let's use two of the examples I gave above. How would you go about modifying the silicon to support new features in the boot rom, and how do you get around the pigeonhole principle when that silicon vendor doesn't ship enough bits of OTP memory to store multiple keys? M-x butterfly doesn't quite get there, but maybe there's another Emacs command I could use?
You choose different silicon? You get a different vendor? You build your own boards and chips?
For a vehicle that costs $100k+ it shouldn't be hard to double or triple the budget for onboard compute considering it is vital to the operation of the entire vehicle.
Note that I never claimed the automotive world has functional people processes. Functional people processes make a lot of technical issues much easier, but they're usually off the table in traditional manufacturing. The security team insists on x requirements for security reasons. Hardware team insists on this chip because it's the only one that makes the budget work.
A shockingly large part of my job is telling both that one team won't be getting what they want and to work it out among themselves. Rinse and repeat between dozens of boards because the relevant teams don't talk to each other and none of them read the "design requirements to ensure we don't have to tell you no" doc either. One time they didn't even tell us there was a board until the end of December, when delivery was scheduled for Feb.
I live in Detroit so what you’re saying does not surprise me in the least bit. I have some friends in automotive at some of the big 3, and suppliers, and I’ve heard some really terrible stories.
It'd be pretty silly to implement an OTA scheme that didn't check signatures before installing updates. That would mean any random attacker could soft-brick the module by sending an invalid image, which a development image should be to a production vehicle.
You could get this situation if the application code accepted signatures the bootloader does not though. I can imagine that accidentally occurring.
Sure, but it could also be signed binaries inside the OTA, a variant of what you are talking about.
IE OTA is a signed package, inside package are also signed binaries.
OTA itself is properly signed, a single binary (infotainment) is signed with wrong key.
While most OTA verifiers will verify the OTA signature (which this would pass), most don't verify the individual inside-package binary signatures at install time, only at runtime.
That's not how I've ever implemented OTA, but I'll grant that it's possible if it was designed by someone with no idea what they were doing. Certainly not a good OTA process though, for this among many other reasons.
It makes sense for each component to check signatures of its code to prevent various kinds of attacks -- e.g. someone coming and reflashing just infotainment or motor controllers with something malicious.
So, OTA update comes in, containing a bundle of software for different subsystems. It's sent to different subsystems. Then, those subsystems check integrity at startup, but one subsystem's bootloader isn't happy because the firmware looks to be invalid.
You can only prevent this if the OTA knows how to do equivalent verification for every subsystem in the car that checks integrity. (And, of course, even if you do this, there's other ways you can go wrong that aren't specific to integrity checks).
OTA is generally developed by tier 1, so this is probably a bug in the tier 1's code. (Samsung, Panasonic, Sony, etc are common tier 1s in this space.)
I am still not sure why I would update software on car, a piece of hardware that, IMHO, shoupd be able to run air gapped 24/7. Exceptions: recurring bugs, GPS maps and security updates. All of which can be done either during service (preferred, if they brick it, they are liable) or by plugging in something. OTA updates just seem completely pointless...
Edit: Also, why the heck isn't the entertainment system completely air gapped from the software running the car?
Rivian consistently ships a lot of new features and improvements, you can see the changelogs here [1]. I think you can pretty fairly critique a lot of them with: They are just solving a problem that they created by making it too techy, or they are shipping stuff they should have completed before releasing the product. I do think its hard to argue that the updates aren't adding anything of value though. There's convenience stuff like pet mode or bird's eye camera view that were added after release, but there are also things like new driving modes (soft sand and snow) or improved DC charging curves and smarter battery conditioning that legitimately improve the quality of the product as a vehicle.
> Edit: Also, why the heck isn't the entertainment system completely air gapped from the software running the car?
As for this, the entertainment system can control basically every feature of the car and is often the primary or only way to accomplish certain things. Even in much much dumber cars the infotainment is still part of the CAN bus and is able to interact with the rest of the vehicle.
Funny, our 2020 MY Jaguar controls car functions from the digital screen in front of the driver, the middle console screens only control AC, entertainment, phone, navigation and other non-car related stuff. No idea how the architecture looks behind all that so. But seriously, even if on the same bus, just don't the media player, radio and connected phone access to the systems actually running the car from engine to brakes. And please, please, finish developing the embedded software running on car before shipping said car. Then it can be air gapped, if not it requires OTA and internet access, raising all kinds of security issues...
And please, please, finish developing the embedded software running on car before shipping said car.
Sorry, but things no longer work that way, and never will again. This is a good thing, as long as processes are improved to avoid situations like this one.
It's supposedly a good thing because more features can be added over time, but if they were features worth adding in the first place, the car should have shipped with them already.
That is the big difference between hardware and software engineering: Once hardware is shipped, there is nothing you can do besides repaurs and retrofits requiring a workshop. Software can updated and changed today anytime.
My biggest issue with modern cars, and it seems this is spreading to other embedded systems than cars, is treating those as software: connect them to the web and run OTA updates everytime you need to fix a bug. That requires some form of inzernet cinnection, which requires regular security updates, which require OTA update capability. All that because, bluntly, software devs cannot be bothered with just finishing software running on non-connected hardware that just works, not can they be bothered following hardware development in case hardware is the main component, as in cars. And no, no car manufacturer is a software company, nor phone maker like Apple.
The proper way of fixing software bigs in automotive used to be, again thank you Tesla for breaking something that just worked, to recall the affected cars to a workshop to conduct the software update. Honestly, the only thing on a car that should be updated by the owner is maps for the GPS unit.
You may not have a say in the matter. Most cars are on some kind of IoT private network with their own cell modems and if the manufacturer decides to push an update there isn't much you can do about it short of ripping out the cell modem. Which may well have unpredictable consequences.
Is not reducing the effective cost of a bad update by 10x or more worthwhile?
Sure, but if you are rolling out to 1% of users per hour, you detect the problem in a couple of hours and much fewer than 2% of users will have applied the update. This is a relatively small support problem.
While if you roll out to everyone at once, you'll detect the problem sooner (within an hour) but have 10x as many affected.
It's easy to underestimate how hard and expensive it is to build, deploy, and remotely upgrade software that runs reliably on a fleet of diverse cars (different models, different years, slightly different components from batch to batch, etc.). It makes updating a mobile phone OS look trivial in comparison.
So far, only Tesla seems to be able to update car software remotely, regularly and reliably. I'm certain it's neither easy nor cheap.
All things considered, physical buttons and dials are probably easier and cheaper, because they don't require software updates!
Forget updates entirely. My car is one of the few places I expect to get software that works the first time.
If you absolutely must have updates, then at least not OTA updates. Have them done at the dealership or service center so any issues can be dealt with immediately.
Come on, is this engineering or hacking? This is a car, not a CRUD app. Get. It. Right.
That’s how things used to be and it resulted in lots of long standing bugs because the update rates were low, and so manufacturers didn’t push updates. Many people don’t live near dealers or service centers or can afford the continued cost (it’s not free usually unless it’s a recall)
OTA is better for consumer when done properly. Other manufacturers manage it fine, and one bad example shouldn’t be what we base things on. It’s what we should learn from and improve on.
I don’t think so. This is the same thought people apply to cartridge based games for example, before software updates.
But the reality is that the same types of bugs would likely exist. Nothing about the engineering aspects has really changed other than more features. All that would happen is future revisions would have it fixed and early buyers would be stuck.
The types of bugs were similar but the bug count was entirely different. Typically on a 16K ROM you'd have two or three known bugs after the product was in the market for a while and maybe years later you'd find another one by re-reading the code and realizing that if you tweak things just so you can get the product to misbehave.
But you're not looking at near endless lists of known issues and 'wontfix' wasn't a thing back then.
eh i guess i disagree. We had that (& still do for some cars) for decades, and it universally resulted in terrible software that you were stuck with for the life of the car. Hard to update == hard to iterate == bad software.
> All things considered, physical buttons and dials are probably easier and cheaper, because they don't require software updates!
I am pretty sure there is a market for a dumb modern car, but no one is building it. I am thinking of an electric car without anything "smart" in it. Modern safety features can stay, if they work completely self contained and without requiring an external connection ever over the lifespan of the car.
This isn't a bunch of Windows PCs home-built from a hodgepodge of components.
They designed, built, and shipped all the hardware. There is ABSOLUTELY NO excuse for not having a database of the exact hardware configs by serial number. They have the ability to test every single shipped configuration.
If they don't, they have already failed as a car company.
I guarantee they have a database with the hardware configs. It's required by NHTSA to do recalls and notices. They'll undoubtedly be using that to inform the right people to come in.
The update servers almost certainly don't talk to that system though.
Well then we need to ask why is their infotainment systems so complex? and does it need to be?
I want my infotainment systems go connect to Android Auto. That is is.
Make it do that, and only that.
This drive to make EV's as complex as possible is one of the reason i am not planning on buying one
EV's are suppose to be SIMPLER than ICE. Make me a Simple Car with simple controls, and just replace the ICE with a battery and Electric Motor, give me an app for my Phone that can do the Charging Trip Calculators and interface with other systems.
That’s a huge plus for me. CarPlay or nothing. BMW is becoming closer to a Tesla like screen. GM is supposed to drop CarPlay in favor of whatever they are doing on their EVs.
I don’t want my ICE/EV to become a SaaS app where I’m paying $500 a year to use my own car.
BMW in particular is an interesting case, a (late) friend of mine drove just about every model the day after it came out (BMW fan) and they spent more time in the shop for software issues than they did driving. To the point that he'd get attached to some of the loaners, it really was that bad.
Updating software is orthogonal to the complexity of the software application being updated, unless you have horribly designed your architecture. I know, because I've made that mistake.
Well sure, but the rotary encoder can’t get moved to a different menu tree by a software update, and I can use it without taking my eyes off the road. I know which I prefer.
Tech junk shouldn’t go in cars, period. Cars shouldn’t be as pervasive and prevalent in society (at least in USA). Yet here we are. Car manufacturers have spent an insane amount of money over decades to get to this point (buying legislators, forcing highway infra, subsidies, profit driven strategy over sustainability)
Decades? Try almost a century. For better or worse, our cities and various economies were built around the automobile.
It's still a free market - these companies could choose not to put tech into their product. But look at the backlash against GM when they announced they wouldn't support Apple Car Play or Android Auto. Consumers want it.
I had to do this once or twice (its very very infrequent in my experience) and one time it was genuinely terrifying, as I had lost blinkers etc where a few interstates all intersect and merge etc.
I still do love the car though.... but a very sketchy moment that I shouldn't have brought on myself while driving in that situation.
Tesla doesn't do that, you're misunderstanding. You can do a hart reboot on the computer by yourself, by holding down both steering will scroll wheels for 10 seconds. Fixes any glitches with the screen.
A lot of silliness has been tossed around about Teslas in this thread so let's be clear about how they work. A Tesla vehicle have two main independent systems:
- The "car" that operates the motors, blinkers, shifting, Autopilot, etc. It also supplies standard readouts via OBD that you can monitor independently e.g. with a phone app. As far as I'm aware, it is impossible to turn this machine off except when the car is stationary, in park, and the brake is fully pressed (or by disconnecting the 400V battery leads under the rear seat).
- The infotainment computer is a small Linux machine that displays the map, speedometer, music, plays the sound of the blinkers through the speakers, and random other stuff. This system can be reset at any time by holding the two scroll wheels.
There are valid things to criticize about this setup e.g. by not having a standard fallback display of speedo, but you can still read speed from OBD at all times. And Tesla has broken things in OTA updates before, but their rollouts are heavily staged. The last major revert I can remember was FSD beta 10.3 in 2021 back when it had reached a few 10s of testers.
it didn't initiate it, I did. I should've waited like..... 30 seconds or until I was stopped quite honestly. I'm fully aware of how the vehicle operates and I would consider myself solely responsible for initiating the reboot.
That said, it was weird that the blinkers didn't work, but it may be that the sound in the cabin wasn't playing and the blinkers were just fine... and in hindsight thats probably accurate.
Point being, I picked literally the worst place to do that reboot. I would consider the need for the reboot in that situation to be on tesla maybe, but the choice to perform it when I did was solely on me.
I dislike elon as much as the next person (i like to think more, but some people definitely me beat).... but the car is solid... i just never thought i'd say "i need to reboot my car's computer"
> (different models, different years, slightly different components from batch to batch, etc.). It makes updating a mobile phone OS look trivial in comparison.
Not really. Vehicle computers aren’t vastly different on every model year and every trim level or option package. These parts are standardized, tested, and carried across model years.
Even with changes, the teams would be expected to have the different variants in their development and test cycles. The 2020, 2021, and 2022 model infotainment systems likely share a lot more in common than an iPhone 13, iPhone 14, and iPhone 15 with all of the non-Pro, Pro, and Max variants.
They invited the multiple combination vampire into their house. They know what devices are being used. If you don't want a dedicated update per piece of equipment, it'll be a large binary with lots of branching. Saying they don't know what device is where is just lazy. Ask the device what it is, and have a branch for it. If the device IDs itself as something unknown, don't do anything.
And it's not like they'd ever abuse that ability, like when someone pokes around in their car and discovers references to a new unannounced model, and then Tesla reaches in, force downgrades the vehicle to older software with no references, and then disables the ethernet port on the vehicle, and for a final fuck you disables its ability to ever get another update.
They'd never do that, except when they did do that.
It sounds like, in this case, the updates clobbered the ssh authorized keys (or equivalent in their system) and so now they cannot access the cars remotely. So they are going to have to go into the shop and have the authorized keys restored.
Bringing CI/CD mindset to cars is probably not a great idea. Software updates to commuter vehicles should have a high bar for operational standards, and a simple thing such as an expired certificate should have never been deployed. Having isolated networks in vehicles helps but doesn't prevent broken updates from, eventually, bricking the cars.
I think this shows more of a fundamental flaw in their update mechanism, than anything.
I don't think a botched update is a big deal. It happens, and should be expected, in a sane design. The fact that the customer noticed is a big deal.
There are many implementations that could be used for an "auto rollback" feature. They either failed to implement that in a sane way, or they were goobers, and assumed things would always be rosy.
I would be pretty pissed if I went out to my garage to head to work one morning and found that a damn software update bricked my car overnight. This shouldn't even be a thing, why does a car need regular software updates to keep functioning?
It doesn't need regular updates to keep functioning. It offers regular updates as they add new features. For instance, in this update a new feature was added to allow for proximity locking at home but disable proximity unlocking. That would lessen the number of times the car would lock and unlock accidentally as you walk in and out of the garage. No one was forced to install the update.
Cars 20 years ago, even most of them 10 years ago, never got any updates unless they got recalled. Nothing broke, nothing got hacked, and most are probably still working fine.
What happened to cars today? I refuse to believe that it's solely because these are electric cars, as if the way the car stores and uses energy dictates that it must be part of the internet of things.
Edit: there were electric cars over 100 years ago. I bet they never got software updates.
Cars 20 years ago didn't have realtime traffic on big touchscreens that you can use to look up your destination and plan out a route that also lets you schedule fueling/charging stops, oh and also stream humanity's entire library of recorded music, books, and podcasts. It's a tradeoff that the vast majority of people want.
But almost every smartphone already has those features, they work fairly well, and it makes more sense for a phone to have those features than a car. (Because sometimes it's useful to know where traffic is when you're not in a car, and stream things while walking around.) It doesn't make much sense for each car manufacturer to replicate everything if they can figure out how to connect a phone to a car, and borrow the phone's features.
As software eats the world, it becomes more and more apparent to the non-developers of the world that software engineering is not, and never has been, a real engineering discipline.
Tech Support: "Oh your garage door is bricked after last nights update? Yeah, apparently the [totally uncredentialed] contractor that wrote that update is only 3 weeks out of coding bootcamp and was just copying and pasting from ChatGPT. Lmao"
At the same time, we're losing tons of mechanical problems that use to go wrong with cars. The amount of time lost to car malfunctions is way down over the past couple decades, even with slight regressions like this one.
If ICE tech was the hot new thing in cars, things like spark plugs would have a chip so that it would fire n times then break, but don't worry, there's a subscription for new ones and they will be automatically ordered when the car says so. If the credit card on file expires, your spark plugs won't work anymore, even if you just replaced them.
I believe one of the reasons it is slow is because it is also updating the firmware on any number of connected ECUs over the CAN bus. This typically means the image has to be sent over a 500kbit/s bus so there is a limit to how long it has to take.
I would naively expect it to just do A/B updates, which unless I'm forgetting something shouldn't incur a speed penalty? (Other than that the update doesn't get applied until restart)
Interesting to note that Ford's approach of updating software is far more conservative and car-like. It can be done fully offline via USB, but requests that you kindly upload the log files written to the memory stick back to them when complete, in the instructions as a necessary step. Presumably so they can track and stop incidents like this before they happen fleet-wide.
Rivian seems more like a "ship it and we'll fix it in the next sprint!" company.
The last time I built something like that, it used partition 1 for the current version, 1 for the last version, 1 with the as-shipped version, and 1 that could restore A or B from the internet or USB.
When will humans be crazy enough to update the firmware of artificial hearts OTA?
Updating cars with new features OTA, even "just" an Infotainment can possibly cost lives, because the driver might get confused and isn't putting eyes on the streets.
It should be forbidden and every change should be made clear to the driver, shown in detail, and should need verification twice before being accepted. There must not be any kind of surprise in a car for the driver.
It should even be possible to skip an update or stop updating at all.
Not updating cars OTA (yes, even "just" the infotainment) can potentially cost lives as well, as security holes would not get patched until the next service appointment.
> where a "bug" has potentially life threatening consequences.
What are you referring to? That is not relevant to this story, and would require a deep understanding of the system to make such a claim of negligence.
“The issue impacts the infotainment system. In most cases, the rest of the vehicle systems are still operational ...”
Based on the photo included in the article, what they're calling an infotainment system is actually two separate components, one of which appears to be taking the place of a traditional dashboard. If that's the case and there's no other way to monitor speed, fuel levels, engine temperature, warning lights, etc., I'd say that's quite a bit more worrisome than just not being able to play your favorite music while driving.
I understand, but the risk of life wouldn't be from the bug, it would be from conscious choice of driving without a speedo. There's a critical distinction there.
In this case, mileage/battery are still present, and I would assume safety critical warnings would still be displayed.
That any car can have a dead
/failing battery/fuel pump/whatever one morning or fail and leave you stranded on the road to an hospital has never been considered a safety risk.
You've never been to death valley without air conditioning Or Russia without heat. I think the infotainment system in this case has a broken climate control function. There are workarounds, but why if you don't have your phone?
This is a great point. I would claim that it would be a bad choice to initiate the update (it's a manual process, requiring intent) when you're in these conditions. But, a less tech savvy person may not understand that updates can be risky, and give it a shot at a remote charging station.
I have been talking recently to someone whose job involves sometimes driving to other continents, and they mentioned that cars more recent than ~2003 were out of the question because outside of the EU you cannot expect random mechanics to have the computers required to interface with the car's computers - required for repairs.
If that's the only problem, then that would be fixed simply by taking "computers required to interface with the car's computers" i.e. a <$200 OBD-II device with you in the car, not choosing a car with 20+ years of wear and tear.
Also, since many of these places have a lot of used cars that once were in the EU or other places with legally required computers, this is changing as now most of the mechanics even in relatively poor remote places have to have these devices as now many of their local customers don't have "computerless" cars any more, and the existing pre-2003 cars won't last forever even there.
Not the specifics of this article, but more generally about the gravity of the situation car makers (and their software engineers) operate under. The very idea that an OTA software update that causes a bug within more critical features of a car could be life threatening. So my point isn't about the specifics of this particular bug, rather the capacity for a bug that could kill.
Rivian "licensee" here. So far all updates have required you to press a button (in the car or on the app) to launch the update installer. Not sure how many weeks you can ignore it for as I never tried.
Stuff like this is why I don't want OTA updates in my cars. Let the car dealership deal with it during regular maintenance. They'll be on the hook for fixing it before handing the car back to me.
I'm pretty sure they are all dumb enough for this. This is why people can steal (modern) cars by ripping a headlight out and accessing the CAN bus through the gap.
AIUI the military uses "CAN bus firewalls" to prevent this. There is also some sort of encrypted/authenticated CAN bus protocol in the works. Neither are common in production cars as of 2022 or so, though they may be soon enough.
Honestly we need a law that says that _all_ keys used in your car (software and physical) need to be given to you. This should include the keys to sign the updates, manage the CAN networks, etc.
Even ignoring the automaker lobby's inevitable opposition, this would never happen because one could use it to defeat things like emissions controls or the coming anti-impairment technology[1].
Regardless of OTA vs dealership-only updates, software bugs can have problematic effects long after the update occurred.
So far I've had to take my Chevy Bolt to the dealership twice due to major software problems causing the "service needed" indicator to be lit (equivalent to "check engine"), and I've owned it for barely over a year.
The first time, some random bug made the car think there was something wrong with the transmission under some extremely specific set of circumstances, and as a safety precaution it would refuse to shift into drive if not serviced within 100 key cycles.
The second time, it was a bug with the software that manages battery health making the car think the battery had a severe problem. In that situation, as a safety precaution, the car refuses to charge above 40%, disables regenerative breaking, limits the HVAC usage, and slightly limits max acceleration.
This is getting very irritating. I bought an EV because I thought it would require fewer maintenance visits to the dealer!
Also worth considering that a manufacturer like Rivian is pretty small. Every town has a Ford dealer. There are many states, however, that don't even have a single Rivian service center.
Coverage is even thinner than I would have guessed. California has 6 Rivian service centers, but they're strongly clustered in the Bay area and LA/OC/SAN. Even in California if you live in Fresno/Bakersfield/Santa Barbara etc you're looking at several hours round trip to visit an official service center.
There is nearly zero regular maintenance to be done on EVs though. No oil, no belts, no fuel filter, spark plugs etc. Even the brakes will likely last entire lifetime of the car
Bullshit. BEVs eat through tires because they're heavy. The air cabin filter and pollen filter need frequent replacement. In the case of Tesla, you better check the undercarriage so you can potentially spot control arms soon to fail. The shield for the battery should be inspected.
BEVs don't eat through tires because they're heavy. They eat through tires because they have fast acceleration and their drivers make use of said acceleration. A Tesla Model 3 weighs around what a BMW M4 weighs and less than all trims of an M5.
Hey if you want to pay a dealer 300-500 euro to inspect your tires and swap air filter I’m not against that - you do you. Also if you buy a german car (ev or not) then yes a mandatory inspection is warranted
I'm not sure maintenance costs are really the relevant part. It seems like the problem is that Teslas are cheaper now and thus Hertz's fleet is worth less than it was before. Additionally, they find that Teslas suffer more damage, likely from collisions or similar. Routine maintenance costs are not mentioned in that article at all.
>In short, the declining value of the Tesla cars in Hertz’s fleet—a decline directly caused by Musk’s price cuts—has hit Hertz squarely in its profits.
>Without explaining precisely why, Scherr said Hertz is suffering a higher incidence of damage specifically with its EV fleet, where the repair costs are roughly double that of a comparable gas-fueled car.
>“Studies of current EV ownership evidence lower incidence of damage and collision than for ICE vehicles, not higher as we are experiencing,” he revealed.
Musk’s price cuts then become an acute problem when one of the Hertz EVs sustains so much damage that the cost of repair is more than the asset itself.
>“Where a car is salvaged, we must crystallize at once any difference between our carrying value and the market value of that car,” Scherr explained. “The [price] declines in EVs over the course of 2023, driven primarily by Tesla, have driven the fair market value of our EVs lower as compared to last year, such that a salvage creates a larger loss and, therefore, greater burden.”
>In short, Hertz then needs to book a noncash accounting charge. Together with the higher repair costs this led to significant profit margin headwinds.
Well if we're talking about Musk-whisperers, they clearly preferred to listen to him when he said that Teslas would be appreciating assets, rather than that.
> A car doesn't need data updates, and definitely not code updates
I don't think this is accurate. Many advanced driving assistance capabilities need access to updated map tiles, which is a data update. They may need code updates to fix errors or shortcomings that can be detected only after deployment on extensive fleets or in response to changes to the environment/infrastructure. This is just one example for why data and code updates are needed.
I think it is more accurate to say that a "dumb" car with mostly electro-mechanical systems doesn't need data updates and definitely not code updates. But that isn't true for vehicles built within the last few years and definitely untrue for vehicles that will be built in the coming years.
> Many advanced driving assistance capabilities need access to updated map tiles
Your phone (or GPS or even a paper map) can guide you; none of the following need access to map tiles:
* forward collision warning
* automatic emergency braking
* lane departure warning
* adaptive cruise control
* blind spot detection
* stability control
> code updates to fix errors or shortcomings
That's what recalls and TSBs have traditionally been for, and the driver can refuse them if desired. I mean, actual lives are at stake here. Would we (or should we) allow 737's to get OTA updates? Of course not. The target is too valuable and surface area too vast to adequately protect it.
The usual "best practice" thing for IoT deploys, is to deploy to "your" devices, what for everyone to go green, then allow that build to deploy more widely. In a well-functioning system, it shouldn't be possible to swap bits between those stages.
Reading between the lines of their public comments it sounds like they did run a full test through their test fleet, their employees, and then were rolling out to customers when the promotion process was "fat fingered". Maybe someone accidentally promoted the wrong release version.
Yes. And also things like rolling out the update in batches, and then also things like golden images, where if there are two crashes or failures in the first 24 hours of the update, change to the last known good software version.
I don’t really like or trust most (if not all) of the established automakers, but there is something to be said for having several decades (over a century in some cases) of experience building potential killing machines vs. a company that’s not even 15 years old. The established players have put out cars which suffered freak malfunctions, but Rivian (and Tesla) seem to be struggling more with QA.
Non-rhetorical question: do companies have safeguards for critical components like braking systems, or are they also prone to catastrophic failure if a software engineer pushes a bad commit?
These all sounded like crap to me as well. I've heard earlier models had much higher radio power (that was possibly illegal) and worked better. They were an archaic tech by the time I was buying the adapters off amazon.
I know I know but then there's a cable from where the phone is to the middle console where the aux port is. But true, it'd for sure be better sound quality.
I have preorders in for the R1S, the Volvo EX90, and the Kia EV9. I passed once already on buying the R1S when they had one in town available for immediate purchase, simply because they refuse to adopt CarPlay.
This incident does NOT give me confidence that Rivian is likely to offer a better alternative to CarPlay, despite their statements otherwise.
I suspect the EX90 will be what I land on eventually.
>This incident does NOT give me confidence that Rivian is likely to offer a better alternative to CarPlat,
I have complete faith that, 5 and maybe even 10 years from now, no auto maker will have delivered anything that can compete with either CarPlay or Android Auto. The fact that an auto maker thinks they can do better is a sign of a really high level of either arrogance or outright greed. Complete deal breaker.
This is actually a topic that I think about from time to time: how to do aggressive changes to software while they are running. In Ruby world you have monkeypatching. And Linux kernel has livepatching.
For example, if you have a distributed system and you want to upgrade a component that every caller uses: you have a large exercise on your hands where you might have to roll out a change over time and then clean up your incremental branches where you have to handle two control flow paths through the code. It reminds me of Google's protobuf required field discussions.
It reminds me of repository-per-microservice and a Java library that other microservices use and updating a dependency and having to deploy the change to every service.
It's like trying to change wheels on a car while the car is moving or refueling a jet in flight.
Unison lang is trying to solve this problem I think, by allowing multiple versions of a function to be available.
One solution I've thought of which is probably overengineered is that API call sites are an abstract object and their schema and arguments is centrally deployed, I called this "protocol manager".
The idea is you write all your code to use a "span" and have contextual data in a span, and you can include or exclude data in a span with a non-software rollout. Your communication schema of RPC and API calls is a runtime decided thing, not hardcoded.
If you have N deployed versions of code and you want to upgrade to X, you have to test 1..N to X versions. So nobody does that.
The database aspect of this problem is particularly interesting to me. I’ve previously built Reshape [0], a zero-downtime migration tool for Postgres, and am now working on ReshapeDB [1], a full database designed from the ground up to tackle this problem.
I wonder if the way Microsoft's XBox is designed may be something to look towards in terms of hardware reliability/fallback. Specifically they utilize a Hypervisor which rarely needs updates, running different operating environments which need frequent updates.
- Better isolation of different parts of the system (e.g. infotainment unit, instrument cluster, et al).
- Better isolation for updates (e.g. run a "beta" update, and a "stable" update side-by-side).
- Automatic error detection and rollback (e.g. if a VM keeps restarting after an update).
- Ease of offering features like rollbacks to end-users.
- Rare hypervisor updates can be held to a much higher standard relative to other VM updates.
The only downside of hypervisor-based systems is slightly higher hardware costs. But even that is largely mitigated by modern architectures that natively support virtualization.
PS - You can also look to any containerization. I specifically brought up the XBox because it is a hardware product, just like a vehicle.
My cars are from 2019 and 2001. I don't use CarPlay or any internet features in any of them. Instead, I just use my Android's screen itself for navigation and bluetooth for phone calls and music.
Perhaps there are advantages to tighter integration with my car (at least the newer one) but IMO they are outweighed by the risks of things like this, or even just getting a software update that borks a small feature that I like.
CarPlay and Android Auto has no risk like this. It is great because it is just a protocol for your phone to be able to use the screen in the car. Your data stays on your phone, and there is no risk of intrusion.
Quite a few reasons but primarily CarPlay doesn't know about your vehicle's charge and road trip navigation requires external inputs to determine your charge route.
Apple could potentially offer an API to have "reverse" CarPlay where the car's app can feed information into iOS. I recently rented an Mercedes EV which had Apple CarPlay and it was a weird experience having to manage two sets of experiences.
Manufacturers should be required to expose an agreed upon spec API that provides range estimate, battery state, etc. So CarPlay and other apps can access the info in a standardized way
Apple holds up their end of the deal by supporting the EV Routing feature in maps. It’s entirely up to manufacturers to expose the information at this point.
100%. I've owned Teslas since 2017 and always found the infotainment to be very good. Though I've really preferred Apple Maps routing recently. The descriptions of when to turn are much more human.
More potential for anticompetitive vendor lock-in (sugar-coated as differentiation), and more opportunities to profiteer from stalking behavior (sugar-coated as telemetry to improve user experience).
..And authoritarian governments can't take complete custody of your movements. Can't shutdown your car either. Seems to be a win-win for both sides (private and public industry).
I wish the economics of mass production didn't turn pennies into millions that need to be eliminated, because I've always thought the "don't disconnect from power" and "update bricks it" type problems could be solved by having extra EPROM to download into, the way linux keeps the previous kernel around after an update.
Or at least the ability to re-init/download from scratch, like a borked macbook disk. And hey, not the extra ability to do that, make it "the way it works" so you're always testing it.
This maybe crazy but if you're writing software for hardware that cost tens of thousands of dollars it should be impossible to brick it with an update, especially if that update is OTA.
This is the new world we will be living in where you enter your car, only to find that something is broken because of OTA. While updates causing some bugs is ok in my phone but I don't want any bugs in my car. What happens if it messes up with safety systems? or what happens if OTA breaks my car that is out of warranty now?
May be I'm the only one that is missing stable software in cars that once vetted, just keeps running as-is if nothing around it is ever changed (ideal scenario for an offline car).
An interesting thought experiment: what happens when these vehicles are out of warranty, and automakers accidentally send a vehicle-bricking OTA update? Isn’t that property damage?
What kinds of changes are generally included in these over the air updates? I have this sudden urge to shake my fist at a cloud and tell the gods that cars shouldn't need updates in the first place, if the car was ever deemed ready for production and then sold to customers for money. But, maybe I'm wrong, and it makes perfect sense. All I can think of would be something like a periodic update to navigation data, is that it?
It’s possible to deem software ready to sell but find improvements later.
Simple example: my Subaru was sold to me with an interesting design decision that caused the radio to come on whenever the car was started. This was not a bug. Every Subaru worked this way for years. A year into ownership I received an OTA update that added a “not playing” state on startup.
This was never a safety issue and was likely not a defect. It was, however, stupid and needed to be changed.
I wish my Mazda had this option! But I would still say that I'd expect them to have included this option before selling the car, especially since radios and user preferences around radio UI are pretty well established.
Sure, but they didn't - and it's not because the software is complicated. The Subaru headunits are very basic (and more-so a few years ago).
When they make an improvement, I like getting the trivial things on my older vehicle. It's better for everyone involved, so long as they do it responsible.
It’s funny I was just talking to someone about a-b images slots and boots the other day and how they had written this test suite because there were so many potential places where partial updates could be interrupted.
Thousands of test points having to be verified was my understanding. That’s before even getting to the confirmed boot/watchdog aspect.
What a hassle, hope they like spending money on labor because it sounds like they are going to need to.
The vehicles are drivable but software and displays go black. It appears that the 2023.42 software update hangs at 90% on the vehicle screen or 50% on the app screen and then the vehicle screens black out. All systems appear to still work except for the displays.
This is what I do with my Prius to get a comfortably distraction-free driving environment. Sounds like a feature not a bug.
Technically, the NTSB could order an immediate recall for all Rivian vehicles due to this issue, as the disabled defroster controls are a critical safety issue in cold and/or humid environments. Tesla was forced to issue a recall notice over the controls being buried in a menu; Rivian’s ”defroster unavailable during driving due to manufacturer error” is far worse — especially given the mass and torque of their vehicles, relative to unarmored road users.
Can’t imagine how much it would suck to be the engineer who fat fingered it and caused a huge crisis for the company, inconveniencing tons of customers and costing millions. Even if there should be processes in place to prevent it in the first place, you’d still know you were the “but for cause” of the problem.
This is the kind of thing that keeps my awake at night.
Does anyone here have some practical tips to turn an embedded Linux machine into an appliance? The kind of system that a botched update cannot brick but only momentarily disable until a non-technical user presses a factory reset button of some sort.
>Does anyone here have some practical tips to turn an embedded Linux machine into an appliance?
Lol
I suppose this is the negative about having sensors that make sure water gets hot enough to be sanitizing, but not so hot that it wastes energy. And I'm sure you can imagine 100 other uses of having a microcontroller/CPU process data and do feedback. (I'm sure there are EE only ways of doing it, but theoretically possible and useful are two different thigs)
/r/Rivian is a class act. I expected a wall of screaming, but instead entered a relatively calm room. People are upset, but there's no seething or flamewars, which is kind-of surprising given the cost of these trucks ($80k+, Range Rover territory).
I think the reason is because they're $80k trucks, not $400/month Tesla leases. Also, they're first generation and I think most of the buyers understand that.
Will insurance carriers cover damages due to botched updates? Imagine 10 years from now the power/control that electric delivery companies would have over retailers like amazon. One botched update away from a complete backup for delivery vans.
Tesla updates are sent in batches and you can opt in for advanced updates I guess to be earlier. Normally when I see that there is an update on Reddit then it takes 1-2 weeks at least to get to my car with the “advanced” updates on.
As somebody who has spent many years doing embedded+iot related to remote fleet firmware updates, this is the kind of thing that lurks in my nightmares.
I'd love to be a fly on the wall at Rivian engineering/operations this week!
need a easy way to do restore to previous version offline. take 100 bucks extra if required to have a backup ssd. Don’t want to be camping and then realizing i’m stuck because of some junior dev not being competent enough
Why would you intentionally upgrade your vehicle software while camping? It’s not like this stuff installs automatically, you have to explicitly accept the installation. Waiting a few days or even a few weeks before hitting “install” is completely normal.
Can I please just buy a car with a motor and battery? Why does every god damn vehicle have to come littered with screens and chips all together like some tentacle monster?
All I need is a gauge cluster screen that can display the normal info like stored and heading while also letting me configure the cars performance and safety features. Then let me mount a double DIN radio that isn't dog shit. I've not seen a single new car with these dumb screens with a sound system that's not tinny muddy garbage with zero adjustment save for "bass" and "treble" settings. I mean all that technology and you can't be assed to put an eq in there. HVAC never needed more than two or three knobs anyway.
I'm going to have a chuckle next time I pass the Databricks billboard on 101 in San Francisco "Rivian powered by Databricks" or something to that tune.
The article doesn't really state what is required to repair the vehicle. I'd assume if it was as simple as loading a flash drive and plugging it in, then Rivian would have provided a way for customers to self-fix. The second a single body panel is removed to gain access to the headunit, it is a physical repair.
So without more info we cannot know if it is accurate or not.
As annoying as this, I find this laughable, too. Rivian updated users on the situation. Then, whines Electrek:
> That’s the last update we had over 10 hours after Rivian customer vehicles were fed the bad software update.
"Over 10 hours"!
I suppose it isn't Tesla, who yeets updates over the fence, that break new things, yeets another update that fixes that problem but introduces another one, then reverts back to two versions prior, before the issue. The Tesla that gets firmware fixes from vendors that have a test harness that should take 36+ hours to run, but says YOLO and flashes it onto a random car they have lying around and emails the vender back 3 hours later saying "LGTM, WFM, thanks!"
Honestly this makes me feel good, just because it always worries me that I don't see this type of issue being resolved more often. having to physically bring in a car seems like a near worse cast situation but it's good to keep this in our minds as a possibility.
This is tangentially Rivian related, but does anyone else see the inherent danger of stylized tail lights that are just a single red bar across the back of the car?
Travelling on the freeway at night I can't really gauge the distance to the car in front of me if it's far ahead and if there's no discernable left and right brake lights. I'd believe Rivians and other cars like that are more at risk of high speed rear-end collisions.
Tesla. Rivian. All cut from the same cloth. A car should be simple. Yet we are stuffing all of this tech junk into it and trying to repackage is as something else to pump the numbers.
Car companies suck at tech. Let’s be realistic. They should stay their lane and focus on improving the car and physical aspects (safety, reducing carbon output, longevity, ease of repairability, reducing supply chain issues)
I'm not aware of any Tesla OTA updates bricking the infotainment system. At least since I've been paying attention. I don't see them quite as similar as you suggest.
I'm not aware of any fleet-wide issues that accidentally bricked Teslas, but as one-offs they do happen; and unlike this Rivian update, a botched Tesla OTA generally leaves the car undriveable and needing to be towed. These Rivians will at least still drive, as long as you don't need fancy extraneous luxury features like a...speedometer.
Rest assured that after the first time I messed it up (which required ssh into each box individually), I wrote a lot of unit and integration tests to make sure that it never failed to deploy again. One of the integration tests ensured that the app started up and could always go through the internal auto update process. This ran in CI and would fail the build if it didn't pass.
While I fully understand that this is hard to get right 100% of the time, a mess up of this level by a car manufacturer is pretty amazing to me.