There are a ton of gremlins in Tesla just waiting to go off like time bombs. I recently got a very expensive repair just outside of warranty where the car simply refused to charge. Paid nearly 3000 for the diagnosis and repair for the car to start charging again - we are talking about a car with no wear and tear parts for the most part costing nearly 3K because a chip somewhere between the charge port and battery failed. My whole reason for buying it was that this would be the ultimate reliable car without ICE or brake wear, leaving tires as the only consumable. Guess I was wrong.
Even in combustion engine cars if ECM buckles up it usually is expensive to repair(at least in my part of the world). I guess with a car like Tesla, rich with electronics & unavailability of parts to 3rd party service centers; repairs can be very expensive.
I think you overestimate the quality of "made for automotive" and underestimate the importance that SW has in cars. These were essentially SW issues. This kind of SW fuckup can will and does happen for bigger OEM's all the time.
Tesla definitively has a problem with hardware design decisions. Remember the Model S touchscreen issue? Somehow their designers forgot that cars heat up in summer and instead of automotive used consumer grade hardware!
Meanwhile companies like Garmin and Magellan manage to build GPS devices that work just fine even though they spend their whole lives in the passenger compartment. They did this since the beginning, I never heard that you should stay away from this or that manufacturer of GPS units because the touchscreen fails.
They're options where: Ship a 17" LCD display (a key feature not the Model S) or don't ship one at all, because there were no 17" capacitive touch display certified for automotive use. Sometimes you just have to ship, especially if your touchscreen is the main interaction component of your product.
The first iPhone didn't have 3G while other phones had it. In order to do something new, you need to cut some corners. It's not like Tesla shipped non-automotive breaks.
My first touchscreen Garmin unit, circa 2007, definitely suffered from reliability issues in FL summer heat. Just keeping it on the dash in direct sunlight while driving was enough to affect its operation.
I’m guessing it didn’t effect their brand much because other companies like Tom Tom were aggressively targeting the entry-level market segment, and then the iPhone came out the same year.
The number people who had these higher-end units in hot southern states was likely just too small to significantly tarnish Garmin’s brand goodwill.
Soon enough most people switched to using their phone once the software was good enough (circa 2009 IIRC).
My understanding is that they use non-automotive grade displays because there simply were none large enough. I can somewhat forgive them for using something almost as good to achieve the design they need. The Model 3 would have to be completely different if it didn't have the large screen.
Is any car company making a simple electric car without fancy features, or at least one where they're optional? Even the Nissan Leaf now has all sorts of high-end features. I'd much rather have a simple, cheaper, reliable car with less points of failure.
Tesla fans will tell you otherwise, but pretty much all of the EVs from "traditional" automakers are more reliable than Tesla vehicles. Cheaper & easier to service too.
I've heard very good things about the Nissan LEAF, Hyundai Ioniq, Hyundai Kona, and Kia Niro. It might be a coincidence but all of these cars are packaged more like traditional cars, i.e. popping the hood allows you to access many of the components.
More reliable? Didn’t leafs of the same time period of the Tesla models being discussed here have horrible battery degradation issues? Hyundai and Kia electrics are too new to the market to know if there will be serious issues
Early Leafs would apparently get their batteries cooked in Phoenix because no active cooling. We’ve got one of the first ones that rolled off the line, and eight years of Seattle weather has ours at ~12% degradation. We otherwise haven’t done shit to that car except drive it. So from my POV, I read the issues with Tesla’s and just shake my head.
All those old Leafs with "horrible battery degradation" from the USA are now taxis here in Ukraine. A car model needs to be VERY reliable and maintainable to be popular as a taxi here, so it's a high praise for the Leaf. The batteries don't degrade straight to zero, they are still very usable. Battery degradation only helped to lower the price of an otherwise very nice car.
Unfortunately if you're looking for a mid- to high-end EV that isn't an SUV or crossover, your choices are still pretty limited, for now. There's the Taycan now, but it's not really in the same price point as the Model S, let alone the Model 3. Maybe the Audi e-tron sedan when it comes out will be a bit more reasonable (relatively speaking).
My Chevy Bolt is like that. It's the most basic (and boring) unassuming car in terms of looks and options. It blends in well with everything else on the road. At the same time it's more fun to drive compared to any ICE vehicle I've ever owned.
The Volkswagen e-Up and its slightly cheaper but otherwise identical counterparts from Škoda and Seat might fit the bill. They now come with a respectably sized battery (32kwh) and lower prices.
Also PSA (Peugeot, Opel (fromerly owned by GM in Germany), Citroen, DS) is coming out with all-electric versions of the Peugeot 208 / Opel Corsa. Slightly smaller than a VW Golf.
On the true luxury end you also have the Jaguar i-Pace. If you have the 80k EUR to spend that is. There are a lot of traditional car makers coming up with electric cars now. With the added benefit that in some cases they can be built at the same factory like their ICE brethren. That would allow them to drive costs down quickly if the demand is there and economies of scale kick in.
ECUs made in the last 10-20 years don’t really ever fail. Bosch has really figured things out and produces reliable ECUs for most OEMs.
The repair on your Tesla will cost more than a motor or transmission replacement on a conventional vehicle.
Tesla owners should file a class action lawsuit to force Tesla to extend their warranty to cover this issue. Tesla will try to fight it and make you pay out of pocket.
ECUs made in the last 10-20 years don’t really ever fail. Bosch has really figured things out and produces reliable ECUs for most OEMs.
Sure they do. Take a look at the FICM failures with the Ford 6.0 diesel. In the gasoline world you'll come across failed injector or ignition drivers if you look hard enough. Failures due to environmental causes (like the wiring harness wicking hydraulic fluid into the control unit itself) are a thing as well.
It's not engine related, but BMW's late 90s-early 2000s five series uses a computer to control its lights. The failure mode is that your lights stay on as long as the battery is hooked up (fail safe). Unfortunately the IC that BMW used was discontinued by Infineon. I think a surface mount version is still available, but ultimately yeah these things wear out and you're largely at the mercy of the OEM.
Injectors and ignition coils are effectively wear parts however on an ICE engine. Though for balance here, I just had to replace the fuel injector on one of my cars, it lasted 30 years however, so I'm not too upset by the 300 bucks it cost to replace it.
I'd strongly dispute that. Why not include the spark plugs as well? And the fuel pump?
The ECU is a computer, a packaged bundle of chips which plugs in to wires and does nothing but communicate to other components via those wires. That's what an electronic control unit means.
The driven devices, amplifiers etc. are under the control of the control unit. They are not part of the control unit. They're located in areas subject to much more heat and mechanical stress, and in most cases (ignition coils excepted) are mechanical in function.
I'd strongly dispute that. Why not include the spark plugs as well? And the fuel pump?
Because...
The driven devices, amplifiers etc. are under the control of the control unit. They are not part of the control unit. They're located in areas subject to much more heat and mechanical stress, and in most cases (ignition coils excepted) are mechanical in function.
I think GP is saying they're physically located on the ECU board / inside the ECU box, not that they're conceptually part of the ECU even if separate. Whether they are or not (physically inside the ECU box) differs by make and model.
ECUs aren't perfect. On my old 2003 Volvo a common problem was solder points in the instruments panel failing after 8 to 10 years, and for the central ECU the connectors could corrode due to moisture. Luckily I wasn't affected by either. Of course Bosch probably doesn't perform "disruptive" software fuck-ups such as Tesla did here, and if they do, chances are good out-of-warranty replacements are free or at least come at a reduced price.
(But at least an EV doesn't have issues like VWs "maintenance free timing chain" skipping a link or two after 70kkm/50kmi)
Late reply, but nope, iirc automotive was one of the industries which were/are exempt from the whole lead-free stuff because the ECU are safety critical.
Non-failing ecus are down to overspecd components, and obsessive supply chain management, given a decent board design. One single capacitor swapped out for a cheaper spec one can brick a huge percentage of correctly produced boards. Tesla is new, and this is a growing pain, and result of not hiring, or perhaps heeding the advice of, those familiar with industry best practices.
It is striking how different the attitude to this is in Australia.
All products and services in Australia need to be fit for purpose. There's no strict definition for this but it basically means if you are selling a "premium" product, your warranty lasts for however long is reasonable in the eyes of the courts and not what the manufacturer says. It is very pro-consumer.
It would be reasonable to assume that a $40k-$100k car would last 10+ years. Obviously this doesn't cover consumables but bad memory cards would definitely be covered.
But isn't that the consumer's fault for agreeing to buy a car with "an unreasonably short" warranty? If Tesla said the warranty lasts however long, why would you feel entitled to legal protection for it to last longer?
If you're saying this part is intentionally designed to fail shortly after the warranty expires, then I understand your point. Is that what you mean?
EDIT: The way I see it is if I buy a car with a 3 year warranty and something fails after 4 years, it's not the manufacturer's responsibility to fix it because the warranty has expired. Why do you think it is?
Fair point. I don't know why I had a mental block and the idea of a recall never entered into my mind.
I completely agree Tesla cheaped out on this part, I just wasn't convinced they are responsible for fixing it. I'm still not sure this justifies a class action lawsuit (I don't really feel strongly one way or the other), but I at least understand why Tesla might be held accountable for their actions.
I think the length of the warranty is kind of a distraction. Say Teslas had a particularly generous 6-year full warranty, but this part was guaranteed to fail 100% of the time after 6.5 years and cost thousands to fix. People would be justifiably upset.
You're talking about whether or not people should be upset. I'm talking about whether or not Tesla is legally responsible. These are orthogonal topics.
Yes, people should be upset with Tesla for a shitty design. I don't think anyone is arguing that. I don't think that automatically means Tesla should get sued for it.
Then again, I've never tried to sue an automaker. So maybe I'm just ignorant of how this is supposed to work.
In a world of perfect information things would be different. If a consumer knew before purchase that a car was going to die shortly after the warranty ends, they wouldn’t have much of a case.
But if Tesla cheaps out on critical items, thus dooming a car to that scenario, and a consumer can’t know that, it introduces a different legal dynamic.
> Fair point. I don't know why I had a mental block and the idea of a recall never entered into my mind.
Because us tech people have been successful at hand-waving away product defects as mere planned obsolesce; we think about it as an option less and less. Point: Intel and all of the speculative execution crap, and specifically TSX (or the lack there of... but marketed as "Secure!" regardless).
Yeah it pisses me off. If you make an under-delivering pile of shit and make no attempt to right the wrong, fuck you and your company.
Your argument is not totally unreasonable - but not within industry norms. Also Tesla is the manufacturer for electric cars in the public's eye, if they get a reputation for unreliability it will set the technology back decades. (see the Oldsmobile Diesel engine from the 80's)
So for a variety of reasons, I'd argue they have a moral responsibility to fix it, INAL, so it's unknown to me if they have a legal or regulatory responsibility too.
<i> it's unknown to me if they have a legal or regulatory responsibility too</i>
Depends on the regulatory environment. In the main though people can expect cars to last for a good long while with maintenance, so a part that was guaranteed to fail well before then no matter how good care the consumer took of it would be a regulatory no no.
One might argue that this flash memory was not the correct choice for the purpose it was being used for, and as an expert, it’s was Tesla’s duty to select a part up to spec for the purpose it was being used for.
> EDIT: The way I see it is if I buy a car with a 3 year warranty and something fails after 4 years, it's not the manufacturer's responsibility to fix it because the warranty has expired. Why do you think it is?
The normal expected lifetime of a new car today is apparently 8 years or 150,000 miles. So I would expect the car to last 8 years on average. If a car is going to last significantly less than that on average (e.g. assuming this means the car needs a fix every 4 years), this should be disclosed in the marketing.
"years" of warranty is a silly concept anyway. If I buy a new car then store it in a perfect environment unused for 20 years (assuming we maintain anything that isn't stable like oil), it should still work as if new. If it doesn't, it clearly wasn't sold fit for purpose.
That's kinda expected. Once a car becomes a pile of buggy software, someone needs to maintain it and fix bugs. That someone is called a SW engineer and happily charges 200-300/hour. Add on top of that the service dep fees that can be substantial (e.g. I pay ~150/hour for mechanical repairs, but the actual mechanic looks like a guy who works for a minimum wage).
The high hourly pay of a software engineer is not significant when that high cost is spread over hundreds of thousands of units sold.
A corrolary of this is that when it is not spread over a high number of units, in a niche product, it will be very expensive so we will see even less niche products as software takes over more functions.
And for many of us Tesla was the biggest intersection between DRM, @InternetOfShit , and #righttorepair . And even just the whiff of one of those was enough to be a hard no.
And for those of you whom know their history, this has all the hallmark signs of anti-trust.
Well now that you're out of warranty you no longer need to go to the dealer so to speak, you can go to aftermarket repair shops. Yes these do exist for Teslas.
Wow Tesla have big brass balls. Charging exorbitant fees for designed-in flaws and intentionally bricking it if you fix it yourself? The next time Elon's house gets vandalized we won't have to wonder why...
> You haven't seen any recent Mercedes Benz then. Full of electronics
I had to dump an 09 VW GTI (I'm in the US) about four years ago with only 60K miles on it. The ABS computer failed; a $4k repair.
As a side note I traded it for a new 2016 Jetta Sport; which was the worst car I ever had, and the last VW I'll ever buy.
I traded the Jetta for a 2019 Hyundai Kona which I like but you can't drive it with the forward collision system engaged. I had always shut that off from day 1.... except a month ago I didn't and that sucker SLAMMED on the brakes when I was lane merging during a busy time--I was Not being aggressive just assertive. Only by luck no one slammed into the back of me. I can't imagine Anyone trusting their lives to a self-driving car.. it makes no logical sense to do that!
Yup. The ICE engine reliability issue is more or less solved. I have built hotrodded engines in my garage that are still going strong at 275,000 miles. The real issue is that tesla reinvented the wheel with everything else too. Other solved problems like rustproofing, doorseals, and component access were overlooked for technology adoption. I admire the bold move, but it was not one destined for reliability. At the end of the day, the tesla is more complex, even in the drivetrain, than my honda civic, and costs quite a lot more, even under ideal conditions.
Sorry, how is the Tesla more complex in the drive train than a Honda Civic? The sheer number of moving parts in the Civic makes your argument highly questionable...
The earlier models were premium not economy pricepoints. Why is the ECU designed around low cost on-board, non-replaceable constructs like this? A premium product, would not nickel-and-time the SD card, it would have engineered it as a field replaceable unit, with an insert,format, wait (for download) and go mode.
Replacing the entire ECU feels like an admission like Apple and increasingly Lenovo (thinkpad), they have driven premium to the wrong place. "please give us soldered on memory and soldered on SSD" is not actually what we said, on our feedback forms to Apple, or IBM (X1 Carbon).
Nobody asked Tesla to shave $1 on the ECU assembly and the incremental cost of recall and fix, along with goodwill now outweighs the cost saving in any sense I can understand it. If the authorities demand he fix for free, it becomes another shit show of the TSLA share price. This wasn't a good choice against all the other choices.
Lenovo still makes Thinkpads that are extendable. My P51 has 3! m.2 (2 of which support NVMe,) 4 RAM slots, easily replaceable keyboard and more. But I think most people want thinner laptops, and the thinner ones have less of this. I don’t think this is a headphone jack type situation: people value portability a lot on laptops, and the difference between a rugadized, user expandable laptop and an “ultrabook” are striking.
My dell xps is super thin and it still has easily replaceable ram and battery. The massive use of glue, rivets, proprietary screws and cryptographic verification in macbooks is not for slimness but for planed obsolescence
Unpopular opinion, but from where I sit the serviceable lifetime of a computer is effectively about a decade, so long as all that non-replaceable stuff lasts the serviceable lifetime of the object, then it doesn't matter how repairable or not it is. Apple charges like 200 bucks for a battery and top case, which is pretty competitive for a replacement OEM keyboard and battery combo.
I think that's about the typical life span of a modern car, too. Anything older than a decade tends to be quite an awful drive. Unless it was built in the days before we decided everything needed to be so expendable.
You'll find the average age of vehicles on the road is around 11 years, and continues to grow longer and longer. Cars cost more, and last longer, so we tend to keep them longer and longer now too.
A car can be maintained well as long as parts for it are available. That people don't want to spend on maintenance of >10 yo cars is a different issue.
I mean.....in Poland the average age of a car is 14 years, and we have very harsh winters with plenty of salt being used. I don't think there's anything weird about driving a car that approaches 20 years old, or anything particularly tricky about maintaining one either.
I think it depends on the quality of sheet metal. I live in a place where it's all mud and salt throughout the winter, and somehow my car doesn't rust even where the paint is scratched. I think it's because of a good-quality galvanized coating under the paint (it's a boring normal non-luxury car made in Korea, if someone's curious).
The frame is what I worry about more than the body. I'd be happy to keep repairing my ~15 year old truck, but the frame is getting to the point where I'm not sure how much longer it's going to pass inspection.
Some poorly-designed models are particularly prone to rusting, but most cars can put up with this for more than 10 winters without losing any structural integrity.
That engine list is about driver-pleasing performance, not reliability. It states "The selection takes into account power and torque output, noise, vibration and harshness (NVH) levels, technical relevance, and basic comparative numbers."
I'd suggest the Cadillac Northstar would be on that list - they pretty regularly need a rear end seal at around 100k, which is something that requires removing the entire engine.
Conversely, there are many engines on there noted for their reliability, Ford Modular V8, for example, GM LS1, Ford Model T engine, VW Air-cooled Engine.
In Canada the AVERAGE car is 14 years old. That means half of them are older than that.
Don't forget, just because a typical owner may sell a car after 5-10 years, doesn't mean it goes away into the aether :).
[FWIW, I have a 2004 Subaru WRX; I make the dealership rounds every year looking for replacements, and cannot find anything that handles better in that price range - in fact, by all accounts modern WRXs are a softer drive]
Your argument is correct, but you might want to retake statistics. b^)
A long tail on only one side (no car has a negative age) means that more than half would be older than the average. You were thinking of the median, not the average.
I (literally) beat the crap out of my car; it’s going on 13 years and a quarter of a million miles. The engine mount is wearing out so it shakes a lot but other than that it drives great. I’d imagine people with similar cars that actually maintain them have even better experiences.
You can also replace the battry on the X1 carbon (RAM is soldered, though). Lenovo even provides videos showing how to remove the battery and other components: https://support.lenovo.com/us/en/solutions/ht508716. No glue, rivets, or weird screws.
It's really mostly the macbooks that are terrible in this regard.
Lighter maybe, not thinner. I There's no reason for laptop to be thinner than ethernet or VGA ports. Give me full size mechanical keyboard instead (minus numpad).
I don't want thinner, but I want the 13 inch form factor. I could change the memory, the HDD and the keyboard by removing a single screw on my old x220, and with zero risk of breaking little plastic legs.
> . Why is the ECU designed around low cost on-board, non-replaceable constructs like this?
What's not replaceable or too expensive about a $400 or so nVidia Tegra board in a $100k car?
> Nobody asked Tesla to shave $1 on the ECU assembly
Are you suggesting it would have cost Tesla $1 per car to go from an industry standard board used in many cars to a custom solution with all parts tested in-house?
Why doesn't any other manufacturer of cars have a similar problem? Did I miss the story where BMW or Mercedes demand $1200 to replace this part? Tesla knew it wanted a highly digital logging investment and they didn't design for use life. Doesn't feel like a sensible place to be.
But to your main point, no. The actual cost per board of getting a design done for replaceable parts would have been higher. Once they hit volume the amortization would kick in. But for the earliest, yes,expensive. Strangely in the world of eg routers, the first gen often has more hackabilty, more removable parts and it's the later ones which shrink down, gets the bill of materials down.
Seriously, they should comp all the replacements for the life of the car, just for p.r. value.
> Why doesn't any other manufacturer of cars have a similar problem? Did I miss the story where BMW or Mercedes demand $1200 to replace this part?
This particular issue may be unique to Tesla, but obviously other car manufacturers have a whole host of issues which their customers deal with and can result in service bills even staggeringly higher than this.
My last Infiniti G35 has an issue with the Bose head unit, where a single resistor would come loose and the radio would lose power when you hit a bump. The problem was the whole center console stack of buttons all were powered through the radio and so you would also lose the ability to change the temperature or turn on the defogger.
Cost was $2400 to replace the unit, and they would typically fail again ever few years. The problem was extremely widespread and despite a campaign to get Infiniti to extend the warranty, they never did.
I ended up putting in an after-market unit but I had to replace the module with all the climate control buttons as well with one from the Japanese market version of the same car, because that one could work independently from the radio.
No other manufacture does over the air software updates. This problem stems from the fact that the firmware image got bigger with each update as incredible new features were added leaving the logs to be written over a smaller area. Also soldered on eMMC is probably less prone to failure than removable microsd under typical automotive conditions.
Barely you still have to manually update the nav system. I pace only implemented partial updates recently due to competitive pressures from Tesla. Frankly I’ve heard mostly horror stories about ipace.
Tesla needs to have stock relative to replacement rate purchased before nVidia's last ship date. So if they used eMMC built in to a part like this then they should be raising problems for their parts logistics.
Not anymore in this case since they replaced this with their own design. Upgrading older cars should be possible, but is rumoured to be prohibitively expensive for Tesla as the moment, since other components have to be swapped as well.
At the rate they were rushing to design and build that hardware. I'm absolutely not surprised they chose not to go through the extra steps of qualifying a connector/mount for a flash daughtercard.
It's very likely the hardware designers didn't expect the software living on that flash to iterate that many times.
In too many ways, Tesla was and still is a hardware startup.
Every write up I've seen indicates the amount of stuff being put into /var is inappropriate for flash, beyond that, those logs should be kept in RAM until shutdown then you commit them to disk.
I'd have no issue with this stuff bricking if Tesla replaced it free (even more galling that they won't on a 50k+ vehicle), you'll have a hard time convincing anyone that a ECU should last less than a decade under normal operation, Nor should you need to replace the ECU before you replace the battery pack on an electric car.
If storage space was alone an issue, then they shouldn't have extended the features of those firmware loads so much - as far as I know the owner has no choice on if you update or not.
If you buy an expensive car, produced in low volumes, with little competition in the independent repair shop and aftermarket parts industry, expect to pay through the nose for maintenance, repairs and body work.
That also applies to other expensive consumer durable goods.
I think normally for automotive components you are obligated to use electronic components that have already been stress tested and are rated for more violent environments (like next to an engine).
I wonder if Tesla maybe skirted this (like "oh this computer isn't in the car it's an attachable add-on).
Source: I interned at an electronics component company back in uni as a chip tester. Mostly same chips to everyone but ones rated for automotive or space tended to get a lot of extra testing (since failures on electronics are bimodal, running tests in extreme environments just for a bit catches a lot of the bad apples).
It's worse. Elon Musk used to be bragging that they used consumer grade electronics instead of automotive grade (which is higher than industrial grade).
They used an industrial rated screen years ago, because no one in the world made an automotive rated screen large enough at the time. They did accelerated wear tests on it and worked with the manufacturer to make some customizations to try to improve the heat resistance, as well as adding cabin overheat protection into the software. Yes, this was not enough to entirely prevent the yellowing issue on some screens.
Look at, for example, what Sandy Munro says about the electronics quality specifically on the Model 3, and you will appreciate they have come a long way since then.
There is no automotive company I would trust more when it comes to quality of the electronics. This is a group which taped out their own custom silicon for their APv3 hardware, designed by famed chip designer Jim Keller, and will field upgrade it into existing cars. What other manufacturer sells hardware upgrades to their late model year vehicles?
Munro said the Model 3 had the best powertrain of any EV on the market.
He had negative things to say about the quality of the rest of their electrical components, noting that based on the cost of the components Tesla should have an 18% markup...because Tesla is using consumer grade parts, not automotive grade parts.
What other manufacturer sells hardware upgrades to their late model year vehicles?
Almost every car sold to consumers in the US today allows you to replace most parts of the car, which includes upgrading nearly every component, if you so desire. Sometimes the automakers sell upgrade parts; usually they leave it to third parties (i.e., the "aftermarket"). It's actually not a good thing that Tesla is the only source of upgrade parts for their vehicles.
Their margins are better than their competitors because Tesla is vertically integrated, allowing them to do things like reduce the number of harnesses by 40%, and design solutions like the "SuperBottle" which Munro once said is his favorite part of the Model 3:
And there's a world of difference between after-market upgrades (of which there are plenty for Tesla) and the manufacturer taking their latest generation technology and manufacturing and designing it so that they can slot it into late-model cars, all while constantly shipping firmware upgrades to their entire fleet which make the cars better in every possible way from speed, handling, range, efficiency, user experience, safety, entertainment, etc.
>> on our feedback forms to Apple, or IBM (X1 Carbon).
Whoa Whoa Whoa ;-)
As somebody who has 9 ThinkPads around his house, including the controversial T25, don't forget Lenovo completely bought and owns all Thinkpad design production branding etc for the last decade.
Would IBM have maintained their priorities if they owned the product today? Not sure. But Applezation of the ThinkPads has happened strictly under Lenovo :-<
The P series is still pretty good; X1 was never about modularity but strict portability; it's the devolution of T series that bothers me personally the most - the T420s was barely thicker than modern T490, but had swappable battery and hard drive. Not "user replaceable", literally push a button and swap it out.
[Disclaimer: IBM employee, but in a completely unrelated part and a ThinkPad fan before my IBM employment; I work in software consulting on large ERP projects and haven't seen an IBM hardware person in my 15 years of service:]
There’s no excuse for this at all. It’s shitty short sighted engineering. Simple as that.
The main firmware should have been on a separate physical device if data was write heavy regardless of “enough space” being left. It would have cost a few dollars to make this issue go away.
The more I hear about the engineering the longer I will avoid buying an electric vehicle. Iterating big problems away on a tangible chunk of engineering rather than a piece of software is stupid.
A couple days ago I found this link [0] to a Reddit post on HN about the software engineering practices at Tesla. Don't know how accurate the story is, hard to tell, but it sure was worrisome.
> The more I hear about the engineering the longer I will avoid buying an electric vehicle.
You can look up the nVidia Tegra board on Wikipedia and see for yourself that the "shitty short sighted engineering" board is used in many gasoline cars.
Yes, but those ICE cars were designed by engineers who understood that you only get ~100ks of write endurance with this sort of nonvolatile memory.
Tesla's use case is unique because they stuff enormous firmware images into the on-board NVM and write to it constantly with very heavy logging. Of course your Flash/EMMC will wear out quickly if you constantly write to it while your device is on - that is the short-sighted part.
The board isn't necessarily a problem, but how Tesla chose to use it sounds like a very predictable problem.
There was once a guy that shared instructions on how to resolder that chip (Get dump of software, root it, solder new chip with more memory). It was instruction on how to root Tesla. But oh, he deleted his content and left some comments:
> As a result I've taken down my howtos. I'm sorry about this but there is some kind of political problem which I don't understand.
I'd be surprised if it's any different to modding your BIOS in a PC motherboard: Buy a SOIC-8 clamp, a USB interface and bob's your uncle. Unless the ECU is potted in epoxy.
Doubt it’s an SOIC. More likely a 40 or 48 pin TSOP or the like.
But if it was an SOIC... just lift the power pin, power the chip externally and find some other points to tap into the clock and data lines and dump away. Then solder the legs of the new one on one side and connect VCC on the other side.
Wear leveling isn’t the issue here as eMMC is wear leveled. The issue is probably just that the total volume of writes has literally just worn out too much of the flash.
Firmware doesn't update as often though so it'd have to be static wear-leveling. I'm not sure of the safety of doing this with firmware files though - could it actually decrease the life expectancy by moving critical blocks more often?
In any case, if 90% of the space is filled with static firmware files then wear-leveling becomes difficult. Probably best to turn it off and let the physical blocks with the logs fail while keeping the firmware blocks intact, or just stop writing logs after a certain point.
> could it actually decrease the life expectancy by moving critical blocks more often?
No. You do normal, dynamic wear leveling. When you notice that the block you want to write to is much more worn than the rest-- say 100 erases extra-- you move a firmware block to it instead and claim that for use. Then that block sits near-quiescent for a long time until it's time to put it back into the rotation of blocks that get actively written. Static wear leveling can just be a tiny, tiny share of write amplification-- a factor of 0.01 or even less-- while keeping all the blocks in play.
You only "lose" from static wear leveling if you end up moving something right before it becomes active again. Whatever your strategy, there's always a degenerate write pattern which will cause useless write amplification, but even relatively simple schemes are robust against basically everything except deliberately malicious write patterns, and you can bound this worst case.
Incidentally, one way flash fails is by dielectric breakdown increasing leakage causing worn-but-idle-blocks to slowly accumulate error. Moving / occasionally scrubbing static, important data actually helps.
Or write the logs to a separate and easily replaceable module/drive. Putting the firmware on the same chip as what you're regularly writing to is probably a bad idea.
This doesn't bode well for classic and antique car collectors of the future, who will want operable Tesla cars.
There are Ford Model T cars that still drive. They were made 92 to 111 years ago.
With undocumented and cryptographically signed interfaces between components, it will be impossible to keep a Tesla running that long. The best you could hope for is to graft the body panels of a Tesla onto the innards of a newer car, merely dressing up the newer car to cosmetically look like a Tesla. Such a hack would not be have vehicle title as a Tesla, would not drive like a Tesla, and would not have an interface like a Tesla.
This goes for any modern car, Tesla is no exception here. Cars contain large numbers of embedded computers, chances are that in every car there is at least some time-bomb like bug that the manufacturers aren't aware of that will kill the car in a few decades.
There is already some development of open source ECUs:
A problem like this could be significantly alleviated by not having the flash soldered onto the motherboard. This is one of my concerns with the current generation of Apple MacBook laptops. Considering the cost of a Tesla as a whole, surely it can't be ridiculously more expensive to put a m.2 socket and discrete flash module on a motherboard.
> surely it can't be ridiculously more expensive to put a m.2 socket and discrete flash module on a motherboard.
It would be more expensive for Apple because people would buy cheap models and later upgrade, instead of buying expensive models or buy new machines early.
I suspect that it may be worse if they didn't solder given that it is in a vibration prone environment. The underspeccing for wear is more of an issue than the soldering in my opinion since if you are dealing with chips anyway you are already pretty damn elbows deep into a specialized area.
Hackers are already ripped apart Teslas and building things from the parts. It's VERY common to use the battery packs (thus their high resale value). Motors are popular as well. Sure there's some crypto/DRM for things like the car's identity (and if it has free supercharging for life). But generally a large part of a Tesla would be usable even 100 years from now, even if Tesla (the company) is dead.
My estimate is that 50 years from now that encryption will be crackable.
Alternatively there may be holes that can be used--such as a 'boundary scan' of one of the chips yielding the key. I don't have a spare Tesla sitting around to test this theory, however (grin)
Might not be that long. In the early 2000s 256-bit RSA keys were common. These days manufacturers don't care about twenty-year-old cars an owners have factored the keys.
1. Let's go back to the good old HDDs. There are even automotive industry ones (I own one)
2.There's a difference (8-fold) between Gb (gigabit) and GB (gigabyte). Memory chip capacities (and data transfer rates) are often expressed in bits (per second), but memory device (RAM stick, storage,...) capacities and file sizes are always in bytes.
Does anyone report on eMMC wearing out in phones, game consoles, TVs, laptops, etc.? What makes Tesla special? Are there really no other car manufacturers using eMMC btw?
Easiest hardware fix would be to replace the part with a larger one which would only be a BOM change. But that doesn't help the units already in the field. I'm sure they've figured something out by now.
>Tesla’s firmware size over the years went from a paltry 300Mb to a full 1Gb over an estimated four year period, which consumes much of the storage space in that Flash. Add on the constant vehicle log updates, and the failure rate of those chips increased exponentially.
Does it really increase exponentially or does 'exponentially' just mean 'a lot' now? Genuinely curious. Also I think I'm allowed to be pedantic when the source is an electrical engineer.
Most people replace their phones and laptops every 2 to 5 years. Too fast to see any problem under normal usage.
Most game consoles and TVs are unlikely to do enough logging to wear out their storage. But if they did, especially if they cost as much as a car, it probably would make the news.
Cars tend to last a decade or two. They shouldn't stop working because of something as silly as excessive logging.
It’s well-known in the embedded software field that you have to be mindful about writing large amounts of data to flash storage. Yes, a lot of other products, including cars, undoubtedly use flash. This is one reason why you turn down logging to a moderate level before shipping, or store log data in a RAM disk and offload it to the cloud rather than store it locally.
> Does anyone report on eMMC wearing out in phones
Phones hit bad batteries before anything else and issues related to them get reported fairly often.
> game consoles,
I think Nintendo had a console that hit max firmware updates, instead of breaking it just meant they couldn't interfere with people rooting it anymore. There was much rejoicing.
> What makes Tesla special
An order of magnitude higher price compared to anything else you have listed. Maybe they should try to sell cars for the price of a new smart phone with the goal of selling a new one every two years so nobody would notice these long term issues.
Assuming it writes log files at a constant rate, the rate of wear is proportional to 1/(1-f), where f is the fraction of storage taken up by firmware. Interestingly, this isn't exponential either - it's infinite at f=1!
> Does it really increase exponentially or does 'exponentially' just mean 'a lot' now? Genuinely curious. Also I think I'm allowed to be pedantic when the source is an electrical engineer.
I mean, exponential over what? Over time? The failure rate (more accurately the cumulative failure count) will probably grow logistically, but would have done so without the firmware size increase. Over free space reduction? That should be linear. More accurately, the amount of data you can turn over before wearing out the flash should have a linear relationship to the amount of unallocated storage available.
Maybe meaner than you, I discredit assertions from folks claiming whatever is happening exponentially as just rote repeated hyperbole...and it still annoys me.
The Model S and X have 8 GB of flash, according to the article. Does the Model 3 have more?
> The MCUv1 units supplied for the Tesla Model S and X, up until 2018, feature an Nvidia Tegra Arm-based SoC with 8Gb of eMMC integrated into the same board, which is hosted on a mainboard.
There is no such thing as a cheap fix for an incredibly complex and proprietary device. You will have to pay a highly trained expert and hope they have stock left of the part.
Well the hardware fix would be replacing a likely rather expensive board, and more sensibly, separating the constant-write kind of load onto a separate block device, preferably something like a microSD matched with a significantly larger eMMC.
There could be a software update to slow or disable the log-like writing or at least warn when the reserved blocks were running low.
I thought that Tesla engineers would have thought of this. Guess it’s hard to be an innovator and inevitably you will fuck up pretty majorly. With cars though these fuck ups are more significant than with phones.
In previous threads I've seen Tesla roasted for doing this; apparently most vehicles that use flash loggers like this make it a user replaceable module.
I know it's hard to include replaceable storage due to the vibrations etc. but hopefully the new Teslas will at least start using something like XFMEXPRESS for semi-permanent storage.
> Guess it’s hard to be an innovator and inevitably you will fuck up pretty majorly.
Or, alternatively, hire people who know what they're doing and take their advice? Flash wear isn't exactly an obscure issue, and it would certainly be on the radar of, say, Bosch (who make a lot of this stuff for the industry).
Right, but not something you might consider until your OTA updates makes your kernel go from 300MB to 1GB. How many car manufacturers do OTA updates? Still, it seems like an extreme oversight and shouldn’t have happened even in the most rushed conditions.
