I have worked in card payment industry. We would be getting products from China with added boards to beam credit card information. This wasn't state-sponsored attack. Devices were modified while on production line (most likely by bribed employees) as once they were closed they would have anti-tampering mechanism activated so that later it would not be possible to open the device without setting the tamper flag.
Once this was noticed we started weighing the terminals because we could not open the devices (once opened they become useless).
They have learned of this so they started scraping non-essential plastic from inside the device to offset the weight of the added board.
We have ended up measuring angular momentum on a special fixture. There are very expensive laboratory tables to measure angular momentum. I have created a fixture where the device could be placed in two separate positions. The theory is that if the weight and all possible angular momentums match then the devices have to be identical. We could not measure all possible angular momentums but it was possible to measure one or two that would not be known to the attacker.
First, wow this is both incredible and crazy! Both the China-side hacks and your side's anti-hack. Mind. Blown.
Second, would have it been cheaper to manufacture somewhere more trustworthy (another country?) instead of spending all this time/money on your anti-hack systems?
This wasn't our device. There was a big, reputable company behind the device. We were ordering a number of those and they would be shipped to us directly from China.
Also, we were basically locked in due to the magnitude of investment in the software we have developed for the device.
Fortunately this only lasted for few months until it was dealt with. It was quite new back then (a decade ago) and it was a surprise for everybody I guess.
> Second, would have it been cheaper to manufacture somewhere more trustworthy (another country?) instead of spending all this time/money on your anti-hack systems?
I'd like to know this too. Has the West completely lost the ability to mass produce microchips at even a reasonable cost for financial applications?
America has fabs, both old and leading edge, but ask industry giants like Gemalto to even bother to manufacture chips anywhere outside of Taiwan, assemble the final product outside of China.
They will never do that, because they look for the cheapest solution.
The bigger the company, the less it cares about things other than cost. This is why Mediatek and Broadcom can usurp the market of network SoCs, while making products with atrociously bad support. I personally dealt with both, and say that they wholely match their popular culture image.
I don't know how it is with USA, but for Russia, the military doesn't care that their chips had frequency measured in kilohertz, and had sizes measured in square sentimetres, for as long as they get them made inside the country.
Gemalto manufactures and assembles many, many products in Europe. The entire European security cluster won't use anything else. I know because I played a part in smart card development there. And all of that was from Europe. Wafers, chips, holograms, mag stripes...
Not exactly, PCB assembly is super cheap everywhere thanks to propagation of chipshooters, what makes the cost go up is logistics - what do you do after you populate the board for your part? Ship it across the world, or to another factory behind the corner?
That largely depends on the size of your board and the total number you want to ship. As soon as you reach full truck loads or full container loads that additional shipping cost is marginal on a board level.
You are right in a certain way. What increases with distance is lead time and as result inventory. Both of which are off set by volume. Cross-border adds customs issues. Complexity is not that much of a problem nowadays, maybe it never was. You are right that all of this oblyakes sense with large numbers, small scale production is better done locally as a rule of thumb.
> for Russia, the military doesn't care that their chips had frequency measured in kilohertz, and had sizes measured in square sentimetres
Well, if your chips are bigger and slower, you will need more chips and mounts/packaging to place them. If you need more chips then the weight of missile/plane/tank will be increased and available space decreased.
So at the end, the 'uncaring' military will receive a weapon which is worse than competition.
I think people should know how stark the differences for assembly in the US are vs outside of it. Something that costs, at low prototype volumes mind you, $20 in China for a dozen boards or so, would run hundreds of dollars in the US and still take the same amount of time. As it scales up, the ratio might improve, but these aren't like 10%-20% differences.
Nationalistic flamewar is not welcome here; neither are political or ideological flamewar. We ban accounts that use HN primarily for those things. Please read https://news.ycombinator.com/newsguidelines.html and follow the rules when posting here.
at first I had the same thought. but i have to question how securely the same manufacturing could be done in a US plant.
the US employee base has its fair share of desperate, ethically challenged individuals. and plenty of incentives to make a quick buck could be offered here too. idk.
OTOH, given US law enforcement's low efficiency what are the chances of being caught? and what if it's merely corporate espionage?
finally, the US is an open society with strong personal freedom guarantees builtin. what if the perpetrator has ties to a foreign country and simply leaves the US after they've installed the vulnerabilities?
You are underestimating the FUN of playing anti-anti-^N-hacks. I have had the privilege to be paid to so anti-anti-^N-hacking on a firewall thingy in the past and it was a challenge and a joy!
Honestly if it weren't programming I feel like this is a movie-worthy story. To me it sounds so thrilling like a spy movie plot but am I just imagining that or was it actually this crazy / cool / integral to way bigger moving parts / things like I'm assuming? Regardless kudos. Your story definitely started my day on a happy note, thanks!
We had MasterCard end-to-end test auditor on site. This is the first time ever you get to do a transaction with real transaction system with real credit card.
Due to requirements we opted to have the only large meeting room to have outside our secure zone. This created an issue as we had no network access from there and in the end we decided to use slow GPRS terminal for the test.
The end-to-end test starts with offline transactions which by their very nature are quite fast (it is negotiated between terminal and card).
But then we went to online transaction and it finished instantly too.
The auditor, bewildered, proclaimed the test failed as he assumed it was incorrectly processed offline instead of going online. But then I pointed out to the printout to show ARQC (basically says it was certified online).
Now, the real discussion started. The terminal was very slow taking quite few seconds to establish GPRS and then even more for the SSL handshake so the auditor said it was not possible to make it work.
How it worked was that I have completely gutted OpenSSL and had entire cryptographic state stored locally (safely, using internal HSM) so the SSL session could be optimistically re-established without another handshake even after TCP connection was closed. The first message the terminal sends is already encrypted transaction message, there is no SSL handshake. I wrote an application to terminate the connection in our data center so that it stored the states of each connection in the database. The entire handshake was only done if the first message could not be decrypted successfully.
The operating system was single-threaded with no multitasking of any kind. This meant that all applications on this device did their operations sequentially. Send network message, print something, display something, etc.
I wrote a cooperative multitasking functionality into the application (using coroutines) so that it could work on multiple tasks at the same time (like talking to network and printing).
I then have segregated all data on the printouts so that it can start printing without having to already have response from network. Hopefully if everything went right, the response would come before it even came to that place on the printout effectively looking as if it was done in zero time.
FWIW, the described technique (or something roughly equivalent) is now standardized as 0-rtt early data in TLS 1.3. (you still need 1-rtt for TCP, unless you can combine this with tcp fast open, or run TLs over UDP)
But am I wrong to have my hackles raised by a) the roll-your-own security nature of this, b) the reliance on a single developer's single stack implementation as what guarantees the integrity of the system? It seems like there are a lot of assumptions baked in.
I, too, would love to see a more detailed write-up--if there's a big idea here (almost a unikernel thought), it deserves to be shared and tried by fire.
When I worked in telecom (a while ago) the manufacturing was shifted from China to Thailand/Other SE Asia due to this. The Thai companies weren't as efficient, but were much more open and honest when problems would arise., plus they didn't blatantly steal tech
See, the article showed that even largest companies are not completely immune to the problem. This was decade ago and payment card industry, not exactly national security matters.
The PLA can lean on factory managers very effectively but they're not going to be interested in small time stuff like credit card numbers. The sort of sophisticated criminal gang doing something like this will have fewer coercive tools at its disposal and I'd imagine would target lower level employees with bribes.
All big and security-responsible companies issue their employees special phones and laptops when they go on business trips to countries like China or Russia and these are quarantined immediately after they return. They get wiped, X-rayed, disassembled and checked, including any accessory (chargers, mice, etc.).
The more critical the field, the more you have to treat those devices as untrusted before attaching them to your trusted zone.
> They get wiped, X-rayed, disassembled and checked, including any accessory (chargers, mice, etc.).
Given how sophisticated these attacks can be, I'd think they'd issue disposable equipment to be destroyed on return, like a cheap netbook or something. I don't see how you could trust an individual viewing a simple X-ray scan to detect some extra microchip the size of a signal conditioning coupler.
Procedures change, as attacks get more sophisticated the next step could be disposable devices. But an attack like the one described in this article won't be mitigated by having a disposable device. On the other hand having your laptop "hijacked" while on a business trip will most likely involve some extra PCB or components that are a little more obvious that something that's "built in".
Then again many companies or public institutions would find it hard to justify shredding each week maybe tens of laptops and phones that still have to be good enough to work on. Basically they still have to be a "standard issue" device with your company's software stack, config, etc.
I'm sure someone can find a good compromise between security and wastefulness.
The obvious solution would be to earmark some number of dedicated "china laptops" that will never be trusted with anything important, and get reused everytime someone has to visit China. If they get backdoored... who cares? They can only spy on things that were getting spyed on anyway due to being in China.
Does the security team have any motivation to reduce waste? Unless you're finding hacks in your devices already I see no reason to think they aren't just making you jump through hoops because it's funny.
The post-trip inspection is not so that the device can be reused, it's so you can (try to) find out if it was compromised. A $3000 laptop is not a significant cost compared to the airfare, hotel bills, etc.
But it's useful to know when/if you're being targeted.
One of the companies named in thr Bloomberg article does. They just deatroy your laptop if it was in the hands of customs without your supervision for any length. US customs explicitly included, which is kind of wierd if you ask me.
That’s perfectly expected. It’s not a stretch of the imagination to think border checks are abused for industrial espionage. If it gives your country a major advantge nothing is off limits these days.
My employer - doing lots of interesting engineering in among others the marine and aerospace sectors - certainly do that; we're based in Europe.
Basically, any device brought to the US and a number of other countries are issued for one-time use; if they leave our custody for even an instant, they are to be scrapped.
> All big and security-responsible companies issue their employees special phones and laptops when they go on business trips to countries like China or Russia
From what I understand, Boeing does this when employees visit France.
They have had problems with men in nice suits going through laptops stored hotel safes.
Sure, we were discussing someone saying their devices came with extra PCBs inside (it's a bit hard to follow but scroll up to the original comment, currently first on the page).
> We would be getting products from China with added boards to beam credit card information.
>> Trying to guess the contents of a box that you cannot open sounds a bit like madness.
>>> Use X-ray? or whatever can penetrate the exterior shell
2 different types of attacks, 2 different types of responses.
Yes, irrespective of country where its manufactured, if there are compliance requirements around an un-openable box, then some process becomes required.
But I think the GP's question is: "Whether it would be cheaper" - in the sense whether such an expensive QA process could have been averted by having a more trustworthy partner. One whom you're not on a race hack after hack.
The point is that if the devices are sensitive with compliance requirements then you must be able to verify them irrespective of who you hired to manufacture them.
You cannot just trust the word of a contractor on this because it's your ass on the line.
The point is that the process was to assure the device wasn't tampered AFTER shipped from manufacturer. Nobody thought it could already have been modified so early in the process. This is the eternal cat and mouse game. When I started in IT in 90s it was assumed that company network was quite safe and you didn't always need passwords, maybe for critical resources only.
I would think that, logically, and as illustrated, "the device wasn't tampered AFTER shipped from manufacturer" means after YOU have shipped it to customers. The anti-tampering system is to prevent modifications in the field.
The manufacturer shipped the device to us from China. We were already customer. The device would already have been locked. We would customize it some more (injecting cryptographic keys, application, placing our labels on the device) and then send them to merchants. The merchants were never customers, they would get it on loan from us. This was the only way to do it as the device could not be re-used with other acquirer so it only functioned as long as the merchant had valid merchant account with us.
> as once they were closed they would have anti-tampering mechanism activated so that later it would not be possible to open the device without setting the tamper flag
You didn’t specify what type of anti-tamper was used, but I wanted to jump in and say usually that means nothing. The US government intercepted packages [0] and put in back doors (removing and replacing the seals), so I’m not sure why you were so quick to dismiss state sponsored attacks by something as simple as an anti-tamper seal. You can learn how to do it yourself at most medium to large hacker conferences too (DEFCON, BlackHat, HOPE, and CCC to name a few, but there’s more with it).
You can just buy counterfeit anti-tamper stickers but if there is a switch inside the unit that flips a bit in some sort of write-once memory, then that would require removal of an entire chip and replacing it with another that may not be 100% the same. You can have a chain of trust in the system where chips will only talk to each other if they all spit out the right hash. Bury the SPI/I2C lines you use for this trust check within the PCB so you can't access it without drilling the card and add a layer of anti-tamper traces that trigger another tamper event if disturbed. Now what was just a quick install has turned into a whole PCB rework job where you are having to swap all the chips with a virgin set, assuming you can't get your hardware in there prior to final assembly.
Use an AES256 key from the factory to hash the chip's burned-in serial number and the time from the RTC. Lock out JTAG interfaces so once the chips are burned, they are inside their own fortress. There are a ton of ways to really lock down the hardware other than a shiny sticker and weird screws. Those keep people from breaking their own hardware, anti-tamper tech in chips keep out bad guys.
The device outer enclosure was tamper evident but the device itself was tamper proof HSM, basically. Any kind of intrusion (melting, dissolving, drilling, etc.) into a secure internal enclosure (separate processor, memory and battery) would cause internal battery to be disconnected from internal SRAM and basically the device would loose all cryptographic material and then self-destruct.
To give a bit of background, when you type your PIN on credit card terminal it is not the terminal application that is really getting the pin (well, except for special credit cards but that is really problem of the Bank that issued the card). The Visa/Mastercard mandate that the application don't have control over the PIN and that the PIN entry uses physically separate keyboard and display.
To achieve this, the keyboard and the display is galvanically separated for the duration of the PIN entry and the PIN is transferred directly to the HSM where it is being encrypted before it is transferred to the application processor for the rest of processing.
Certification. Worked in the same industry, and there were very strict both hardware and software requirements for POS software. Having gone trough credit-card audits, early EMV certification programs, and certification to place non-payment software next to payment software on such systems, I can tell you - it's no joke :)
It's not your problem. For the transaction to take place it has to go to your bank and your bank trusts Visa/Mastercard to manage risks. You would be surprised to know most frauds are absorbed by banks. Nobody is interested in people fearing plastic because it causes people to increase very expensive debt plus additional interchange fee from every transaction.
That, at least, is a problem that cab be solved by tightening security at supplier site.
The safeties are mainly to guard against the rest of the world. For example it prevents tampering in transit, or even in our own company - disgruntled employee can't do anything.
Or think for a second about the fact that we leave the device for, hopefully, entirety if its life at the client site. We had clients that were shady businesses like strip clubs that no other companies would touch with a stick.
Those VISA/MasterCard rules can't be universal because there's at least one bank issuing merchant terminals that run Android and take the PIN on the touchscreen:
Clover CEO here. Won't comment on a competing device but this may not work the way you think. In Clover's approach the touch controller input isn't reaching the Application Processor running Android when in PIN entry mode. You can do patent search if you're interested.
That would be in line with the requirements. You go through stringent certification with the software and hardware that has access to the actual PIN and then show that the application and application hardware never really has any access to it so that you can customize/update your software.
This is the easy part.
The hard part I remember was establishing secure communication between all components in the system (initializing HSMs, injecting keys). I remember helping designing the process and writing hundreds of documents describing various security-related procedures like how the HSM racks are inspected, how the keys to the racks are fetched from the safes, how there are multiple safes for multiple security officers, how the officers are prevented from ever having access to other safes, how fetching anything from safes requires logging and using tamper-evident containers, how the logs are inspected, and so on.
I have designed a special cryptographic protocol so that we could generate and inject keys to the devices in KIF (Key Injection Facility) and separately to our database (to establish communication with the terminal). Fun.
Agree the people and process side is very difficult to do well. Familiar with all those and more -- we have extremely good, dedicated employees who care deeply about doing those things right.
We have some fun stories on this topic, like when we were using our PCI PIN approved secure room in our development office for the first time. We papered over the cage to prevent a security camera from being able to see employees entering PINs on the HSM. An eager employee papered over this cage a little too well cutting off the natural flow of air. And then there was a bug in our offline CA code and we spent 30 minutes in that air deprived cage while debugging occured :) finally the bug was fixed, we issued the cert on our first production device, and stepped out to get a breath of fresh air. Obviously this isn't our daily driver secure CA room :)
I have few more stories like the time when I closed the HSM rack door a bit too energetically and caused outage to entire company as we had to bring in third security officer to re-initialize it.
We also had special screens created for all cameras in the datacenter to block view on the HSM racks.
The biggest issue was, just before end-to-end test we figured out we forgot one of critical procedures (it was establishing authenticity of the HSM used) and we had to scramble to get new HSM and to re-establish all cryptographic material (so new storage keys, etc.)
What you're describing sounds like another backdoor of its own!
The general problem with most "industry security" approaches is that they simplistically attempt to wrestle ultimate Godmode-control for themselves, rather than working towards eliminating it.
Those are universal PCI council rules the fact that the terminal runs android doesn't mean anything it still needs to be compliant with the PIN entry device requirements as well as PTS and POI requirements depending on the exact nature of the device:
The requirement for the tamper proofings is literally the first requirement in the PED standard:
A1 Vendors must comply with all components of A1.
A1.1
The PED uses tamper-detection and response mechanisms that
cause the PED to become immediately inoperable and result in the
automatic and immediate erasure of any secret information that may
be stored in the PED, such that it becomes infeasible to recover the
secret information. These mechanisms protect against physical
penetration of the device by means of (but not limited to) drills, lasers,
chemical solvents, opening covers, splitting the casing (seams), and
using ventilation openings, and there is not any demonstrable way to
disable or defeat the mechanism and insert a PIN-disclosing bug or
gain access to secret information without requiring an attack potential
of at least 25 per PED, exclusive of the IC card reader, for
identification and initial exploitation as defined in Appendix B of the
PCI POS PED DTRS
But this device doesn't need a way for the user to enter the pincode. So, all the sensitive part of the terminal is probably completely isolated from the android part.
I don't know this device internal and the PED/PTS exact requirement but it seems plausible for me.
You have something like a physical compartment who include the NFC and everything needed to process it like in a classical terminal. This compartment is highly secured as requested by the specification with just a very simple interface for the android part to send the amount to bill.
I've seen a lot of each-machine running on windows. Doesn't they work like this with the windows machine just managing the display buttons to select the amount and sending this information to the secure part who handle card interaction, pincode and delivery of the money ?
Yes this is a common design in fact most current solution segregated the POS and POI completely anything that handles the actual credit card whether it's C&P, Track2 or NFC is a closed black box with the required PED/PTS/POI and P2PE certifications the merchant never sees what's going on they only can talk to the thing in bill the next card X and get a confirmation of the transaction that's it they don't see any of the card data they don't even see any card holder data unless they collect it in a side channel e.g. a loyalty program.
Now none of these certifications or standards is bullet proof but people have a very skewed vision of the PCI certification process likely due to bias of only having interacting with the PCI-DSS requirements for merchants and low levels to boot meaning they didn't had to do anything but to fill the SAQ themselves and be on their way.
Worked in the payment industry for years. Visa/Mastercard do absolutely nothing to verify that companies are not storing Pin codes. The HSM is required for communication with them only.
HSMs are required so that the company does not need to have PIN codes exposed anywhere. Not having PINs or full credit card data makes your life easier as there is nothing to steal from you in the first place.
If your company stored PIN codes it means you were in breach of the contract and it had to lie to the auditors to pass the certification.
It is correct. Incompetence abounds. You are correct about the HSM but it does not enforce anything except for the exchange between whoever and MC. You do realize that pin codes are entered into a UI and phone system as plain text right? There are PCI audits but they are a joke.
You realize when we talk pins we mean authorizing your credit card chip&pin transaction? Nothing to do with your phone pin or maybe som other pins.
The pin we are talking about is what is customized on your credit card (directly in its memory) or its equivalent in your bank's HSM for the sole purpose of performing CVM step negotiated by yor card and payment terminal.
That's not correct the QSA will validate that the device does not store PIN codes or the that the merchant does not store anything they are not allowed.
Devices that accept cards need to comply with PED/PTS security requirements including very strict physical security requirements which are validated by PCI council approved laboratories and firms.
The POS and the PED/PTS isn't the same thing the POS can complete the transaction without touching the credit card in fact most of them do exactly that the only thing that it does is communicate with the PED/PTS to send the amount and get a confirmation/denial.
It's not CNP doesn't use the pin it uses the CVV2, you also can't use the chip and pin or track 2 swipe data for a CNP transaction.
I think the GP is confused on how a POS works, POS isn't a POI most of them don't touch the credit card they just talk to the reader, most readers today are P2PE closed loop solutions so the only thing the POS does is sends to the reader charge the next card $X the reader will then reply if the transaction went through or not and that's it.
The reader itself will talk to the acquiring bank or the payment provider in a point to point encrypted closed loop and the merchant would never see any credit card details.
SAQs don’t involve QSAs.
They are also intended for merchants which are a rounding error also there is no SAQ for PA, PED, PTS etc. certifications only for merchant PCI-DSS.
You can totally fake your way through PCI audits. I know of a company that did it for years using a fake network and servers. Not sophisticated at all. Most auditors do not find all of the compliance violations. They have one person do it. It's all about money.
You can fake a lot of things so what? That’s not the point, also PCI DSS is pretty crappy but the hardware vendor, payment provider and P2PEE certifications are a completely different story good luck faking it.
Sure you can send fake devices to be certified and sell something completely different but the same can be said for any certification and if you get caught boy or boy...
Wrong. PIN codes are entered into a damn mobile app and passed through an API. Billions of times per day. You guys are clearly missing the card serciving aspect of the industry.
I never said a nation state couldn't attempt this but just that if you try really hard to lock down your hardware, it becomes very expensive and risky. You may be better off attacking the system in another way other than cracking it open and soldering your payload in there.
Payment systems are typically better defended than by just a sticker.
It’s not surprising to see a ton of tamper switches, vibration/shock sensors, even light sensors. And they’re all powered by an internal batter and separate MCU that will brick the device upon open.
Depends on the hardware and the anti-tamper measures. I've seen POS terminals where the pcb was completely encased in security plastic, where any attempted breach would wipe the internal security keys, which meant the hardware just became a useless. They're so sensitive that these things enter "tampered state" from time to time without any tampering. I developed software on these things - and bricked multiple devices by accident, even though this was 'development' hardware.
I have caused production outage during one of our reviews of the rack with the HSM. The procedure required opening the rack, inspecting and accounting for all devices and cables and then checking the status of the HSM itself. At the end of the procedure the rack had to be closed.
Visa/Mastercard required that there are two people present and that there are two people required to open the rack.
We had it modified so that it has second lock.
The additional lock did not fit perfectly. During the procedure when I tried to close the door the door snagged on the lock and then slammed shut. It wasn't a lot of force but it caused the HSM to loose its keys and required a lengthy procedure to get three security officers to initialize the HSM with components stored on their smart cards during which the whole system was unavailable.
Yup. This would be the modern equivalent of Air America and other schemes by the CIA to raise money to operate by involving themselves in illegal activity like the drug trade.
People said the same thing about personal detail metadata as well. While theft would be foregone foolishness tracking transactions could have value to them. Granted it would probably be easier to snoop on the fiscal institutions.
Wait a minute... So your company has a Chinese equipment supplier, finds out that the supplier is tampering with your purchased equipment, and your solution is to add criteria to the incoming inspection?
No wonder China keeps screwing with you guys. You aren't supposed to eat that cost! Write a PO with tons of fine print that says "We will disassembly units at random for compliance inspection. Non compliant products will be returned at the suppliers expense." And then add a clause that says ">3 non-compliance events in under # months will result in the entire PO (10 or 20 units) being returned and all contracts cancelled."
I cannot believe you are getting screwed by a company you choose to do business with and yet you eat cost to ensure they aren't screwing you. Just get a new supplier! Do on-site inspections at their facility. This is nuts.
There were other considerations like the fact we were actually buing it from large reputable company and what happened was that some employees were doing it with no involvement of the company.
The fact is, doing any kind of hardware production in China, you have to be aware Chineese have different value system and you would not be suited doing any business if you throw tantrum at any sign of apparent dishonesty (assuming the company was involved which they could not have been as they have been the ones damaged the most).
If the company does screw you (like replacing components for something cheaper) they typically will not be thinking they are doing anything wrong. They are just testing if you notice and if you do not they will say it makes no difference for you but saves them costs.
The way to work is then verify everything and politely point it out. If you notice they will correct apparent mistake.
All employee actions are company actions. You partnered with a company that can’t control what it’s employees do? No internal audits to make sure their reputation wasn’t being tarnished by a few employees?! Your loss.
I understand the indignation etc etc. And the suggestion to not use these kind of companies anymore.
And that sounds really reasonable, until you realize that pretty much all contract manufacturers in the Far East will source cheaper or off-spec components than those on the BOM if they can get away with it.
One of my friends supplied small widgets for a well known consumer electronics maker. He routinely gets widgets returned to him as defect for inspection, which then inevitably turn out to be clones of his widgets.
The only way to make sure that your product rolls of the product line as expected, is to have people on-site with continuous inspection (and pray that they're not the cousin of somebody who's on the other side.)
If you want the benefit of dirt cheap manufacturing, you need to have a system in place to deal with these practices.
How much do you think you would have to pay them to make it worth their while to not use cheap components (so as not to risk losing you as a customer)? It’s amazing how people never consider using economics.
Edit: But let’s not get away from the matter at hand. The issue wasn’t cheap components, it was full-on credit skimmers installed in yor hardware (IIRC). Should be easier to incentivize the conpany into halting those before they get shipped to you, no?
As somebody else noted: it doesn't have to be the company that does it, but a lower level employee who cuts a deal. So paying the company more wouldn't solve the issue completely.
All of these things are extremely cost sensitive. Your suggestion that people don't consider using economics is simply wrong. If you can manufacture a million pieces for a few cent less per piece, and the only negative is having to paying a bunch of inspectors who are fixed cost, it's an easy choice.
And again, you should make it the company's problem if they can't control their own employees. They can learn to put some controls in place once it starts hurting their bottom line.
> All of these things are extremely cost sensitive.
Say their profit margin is 1%. If you offer to pay them a 1% higher price, you're offering to nearly double their profits on whatever they sell to you.
Admittedly, that only works if you are a sufficiently large portion of their business. If you're a tiny percentage, then it may take a lot more to motivate them if their other customers aren't willing to pay more.
If the solution to the problem involves line checks, independent audits, personnel checks, increased pay, etc., thenyes, throwing money at the problem is a large part of the solution.
You have to pay them as much as it would cost to manufacture in the Western country doing the out-sourcing...so unless the cost of these extra quality checks outweighs the cost of on-shoring you might as well measure angular momentum.
IMO this is not indignation, it's supplier sourcing 101.
> If you want the benefit of dirt cheap manufacturing, you need to have a system in place to deal with these practices.
I will definitely agree that holding suppliers to standards regarding consistent output, unadulterated products, and conducting audits all make production much more expensive.
But if you're a device manufacturer, these sourcing controls are key to shipping quality products.
But make no mistake, this yet another globalized race to the bottom, but one that hits capital (or the end consumer), rather than labor or the environment.
>The fact is, doing any kind of hardware production in China, you have to be aware Chineese have different value system and you would not be suited doing any business if you throw tantrum at any sign of apparent dishonesty
Sounds like the solution is not doing business with them and pushing for a ban on others doing business with them (since this largely has a socialized cost when things go wrong, such as individual people having their credit cards stolen).
I used to work on the server-side stuff for telecom devices. We designed hardware that went into customer homes, but only downloaded the (encrypted) firmware upon home activation as otherwise the Chinese manufacturers would have ripped us off and sold them to telecom companies without our cut.
So, realising they could copy our hardware, but didn't have our software, they responded by trying to hack my servers, multiple times, from the same IP they sent manufacturing data from. A quiet word with their management would stop it, and it'd start again a couple of days later.
These people have no shame, and if we are going to go for lowest cost at all times it is what manufacturers should expect to happen.
OMG, that is so so so so brilliant! I'm going to go tell my boss right now that the suppliers better suite up, otherwise we are going right to their more expensive, less experienced vendor and will delay our product launch for a year....and likely still suffer the same problem. Those Chinese vendors better shape up or we're going to really teach them a lesson by driving ourself out of business right quick!
Just imagine for a second if someone was advocating on HN for the use of a web framework that has knowingly allowed itself to be compromised to steal passwords because it was cheaper to use.
How about a compromise? If any of your customers are a victim of a crime because you continued to use a shady but cheap vendor after seeing them trying to slip past tampered hardware, your company is held fiscally and criminally responsible.
Leaving an abusive relationship tends to be scary, especially if the victim is convinced that the abuser is the least bad. Doesn't mean the thought of ending that relationship should be dismissed out of hand.
You can be as sarcastic as you want, but these stories are absolutely indicative of a much larger problem. It might be beyond the capacity of one company to fix, but in the aggregate they represent a serious political and economic threat and need to be dealt with one way or another. Preferably by literally anyone other than Donald Trump.
Speaking as someone who is _not_ a citizen of the USA and is well aware of his issues, he seems to be first US President who actually appears to have the guts to do something about it. I doubt an establishment politician would ever have imposed tariffs on 200 billion dollars of China imports.
If he stuck to just China I would agree. But by trying to change up numerous different agreements, he lost the goodwill needed to have a collective crackdown on China. Even if he is the first to be willing to pick these sort of battles in a long while, he has a new issue of picking too many battles. Seems like a Goldilocks problem.
The tariffs aren’t the problem, it’s the stupidity of going it alone and alienating allies that could have turned the thumbscrews with you on China to get them to really capitulate.
> some employees were doing it with no involvement of the company
Again, I don't see why that matters. I understand the latter part, but if this is one employee without the knowledge of the company, then pushing it on them forces them to add checks, not you.
>There were other considerations like the fact we were actually buying it from large reputable company and what happened was that some employees were doing it with no involvement of the company.
It's their responsibility to police for that. From the perspective of their business arrangement with you, it doesn't matter whether their left hand or right hand is evil; it's not your problem either way.
Is it really hard to find alternatives? Are they really cheaper than non-Chinese sources when you account for these inspection costs?
the indignation was not that. you are still missing the point.
after the 1st device found, you should have contacted the manufacturer and said that you will start a department that has the capability of opening the device, inspecting and re-sealing in a way that it won't impact any guarantee the factory provides. If they denied this very sensible request, you had proof that it wasn't a isolated employee doing the hack.
Not really familliar with PCI DSS but it might be that the card-readers/terminals aren't PCI-compliant if opened? So not the manufacturer's issue but the customer's.
> Not really familliar with PCI DSS but it might be that the card-readers/terminals aren't PCI-compliant if opened? So not the manufacturer's issue but the customer's.
I think that's the case. The EEV Blog guy did a teardown of and old one once and pointed out the numerous tamper-detection features that would clear the device if opened.
However, if I were the customer here, I'd tell the supplier that from that point forward they need to supply me free extra product with my orders, so I can do my own random destructive testing to look for implants. I order 100, they send me 105 for the price of 100.
PCI DSS allows for "Mitigating Controls" if you need to deviate from specified requirements, provided it is well documented and is equal to or greater in security. Doing teardowns to review circumspect hardware, and applying one's own tamper protection deal (and with accompanying documentation and tracking/logged information) would very likely be sufficient to maintain complaince.
I am reading it as "we don't any any option outside China and we have to live with this". I guess even after adding cost of doing additional of products you buy from China its less than what you would pay to buy it from other place.
Thinking about it more, what you are doing make sense also because tampering can happen with any supplier, irrespective of factory location, given how aggressively states are pushing for spying on each other and their people.
>If the company does screw you (like replacing components for something cheaper) they typically will not be thinking they are doing anything wrong. They are just testing if you notice and if you do not they will say it makes no difference for you but saves them costs.
They don't pass the savings to you so they know very well what they're doing, just playing fool to see if it slips.
> they typically will not be thinking they are doing anything wrong. They are just testing if you notice and if you do not they will say it makes no difference for you but saves them costs.
This is how children behave. Still feels like you’re rewarding bad behavior. You’re enabling them.
I think it’s more that you’ve valued the low per unit price over having a healthy contract. Your vendor is incentivized to make all of their money off of externalities and your company think it’s cheaper to outwit them than to demand QC on their end. Whose brand will be sullied if you miss some of these units and a customer finds the spyware? Not theirs.
And second thought, shouldn’t serial numbers be coming off the line in ascending order? The kind of work they are doing would require taking parts off the line and putting them back later so odd lots of SNs are the ones you need to verify.
You could also be mandating how many boards are allowed or that the SNs go on early in the build process.
This is just how low trust societies function for hundreds if not thousands of years. This is also why families tend to be much stronger in these places. The American idea of a strong family is different from what I'm describing, since it just centers on immediate family. My definition of strong family bonds, implies both closeness and dependency not just on your immediate family, but even on your distant cousins on both sides. It's close if not the same to how Mediterranean and Hispanic cultures view families. I guess you can call it a clan mentality. I'm now wary of countries where families are really strong ie clans. It's not always the case, but it tends to mean the rule of law is weak and corruption is crazy.
When you can't even trust that food is real, and when you have to bribe even low level provincial government employees (can you imagine needing to bribe DMW workers or even police?) - what else is an average person to do except to treat it as the norm in order to survive? Unless you're an elite, the only other option is to leave, and not everyone has that choice. Very little is considered wrong over there, aside from criticizing the powers that be; as long as it helps your clan. It doesn't help when Western companies and governments look the other way, as we see in past news stories and even the comments here.
Not much to add, but want to note that you're spot on. I've encountered this SO many times over the years and this is always the explanation I'm given. At first it just made me indignant, too, and to some degree I guess it still does, but now I just try to let it go.
It's not rewarding bad behavior when you take it into account in the deal. Moralization is useless when you do business with different cultures. When you notice something what would be out of ordinary in your culture, you estimate the cost and find out how locals solve it. Different cultures have different kinks and people living there know to adjust to them. It can be fun to figure out how other cultures function.
From a naive Finnish perspective you could see American business culture as dishonest. People lie all the time and promise things that are false and you need massive amount of legalese or they fuck you over. When they say something, you can't trust it as much as you can trust a Finn (or Swede). American way is to lie to your face in pre-negotiation but then be truthful to the legally binding agreement. Germans or Belgians are more truthful beforehand and they stick to deals better without legal enforcement looming over them, but require more detailed written deals than Finns. Who is right? Should we punish Americans for their behavior or adopt their culture?
Of course not. You must understand that that the level of acceptable exaggeration, overpromise and bullshit and trickery is strictly culture dependent and relative. Someone is dishonest if they "lie this amount above average" in their cultural context. After you make an adjustment, you know how to get the truth.
Chinese manufacturers have the capability to stick to standards, take responsibility and deliver high quality stuff. It's just more dependent on personal relationship. Just placing an order has less trust as default. Either you develop "quanxi" with your suppliers or work trough some third party that understands both worlds. Third party understands what you need and knows how to get it from the Chinese suppliers because they have a relationship.
Chinese pay the cost that comes with their business culture. They can't network as fast as you can in open western cultures with more default trust between strangers.
Altering components to something cheaper is common practice for every company that is willing to exploit the fuck out of their customers. Look at the food industry. In EU there is control, in US thare are sanctions. But it is up to the governments because companies are to opportunistic to change. As long you can (re)sell the junk you get, you're fine, if you can't you upgrade "QA".
Both the EU and US have internal controls on food quality and sanctions.
Contrary to the impression your post gives, the EU has much more "sanctions" against food (esp. imports) than the US does, and is generally taxing much heavier and restricting many more things. Importing essentially any milk product into the EU is only possible on an exception basis. Trump has a LOT more work before extra sanctions in the US will become comparable to the EU ones.
What is often not said is that the EU not only has sanctions on its borders against food products, it actually has internal sanctions and regulations that are purely political in nature.
For example "champagne" in the EU can only be made within a particular region in France. Which leads to funny, in a sad way, consequences:
For your other part of your comment, I am unwilling to give a real answer. Obviously the US checks it's food quality. It's made different choices than the EU, famously the US allows hormone "enhanced" meat, but it does check food quality and safety. There are heavy penalties for breaking them.
Champagne is literally the name of the area where the product comes from. It makes perfect sense that anything that doesn't come from the region Champagne, isn't champagne... Call it sparkling wine, it's fine, but not champagne. And it applies to most local products. Camembert? Comes from a specific town, which is called Camembert. Beaujolais? The name of the province. And the list goes on. It doesn't sound "absolutely ridiculous" to me.
Did you look at the map? It's nearly all things with a place name in the product name. For example, "Welsh Beef". Why is it reasonable for a consumer to buy Welsh Beef and for it to possibly have come from somewhere other than Wales?
It seems to me, more like generalising the concept of trademarks to place names rather than just company names and brand name.
Not all cultures are great. If you are claiming “all cultures are great”, then perhaps “great” has lost any useful meaning.
Discrimination, as the ability to discriminate features to identify useful pattens, is not *-ist; it’s a critical skill that is being sullied by some neo-intellectual BS that passes for “equality” or something.
The ability to identify patterns of behaviour prevalent to a class of suppliers and avoid them, rewarding other supplies — even from the same area of the world — who are better; even “great” perhaps, is a valuable skill...
But perhaps I’m wrong. They’re all great. Carry on.
Give one example of a non-ethnic Chinese person who is considered to be culturally Chinese.
It's the norm in the west to not conflate culture with race. But to think that this is the norm in the rest of the world is to project your value system on others.
I had the same initial gut/emotional reaction, but after a re-read, not _sure_ if it's justified.
If we define "a culture" as set of sociological customs, value systems, expected behaviours and stimuli/response criteria;
and if we define "culture greatness" as such sociological customs, value systems, and behaviours that lead to specified and asserted goals;
than one can reasonably compare and contrast cultures in meeting (or even striving) to these goals, and thus discuss their greatness.
Only to the extent that the relationship between race and culture is causative rather than correlative.
If forced at gunpoint to trust one person in my supply chain, I would instantly choose the Taiwanese-American in San Jose over the mainland-Chinese in Shenzhen, every time. Likewise, I think exactly the same Indian employee will be more productive if physically transplanted to Silicon Valley and surrounded by assimilated Indian-Americans than if set up in an office in Bangalore surrounded at all times by other Indians and Indian businesses. And equally likewise, I have noted a large difference in my own work productivity doing similar tasks under different employers, or even in different work groups under the same employer.
So it isn't even national culture, racial culture, or regional culture, but also corporate culture. Breaking news on the Obvious Channel: social apes found to be heavily influenced by their peers.
Ancestor post was obliquely referring to the oft-reported tendency of mainland-Chinese manufacturers to try to cheat their customers. They have been caught copying protected or trade-secret IP, counterfeiting, substituting cheaper out-of-spec parts, and inserting hidden hardware backdoors. Outside of China, engineering/construction companies tend to import Chinese subcontractors rather than use any local labor. That's all just PRC-Chinese business culture. If you pluck individual employees out of a Chinese firm, and hire them into a multinational based out of another country, they will be perfectly fine, aside from the slightly elevated possibility that PRC intelligence will aggressively recruit them as an asset to steal exploitable business secrets. If you can manage to pull everyone they care about out of the reach of the PRC, they will likely be a great investment for you.
Clearly, the national-political-business culture of mainland PRC China is hostile to businesses that have significant interests outside of China. The government there is racist, controlling, and mercantilist-protectionist, and the businesses with a physical presence have to fit in to that culture in order to exist, along with the traditional system of guanxi social capital. It makes them difficult to work with, but as long as the price is right, it's still worth it to try. You just don't give them access to any secrets, and make sure your QA department is full of no-nonsense hardasses. And as mentioned elsewhere in this topic, that means you may have to measure total mass and moment of inertia around multiple rotational axes in order to find any unauthorized modifications that might expose you to company-bankrupting liability in your business culture. You can't "trust, but verify"; you might just have to check every unit of every shipment.
As for culture culture--wushu martial arts, the action movies including it, silk, noodles, jadeite carvings, fine porcelain, traditional architecture, religions, herbalism, the woo-woo pseudo-religion attached to the medicine traditions, etc.--Chinese culture is indisputably great, especially considering that it's what's left after the PRC government tried to annihilate most of it as part of their revolution.
There are many important and worthwhile discussions that our empirical tools are not able to significantly contribute to.
That's not to say that there is a such thing as single-dimension "greatness" metric. However, outlawing value judgments is going too far in the other direction. Of course, such things can be quite sensitive, and should be done thoughtfully.
It is called business. The company's ONLY duty is to bring profit to its investors and management is legally bound to maximise it. Now, the definition of profit may differ as well as what is profitable and what is not, but the company is legally obligated to work to bring profit and if you throw a good deal because you don't like it you better explain it to your investors.
Whether you like what your supplier does or does not is of no concern. If couple of units can be detected and written off it is treated as cost and you move on to decide if it is still profitable and whether you can get better deal somewhere else.
You let you emotions rule and it just means you are not fit for the job.
"There were other considerations like the fact we were actually buing it from large reputable company and what happened was that some employees were doing it with no involvement of the company."
I fail to see how that has anything to do with it. The company is responsible for the product, full stop. If they can't stop their employees from tampering with the product, that's entirely their fault.
he's saying that his company knowingly allows a percentage of fraud, identity, and financial theft to occur against his customers' customers because there are enough layers of indirection where they (he and his bosses) probably can't be held personally responsible. Enough efforts (fancy tables with big receipts, etc) have been provided for coverage in case of a court case or media blitz but actually using a reputable supplier is clearly too expensive.
I can't believe the parent comment is so brazen. It makes me physically sick.
But it doesn't stop with the cards. ATMs, Tax Fraud, ... almost all businesses within banks have to deal with fraud and there is some threshold that they just allow to happen.
like shoplifting. Most large chains know exactly how much "loss" they experience, and they just raise their prices to cover it. Theft and fraud is a part of life unfortunately...
This problem is not specific to China, it's a problem of going with the lowest priced supplier. Your proposed recourse presumes that the supplier is making enough money on the deal to engage in a contract with teeth... many are not. Also, even if they moved everything to the U.S. they'd likely have different flavors of the same types of issues (i.e. did the NSA embed someone in the company to implant backdoors etc.) Fraud, sabotage and espionage aren't confined to specific countries. When greed drives you to move operations to the absolute cheapest places you can find on Earth, that's not going to be without its own set of risks and problems.
Honestly, to anyone bashing GP, look into your pockets, laptops, watches, cars, TVs, routers, CCs, singing toys, bitcoin mining ASICs. All with sealed black-box chips. Sure we can x-ray a couple of randomly stripped chips, but each one?
Now show me alternatives when most consumer grade electronic parts are fully or partially made in China.
Maybe you and me are missing something here. It seems crazy that somebody would go to these lengths without switching suppliers, there must be an underlying reason that is assumed to be understood by the informed reader. (But isn't understood by me and you it seems.)
The product is different, but the problems are the same.
* They drop changes and problems at the last minute, so you're over a barrel with your customers.
* Relationships take months, maybe years to build. Switching suppliers is a long and costly exercise.
* Often suppliers themselves are in communication, so your attempt to build a new relationship is scuppered by your current (corrupted) supplier before you've got far.
* The new supplier will 100% play the same games.
* Suppliers/factories are often expert at these games - they're not idiots, their bread and butter is manipulating managers and big companies into corners and fleecing them along the way.
* Large suppliers often have strong local/regional government connections. Once you start causing serious problems, they'll have no trouble causing massive, massive problems for your business.
* Are you going to admit to your customers and bosses that your products were faulty and you knew?
It's a poisoned chalice. Companies want low cost manufacturing, and sometimes China is the only place a product can be manufactured competitively. But this kind of behavior is standard for Chinese companies.
> * Are you going to admit to your customers and bosses that your products were faulty and you knew?
If you can't answer that with a yes, maybe you don't have the backbone to work in anything critical.
When you discover a fault in something, particularly a fault that might hurt someone, you have a moral obligation to speak up. To do otherwise is cowardice.
Failure to speak up when we see shit is how stuff like the VW emissions gate happen, and also why security professionals can make a career out of ferreting out your mistakes and engaging in responsible disclosure.
All this tells me is that Made in the USA is more valuable than I once believed.
I think you’re right about the value of Made in America and societies where honesty (even about problems or risks) is valued.
The book I referenced included a number of instances where factories actively undermined individuals so that they had very little choice than to either go along with the situation, or press the ‘nuclear’ button and scupper the entire arrangement, including their own livelihood and career. That’s a tough choice for people to make.
Or: you can expect this kind of shit from pretty much any supplier. Even if you go to a western company, they all source their components from China anyway.
Also it forces their customer's customer to essentially subsidize production cost.
Some amount of financial theft happens to random consumers who don't know where it came from and this stolen money is paid in bribes to employees as a subsidy to their low wage.
As a result this immoral company externalizes aspects of their cost structure to greater society.
I've heard very similar stories from people in the card industry before. I don't think it's made up, unfortunately. Card readers from China are systematically untrustworthy.
Using the technically incorrect term "angular momentum" rather than the more correct term "moment of inertia" made me think the same thing. I would think a person tasked with building such a device would know their physics well enough to use the right term, but I may be wrong.
> We could not measure all possible angular momentums but it was possible to measure one or two that would not be known to the attacker.
You mean moment of inertia, not angular momentum.
You could measure all of them! Given the moments for the three principal axes at any point, you can use the parallel axis theorem to calculate all the rest. In general, there are 10 degrees of freedom: 3 for the position of the center of mass, 3 for the axes, three for the moments, and for the the total mass.
For a nicer way to count and to do the math, you have the inertia tensor at the CM (a 3-dimensional rank-2 symmetric tensor, 6 DoF) plus the location of the CM and the mass.
In any event, this is a cute tampering-detection trick, but I would have started with an X-ray or CT scan.
Another implication of the parallel axis theorem is that the attacker could perfectly mimic every moment of inertia by shaving plastic. They wouldn't have to know which two axes were being tested because there are only three real numbers worth of information in the system to begin with (once center of mass and total mass have been dealt with.) In the whole MOI tensor there are only six free numbers which sounds like a reasonable number of parameters to fix by adding and removing small amounts of material.
What was this company doing in hiring an untrustworthy manufacturer to handle secure devices? That's playing a game you've lost from the start. Not every problem is technical!
> What was this company doing in hiring an untrustworthy manufacturer to handle secure devices? That's playing a game you've lost from the start
You're assuming that there were feasible alternatives; from their comment below:
"wasn't our device. There was a big, reputable company behind the device. We were ordering a number of those and they would be shipped to us directly from China. ...
Also, we were basically locked in due to the magnitude of investment in the software we have developed for the device.
...
Fortunately this only lasted for few months until it was dealt with"
Business lesson to include a 'no China' if the hardware handles anything sensitive, with a large fee for violating it for the middle man servicing the contract.
Another problem is anti-tampering measure is applied before the initial tampering check have been applied. Anyway, that wouldn't solve everything considering that if the chip itself is tampered with, that's undetectable short of an electron microscope analysis and even that wouldn't solve the problem of backdoor in the original chip design.
As you say, that's un-winnable. The only way to really build trust is the capacity to sue your manufacturer to oblivion - I mean the real oblivion (destruction of shareholder value) with criminal charge for the company officers, not the lame single digit percent of a single year of profit with a discount if you settle.
I reckon this could be addressed with contractual guarantees in exchange for doing business in the first place. Make it a contractual requirement that the supplier agree to adjudication in the United States in the case of security related concerns or other tampering with the final product or with the designs. They could still refuse if something happens, but I reckon at that point you can just work to shut them out entirely from the US market.
At this point, I think it's fair to establish an industry watchdog group that works as a clearinghouse for knowing all the American companies sourcing parts from the PRC and which companies they source from. This watchdog organization would be responsible for blacklisting suppliers from the entire US market if they don't agree to adjudicate in the US or if they fail to meet that contractual obligation).
This won't bankrupt them outright, but it will sever them from the US and if this watchdog group expands to aid non-US companies, could help shut them out of the entire global market.
I would also require registration of all top executives of these corporations so that you would make the punishment sticky if they moved to other companies. Overall, you've got to set a high punitive cost to non-compliance and non-cooperation.
You can't lock China out of the market without diplomatic implications. Even assuming you managed to get a purely private boycott of China together without somehow violating anti-trust law, by the time it started making a serious dent in the market the Chinese government would have started making representations to the US State Department. Where it goes from there is anyone's guess.
I honestly did not know about that. I thought that if you move any mass (remove non zero mass and place it somewhere else) there must be at least one axis which you can use to detect the change in moment of inertia.
If the mass had to all move to one other place this would be correct (the center of mass would have to change), but the attackers are able to move the mass to multiple other places.
I enjoyed your comment. But you are looking at the difference in the inertia tensor of an adulterated board from a non-adulterated one. Not the inertia tensor itself. Signal to noise problems with the measurement.
Seriously, why are we still outsourcing chip manufacturing to other countries? Sure it's cheaper, but we sacrifice a lot to have a society of corporate slaves build our tech. Security, core domain knowledge, capability, corporate secrets, patent rewards and enforcement, etc... All of it you throw away the minute you ship your manufacturing out of the country. I've seen enough board printing machines out there to start working on our own. As a country, we need to close this gap, more automation and capability and there will be no need to outsource circuit board printing and manufacturing. We will be much better off.
Canadian steel is considered by this administration to be a national security risk. But Chinese made boards and chips installed in weapons systems and crucial data centers? No problem.
The thing being called a national security risk is a lack of domestic production capability. No one is saying Canadian steel is sabotaged or something.
Yes, obviously- but one of our closest allies shorting us on steel supplies in a time of crisis has never happened, while espionage from China-produced tech is happening right now. Which should be our priority?
I love a good Trump bashing moment as much as the next guy, but this is inaccurate. The DoD has stringent requirements and quality control procedures in place for their chip procurement. Not to say they couldn't be improved, but the DoD has been aware of this threat for a while, and seems to be mitigating the risk fairly well.
Also, as far as I understand the argument, it goes beyond "Canadian steel is a national security risk". A couple of years ago, Mexico was caught laundering $2B of Chinese aluminum to avoid US taxes.
If the source is China, in a hostile period they might stop sending it. But according reportings I have read, the US DoD only uses about 0.5% of steel in the US, and only ~30% of US steel is imported.
Their argument is that you need strong domestic steel industry to build tanks, ships, etc, in case of war. Not saying I'm agreeing with it just pointing out the stated rationale behind the tarrifs.
The concern was the depression in prices was cratering local high-quality smelting capacity.
However, the DOD issued a memo (penned by Mattis iirc) indicating that there was no supply related concerns. The lack of aluminum used to justify the tariffs was in fact just the result of the LME's anti-market-tampering rules creating an incentive for metals traders to stockpile the material in private stores rather than in LME warehouses.
There was a nice article about this yesterday, but I don't have the link.
It doesn't. They claimed that so they could enact the tariffs, otherwise it would be a WTO illegal tariff. I think Canada/others are arguing that it is not a security risk and therefore is indeed an illegal tariff. This is what I remember from some articles. Please correct/elaborate.
Maybe because it undermines fundamental US business interests, and therefore wellbeing? If so it still seems like a stretch to me. Doubtful we'd have no where else to source steel if war or disaster struck.
Counterpoint: even if we ignored the fact that you cannot possibly produce the volumes of chips necessary at the price necessary in your country rather than in "we don't have to acknowledge all the human rights violations" countries, why would you believe this problem goes away if chip manufacturing were done in your own country, rather than another?
The moment the option of taking control of a production line of something _this important_ becomes available, your local specialized organized crime outfits will start to figure out ways to insert themselves into those production lines, learning the ins and outs, and figuring out a way to get something, anything, in there that won't be noticed but will give them a hook into millions of systems.
The law does not prevent crime. It just puts a price on it. While that price is typically too high for individuals, for organizations that have no business registration to revoke, and no CEO to drag to court, it is an entirely trivial cost.
>Counterpoint: even if we ignored the fact that you cannot possibly produce the volumes of chips necessary at the price necessary in your country rather than in "we don't have to acknowledge all the human rights violations" countries, why would you believe this problem goes away if chip manufacturing were done in your own country, rather than another?
It's certainly easier to enforce laws and observe manufacturing processes at domestic factories than it is at factories thousands of miles away in a country that actively encourages IP theft and other wrongdoings, don't you think?
Not really, no. Why do you think it be easier to police? In concrete terms: who do you expect to do the policing? Because this is the kind of work that requires an expert, to catch an expert, messing with a product created by people 100% ignorant of _what_ they're making. Just _that_ they're making them. Where are you going find enough experts to check the work of even a single production line?
Because "thousands of miles away in a country that actively encourages IP theft and other wrongdoings". It makes perfect sense. If you don't think that, try to run any business in another country. It's just obvious that it would be easier to do domestically. You just list other set of problems. Those problems only get compounded if we'are talking about another country.
You can prosecute someone in your home country and even change policies with enough political support. Best of luck doing this in China without getting 're-educated'.
But note that as always: that's too late. By then, the damange has already been done, and the person you take out of the system is simply replaced by anyone else. Card skimmers sure as hell haven't gone away just because we wised up to them: we now just pay A LOT MORE on law enforcement, and all that gets us is law enforcement now at least staying in lockstep with the skimmer's increased sophistication.
Now imagine how much it'll cost to stay in lockstep when the level of sophistication becomes "a single component the size of half a grain of rice".
Not really: I'm letting the practical dictate the process. Catching this kind of inflitration requires experts to scrutinise the work of people who know nothing about what they're making, to catch experts with a much more vested interest in success, and you aren't going to find enough experts to take that job, at a good pay, to check the output of even a single plant. Let alone an entire industry.
If the question is "why don't we move chip production to the US, so this doesn't happen" the answer is "because you might be able to do that for one or two plants. And it'll probably drive up the price by an order of magnitutde, so they'll go out of business soon after. And you sure as hell can't move the entire industry because of prohibitive cost and shortage of manpower to do the bit that you want tacked on that can't be trusted to be done abroad".
It's hard to compete domestically against low prices caused by China's completely different standards for wages and human rights for laborers. Plus, here we don't like the idea of manufacturing industries being subsidized by the government (except in the case of "defense" of course...), while China obviously has no qualms with doing so.
Business culture is different in the US and China. If you tamper a chip in China and go unnoticed you are considered smart & the winner. If you do so here you'll be getting a more negative reaction.
Western manufacturers have a tendency to buy from the cheapest supplier, even if it isn't physically possible to manufacture at that price point without cutting massive corners or making profit off of some externality. However, when Chinese manufacturers build their own brand, you'd be surprised how the business culture suddenly creates a different set of incentives.
Trust alignment of incentives, as opposed to hey-they-promised-an-actually-impossible-pricepoint-on-paper.
I think I saw this on another thread. You're saying if someone bids that they can do something for a price that's not reasonable, it's my fault for believing their mischaracterization? "Fool me once, shame on me."
I much prefer the other mode of operation: "Fool me once, shame on you. Fool me twice, shame on me."
You're conflating a few things there. Whilst companies like to make noises occasionally about saving the planet, doing the right thing, making (your country goes here) great etc, it's just horseshit. They exist to make money. That's it.
Very interesting methods you used to detect the changes! Out of curiosity: was there a reason that taking an xray of the devices was not an option? Industrial/veterinary xray machines can often be had quite cheaply...
This feels cargo cultish. Products drop from the sky. One day they become poisonous. You have no idea how to reproduce them locally. So you come up with hacks to make then less dangerous.
We really need to get back into manufacturing if this is our brave new world.
Get an x-ray machine? They are surprisingly cheap pieces of hardware if you are willing to deal with a small area, low penetration image. Low penetration means no lead, which makes for something that's about as cumbersome as a large bar fridge.
It's mentioned in the article that X-Ray didn't help much: 'Gray or off-white in color, they looked more like signal conditioning couplers, another common motherboard component, than microchips, and so they were unlikely to be detectable without specialized equipment'
This comment is specific to the parent talking about their experiences producing credit card terminals that ended up with PCBs implanted in them. Here it is appropriate.
And as soon as they know you are using X-Ray to test ( I would be surprised if there were no insider information leaking on the OP's end ), they would develop a way so that somehow the X-Ray images looks identical. May be a heat spreader like metal as cover up.
For the folks commenting below that we should bring the manufacturing back to the US, why wouldn't the bad guys just start bribing American workers to insert the attack hardware into devices made here?
It's not like Americans are somehow above being bribed.
The bad actors could be brought to trial in a United States court, which is a level of deterrent not included in offshore manufacturing. If a US manufacturer was found selling tampered chips then the company itself could be held liable. This would create a general pressure to increase onsight security.
There are other benefits to us based manufacturing, but you only brought up the crime aspect so I will leave the other benefits unsaid.
Americans can be bribed to do anything. However, you don't really need to bribe or ask someone to betray their country and fellow citizens when it's done on Chinese soil...
It's possible, but it's much, much, much easier for an American company to hold another American company accountable when something like this happens. Instead of having to go through all those hoops, you sue the pants off the manufacturer.
I think people here are just suggesting that the folks who believe producing things in America would avoid a culture of abject amorality and greed seem naive, at best. Even ignoring the current obvious examples, Americans frequently engage in profitable criminal behavior: https://www.businessinsider.com/mcdonalds-monopoly-game-rigg...
This reminds me of a story about, IIRC, Soviet intelligence personnel determining that a photocopy machine at their consulate in the USA was bugged by the CIA by measuring it's weight and comparing that value to the standard value published by Xerox.
is measuring the moment of inertia not standard? I think any mechanical engineer would suggest it for that kind of problem (disclaimer; I've suggested it before for the same problem)
Why don't you guys consider to expose this by suing the bad manufacturer? I believe this could help other truely honest manufacturers both in and outside China to beat the wrong doers.
The Chinese operation would simply shut down and reopen under a different name. And the credit card companies are always very worried about their brand image, so they are not interested in any negative publicity.
But I don't think it's that good to keep playing the rat-cat game.
> the credit card companies are always very worried about their brand image, so they are not interested in any negative publicity.
Letting the bad guys keep doing what they do while putting it's own customers under shadow, to me, it sounds like the credit card companies are helping the bad guys.
As a Chinese myself and a credit card user, I'm more worried than you guys do, because we are more likely to receive that kind of treatment (compromised credit card and computer chips etc). I really hope somebody can teach those bad doers a unforgettable lesson.
Huh? This would be between you and the Chinese manufacturer. If you found s tampered unit, you don’t pay them. How should that make you go out of business?
They need the units from somewhere, with a deadline determined by their own runway, and presumably only some of the Chinese ones were compromised. Sure, they could avoid paying, but then they have to spend months getting the production run going somewhere else .. which might also be compromised.
I do not for one minute believe, that china, whose production floors are spotless, watched like a hawk and every product tested, did not know this was being added. They would have been able to tell themselves by weight. Yes, product weight is used for shipping manifests too.
This is obfuscation of the fact, that china as a state actor has perpetrated this crime against our country period. full stop.
What good is angular momentum when the producer can have fluctuations in its supply chain? Yes you can see when devices are not the same, but what if that happens all the time, legitimately?
Gosh, I would love to read everything about this. I know there are some videos about anti-tampering card readers on youtube, but not on the feedback race between hackers and security.
Maybe a setup to measure inertia tensor and center of mass (in that setup's axes) will be easier and I think it's what your call "measuring all angular momentums".
Was the secret service contacted? Card skimmers are a big no no. Family friend works for Dept of weights and measures and finds skimmers all the time on gas pumps. Scary stuff.
Thanks for sharing this story, and I hope you aren't put off by the huge thread of people second-guessing your competence. We need more of your kind of story.
x-ray would have been easier, as others have said.
but you're talking about the addition of an entire board! probably on the order of 10% of the size of the main boards.
in this article, perhaps dumbed down or altered, they are talking about the addition of a single, tiny chip, too small to even be an MCU let alone have wireless capability (which BTW requires an antenna).
The main lesson I learned from my Dad’s employer (twice) was always have two vendors. You can play them off of each other. When I did contract work I saw how powerful getting out or putting for away the checkbook can be. Large vendors ignore you if the checkbook isn’t moving.
I say “twice” because they were also the biggest employer in town. I got out of there. Slim pickins for career opportunities that didn’t revolve around BigCo.
Sorry, but this doesn't sound as true or there is huge mistakes done in choosing supply chain for such sensitive matter.
How come company keep ordering devices from some unverified sources from China, and after hitting a wall keep doing same?
How do you accept shipment of such devices without randomly opening and inspecting sample(yes losing all data, but electronic inspection can be done).
How you didn't investigate that with Visa/Mastercard? Whoever does that, he will lose his payment terminal certification after such incident, because they will track them down by IC serials very quickly.
What if vendor changed power supply board or even components type on it, and your momentum or weight test will make false positive?
Unless... your employer or you buy single devices, on demand, from some shady aliexpress seller. But then, it is plain suicide.
While it may sound sensational this was more of an operational issue, really. We were told by Visa and Mastercard that it is not even a question if we are going to be targeted. If you work in payment card industry you are constantly being attacked and the only way is to make it part of the process to deal with those things. Our network was hacked but what was important was tight, almost mathematical processes around protecting very specific material like credit card data and PINs.
For example, PINs are only ever being in unencrypted form inside of Hardware Security Modules and only for the purpose of being encrypted with Visa/Mastercard exchange keys. The process was designed so that nobody has enough access to ever get enough cryptographic material to be able to decrypt anything, at least two or three people would have to collude to do anything.
It also happens that we put all our resources in investment in software for the platform locking ourselves in. It would be rash decision to change the platform and it would probably kill our company. Also we (correctly) gambled that it would be dealt with quickly.
Look, there are enough supply chain problems with counterfeits already, you don’t want to start thinking about malicious implants. Just google for it, it’s massive
Presumably the organisations responsible for hard coding backdoors in chip designs know how to test to confirm their presence.
Presumably some adversarial nation-states have moles inside these organisations > know how to remove them prior to fab.
Presumably these adversaries export genuine chips to their adversaries, thereby tricking them in to thinking the backdoors made it through the fab process, and only use chips that have the backdoors removed in their own critical infrastructure.
Presumably.
I’ve always had this fantasy of being a hextuple agent involved in this type of deep espionage.
Is there an article that describes a bit more in detail what the chips actually did (or were capable of doing)? They only say "the microchip altered the operating system’s core so it could accept modifications.", which I might interpret as circumventing signature checks to allow installing modified firmware on the systems? But how does the chip connect to the network and how does it receive commands?
That said, it's pretty scary that you can hide so much malicious functionality in such a small device, makes me wonder what might be hidden in my Lenovo. In any case it speaks highly of the auditing firm that they were able to locate this. I wonder if they performed an x-ray analysis of the board, as given the size of these chips it should be possible to embed such devices in one of the internal layers of the board as well, making them essentially invisible to optical inspection.
SuperMicro hardware has very extensive IPMI integration into the mothebroard, which amongst other things can take over and inject frames into the network interface, emulate a VGA device, talk to the CPUs serial lines directly, flash firmware, control the state of a number of physical devices- and this is what it supports just from the web interface it presents by default with the password "ADMIN:ADMIN". My money, based on experience attempting to harden their devices, is that any modification were injected into the IPMI hardware where most of this was already supported.
This stuff ends up being extremely difficult to disable. The naive approach would be to not connect to the dedicated NIC that's indicated on the back and in the instruction manual, but if you do this it masquerades onto the main NIC invisibly to the OS and DHCPs on its own to open up an administration port, web interface, and some assorted call homes. You have to explicitly tell it to use the non-connected port, change credentials, and modify it so that it is not accessible within operating system as well. Hopefully while the machine is offline to prevented any automated scanning finding it within your network.
The number of times I'd end up nmaping our local networks and being able to remotely access production hardware with an interface that allowed me to reach this interface was maddening. The system is basically designed to be as insecure as possible by default, and allow for the maximum possible persistent threats with BIOS flashing, IPMI flashing, and other completely nu-authenticated avenues exposed. The course of action was always just to write off the hardware and bin it, because god knows what impact you could actually have using that interface.
> The organizations behind the new project each have already made substantial contributions to creating open source baseboard management controller (BMC) firmware. Now, working together, they will define the vision for a standard stack that can be used across systems and computing environments.
LinuxBoot and Open Compute OSF are working on open-source server firmware that can be measured on every boot and validated against hardware root of trust keys controlled by the server owner instead of the server OEM, https://www.platformsecuritysummit.com/2018/speaker/hudson/
Wouldn't help. The BMC hardware has direct serial access to CPUs and other hardware in the machine. Communication is unencrypted. A hardware modification attack wouldn't touch the firmware at all and could still compromise IPMI functionality.
Even now, supply chain hardware modification attacks remind people of fiction. However, the number of people known to be affected by buggy BMC firmware is orders of magnitude larger, as described two comments upthread:
> The system is basically designed to be as insecure as possible by default, and allow for the maximum possible persistent threats with BIOS flashing, IPMI flashing, and other completely un-authenticated avenues exposed.
SuperMicro hardware in particular always struck me as such. Asking users to pay a license fee to be able to update the BIOS on their devices (after paying tens of thousands for the hardware itself) is a kick in the teeth.
To be clear, this is only for the ability to update the BIOS over the BMC interface, not for BIOS updates in general. (unlike some other vendors, where you need a support contract to be even able to download the updates)
At least on some boards you can boot the USB drive image containing the BIOS updater through the BMC and do a remote update that way.
Supermicro doesn't publish changelogs for their BIOSes. Their disclaimer on their BIOS download page says, "don't update the BIOS unless you have to". It's a ridiculous stance for a server company. You buy Supermicro over Dell for the price. That's pretty much it.
For the models I was looking at a number of years ago, the options were through IPMI or via a USB Floppy Drive. Perhaps my memory is failing me here, but I seem to remember being quite enraged at the prospect and would have done a lot not to have to pay the license fee on principle.
This is only really valid for protocols or products designed before the Morris worm of 1988. Anything designed beyond 2000 has no excuse for not thinking about internet security.
Well, IPMI isn't supposed to be exposed to the internet. Best practices have you running your BMC's on a completely separate, highly locked down administrative network.
I did a security audit on a company that had a setup similar to this.
After popping an internet facing web server, I was able to compromise the IPMI system and use the management network to bounce around to any server in the enterprise completely bypassing all the firewalls and segmentation on the production network.
Best practice would be for the BMC to not have access to the regular network ports when it has a dedicated network jack. All the ones I've looked at don't have any kind of physical interlock or switch, it's a software interlock.
Now even a hardware interlock could be subverted, but that's harder than sticking code in the bmc firmware, which does tend to get updated during the life of a server.
Well yes, typically you'd 1) configure IPMI to use the dedicated port, 2) put those ports into a VLAN with no outgoing internet access. But since this is BMC, what's stopping it from just using your management or production port to fire off its secret phone-homes and whatnot?
The problem with an IPMI BMC is that if you have malware that roots the OS from the BMC that in turn roots the BMC to reinstall itself from the OS, then you can never actually get rid of it. I actually opened a bug with a major clustering vendor in about 2012 because of this. Their response was a docfix.
If people are interested in digging into how IPMI works every SuperMicro board I've ever seen uses some variety of ASPEED Baseboard Management Controller (BMC). Facebook uses the same chip in their open hardware projects.
> The naive approach would be to not connect to the dedicated NIC that's indicated on the back and in the instruction manual, but if you do this it masquerades onto the main NIC
cool, thanks for that info.
> just to write off the hardware
maybe you could just standup the mgmt network but blackhole route it at each switch port. The mgmt NIC thinks it's working properly but it can't talk to anyone nor can anyone talk to it.
No. SuperMicro make some models which appear to be for government application which lack the ASIC completely, they were available on NewEgg with no further description of purpose at one point.
No, it boots prior/separate to the board itself. It's basically a mini-PC embedded in the board that has it's own CPU/Memory and tentacles attached to everything in the mainboard.
No, the controller runs it's own totally separate OS, has connections to most/all the buses and is able to emulate devices whether you have drivers installed or not.
But they were capable of doing two very important things: telling the device to communicate with one of several anonymous computers elsewhere on the internet that were loaded with more complex code; and preparing the device’s operating system to accept this new code. The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off.
To me, that makes it sound like they could download from a remote host and inject code and do literally anything.
If this is indeed the case, I'm surprised someone didn't catch something earlier when the device was calling "home" over their network. I'm wondering if China stole BlackRidge First Packet Authentication tech [1] to keep things dark. BlackRidge is... "involved" in IC and defense projects.
5. When a server was installed and
switched on, the microchip altered
the operating system’s core so it
could accept modifications. The chip
could also contact computers controlled
by the attackers in search of further
instructions and code.
So, in typical vulnerability/payload/exploit fashion, the board's bus is vulnerable by default, because the chip pierces all the usual lines of defense protecting against network and operator I/O. It carries a payload intended to target very common features used everywhere commodity servers are used, one that likely listens for DMA traffic on the bus, and alters the signal stream, by escaping upon the occurrence of a magic sequence, and inserting its own signal, before resuming the authentic stream in flight.
The payload could be pretty small, since the server boards are likely using OS packages that match the chipset. This limits the software to a small set of well known targets, Linux, Windows, Apple. Target their kernels, and you only have to snip out a small chunk of bytes, and splice your own pre-defined package in. Splice in a miniature runtime, that operates a turing complete set of operations, and open up a listener that waits for network access, and now, your payload can enable arbitrary code execution, irrespective of permissions.
Now, to exploit, the payload needs to time the opportunity to splice itself onto the disk correctly. If certain well-known chunks of code will always exist in each given operating system, then with every disk access event one just needs to wait for the inevitable moment those magic system-specific bytes travel over the bus, in order to replace the known bytes with the poisoned modification. Events might target when the bytes are originally installed with the OS, or every time the OS reads those known bytes back into live memory, from any source.
The total payload package could probably fit inside a couple of megabytes, pack on a few more for the "listen & splice" part of the attack to round out the entire mass, and all we know how much data an SD card can fit into say... five grains of rice?
It's not particularly magical, there's consumer chips around which are not a whole lot bigger (though obviously in a more standard package). You don't get a lot of resources, but you don't really need it if all the other frameworks are in place in other software. If this sort of thing is something you can buy on Mouser for a few cents, the espionage grade material is probably an order or magnitude more higher quality.
For example, looking for ELF or Portable Executable headers, as a crude estimate to determine attack opportunities. In this case, the magic numbers would probably be more selective and sophisticated, but still have an aspect of hard-coded values, since we're talking custom silicon.
This sounds like speculation. I'm quite capable of coming up with my own unfounded speculation, but there is a real report out there with the actual details in that really needs to be made public, legally or otherwise. There ought to be a CVE about this. Where is it?
That would make a lot of sense and would give the attacker a way to interface with all of the other hardware (network, disk etc.). Do you have a source for this information?
I looked up supermicro blade motherboards, and saw that the chip was right near the IPMI chip's line to spi flash.
And prior to that, there were already persistent rumors in the Chinese interney of certain Chinese mobos sending "weird garbage on ICMP," and "BMCs that somehow boot and work with their flash memory soldered off"
Remembering that, I might even suggest that this is not a modchip that does something with signal on the go, but just a very tiny flash chip that has the modded firmware.
Going further from that, to pack, say, 16 mB on a sandgrain sized chip, the densities need to be like that of best flash chips out there, which also means that they have access to last gen flash fab.
The recovery overrides the primary if detected by default.
The place they put their "filter cap" is right on top the empty TSOP8 pad for the recovery flash. And they probably ordered the factory to sneak the traces just a little bit more, or put hidden vias under it, or simply had somebody very dexterous to solder it to pads with hair thin wires.
This is the most plausible theory I've read in this thread. Assuming the image in the article is a stock image (there isn't yet a clear image of a definitely compromised board), then the added part could simply be another TSOP8 Flash part. This implies the firmware to the AST2400 is unsigned (which it appears to be, as there's coreboot options for it).
That makes the whole thing gloriously simple.
A part "stuck on" afterwards is obvious. A part fitted into a no-fit footprint after optical inspection is not, it looks exactly as if it was meant to be there.
Well, it means that provision for the second flash was already there, and PLA simply exploited that fact that Aspeed chips are virtually omnipresent in higher end servers.
It also means that the extend of intervention into board design was minimal, and that a trivial automatic xray would not have picked it up. And as implied in the article, later they buried the bug to beat the AOI, if it was done higher upstream.
So, they would've been screwed even if they were doing board testing outside of China.
That's a clever trick.
But the sole fact that the chip has "to phone home" makes detection trivial, and puts the usefulness of the method to nil - anybody sees the router blink when it shouldn't and your bug's cover is blown.
There really, really needs to be. If this story is real and things have been known compromised since 2015, it’s unacceptable that this information isn’t shared more widely. The “we’re going to follow them” idea doesn’t hold water over 3 years, countless small vendors using Supermicro and not being well-connected enough to know (like Apple or Amazon) about it.
Hm, but DMA messages get distributed over a parallel bus and this chip seems to employ a serial interface, so I would assume that it's not directly connected to anything that requires high throughput (i.e. memory, disk and peripheral access).
They attacked the Base Management Controller. There's an article by Bruce Schneier from 2013 warning about exactly this attack. Quoting:
"Basically, it's a perfect spying platform. You can't control it. You can't patch it. It can completely control your computer's hardware and software. And its purpose is remote monitoring. At the very least, we need to be able to look into these devices and see what's running on them."
Well, you cannot patch the vast majority of the software in your computer (assuming you are like the vast majority of users using proprietary crap for everything). That does not mean it is all unpatchable. If supermicro care, they could release a BMC update, for example.
"17 people confirmed the manipulation of Supermicro’s hardware and other elements of the attacks. The sources were granted anonymity because of the sensitive, and in some cases classified, nature of the information."
So not a single source. Might as well be going to war over invisible weapons of mass destruction. Also, Apple and Amazon both has said they do not agree with these claims. So far this is nothing more than propaganda.
so you prefer to believe official corporate statements saying they didn’t have security leaks, rather than a news agency with 17 independent sources confirming they had ??
17 independent sources doesn't mean anything. Again, this is the same as the sources that said their were weapons of mass destruction in Iraq. Companies and government use propaganda all the time, so until you have more information confirming these claims, then it is best to hold off before trying to say that Apple and Amazon denying the claims str proof that they are real.
Def possible they are lying. There are a lot of companies competing with white box manufacturers. If you can't name a single source, and you have two major companies saying these claims aren't accurate... then maybe you should hold off before making judgments.
"Two of Elemental’s biggest early clients were the Mormon church, which used the technology to beam sermons to congregations around the world, and the adult film industry, which did not."
To be fair, removing the last three words could make it sound like the Mormon church was beaming sermons to the adult film industry which would probably be even worse...
Interesting about how some of the trojan chips were hidden in PCB substrate layers to avoid optical detection. For a much stealthier approach again, it was shown in 2013 that slighting changing the dopant mask for a few gates on Intel Ivy Bridge chips could render RNGs insecure[1][2]. Mask inspection systems[3] are used to detect mask manufacturing defects, but the question then is, do those inspection systems use Super Micro motherboards?
We need to get the fuck out of China. It is becoming less credible to throw our hands up and say "China has all the silicon manufacturing, guess we have to put up with it!" - this is national security, both directly via hardware in the DoD and through our economic stability.
Saying "Well the Chinese companies are different" or "It's just rogue employees" or "We just have to accept it" is not good enough.
We need a little bit more economic nationalism and realize that this shit matters. How is it so shocking? How did a Canadian security firm find something before the CIA/DoD? How is this not a NatSec program to be inspecting this stuff through shell company orders and honeypots, like we have on the Internet?
> It is becoming less credible to throw our hands up and say "China has all the silicon manufacturing, guess we have to put up with it!"
The thing is, China doesn't. Most expensive chips are fabricated outside of China in nearby countries and shipped there for final assembly.
What goes on in China is usually human assembly. And that's what seems to have happened here -- in the process of putting chips on motherboards, someone added some bonus chips. It seems like it would be hard to do that without dedicated traces / solder points.
Why does that matter at all? How do you know the CIA didn't use Chinese spies to do this? Although from a different agency, the motto "Nothing Is Beyond Our Reach" is telling.
So the chip shown in the article looks like a typical SMD balun, it is a type of transformer used to adapt impedance between two transmission line. It’s designed to replace a series a lumped element (capacitor, inductors, resistors) normally used for impedance adaptation (in a T or Pi network). The most common used for the device is directly between an antenna an a RF front-end to serve as an antenna tuner.
Technically you could embed and power an RF front-end inside a “flavored” balun to intercept or alter any communication passing through that front-end or even use the antenna to communicate during the down time. So this literally would hack Wifi / Bluetooth at low level and inject code and at the same time create a mesh network of malicious devices to relay information. Welcome to IoT Cyberwarfare.
But this clever hack is probably not limited to RF and is likely to also be embedded in transformers used for isolating Ethernet lines. Common mode chokes (some SMD chokes also look like the chip they are showing) or even some integrated ESD protection solution would be an ideal target as they are inserted in series of the signal.
"But this clever hack is probably not limited to RF and is likely to also be embedded in transformers used for isolating Ethernet lines."
I was thinking along the same lines: a balun could harvest energy and use it for other purposes, but it should either store it for later use using maybe a tiny supercapacitor inside, or inject it into the data stream on the fly not unlike RFID dongles do. Or maybe just have enough power and memory to store a few hundred bytes and alter a few fields here and there, so that for example a network frame coming from a compromised machine can have its source field rewritten as it came from a trusted source.
In theory small pull up resistors also could be swapped with lookalike malicious parts capable of tampering with i2c traffic between boards peripherals so that devices can be activated/deactivated no matter what the CPU tells them.
We're going straight to the point when there won't be a single device in the world, from toasters to supercomputers through top brand network gear, that can claim and guarantee to be secure; inserting malicious hardware and firmware is getting just too easy for those with the necessary knowledge and resources. A technically interesting and scary scenario.
I don't know enough about this, but isn't the article saying that the appearance is deceptive:
The chips on Elemental servers were designed to be as inconspicuous as possible, according to one person who saw a detailed report prepared for Amazon by its third-party security contractor, as well as a second person who saw digital photos and X-ray images of the chips incorporated into a later report prepared by Amazon’s security team. Gray or off-white in color, they looked more like signal conditioning couplers, another common motherboard component, than microchips, and so they were unlikely to be detectable without specialized equipment.
If the photo in the article is real you wouldnt be able to identify this component as compromised "just" by visual (even xray augmented) inspection. TVS Diode Array looks the same from the outside, whats more its build in same way with silicon die embedded in tis structure. Other than signal analysis it would take decapping every single component of a motherboard to find this implant.
The article specifically says that the attack goes after the Baseboard Management Controller (BMC). In Supermicro equipment this takes the form of a separate SOC made by ASPEED that runs it's own operating system entirely separate from the main CPU but with access into the main system through various mechanisms. Facebook uses the same ASPEED chips in their open hardware projects.
Yes, TVS Diode Array (image google to see they look just like in the article) for ESD protection is the obvious candidate, inserted on serial communication lines - perfect spot for signal interception/injection.
It seems to me that embedding an RF front-end, back-end, and signal manipulation logic in a chip of the size shown in the article would be extraordinarily difficult. Assuming that feat was achievable, it would likely result in serious performance hits on that transmission line right? Signal latency would probably be a dead giveaway for something like EEPROM reads being manipulated.
you would be extremely hard pressed to fit that amount of logic (and code) into that small of a package. and to operate intelligently, as you suggest, it needs a CPU clock, not available to it from where it would sit. And it would need to do signal analysis in real time, with no power and no clock.
Honestly, I'm too ignorant to judge the rest, I just think you weren't accounting for (1) you don't need a clock signal for fancy logic and (2) you can fit a lot of logic in a cubic mm. The other points may rule this out, I don't know. Note that the async CPUs I've read about tend to be lower power than clocked ones.
It's been a few years I've given up on the idea of privacy with technology.
The number of security flaws that get discovered daily is only the tip of the iceberg.
I'm pretty sure some governments (or organizations) have had backdoors, be they hardware or software, in place for more than 20 years. We simply don't know about it yet (and probably never will). Would that actually be that far-fetched? I think not sadly.
Even the Intel Spectre et Meltdown fiascos are a sign that we have no idea how to actually secure this stuff. And that's normal, the very definition of IT security is that nothing can be secure.
Take the whole antiquated concept of processor rings for instance, we are adding a new level every other year now it feels like...
I find it way more interesting (even if it is ultimately "worse") to adapt to the mentality that "nothing is secure" than "let's try and make it secure", which as stated is in itself a fallacy...
The more I understand software the less I trust it (given the current state of engineering practices). Meanwhile all my friends/family are scrambling to install all the latest new "smart home" gadgets and I just look like a paranoid kook trying to talk them out of it.
This. I've even had an in-law say, "My brother works for the Defense Intelligence Agency, and he uses smart devices in his home, so they must be safe!", with no consideration that tech may not be his specialty, or he doesn't follow the daily IoT fiascos, or maybe he just thinks he won't get hacked. Dunno. Meanwhile, my year-old thermostat still wants me to connect it to wi-fi, and that will never happen.
> It's been a few years I've given up on the idea of privacy with technology.
The problem with tech is that it's modularity and dependency on other people/tools/hardware/etc ultimately requires trust.
> in place for more than 20 years.
Far longer than 20 years. The idea of trust, privacy and security has been discussed for a long time. The complexity of security on just on one specific technology ( compilers ) was discussed in the early 1980s by Ken Thompson.
Compilers are just one part of a complex ecosystem. Now imagine having to check ABI or the physical chips/hardware themselves. Where you require sophisticated hardware.
I think the root of the problem is backwards compatibility and the fact that initially all these components were not designed with security in mind. So we're adding more insecure components in spirit of "move fast, break things" than fixing the debt incurred.
Yeah, I mean that's what I would say too if some government intelligence agency told me I cannot say a word about this and have to deny it vehemently! haha :)
There's definitely something afoot. Bloomberg probably wouldn't have gone forward with just an in-depth piece referencing so many major tech companies unless it had substance. But generally even when legally compelled, companies tend to prefer silence or curt denials over lengthy detailed contrary pieces.
Is there a federal investigation going on into some sort of sabotage by the Chinese government? Possibly, and if so, there's a lot of reason these companies could even be willingly participating in covering it up. All of them benefit from the investigation being uninterrupted, and nobody needs the bad press.
Could the story be, in fact, fabricated? Sure. Russian trolls have seen to sowing discord in far more than just our election. They stoked drama about The Last Jedi even, as recently claimed. Is it possible they're seeing if they can get a major news outlet to publish something with absolutely no basis in fact? Maybe.
There's a lot of interesting possible angles here, but with the number of large public companies involved, this morning is probably the start of this drama, not the end of it.
I think it is a win-win for Apple and Amazon to spin it this way. They get to deny that they were compromised, and if that gets disproved they can easily say they were cooperating with national security operations and get away without any damage.
What is more telling for me is that Apple has cut off all contracts with the company involved and will have nothing more to do with them - despite their denial.
If they wanted to cut ties for a different reason, execs could have just said "find a problem with some supermicro boards, no matter how much it costs to do so, and never repeat this conversation". They are complicated human-designed and -made artifacts; of course some problem exists. This is known in other situations as "parallel construction".
hard to believe the companies are knowingly lying.
If you're working with the feds on a national security incident, and the PR department comes round because some reporters were asking about it, the correct answer is "I don't know what you're talking about."
Of course. Apple/Amazon can be coerced into denying this (via government requests or otherwise). Bloomberg can't, nor is there any advantage for them to publish false information.
> nor is there any advantage for them to publish false information.
of course there is. It's called ad revenue and page views, both of which they're raking in today with these allegations. This story is all over the place. I don't trust Bloomberg with anything Apple at all and haven't for years now.
Yes, this too. Some of the Bloomberg and WSJ "leaks" feel like their primary motive has been subtle stock manipulation. In particular, the "demand is slowing for $PRODUCT" articles that we continue to see.
However Apple or Amazon didn't back up their claims. If Bloomberg published an article of this magnitude which could be demonstrably disproved, it'd kill Newsweek's reputation.
More importantly, think of what would be Bloomberg's reasons to publish a false story versus Amazon and Apple's reason to deny a true story.
Even if a national security letter wouldn't compel Apple or Amazon to lie, couldn't it compel the security team not to inform the executives or lawyers? Or at least this is an argument that's been going on for a while, can't remember if that issue has been settled.
Fascinating. A company I used to work for dismissed this sort of situation as a problem and ended up using SuperMicro boards in a security crucial product. Their hardware has always been notably very crappy, with the IPMI interface defaulting to world-open unsafe parameters, but I'd not expected it to be this cleverly hardware backdoored. It's possible to neuter Intel ME, but that's only a small comfort with these motherboards.
>Amazon’s security team conducted its own investigation into AWS’s Beijing facilities and found altered motherboards there as well, including more sophisticated designs than they’d previously encountered.
>...the malicious chips were thin enough that they’d been embedded between the layers of fiberglass onto which the other components were attached
>...that generation of chips was smaller than a sharpened pencil tip, the person says.
I hope this corrects the mistaken believe that China can't home grow sophisticated tech.
This sounds strange but it's commonplace to make parts like this. Some DCDC converter modules have silicon dies embedded into their PCBs to save physical footprint, most notably there's a family of Texas Instruments ones which use this technique.
> I hope this corrects the mistaken believe that China can't home grow sophisticated tech.
There's nothing particularly sophisticated about what they did, especially given what China has access to as one of the central hubs of tech manufacturing. There are two dozen nations (or more) that could do this from a strictly technical standpoint (few have the kind of required supply chain access to pull it off at scale in actuality). It's the audacity that is primarily impressive. China is encouraging all of their richest trading partners to further isolate them when it comes to supply chain and advanced tech. They're confirming for the 107th time what everyone already believes.
One question though. Would a politically correct, by-the-book US president have had the balls to sanction China? Considering such sanctions could affect the US economy.
The goal, explicitly stated and often acted on in many contexts, is to disrupt international order and cooperation (including trade), and promote nationalistic competition by all countries. Recently his UN speech, for example, advocated it.
Given that there’s video from 4 decades in which Trump advocates for tariffs, during which time he wasn’t a politician, it would seem that they are something he believes strongly in.
IC fabrication and packaged component assembly are very different things. China doesn't have anything near this level of sophistication, at least not without any outside help.
This seems either incredibly suspicious to me or indicates that Apple is way more protective of security and distrustful of the USGov than I would have thought: “Because Apple didn’t, according to a U.S. official, provide government investigators with access to its facilities or the tampered hardware, the extent of the attack there remained outside their view.”
Why wouldn’t the FBI be able to compel Apple to turn over such key evidence in such an important investigation? Or why wouldn't Apple be willing to do it even without compulsion?
Apple seems cagey about its reputation with the Chinese government. Their balls are directly in the Chinese government's hands, they don't want to do anything that might make the government upset.
Beyond having their manufacturing there, it's also the biggest untapped market in the world.
I hate leaving a brief unsubstantiated comment, but this is the new Cold War and I think it needs to be acknowledged as such and brought out of the dark.
I don’t want my intelligence agencies fighting a war we aren’t even publically acknowledging and subjecting to public congressional oversight. I don’t trust Congress, but I trust the NSA/CIA far, far less.
The economic sanctions against China are disproportionality minuscule if they are meant as a response to this level of attack.
If this is true, it should be proven in a court of law and trade sanctions should be a positive integer trillions of dollars.
This is what Bloomberg's article refers to as "unrelated reasons" for Apple cutting ties with Supermicro in 2016.
> Three senior insiders at Apple say that in the summer of 2015, it, too, found malicious chips on Supermicro motherboards. Apple severed ties with Supermicro the following year, for what it described as unrelated reasons.
And then as an "unrelated and relatively minor security incident" later on.
> One country in particular has an advantage executing this kind of attack: China, which by some estimates makes 75 percent of the world’s mobile phones and 90 percent of its PCs.
Intel and AMD are both USA based companies.
Is it conceivable their processors contain backdoors in a similar vein?
I'm pretty certain they do. AMD's PSP and Intel's ME are really shady and are still enabled in every CPU sold today (and full of disclosed and undisclosed flaws for that matter).
Not to mention that these back doors can be easily disabled, e.g intels high assurance mode, but normal "owners" are specifically not allowed to disable them.
No, it is totally inconceivable AMD and Intel CPUs are backdoored this way. Inserting a microcontroller somewhere on the Ethernet traces and then using IPMI is not as sophisticated as this article wants to describe. Sophistication is necessary from the payload to be stealthy but not the hardware.
There are at least two problems with China messing with CPUs: a) they are not made there. TSMC is Taiwan, the Asian parts of Global Foundries are in Singapore, Intel is manufacturing CPUs mostly in the USA with some in Ireland and Israel, only 3DNAND / 3DXPoint ICs are made in China. b) the hardware knowledge to change here, well, think of it, the microcontroller on the motherboard is on the few millimetres scale, the parts here are on the few ten nanometres scale so as a rough but not unjust quantifications would say it's 100 000 times harder.
Whether the platforms are hackable / hacked, that's harder to say thanks Intel ME and such but the CPUs aren't.
I don't think the parent was talking about this specific type of attacks but other types of hardware backdoors. I think the answer to that is very obviously yes, and given what we know about intelligence agencies I'm even willing to go as far as saying that it is likely (or at least, if you have reasons to be worried about Uncle Sam getting to your stuff you should consider it a very real possibility).
Modern ASICs are so complex that I'm sure that sneaking a tiny backdoor into the behemoth that's a modern CPU or embedded SoC would be almost trivial. They'd also have great plausible deniability in case they're found, if something like SPECTRE was an intentional backdoor how would you ever prove it for instance?
> Modern ASICs are so complex that I'm sure that sneaking a tiny backdoor into the behemoth that's a modern CPU or embedded SoC would be almost trivial.
I suspect putting in a backdoor would be difficult because they are complex. Wouldn’t it be far too easy for the backdoor to inadvertently cause reliability or performance issues? And the bug would have to be useful enough to warrant potentially destroying the semiconductor business of a nation, not just some difficult to trigger side channel.
Welcome to HN. You’ll find that kind of negative comment which doesn’t really add to the discussion isn’t valued here.
It wasn’t wild nor uneducated; I have some experience with FPGAs and the challenges of complex circuit design - regardless, I was posing a question rather than stating fact.
Do you disagree with what I suggested? Educate me, correct my speculation.
> No, it is totally inconceivable AMD and Intel CPUs are backdoored this way.
That’s a fairly strong statement.
It could be argued it is conceivable as evidenced by the fact we are here talking about it.
Also, I’ve read that it’s impossible for any one person to fully comprehend the circuit diagram of a modern CPU. In light of that it seems conceviable any number of wayward circuits could be hidden.
can you be sure that fabs haven't been infiltrated and masks changed between design and production in any factory, be it tsmc, samsung, glofo or intel?
Presumably the organisations responsible for hard coding backdoors in chip designs know how to test to confirm their presence.
Presumably some adversarial nation-states have moles inside these organisations > know how to remove them prior to fab.
Presumably these adversaries export genuine chips to their adversaries, thereby tricking them in to thinking the backdoors made it through the fab process, and only use chips that have the backdoors removed in their own critical infrastructure.
Presumably.
I’ve always had this fantasy of being a hextuple agent involved in this type of deep espionage.
It makes me wonder how compromised the prosumer/enthusiast motherboards are. All of my important home systems are homebuilds using off-the-shelf motherboards purchased in person, and lack any IPMI access. Sure there's AMD's PSP (grumble, grumble) in my Ryzen box, but I can't help wonder what might be lurking on the motherboards, or for that matter in my network switch, router, access point, etc.
I've sometimes thought about picking up some type of used rackmount server, but hacks like this give me cold feet - aside from the usual issue of 1U/2U boxes sounding like jets taking off.
The denials by Amazon, Apple, Supermicro and the Chinese Ministry of Foreign Affairs [1] are relatively pro forma, both directed by respective nation states involved in this matter. One of the reporters interviewed on Bloomberg noted Amazon and Apple could be directed by US national security interests to deny to protect the ongoing US investigation. Supermicro could similarly be directed by Chinese national security interests to protect plausible deniability.
There was a sense of realpolitik by one UK guest commentator on Bloomberg, comments along the lines of "hey, spying happens since time immemorial, put on some big boy pants, yes there is shock but not horror when the Snowden revelations came out, the US does it, too, etc.". I disagree with this sentiment, as while the attack was quite targeted, it puts into question a quite large supply chain network.
Kudos to Bloomberg putting in the 12+ month investigative journalism to pull off this scoop. Yet another validation of the reasons I subscribe to listening/watching them.
> Supermicro could similarly be directed by Chinese national security interests to protect plausible deniability.
Supermicro is an American corporation, headquartered in San Jose. They're not directed by Chinese national security interests, they'll do anything the US Government tells them to do when it comes to US national security.
As someone who has used a few SuperMicro boards in custom builds, I would like to know which motherboard models have been affected. It's not clear to me from the article whether that is even known at this time.
I'd also like to know of a recommended work-around. I have always had IPMI "disabled" as much as I can; assigned a non-routable IP address in the BIOS in case it flips to a failsafe mode by piggybacking on the main NICs. But is that sufficient? Is BMC ~= IPMI?
Has anyone found more technical details yet for system administrators who don't work at the thirty companies contacted?
Technically BMC is the hardware and IPMI is the software, but yes they're the same thing.
It's good to change the network port settings to "dedicated" so it doesn't failover... but if it was malicious, and it clearly has the ability to piggyback on your production interface, why wouldn't it? There's honestly no way to secure this without cutting off your hardware from the internet completely (or firewalling it off similarly).
This is just the hack that was discovered because there was macroscopic evidence of it. All it would take to pull off a similar hack that was undetectable is one well placed mole in the company that designed a key piece of silicon or software.
I have worked in both chip design and security so I feel well qualified to make this assessment: a single mole in the right place who knew what they were doing would definitely be enough. Security holes get past design review all the time by accident. A skilled hacker could easily insert an intentionally obfuscated one that would escape detection.
Yup. And that single well placed mole could have 100 or 1,000 security experts working on their behalf. It is not as though the mole would have to design the hack, only pass necessary documents to a military grade hacking group then implement their modifications.
Why aren't these attacks constrained by normal corporate firewalls? How does a random server on a navy ship start contacting baddie.china.com without raising red flags?
Effectively nothing can be constrained by a whitelisting firewall if you have a sufficiently bored actor. You can smuggle data through a variety of benign looking protocols, things that wouldn't matter in the least generally. Your average server contacts hundreds of different public NTP servers, binary repositories, domain name servers every day. If the keys to the kingdom are a 32 byte ECDSA private key, you've lost if you think you can protect this from reaching the outside world.
A method that wouldn't show up on any firewall in the world is simply to delay or drop certain SYN packets. Even if you only intended to transmit a bit at a time through this, any unauthenticated host on the internet could use this without raising any suspicion or even printing log lines in most environments. As soon are you're making an assumption that you're trying to prevent what's inside from getting out things become substantially closer to impossible than anybody would like.
Firewalls in high security environments aren't just port/protocol based. You lock everything down - source ip/port and destination ip/port. You should know where it is coming from and where it is going to.
In the parent I described a system which would be able to communicate through those restrictions to another compromised host (remember we're assuming everything is compromised for the sake of this article, which actually seems like a good assumption now).
> remember we're assuming everything is compromised
I think Bloomberg (and all related) web servers displaying the article are compromised and they're leaving out critical facts the point the finger elsewhere.
Right so all the stars need to align for it to go unnoticed - compromised server, firewall and other alerting/monitoring tools.
I would have thought one single unexpected packet in these high security environments would raise significant alarm bells and any anomaly would be found very quickly.
Only if you’re lucky. Priority is checking boxes and meeting audit requirements.
If you’re running an IDS on a big 100 Gb datacenter network, you’re literally processing millions of events. Very few places would notice such a thing unless they were investigating something related, and the ones with the capability are going to be for static workloads as getting anything done will be slow and painful.
We had something similar where anything outside "normal" generated a ticket. It was disabled after 1 week because the support teams where getting more then 5000+ tickets each day. And this was after filtering etc....
Now this does not make it impossible, just very complex. In a more "controlled" environment such as a naval ship, i could see this actually working better, especially if the system is supposed to talk to very few external systems.
It does. I've deployed systems that would not only notify staff when novel packets were observed but immediately isolate anomalous hardware through a combination of powerdown and network fabric reconfiguration.
The only way to create such an environment is to totally disconnect it from the outside world -- I'm talking even power source, phones, internet, all of it has to be disconnected or else I can exfil data all day long and nobody would ever know.
Dear snarky anonymous coward, as stated I left in 2015. I think you will find referenced issues occurred subsequent to that date under very different technical leadership.
Intrusion detection involves connections coming from the outside. These attacks originate inside the network, from the compromised equipment.
Except on extremely controlled networks, this would be very hard to detect. It gets even worse when you consider that the Chinese had/have a distributed network of compromised machines. Imagine using a Google edge server as a dead drop...
We need open source hardware designs that can be built locally (where ever your local might be). This black box hardware crap has to stop. Smart people who know how all this works need to dump all their knowledge in to a design and a process. Trade secrets are keeping us not only limited in choices but exposed to bad actors who can control a link in the supply chain.
My gut feel about this for many years has been outsourcing chip manufacturing to another country is a serious security risk. This was a hardware device, hard to detect as it was, how much code is in the chips we already expect on a motherboard?
It seems flat out foolish for one country to own the world’s computer manufacturing.
This is the logical conclusion of companies giving in to shareholders frothing at the mouth for increased profitability by any means necessary. Manufacturing crucial components like motherboards in China is just absurd.
Rather not, China's economic rise started in Jiang's era was both due to it being cheap, AND due to Jiang's era officials being more competent and business friendly than those of an average bantustan (really sorry having to use the term)
I am sincerly impressed by the amount of supply chain analysis and operative supply chain management that went into that hack. And once the Chinese identified a distribution node in that particular supply chain, supermicro, they opted for a brute force attack by seeding these backdoor chips into supermicros servers and wating where they ended up. That was one hell of a hack.
It also gives you pause. Did that happen only once? For how long? Where esle did these chips and servers end up in the end?
Im impressed as well. But I'm sure its not as impressive as what the NSA is capable of. If you're reading an article about this - its because the US government wants you to know about it. The US has been doing this kind of stuff well before the Chinese or anyone else... And yes, totally agree that this is probably fairly common place. Some comments have shown that this happens with run-of-the-mill hardware like credit card scanners.
This stuff also reminds me of the Snowden revelations of hard-disk firmware hacking. Very similar conceptually to what is described in this article - albeit without custom hardware.
Every major major nation can do this stuff one way or the other. But somehow I am used to software hacks by now. That someone is basically managing aphysical supply chain (with suppliers, production and all of that) within an existing one is a first for me. The only thing similar are stories of the Italian mafia getting stuff by customs by duplicating shipping containers, still what the Chinese did is different level. But the, being a supply chain guy, I have an easier time understanding this as compared to what NSA and GCHQ are doing.
if they really are widely seeded (and it’s general knowledge/easily assumed that the pla leans on other manufacturers), this gives me a few ideas about what could happen in the event of conflict between china and the US. turnkey shutdown of the entire economy.
It reminds me of that scene on Armageddon. "Russian parts, American parts, they all come from Taiwan." Now it's China. It is easier to spy when basically everything runs on hardware made in your country I guess.
Just as a sidenote: X-raying PCBs and then diffing them against clean PCBs is a worthwhile thing to do if you're concerned about hardware backdoors, or 'interdiction' of hardware in a supply chain. I do this sometimes when ordering super-critical equipment like Thinkpads from the U.S as you never know what lurks on the motherboard (keyloggers, etc).
I have a clean Thinkpad that I use to compare against potential backdoored devices. So far I haven't spotted any differences in the PCBs. I guess the intelligence agencies have not marked me as important enough to target. That being said, I imagine there are people working in the cryptocurrency space who have a lot to hide (if you own their boxes, you could be looking at thefts worth millions of dollars, or whatever the equivalent is in the cryptocurrency they are developing).
Once you do a thorough sweep and decide one device is clean all it takes is a small PCB revision and you have to start over. It becomes a very expensive exercise after a point (time and money) that is reserved for the most critical of situations.
The problem is a good hack doesn't need the most critical of situations to succeed. It can creep in through the most innocent cracks and climb its way up the chain.
And some hardware hacks are beyond what a regular Xray PCB check would outline, like a modified piece of silicon in a chip. Xrays are great for more obvious changes, not necessarily for something that was inserted at the very start of the supply chain.
> X-raying PCBs and then diffing them against clean PCBs is a worthwhile thing to do if you're concerned about hardware backdoors, or 'interdiction' of hardware in a supply chain. I do this sometimes when ordering super-critical equipment like Thinkpads from the U.S as you never know what lurks on the motherboard (keyloggers, etc).
I'm curious - how does one go about buying or getting access to an x-ray machine (and how much does that cost)?
"In one case, the malicious chips were thin enough that they’d been embedded between the layers of fiberglass onto which the other components were attached, according to one person who saw pictures of the chips."
> That left the decision about where to build commercial systems resting largely on where capacity was greatest and cheapest ... “You can have less supply than you want and guarantee it’s secure, or you can have the supply you need, but there will be risk. Every organization has accepted the second proposition.”
Once the critical mass of electronics manufacturing moved outside the US and manufacturers outsourced the bulk of work, there was no longer any alternative.
Economists say this is ok or even really good for global welfare.
But do any economic models accurately account for this type of economic damage, this type of cost to government, consumers, companies like Supermicro, Apple, Amazon, etc?
And this is one of the major reasons why I support that the US is going after China.
It's a disgrace that previous politicians from both the EU and the US have just let China steal from the west through technology transfer and not actually enforcing IP.
Side note: this is why private blockchains for supply chains don't have any sense because there is no way to probe that you are delivering what you say.
Most likely it is using SPI https://en.wikipedia.org/wiki/Serial_Peripheral_Interface, that requires four pins and the two remaining ones are power and ground. SPI what is used to access EEPROMs and flash memory, so an attack that you can do is daisy chain such a device in the path to the EEPROM the board management controller uses as its firmware storage. Then you can very easily insert your own instructions and get the board management controller to execute whatever you want.
So it’s sounding like the Chinese government may have invested a large portion of its war chest on subsidizing computer manufacturing in order to gain this type of strategic position. It feels like chess, and seems brilliantly scary.
I'm not sure exactly what "delisting" entails, but if my billion dollar company was about to be in the news for having its supply chain compromised by a foreign state actor, I would be happy to not see the stock crash triggered by a public order book.
Stories like these make me think there is a real need for electronic devices made entirely in the US (or your preferred trusted country).
Products like the Yubikey are apparently made in the US / Sweden, but I wonder how many components are actually made elsewhere and just assembled in these countries. Are there any good examples of consumer devices actually made entirely in the US down to the microchip?
Like data breaches, this will be big news that some devices were compromised at some time. Then followed by a slow dribble here and there, that, oh yeah, well more than we initially thought. Then a few months later it will be, "Yeah, it's really a lot more than we thought." Maybe a year from now it will basically turn out to be everything. But numbers get so big so fast people don't really know how to effectively parse the difference between 20 million devices and 20 billion devices. So it won't matter to most people.
And really, it shouldn't matter to most people. There's nothing the vast majority of people can do about this. It's not like we can just go buy stuff made in the U.S.
Building in China may be cheaper, but that’s a stupid consideration. After all the Greeks charged the Trojans absolutely nothing for a fabulous wooden horse.
what they describe isn't possible. which is perhaps why Apple et al. are easily able to refute it.
what actually occurred must be something a bit different, that they didn't understand and/or aren't conveying accurately or didn't receive accurate info due to it being classified. or you know, because the general press is still at movie technology ... the hacker is in the building!
A chip the size of a pencil point can't be inserted anywhere useful or do anything meaningful. Something that small amounts to a discrete component, not a "chip", if packaged. If bare die, it still can't be vastly complex and needs to be mounted in a way that is very obvious (epoxy blob).
So, it must be an additional discrete that inhibits the burning of a W/O fuse in the BMC or some other management function. Thus preventing the disabling of some debug function.
There is no way it has communication capability, etc, on its own.
EDIT: ah, @baybal2 perhaps figured it out. it's likely flash memory and the bmc already has provision to read it as the "recovery" flash.
The Chinese government didn’t directly address questions about manipulation of Supermicro servers, issuing a statement that read, in part, “Supply chain safety in cyberspace is an issue of common concern, and China is also a victim.”
Essentially China ils saying "it was not me". Plausible
I Read that as "The US is also attacking our hardware supply chains". That is, the statement concerned supply chain attacks in general, not this specific one.
China is sourcing virtually nothing from US. If US is attacking the supply chain of China, it's doing it on Chinese soil. Which brings us back to my interpretation
If you scroll all the way to the bottom of the article you will find the disclaimer
Bloomberg LP has been a Supermicro customer. According to a Bloomberg LP spokesperson, the company has found no evidence to suggest that it has been affected by the hardware issues raised in the article.
Why did Bloomberg not buy a bunch of SuperMicros (new and used) and find the chip? That would be difficult but would turn this into a HUGE story. Even if they didn't have the technology to do so in-house, there are many companies they could hire to do a forensic investigation. The weakness of all bugging is that it has to communicate to the "outside world" at some point to be useful and that communications is discoverable. Even Stuxnet, which was much more narrowly targeted than this, was eventually discovered.
There are tons of old SuperMicro computers on eBay for very reasonable prices. They could have bought lots of them of the appropriate vintage (2015 and prior).
> There are tons of old SuperMicro computers on eBay for very reasonable prices. They could have bought lots of them of the appropriate vintage (2015 and prior).
I think buying a bunch of old servers and looking in them for implants is so far outside Bloomberg News' core competencies that it's unsurprising they didn't do it.
I take a different view of this. Some 30 companies got this hardware, two are named as having the resources to find the offending hardware. By this logic you’re still safer using a company such as Amazon.
My point is that you should care about privacy, even if you're using, for example, apple services. Thinking that nobody has the resources to hack a company that big isn't an argument and has just been disproven.
In other words, if you have plans to ever be a politician that china may not want in any position of power, don't store your nudes on icloud/dropbox/google drive, or they may suddenly get leaked when you least expect it and ruin your career in favor of a more... Shall we say "convenient"? ...alternative.
I’m not saying nobody has the resources to hack a company. I’m saying few companies have the resources to detect such a sophisticated hack. Therefore my infra is probably safer with, say, Amazon vs my local colo.
You are correct that individuals seeking personal infosec against state actors must be eternally vigilant.
> In order to get further down the trail, U.S. spy agencies drew on the prodigious tools at their disposal. They sifted through communications intercepts, tapped informants in Taiwan and China, even tracked key individuals through their phones, according to the person briefed on evidence gathered during the probe.
So this is still possible ? And this is slipped so lightly in the article.
"Two of Elemental’s biggest early clients were the Mormon church, which used the technology to beam sermons to congregations around the world, and the adult film industry, which did not."
Amused to see some element of humor added to an article on a subject that will have extremely serious consequences, the likes and extent of which are hard to estimate at the moment
Where is the _actual_ explanation of how the chip works? I assume it's been dissected to bits, and that's way more interesting than just "it's been found on lots of boards". What does it actually contain that apparently lets it do things that apparently no one else has managed to achieve at that scale?
Do we know how this attack actually worked? From TFA it seems to involve the BMC (which afaik everyone already assumed was untrustable), and also involved the capability to "phone home" (also notable since in security-conscious deployments the BMC NIC would never be allowed public Internet connectivity).
"The resulting software dramatically reduced the time it took to process large video files. Elemental then loaded the software onto custom-built servers emblazoned with its leprechaun-green logos."
I didn't realize Pied Piper was based on a real company...
Probably much bigger. It's naive to assume Supermicro was the only vendor compromised in this way. Similar implants probably exist in equipment from every major vendor of information-processing equipment. Routers, mobile network equipment, you name it.
This is the leverage you give another nation-state when you let them control your supply chain.
Why didn’t they ask any of their sources to point out one of these chips on a specific (common) Supermicro board? That’d allow independent verification by everyone and way less vague speculation.
Well it's nice to the see the specifics on how they did it, but anyone was really surprised by this? Most electronics in the World are made there what people are expecting gonna happen?
Where are Vcc and ground for the chip? These leads would show up on a simple micrograph of the board. How does this chip make connections that enable spying on data in the server?
As long as you are powerful, you can get away with almost anything. This goes from individuals to countries. Had China been weaker, it would have been invaded already .
Maybe it's worth talking about how we can better secure the chain of custody in shipping? I asked about this a year or two ago and my question got the thumbs-down.
I wonder, given the fact the C&C were identified, wouldn't be external firewall configuration much cheaper mitigation method than motherboard replacement?
Maybe it's worth talking about how we can better secure the chain of custody in shipping? I asked about this a year or two ago, here, and it got squelched.
Isn't this or alike chip put all the Chinese hardware other people and enterprises buy too? Or at least in all the SuperMicro mainboards (I use these too!)?
As a nation, we need to unite against the external threat of China.
Russia and Israel are fine.
China is the number one problem sowing discord in this country.
> The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off.
Obviously this would still be possible without IntelME, but having an always on, highly privileged and remotely accessible baseband definitely makes the modifications easier and smaller...
Elemental servers sold for as much as $100,000 each, at profit margins of as high as 70 percent, according to a former adviser to the company. "Two of Elemental’s biggest early clients were the Mormon church, which used the technology to beam sermons to congregations around the world, and the adult film industry, which did not."
Church would certainly be more interesting if they did, though.
Other motherboard vendors (e.g. Tyan, Asrock, Gigabyte make server boards), or buying entire servers from Dell, HP, ... And hoping they don't have issues like this, despite probably all manufacturing partly in China.
None of those guys offer the same variety of form factors and features that Super Micro does. If you are an OEM that needs an Intel Xeon-D board in mini-itx form factor for example.
Holy molly! This sounds similar to what I thought a couple of days ago: China is the major electronics supplier to most technological companies, what if China added some sort of sniffing or malware on the chips to spy on their customers?
As something I heard: In China, business a fight for survival that is brutal to the extreme. You will be cut down, cheated, extorted, and broken with zero hesitation.
Doing business in the West is comparatively like a walk in the park.
... huh. For the first time, I'm considering the possibility that the Trump administration's tariffs on China might actually result in a positive outcome: forcing US companies to look elsewhere for hardware, possibly creating new supply chains with other (hopefully more trustworthy) partners out of sheer necessity.
1) I wonder how wide spread this is in electronics produced in China. Say, in those cheap usb blutooth radios, for example.
2) I wonder if this is spurring the current trade war push, besides just Trump.
3) Seems time for more transparency on supply chain integrity and security.
"Over the course of the past year, Bloomberg has contacted us multiple times with claims, sometimes vague and sometimes elaborate, of an alleged security incident at Apple. Each time, we have conducted rigorous internal investigations based on their inquiries and each time we have found absolutely no evidence to support any of them. We have repeatedly and consistently offered factual responses, on the record, refuting virtually every aspect of Bloomberg’s story relating to Apple."
I wonder how legit it would be for an US company to set up an internal gag order? Running on compromised hardware is practically a nuclear meltdown (pun intended) and publicly admitting to it as well.
The trick seems to be having sufficiently uninformed people in all positions that might get to write that kind of response. No need to feign ignorance when you can have the real thing.
Qualification is no hindrance to being or of the loop though, deliberately or not. It just makes it more embarrassing. Ignorance itself is still easily attainable, just "shoot the messenger" (this will make you seem tough and thorough to your higher ups) a few times when bad internal news escalate to your level and you will remain in blissful ignorance for the rest of your tenure.
How does that actually work? How far down the chain of related facts to the national security incident are parties allowed/required to lie? If facts can be used to triangulate the secret, that can't be disclosed, right? Are incidents like this like a little fact-bomb which can be used to legally hide other institutional facts under its cover?
I assume it’s like national security letter. Only people in the company that has knowledge would be the ceo, general counsel, and people working directly to mitigate the issue. PR and corporate communication wouldn’t have any knowledge on the incident. I wonder how you collect insurance for these types of incidents if you can’t disclose them.
I mean what are they gonna say? "Yes, we have been aware that an unknown but possibly huge number of our servers have been compromised, but decided to keep our customers in the dark"?
I think the point is that actual honesty from these megacorps would be so surprising that even raising the possibility of it happening is so absurd it feels like parody writing.
This may be the first time in the history of the internet a statement from Facebook has ever been held up as an example of honesty and transparency from a corporation in America.
Mostly because I believe that, while it's important to maintain healthy skepticism around privacy and security issues, I also believe that we don't benefit from cynical hyperbole.
The Snowden leak confirmed Apple's statement. It showed that Apple worked with the FBI on a system to process court ordered user data requests. Apple would obviously have no knowledge of the code name of a downstream NSA system that processed that data.
The snowden leaks among others show that most companies aware of PRISM ended up flat out lying about it. Either it's a type of NSL we haven't seen before or employees receive death threats, etc.
No, none of the companies lied about it. The companies worked with the FBI's Data Intercept Technology Unit. They would obviously have no knowledge of a dowstream data processing system like PRISM.
Couldn't an NSL have been served to datacenter operators, along with the notification of the attack, and the organisation's management simply be unaware?
That assumes that 1) the intelligence community has the power to stop it and 2) that Apple believes this to be the case and 3) that Apple is confident that the intel community would use that power to protect them. That seems like a reach to me.
#3 isn't the intel community protecting Apple, it would be protecting themselves, which is a lot more plausible. They don't want detailed information about the techniques coming out. Odds are good that the Bloomberg story is still incomplete in some critical way, and decent that even if the story as a whole is broadly-speaking "true" there's still an outright lie contained in it. My guess would be the way in which it was discovered.
I work for a company that sells network appliances, and I've been questioned by customers as to why I'm doing an SRV DNS lookup instead of a standard A DNS record lookup in some software I wrote, and had every detail of how I use TLS picked over by some customers. (More power to them. Not a complaint.) Some people run really tight networks. I wouldn't be surprised the real discovery mechanism was someone noticing the packets heading out that had implausible source-dest pairs ("why is my internal network that barely knows the internet exists trying to send packets to $RANDOM_LOCATION?"). If the people discovering this were actually the intel agencies themselves, for instance, they'd find another story to tell rather than reveal that. I am absolutely, positively not claiming this is true; I have no more evidence of it than anyone else. I'm just giving an example of the sort of thing I mean. It's also possible the intel agencies slipped a hint to someone about what to look for; again, I have no info to that effect, just an example of why they might not want something to go to court.
Although that nearly went in the other direction. The people involved were nearly sent to jail for shipping arms to Iraq which they had been doing at the behest and with the complicity of the UK security services.
The power of government agencies to turn up to people's offices and tell them "you need to stop doing what you're doing, and it's illegal to mention this meeting" should not be underestimated.
Not at all. It would be quite damaging to their reputation if it came out later that they were affected by this, knew it, and lied about it. Especially since the privacy of customer data is a key part of their marketing message these days.
Well, what was Apple's response to the PRISM revelations?
Apple: "We have never heard of PRISM. We do not provide any government agency direct access to our servers, and any government agency requesting customer data must get a court order."
PRISM: "Collection directly from the servers of these US service providers: Microsoft, Yahoo, Google, Facebook, PalTalk, AOL, Skype, YouTube, Apple" A later slide claims that Apple joined the program in October 2012.
So, was Apple lying about PRISM? The Snowden documents certainly seem to support that position[1]. If they lied about PRISM, why would you trust them now?
[1]: Obviously there are possibilities like Snowden documents were wrong, or deliberate hoaxes, etc. Spy games are fun!
This article is more or less total bullshit. At _best_ that device might be a mechanism to cause failure intentionally. And there are tons of ways to detect it with commodity technology, and plenty of vendors who implement that technology for assembly manufactures commercially.
That’s what I thought but then it says it’s hooked to the BMC bus. It’s basically a small IME device with no java bloatware to run. I’d think it’s reasonably credible
This breaks the site guideline that asks you not to insinuate astroturfing or shillage without evidence. Please don't do that—it's a toxic trope that leads to dumber threads.
Edit: looks like we've already warned you about this more than once. If you keep doing it we're going to have to ban you, so please don't post like this again. Ditto for unsubstantive comments in general.
What evidence do you want? They only have 5 accounts with the ability to downvote. This is why comments always have -4 with anything critical of apple.
>Each time, we have conducted rigorous internal investigations based on their inquiries and each time we have found absolutely no evidence to support any of them.
An uncharitable reading, but this statement does not exclude the possibility of investigations by 3rd parties hired by Apple.
Neither does it exclude the possibility of internal investigations of which Apple's press office is unaware. If Tim Cook simply said this never happened I'd believe him. This elaborate denial suggests otherwise.
One of the company I worked for once received request from a news outlet about potential rumor around us, and bullied our CEO into an interview to disprove that rumor.
Then the journalist picked several quotes out of context as proof that rumor being true.
While I don't trust megacorps, I don't know if I can trust journalists more when a major breaking news is on the line.
They have to deny it. If the allegation is true, it means the Chinese CCP Gov knows which computer parts are produced specially for US gov or certain big companies and target precisely. There has to be some deep link for information flow to allow that. Anyway, worth further digging.
The article says they shipped the boards to a company in Ontario, lawyerly:
>…At no time, past or present, have we ever found any issues relating to modified hardware or malicious chips in SuperMicro motherboards in any Elemental or Amazon systems. Nor have we engaged in an investigation with the government.
I mean:
> The companies’ denials are countered by six current and former senior national security officials, who—in conversations that began during the Obama administration and continued under the Trump administration—detailed the discovery of the chips and the government’s investigation. One of those officials and two people inside AWS provided extensive information on how the attack played out at Elemental and Amazon; the official and one of the insiders also described Amazon’s cooperation with the government investigation.
This reminds me of that old story about the Xerox copy machines that the Soviet Union bought.
Where each unit was planted with a image recorder. And for years, the American spy agencies had a great laugh, that they were able to intercept all the documents that the Russians made a copy of.
Back then, this was an off-network infiltration. Where the copied images, were retrieved during regular servicing intervals by a Xerox technician.
I was born raised, and went to CS undergrad program in China before come to US. Yes. I can testify that the "Xerox Copy Machine" is the first thing they teach you about information security :p
I had never heard of this...but now I can only imagine how exciting this must have been as one of the engineers-- working on a top secret project for the CIA in an abandoned bowling alley: https://electricalstrategies.com/about/in-the-news/spies-in-...
edit: whoops, looks like amatecha beat me to posting more info
It's important though to make sure that your team keeps tactical advantage so that it can continue existing. Maybe someday mankind will find world peace but in our current world there are a lot of nations that hate each other still and wouldn't hesitate to take advantage of weaknesses of other nations for personal gain.
In the developed world, hatred is often manufactured to gain power... The leaders of a nation can cause more harm in their quest for glory than the average citizen of an opposing state.
If our team is resorting to unethical and immoral ways to gain that advantage, then we can't take the moral high ground and also can't complain when the other team also does "whatever it takes" to gain an advantage.
Also, security through obscurity is, as we know, an illusion. Information always finds a way out.
I understand your point, but there should be limits.
What do you tell your small human about the current behavior of many prominent adults, or if the small one is too young to see what's happening, what will you tell them?
But once you create the technology and hand it over to someone else, you have no assurance that it's only going to be used against the "bad guys":
Judging by the number of parts ordered from Xerox, Zoppoth believes that spy cameras may have been installed in photocopiers all over the world, to keep an eye on U.S. allies as well as enemies
> Exciting, but not so ethical. We owe society to put our knowledge towards making it better for all people, not just "our team".
Our team promoted liberty and democracy worldwide (see below). The other promoted totalitarian dictatorship, labor camps, etc. If your team is liberty and democracy - self-determination for all - then it's not 'our team', it's everyone's team.
To address the elephant: Yes, there were many, many exceptions to the West's support of democracy, and many of them were awful (Indonesia, the Mideast, Zaire/Congo, and many others come to mind) but beyond a doubt, during the Cold War and after democracy and liberty exploded - stop and compare it to any other time in human history. Lots of bad things happened, but compare that with Soviet-dominated areas such as the USSR itself and Eastern Europe.
+1, couldn't agree more. I wish more nations/Governments would think like this, but too many are motivated by more wealth and power. At some point, you would think, the animal would satiate itself. It appears that is not the case.
That the Chinese military is technically competent enough to pull off such a thing.
Or that they are incompetent enough, to not secure their own back doors and networks, and allowed the FBI, NSA, and other American government organizations, the ability to counter-hack them, and monitor all their internal communications.
The truth is somewhere in between.
So, this article is basically saying: China got clever, and spied on us. We spied back on them, and we also hacked all of their internal communications too, so basically, we own them.
Well, at least America is finally admitting, that they too, actively spy on other countries. And can easily infiltrate any other country that they want.
The US has never denied that it spies on other countries’ citizens and state apparatus. Every nation does.
The only controversy has been about whether the US intelligence community spies on US citizens without warrants, which used to be considered out of bounds, but is now awash in murkiness.
One doesn't even need Snowden to know this. Given even a little reading of the IC's (intelligence community), and more specifically CIA's, history. What else do you think they do all day, if not for spying on other countries?
As mercutio2 mentioned in a reply to your same parent, the controversy w/r/t Snowden is domestic spying.
> Or that they are incompetent enough, to not secure their own back doors and networks, and allowed the FBI, NSA, and other American government organizations, the ability to counter-hack them, and monitor all their internal communications.
Wait.. did I miss something in the article?
The closest thing to that, I remember happening, had the roles reversed [0], and should be an important lesson about not underestimating the opposition.
> we also hacked all of their internal communications too
my take on the article was that US counterintelligence sort of cobbled together some hacked phone communications with some info from human agents working in or around or near Chinese factories. to say that "we own them" maybe goes too far?
Not surprising, when you consider that every intelligence agency spends at least 10x more money on offense than defense. I first became aware of this when I worked at a company where we had the NSA "defense" as a customer. We tried and tried to get contacts on the offense, because it was a much bigger market opportunity, but the company went under before we succeeded. So it's our Ferrari offense against their Fiat defense, and vice versa. (Not at all a comment on skills or motivation BTW. There might be an element of having one's pick of a larger talent pool, but mostly it's about resources.) It's practically inevitable that we'll both score lots of points off each other.
Who else found the design of the page made the article difficult to read? I know darkness, spies and hacking go hand-in-hand but it's a bit too much imho.
This is because, contrary to what a lot of people say about dark themes (even though it's mostly a meme at this point), dark letters on white background are better to read than white letters on black background.
"People with astigmatism (approximately 50% of the population) find it harder to read white text on black than black text on white. Part of this has to do with light levels: with a bright display (white background) the iris closes a bit more, decreasing the effect of the "deformed" lens; with a dark display (black background) the iris opens to receive more light and the deformation of the lens creates a much fuzzier focus at the eye." - Jason Harrison – Post Doctoral Fellow, Imager Lab Manager – Sensory Perception and Interaction Research Group, University of British Columbia
This quote describes the situation well I think, why the opinions differ and why neither dominates UIs really.
I know this is going off the main topic. But it is very strange that I discover Black background, as in Dark mode in macOS, doesn't work at all despite what I have always thought it would be cool. But Black background on mobile devices work absolutely great!.
And I have no idea why, I searched on Google and couldn't find any decent answer. All results were either OLED being battery friendly with switched off pixels or other mobile UX for Dark mode. None of them describe the difference felt in Desktop and Mobile. Any link or explanation would be much appreciated.
As a fan of dark color themes and displays, I've shared a bunch of thoughts related to this on my personal blog. But I'd like to share a few quick thoughts below. I am not an expect on these matters, so these are just my opinions:
1. Glare. When reading on a small device, the amount of glare reflected in the screen space occupied by black pixels may be fairly minimal depending on your reading environment and positioning. Also, although mobile devices use glossy display surfaces, they tend to have several anti-glare layers in the display stack. You said your computer is macOS, so I suspect you're unlucky enough to be reading on a glossy laptop display. With "dark mode" color schemes, your eyes can more easily see the reflected scene (maybe your office lights, an exterior window, or even your own face). And the focal length of that reflected scene is 2x your reading distance to the screen. That reflected scene at an extended focal length is more relaxing for your eyes to focus on. So in order to read, you need to fight your natural tendency to relax and look at the reflected scene.
If you are lucky enough to be reading on a matte desktop display (typically a professional or prosumer monitor, such as a Dell UltraSharp or LG 43MU79-B), the glare will be minimal and it should be much easier to read.
2. Pixel density. I contend that one reason dark themes have become more prevalent in recent history is thanks to wider adoption of high pixel density displays. At a legacy density of approximately 75 dots per inch, the stroke weight of bright text on a dark background appears too faint if the strokes render as just one pixel in width. Higher pixel density allows for the strokes of letters to be wider than a single pixel, allowing for greater clarity. If you ever designed a dark theme in the days of ~75 dpi displays, you might naturally favor bold text as the default because it was considerably easier to read. (Interesting sidebar: many console oriented bitmap fixed-width fonts historically used two pixels for stroke weight, especially in the horizontal dimension, presumably because they were designed to be used bright-on-dark.)
Now, you did again say you were reading on macOS, so your display's pixel density is probably higher than ~75 dpi. But a MacBook Pro is still only ~220 dpi. A Surface Book is ~260 dpi. A Dell XPS laptop with a 4K display will be a little higher still (maybe ~300 dpi). But many phones are using 450+ dpi displays. The stroke weight of a character on a mobile device is several pixels wide, so it's highly defined and clear.
3. Font selection (related to above). Bloomberg has selected a serif font, presumably because they are a media organization and serif fonts are typically used for article bodies. However, combined with pixel density, the serifs will lose a lot of their definition and (in my opinion) reduce readability versus a sans-serif font. As an experiment, pull up the dev tools and change article[data-brand="businessweek"] .body-copy p to use sans-serif and see what you think. It may be marginally easier to read.
4. Don't discount OLED. The contrast that Bloomberg selected is maximal (pure black background and pure white text) and that works well for OLED since the background vanishes entirely. However, since most desktop and laptop monitors are not OLED, you're still getting backlight bleed, so the contrast is imperfect. Especially combined with the glare factor above, my experience is that given LCD backlight bleed, it is better to use a dark gray background instead of stark black. This makes the backlight bleed less distracting, for lack of a better word. The background ends up looking more uniform.
As with above, try adding "background-color: rgb(40,40,40)" or similar to .body-copy and see what you think.
While you may find a particular attack if you're looking for, in general, it's impossible for even the most thorough audits to check for the whole class of such attacks. You're not going to look into the chips. Well, you can, but that's prohibitively expensive and destructive - even if you could check that this chip was okay, then you still have to throw it out after analysis and plug in a different one.
The only feasible thing to do is thorough audits of all the supply chain for every component in your system, ensuring that your supply chain does not include even a single chip from an "untrustworthy" supplier, and even then it reduces the chances of an attack but does not eliminate it.
> Faggin: Yes, we were concerned about others copying the Z80. So I was trying to figure what we could do that that would be effective, and that’s when I came across an idea that if we use the depletion load the mask that doesn’t leave any trace, then I could create depletion load devices that look like enhancement mode devices. And by doing that we could trick the customer into believing that a certain logic was implemented, when it was not. Then I told Shima, “Shima, this is the idea how to implement traps. Put traps, you know, figure out how to do the worst possible traps that you can imagine,” and then Shima with his mind, that was steel mind, was able to actually figure out a bunch of traps that he could talk about.
> Slater: You want to tell us a little about that Shima?
> Shima: I didn’t count [on] talking about that mostly. I placed six traps for stopping the copy of the layout by the copy maker. And one transistor was added to existing enhancement transistors. And I added a transistor looks like an enhancement transistor. But if transistors are set to be always on state by the ion implantations, it has a drastic effect on very much. I heard from NEC later the copy maker delayed the announcement of Z80 compatible product for about six months.
The logical extension of this is to embed a sleeper chip between board layers that can be activated by a radio signal, then try to disperse it widely within the DND.
Any serious conflict with China would then look like the first day of the second Cylon war...
Of course, they would probably have to get a Chinese boat really close to a US warship a few times beforehand to test the system. All they would need to do is to receive some sort of a ping back to confirm receipt of the signal. Could be simple as an innocuous visit to a particular page of a website.
It would probably look like brinksmanship or posturing, or a navigation error of some sort.
Doesn't the printed circuit board also need to be redesigned, to allow for the spy chip to be inserted?
So, this means that China forced multiple companies to modify their processes, in order to pull this off.
1. The PCB designer
2. The PCB printer, if it's a separate company
3. The company assembling the final product
And if they forced the PCB to be redesigned, then wouldn't this be a immediate red flag? But this means that the customer, Supermicro, would have to audit the PCB results as well.
They wouldn't need the PCB designer. They would need to suborn (employees of) the PCB manufacturer and the assembly house. These will be supplied with the PCB layout (in Gerber or ODB format, for making the PCB) and the bill of materials and pick and place file (for assembling it). With these files you can reverse engineer the schematic and carry out the modifications described. The PCB manufacturer might well be a sub-contractor to the assembly house.
Most electronic engineers don't have the software tools for or experience of sophisticated reverse engineering but there aren't any major conceptual barriers. They would have to go from the geometry of the tracks and pads to a connectivity graph (very automateable), then collect the pads into footprints for components (probably partly automated), then identify those components and the functions of their pins (easy with a complete BOM) and then workout the circuit function (should be straightforward for standard parts and circuits).
There might not be automated tools for making the desired changes, in which case they will have to manually draw the new track geometry on the Gerbers, add the parts to the BOM and pick and place files and change or nobble the test criteria / files. Hard work but quite straightforward.
These skills will be developed by people doing legitimate industrial reverse engineering as well as espionage / intelligence. I would think there are also unfortunate cases where firms have to reverse engineer their own products after losing the original files.
"Two of Elemental’s biggest early clients were the Mormon church, which used the technology to beam sermons to congregations around the world, and the adult film industry, which did not."
Because it's so awesome. This must be one of this journalist's career highlights, to be able to put something like this in a mainstream serious reporting piece.
> Two of Elemental’s biggest early clients were the Mormon church, which used the technology to beam sermons to congregations around the world, and the adult film industry, which did not.
Looks like SuperMicro was delisted by Nasdaq just over a month ago. Wonder if it has anything to do with this under their pretty poor surface excuses for not being in compliance.
This story could easily be interpreted as anti-China propaganda. Could you think of any other sovereign power who would want a backdoor into servers used by billions of people across the internet? Hardware comes from China. This doesn't mean that the Chinese government orchestrated the attack. The United States government is having a trade war with China. This article's publication isn't just coincidence.
Further, those with insight about how to use the backdoor definitely made use of them. This is one of the central issues with government-mandated backdoors. They are introduced for one group but used by others.
It could be interpreted that way, but this sort of activity isnt novel or unknown.
The Feds run key hardware through third parties to detect counterfeit hardware. The “new” thing is the targeting of devices headed for commercial customers.
your comment looks like an pro-china propaganda. In china the state has heavy control on all the companies. The hardware is manufactured in china. I don't think something like this is happen without the state's knowledge.
It seems like many people take communist-like propaganda at face value.
The Chinese government would like to have almost-complete control. The Chinese government publicly says it has high levels of control.
But when they can't effectively regulate their medical (mass HIV infection from blood plasma needle reuse), food (tainted milk), or chemical (unlicensed mass CFC production) industries... reality seems to differ.
So, suppose you're Supermicro. When some CPC official comes around to tell you to make your technology a little easier for the intelligence department to access, you're going to do it.
Companies in China (especially those in the tech sector) have to keep close ties to the government, and most of their leaders are members are of the party. You don't GET to be a multi-billion dollar tech company* in China without toeing the party line [1] [2] [3]
The issues around regulating food safety and vaccines that you mentioned are irrelevant.
* Foreign companies must operate Chinese subsidiaries to run their operations in China.
I see your *, so I assume you do know SuperMicro is a US company? And that the breaches reportedly happened at local Chinese subcontractors?
Everything you say seems to be valid for Chinese companies. Which SuperMicro... isn't.
If they want to start auditing their incoming supply from China more closely, or even shift production elsewhere, there's nothing except cost stopping them from doing so.
And the rub is that any competitor also using Chinese supply (for cost savings) is vulnerable to the same attack. SuperMicro was presumably targeted because of their size and global customers.
Apple is not Apple Inc. in China, it is Apple Computer Trading (Shanghai) Co., Ltd. SuperMicro will have its own subsidiary/subcontractors to handle their operations in China.
As having worked for a subsidiary of a large multinational company in China, I can tell you that a lot goes on that you might find surprising. Middle management and those involved with establishing the deals with subcontractors would often have arrangements to make money through various means, such as through IP transfer or property theft. The subcontractors themselves are Chinese companies.
At our place a lot of shady things were going on and were, thankfully, found after a number of years. The global HQ had to step in and fire around 1/3 of employees working at the China branch.
Now, there is no reason to suggest that this was related to any government policy or request. But it should be clear by now that the CPC are not exactly opposed to shady things happening to foreign companies. My point is that this is an an environment where something like this can happen quite easily, especially when a lot of technology companies have close ties to the state.
Anti-China propaganda? These chips were designed by the Chinese military and inserted by PLA agents. Do you have to start calling things propaganda to make it seem like this is baseless criticism of China to those who have not read the article?
> Nested on the servers’ motherboards, the testers found a tiny microchip, not much bigger than a grain of rice, that wasn’t part of the boards’ original design.
> During the ensuing top-secret probe, which remains open more than three years later, investigators determined that the chips allowed the attackers to create a stealth doorway into any network that included the altered machines. Multiple people familiar with the matter say investigators found that the chips had been inserted at factories run by manufacturing subcontractors in China.
The only interesting thing about this is left out. Who planted it is clear (someone told to do so) but not a single time is it questioned who they planted it for. Smells like false flag to me.
We think China does X Y and Z but we know the US does X Y Z and the rest of the alphabet. So unless something specific is leaked that shows who actually ordered this, logic would point at the US.
The article specifically mentions the PLA as the bad guy. So that part was not left out. The whole story is hearsay so far. It's plausible, but best read with a good dose of salt.
If we believe the story up to the point where subcontractors are forced into planting the chips, we must accept that it's easiest and least risk for Chinese government actors to force them into it. Mounting a false-flag attack at Chinese subcontractors would be exceedingly difficult for foreign agents. They'd likely blow their cover when trying to represent Chinese officials.
If you want an alternate version, I suggest you start with the easiest: The whole thing didn't happen. And Bloomberg is a victim of propaganda.
> ...but not a single time is it questioned who they planted it for...
The article and some accompanying reporting on Bloomberg audio/video segments says the attack seems targeted relatively specifically towards nearly 30 organizations (only US-based organizations were mentioned as targets, unknown if the list included organizations based in other nations). One known vector was through four subcontractors in China that built the boards for Supermicro's main Shanghai factory, specifically by bribing and/or coercing managers of those subcontractors' factories to go along with accepting the chip shipments and to make changes to the plant floor from the design to perform the chip insertions.
Designing and building a chip like this and then mounting the logistical effort to performing the insertions costing some non-trivial funds, coupled with the known targets, (Amazon didn't seem specifically targeted, Elemental a company they acquired was, who notably has US national security clients), form the circumstantial allegation that a PLA spy unit was behind the attack. You are correct that this doesn't entirely rule out a false flag possibility, but until we get more details about this, we're operating in the dark.
A false flag is an interesting supposition, but how would the US benefit from successfully convincing the world of the false flag's cover story?
The US has a few things to gain from this story: Economically, because Chinese products are perceived as compromised. Politically: because the Chinese government is seen in the offense.
The US has something to loose too: Being perceived as dependent on Chinese manufacturing and potentially compromised down to military hardware. (The first everybody knows, the second would be devastating for trust.)
All in all it would be a weird angle for a false-flag attack.
But then, since the logic points to the US it only stands to reason that is a false false flag! Why those wiley spies in Beijing really are clever. Discredit The USA and plant spy chips in important computers!
https://news.ycombinator.com/item?id=18142277
https://news.ycombinator.com/item?id=18138990
https://news.ycombinator.com/item?id=18143569