My favorite debugging challenges are the "Pfft, computers simply don't work that way" -> user reproduces symptoms in an inarguable manner -> "Hmm..." -> "Well, apparently when thirteen levels of stacked prisms are arranged in the right way, computers do work that way. Fixed (on Tuesdays)."
Edit: Or, you know, hardware failures that nobody expects anymore.
When I interview developers, I like to ask them about their favorite bug. I want to hear a story. Most people have a good one. It's usually intermittent, hard to track down, and takes a long time to solve. It becomes a developer's nemesis, taking on an identity. The relief, when it gets solved, is huge. Good developers tend to talk about such bugs with passion.
This story is a beautiful example of such a story. Great read, great bug, great analysis!
Years ago I worked on a stock control system for a popular clothes retailer. Where previously store managers had to spend hours on the phone ringing the depots to see what was in stock and reordering popular lines, now they could fill in orders an they would be emailed to the depots in machine readable format, with the order balanced across multiple depots depending on availability of stock.
After the system had been in operation for a year or so, managers started to complain that some orders weren’t turning up. After a bit of digging it only seemed to affect one particular depot, but I couldn’t initially see a common factor between the orders. It had nothing to do with the size of the mail, or the originating store. I could tell from the logs the emails were being formatted and supposedly sent to the depot, yet they never translated into orders. The system would silently fail after we’d handed the email to the depot with no obvious reason why.
So I wrote a quick script to collect all the orders for a day from all the stores and compile a list of all product lines that were present in the orders that went missing, but not present in the orders that succeeded, just to see if there was a pattern. This turned up a few dozen items and after a quick visual scan of the product descriptions one stood out slightly, so I phoned the depot on a hunch.
The depot had an anti-spam filter on incoming emails that silently dropped messages with pornographic words in the body or any attachments. The retailer had just received a new seasonal product line, which was selling well, and the product description was “bondage tights”.
After getting the depot to whitelist emails from our system, orders for this product line started working.
Our neighbor, who started as an elementary school teacher but moved into elementary computer education (a roving teacher with an Apple computer lab cart) had a similar problem. Her title had been changed to something like Computer Education Specialist and her email would be blocked when sending to certain other government entities. When the school district upgraded their spam filter, her emails started to be blocked within the district as well. It turns out her title was being identified as Computer Education Spe CIALIS t.
We ran into a similar but much easier to identify problem with PayPal - any products we sold containing the words "Cuba" or "Havana" in their title or metadata, regardless of the context, would cause PayPal to reject the entire payment with a meaningless error and no indication of what had caused the problem.
PayPal refused to admit to us that this was the cause and insisted they couldn't shed any further light on the issue despite the fact we could trigger it 100% of the time.
In the last few weeks it's started randomly choking on the word "Pharaoh", God knows why, and God knows why it only does so about half the time, but I'm not going to bother asking PayPal about it.
Fraud detection and compliance systems have a false positive rate, unfortunately. And a company confirming something like that would be tantamount to telling how to avoid the filter, which also raises their false negative rate, which is not in the company's interest, whether the party in question is a fraudster using it for their own interest or a non-fraudster that spills the beans on HN.
But yeah, it sucks that their system can't learn to give you an exception after it has been reviewed once or twice.
Tough one. I know exactly what you mean, I know I have been "there" many times, but I surely would not be able to retrieve any of those stories from memory even if my life depended on it.
We seem to condition ourselves to quickly forget the excitement because even the most epic bughunt will tend make you lonely if accidentally used for conversation in polite society.
I think this goes for a lot of niche hobbies; a bike nerd might not be able to casually converse about that time his gears stopped working properly in the middle of a race, but to another bike nerd it would be a hilarious story! The same for us programmers, it's hard to relate to for non-programmers. Perhaps also why it's important that your friends share your interests.
Early development of a numerical solution for roots using Newton-Raphson in Fortran IV on a DEC 10. My code was correct, I walked through it manually many times and I had printed out many variables and yet it never found the roots.
In one of my subroutines I changed the value of a passed in variable. In my code there was a point where I called that subroutine with the value '0' for the variable.
The Fortran compiler was pass-by-reference, so the compiler accommodated me by creating a reference to the constant 0 and passing it to my code. In my subroutine I changed the value of that passed in 'variable'. Since the compiler was using that "constant" elsewhere any conditional that looked for 0 would fail after that point since the value of 0 had changed.
My professor and I only found it after we printed out the machine code generated by the compiler.
Old-timers please correct me if I'm wrong, but modern-style full-coverage tests of code weren't trivial to write or add to fortran projects on dec10's...
It was possible to write tests but pass by reference is a "feature" of Fortran IV.
A subroutine receives values from the caller through
the parameter list; likewise, the subroutine returns
values by modifying the variables in the parameter
list. Subroutines are invoked in a FORTRAN program
as a statement, not an expression. The syntax used
is,
CALL name(actualParamList)
See you modify the parameter to send a return value. Really ancient stuff. And if you're working on hardware that is challenged in not having a "load immediate" instruction for floating point, well you get "optimizations" where the compiler creates what is essentially a pre-initialized variable and uses the address of that when a constant is used.
I mean, I don't understand how the compiler got out the door if passing an integer to a subroutine caused inexplicable disasters. It seems like ChuckMcM is saying the equivalent of
int increment(int x) {
x = x + 1;
return x;
}
int y = increment(2);
Would blow up your program's concept of 2, and I can't imagine that being missed before sending it out to users. But I've never used Fortran, so I may be seeing false equivalences to structures and practices in modern languages.
It's almost always motherboard hex spacers. Source: have had multiple computers with such problem. All were due to surplus spacers having been put in which subsequently contacted solder pads on the board underside.
The worse version is broken in-board signal paths.
Both ultimately result from mechanical distortion.
I've been caught out by "ground" not really being ground before; two different sides of a whole floor computer / comms centre had different potentials. So most of the power was hooked up to ground on one side of the building and a small number of non computer / comms gear was hooked up to ground on the other side (think lights etc that were too difficult to rewire), but the other gear used wall sockets that were a different colour to make sure people didn't plug computers or network gear into them. This was all discovered after a temporary 20V spike difference in the different grounds trashed an expensive router. The electricians suspected that because we used a largish number of microwave links on the building the copper bars that run down the building to ground were acting like antennas and picking up the energy. So we defaulted all the sensitive gear to the "ground" that was least problematic. It was cheaper than re-doing all the ground circuits in the building with RF shielding. The joys of old buildings.
A popular train line out of London has an electrical problem like this with one of the train models run upon it. Whenever these trains, travelling about about 100mph, cross from one power section to the next with the front the train entering one, and the rear leaving the other, there is some weird electrical event that causes a safety mechanism to violently retract the rear pantograph[1] (maybe more than one). This causes an almighty WHACK on the roof which scares the bejesus out of any visitors not used to impact noise. I've seen at least one coffee ejected from a cup onto someones trousers because of this. The train progresses unharmed.
Are you sure this isn’t just the pantograph passing through the neutral section? They have these on long runs where the power is supplied from different sources (resulting in sync issues). There are circuit breakers at these points which on some trains makes a very loud ‘clunk’, which stops the train trying to draw current through . The other type of ‘clunk’ you get on some trains (although not the ones you are describing) is the pant going up/down on combined AC/DC lines.
It could be circuit breakers, yes. They would need to be big devices, though. It is very loud. Best I can come up with is someone dropping a shopping trolley on the roof from a storey or two high. I got the pantograph information from a rather "train-nerd-I-love-my-job!" sounding driver over the tannoy one evening perhaps two years ago. I may have misheard!
They are the circuit breakers. Our trains in Brisbane, AU run 25kV (Thameslink seems to be the same 25kV overhead, with a 750V third rail, but this will be the overhead section) and the circuit breakers trip during the entry/exit of a neutral section, causing an almighty bang as they kick out and back in. If you're sitting in the wrong carriage directly underneath them it's very loud.
For reference, 1kV arcs across about 1cm of air, so 25kV needs a full foot of space in air (obviously less with a dielectric, but you get the orders of magnitude).
Source: I design trains for a living, though on the mechanical side
> ... faulty train signalling hardware on PV46 was emitting erroneous signals in addition to the ones it was supposed to emit. These erroneous signals occasionally prevented trains travelling in the vicinity of PV46, including in the opposite direction, from properly communicating with the trackside signalling system. This loss of communications led to the activation of the trains’ emergency brakes.
Sounds like it may have been sending an (off-by-one, inverted, etc) signal indicating it's on a different track, or moving a different direction or speed, than it was in reality.
While the press release says these were "in addition to the ones it was supposed to emit", it's not hard to imagine that a similar error could happen where the wrong signals were sent in-lieu-of the intended signals, and thus leave trains rolling that should have detected an emergency-stop situation.
They might have problems reproducing such an intermittent problem without running the train. Often times such a problem clears up when someone physically messes with the hardware. Under such circumstances the problem would probably be solved by taking the circuit board that does the communications out and then carefully placing it in the garbage, possibly physically destroying it first.
Not all problems can be troubleshot and there are some things that remain mysteries forever.
It would also be interesting to see how the nature of the cause can explain the one statistical surprise that remains: PV46 is no outlier at all in figure 3, "frequency by train ID". According to the preliminary explanation, PV46 should either be so massively overrepresented there that the bughunt would be over before it really started (it is always close to itself), or it should only be affected by the estimated 5% of outages not related to PV46 which it clearly isn't.
Except that figure shows the frequency with which PV46 experienced an interruption, not the frequency with which it was near an interruption. As stated in the article, "We could not inspect the detailed train logs that day because SMRT needed more time to extract the data", implying that there was no position information for trains that were not actively experiencing a disruption.
Arguably, based on the preliminary explanation, it seems like PV46 might be over-represented (it had 4 interruptions) -- after all, it never travels in the opposite direction to itself.
From the bottom of the linked press release, which summarizes the overall investigation:
"In particular, I thank the engineers and data scientists from DSTA and GovTech respectively, without whom we would not have been able to theorise the possibility of a faulty train, and identify PV46."
This is fantastic. I was in Singapore a couple of weeks ago and after mentioning that I'd narrowly missed out on getting an SMRT job earlier this year we got talking about the SMRT in general. He commented on this exact phenomenon of emergency braking for 'no reason', saying that the people of Singapore were losing faith in SMRT and their ability to keep up the reliability of the network after a spate of these issues due to 'interference'.
It's truly wonderful to see how they went about identifying the problem. This is fantastic work and the team deserves major kudos for nailing this.
Not trying to be too negative here, but why wasn't a broad association analysis done on this dataset first?
The number of outages correlated with train activity should have popped up like a sore thumb, no?
Hindsight is always twenty-twenty, but I feel the author just wanted to show of his cool visualization knowledge, while the same could have been achieved with a much more boring association table.
EDIT: I am corrected by "Rifu". Activity data of all traincars was not available in the first instance, so a broad association analysis would probably have yielded nothing. The type of analysis the authors use seems warranted. Although I'd still recommend doing broad association analysis on any dataset, before moving into complicated visualization techniques.
As per TFA, they did say that the only data they had on hand were from the trains that broke down, so it would have been impossible to even detect the "rogue train" from a simple analysis of the data they had. Though it also said SMRT was slowly extracting the train logs from the incidents so I'm sure they would have gone with that if their preliminary analysis yielded nothing.
I believe you are actualy "right", the article states "we were given ..." but WHY didn't they ASK for single train activities? These data is normally available in any railway, as it is used for several things, including programmed maintenance.
In other words the Authors did IMHO a very interesting analysis of the data they were given, but - had they actually asked for the "right" kind of data - it would have been much simpler and faster.
If you prefer they approached the problem from the viewpoint of data analysts (which is what they are, and evidently very good at it) but not from the viewpoint of an accident investigator.
It was done in a single day, OK, but the train usage data is not "extra data" it is something that is "ordinary data" that you have typically for two to four weeks in advance (as a planning) and daily or at the most weekly afterwards, let alone - in November - the actual usage data for August-October.
Actually I know nothing about the specific Singapore train system, but the basics are "trans-national" and "well established", a metro is an extremely complex structure to manage, but the principles are well established on the experiences of early railways.
You have to manage a time table, the personnel and the maintenance of both the rails (and station and power lines, etc. i.e. the "static" parts of the infrastructure) and of the "moving" parts (the trains).
In order to do so you have "allocation" tables, usually made in advance two or four weeks at least, stating "who" (personnel) and "what" (train) are "where".
These "programs" are monitored and changes to it (think of people not showing up at work, a train having a malfunction, etc.) modified to allow for these impredictable events.
So at the end of the day (each day) you had a printed piece of paper with modifications - if any - scribbled on it.
Nowadays the same thing is done on computers, possibly in a much more detailed way, but the basic data "which train was in service on which line at what time" has been available on paper since day one (or two) of any railway in the world.
While I can (barely) understand how these basic data for the 5th of November was not available on the 5th night, there is no real reason why data since August and until - say - the 4th of november was not available.
As said, the data analysis carried on the data provided by SMRT has been carried on in a clever way, as finding the culprit from that set of data would have not been at all easy through other simpler analysis methods, but somehow SMRT (and LTA) provided the "wrong" set of data.
I don't think they particularly have minute by minute timetables on the MRT, just first and last train times and number of trains operating per line during peak/off-peak periods. On the circle line the trains are driverless and operate with a moving block system (Comms-based Train Control), so the headway between trains is fairly constant, but the absolute timings are dependant on the train ahead travelling at the ideal speed, not stopping for too long etc. (e.g. if someone holds the doors to jump onto the train last minute you'll be delayed as the doors re-open and attempt to close again). So I think you would have to extract logs from the control system or the trains themselves to establish exactly where they were at any particular time. And bear in mind that this information is probably rarely used (at least for this purpose) and may not be in a format suitable for the kind of analysis they needed to do. Funnily enough it was the use (and failure of) of the CBTC system that caused this problem.
Also remember that until they begun the analysis it wasn't clear that a single train was the cause. I imagine they were investigating trackside issues as well as issues with those trains that stopped, rather than an independent train that wasn't experiencing problems.
TLDR; It's not a case of looking at pre-defined timetables.
Sure, but minute-by-minute is not needed.
If you look at the last graph on the page, the one after "The pattern was especially clear on certain days, like September 1. You can easily see that interference incidents happened during or around the time belts when PV46 was in service." it is the most simple data, drawing points (the incidents) over a service timetable.
These data (when the single train was in service or not) must have been available.
I believe that the tricky part was the "delay" between the PV46 passing and (some time later) the "wrong braking" of another train.
I'm not sure if I understand correctly. I assume that the trains are in service the majority of the day. Wouldn't that make it very hard to correlate activity of a single train to the outages?
Not really, I believe the article states that 95% of the outages could be attributed to the single faulty train. That's a really big effect. Correlating traincar activity to outages will only NOT work when the traincar is active on the same track all the time, which it clearly isn't
I was expecting something more like the 'A Subway Named Mobius' short story [1] - it's one of my favourite topology related bits of SF - OK, possibly the only topology related SF?
Yep. Singapore Circle Line is a CBTC (Communication Based Train Control). Trains receive their movement authority through wireless transmission. If due to interferences the transmission is interrupted for more than some seconds, then movement authority expires and the on-board systems will bring the train to a stop through safe measures (typically, application of emergency brakes). It is quite common to have interference issues due to some geographical problem (e.g. incorrect installation of access point antenna) but much less common to have a jammer train. It is also quite impressive that the jamming would be long and wide enough to exhaust all the redundancies put into such system.
Super cool to see how the team isolated the cause. I travel along the same route as well and this train caused half the circle line to be shut down during one morning.
For the next few days after, they shut down the telco signals in the train tunnels to determine if it caused the signal interference.
I heard that they are as competent as private sector.
They are much, much better than that. They hire the best grads and pay lots, but expect a lot too:
One thing that stands out in Singapore is the quality of its civil service. Unlike the egalitarian Western public sector, Singapore follows an elitist model, paying those at the top $2m a year or more. It spots talented youngsters early, lures them with scholarships and keeps investing in them. People who don't make the grade are pushed out quickly.
The government is the highest-status and highest-paying employer in Singapore. The education systems are extremely strong, they send students all over the world for higher education, and those students usually return to Singapore to work for the government. The fact that their Prime Minister was the Senior Wrangler [1] at Trinity College demonstrates a lot about the overall way their government works.
Since moving to the US, my health insurance is handled by the private sector. Competent is not how I would describe it. (The more accurate term is 'Kafka-esque')
trains in singapore themselves are private sector. SMRT is a publically traded company. If their trains keep exhibiting this problem, they run the risk of losing their contract to run the circle line to ComfortDelGro (or possibly an upstart competitor).
Probably the handoff to data.sg was probably done because it doesn't necessarily make sense for them to maintain a separate data science division within the company.
The parallel with epidemiology is striking. And I keep on being amazed by the power of the visualization of data. The problem began to unravel when they "zoomed in" on a pattern that was not clearly visible initially.
Wow what a digital Sherlock Holmes tale! Beautiful story. Incredibly hard if the data was not visualized that way. I learnt some good things. Thanks for sharing here!
Good work, though I can't help thinking that a defective train could just as easily have been caught through performing regular maintenance. Rolling stock and other physical things are subject to both manufacturing defects and wear.
I wonder why the rail managers allowed the defects to inconvenience passengers for so long instead of rethinking their maintenance procedures.
I wonder if some "AI Gen 3.0" machine learning / cognitive machine system could have caught this, without human intervention?
I just yesterday attended a presentation by a ex NSA guy (who's of course pitching his company Adatos, so grain of salt) who claims you could feed the raw data in and the pattern will be found.
You may have heard its previous name being used - iPython notebook. FWIW it took me a year plus to actually start using the name Jupyter. Wasn't until one of my students asked me why I kept saying "iPython" when the tab says "Jupyter"
Jupyter notebooks can run with other programming languages too. Julia, for example, is designed to interface with Jupyter for a more matlab/mathematica -ey experience.
TLDR; There was a single train emitting both correct and erroneous signals. This caused trains in the vicinity to lose data link which triggered an emergency brake safety feature. The article doesn't make clear what medium the trains communicate on which was affected (e.g. rail, wireless, powerline). They have not yet determined what caused the faulty hardware (or software???) on the faulty train to enter this state.
Think you're looking at this story the wrong way; Its not about the fault but the bug hunting process.
I propose:
TL:DR - They had a strange hard to identify bug, but used a limited data set and interesting techniques to quickly find the esoteric cause of the problem.
imho his TL;DR is better than yours, but your improvements could be made to his. In my education we was taught that in science and business, the abstracts should tell the whole point of the story; the "spoiler" should not be saved till the end even though that makes for more fun; this is to allow a busy person to get the takeaway quickly and assess if they need to read all the detail. Whether anybody agrees with that or not, it's been burned into my way of looking at the world, so even when reading this article, all the way through I was gritting my teeth thinking "I wish I had some clue as to what I'm reading about."
so to add in your point, to his TL;DR I would put in "the intial dataset included only the trains that had suffered the fault, but as the fault was caused by a functioning train, a more comprehensive dataset was necessary to find the problem; had it been provided initially, less detective work might have been necessary"
I'm not sure if you meant just the communication system was underground but some sections of the routes go overground (e.g. a large section of East West line is overground).
