I swear this guy exists at every company I've ever worked.
He's the guy you go to when you find some legacy code which you have no idea how the hell it works and end up getting a 2 hour history lesson into a decade of company politics and failed replatforming projects.
"... And the AIX machines were a real beast. We had 20 of those things, and each had its own 20 amp circuit. Had a tendency for the grounding to the steel case to fail. If you wanted to upgrade it you'd first need to grab a pair of thick leather gloves...."
You kids sitting there. All you do is change your EC2 instance size and restart it. Back in my day I had to go and install the RAM by hand. And you know 128Mb of RAM then took up a whole suitcase and weighed more than your laptop
The I fall asleep at my desk, periodically waking up shouting "STOP-A!" and hallucinating about being bitten by thick ethernet vampires.
When the Sun 4/330 I used to have needed a RAM upgrade someone flew up from London to Edinburgh with the RAM and installed it. Fortunately we were in the Grassmarket about 10 minutes from the Castle so he had plenty of things for him to do to keep him amused until his return flight.
Edit: I checked and the max the 4/330 could take was 96MB so I suspect the upgrade was an awesome 64MB. That machine cost more than most cars I have owned - even without allowing for inflation.
I worked in some ex-USSR government facility, and they had an old hp-9000 system, that needed ram upgrade. HP wanted some ridiculous thousands of $$ for like 16MiB of RAM. After some inspection it was discovered that the RAM is nothing more than 72pin SIMMs. We took some ram from nearby old 486 and it worked.
> HP wanted some ridiculous thousands of $$ for like 16MiB of RAM. After some inspection it was discovered that the RAM is nothing more than 72pin SIMMs.
Yeah crazy money. My father used to import RAM when it was really really expensive in the 1980s. He used to take it on planes from Taiwan as hand luggage handcuffed to himself because it was that valuable.
This brings back memories. I got my start in parts wholesale selling 30pin SIMMs in static baggies out of my leather jacket in a Taco Bell LMFAO. It took some convincing to convince the management I wasn’t selling drugs, but the getup helped me to not be noticed by people that might otherwise know that I was carrying thousands of dollars in cash and chips daily.
I would sit there and wait for RAM hungry customers that would call me on my 3 lb cellphone lol. Each one has to buy at least a $0.39 taco so I wouldn’t get kicked out by the manager. They had armed security, who I tipped 20 dollars a day. I hung fliers all over town with tear-off tabs.
Then I went to COMDEX, it changed my life. Within 2 years I was in my early 20s and moving 2-3 million in hardware a year. It was a heady time.
Then, the business dried up overnight with online sales.
Fortunately I saw that coming and had moved into technical services and contracting, and sold the parts wholesale business to one of my competitors lol. The local wholesale distribution biz was dead within 8 months.
Hahaha hooky RAM dealer. Much respect though for being there at the start. I used to do something similar with pirated Amiga games but that was a limited market!
At first, it was an opportunistic grift by a teenager looking for party money.
As prices fell rapidly at the supply level, local box stores would occasionally sell off stock at deep discounts (reflecting the real market price). Being a teenage sociopath, I would buy every last bit of it in town and then coast on the uninformed price perception for a week or two.
Prices were falling so fast at that time that the stores were always wrong footed, so I was able to keep up that racket long enough to get to the bottom of that mechanism and start ordering in bulk directly from city of industry California.
Place was called Ma labs, iirc. After that it was obvious to start moving disk drives, video cards, processors, and later everything else too. I was just exploiting basic market inefficiencies , since the only other sources were the big box stores that were constantly 2-4 months behind the price curve.
Then COMDEX happened. It got me tied directly in to motherboard and peripheral card manufacturers. I also got introduced to the cocaine and extremely friendly booth-girl shmoozefest, and NGL for naive Alaskan me it was heady stuff. The real deals and the points negotiations were made in the afterparties with piles of coke and very persuasive “account representatives”. Memorable times were had in the hedonistic environment of an exploding industry.
At that time there were a lot of small manufacturers in industrial parks. They were usually pretty small outfits usually just a shop with a few hot air stations for warranty work and a wave soldering table, 15-20 employees at jet labs, for example, which was one of our big suppliers of motherboards, video cards, modems, lan, and other peripheral cards.
My activity festered a plague of tiny pc builders selling systems, and launched the careers of at least 50 technicians and small businesses. We used to put on big public LAN parties and public PC repair clinics as promotions for our “dealers”. I worked with a training/testing outfit to get new dealers A+ certification to reduce the damage to society and actually grew into a semi responsible citizen during that time.
Fond memories but I’m glad to leave that hustle behind.
After that, I unwittingly became a software developer for a major criminal organisation. I thought they were a car dealership, but gradually it became apparent as I built out their backend that the dealership was just a small part of a very large laundromat.
It took me years to extricate myself from that and for years afterwards they would give me gentle reminders that they were keeping an eye on me. Free 500 dollar an hour lawyer’s though. They never let me get near a courtroom or a police interaction without “proper” representation showing up even for years afterwards (statute of limitations?)
The fact that they always knew when I brushed up against the legal system in any small way, even traffic tickets in rental cars, gave me great respect for the scope of their concern. That and their obvious love for mid range turbine helicopters.
I worked in a shop in the early 90s with a bunch of Suns, and we had hired this new sysadmin. One day we noticed that half the RAM was missing from a workstation, and few minutes later noticed half the RAM was missing from all the workstations. Turned out the new sysadmin was a cokehead and had stolen it.
Oh man that's funny. Only because something similar happened to us. Hired new cleaners circa 2001. Got into the office one morning and all the PC cases were open. All the RAM and CPUs had been stolen.
In the mid-90s, when CPU heatsinks didn't need to be screwed in, thieves discovered that the new ZIF/zero-insertion-force sockets also make good zero-removal-force sockets.
When I was in Korea in the 90s you could get LG RAM for about half what it would cost in the US. I had a scheme to buy a bunch of it to sell when I came home, but in retrospect it would have been the nerdiest case of "guy goes door to door trying to sell rapidly expiring meat".
I went to the closing sale of a local computer store that had been in business since XT days.
Owner lamented finding 4MB sticks of RAM that had fallen behind the workstations that had cost hundreds of dollars back in the day, as he was selling (and this was a while ago) laptops with 1-2GB of memory).
Continuing with the Monty Python vibe, the first code I wrote was in machine code (ie hex, not even assembler). Though I worked with a guy who wrote machine code on a magnetic drum and they used the delay of the rotation of the drum for their timing. The good ol' days.
My thumbs still have blisters from manually stuffing memory expansion boards with 1Mb of DIP RAM for our 80286 PCs. We were doing image capture and processing, long before it either usable or useful.
"Back in my day, we only got one compile a day with our punch cards. Now you can compile willy nilly without having to put in the forethought we did" - ex coworker
> Back in my day, we only got one compile a day with our punch cards
You were lucky! In my computer class at school in 1979 we got one compile a week. Punched into cards which were taken to the local insurance company who actually had a computer.
Stop-A is the key combo you hit on Sun workstations to drop into the ROM monitor. Usually when they are locked up solid with a crashed Xserver and need a reboot.
I did a real internship (as in it was for my diploma and lasted only a few months) in a medical facility operating Big Machines. I remember a guy soldering a new battery to some old DEC/Digital hardware.
"They were afraid to turn off or reboot the DEC Vax for fear that if the disks stopped spinning they wouldn't start again. I saw it once at the retirement ceremony and it didn't look like a server but more like a set of industrial washer and driers. The users transferred files using the Kermit protocol and printed locally by a telnet client that could understand the special characters that redirected character output from the terminal to a lineprinter."
An old IBM tech told me that hard drives failed when power cycling equipment could be temporarily resuscitated by placing the drive on the floor, twisting it hard by hand, then as quickly as possible connecting it back into the machine and powering on. Apparently getting the platters to spin helped them get around a worn out motor that couldn't make the initial spin-up.
I remember upgrading from a 4.3 GB HD to a 20 GB HD around 2001. I had the old hard drive sitting on my desk, and the new hard drive sitting in the case, both attached to the motherboard via IDE ribbon cables.
Most of the way through backing up the 4.3 GB HD to the 20 GB HD, I heard a screech and the old hard drive twisted slightly on the desk... conservation of linear momentum. The drive was visible to the BIOS, but refused to spin up after that.
I put the hard drive in a ziplock bag, put it in the freezer to give the parts slightly more clearance from thermal contraction, and slammed the hard drive nice and hard on the desktop to free the stiction. That revived the drive long enough to finish my copy.
I did have backups of all of the really important information, but restoring from a stack of floppies is more tedious, slower, and less fun than percussive maintenance on a hard drive.
Back when I had an internship working on MEMS gyros for GPS guided mortar rounds, we had to sometimes perform similar percussive maintenance if static electricity had caused the moving parts to contact the substrate. We took the gyro, and smacked it hard on the desk in an "eyeballs out" orientation to give it something like 10 to 100 Gs of acceleration in an attempt to un-stick the MEMS gyro.
Yeah, did same in 1999 or 2000, but we _did_ _not_ take the HDD from the freezer, just pulled the ribbon and power cable through the narrow slit between the door's rubber gasket and the fridge's body.
Yes, stiction is a real issue in old hard drives. In fact something that used to work well if you were cold booting an old server even in the mid 90s, with the expectations of an immediate migration, was giving the drives a good sharp slap to help break stiction so they could spin up. The only issue is it could cause a head crash. Another fun trick from back in the day was sticking hard drives in the freezer to cause the platters to shrink enough to dislodge the head and restore the air gap if you had a head crash so you could try to recover the data.
As much as I love hardware, I much prefer our current solid state wonderland.
> to shrink enough to dislodge the head and restore the air gap
Huh?
What I heard is what a cold drive would be a bit more magnetically stable and IMMSMV that surely worked, because I never had a drive with a stuck heads but I did had drives what would just abort the read or return gibberish, but after an hour in the freezer they would read just fine, till they heat up again. That's for the drives mfg after 2000, if that matters.
My understanding may be incorrect, after all the freezing the drives trick is mostly something that was shared among sysadmins like an old wive's tale. Nonetheless, it does work (or did).
The way I had always understood it was that if you had a head crash (which was caused by the head physically contacting the platter, overcoming the resistance of the air gap between the head and platter, usually due to physical impact) that the magnetism of the head would prevent it from lifting back up on its own, and that freezing the drive would cause the metal in the platter to contract away from the head, which was restricted in movement by its armature, thereby restoring the air gap and lifting the head away. If you started the drive spinning before it heated, the head would stay out of contact and you could successfully read data (some of it, for awhile).
Every Friday I had to take a bunch of magnetic tapes to the bank so they could look after our backups ( which was a good excuse to pop into the pub next door to the bank...)
Oh, and we never ever tested that the backups actually worked
My first job out of school was engineering consulting, writing software models of pre-production military radio hardware. The client was a military contractor, and they paid a crazy sum in order to have 3 DAT tapes: one tape always in the drive used to make the daily backups of the shared working directory, one tape always stored in a bomb-proof bunker, and one tape potentially in-transit via courier. I think it was once a month that the client's sysadmins rotated in the "unused" tape, sent the latest tape to the bunker, and waited for the old tape from the bunker to return as the "unused" tape.
Due to absurd government requirements, we were not allowed to use any version control software other than PVCS, and PVCS could only be used for storing official releases of the models. So, the last person in the office on Friday needed to make a dated zipfile of the shared project working directory.
So, one day I convinced the project manager to just let me rename our working directory, create a new empty one in its place, and tell our sysadmins that I accidentally deleted the contents of our working directory. It turns out that the client's sysadmins hadn't set up anything to expunge the oldest backup and had dutifully ignored the alarms that the tape was full. So, the latest available backup was from about a week after they last changed tapes!
The senior sysadmin prominently had a sign on his cubicle reading "Programmers are the ditch diggers of the 21st century." Those same sysadmins were supposed to keep the server room door open while I reinstalled Solaris on a Sunfire V1280 via serial connection from my laptop, because there were cables carrying TSSCI (Top Secret - Secret Compartmentalized Information) traffic. However, the V1280 consumed so much power that the server-room was under-cooled, so the sysadmins illegally shut the door to keep their cubicles cooler. They had top-secret clearances. They knew their obligations to keep me away from those TSSCI cables. Those lazy lazy arrogant sysadmins.
Anyway, unless you have recently passed a restore-from-backup drill, you don't really have backups. Even if you pay absurd amounts of money for sysadmins with top-secret clearances and absurd amounts of money to store your backups in a bomb-proof bunker.
Do you know why I am giving you the two hour history?
First, I recognize that I need human interaction, but do not really like people.
Second, I am hoping that you will learn & understand instead of just memorize & regurgitate so in the future you can resolve it yourself instead of bothering me.
Now, go away; I had my week's fill of socializing.
For me, it's usually trying to rapidly justify the existence of whatever Rube Goldberg machine I'm having to explain. There's always reasons behind the madness, even if they're not particularly good ones.
> Second, I am hoping that you will learn & understand instead of just memorize & regurgitate so in the future you can resolve it yourself instead of bothering me.
Where does all this hope come from? I’m barely 15 years into my career and I’ve nearly given up…
I've started to attempt to gently grind that into juniors. After a certain point, hope is a vice you cannot afford. It is simply something you outgrow, if only because of the unceasing retirement from the mortal coil of those that can bail you out.
Hope is for the young and inexperienced. Everytime I come to your rescue, you'd best pay attention. I have an expiration date.
I’m not trying to explain why you shouldn’t change it. I’m trying to help you understand what competing forces made it that way in the first place so you can decide if any of that’s still relevant.
I’ve seen a few of those. Some are worth their weight in gold. Most are set in their ways from the trauma of trying to make everything not fall apart for so long. Almost all have stagnated and are incapable of bringing in new ideas. Almost all are husks of human beings from putting up and personalizing so much of company dysfunction
This reminds me, in a sad way, of a friend of mine who stayed on when a startup we worked at was acquired by a huge corporation. He has an amazing mind for technical detail, and at the startup, he planned and executed some hugely ambitious technical initiatives. But management at MegaCorp treat him as an obstructionist. They want to announce an initiative, launch it, declare it done, make up a big dollar figure for the "savings," and gather a promotion. They don't need the initiative to make sense or work in a technical sense. But he understands the immensely complicated legacy context, all the work required to accomplish anything, and all the things that will break if they leave out parts of the work, and he won't keep his mouth shut. He would love the challenge of running a big ambitious project, but the only way to get anything big approved is to drastically underestimate the effort, so he's stuck rearranging deck chairs until they inevitably eliminate the product and lay him off.
A real life application of Odysseus's Oar. I have mentored junior devs who have "heard" of SVN and also think CVS is just a pharmacy. It was a real gut-punch the first time.
They also tend to be brilliant engineers. It’s a heck of a lot easier to always be writing 1.0 of something. It’s a hellaciously difficult engineering task to improve a system that’s been improved upon for decades. These folks should be admired and respected, they’re by far the most crucial people at any company older than 30 years old. Instead we get condescending garbage like the linked article.
I just wish hiring managers saw it the same way. It's hard to explain how your last major project was a web enhancement that would have taken a couple days on a virgin codebase, because for you it involved a month of excavating through multiple strata of code dating nearly two decades old, spread across three codebases, two backend languages, a code generator which no one working there understands anymore, and a library who's documentation now only exists on the wayback machine.
A secret I’ve learned is you only need one job, so you don’t need to take just any job or be hireable at every job. It might take longer, but finding a manager that “gets it” is very important. The most important thing is to find someone who recognizes further that a team is more than the sum of its parts, that each individual is unique and not a cog to be evaluated on a metric rubric and forced into compliance with some idealized engineer. They do exist - I was one for many many years. I’ve recently given up in disgust though and switched back to IC, but senior enough I can shape how managers manage engineers. Regardless, my advice is, if it’s helpful, find the right job with the right team and the right company. It’ll take longer, may pay less at first, but finding the right place that respects work at its value not at its marketability is key to any successful career.
Or a library who's code now only exists on the wayback machine... and of the modified version that you actually use only a single copy exists on an old machine that one of the devs in the company never bothered to hand in for recycling...
Never mind that it's part of a decade old contract stipulating a quarter century support...
I wish people like this would architectural decision records so people inheriting it later wouldn't be left in the dark by the decisions made in the system
Because sometimes on the outside systems look confusingly written, but it could be a reflection of the scope of confusing business rules to begin with that they had to scope out
What I propose wherever I go is writing a deprecation plan before production release. It may not be kept up to date over the lifetime but at least a plan for what it would look like to exit to another platform exists documenting assumptions and one way doors, why things were done, and what a replacement would look like and why. I’ve never had great success though, people are too eager to make the next 1.0.
It doesn't seem like a bad way to go. If the company is stable enough and you can cash in your engineers salary for 25 years and have crazy good job security.
Seems better than being a 50 year old trying to wow a 28 year old interviewer with your skills on whatever hot new framework just came out.
...these OS/2 machines were once interconnected by a networking technology called 'token ring'.. no joke. It was basically an entire network where if one single node on the network misbehaved it would bring down the entire network - they didn't self recover either! you had to reboot them all. I remember once when I misconfigured a token ring card on a PC and it brought down the mainframe! It shut down the bank for 30 minutes"
I used to work in a place where every office on the floor was connected to the same ethernet hub. We also tested a server product that became a DHCP server if you checked the wrong box, so periodically everything would stop working until the team lead came out of his den and went to yell at the new guy.
What do you mean hubs? Back in the day we manually pierced thick Ethernet coax to get access; of course you could ruin the whole segment running tens or hundreds of meters if you didn't take care.
Hard to be anything else if the entire job is spinning up a yet another microservice and then spending most of your time debugging in production because shit keeps breaking all the time.
Maybe we're thinking of two different archetypes, but I wonder if their "hard not to love" nature is part of the survivorship bias of why they're still there.
They are worth their weight in gold, truly. I had the pleasure of interacting with one such man at my previous company weekly for an extended period of time while planning/designing a rewrite. Listening, thinking, and questioning his stories and perspectives allowed me to formulate problems and architecture in a more first-principles, fundamental way and I believe that experience really helped me mature as a young engineer (as well as the project).
This is where another one of your Pied Piper boxes would go. Okay. Let me show you the next location where we would install one of your Pied Piper boxes. ... okay
Not necessarily hate...there's always the 1-2% of new hires that are highly impressive and gives us hope for the future. They tend not to last too long. The rest seems hell bent on never even cracking open documentation, and keep asking "where are the architecture videos?" (at least where I work).
This is actually something I've identified as a major con of working at startups (there are a lot of pros too, and I'm at a startup right now that I love): It is impossible to have these people, because the company is just too young. And also I think this kind of person doesn't usually join a startup for its first, like, decade.
"Hard not to X" is not the same as "Easy to X". It may be easy to do something but also easy not to do it. It would be easy for me to yell "Elephant!" right now, for example, but also easy not to do it.
I often search for a clearcut answer to a technical question and I'm met with a 2 hour history lesson into a decade of company politics and failed replatforming projects.
Yeah, thanks for telling me why John from accounting was a dick 10 years ago and you had to code this module in a certain way. I really don't care. I'm new to the codebase and I just want to know how it (the codebase) works.
I'm currently in this situation and a colleage never gives straight answers to anything. It's always some little rant about something and when it's done, I still haven't got my answer.
> I'm new to the codebase and I just want to know how it (the codebase) works.
You should want to know _why_ it works that way too if you want to do any meaningful work with it. Context matters. I’ve seen many cases where the way something works seems dumb, only to learn later they had already tried the “smart” way but ran into some obscure problem which the “dumb” way solves.
This is important. As soon as we come to the understanding that the coders that came before were not all idiots then we are forced to ask the "why". There is generally a decent reason why something is coded the way it is. Could be as simple as it was an emergency and meant to go back and fix it but never had the time or it could be a valid business edge case that absolutely had to be wedged in. There is almost always a why. Otherwise you tell everyone about a fix you made that 3x performance and you see their faces go gray as they explain you just shut down some archaic but mandatory process in Singapore.
It probably depends on the person, but I personally enjoy such tales. Gives me more context about why a piece of code was written the way it is now. But then again, I also enjoy scrolling through the commit history of a repo like it's an archaeological dig site, so I might be the weird one here ahaha.
It's full of "fix" or "fix typo" or "update blah.c"... tiny commits straight to the master branch without CI.
Anyway, I really don't care about the past in this way. A simple "accounting needs this for that" is enough for me. No need to explain that Adam was getting divorced at the time, so he was grumpy, and, and, and...
Easier said than done, but try telling them that. "I appreciate you sharing context, but it's too much to take in at once. It would work better for me to get a more clear-cut answer."
Sometimes, the 2 hour story isn't worth the time it's told. Sometimes, the 2 hour story gives you the insight necessary to satisfy Chesterton's Fence. It seems like in this instance, that's not the case, but I'd definitely encourage you not to dismiss stories in general because you don't think you need the history to achieve your goal.
Because I don't want to hear about personal stories? How is that related? I do very much want to know how code works. It's the very core of my argument.
And this propagates the problem. Getting an answer without understanding. Sometimes the history is necessary in order to not repeat botched attempts at "fixing" something that people have already tried.
He's the guy you go to when you find some legacy code which you have no idea how the hell it works and end up getting a 2 hour history lesson into a decade of company politics and failed replatforming projects.
They're hard not to love.