The reason mainframe persists is it's a pretty slick development and deployment environment. A lot things you might cobble together as dependencies - like maybe a database or a message queue, or observability facilities or even deployment strategies like hot-hot deployments - they're all just built in to the platform. That means they're trivial to consume and they're fully supported by one vendor. It's like the worlds most comprehensive application development framework.
Going back to the hardware that everyone likes to focus on, it's less radically different from normal servers today than it was historically. The mainframe today is a 19 inch rack like any other. By that i mean it is not only a 19 inch rack like your x64/ARM servers are but also the same power density (32 or 64 amp racks), cooling requirements etc.
The most interesting bit is the software, not the hardware. There are cool hardware aspects too - but focus on them and you miss the real reason these things are popular in certain environments.
Most of this Cloud technology we like to go all atwitter about is a retread or a redesign of software that's been on mainframes for thirty years. The old farts must laugh themselves silly looking at us acting like we discovered a large white object orbiting the earth.
Oh yeah, in a past life I was a CPU logic designer for walk-in, refrigerated mainframes. I was always amused by youngsters thinking they had invented things like pipelines and branch prediction. Seymour Cray and Gene Amdahl were prolific inventors.
I posit that this retreading of old paths was very good for the industry.
All the truly important CPU design features were first invented using discrete logic for mainframes and minicomputers. Then, 20 years later when it became possible to fit them onto a single chip, they were all out of patent, which is precisely what made the early microcomputer industry so competitive. When none of the important stuff could be patented anymore, you could only compete on merit, not by getting a government enforce a monopoly on some critical piece of your design.
I've long been a proponent of computing history as required part of CS.
Progress is supposed to work by standing on the shoulder of giants. But that only works if engineers are aware of what came before, how it worked, what was good and bad about it.
In the software world the same things are continuously being reinvented but the lessons of the past are not integrated because the inventors have no knowledge of the past 10 times the same thing was done. So every lesson has to be learned anew in production. That's not progress.
The distributed computing class had a history of distributed computing component.
But it was an elective, and as I found once I got into the real world, one that most people either opted out of or didn't have the option to take. I spend way too much time providing Cliff's Notes to a class I took almost thirty years ago.
Some places have them. I took an elective class at university which focused on open source philosophy & licenses. Read and discussed a bunch of papers written by the guys involved in the early open source movement. Stopped short of code review though: which would have been a nice addition.
A very very good paper, and one that was clearly a passion project for the prof.
I'm not old enough (or been in the appropriate industry) to have worked with mainframes. I started in the heyday of the workstations (Sun, HP-UX, AIX, SGI) as a reaction to the annoyances of mainframes.
Why pay for disk and compute access when you can have a workstation! Why have limited access to the system when you can have your own! Was the vibe.
It is amusing/sad that in the AWS world we're right back to basically mainframes. Centrally hosted massive compute that you don't really have full control or access and have to pay for use time.
I wonder when the next swing back to personal computing arrives?
You're old enough to have noticed that we cycle between client-server and peer-peer about every decade. Sometimes it's hardware, sometimes it's software.
Had the US government not forced IBM to get out of the centralized provision of computing by consent agreements and eventual divestiture, and had the US government not forbidden the Bell System to enter the centralized computing business by the FCCs Computer Inquiry 2, its likely that computing and networking would have evolved in a more rational way that did not include personal computing.
All this is true but you forget the cost. Mainframe do not allow to start small. I have no idea how much the cheapest mainframe cost but it's porbably more than a dirt cheap x86 server with a bunch of free software so important for student projects, training, hobbies, start-up and cost constrained industries.
Sun was way ahead of its time. Too far ahead in fact, in the sense that they brought products to market that the market didn't yet realize they wanted.
The Javastation network computer for example, it was a bit like a Chromebook (though incredibly slow -lol)
I think Sun should probably have bought Azul systems during the tech downturn. Even if it turned into a reverse merger (which history suggests may have been the case), it might still be around today.
> Going back to the hardware that everyone likes to focus on, it's less radically different from normal servers today than it was historically. The mainframe today is a 19 inch rack like any other.
The Main Frame of yore was ... one of the 19 inch racks the computer was made out of (the main one, typically with the ALU and registers in it). And those racks were used because that was how phone systems were built (phone systems had a lot more repeated / regular structure than something like a computer, though of course everything has converged). The "frame" nomenclature came from the phone system too.
So "normal servers today" got the rack from the mainframe, the mainframe didn't adopt it because of how datacenters were built. All the familiar stuff (raised floors, chillers, backup power, fire suppression, controlled access doors and so on) come from the mainframe world.
The granddaddy of mainframes, System/360, was never in 19" racks. IBM first introduced the use of 19" racks in their midrange lines. What we currently call IBM z didn't adopt standard racks until the 21st century.
The hardware is quite radically different. It's not the mechanical form factor or bulk electrical and cooling of it, but the silicon.
The processor checkpoints state, and if an error is detected it can be rolled back and the checkpoint moved to another physical processor and re-started transparently to software. Stores are tracked all the way down to "RAID memory" (actually called RAIM), whereas in other CPUs they get punted off at the L1 cache. Practically every component and connection is redundant. There are stories of mainframe drawers being hauled to particle accelerators and the beam switched on while they're running. Quite amazing machines.
Not to say that has more value than the software (I don't know one way or the other), but the hardware is no gimmick.
I think its hardware is what enables such reliable software.
Hot plug CPUs, memory modules, and DASDs. All survive the failure scenario gracefully.
Good luck trying to do that on a rack of x86 servers.
I think as in most engineering things. They do a different trade-off and cater to a different niche while x86 servers are the mass market consumables that most workloads -should- be using. The mainframe CPUs go for the widest IO capabilities, insane cache sizes and hardware offload because that’s what their target audience(banks, insurance companies, airline ticket systems) need.
Their parallel sysplex clustering solution is an engineering marvel but also tightly coupled to their hardware.
In some ways, the IBM mainframe is the Apple of niche critical enterprise computing. They are also the ancestor of cloud computing in paving the way for “pay per use” cost models and multi tenancy.
x86 just made it so the whole computer could drop out of the “cluster” which greatly simplifies the hardware design but the result is NOT a “virtual mainframe” as the software becomes exponentially more complicated if you want “all power”.
In a way, yes, swapping out entire computers are how one is expected to work with x86 machines. Ask anybody running a critical state full workload(Relational database with lots of writes for example) and they will tell you how nice it is to have a system so reliable to run their writes onto.
Running a Postgres cluster or a Kafka cluster on a bunch of x86 machines sure gives you scale, but definitely brings the pain of handling availability scenarios for write consistency. Distributed locks and distributed data stores that use Raft/Paxos serve as a good reminder of how hard it is to achieve them.
If IBM could do it at hardware level, some Enterprise customers are willing to loosen their rich wallets for that comfort.
It is possible to move that complexity into the platform and outside the applications themselves. When I build a web application I don't need to do anything special about failovers, as long as I remember the local filesystem and state is fungible and it only is persisted when I persist it someplace else (that also employs the same safeguards).
I liked to use Google's App Engine classic environment (the Python 2.7 one) as a good example. It really hammered down some good practices (such as keeping the filesystem read-only) and gracefully handling degraded modes such as a datastore that becomes read-only for no reason or the cache that's supposed to be there that just isn't. It was a great teaching tool.
It persists in many places due to the deeply ingrained belief that, for one reason or another, it would be technically, practically, or economically impossible to migrate functionality off the mainframe. Somehow the people in charge of these systems have managed to convince large enterprises of this for decades. And now we are in a situation where their long held beliefs have become true because nobody is around that understands how or why the software works the way it does. It's a disgrace.
> It persists in many places due to the deeply ingrained belief that, for one reason or another, it would be technically, practically, or economically impossible to migrate functionality off the mainframe.
It is never technical or pratical reasons, always economic reasons. To use contemporary vocabulary, you "just" need to replicate a "multi tenant HA cloud environment" to migrate from a mainframe at great cost.
> And now we are in a situation where their long held beliefs have become true because nobody is around that understands how or why the software works the way it does.
Statistically unlikely, most of the business software that goes onto said mainframe is in fact very simple and straightforward and can be easly ported out, however, the real deal is all the integration and high availability logic built-in into "the mainframe platform". You're not porting that out without replicating said "multi tenant HA cloud environment" at great cost.
>most of the business software that goes into said mainframe is in fact very simple and straightforward
Yeah no. I worked on mainframe code for 15 years of the last 20 and can tell you that you’re wrong. Even reasonably simple business lines have cobbled together insane cobol, batch and cics screens that would make spaghetti blush.
When you open up a 100,000 line batch program that’s just 1 of 30 cobol programs in that run which is a mess of gotos, global mutable state and basically every single other generally frowned upon software practice we know about, you start to understand the issues.
Moving off of mainframe is practically impossible because of it. That other person being voted down is actually correct, nobody in the organization actually knows how any of it works.
When organizations try to transition away, what ends up happening is they get these insane promises of “5 years” from vendors (like IBM. Ask me how I know!), but then it takes 10 years to transition single systems.
Nothing is documented. So you have business people who have no clue how anything works providing high level work flows. But then there’s always 50 random edge cases or interactions they never considered cause the 40 year old code just does it.
IT contains most of the business knowledge, but the mainframe grey beards are… stubborn. The business hangs on to these people because they’re seen as insanely valuable, not realizing that the grey beards have actually been actively suppressing knowledge to only their hidden documents, and they’ve got 8 years till retirement and are going to make sure that their job exists at least until then by actively sabotaging efforts to change. But also, for some reason, the business manager love the saboteurs. It’s insane to watch.
> But also, for some reason, the business manager love the saboteurs. It’s insane to watch.
Because they keep the shit running. If you're a bank and responsible for trillions of dollars in assets and billions of dollars in daily transactions, the absolutely last thing you can afford is downtime or, even worse, data loss. You'll not just have to handle a ton of angry customers, but in the worst case attract regulatory attraction or getting dragged in front of Congress and the courts.
Here in Germany, Deutsche Bank is integrating 12 million accounts from Postbank to their own IT systems, and it has been an utter utter shitshow [1], attracting a lot of bad press and the banking regulator BaFin is also investigating [2].
What happens when there are new requirements? An example, fiscal legislation changes and programs dealing with money have to adapt. Of course a way out is to lobby for changes that don't touch the software, but what happens when the software must be touched?
The estimate for carbon tax legislation was 700 hours and it took 1600 hours and didn’t work correctly for 9 months following.
Edit: and for the record, at that point, we were already splitting off carbon transactions for their own processing because, surprise, energy has lots of regulation.
The risk in a reengineering project that comprehensively replaces the mainframe is enough to discourage most people making that decision. It could be a $3M project that pays pack in 4 years after scrapping the mainframe and running out the maintenance contract, or it could go over budget and take forever to pay back. Your IBM rep has undoubtedly run the numbers.
I'm not sure how much water the multi tenant HA argument holds, these institutions are already doing that on AWS / Azure / k8s for the more modern apps. Rewriting all the old crusty COBOL code might be a different story.
Getting stupefying amounts of I/O on AWS is as simple as turning a knob and getting out your credit card. The kind of throughput quoted in the article—48GB/s—is achievable on AWS for $250k/year, list price.
But let's say you want that data compressed and/or encrypted, you're going to need an awful lot of compute to exploit that 48GB/s if it's possible at all.
Whereas on mainframe the service processors can deal with it transparently to the CPU using specialized hardware.
And let's say it's financial transactions - fixed-point decimal arithmetic. On mainframe, that can be hardware accelerated too.
Yes, and it's interesting how Intel is moving in that direction with their Xeon line, integrating the QAT peripheral for compression and encryption offloading. Unfortunately these things are unicorns and that aren't generally available in the cloud, either.
Well, the general approach over the past 30 years or so has just been to mostly go with generic x86 hardware which would get predictably faster year-over-year and software could be written pretty generically for those platforms. With performance growth slowing, one of the answers has been to have more heterogenous and special function hardware and the software people will just have to deal with it.
Getting (and maintaining) a local mainframe in the first place is also not cheap though. I won't hazard a guess to which one might be easier to keep up and running, but at least in my bubble it would be 100x easier to hire more k8s admins than to hire qualified mainframe admins.
>> It persists in many places due to the deeply ingrained belief that, for one reason or another, it would be technically, practically, or economically impossible to migrate functionality off the mainframe.
> It is never technical or pratical reasons, always economic reasons. To use contemporary vocabulary, you "just" need to replicate a "multi tenant HA cloud environment" to migrate from a mainframe at great cost.
When your business processes are so ossified by a mainframe that you have, say, groups preparing Excel spreadsheets -- with data formatted from one mainframe screen in 3 columns -- for use by another group to input back into the mainframe -- with the data reformatted in 4 columns -- you know you're never escaping the gravity well of the black hole at the center of IT that is "The Mainframe." This is just how it's going to be until the company is so technically irrelevant that the remains are picked up in a fire sale by some PE to sell off to some other, tangentially-related company, solely for the IP assets.
Or the people in charge are of the belief that things work and they're a tiny part of the overall organization's costs--and any massive migration will run over schedule and budget, won't necessarily be better at the end, and will be unnecessary and disruptive. They may or may not be correct but it's not an irrational belief whatever those advocating for moving everything to a distributed platform think.
Have you ever migrated a decades old application off of a mainframe?
Most of these kinds of systems have decades of codified knowledge that applies the legal requirements of many disparate jurisdictions in these systems. Laws vary from city to city, county to county and state to state, and that's only for the US. Some of these systems implement code that applies legal requirements for dozens of nations.
In general, you're going to have to spend at least a year just to document the existing system. Another six months to a year generating test cases and scripts for validation. And that's before even starting to build the new system. And that is ALL optimistic, assuming everything goes right.
I had to once write a relatively simple application for "vacation bidding" where in employees will get a position to request certain weeks off during the year. This was a legal requirement and happened to involve the company, two other companies that had been merged in and three unions. Each with different rules regarding how seniority is calculated. The contractual specifics were over two thousand pages long. That's just one very small piece of software in one of these very large companies. It took the better part of a year to implement. Now imagine taking that a couple decades later after it's evolved and migrating to an entirely new platform and paradigm. Now imagine the cost to do that.
Migrating a suite of programs written w/ COBOL, CICS, DB2 from z/OS to Linux is fairly straightforward. MicroFocus COBOL compiles IBM COBOL, DB2 runs on Linux or you can swap out another RDBMS. The CICS part requires a simple runtime easily implemented in C plus a new scanner to replace the calls in your programs.
I was part of a 4 person team that knocked this out in under a year. 1,000's of mainframe programs running w/o change (at the time) on Unix. This was in the 90s.
> DB2 runs on Linux
The DBA's I interned with would yell at you for this claim. They'd say UDB runs on Linux, but DB2 only runs on z/OS.
I worked with both DB2 & UDB during my internship 2 decades ago, and while largely compatible with core SQL functionality, that changed quickly after you got off the "normal" path. UDB was released at a much faster cadence, and if you developed to the latest features of UDB, you'd often find yourself unable to deploy against DB2.
A few of my last projects as an intern: 1) storing BLOB objects in DB2 from a .Net app and 2) porting an "interactive" batch-processed COBOL app to ASP.Net form 3) getting DB2 Connect clustering on windows working (I never did get this working despite spending an entire summer on the phone with IBM support).
I've some fond memories of working with the mainframe. Like the time I managed to crash the entire development LPAR with a specific SQL query that broke the DB2 query optimizer... The phone call from the NOC was immediate and went something along the lines of "I don't know what you did, but don't do it again, you just took down all of development". Development was something like +10k users...
It'd probably be more accurate to say users than developers on the LPAR. I won't name names, though. It was at a Fortune 500, though. Manufacturing sector, not finance.
Worth noting, in the 90's general computing was still increasing at a very rapid pace, where you could replace a late 80's mainframe with an early 90's x86 or PPC server. You won't have the built in layers of redundancy, but that isn't always necessary for any given environment. The cases of many mainframes still in use today are on much faster/newer systems where that critical uptime is much more important.
I'm not really a fan of mainframes, only pointing out that the companies still on them are generally on them as a direct replacement to general purpose servers isn't purpose fit, and conversion is excessively costly, even compared to IBM mainframe contracts.
The env I helped set up was online only, no batch jobs. JCL is essentially a declarative shell language, so it's not that hard to map to Bourne shell et. al.
People may assume that a lot of this mainframe software is "running just fine" but nothing could be farther from the truth. Often you will find that this software was fine when it was written 40-50 years ago, but now it has a lot of deferred maintenance and upgrades which are now a "big problem". Is it a problem that an application that can move millions of dollars is only secured by a strictly 8 character alphanumeric password? You be the judge.
Certainly. A good reason for a migration is that we are in a place where we need to do something major--whether that's modernizing the software on the mainframe that isn't doing the job, moving everything to new platform(s) (including potentially cloud), or some combination of the two--the latter being a pretty common strategy these days. At that point, just adding more duct tape is probably the wrong approach, at least in the medium to long term.
You have no idea. This is not some database field that you can just alter with one statement and update a field on a screen. Things like this get hardcoded into hundreds of thousands of lines of assembler and cobol and need to be changed and tested everywhere. With no automated testing or other tools to mitigate the risk. It is an absolute nightmare. And 5 or 10 years from now it will be way worse because working on a mainframe is a deadend career and very few people are learning it.
I’m not saying it’ll be easy. But what’s even harder is rewriting 50 year old software, with all the bugs that others relay on now (both people and software), wanky and complex business logic, etc, etc.
I’m not saying that you should stick to mainframe. But rewriting it because you need to support strong password is not a reason to do it.
Over my 30 years, mostly in Fortune 250's, I've seen 3 mainframe migrations fail. If you're part of the team, the trick is to get another job before people realize -- in year 4 of your "5-year" migration -- that it was never going to work, because managers in other parts of the business will never allow their headcount to be reduced by better IT workflows.
Having worked adjacent to some of these kinds of projects, can't upvote this enough. Not to mention that business doesn't take the suggested 1-2 years to simply document and define test cases for the old/new system to ensure such a new system works, and simply start working on the new system based on what the business/da types drew up. Missing literally tens of thousands of legal edge cases defined in the legacy codebase.
Some people just like to see the world burn. Move fast, break things. Like my bank, ATM, pharmacy, shipping mgmt, hospitals, paycheck services, utility grids...
And then in 5 years someone can rewrite it all again because something else will be cooler and my code will be technical debt.
Well, for one, lack of capable personnel. Nobody learns IBM Mainframes in school, and most jobs for them want experience. Furthermore, they're not sexy but boring, with paywalled docs and similar shitty "enterprise" techniques, making it so very few people actually want to or manage to learn them. And it's a dead end, career-wise (very few companies use them, so changing jobs isn't as easy as more widely used tech).
So what will happen when in 10 years the last people you have that know how it works are retired?
Which part is untrue? That there isn't enough personnel or that it's a severely limiting career choice? Because what you're saying confirms the first one, and for the second one, not a lot of people dream of working on legacy stuff in a big organisation, and those are your only choices if you're a mainframe programmer. Say in the US, for mainframe work, there are what, 30 banks? 5 airlines? 20 insurance companies? Not a lot of choice, and it's doubtful they'll offer "perks" such as remote work.
Lots of government too. I worked at a government agency that ran it’s business off of mainframe. I worked there as a web developer and we had to consume data from the mainframe in our web apps. Very tedious process.
Yeah, the hiring process is often 6+ months long for half the pay of private sector jobs, and 1/5 the pay of the consulting companies they bring in to do the bulk of the work anyway.
They are super wanted but it's true that it's very hard to get into.
Want to learn Windows, Linux, cloud whatever? Just build a cheap home lab and play to your heart's content. Because really, training is not how you get good. Nor is certification. It's the hands on experience that comes from running into the crap the vendor doesn't want to tell you. All the little annoying flaws they keep out of the documentation under pressure from marketing.
Good luck doing that with a system Z. Try to find one on eBay for peanuts the way you can get a nice ESXi server.
The greatest hurdle for a project to be replaced is that if its works. I assure you that the users of IBM mainframes would be looking at at least a multi million dollar project to perform a decommission and if even approved it would be first on the chopping block to pause / eliminate the project.
You are correct. There was a time when companies ran projects to replace their mainframes. They all largely failed. The trend now is to integrate around the mainframe. Build new functionality on modern systems and leverage the mainframe where it makes sense.
I worked at one company that is trying to replace their mainframe. The project is now in its second decade. The company has the burden of supporting the mainframe application and now the 'modern' application. The project is still a few years away from being able to completely shut off the mainframe.
Oh and the modern application is also showing its age as the business has changed and the modern system built assumption in from the mainframe system and in many ways is just as inflexible to change as the mainframe app was.
This is why companies aren't scrambling to replace their mainframe systems. They are happy to continue paying for the high cost of compute on these systems because its still cheaper and less risky than replacing the system.
I've been on a couple of successful migration projects, both of which moved things to unices. One was motivated by the $5 million/yr costs to maintain the mainframe, which was replaced by a handful of lowest cost bidder unix boxes. The second was regulatory-agency-mandated cleanup after a financial fallout, and unlike the previous two or three times that company had tried and failed to consolidate their risk reporting onto a single system, this time it worked. Because it had to. The threat of an external regulator was sufficient to break down the previous barriers of balkanized system owners protecting their turf.
I bet that the "modern" application, given its 20 years vintage, is probably a J2EE application running on whatever is the modern incarnation of IBM Websphere and it was written as a mind-boggling soup of EJB 2.x Entity and Session beans.
I have worked on an RFP for migration. It persists because the cost for migrating can be quite high -- many mainframe environments have many applications with complex sets of dependencies (after decades, not surprising). No one doubts that it could be migrated, but it is not a trivial undertaking.
When folks have replaced significant enough chunks with other apps, or otherwise made the prospect less daunting, then it may happen. It just comes down to cost and risk management.
Notice they didn't mention how incredibly outdated all these applications are? Our financial money transfer system is an ancient joke. The IRS has systems that are hopelessly out of date and limited in capabilities.
It's really easy and safe to sign a contract for a newer mainframe; porting a legacy app is fraught. Middle-upper management goes for what's safe, not what could torpedo their career.
The hardware might be whiz-bang but the legacy applications are slow, outdated, and extremely hampered in functionality.
In my state, the birth records system is electronic, and so is our RMV. But in order to get a RealID, I had to get a paper copy printed out to bring to the RMV because they can not, or will not, integrate the two systems. Meanwhile other countries have things like electronic ID cards with personal certs so you can electronically sign documents and identify yourself.
The whole thing is a giant puff piece for IBM, reading like a sales presentation transcript.
Inline SQL is something I've always missed elsewhere. Not sure it's viable when you've got a billion different incompatible databases supported on a platform, so some of the limitations of mainframes have advantages.
God, I hate LINQ. When dealing with code that uses it, it's like you're reading a detective story where suddenly TYGER TYGER BURNING BRIGHT IN THE FOREST OF and then back to murders until WHAT IMMORTAL HAND OR EYE and then murders again.
I've always looked at IBM like an even more extreme version of Microsoft. I'd probably have us on their path if the cost and knowledge weren't a massive barrier to entry. I know IBM has some cloud thing for us baby startups, but I sure as hell can't figure out how to use it or if our customers are even aware of it.
Vertical integration of the business into one magic box is a perpetual dream of mine. Bonus points if I can yell at exactly one vendor if anything goes wrong.
IBM probably doesn't make mainframe LPARs available over the internet because the cost of the LPAR would be cost prohibitive to any potential customer. On the other hand, a z/VM guest or an LPAR on POWER should absolutely be an option.
They do. A z/OS with a single CPU and 4GB of RAM costs around $2000/month. It's a bit of a hassle, as you need to connect to its VPN and will need to add a public network gateway for your machine to see the world, but that's it. It could, of course, be just a tiny slice of z/OS running under a larger LPAR, but IIRC, LPARs can get really small. You can go as far as 16 vCPUs and 128GB of RAM for roughly $30K/month.
Last time I tried I had issues with the VPN and couldn't connect to it. I'll try again later. Another good thing about IBM cloud is that they embrace Terraform and you can download templates for just about everything they offer.
And the bulk of the Telum processor is an inference engine, in essence a very large vector processor (although one tailored for AI inference workloads).
I never worked on z/OS, but I did work on AS400 (Series I, or whatever it's called now).
I think the main things missing is how much IBM really brings to the table here.
> If one crashes or has to go down for maintenance, the other partitions are unaffected.
Effectively, IBM often does that for you. The machine detects an issue, and calls IBM, who sends someone out (in my day it was immediately), and they fix it. Then it's fixed and they leave and most of your staff has no idea they were there at all.
Plus, there's not enough that can be said about using a dedicated stack of hardware and software owned by one company. If there's an issue, it's IBM's. They are the ones who need to fix it. No trying to get HP vs Microsoft to agree to take the blame (which can take literal weeks). Just call IBM, and they take care of it. (In theory)
Oxide seems to be trying to build a similar arrangement with customers. That's half their motivation for switching to open firmware for the little computers hidden in your machine. There's a big game of fingerpointing these days where you call your vendor and they blame one of their vendors and can't/won't hunt down the issue for you.
The ways I've heard that explained sound exhausting. Paying anyone who you can say, "your machine broke come fix it" and they actually do, is probably worth the money. Right now Cloud providers and IBM are the only ones really providing that service. I suspect history will say that people were not running to the Cloud so much as running away from bullshit hardware vendors.
Yes! There are lots of ways in which we are nothing like these big IBM systems (we are, for example, entirely open -- like POWER, but very unlike AS/400 or Z series machines), but we definitely admire the robustness of these machines!
Indeed, arguably to a fault, as I likened us to the AS/400 in a VC pitch[0] -- which (despite the potential accuracy of the historical analogue) is as ill-advised as it sounds...
IBM big iron is a thing of beauty. There is nothing ill-advised in pushing a platform that's comparable to it in the good parts, and lacking the icky ones is a huge plus.
And it's a shame POWER is soooo expensive. But, then, so are high-end x86 and ARM, so I guess it's not that it's expensive more than I'm not paid nearly enough to have one. And, between buying a high-end POWER box and a yacht, I'll always take the second.
I do like the idea that parts of a system can simply be turned off and when a number of faults have accumulated, a tech is assigned to deal with them all at once, amortizing the maintenance costs over more tasks.
Better overall to replace three drives at the same time, especially if the servers are a half hour away from you.
I have a really hard time with the game theory around this.
Every time I run the numbers, I arrive at: I would much rather be in a situation where I can place blame at the feet of one party, especially if they are under some contract to provide a certain level of service. There are things that I absolutely cannot control and I'd rather push as much of this as possible into one bucket.
What I can control is how we actually use the technology relative to the business. If our sole vendor doubles the price on a certain service, then we will explore alternative services (within the same vendor) or ways to use that service less (maybe we send emails instead).
I know the vendor is going to try to turn the economic screws. That I can work with. Business as usual. What I don't enjoy are those "unknown unknowns".
There was one other aspect of these machines I missed. They run forever.
We had a machine in there that by the time I left was running nonstop for 10 years. I was their only IT person for 5 of those years, and I basically visited them a few times a year to work on MSAccess reports, and never touched the server (I may have never even logged into it!). It just chugged along running their entire business with no maintenance.
Throughout my career, the term i heard most often for this type of scenario was: "Which neck to choke when stuff fails...", or something like "...the least number of necks to choke when stuff fails...", etc. lol :-D
I've always heard that as "scheduled uptime" or "unscheduled outages". When I worked in a mainframe shop, they used to IPL (reboot) the mainframe every Sunday morning. That down time was never considered as part of the SLA.
Not disagreeing, just commenting that I would have assumed that taking the entire system offline for a reboot on a weekly basis would be untenable from a business perspective. It seems at odds with the concept of all the redundancy built into mainframes to ensure high availability and uptimes. Given that many mainframe-based systems (e.g. airline reservations) generally need to be available 24/7/365, I would have assumed that while one part of the system is being rebooted, others are still available so the overall application can continue to run uninterrupted.
If you zoom out, the ecosystem is very comparable to running on AWS or similar. It's an opinionated environment with proprietary answers for how to do things like scheduling, short-lived tasks, long-lived tasks, storage allocation, networks, monitoring, forced version upgrades, etc.
I suspect that for most organizations that use mainframes today, there are so many integration points and so much data is involved that the economics that drove a lot of early-on timesharing no longer apply.
It's one of the major ways to get a mainframe these days, even companies you'd expect to have one on premises might actually have a lease on one running in IBM datacenter with VPN to internal network.
I believe the real reason for its survivability is the fact that you can pull a tape from the seventies and those binaries will run without any modification.
It's not only that you can easily recompile your COBOL from the '70s, the binary is still compatible. You've never been pushed to migrate to another technology.
Imagine the effort and 'knowledge' included in those evolved programs. The banks don't even know, and are conscious of it, how many laws and regulations they have encoded in there.
As someone stated in another comment, the software is the impressive part.
This is both good and bad. You have to consider bugs a kind of feature like anything else. One of my coworkers showed me a bug report he'd opened 30 years prior that IBM still refused to fix because people depended on the broken behavior. So porting off the mainframe also means bringing along those quirks or rewriting to specs that provably don't regress performance and behavior. Writing or rewriting software is easy, but migrations despite how they first appear are not really "green field" development.
I've never used or even seen a mainframe in 26 years in the tech industry. My brother in law works for a bank and basically the business runs on it.
The hardware and software are certainly impressive but does anyone use a mainframe for a new project and not just upgrading or expanding an existing system?
I'm in integrated circuit design and we have compute clusters with thousands of CPUs. For some jobs we use a single machine with 128 CPUs and over 2TB RAM. Some steps get split over 30 machines in parallel. All of this EDA / chip design software migrated from Sun and HP workstations in the 1980's and 90's to Linux x86 in the 2000's. I think some of it ran on mainframes in the 70's but does anyone use mainframes for scientific style calculations or is it just financial transaction / database kind of stuff?
You'll most likely find it in large companies that operated in the 60s or 70s that haven't switched to anything new, mostly because their core business runs on it.
I know of two companies, and at least one still use it, had several summer jobs there. They make sheet metal rolls by flattening out train cart sized hunks of steel, and while the mainframe system didn't run the machines (operators and PLC handled that) it kept track of everything, inventory, logistics and planning. I used it to plan rail shipments, where to put each roll of sheet metal on a train and loaded them up.
Oh, fun times. I've been on the other side of that business in my past life, where I had to "revive" a business critical program written in VB3 (yes for Windows 3.x) after a computer migration that was used to calculate the weight of an aluminum or steel coil/roll via its dimensions so it could be input into the PLC for the feeder mechanism at the beginning of a production line that did forming/extruding of metals.
So on one end mainframes, on the other ends software written for DOS and Windows 3.x still being used in (at the time the 2010s) to keep critical infrastructure for manufacturing running.
That could have been a mainframe, but I think factories are much more likely to be using AS/400 aka IBM i. That runs on regular IBM Power servers these days.
> I've never used or even seen a mainframe in 26 years in the tech industry.
40 years here. Although in theory I've used a mainframe in college, but it wasn't IBM (Burroughs) and just seemed like "a computer" at the time. I also worked a bit on a terminal emulator for ICL mainframes, so probably at least logged in. So 40 years of saying "I wonder if the mainframe people already have a solution to this?"
Probably mainframes haven't been much used for HPC since the transition to Cray in the 70s.
I started my tech career as a student worker at my school district's office. They had a Unisys mainframe managing student and employee records; around 1998, they replaced the old system which was about 3 feet tall and maybe 20 feet long, with a 4U dual processor Pentium Pro running NT 4 with a Unisys emulator. Seemed to work just about as well, but the operator console seemed a lot less fun. Still interfaced with the giant impact printer to print out grades and payroll.
"Today’s IBM mainframe CPUs are based on a highly evolved version of the POWER architecture that IBM has been developing for over 30 years. "
I've heard a lot of largely clueless people who weren't aware of the differences between the zeries and pseries say something like this, but its generally been entirely false (especially 15-30 years ago which overlaps with some time I myself spent at IBM). Given the rest of the article I wouldn't presume the author is in this category.
So has something changed? or is the implication stretching the truth? I mean I guess you could strip a POWER core down so it only runs s390 microcode/etc, but that likely won't actually yield a performant machine and the process of evolving it would likely fundamentally change the microarch of whatever RISCish core they started with.
I mean they are entirely different Arches, in the past utilizing entirely different microarches. I can see some sharing of a RTL for maybe a vector unit, or cache structure, or a group doing layout, but that doesn't make the zeries processors any more derivative of POWER than Itanium was derivative of x86, etc.
PS: the bit about zos partitions supporting linux seems like a bit of confusion too, earlier its correct about the LPARs being capable of running linux directly, but ZOS isn't providing the lpar functionality, and is generally just another guest alongside linux, ztpf, and various other more esoteric "OSs" that can run natively. There is a unix system services in zos but that isn't a linux kernel/etc.
I wonder if the article meant that IBM had worked on the POWER architecture for over 30 years? IBM did work on the eCLipz Project [0][1], combining / sharing tech from IBM Power/pseries, AS/400/iseries, and Zseries. This was around 2005-ish. I assume that collaboration has continued, but I don't know if that counts as 'based on ...'.
"The z10 processor was co-developed with and shares many design traits with the POWER6 processor, such as fabrication technology, logic design, execution unit, floating-point units, bus technology (GX bus) and pipeline design style, i.e., a high frequency, low latency, deep (14 stages in the z10), in-order pipeline.
However, the processors are quite dissimilar in other respects, such as cache hierarchy and coherency, SMP topology and protocol, and chip organization. The different ISAs result in very different cores – there are 894 unique z10 instructions, 75% of which are implemented entirely in hardware. The z/Architecture is a CISC architecture, backwards compatible to the IBM System/360 architecture from the 1960s. "[2]
For the POWER
Yeah, I don't know where the author is getting the POWER arch connection.
I thought the IBM Z Architecture was the CISC based System 360 / 390 architecture from the 1960's. At least that is what I remember my one friend who has some mainframe experience was telling me.
> Mainframes descended directly from the technology of the first computers in the 1950s. Instead of being streamlined into low-cost desktop or server use, though, they evolved to handle massive data workloads.
I think the first sentence is 100% correct, but the second one not so much: current desktops and servers (not to mention laptops, tablets, smartphones etc. etc.) evolved from the first microcomputers introduced in the 1970s with the idea of having a computer (albeit initially a not very capable one) that anyone could afford. These then quickly evolved during the 1980s and 1990s to cover most of the applications for which you would have needed a mainframe a few years earlier.
It's still somewhat true. Desktop CPUs have hardware acceleration for things like video decoding, mainframes have hardware acceleration for things like encryption / decryption, compression / decompression, fixed-decimal arithmetic, etc.
Each person has a box in which they are essentially the sole tenant, versus a big box that has to have a bunch of sophistication to handle multitenancy.
I was involved in an IBM mainframe to RS/6000 migration in the mid 1990s for a major aerospace company. We moved the engineering functionality to a client/server system, including many custom programs and CAD/CAM/CAE.
Like the article mentioned, migration was a Herculean task done over a period of years, with a good chance it wouldn't succeed. I worked many 70 hour weeks, I had to drive to the factory at 2AM some days to monitor jobs because only certain people had remote access. Floating point and text encoding was different between the systems, decades of legacy engineering and manufacturing data and code (having cost hundreds of millions of dollars to produce) had to be translated and verified. There was a rivalry with the mainframe group who wanted it to see it fail, reluctant programmers, engineers, and line workers. On and on and on...we had to apply start-up levels of creativity and ingenuity while working in an MegaAeroCorp bureaucratic atmosphere.
Ultimately, the project was a success, although the system soon moved to PC-based desktops as graphics cards became more powerful. I got a company award from my internal engineering customers. Shortly after the dust settled, I got called into a secretive meeting with my boss and the department head, expecting a promotion and a raise, or at least an "Attaboy!" No, I was reprimanded by an HR rep with a formal note in my records about my attitude ("You risked everything by cowboying!"). I was admonished for having the gall to take a couple of sick days during the ordeal (my coworkers were concerned when they found me passed out at my workstation, my boss warned me after my time-off "If you are sick be sick" and called me a filthy name.) I was put on double secret probation.
Our whole department was eventually outsourced, and I ended my tenure there by being escorted by security to the door and told not to let it hit me in the backside. I guess I should have joined the mainframe group on the business side, in retrospect, they are still around grinding out profit statements. It was a challenge to get rid of a mainframe, maybe more so than eliminating our entire engineering support team.
After the migration, when they were "decomissioning" the mainframe itself, the company photographer took a picture of the migration team gathered around the mainframe. It was unplugged, being gutted, with copper pipes sticking out. Years later, looking at a print, it reminded me of the old pictures of a whaling crew gathered around a whale carcass--don't get me wrong, whales are much more valuable and important. It does seem like we helped destroy something we really didn't completely understand the value of.
The only thing not mentioned are the co-processors that handle a lot of ancillary tasks and thus keep the main CPUs free for more work. For instance, printing a report might require just 1 interrupt per page, the rest being handled by the printing controller. The TN3270 terminals talk to a separate communications controller before being passed to the cpu, and the disks used to be able to return results in a sorted order, not sure what they do now...
> Communication is through Kafka events or Java Messaging Services, and new server instances can be spun up in seconds in AWS or Azure clouds to provide additional capacity, which is needed for high-volume processing.
I'm honestly surprised that a bank would agree to run anything even remotely serious on someone else's infrastructure.
Many banks, including very large ones, already have some amount of stuff running on public (& private) cloud infrastructure. Many other banks are preparing migrations. I, too, was surprised by this, considering only a few years ago they were very reluctant to even buy software from anyone who wasn't on a very short whitelist.
In the end, I guess it's all about the money ... and that includes the money it takes to find and train mainframe devs.
Source: spent a significant part of my career in fintech.
Me too. We sold a lot of dedicated fiber to banks in the early 2000s because they feared the packet switched nature of ATM. They thought packets would randomly end up at competitors.
I don't know if they ever realized even dedicated fiber links were software switched by that time so it was easy to misclick and send an entire fiber stream to their competitor too :)
My partner does very technical systems hacking on z/OS. They have a huge problem hiring new programmers. His colleagues are all well over 65 and are inevitably retiring (or worse). It seems like a real problem for the future of the platform.
Lot of people in Taiwan are learning COBOL and mainframe programming specifically for this reason. Surprisingly big (or small, depending on how you look at it) popular of young people learning it.
Big offshoring companies like TCS went big with the Y2K bug correction, and handling COBOL development.
Sometimes that is how one goes big, by taking care of stuff no one is interested into doing, and when there is enough money in the bank, go after everything else.
It may be my mileage.. But I frequently hear about the shortage of mainframe professionals and, some years ago, I seriously considered getting into it. However, after some attempts to find training on topics I typically see in job postings (CICS, JCL etc), I gave up after finding no courses also providing access to a mainframe "account" for practicing. Maybe I didn't search it properly. I've been advised to try an entry-level position in a company hiring juniors, since they usually provide the training during the first months. But, after over 20 years as an engineer, that felt bigger a step back than what I was willing to take.
I still feel interested though. I'm aware of the current state of chaos on mainframe development, but honestly, I don't think that the current web/mobile situation is much better and, besides, I personally hate them really bad.
Early in my career I worked on IBM mainframes and wrote JCL for them.
Let's say someone knocked on my door tomorrow and said "We're looking for someone with any experience at all with JCL on IBM mainframes. We'll pay a million dollars a year for a 10-year guaranteed contract. Are you that guy?"
I would say "Nope. Sorry. I'm a plumber. I don't know nuthin' about computers."
(And then they would say "Ha! GOTCHA! How'd you know we were talking about computers? Now take your million-dollar check and watch your head as you get into this nice black helicopter.")
> Sadly only Microsoft and a few others share this attitude…
Since the introduction of Windows 10, this is no longer true. For example, games that supported Microsoft's "Games for Windows – Live" often do not not work out of the box anymore, as required DLLs are no longer installed. And on the 2017 Microsoft Answers thread about what to do to work around this[0], users are reporting the steps no longer work.
Believe it or not: I never worked on an IBM mainframe, except in college. It always seemed like the prejudices against it meant that, once you get into that world, you can't get out.
That said: what the business types like is the belief, whether it's true or not, that when you call IBM for service they're there before you hang up the phone.
Also, a long time ago at Oracle, I sat in probably the most boring meeting of my life: a group called MOSES, composed of sysadmins for Unix. Their complaint was that, supposedly, all mainframe sysadmins did things the exact same way, so if you hired a new one, there was no training. Whereas in Unix, everyone did things differently, so a new hire couldn't be productive right away.
I worked on a mainframe for a major airline over 20 years ago. I always found it amusing that Sabre detailed my programs/jobs to cost the airline almost 7 figures a year; I never knew if that was real dollars or price to be negotiated down. Most of my programs was to repackage yesterday's flights and passenger data into CSV files so an external program can pick them up and ETL into a data warehouse. At the time, there was no utility like ODBC. I had to use JCL and SAS and my own created utilities. I even had to write a utility to output to CSV (there was none offered out of the box by IBM or SAS).
I ran Internet banking for a small bank in the early 2000's. We had our IBM mainframe right next to a bunch of HP/UX servers. It was just massive and had a nice red button which they warned all of us to stay clear from. I recall that was around the time running Linux on the mainframe was becoming a thing and we tried to run Websphere and our J2EE app on it. It was not successful (too slow) and we kept running on HP/UX.
Back in the early 80s, I visited the Police National Computer Unit at Hendon in North London. The computer room was the size of a football pitch. Most of the floor was covered with washing-machine-sized disk-pack drives, along with quite a lot of dedicated IO processors (also washing-machine-sized). The walls were lined with storage for offline disk-packs. This system was a triple Burroughs B7800.
What distinguished mainframes from minicomputers back then wasn't their awesome processing power, or their resilience; it was their huge IO capacity. They could handle hundreds of storage devices, and hundreds of thousands of terminals. At least, that's how I was told it; I never worked on mainframes.
Nowadays a single connection to the internet can connect you to a similar number of "terminals", and storage is also accessed over a network. So the advantages touted by the author are quite different from what they used to be; IO nowadays isn't the issue it was back then.
Here’s a recent-ish Changelog podcast with one of the few professors who teaches about IBM mainframes and COBOL and is part of the Open Mainframe Project, https://changelog.com/podcast/524
Ah, good to hear that the IBM mainframes are still around, are more powerful, and that the software for the old IBM mainframes will still run on the new mainframes.
So, yup, JCL still works! An 3270 series video terminals with 24 lines of 80 characters each driven with CICS (customer information control system or some such) is still used.
But I didn't see that
(1) Rexx, often a good replacement for JCL,
(2) XEDIT, and the PC version KEDIT, my favorite text editor,
(3) and VM/370, an impressive virtual machine supervisor, upgraded from the original CP/67 for the IBM 360/67 mainframe
are still supported and used. I would be shocked if they were not
still supported and also still used, especially for development and analytical computing tasks.
For the software to schedule the fleet at FedEx, I wrote in PL/I on CP/67 (control program, i.e., virtual machine, for the IBM 360/67, with virtual memory) CMS (conversational monitor system, the user interface) on a 360/67 at National CSS time-sharing in Stamford, CT. Right, VM/370 was from CP/67! I wrote the code from my living room while teaching computer science at Georgetown U.
Then in Memphis, one evening used the code to develop a schedule for all planned 33 airplanes for all planned 90 US cities. The next day two representatives of BoD member and crucial investor General Dynamics went over the schedule carefully and announced to the BoD that "it is a little tight in a few places but it's flyable". Until that schedule, due to concerns of some BoD members, scheduling had been a show stopper. But with the schedule the BoD was pleased, crucial funding, equity and also counting loans on the planes, etc., $55+ million, was enabled, and FedEx was saved until the next such BoD crisis.
I solved the next such BoD crisis with the differential equation
y'(t) = k y(t) (b - y(t))
and for this one, for the computing for the data to draw a graph of the solution, used my new HP calculator I'd paid $400 for! There the BoD wanted some revenue projections.
NCSS was expensive, but I enjoyed using it AND PL/I! It didn't use 3270 communications and, instead, used asynchronous bits on a phone line and a portable terminal with heat sensitive paper in rolls. The terminal was from Execuport and supposed to be portable. Actually, it was delicate!
It is just staggering how far computing has come since then! Hardware, operating system software, APIs, user interfaces, communications, YES, but applied math, algorithms, and programming languages, not as much and, thus, seem more fundamental!!!!
Absolute fringe opinion(or is it?): The performance and reliability are awesome, but have nothing to do with the real reasons IBM mainframes are still around.
Backward compatibility helps - you can still run code for the IBM 1401 on these machines if you need to. When they came up with System 360, they effectively crystalized software development... taking what used to be a forced march to new hardware every few years as you got a new, faster, incompatible machine, and were rewriting the code for the better faster hardware.... and just FROZE that stream of development, which is why Y2K really happened.
But even that's not the real reason IBM mainframes have stuck around so long... it's the same reason Virtualization and Containers took off.... they give no default access to files or disks, the administrator has to plan out which disks and other resources a Job, (or Runtime, VM, or container) will access. (Example, the DD statement of JCL[1])
This has the effect of being a very course grained capability object system.[2]
The default access to everything on a Windows/Unix/Linux system just doesn't work in the modern era. It is because side effects can be strictly controlled, that Mainframes and later VMs and Containers have so much value.
-- a small rant:
It's 2023... consider the humble AC wall outlet, you can plug almost anything into it, and the circuit breaker or fuse will prevent bad things from happening to your wiring, house, etc.
Why can't we protect against a rogue device plugged into a USB port? Why can't we just run any program we please, like we can plug in a lamp we bought at a garage sale, without risking everything?
The problem with most projects of mainframe migration that I've seen is that most of them suffered the bad luck of being started around the 90s and 2000s, a really complicated age in Enterprise Software, the age of COM/DCOM/COM+ and EJB 2.0.
Any normal person would prefer to program in S/370 assembly than be forced to use EJB 2.0 Entity Beans.
I remember in my late teens, working as a temp... literally looking up individual accounts from a "past due" report to make sure they were still past due before they received a call from collections dept. Why, because paying a temp was literally less expensive than re-running the report on the mainframe.
One of the reasons Mainframes continue to be used is depreciation. I know a few companies who love their mainframes because they are fully paid for and from a budget perspective they are seen as “free”.
This can be very attractive compared to the never ending OpEx for running cloud computing.
Doesn't make sense to me. I believe that IBM has found the forced upgrade path for hardware/software like Apple has perfected. You may not be able to run old versions of IBM's software and still have it be supported by them. Same for the hardware. I don't think you would commonly find and old mainframe (the hardware) in operation. At the last place I worked that had mainframes, they would upgrade to new models every 2-3 years.
I call BS on this. I never heard of upgrading mainframes every 2-3 years. It makes no sense. Mainframes are huge investments meant to last a long time, it is not a PC you replace in a few years.
Call BS all you want, I don't care. I know what I witnessed. I don't know if the mainframes were owned are leased, but they were swapped out for new ones every few years.
I’d like to hear more then. Mainframes are traditionally a big investment where you consolidate tons of load and jobs and LPARs onto one honking big machine. A very expandable machine in terms of any resource you can think of.
Why would you turn over something like that every few years? Your narrative sounds more like taking about ephemeral cloud stuff, not multi million dollar main frames.
I think in my former employer's case they leased the mainframes because they found it more favorable to have OPEX over CAPEX. Every few years IBM would pitch their latest and greatest and often times it made financial sense (lower costs) to upgrade to the newer models.
The trick to surviving in Enterprise environments is consistently and credibly beating the longest horizon dtrayegic rate of return budget (luck has it enterprise's stategic horizons are not that long). At any point it is still better to stick with the current than to change. You boil the frog, but very carefully.
Oxide could probably be considered a "modern mainframe" if you squint. The architecture is of course much different, but server CPUs from the likes of AMD are gradually catching up with the purpose-built stuff in raw power and I/O capabilities.
Oxide hardware is more comparable to a midrange computer. Though they are apparently planning to add support for having multiple physical racks in a single managed "silo", which enables HA scenarios and gets a bit closer to what mainframes provide.
Yes, like many kinds of comparisons, it really depends on what you mean by the thing. The architecture of Oxide is very different than a mainframe, but the "big computer that's a fully integrated system" vibe is kinda similar.
This is an interesting overview of mainframe vendors and how many customers they might have: https://arcanesciences.com/os2200/app1.html
Most of the "dwarves" are now using emulation...
tldr: This awesome professor has an amazing track record of getting seemingly "average joes" trained up for mainframe programming. Because of how uncool this process is perceived by the average SV developer, it's literally shunted off into the dark recesses of the world, despite it being an awesome jobs platform.
Going back to the hardware that everyone likes to focus on, it's less radically different from normal servers today than it was historically. The mainframe today is a 19 inch rack like any other. By that i mean it is not only a 19 inch rack like your x64/ARM servers are but also the same power density (32 or 64 amp racks), cooling requirements etc.
The most interesting bit is the software, not the hardware. There are cool hardware aspects too - but focus on them and you miss the real reason these things are popular in certain environments.