20 years ago, when I was trying to enter the field, sysadmins were almost all former developers who had a pretty fundamental understanding of operating systems.
Somewhere along the way "sysadmin" became a euphemism for helpdesk, and some folks thought "devops" (as a job title) was new, but it was just the same old sysadmins really, or new devs becoming new sysadmins with the fresher title, they felt themselves the vanguard of a new paradigm, I'm sure.
Now we're in the same situation again, where "devops" (as a job title) can have no development experience. So we need a new term, apparently.
Maybe gatekeeping is useful to stop this job title treadmill.
I think part of the reason this trend has been going on, and I have observed a similar thing as you point out here, is that businesses have tried to get rid of this “role” for a long time now and by constantly reinventing and renaming it. They have failed miserably. When people ask what I do, I call myself a systems engineer or “backend engineer” if that makes more sense to them. it’s more apt for what is actually needed and asked of me.
Things went wrong when we started calling Computerization, "Information Technology".
This was a totally misguided re-branding of the field. It attracted bureaucratic personalities and classist managerial types, and utterly destroyed the field.
Like you, I couch my job description: I'm a "Systems Software Developer, I develop software systems" .. because its true, thats what I do, even if its a majority of the time working on embedded projects.
And if someone then responds with "ooh, so you're in IT", I immediately go to another corner of the party and avoid them like the intolerable bores they are ..
> And if someone then responds with "ooh, so you're in IT", I immediately go to another corner of the party and avoid them like the intolerable bores they are ..
No need to be arrogant towards people outside the field. My grandma doesn't know the difference between the random IT guy that updates your Windows printer driver and a Linux kernel dev, but I'd still not call her an "intolerable bore".
I go a step beyond. When I’m interacting with “non-IT” folks, I say just I’m “in IT”.
It isn’t a problem for my ego to do that, and it’s something that’s both relatable and typically stops the conversation from revealing anymore about my work (not that there’s really anything to hide, I just enjoy my privacy and no, I don’t want to meet your “computer genius” nephew or fix your phone).
If I’m in a clearly technical group, I’ll go into more detail. I don’t identify myself with the DevOps term though, or sysadmin, or developer. I don’t clearly fit into any of the -currently assumed- definitions of those.
I always default to "I work in IT". If that person is in the space I might go into specifics; but even then majority can't relate or understand the role i spend my days working in.
It also has the benefit of coming across as more blue collar and I catch less grief, as the disdaine for tech workers continues to grow.
In social situations, I usually just say "I'm into computers". Most people who ask what I do don't really care, they're just engaging in ritualistic small talk anyway. If someone does care, they'll ask clarifying questions.
Guess Problem is that a lot of managers also don't know, and have same understanding as your Grandma. I've seen plenty of companies take a very highly paid engineer and have them install printer drivers on peoples PC's.
I was getting paid £200k/yr as a Software Engineer for a company with an unresponsive offshore IT help desk. I would often spend a few hours (!!) a day helping various stakeholders with their desktops/printers/email. This was an insanely ineffective but absolutely necessary use of my time (I was managing a small development team with stakeholder satisfaction kpis), sadly my IT efforts were clearly more appreciated than development.
Hard to convince management that a dedicated onsite IT would be required - especially when they think it is your job already.
I suppose there is a difference between companies that are tech to the core (where everybody in the management chain up to CEO is actually a sw eng) or a company where software engineering / programming is just a dept. somewhere on the side.
Gotta appreciate if you're working in the former. Drawback is you can't BS your way through issues. Everybody above me in the chain (which is like 4 people) could write a quicksort blindly. If anybody tries any BS they don't have their job much longer.
The term 'wage slave' is commonly used. It is not always a humorous take on the situation. If you live or have skills where you have a lot of options. Then that is good for you, not the default for everyone.
I disagree. Data center ops have been completely reinvented by the devops practices. Sure it's a bunch of sysadmins at the start, but a bunch of sysadmins really trying hard to take themselves out of the loop so that things can scale.
A good example is how software releases work. Where I worked 15 years ago, releases were something we could afford to do about four times a year, at night, on the weekends, doing the rollout by logging onto machines and starting them back up, babysitting the rollout. Because rollbacks were catastrophic, an insane amount of time went into QA, so the time between a code change and a release could be easily multiple months. Now I'm at a place where releases are done every day of the week. Rollout steps are largely automated. Rollbacks are mostly drama free. Risk is mitigated through automated canarying (and automated testing before that).
And certainly the business has successfully gotten rid of paying people to log in to servers on a Saturday night to perform a release (which frankly not a lot of people are into in the first place, so everyone wins).
From what I've seen recently is the inverse of that.
There are scores of developers who have little understanding of OS scaffolding or networking, but are good at writing merge sorts in Python.
Most Devops people I know are not the ones pushing "vanguard of a new paradigm" BS, they are just ops who can code and understand performance tradeoffs, networking, subsystems, etc.
I guess we can blame Devops for one thing, the paradigm has encouraged the behavior of "real devs" (TM) not understanding anything about the underlying platforms they are writing code for. That was the point, but it's not all gravy.
[Weeps] That wasn’t the point, the point was exactly the opposite of that, to get developers and ops working as a single team, building an understanding of each other’s domain in order to make things better. Sadly the point was lost long ago, and devops engineers and devops teams have taken the place sysadmins and ops teams in providing a nice little silo you can chuck your terrible code over the wall at and not have to think about how it works.
But you know what? There are tons of developers who are _perfectly fine with this_! There are tons of developers who just want to write business code in their CRUD webapp and call it a day, the more abstractions you can give them, the happier they are. And while I don't like it, I don't blame them.
Absolutely agreed, and if they have to think particularly hard about things outside their application space for a CRUD web app then either you’re in an extraordinarily high throughput environment, or the platform engineering needs a bit more work to abstract things away. That’s the ideal outcome here, developers can develop, and when they’re done they hit the Deploy button and off it goes.
Where this falls down is when development teams are building complex systems and then expecting the ops team to divine how that’s meant to work in production.
Do you dislike it when application developers don't think about the way CPUs work at the silicon level? Or the way TCP/IP works under the hood (handling packet ordering/ connection negotiation etc.)?
It's not immediately obvious to me that an application developer should have to worry too much about how their software is delivered/hosted etc., though I'd certainly agree that understanding it better generally makes you a better developer. And that the reality of modern O/S's and hosting platforms still means that implementation details inevitably leak their way into application space, unfortunately.
assuming the code is reasonable and can scale they should be allowed to and enabled to do this. as a dev I dont want to think too hard about the infrastructure underneath it, I just want it to run.
the ole “runs on local therefore it’s good to go” should be good enough.
You can ignore some of the details, but you can't ignore everything.
Which database you choose and how you scale it will affect your application code.
You can't write code that makes use of the local file system (or in-memory data for anything meant to be persistent) if you have more than one container serving your app. (And yes, some people need to be explicitly taught not to do that.)
Yep, you can think of it as overlapping circles in a Venn diagram, or if you prefer T-shaped people, imagine an entire 4-row block in Tetris. How do you fill out expertise in the entire domain but also have broad knowledge of it across the team. The business side hates it because people with expertise aren't interchangeable widgets they can throw at any problem.
Having done this job once upon a time, I think the downtrend in title is inevitable.
What tends to happen is that at some point in time, an organization finds itself in a bad place. Where pagers fire regularly, and the skills you need to manage the noise are "fast context switching", "good communication", "debugging", "ability to work insane hours". Very little of this involves coding, or building software.
Teams may start by assigning some engineers into this role as an elite team, but if you come back in a couple years you'll find that the elite team has burned out or forgotten how to write software and due to the impossibility of hiring new elite team members - has resorted to hiring people to carry a pager.
There are a few companies which invest in big platform orgs, but due to the lack of revenue generation - it seems that this can only be sustained with very strong leadership buy-in.
“Having done this job once upon a time, I think the downtrend in title is inevitable”
I absolutely agree with this. It seems like a it inevitable for anything that a business needs to do but doesn’t really want to do. Another place where you see it is with project management, where there have been many different titles for the same role.[0]
I do think there are differences. If you read LISA papers there is obviously something fundamentally different about what we did when we were practicing system administration than what we were doing when we did devops.
One of the fundamental shifts in my opinion was building APIs on top of sysadmin tasks. Provisioning a VM, installing an OS, getting software onto the server, configuring ingress/egress, handling process restarts, etc.
These all were turned into tasks you could run from a developer’s laptop.
But it was still a different skillset and devops dumped that skillset on developers (“Hey! Now, instead of opening a ticket, just write a 1000 line magical incantation of poorly understood YAML and the system can do it for you!”).
Platform Engineering grew out of that pain. It’s the layer we are building on top of devops in a way. We are taking those APIs and treating them as foundations to build products that manage the SDLC lifecycle for developers.
I think at its core, deveops was about “how much sysadmin work can we turn into an API and give to developers?” The answer turned out to be almost all of it. Then Platform Engineering is “now that we have APIs, how much can we automate away for developers.” We don’t know the answer yet, but suspect we will find the answer is also “almost all of it.”
Site Reliability Engineering seems to be following a similar story arc and converging on Platform Engineering.
The problem with handing over infra provisioning and maintenance to developers is they don't have the expertise or the time. Unless your platform is REALLY solid, and onboarding is easy and has enough guardrails, you end up with a dozen different little platforms that are like Galapagos island birds.
At my current company theres literally no coherence. Some people deploy on EKS, or ECS, or Anthos, or regular linux hosts. We use every database on earth. Things on Azure GCS and AWS. Databricks and Snowflake. The list goes on!
I work as a SRE and have done a lot of platform engineering in my life. I call the sysadmins you're talking about "operations". This role proliferated because system administrators/engineers that were coding were paid the same as a SWE while ops people came at a discount. Every contracting and consulting firm ever had an "ops as a service" contract that they'd offer large businesses. This was the norm for a little over a decade I think. Things have changed now and it's harder to exist in the SRE space if you don't code, but finding people who know systems, infrastructure, and application code to a high degree is pretty difficult and, again, comes at a premium. This is the result of more companies reaching scale that cannot survive on blueprint infrastructure and lifecycle designs, and things like tooling mattering so much to DevXP. Companies see the value in people who work on this kind of stuff now.
A lack of gatekeeping I don't think was ever the issue; the incentives were just perverse.
I spent time working around the space industry. Almost a decade of my life.
The big contractors in aerospace have a _really_ difficult time finding people who know enough about the hardware and the software to be competent engineers. They’re basically unicorns.
Not that they don’t exist, but if you’re looking for someone who is demonstrably good and has a validated educational background in both, you don’t have a huge pool of talent to go with.
So they hire them when they find them, but they also just tend to hire experts in each field and let them collaborate. They didn’t try to reinvent the wheel with some new “paradigm shift”.
As a former systems engineer who was still in that role when DevOps became a thing, I think there's a a more fundamental difference.
The team I was on was made up of ops/engineering[1] people who knew how to code. We built tools to fill gaps or let us do things we couldn't do otherwise, but would typically use out-of-the-box functionality in third-party products by default because it was cheaper in the long term than building and supporting custom code. Maybe there's a better term for this, but I think of it as "OpsDev", and AFAIK it's more or less extinct.
DevOps has always struck me as being developers[2] getting fed up with having to deal with a separate ops team, and building tools and platforms to automate away that ops team entirely.
The two things can look superficially similar, because they achieve similar goals (automate away repetitive work, let people do more work), but the approaches are almost totally different. Kubernetes and Terraform do amazing things, but they are very obviously tools written by and for developers. I think it would be difficult and unwise to hand off first-tier support for issues involving them to a help desk, for example. It's pretty safe to write instructions for first-tier support to use a GUI or run a command but substitute the name of a server farm and a software package in the arguments, especially if the command is a script written by the engineering team to verify that the work makes sense before executing it. It's a lot less safe to try to have non-developers hand-edit source code, a script or even a YAML file and push the result to production.
Both approaches work. They just require different kinds of people, and each has their own benefits and drawbacks, and IMO it's a mistake to put them both under one label.
[1] "Engineers as in Geordi LaForge or Montgomery Scott": the people whose primary job is to keep the engine(s) running, respond to emergencies, make improvements where possible, and generally be a smart-person-of-all-trades.
[2] "Engineers as in Burt Rutan": the people whose primary job is to design and build new products (for lack of a better word) from the ground up.
I'm not sure, but I haven't seen places where devops team hire people without exp. I myself tried to get some interviews (I do have some dev exp, albeit data engineering ones, not backend) to no avail. To this point I believe most companies don't hire green for the devops team.
No programming experience required (only some minor scripting in shell languages & a reference to education), in fact this job ad reads identically to a sysadmin job from 2014; in the middle of the "sysadmins can't code" zeitgeist, when all developing sysadmins had moved to become devops engineers or whatever.
Copied from the ad:
----8<----
Your tasks:
* Containerization and orchestration of our products for development, testing and production environments designed for cloud and on-premise solutions
* Produce reference architectures and standardize solutions to streamline efficiencies within the business
* Mitigate against Cybersecurity threats by ensuring systems are safe, secure and compliant to best practice recommendations (e.g. CIS Benchmarks, Vulnerability scans and Penetration tests)
Your profile:
* Finished studies in Computer Science or software engineering (Bachelor, Master)
* Proficient understanding of virtualization, containerization and orchestration (e.g. Kubernetes, Podman)
* Experience working with CI/CD tools such as Azure DevOps, Jenkins
* Experience of Infrastructure as Code tools such as Ansible, Terraform, Helm
* Scripting knowledge including bash, PowerShell
* English fluently, German is an advantage;
* Improvement-driven, structured and proactive team player, high sense for quality, open minded
Requiring a Computer Science/Software Engineering degree for a non programming job is a little weird.
The point is coding skills are not required to do the job. Sure it would be nice but if somebody ticks all the other boxes then being good at programming isn't important.
Those education requirements almost always go out the window or can be ticked by passing Cisco certifications or something else vocational. (speaking from experience)
What I'm trying to drive home here is that you do need programming experience to be a good sysadmin/devops/platform; back when devops as a job title was becoming "a thing"(tm) I was told over and over again that sysadmins couldn't code and thus we need devops.
Now it seems devops don't need to be able to code (which you seem to think as well). So we will come up with a new term to cover programmers who understand systems.
But the criticism was that the job ad was not requiring knowledge of how to write code, and I was pointing out that the degree requirement (strict or not) indicates that that's not true.
If you don't have the required degree but can demonstrate equivalent skills, then that's fine. Kudos to the company about being flexible on the formalities side.
There is a line that is universally ignored in job ads. This is that line.
if I showed no ability to write a for loop or a linkedlist. I could still get that job.
I see it time and time again, it's like we must add that line to job descriptions, but it's never checked, never verified and ultimately it serves no purpose. There will be no difference between someone with a BSc in CompSci, an MSc in CompSci or no degree at all.
The thing that will get you that job, likely, is discussing how http works, how tcp works and perhaps something to do with exit codes.
My experience with sw eng job interviews is very different and I've been interrogated quite intensely about data structures, design patterns, computer architecture internals, and have done the same on the other side of the table many times. It brings a good perspective on whether the candidate knows their way around the box and has a deep understand of what happens inside it. And I also appreciate if my colleagues know how an L2 cache works, what an RB tree is and what's good and bad about microservices. Whether that's been acquired through a university degree or through self-learning or experience on the job, I don't care though.
If I'm interviewing person A and person B, all things being equal, I'll go for the one that has a degree on CompSci (or equivalent). That's it. It's not that difficult. Obviously if person A doesn't have a degree but knows much more (based on the tech interview) than person B who has one, I'll go for person A.
The only european country that takes titles even more serious is Austria. So without a M.Sc. in CS or a similar M.Sc. your chances may be quite low to even get an interview.
Especially since university is basically free here so many people have a B.Sc. or M.Sc. in germany.
I have a computer science degree and barely mention it in my resume because it’s just one of those “checkbox” things that doesn’t actually matter. At my current job I’m not sure anyone even knows that I have one from a decent school. it’s just credentialism. you don’t need a CS degree to be a good systems engineer. It can help you solve weird problems, and IMHO it allows me to think about certain things from a very particular lens a lot better than others, but you absolutely do not need one.
> The point is coding skills are not required to do the job.
I’m sure this is the case in some places. This can’t be farther from the truth in many others. At every organization I’ve been a part of, the best “DevOps” on the team was usually the person who was the best at coding, or at least understood it the best - and coding is one of those things that you can’t really understand well if you don’t do a lot of it.
Any by the same token the best developers have a good understanding of Ops. Often these will be the Senior ones and in some cases they built the platform before Ops people were hired.
But plenty of developers only have a vague picture of what production looks like especially in a complex environment. However they are perfectly able to do their job okay and produce working code.
In my experience they also don't want to have the computer make a api call and wake them up at 3am.. but are happy to let the DevOps guys take that sword.
Being great at DevOps means lots of Production experience. The level of experience required to go from thinking about working software (code) to running software (Production) is immense. That's why it is so hard to level up DevOps talent in the current dichotomy. In a silo DevOps engineers won't run enough software to understand their customers (the engineers in your org).
Assuming these are the tools and processes that build and deploy your main product, I don't understand this comment.
Sure, maybe some of the work isn't writing software. But some of it is, and it shouldn't be written by amateurs. Also, they are building these things for people who do write software. It would be hard to produce something that works for them without understanding what they do.
As somebody who does a job like this almost none of it involves writing software. 90% of what you write will be terraform and ansible. The remaining will be 20-50 line bash/python scripts.
Reading code, understanding building and deployment are all useful and required to some extent (especially the latter). But specific programming skills are not required any more than that would be for say a technical writer or network engineer.
I believe you, but I think this experience/job varies pretty wildly. That is, there are organizations where your job does involve writing software that's more than short scripts. If a company chooses "best of breed" for their pipelines, for example, there's quite a lot of integration between SCM, build tools, security tools, deployment tools, and so on. Doing that in a way that serves the end users well isn't always trivial.
Yeah, a lot of JD reads like that, but hey why would anyone be proficient in these skills without dev exp? Or they are already a devops anyway.
Proficient understanding of virtualization, containerization and orchestration (e.g. Kubernetes, Podman)
Experience working with CI/CD tools such as Azure DevOps, Jenkins
I believe most companies don't hire green for the devops team.
Most companies do. Perhaps not top Silicon Valley tech companies with cutting edge needs and paying over $150k, but overall I've found that dev ops teams have plenty of jobs for 'green' team members. You just have to accept a 'green' salary and 'green' job responsibilities.
"To this point I believe most companies don't hire green for the devops team."
Yes. Devops people come from a SWE background generally + they understand infra. They usually have 10+ years experience. Most specialties work this way in software: security, devops, AI, ML, ... etc. You need to be a SWE first, then specialize.
This is not what I've seen at many places I've worked it. Plenty of devops people with only basic programming background plus a AWS/Azure certification course or two.
Also I work in AI/ML and hardly anybody has 10+ years of SWE experience before specialising. Most people start fresh out of a PhD or postdoc program or come from some sort of math or physics background.
In my experience, it used to be 60% former sysadmins that didn't know how to code or approach problems from a software engineering perspective, and 40% SWE who liked doing platform stuff.
Now it's actually 80:20 due to software engineers becoming frustrated and disillusioned with the current state of things. Not only due to specialization and the push towards tool standardization (you don't get to build many custom components because we k8s all the things so most of the time you're just writing yaml/hcl), but because working with regular sysadmins gets extremely frustrating.
I now saw it several times of when they needed a DevOps team, what they meant is what I think is different from the typical sysadmin. For doing the change itself on the Salesforce platform is easy, but in the end everything has to be merged, deployed, etc. The new DevOps teams are doing that stuff. They're not concerned with the dev part of the change, but all of the deployment/release tasks around it. This is actually novel for a lot of organizations, because their pipeline was so much simpler, e.g., a three system architecture, where every project deployed to one test system.
Somewhere along the way "sysadmin" became a euphemism for helpdesk, and some folks thought "devops" (as a job title) was new, but it was just the same old sysadmins really, or new devs becoming new sysadmins with the fresher title, they felt themselves the vanguard of a new paradigm, I'm sure.
Now we're in the same situation again, where "devops" (as a job title) can have no development experience. So we need a new term, apparently.
Maybe gatekeeping is useful to stop this job title treadmill.