In the tech-related (but now fully mainstream, unlike a few years ago) news these days, there are a number of recurring themes.
1. Massive security breaches affecting major corporations, governments, and so on.
2. Massive IT outages, affecting corporations, governments and so forth.
It doesn't take a lot of incidents to mark them as 'massive' since things are strongly centralized in this world.
Meanwhile, in the last 15 years of my IT career, never have I seen such a strong push towards offshoring. One would think that with such a vulnerable IT landscape mixed with an unprecedented dependence on IT infrastructure, that CTO's would want to spend more and not LESS on operational costs.
> never have I seen such a strong push towards offshoring
As I've been working in the airline IT industry, I've observed that the reality is more nuanced.
Core booking and ticketing systems are 'outsourced' to solutions from Amadeus and Sabre. This makes sense not only for many strategic reasons (easier to have airline alliances when the airlines are all on the one system), but also because airlines are in the business of flying planes, not building ticketing systems.
Airlines are often full of loads of legacy and old systems which you just can't hire developers for (in the numbers, and costs, desired) - this is where Indian outsourcing firms come into the play and resolve the shortcomings. They're taking all the jobs and work that no one really wants to do anyway.
However airlines, and many other industries (banks come to mind), are realising that their digital offering is just as much, if not more, a core part of their offering and those are skills they need to bring in house and own if they want to be competent. I'm seeing more and more web and app developers be brought in house so airlines can establish these competencies. Digital/Product agencies are 'stepping in' in the meantime to help out (like Virgin America and Work & Co).
Airlines are logistics companies that fly planes. They have to have a functioning plane at the right location - and only the right location - at the right time, or they bleed money.
It baffles me that they ever thought outsourcing part of that logistics was a good idea. But then I've worked at tech companies that tried to outsource their IT departments too.
No, it isn't. The point of sale system is about taking payments. If your core business isn't 'the processing of financial transactions', then buying that is the sane thing to do. IBM being able to field a POS system doesn't make them a competent retailer. But they handle regulatory rules for you and let you be a competent retailer.
Airlines aren't just responsible for flying planes. They are responsible for the planes. It's not enough for the plane to show up in Chicago by 5pm, it has to be there and all of the necessary maintenance and safety work has to be done by 5pm. Anyone who could figure that out for them would simply go into business for themselves and compete.
It seems that companies would rather buy "insurance" rather than spend money on actual security. They have accepted that they will eventually be hacked or have a really expensive outage, so now they want a cushion of money to pay for any penalties and lawsuits.
I get the impression that for the most part, no one really believes in security, and doesn't understand that it's a process and not a product. They already tried throwing money at the problem by purchasing security products, which ultimately failed to deliver security. Now they're trying the opposite by cutting costs. Meanwhile there are very few consequences to losing control of people's data.
Why aren't there Security Process Engineers (SPE)? I was recently waiting on a massively delayed flight and observed three instances where employees left the computer behind the check in desks unlocked. I'm sure these computers checked off all the requirements for having a firewall turned on, anti-malware software, auto-updates turned on, etc. But the front door was left wide open because the employees were forced to move between their counters and the actual jet bridge. Since they get penalized for late flights, but not for security breaches it seemed as if they didn't care. This would be where a SPE would come in and propose something to remediate the process failure.
"They already tried throwing money at the problem by purchasing security products, which ultimately failed to deliver security."
So true. I used to wonder why corporates can't build a real security team. One would spend $1M on an annual contract but perhaps you will do okay for $700,000 for a few security engineers. Then I realized that most corps are indeed looking for insurances.
You see, managed security vendors have a team. You don't manage the hiring, the tooling building etc. Obviously you can't just sit back and watch them put out the fire. You train them to understand a bit about your environments, you work with them on triages and resolutions, and you work with them to integrate your systems with them (e.g install an agent etc)
But it isn't most enterprises' interest to build a strong security team. Many of the in-house security team's role is to manage incident response. Many of them don't really have a say, they are often consulted and that's it.
Many of the product demos sound exicting and if you are not careful during your POC you will end up with a medicore product. Even if you did your best during POC yo evaluate the product, you will realized the product is really medicore. It caught the low-hanging fruits and are full of false positive. You ask for better analysis but because they come in as black-box, there is a lot of back and forth before you can act on the issue. So at work I would get a ticket from the vendor and I would end up doing a lot of the analysis. That sucks because while I enjoy doing security triage, that's not my role, but the conflicting side is at least they caught something. I would have to engineer or deployment some solutions and monitor the whole thing, since no one man or team can do everyone's job.
Since no one wants to be responsible for other's job, that's the whole point of having a secuirty team. It just happens that team, the people who are building tools and monitoring incidents are outsourced.
Most importantly, there is very little control of what your vendor can do. Want a new feature? Want better reporting? Want to change some configuration? A lot of time you are out of luck, or you need to be patience.
But, seriously, security practice is not magical. There are best practices such as SSH to server using key or cert authentication, not password. Least privilege to run processes, so system admin/devops should create a checklist on what's in place what isn't. It does take a serious committment from management to move forward though.
And security is an iterative process. Evaluate the best most secure option possible for the moment being, and put a realistic plan together for the remaining unresolved concerns.
that CTO's would want to spend more and not LESS on operational costs
The incentives are the same as they always were. A well run IT organization can coast on momentum for a long time - everything is neatly automated. So they sack their good engineers and hire an outsourcing company and save an amazing amount of money just long enough to get that next bonus or promotion, and they're long gone when the momentum runs out and the wheels come off. Repeating the same scam at another company.
In the derivatives world (that I started my career in) there's a common theme: you can sell options, and generally you will make money. The only issue is, now and again you lose a lot of money. Often enough to wipe out everything you made.
But incentives matter. If you come in as a new guy, you want to impress by making money. And come up with some story about how you'll know when not to sell options. You also don't want to be the guy losing money when the other desks are making it. If there's a crisis and everyone loses, that's fine. Unforeseen, right? It happened to everyone.
The CTO is facing a very similar choice. Save money by not upgrading, work the security guys a bit harder and let them update the OS a bit slower. Don't think too hard about redundancy. Or code review, that takes away dev time. You're saving money.
But now and again, something happens. Maybe you have to leave, but you'll still have "saved money" for several years before. You'll land somewhere.
There's some evidence in software engineering research that offshoring isn't the problem but that the distance between team members on an organizational chart, however, is much more important. It's ok if an external party comes in as long as the resources are treated somewhat like equals to existing engineers. But when they become "that team" you will lose collaboration and things will fall apart partly because that team's manager may have different priorities than the local team.
When it comes specifically to IT, however, we're looking at almost a race to the bottom to save money immediately, sometimes to make more budget for software engineers that are revenue centers. Furthermore, a lot of IT folks want to stay current on technologies just like many developers do, but IT constraints are oftentimes even more than what their developer peers in the same company would have. This has led to a concentration of lower performers staying for a long time (not indicating skill necessarily - their environment is hostile to professional growth) and more ambitious technical folks move elsewhere.
It's been puzzling for me why IT orgs don't focus upon automation obsessively like most industries trying hard to cut costs have. The US coal industry automated a lot of labor and it's not like coal miners were making handsome salaries like most sysadmins did in the late 90s. The low performers all seem to have a strange obsession with creating as much manual processes as possible instead of, I dunno, writing some orchestration in Rundeck jobs or Ansible playbooks maybe. One decent automation engineer can replace dozens of lower skilled junior and mid level sysadmins even in bare metal environments.
The key is incentives. An IT outsourcer's goal is generally to fulfill their contractual obligations (to the letter not the spirit) for as little money as possible.
So if something like security or reliability isn't clearly stated in contracts (and it's extremely difficult to do that well) it is disregarded.
An outsourcing company is not going to spend money it doesn't have to, simple as that.
SAAS like gsuite is an entirely different prospect. you're buying a service from Google not outsourcing your IT to them. If google's service was insecure or unreliable no one would buy it.
I've worked in outsourcing for 9 years in Romanian companies as a software developer. I had the impression that the goal was to have the customer satisfied, not to fulfill a contract (as long as the customer was also reasonable, of course) and for pragmatic reasons: a happy customer pays you longer. My longest project was for 4 years, with the same customer. And my former employer keeps working on that project, 2 years after I left the company. In these 5 years the customer's company got acquired and for some reasons they had a pretty high turnover with their own employees, but they kept working with my employer.
When I joined the company there were 60 employees, when I left it 5 years later, there were about 500. I think they could only achieve this growth based on their high work ethics and the quality of their hiring their system.
To make customer happy you don't have to build reliable software. Corporations are not machines, there's a lot of people with their own interests, who make important decisions. You have to make these people happy, first of all. They can be bribed, convinced or fooled to achieve the desired outcome for your business. And if something bad happens, you have to dodge the bullet together, because you are in the same boat. So you blame other contractors, hackers or the guy who left the customer company some time ago, and do a lot of other things to convince the customer management, that it's not your fault, you actually saved them from bigger losses somehow and you can help to build better software. And they won't be happier, but they will think that you are good guy trying to solve their problems. And that's the only thing you need to sell another contract.
I wouldn't say that an outsourcing company wants to do a bad job, of course they don't, but things that are often invisible in the product (like security or reliability) will likely get less focus in a cost sensitive environment.
For example, completing a full security development lifecycle can add 10%+ to the costs of the final product. that's not a cost that a company will incur unless they have to.
In a bid for work, everyone says "we take security seriously" and the client probably can't evaluate the difference between someone who really takes all the necessary steps, and someone who pays lip service to that concept.
So the cheap company (who doesn't spend that 10% extra) looks just as good as the one that does, and they're likely cheaper.
Guess who's more likely to get the work (all other things being equal)...
Back in the 2012 I asked one of the department managers: are the customers willing to pay for automated QA tests? and he said "they usually do, because I give them the price per project and tell them 'in this price X it's included automated testing for the functionality. We can lower the price if you don't want the automated test suite".
I liked the approach and I think the same could go with security. Include an external security audit in the initial project price.
I don't see why an IT outsourcing company would not have the greatest incentive to take security and reliability extremely seriously.
Is there anything worse for an IT service provider than being blamed for a massive IT outage at a global corporation? This is headline news.
And I don't see any difficult contractual issues at all. On the contrary. A massive outage, by definition, means that the contractual obligation is not being met.
Security costs money, reliability costs money. If it's not in your contract, you don't pay for it.
if there's any outage that is because the customer didn't ask you do do something that's their fault not yours.
For example would you as an IT outsourcer pay for a redundant datacentre if your contract didn't call for it?
Would you patch all your systems immediately even if it caused availability issues if it wasn't explicitly outlined in the contract?
would you explain to your shareholders that your profits were lower this year because you undertook activities not specified in your contracts because they were good for the security and reliability of the services you managed...?
When outsourcing contracts are bid for there's a common experience of lower costs win. that inevitably leads to items that aren't strictly required being excluded.
>For example would you as an IT outsourcer pay for a redundant datacentre if your contract didn't call for it?
I would assume that the contract calls for particular service levels and that downing the entire fleet of a large carrier for days is in breach of that contract.
If the contract says "provide service X with 99.999% availability" the service provider cannot come back and say, oh but you forgot to specify that we should run a redundant data center to guarantee that availability.
>If the contract says "provide service X with 99.999% availability"
If you read those contracts, you have to read what the consequences are for breaking that uptime guarantee. Usually it's something silly like 10% off your next bill.
have you been involved in many IT outsourcing agreements? Cost is always a major factor in procurement, and in IT where there can be a strong market for lemons, it's often the key factor.
A company that's more expensive and can't clearly demonstrate in a bid scenario the positive impact of that increase in costs, will lose bids, a lot.
But the problem is that factors like security and reliability are often invisible in outsourcing contracts, as it's very hard to specify things like security exactly in a bid contract, and very hard for customers to tell the difference between an organisation with higher security and one with poorer security practices.
Everyone will say they take security seriously, but the cost of actually doing a good job on security is much higher than paying lipservice to it, so it doesn't really make commercial sense to do it well.
In general in fact I'd say that a lot of IT outsourcing contracts tend to lead to a market for lemons. It's very hard for customer to assess the quality of a companies staff for example, so the company with the cheaper staff can afford a cheaper bid, which looks just as good as a company with more expensive staff...
You are right, but they all do it because outsourcing is a signal that corporate management has lost faith in its ability to manage IT and now has but one method of judging value - price.
If the possible negative consequences to company's image is the only incentive, they will rather try find an excuse, since they've fulfilled the requirements of the contract. Can be anything, from missing security requirements (under the assumption that their customer will take care about security himself) to incorrect use of provided software etc. And they'll probably be right, especially if they actually tried to sell the security. And they'll probably be able to defend their position in court and sue for defamation.
You're not wrong that delegating operations COULD improve security and reliability, but it could also suffer from the same issues that branch office suffer from -- out of sight, out of mind, distance from the corporate HQ culture and scrutiny, etc.
Furthermore, often these issues rear their ugly heads when they sack the existing staff who are keeping 1000 balls in the air and rarely dropping one. In the migration to an outsourced team, or an offshore team, or even just another equally competent team in the same office, details will get missed. It's simply not possible to do a complete knowledge transfer. Things will get missed, and sooner or later one of the things you miss will turn out to be a landmine.
Of course, the original team wasn't perfect either, and could have also suffered a large outage, but the new team is more likely to do so, at least in the short term.
In a best-case outsourcing scenario, you either hire better people, or more people, or a vendor with better processes or understanding. You do the migration very carefully and miss as few details as possible, AND the quality of the new team is such that, within 3 or 6 or 12 months, they're actually outperforming the original team. AND you manage to avoid hitting any landmines during the transition period where they're underperforming. This requires a great bit of planning, execution, and luck.
Or maybe it's what is repeated here so often: don't let the perfect be the enemy of the good.
Perhaps it really is cheaper to have a few outages than it is to built the sort of systems that never go down. Every "nine" you add gets more and more expensive.
Or perhaps the systems are so complex that it's just impossible for them to be fully reliable, and this is just one of the costs of having the ability to fly anywhere in the world, cheaply and at the drop of a hat, and to have it work 99% of the time.
Or maybe it's what is repeated here so often: don't let the perfect be the enemy of the good.
Is that what you tell hundreds of thousands of passengers sitting in a terminal with canceled flights and confused airline employees during one of the biggest holiday weekends?
If your credit card number gets stolen or worse, your identity, and it's up to you to get that straightened out, are you just going to shrug your shoulders and write that off as the inevitable impossibility of perfection?
I'm saying yes, maybe that is the case. Maybe those scenarios are either a) cheaper to handle on occasion than building systems which will never fail, or b) unavoidable (with some low probability).
Air travel itself is not 100% safe. Planes crash. People die. Yet we feel that the probabilities are enough in our favor that we accept it.
What? "We" don't accept anything: strict regulations and processes were put in place to reduce the probability of accidents to the absolute minimum. Did this stop manufacturers and airlines to spearhead this or that new plane? Maybe - very few companies can carry that sort of compliance effort; but it's why the public trust air travel as fundamentally safe (statistically speaking, it's probably safer than cars).
Ignoring the offshoring part of this for a moment I think you're on point about the CTOs, it baffled me too at the beginning. When I started doing sales for a startup service and was targeting CTOs I discovered based on my interactions that outside of Silicon Valley companies or obviously tech centered companies, CTOs in other countries and industries are in many cases just the usual political shark type person interested in showing how they were able to cut costs to help improve profit and often there isn't the real interest in tech to understand why doing things better is so important.
As an example a person in a tech leader role at a major company in the UK who I thought may be interested in hearing about a product was quite rude about me never contacting them again and refusing to take any calls or emails before even hearing about something that other CTOs has at least looked at and some had even become customers. She happened to have a personal blog that I checked out and all it mentioned was her passion for swimming, nothing at all, not a single thing about tech. Why is this person in this role then? Beats me but it is sadly commonplace, until you get people at the top that truly understands tech and was actually a software engineer themselves at the start of their career and are things won't change.
> As an example a person in a tech leader role at a major company in the UK who I thought may be interested in hearing about a product was quite rude about me never contacting them again ...
You didn't spam them did you?
Your description of their behaviour sounds a lot like the standard response to spammers. eg FOAD
No I did not, and that's the issue, a lot of these people are not actually interested in making things any better, they'd rather not be disturbed instead. And then something like what happened today happens. And then people are misdirecting blame to others. The leadership is in most cases to blame.
In that particular case I contacted them by email, introduced myself and explained why I had contacted them specifically, what I knew about what they were doing and the problem I was solving and asked if they wanted to discuss it. They responded and said they were not interested. I replied and said that's fine, if they change their mind to get back in touch. They wrote back again with the 'foad' response as you suggested, really quite a shock considering I had expressed already that it was all good and I had considered it the end of the conversation and didn't actually expect a reply. Unless I actually share the emails with you I'm sure it just sounds like he said she said but I do think I'm describing this quite accurately, most people did not reply like this, especially people who I had researched and were generally interested in software and tech, they were either interested in discussing the opportunity or calmly explained why they were already good. I am just providing my experience of some bad leadership at the top when it comes to software/tech.
Note that this attitude is what pushes some sales teams away from reaching out to line management, and towards C-level-targeted sales. If you hate "golf-cart sales" outcomes where a C-level forces a solution down your throat, then learning how to communicate with sales people will richly pay off. It is part of managing upwards, by short-circuiting sales warm approaches to the management you report to, and turning them into your allies to help you pitch your priorities that happen to align with their sales goals.
I am up front with the sales people who approach me, and tell them when I anticipate the problem space they solve will rotate onto my front burners, my anticipated budget to switch (usually in the form of "if I switch to what you propose, I only have $X to do it with, all in, software and services"), the benefits from my current solution I want to ensure stay in place, and the pain points I want to solve. That usually ends the conversation right there and then, I'm tagged "no-contact" in their CRM, and the spam disappears.
In my personal anecdata experience of observing C-level sales efforts from the inside follow this general pattern. I encourage you to look at these situations from an angle other than "all/most sales people contacting C-levels are spammers, and C-levels who respond neutrally/positively are idiots for not recognizing spammers".
Generally speaking, C-level relationships are cultivated over a long time with the kind of sales and marketing budgets you've read of; dinners, sporting events, wine tastings, etc., with some proven sales account manager. However, it takes a lot for some sales opportunity to rise to the level of bringing it to the attention of this relationship. If you want to stop what you call spamming, then your job is to manage upwards by ensuring pain points do not rise to that level. This keeps the sales account management activity focused on the numbers when the support contracts come up for renewal.
However, if you have not been successfully managing upwards, then lines of communications have broken down between C-levels, or between CIO to your level. The first you can't do much about. The second is partly within your control. If you ensure your manager has nothing from the business to complain about that can be addressed within the solutions you are responsible for, then you've done what you can about it; you can't fix what you aren't told, after all. Where most technology-oriented staff misstep is they think they need to hear it from their management; the staff who successfully short-circuit pain points coming up to the C-level's attention realize that they can also ask, and more importantly, demonstrate they can communicate and coordinate between departments and people to help address those business-oriented pain points before they percolate upwards.
Where these sales account managers pounce is when the pain points become so grave that the C-levels hear about it and feel they have to "do something" about it, and "it" is a competitor's solution. And if you really like that competitor's solution, and hate the idea of switching away, be on the lookout for a call from one of the sales reps assigned to help that sales account manager. If you've even heard about the pain point, then you will be able to help drive the discussion, and most of the time if your favored vendor is on the ball, close the window of opportunity for the sale to be floated up.
You're conflating real
sales emails and spam again and really seem to hate sales people. How are the people who respond idiots for actually being happy to hear about a product or service that solves their problem and signing up.
I'm a C-level executive. I am more familiar with my infrastructure and pain points than any cold-call salesperson will ever be, so if I need a third party solution to a problem, I use the internet to find it.
I'm not going to waste my working time communicating with a salesperson back and forth for a product I'm not interested in. One, it only validates that I read their email, and two, the end result is the same - I don't buy the product.
Ahhh. That's a misconception on your part. There is literally no difference between any kind of sales email, and spam. All sales email to people that haven't opted-in previously, is spam. 100% of it. Thankfully, some countries have even introduced laws against it.
Good point. You've convinced yourself the people you're spamming are in the wrong, and they should trust someone who's already shown to have a distinct lack of ethics.
Literally, by spamming them.
It's absolutely no wonder they're telling you to FOAD, as that's the correct response.
Please, change to a different profession. Preferably one that contributes to society in a positive way instead of your present one. :)
In one way the CTO gets the 'purchase' of insurance here. Not against the company having issues but for their job because they can say, "Well I hired the best company to do this. If they couldn't do this, then we definitely couldn't!"
Don't manufacturers face the same sorts of issues with supply chains? I mean, if there's a train wreck or an ice storm, the people who needed those deliveries are pretty much screwed.
I think we like to pretend that this is a problem unique to informatics.
1. Massive security breaches affecting major corporations, governments, and so on.
2. Massive IT outages, affecting corporations, governments and so forth.
It doesn't take a lot of incidents to mark them as 'massive' since things are strongly centralized in this world.
Meanwhile, in the last 15 years of my IT career, never have I seen such a strong push towards offshoring. One would think that with such a vulnerable IT landscape mixed with an unprecedented dependence on IT infrastructure, that CTO's would want to spend more and not LESS on operational costs.
I'm slightly baffled.