Docker is being used to cure diseases, keep planes in the air, to keep soldiers safe from landmines, to power the world’s largest financial networks and institutions
'We really need to land, but the container responsible for landing navigation has crashed and I can't restart it because Docker complains that the endpoint is already registered on the overlay network'
Jeez, everyone from Linus Torvalds to Intel to the freaking power company could take credit for these things. Maybe tone down the "we're saving the world" rhetoric, Docker?
It makes more sense in the context of the criticism that Docker gets. It's a counter to the inane "Docker isn't stable enough, Docker isn't secure enough, Docker isn't whatever enough to use in production" critique that some obstinate old ops person will throw up because they'd rather keep their job comfortable than try to accommodate a need beyond their own. I don't read it as Docker does all those things so much as Docker is used to do all those things and diseases are getting cured, planes aren't falling from the sky, soldiers aren't getting unnecessarily blown up and financial networks are only crumbling due to their own internal negligence. If it can be used in all those situations, it will probably work for your crappy pool-of-app-servers-on-top-of-a-database webapp.
> obstinate old ops person will throw up because they'd rather keep their job comfortable than try to accommodate a need beyond their own
Oh, come on. This is why I hate Docker. Right here. It's like a heat seeking missile that looks for the friction points between engineering and operations, then explodes in a fireball of "do we really need operations? They're just old and obstinate and in the way of progress," without even really stopping to ask what progress is from the perspective of operations.
Some of us like to think of legitimate technical reasons to push back on Docker in our own circumstances and deployments (some of us even read code to help make that determination), but I suppose your dichotomy would be fun to navigate for everyone involved. I've definitely been in environments where one side characterizes operational pushback as change reticence, which immediately personalizes the discussion. That's a tough environment in which to argue, especially when you're arguing with people who will not be on call for the crap they are trying to deliver to you.
The casual ageism in your comment is, sadly, expected. I'm growing accustomed to HN considering it acceptable to go after "old" and "operations" with equal vigor, which speaks more broadly to worrying trends in our industry. I'm 31 and have devoted my career to operations, and I already have an exit plan. That should say something about the industry, and I think this comment is a fine microcosm of that.
For what it's worth, I'm older than you and started in ops. If it helps to put things into context, I spent a ton of 1997 using a chisel to shave a bit off standard RAM to help avoid Sun's near 10x markup. The startup I worked at was provisioning around 20 Sun boxes per week and we had to patch together tools to manage infrastructure of that size before off-the-shelf tooling to do that really existed. I wrote a lot of expect in 97-98.
Where I get frustrated with ops is when they forget that they're in the service business. They're there to serve application developers in the same way that the application developers are there to serve customers. An application developer pushing back at a customer ask with the line, "without really stopping to ask what progress is from the perspective of developers" would be rightly lampooned. And yet somehow there's people that think the ops mindset is reasonable. In my view, "no" is not a valid viewpoint. Application developers are telling you they want the kind of functionality that Docker offers. That makes it your job to figure out how that happens, not if it happens.
Where I get frustrated with ops is when they forget that they're in the service business. They're there to serve application developers in the same way that the application developers are there to serve customers.
In my view, you've got that completely backwards. Both departments serve customers in a roundabout way. Dev is responsible for turning user stories (generated either by other devs, focus groups, and so forth - which may, but doesn't necessarily involve an actual paying customer) into code, ops is responsible for keeping that code running.
If dev screws up, user gets bugs, or the app goes down. Same for ops - it's just the class of bugs are different.
Application developers are telling you they want the kind of functionality that Docker offers. That makes it your job to figure out how that happens, not if it happens.
As long as me and my ops team are the primary people with skin in the game (read: getting 3AM wakeup calls), me and my team will maintain a say in what tech stack we're responsible for the care and feeding of. The dark days of dev throwing code over the wall to ops is long gone - I am gobsmacked that you'd suggest that this is for some reason improper.
What dev often forgets, and the cause of much strife, is this justification. If you want a team of n people to learn this new thing and its new ways of operating, then you need to do an honest, dispassionate cost/value analysis. And show your work, dammit. When this never happens, it conveys an "Oh great, Dev is on one of their 'Ooh, shiny!' kicks again" mentality to the rest of the team, especially those that aren't politically connected enough in the org to know any different. I know it's because you're busy and have insane deadlines to meet. Meanwhile, the phones ring at 3am, and the message is lost.
The result is a strained and adversarial relationship, often to the customer's detriment.
I work in IT, so not exactly the same, but I can relate so much to your pain. Teams are constantly bringing in some half-baked app that "Worked in the demo" and give us the registration key and a date to have it running by. Invariably it become the biggest headache for us, because it never works, fails at the worst times, etc. And when it doesn't work, who is the finger pointed at? The person who bought into it wholesale without any testing, or the team who's been working 60+ hours every week trying to string it up with paper clips and rubber bands?
That's exactly the same. Don't sell yourself short. You're doing operations with a different audience, and that type of behavior is just as common in external-facing operations. Hell, it doesn't even have to be an off-the-shelf product; I've been through that song and dance with in-house software.
> As long as me and my ops team are the primary people with skin in the game (read: getting 3AM wakeup calls), me and my team will maintain a say in what tech stack we're responsible for the care and feeding of. The days of dev throwing code over the wall to ops is long gone.
Here's where we're in violent agreement. I believe the best way to ensure that the pager doesn't go off at 2am is to give that duty to developers. Far and away the most likely reason for a production issue is new code that went live. No matter how good communication is within an organization, ops will never have the same context for issues like this that dev will. So dev being the tip of the spear with regard to incident response will result in faster resolution. The job of ops is to be an enabler of this responsibility. They should be building tooling and infrastructure that makes it easy for dev to do the right thing but doesn't force decisions on dev. Ops is an abstraction layer that gives developers higher-level primitives for infrastructure that allows them to solve the customer problems more easily.
But there's no substitute for the dual benefit that you get from a) oh...I wonder if this problem is related to that change I reviewed yesterday and b) hmm...I've got pager duty this weekend, so perhaps I shouldn't just LGTM this PR. And for that reason, developers and developers alone are responsible for a working application. Ops is only there for support.
> Where I get frustrated with ops is when they forget that they're in the service business. They're there to serve application developers in the same way that the application developers are there to serve customers. An application developer pushing back at a customer ask with the line, "without really stopping to ask what progress is from the perspective of developers" would be rightly lampooned. And yet somehow there's people that think the ops mindset is reasonable. In my view, "no" is not a valid viewpoint. Application developers are telling you they want the kind of functionality that Docker offers. That makes it your job to figure out how that happens, not if it happens.
Nope, nope, nope, nope. I quoted this entire paragraph at length because it's all just completely wrong.
Read Site Reliability Engineering that Google just published for an alternative perspective. SRE at Google is specifically and vocally organized to have authority to push back on engineering. Without exception. "No" is absolutely a valid viewpoint, particularly when you're asking a team to go on call for your code. It's not you sleeping in the guest bedroom and fucking up a marriage, now is it? No, it's us. Let's reframe this: you are responsible for software. I am responsible for production, which happens to run your software along with, likely, a bunch of other things that you don't know about. You said in another comment that operations exists solely to abstract infrastructure for developer consumption, which is such a heavy marginalization of what we do that I'm amazed you even identify yourself as an operator of any kind.
I'm not saying this to be an impediment to you. Both our paychecks flow from the same mission. I just need the authority to push back when your decisions threaten production (especially when you don't understand why). It's contrived, but on one end of the spectrum, if you said let's rewrite the stack in ColdFusion, it's not my job to figure out how to enable you. It's my job to figure out how to keep the lights on, and defending that properly means investing absolutely zero in your ColdFusion rewrite. Escalate to my management if you choose; if it comes down to the ColdFusion rewrite or my continued leadership of operations, I'll go somewhere else. So the more you ask operators to be "yes people" for engineers, the more you end up with operations outsourced to Tata and Infosys -- who will say yes to everything you want. Have fun with that.
Seriously, operations will be the equivalent of COBOL consulting by 2030. Just wait. I'll hit my mid-40s, pull a Mikey Dickerson, and build a consultancy to swoop in and clean up the mess that you're creating by marginalizing operations as strongly as you are and laugh all the way to the bank.
Particularly in SRE, I'm a software engineer who happens to enjoy the plumbing that keeps your code running. I do not exist to serve at the pleasure of application developers, nor do I accept any imagery of my being inferior because I don't spend my days writing red/black trees to shave a few milliseconds off the news feed. I can, and have, do/done SRE without a single application developer customer, and I like to think of SRE as a little bit better than application engineering: you push to a repo and wait. I push Enter and 10,000 things happen instantly. Who's having more fun?
Tossing lame code over the fence is how to sow bad seeds with operations. That you've then had negative experiences with operations, enough to generalize us as a species (are you hitting sysadmins? ops? sysops? devops? hwops? netops? SRE?), is only a reinforcement for that to me. I would say between a combination of educated guessing based on your opinion, prior experience, and a few other inputs that it's pretty likely that your gripes with operations stem from flawed expectations on your part.
To me, "old" as a prejorative has about as much to do with age as "gay" as a prejorative has to do with homosexuality.
The meaning people use the word to get at is "from another time when problems were solved differently, and staid in their opinions toward new solutions vs. those from the previous era."
I've heard myself called an "old man" online for liking SQL stored procedures. I was 23 at the time. :)
The reality is, as far as I can see, there's just not much benefit in using Docker in a crappy (read: small) pool-of-app-servers-on-top-of-a-database webapp. I've had a lot of newer devs pushing to use Docker. I haven't had these newer devs give a solid reason why relatively low-volume, simple web apps need an additional layer of complexity over an easier-to-manage VM setup with automated provisioning.
The largest financial systems are running on COBOL/mainframes.
Planes are running on embedded microcontrollers still written in C or Ada. (They can't even get the approval to write C++).
Docker is not 1% ready for critical use. The day these things will use Docker, planes will fall and we'll have the next financial crisis.
As someone who worked both in finance and aerospace, handling systems that traded billions of dollars and planes that were carrying hundreds of passengers. This is my testimony on the year we tried to use Docker in production:
http://thehftguy.com/2016/11/01/docker-in-production-an-hist...
But I want to give them a fair opportunity to justify this statement. Can someone give me examples of diseases being cured via docker, or planes being kept in the air (etc)
Reproducibility is a big problem in analytic pipelines, specifically genomics. Researchers tend tweak their tools here and there and it has always been quite difficult to "package" a pipeline. Docker makes that much easier. We've seen a huge movement in the containerization of genomics pipelines, the results of which (we hope) help cure serious disease. In my world, we're using Docker every day to try and find cures for pediatric brain cancer. It's obviously not our only tool, but it helps.
Not in genomics but the same in true in spatial pipelines. Trying to get a machine setup with the correct set of libraries and packages across various languages is almost impossible on dev box much less on a prod cluster. Docker makes this easy to do once locally, validate the build and then use the same container to do the production runs.
Neural networks that construct ontologies from gene similarity matrices for discovering cancer pathways are tools that help cure cancer. There are a myriad of technologies supporting these type of applications; Torch, Python, Tensorflow... Docker is a minor detail that has as much to do with curing cancer as JSON or HTTP. When you make a claim like the one Docker is making, you should be a critical piece of the pipeline that actively contributes towards the end goal, not a piece of infrastructure. As a scientist, I find it disgusting that they're using cancer research as their buzz word for stability and importance.
> Reproducibility is a big problem in analytic pipelines, specifically genomics... [Docker solves this problem].
There is a legitimate fundamental issue with reproducing people's setups and Docker solves this. Ergo docker is valuable technology for data science.
Literally typing `docker run...` to reproduce results (vs. VM setup, installation etc.) seems important enough to justify their phrasing.
Fact: there are handfuls of data science startups that essentially rebrand docker and have been funded in the hundreds of millions of dollars specifically to solve this problem.
I assume they mean "minimize time stuck on the tarmac" vs "literally flight critical", but I doubt even that.
The former would imply that something like dispatch functions are in production running on docker. I suppose that's possible, but it's not an area that lends itself to tech that's not very solidly proven for the space.
Because containers make life every so slightly less terrible? And container orchestration allows for better efficiency of resources?
The following government entities are known to use, or have used (Docker) containers:
* USAF
* US Navy
* CFPB
* CDC
You can look up their RFPs online and see why they want to use containers. IMHO, these are pretty direct ways of keeping soldiers safe, and curing disease.
It probably just means some company that works in pharmaceuticals uses Docker for something, and another company that works in aerospace uses Docker for something, etc.
Somebody should buy a television ad for them: "Feeling tired?
Ask your doctor if Docker is right for you". Include an actor in a lab coat, with a quote from the CEO saying "Docker is being used to cure diseases".
The docker marketing team does seem to be accomplishing their intended purpose, but their preeminence in the company's decision-making does rub the cynical dev crowd the wrong way!
The thing is that all these things can be done w/o docker at all. It may make it simpler (for some definition of "simpler"), but it is not necessary to do any of these things. Now the Linux kernel on which docker is built OTOH is probably not so easy to take away.
The same could be said of almost anything though: "[Linux|Raspberry Pi|Angular|Python] is being used to cure diseases, keep planes in the air, to keep soldiers safe from landmines, to power the world’s largest financial networks and institutions
For Linux, the Raspberry Pi and Python it is probably actually true about the first two, not sure about the landmines, Raspberry Pi in finance I doubt but Linux and Python for sure.
Because LXC on a technical level is much simpler, doesn't make mess in network
and filesystem setup, and is easy to understand. Docker does some heavy magic
for things to work for programmers (the ones who can't be bothered to learn
how ethernet bridging or IP routing work), so the whole thing feels brittle.
I've seen a test version of an app cause issues such that the launcher would crash (i.e. "respring"). Probably some intersection of particular api usage and OS bugs.
To the unobservant user it looks like the OS is rebooting, although it's much quicker than a real reboot.
psh, iOS can be rebooted from safari by web pages - just send them enough images with 3d transform + filters and watch the memory allocator kill the whole system trying to compose the thing.
Yes, I believe they can. I haven't had my iPhone (6S) crash more than 5 times that I can recall in the 14 months I have had it. Every time it has rebooted I have been in a third party app. The phone begins acting strange, gestures may stop for a few moments then it goes black and the Apple logo shows up on the screen as it reboots.
Two of them have been in the last month or so. Unfortunately the phone doesn't just reboot. It shuts down completely and refuses to boot back up with a low battery warning. I believe this is actually hardware related but I have had it crash when trying to call an Uber. I think Facebook may have caused a crash too.
Uber was especially painful because hailing taxis in my city (Seattle) is essentially impossible and pay phones are not a thing that exist in 2016. Luckily I was near a bus stop and know how the lines work without my phone.
It was a shocking illustration of how dependent I have become on my smartphone.
Evidently they can. I witnessed it myself, even did a cold start (hold power and home until you see the Apple logo) and it still happened on a completely up to date iPhone 6.
I have worked in teams without a formal tech lead and yes, the most senior/confident developer would work like that (even without extra pay).
However, these teams also tended to be most consensus-driven. Decisions would take forever because there would be a tie and people would just put it aside, or they would avoid hurting other people's feelings directly (it often turned somewhat personal), etc. It was very inefficient.
I'm in London, and public transport is good enough for most journeys, and I loathe black cabs far more than I feel anything (except distrust) about Uber.
Uber works wherever I am in London, it's cheap enough (realistic compared to black cabs).
It's good enough, but not good enough that I'm fine with their various invasive activities. One not mentioned elsewhere is how they use both rider and driver mobiles as a kind of distributed cache/database. Which is the thing that started me off on "kill the background process", initially to spare my own data and battery.
Don't count on Ruby getting 10x faster anytime soon. The meta-object model and method-missing reliance of many of the libraries make performance optimizations of JITs a very limited enterprise.
While your work is very impressive, and I'm a big fan of Graal and systems like it, I think it's a mistake to say that something is solved when there's a single research system that gets good performance. As another example, there have been systems with fast continuations for 40 years, but I still wouldn't call that solved. I've built a JIT that's very effective at optimizing contracts, but I don't think that's solved either.
Well the statement was that it was not possible to remove the overhead of method_missing (well they didn't actually say 'impossible' but that's how I read it, and I also took them to be referring to peak performance, because people usually are).
I solved the problem of showing that it is possible to do that. Before people thought it was impossible, and now they know it isn't.
I'm not claiming I solved other problems such as removing the overhead with low memory or removing it in corner cases.
I hear the Truffle+Graal combination is getting very good results about Ruby. Here is video about it (pointed to the missing-method example): [1]
They're hitting 35x in some benchmarks [2]. Most benefits being in workloads where Ruby has to continuously talk to C; the result-passing part gets compiled away.
It is available in Java 9, if I'm not mistaken. LLVM IR bytecode ingestion is in preparations too [3].
Three of the biggest Rails Shop, ( this is assuming the most popular, resources heavy usage are Rails related. ) all run on CRuby. That is Cookpad, Shopify, and GitHub. ( Basecamp in rather small compared to these 3 )
I wonder if there are any popular webshop ran on JRuby, or why those three dont use JRuby.
It's true that many libraries would perform poorly because of a reliance on method_missing. But what can be made quite fast is code that either uses define_method instead, or uses define_method on the first failed call.
In fact, a tracing jit could even make the method_missing case fast by automatically specialising and optimizing missing coupled with a vtable/dispatch table.
My forever-in-progress ahead-of-time Ruby compiler uses vtable based dispatch for every method name that's seen in the program at least once, and that's usually most of them (unless people e.g. construct method names dynamically at runtime).
To handle method_missing, it creates thunks for each method name and fills the missing vtable slots with those, which is similar to what I suggest above - the next step of dynamically optimising method_missing for called symbols and replacing the vtable thunks would be a relatively minor step in a JIT.
I don't think anyone expects Ruby to get dramatically faster in the very short term, but there's lots of opportunity to make most Ruby code much faster in the medium term.
I think we'll also start seeing implementations that do things like JIT influence how people write Ruby, because a lot of things won't matter much for MRI performance but will be a huge deal for implementations that uses compilation techniques, so it's possible to get a lot more "compiler-friendly" Ruby while still writing clean Ruby that runs well on other implementations.
e.g. the above "define_method on method_missing" basically boils down to (pseudo code):
def method_missing sym, *args
raise suitable exception if sym doesn't meet right criteria
define_method(sym, args...) do
... whatever ..
end
send(sym,*args)
end
If it lets an implementation speed up its (ab)use of method_missing enough, you'll see people adopt stuff like that.
Smalltalk has those same features (which is where Ruby got them from). Almost all of the high performance JIT technology we use these days was originally invented by Smalltalk VM hackers. They invented that stuff specifically to make that kind of code go fast.
The 3x was compared to Ruby 2.0, on some slides they claim Ruby 2.4 is already 30 - 40% faster then 2.0, so effectively you will only get around 2x more speedup.
Java with Ruby's syntax and undeclared typing would make Java programs so shorter that it would be worth it. But there would be nothing left of Java, except the object orientation. It would just be a simpler more static Ruby.
No because it's statically typed, too much work programming like that and it's not worth it for the kind of software I develop (backend of web/mobile apps), but thanks. Suggestions are always appreciated.
It's object oriented and runs on the top of the Erlang VM. It maps classes on gen servers, which is pretty natural and it spares the developer all the boilerplate of creating a gen server module. Unfortunately the language designer killed it because "it's pretty bad to have two competing niche languages with nearly identical features." The other language was Elixir. However the two languages are very different. Elixir is 100% functional, Reia could have been the language to transition millions of OO developers to a functional and concurrent world. I could have used it instead of Ruby.
This is an odd stance. Backends are where static typing shine, and where those who like static typing sing its praises. Maybe you just don't like static typing?
I really don't find static typing to be more "work". You still have to think about types in dynamically typed languages, you just get to skip specifying them.
Skipping that saves a lot of keypresses and time. I always found very little help from static typing. I know what I'm doing if I type hash = {} in Ruby and there is no need to declare it as HashTable (what else could it possibly be?). About the types of the keys and values, they don't matter much in my experience. Maybe it's Ruby or it's Rails, but they tend to fall right into place without surprises.
In case you're not aware, many static languages have implicit typing nowadays. I still write Ruby most often, but for example in Go you could just write `something := "string"` and with that := operator it'll manage the type assignment for you. Lots of other static languages offer that now also. I'm not well versed on modern Java, but the type system in Go pretty much gets out of the way for the things I've done with it.
> I type hash = {} in Ruby and there is no need to declare it as HashTable (what else could it possibly be?)
In java you have options other than a HashMap, there is LinkedHashMap (which preserves insertion order) and TreeMap (which stays sorted according to a comparator). There are also extensions for things like Immutability. The checking can also guarantee that it will only have one type of object, or a common interface, or you could allow it to accept any kind of object.
So, yeah, there's quite a few things that an associative array could be, and there's reasons for all that, and i've used most of these on web backends.
There are other ways of meta programming that are still performant.... that is why many Ruby devs see Elixir/beam and Clojure/jvm as good language/platform alternatives.
If you don't mind starting from scratch and using a statically typed language, crystal is amazing. However, you will not get vary far porting any large ruby project to crystal without a rewrite from scratch.
However many of the libraries list under "third party" are either unmaintained or not really ready for production. The stuff in the standard library seems pretty good though (although I've not used Nim for serious production work)
Except it's nothing like that, it's like an interpreter framework written in Python with a reference implementation that happens to be a Python interpreter. Write your interpreter in RPython, use the right decorators and boom you have a compiled interpreter with a free JIT (with speeds that beat C in some workloads, apparently).
Why do this?