I remember going to AWS Reinvent last year and having some beers with a bunch of people who did devops. We started talking about tools, and they were utterly flabbergasted, that we had not embraced docker. They went on an on about how simple docker made HA, and handling fail overs/maintenance. More or less made it seem like it was the greatest thing since sliced bread.
Me and a few coworkers decided to try and dockerize some of our serivces, and move our staging ES cluster to docker.
For the most part building our own containers was easy enough, for the various services. The biggest issue we had was with Elasticsearch, since we have 4 types of nodes. So we ended up building 4 specialized containers for each node type.
Then came the issues:
* Making containers work together, for example we use a logging agent, we decided to make that its own container. Then actually getting a way to share a log directory with the agent, was very painful and unclear. (Honestly the single most frustrating thing i recall was asking for advice in irc and more or less being told i am doing it wrong)
* Containers would randomly crash under load exiting with error 137 out of memory. Apparently a few of our services would randomly leak memory, but only when running inside a docker container vs ubuntu 14.04. (I never figured this out)
* Containers would randomly become unreachable, forcing me to kill them and restart them.
* Random hanging issues, suddenly docker exec is hanging. Being told to revert to an older version, or install a newer version is tiring and makes my environment very inconsistent.
* Trying to debug why our app in the container isn't working is not fun at all.
However the single part that killed me was, i was chatting with one of the people who i met at Reinvent, and i mentioned my issues. He acted like it was completely normal for these kinds of issues.
After a solid 2 weeks of random issues and the constant barrage of pagerduty alerts, i just rolled everything back to static EC2 instances for staging, and have ran into 0 issues. I want to try containers again because i want them to work, but i have just had too many issues.
We have similar stories. We even brought in a devops consulting firm, who insisted on a pure AWS / Docker (ECS & co) stack. After a couple months of shocks, crises and occasional all-nighters, I just started deploying backup instances on Heroku, so that the QA and design teams wouldn't get blocked by the weekly clusterforks. After a few weeks of smooth sailing on that front, we just activated logging and auto scaling addons, and blessed the Heroku stack with the production domain name and CDN.
Heroku gets expensive quickly at scale, but the engineering required to make "the future" work on AWS was unmeasurable (because it never really succeeded for us).
I don't even know what to blame for the whole episode. Docker's incomplete architecture (2015)? AWS's inability to abstract and manage complexity into something that just works as described on their product pages? The consultants? Myself, for putting faith in that triad of unfamiliar but crucial people, services and products? Whatever, it's no longer a current problem for me.
It's exactly these kinds of issues that add to my impression that containerisation, and Docker in particular, is more of a religion than a solution.
I don't hate it per se, but I'm just a bit fed up of people telling me I should be using it without being able to articulate what problem it's going to solve for me.
At the moment it feels like just one more thing to learn, and one more moving part, that isn't strictly necessary in order to achieve what I need, which is basically to deploy software without a heap of aggravation.
For me, with appropriate base containers, the real benefit is load time... booting a full VM has a lot of overhead, and takes quite a while. Loading a dockerized app is much faster. If you're extending this for your testing environment, that can be a huge win over a larger team.
Being able to load more containers than VMs onto a server is another piece...
Beyond that, it's not too much different than running VMs was earlier on, it took orchestration and tooling to get working and there were growing pains... much of that abstracted well before becoming popular.
In the end, containers are just another step towards wherever we are going... I think it's pretty nice, but the tooling is just now starting to catch up.
I think that's a legitimate argument but deployments to VMs that are already up and running can also be fast, and for me they are, so it's not overwhelming. My feeling is that one day I'll reach a level of complexity/deployment time where the benefits of containerisation become clear, but I'm not there yet, nor anywhere near it.
That's a discipline issue though... shared nothing start from zero... The "docker" method doesn't upgrade an app already running in a container, it creates a new one.
I wouldn't use it for everything, but whenever I encounter a problem where applications crash or behave weirdly - the problem is with the app, not the container technology. It's not because the problem only manifests itself in a container that the problem is suddenly "docker".
And as with every technology, you have to understand it's strength and weaknesses. I use Docker internally and in production with very few issues other than it advancing way too fast at the moment, but am always amazed how many people dive in head-first into a Docker adventure without understanding what it is and how it actually works and it's limitations.
This is from the perspective of a relatively small development company with applications where scaling is a non-issue. Our problem is that we have a ton of active projects. To give you an idea, our internal CI now still has over 500 active build jobs after a recent cleanup.
This CI is the first thing where Docker shines, it was an absolute god-send. I got rid of tons of frankenstein build slaves with unpredictable configurations, and replaced them with one huge VM running docker, with build images per project. This made this massive mess suddenly perfectly manageable, documented and version controlled. Need to build something in a build configuration from 1 year ago? Not something I want to do every day, but not completely impossible either, since I still have the exact same docker image in our internal registry.
Other than that, upgrading internal tools became a lot easier. Everything used internally (redmine, jenkins, ...) is containerized, which means it can easily be tested, migrated, cloned, ... It enforces data separation, which means it's clearer and easier to backup/restore and test these things. It means that now whenever a new version is released I can easily test this with our real data to see if there would be any problems in our configuration, and if not, quickly deploy a new version.
I feel that sometimes I'm pro containers, and sometimes I'm very much against them. Docker does make deployments and upgrades very easy once you have the initial infrastructure set up, and you can deploy applications directly from upstream as "units" so that each deployment is exactly what you want; deploying containers helps prevent hosts from differentiating too much.
If you have the infrastructure in place to rebuild your virtual machines though, containers offer little benefit other than perhaps not having to package your software.
At work we're mostly deploying docker containers using puppet, and often when docker fails to work, I can just obliterate everything and run puppet once to re-download images and set up everything again. I would not trust docker to manage any data that I can not recover from elsewhere.
My biggest gripe is that configuration management feels like an unsolved problem. Most examples I see on the web seem to simply ignore it, or do the usual "mount a host directory inside a container" thing which has issues with file permissions and host/container UIDs clashing, and just feels inelegant.
I'm also developing a dislike of Docker on account of it simply failing to work at times for no discernible reason, and having stupid issues like IPv6 simply not working correctly.[1] It's also rather inelegant when used on a systemd-based host because it wants to reimplement most systemd functionality.
I'm still waiting for rkt to mature a bit, maybe it will be better.
Trying to debug why our app in the container isn't working is not fun at all.
My experience with AWS was similar, and I got to a point where I had to stop accepting clients who had built (or insisted on building) their infrastructure on AWS. It actually kind of reminds me of a pyramid scheme where they ensnare you with a seemingly good deal, but in order to get that you've gotta buy in just a little (time, cost, etc) more and more -- just to be able to debug -- and more until you're calculating sunk costs[1] bigger than any company should have to spend on something as trivial as, say, a Wordpress site. It's highly unlikely that 95 percent of the SMB websites out there need the overkill of AWS infrastructure / EC. Do a cost benefit analysis.
I think the assumption arose not from the demographic of people on Hacker News, but from the username of that poster. "Shawn" is a fairly common name, where one in every 2000 people will be named it. [1]
Meanwhile, "Shawnee" is a really rare name[2], one you may not be aware of if you didn't grow up in the US (or in particular parts thereof). (Apparently only 4000 of them are alive today!)
Just as an aside, using "they" by default can be very confusing, especially when plurals are involved. I use it sometimes, but prefer to use the passive voice instead ("the parent poster", etc.) I definitely do not have time to check every person's profile when I comment on HN.
Thanks for pointing out where the name confusion may have come from.
> I definitely do not have time to check every person's profile when I comment on HN.
At least then we should not make explicit such gender assumptions, regardless.
I have never heard the name "Shawnee" and generally do not assume usernames on web forums are indicative of real-life anything (e.g. you are 'striking', but apparently that is not your name).
I agree 'they' is an awkward construction in English, but for many people the alternative using passive voice is a little more complex to use (cf. non-native English speakers).
As a Polish immigrant, I personally think "singular they" is worse, because it creates ambiguity where there doesn't need to be any... which makes things a bit more confusing. Meanwhile, I think my proposed alternative is native to most other languages, and should translate fairly easily.
I'm not sure I see the point in thinking this long and hard about pronouns, but to each his own.
Yes indeed, to each their own. The singular "they" can be confusing in certain cases, but that's not really a problem unique to "they". "He", "she", "it", "we", and "you" can all be confusing in relatively common circumstances.
But you're going to have to get used to the singular "they", because it seems to only be getting more common (historically, singular "they" was acceptable until the late 19th century, so the strictly plural "they" might be seen as a 20th century anomaly).
I, too, think Docker is the bee's knees. But I don't think it would be good fit for a datastore (even though zeitgeist is to Dockerize all the things).
Docker seems to work best in an SOA type environment where you have a set of stateless services that might use different stacks. Docker simply unifies their provisioning and deployment.
My impression is that it really shines when you need to scale and you can easily spin up and kill large amounts of nodes/compute without any impact to your service–when you're at the point where you're thinking about the health of the service rather than individual nodes. The whole pets vs. cattle ideal seems to be discussed most in configuration management contexts (because it's widely applicable to most architectures) but it translates into big benefits when operating infrastructure for something like a scaleable SOA.
So to give them the benefit of the doubt, they may have made some assumptions about your scale/workload/architecture. Perhaps for them containers occasionally crashing is just a small blip that will be automatically corrected so while they need to be aware of it to monitor for trends they're not generally concerned with them.
The problem is you are trying to run a production environment with a development tool. Manually doing "docker run" is not how you deploy containers. Kubernetes is designed to run containers and addresses the most common issues.
> Making containers work together, for example we use a logging agent, we decided to make that its own container. Then actually getting a way to share a log directory with the agent, was very painful and unclear. (Honestly the single most frustrating thing i recall was asking for advice in irc and more or less being told i am doing it wrong)
kubernetes handles shared directories easily with all containers in a pod. If you need to share across pods you can use persistentVolumeRequests
> Containers would randomly crash under load exiting with error 137 out of memory. Apparently a few of our services would randomly leak memory, but only when running inside a docker container vs ubuntu 14.04. (I never figured this out)
Kubernetes provides replication controllers that will always re-launch or provision the number of desired pods for a service. It also provides health checks just like an aws elb to determine if a pod is healthy. You can also set resource limits (cpu and memory) per pod.
> Random hanging issues, suddenly docker exec is hanging. Being told to revert to an older version, or install a newer version is tiring and makes my environment very inconsistent.
Docker exec should not really be used on a running service in production, all of your provisioning should happen in the dockerfile to build the image
> Trying to debug why our app in the container isn't working is not fun at all.
If your application is 10-factor, you can easily tail the logs of any container at anytime
Yes, Docker is still very young and has a ton of issues like these. I think it will take another 2-3 years before that whole ecosystem has emerged, matured, and is ready for a medium-stability deployment.
Unfortunately for us, such considerations don't stop tech fads. Because containers can allow many more applications to cohabit on the same "hardware", it has business momentum behind it too (lower infrastructure expenditures). "Docker" will be the buzzword until such time as it's actually practical and intelligent to deploy with it.
I'm confused about the need to have a web app that 'runs anywhere' when you know exactly 'where' it's going to run. When you spin up an instance, don't you get to decide what OS and services you're going to use?
I could maybe see the advantage if you had no control over the environment that the app was going to be run in like you might have with a downloaded executable for example. However, if your product is just some kind of web application, I don't see the need for containers.
Being able to leave your current hosting provider quickly is important, but not over engineering abstract hotswapping interfaces is pretty important, too.
What docker promises is truly amazing and is something that I think everyone wants. However, docker itself still has a lot of problems.
In particular, docker's new builtin swarm (with 1.12) has tons of issues. I've experienced __so many__ problems coordinating container startups on a swarm cluster.
From the documentation:
> Swarm can build an image from a Dockerfile just like a
single-host Docker instance can, but the resulting image
will only live on a single node and won’t be distributed
to other nodes.
This is a serious weakness.
You have to use __manual scheduling__ if you want to ensure services are started on the same node. This is problematic if your inventory of nodes is dynamic and frequently changing. Allegedly, kubernetes solves this problem.
Swarm (and the whole ecosystem) is still not as mature as it could be, but I think the end result is going to be very awesome and useful.
That's probobly a bad pattern to use the local registry like that. The goal is to be as stateless as possible, so pushing and pulling from an external registry would fix that.
>Containers would randomly crash under load exiting with error 137 out of memory. Apparently a few of our services would randomly leak memory, but only when running inside a docker container vs ubuntu 14.04. (I never figured this out)
Did you identify that the software definitely did not leak on ubuntu? Or was it that it ran on ubuntu because it consumed swap?
Was it for sure a leak? I have an app that doesn't have back-pressure so when too many requests come in, it fills up memory waiting to push them all through to the slower database. Normally it uses 256m. Spikes make it hit 700mb. If I tell docker -m 300m, then the process gets 300m ram + 300m swap, so when it tries to use 700m it gets killed. I could tell it to use 350m and then it will run, but it would be swapping furiously.
Reminds me of the early days of asterisk and open-vz virtualization. Everyone sane stayed physical but a few of us crazy enough were able to push through and reap the rewards early on.
flabbergasted? Wow this is a new word for me, I have never heard of it. When i first read it in my mind I thought it was a Spanish Football player that used to play for Arsenal, Then Barcelona, and now Chelsea. LoL
Is this word even used anywhere else beyond US? Never heard it used in the UK.
I was born and raised in the midwest part of the U.S. and I've been using "flabbergasted" for all my adult life (20+ years). Of course, I've always had a propensity for peculiar or anachronous verbiage.
Me and a few coworkers decided to try and dockerize some of our serivces, and move our staging ES cluster to docker.
For the most part building our own containers was easy enough, for the various services. The biggest issue we had was with Elasticsearch, since we have 4 types of nodes. So we ended up building 4 specialized containers for each node type.
Then came the issues:
* Making containers work together, for example we use a logging agent, we decided to make that its own container. Then actually getting a way to share a log directory with the agent, was very painful and unclear. (Honestly the single most frustrating thing i recall was asking for advice in irc and more or less being told i am doing it wrong)
* Containers would randomly crash under load exiting with error 137 out of memory. Apparently a few of our services would randomly leak memory, but only when running inside a docker container vs ubuntu 14.04. (I never figured this out)
* Containers would randomly become unreachable, forcing me to kill them and restart them.
* Random hanging issues, suddenly docker exec is hanging. Being told to revert to an older version, or install a newer version is tiring and makes my environment very inconsistent.
* Trying to debug why our app in the container isn't working is not fun at all.
However the single part that killed me was, i was chatting with one of the people who i met at Reinvent, and i mentioned my issues. He acted like it was completely normal for these kinds of issues.
After a solid 2 weeks of random issues and the constant barrage of pagerduty alerts, i just rolled everything back to static EC2 instances for staging, and have ran into 0 issues. I want to try containers again because i want them to work, but i have just had too many issues.