I'm a heavy cc user for writing code, reviewing documentation, brainstorming, updating jira tickets etc etc.
For the past few months, I started experimenting with managing a team using cc. As a team, we got together and decided to experiment with a new way to run the team and now that we're looking at some good results, I wanted to share our learnings here
> The problem with the "everyone" model being pitched here is that it may as well be a synonym for "nobody."
Can't agree with this enough!
Thanks for your inputs. A lot of it resonates with what I've observed which translates to the fact that this is as much a cultural/people problem as much it is a technical problem. If teams took ownership by just building visibility, then it'd be an easier problem to solve.
You bring up a good point of doing canary deployments for solving this problem. I'll check this out.
But its interesting that you say ".. if it is a substantial risk in your domain". Isn't this a problem that most engineering teams are struggling with, especially in last few years? Being part of a few DevOps meetups in my area(Seattle) for a while and having attended a bunch of conferences in last couple of year, I've noticed cost coming up as one of the most recurring discussion topics.
Just curious why cloud costs wont be a risk in any domain.
It is a risk for any company, but the possible harm is variable.
At a prior employer, cloud costs could have doubled or even gone up an order of magnitude and because the margins were so good and the tech costs so low, it wouldn't have mattered and may barely have been noticed. Compute wasn't a substantial business cost in any way, as customers were paying for domain expertise in the product.
At another prior employer, costs scaled with revenue pretty linearly, so while bad, it wouldn't be catastrophic before being noticed as it would also mean increased revenue.
However, for say a company that does video streaming where cloud costs are already enormous, poor cloud usage can cut months off runway. Same with AI, where the money is overwhelmingly being burned on compute.
Cloud waste can happen anywhere, but the harm can range from still a tiny number to destroying the ability to make payroll depending on what you are doing.
I agree that building visibility makes accountability easier. Its relatively trivial to build observability for individual services and we have achieved some version of it.
The problem is when 30 odd microservices (each team owning between 5-10) talking to each other. In pre-production setup, changes in few of these services might not have noticeable impact on cost which will become quite apparent in production. When this happens, we definitely notice an increase the cost and the unit metric. But then we dont know where to start fixing this problem from. Right now, this becomes a war-room situation based on the severity but I dont think this is sustainable.
In comparison, if we take API latency as a metric, the accountability and ownership are clearly defined: If an API slows down, the team that owns it, fixes it. They can work with anyone they need to but its their job to fix it.
Did you face similar concerns/issues? Not sure if this is a problem other engineering teams are struggling with or even considering as a real problem to invest it.
I'm also not sure if theres a "standard" way of doing this which we should be thinking about. So, looking for ideas and thoughts here.
I see what you are describing. The pre-production/staging setup may not bring out cost increases caused by application changes, and by the time it is running in production for a while it has already caused a cost explosion.
We did face similar situations - but we fixed them after the cost went up on prod. I guess this has more to do with how much and how fast an "undetected" cost in pre-prod can explode in production. We used to keep an eye on the prod cost numbers after a deployment, and then tackle each one, because the increase was not that quick.
I'm not sure either about a "standard" way, so I'm just thinking aloud here, and I've not tried this myself:
For application changes, measure the difference in cost in pre-prod, in terms of percentage increase, between the previous deployment and the current one, and use that to estimate the possible prod increase. I suspect this will become messy very fast as the other factors to include would be num requests, CPU/memory usage, and so on.
Already done! This actually works pretty well when two conditions are true
1. cost becomes an engineering-org wide pain point
2. there are a bunch of low hanging fruit (another way to say you're just starting off your cost improvement journey)
Since cost is not really a "deliverable" for non-platform teams, this incentive doesnt go far. Especially after a few iterations of this are done, saving 1K is hard.
We did a short program(similar to a bug bash) for a couple of sprints to dig up all the improvements we can do to reduce to cost. This was early in our cost reduction journey. This did help us get a huge list of what we can do and we picked the most impactful items from this.
Standards enforcement like tagging, TF structure, pipelines etc is currently owned by the platform team. We also have mechanisms to figure out the cost change (approximately) with each Infra PR.
The struggle is to attribute cost changes in application-only changes and to identify them early in the lifecycle. These would be PRs for microservices that add features, fix bugs and handle tech debt.
Esper | Multiple Positions | Seattle, US | Full Time | Visa Sponsorship
We're industry’s first DevOps SaaS platform designed to provide a simple, safe and secure way for engineering and DevOps teams to release applications and manage fleets of smart Android devices in production. Our platform enables developer, mid-market orgs, and enterprise fleets of 100,000+ devices to deliver their software as a service.
Read more about our recent round of funding on Techcrunch: https://techcrunch.com/2021/05/20/esper-raises-30m-series-b-...
We take pride in being an inclusive, transparent, growth focused and customer obsessed team. We encourage learning from our (and others) mistakes, taking bold risks and challenge each other. Read more about our culture from our team: https://www.builtinseattle.com/2021/06/29/seattle-companies-...
We're hiring for following positions in our Seattle office
Our stack: Python/Django, Reacht/TS, Java, Android and Go
If you're passionate about solving challenging technical problems and working in a great startup culture, check out our other open positions at: https://jobs.lever.co/esper-3
Have worked on Mac's, XPS's and HP's in the past but once I got on Thinkpad, never looked back. My current system(for past 5 years now) is a trusty X1 Carbon that has seen some rough times but has never failed me. Thinking of upgrading later this year to a newer version of the same line.
Go for it - I went with an X1C6 as a replacement for my old macbook pro, and I couldn't be happier. Excellent keyboard and no issues running linux. The only setting I've changed from my default ubuntu was to install and use the kde plasma desktop to avoid scaling issues, and it's perfect. You'll love the form factor as well.
I've been using an X1 Yoga (1st gen) for almost 3 years and it's been great as well. I would probably go Carbon next time as I don't use the yoga function that often. But I don't think I could switch to anything else after using this laptop keyboard (and if it breaks I can just replace it myself).
How's the touchpad? What's kept me on Apple is the ability to actually use the MacBook Pro as a portable computer, not just a crappy desktop you can drag between the power and mouse+keyboard stations that you need to get work done without swearing at it constantly or frowning while you watch the battery life indicator plummet.
The keyboard and port situations are both pushing the MacBook far enough out of that "close the lid and go, no worries" sweet spot it used to fall so solidly into that I'm eyeing other options, but I have some very bad memories of both battery life and trackpads on non-Apple machines, and my (limited!) interactions with such more recently haven't made me optimistic.
Just to offer another perspective on the battery front, that only really matters for casual browsing. Anytime real work is done on a machine, you'll be plugging in within a few hours on everything anyway.
If you're doing CPU/IO intensive development work, they all fall down so quickly that I don't think it matters on the differences. It's really a browsing/videos metric, which is relatively unimportant to me when buying a workstation class laptop. I don't think I'd buy any laptop based on battery life as a primary factor, and the overall usability situation for most stuff on the market isn't as grave as it used to be (which your post accurately recalls).
I've owned the XPS15 with 6-core 8th gen Intel (maxing out the scales at 97Wh battery) and have a Thinkpad X1 Extreme. I've had MBPs in the past too.
For the touchpad on Thinkpads, I think they're pretty good if you're on Windows, most are anymore with MS Windows Precision drivers as the Thinkpads use. For me, a major driver for me towards these is the Trackpoint.
Depending on your usecase, my favorite machines on the market today are the X1 Extreme (development), X1 Yoga (consulting/development), Samsung Notebook 9 Pen 13" (home), and Macbook Air (home).
I admit that the touchpad isn't as smooth and gesture driven as the mac but I'm not the user those are really meant for. I spend most of my time on the keyboard writing code and living in the "cloud" which makes using mouse a drag anyway. So all I'm looking for is a powerful machine that doesn't hog resources on useless stuff, with sturdy keyboard(physical keys please, none of that touch bar BS), can support dual/multi boot and works with all the peripherals I need. Thinkpads work better than most without the significant price overhead.
One of my dystopian future scenarios has no Thinkpad's in it :D
I wanted to have a working cluster from scratch which meant not using saltstack and any of kubernetes helper scripts. So I wrote a bash script to do the same here : http://blog.shippable.com/multi-node-kubernetes-cluster
At Shippable(shippable.com) we've been using docker for over an year now for the following use cases:
1. deploying all our internal components like db, message queue, middleware and frontend using containers and using a custom service discovery manager. The containerization has helped us easily deploy components separately, quickly set up dev environments, test production bugs more realistically and obviously, scale up very quickly.
2. running all the builds in custom user containers. This helps us ensure security and data isolation.
We did run into a bunch of issues till docker was "production-ready" but the use case was strong enough for us to go ahead with it
yes, adding that lets you see all remote branches, including PR's. The goal with this tool is to eliminate editing .git/config for each project and make it easier to work with pr's locally. This is just the first step. Definitely planning to add more pr-specific feature.
Do open an issue if there's any particular feature you would like to see.
Thanks :)
https://www.devashish.me/p/why-5x-engineers-dont-make-5x-tea...
Would love to hear thoughts from others who are trying something similar.