We've used LaunchDarkly (LD) for about a year now. We previously used our own Feature Flag library built by our CTO: https://www.npmjs.com/package/dipswitch. LD provided a lot more, and continues to release new features that would've been much more expensive for us to build vs. buy here.
We used to have our own feature flag library built by our CTO as well! We (thredUP) recently switched to Split (http://www.split.io/) and it's been incredibly helpful for our team of 25 engineers. Slow controlled roll-outs, short circuit broken features without a deploy through their dashboard, and we actually use them for A/B tests as well since they have an impression data pipeline and A/B tests are so similar to feature flagging. Definitely one of the smartest "buys" we've made in our last 4-5 "buy vs. build" discussions.
LD provided a lot more ... that would have been more expensive for us to build vs buy
we implemented most of LD in a 24-hour quarterly hackathon we hold
What's the breakdown of 'most' vs the remaining? 80/20? Alternatively, if 'most' was implemented in 24 hours, how long would it take to implement the rest?
From these two statements, I get the impression that either
Initially, I was very confused about why this is a good idea. Before the web, feature "productization" was the always part of design of the system and of each module: in other words, each feature should be disabled based on some criteria (i.e., like not ready for release, insufficient licenses), measured, etc.
But then after a few minutes... I kinda got it. I think this is a great tool for marketing, biz, and product management in organizations where the overall organization structure is kinda messed up and they cannot convince development to do things right. Which is 90% of companies. Very similar to hiring external management consulting services: since nobody in the company is capable to organize this.
>> overall organization structure is kinda messed up and they cannot convince development to do things right
That has nothing to do with why feature flags exist. No amount of planning or perfectly written software will give you insight into how your customers will react to your feature. I'd rather release something to 5% of my customers, see how it performs in the wild, make adjustments accordingly, and only then release it 100% along with any marketing initiatives that support it.
If teams use feature flags to release untested/unprepared code, that's their prerogative and is a perfectly acceptable use case if the business is amenable to those risks.
I've lost count over the years of how many features I've seen significantly modified after user feedback, or altogether cancelled because it was determined the long-term maintenance cost wouldn't be worth it vs. adoption/traction.
The interesting thing is how a lot of the focus in these comments has been around optimizing and A/B testing but feature flags have value in being able to continuously integrate and deploy without disrupting your production environment. For teams that want to merge and release regularly and have launches be easy I think LaunchDarkly is a big win.
As a product owner who uses LaunchDarkly, my preference is to always abstract complexity if its not core to my product. I am always an advocate for tools and services that allow me to get to market quicker, while reducing long-term overhead. An internal service like LD would need to be built and also maintained. I think many people overestimate the complexity of services like this; it seems simple but execution is always more difficult (scale, bugs, features, etc). Cheers to LD for providing this service so my team can concentrate on our core.
We use LD, because (1) we don't have time to make a pretty UI with user management and audit logs, and (2) we like the reliability story of running all of the heavy lifting through fastly's CDN.
They are a little more pricey than we'd prefer, but we're happy.
I've used it. I wasn't super impressed with the features around grouping accounts so that you can match a group with a feature toggle. As I recall, it was mostly regexes, rather then a long running group system. I'm guessing a big infusion of money will help with this.
They should incorporate ML options too. They have multivariate testing + goal/conversion measurement, they should have multi-armed bandit ML methods as an option (epsilon-greedy, bayesian bandit etc).
As a matter of fact, we have experimented with multi-armed bandit 'auto-pilot' to roll out a new feature while watching A/B test results. I'd like to see this make it to production as well; it was one of those things that sounded like a fun idea, but we weren't sure if actual customers would use it :).
nb: I work for LaunchDarkly, if it wasn't obvious from my comment.
Feature flags certainly aren't new -- anyone who has been into SaaS and staged rollouts of production changes likely has built their own or leveraged libraries that implement feature flags. However, the downside of homegrown or library-based models soon becomes visible once you have a more focused organization where each group focuses on what it does best (product vs engineering vs customer success). As Joel Spolsky asserts in defense of NIH syndrome (https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-...), do build what is critical to your business. For a business that revolves around user modeling/behavior/marketing, it would make some sense to build your own. For example, Mixpanel or Totango could probably use their own internal segmentation engines to target users for new features, thereby 'eating their own dogfood'. I work at Upserve, where we provide software to help restaurants and bars run their businesses better. We are obviously not directly in the business of producing feature flags for our customers, but we leverage the hell out of them.
Here are a few key benefits I've seen as an avid user of LaunchDarkly, that are end up costing a lot to build on your own:
- ability to have people outside of engineering control who gets access to what features. Specifically, product management should have control over what features are available, and they probably don't want to go into a dev console to make the changes. Sure, you could build a UI on top of home grown FFs, but why spend valuable time on this?
- rules-based user targeting -- rather than embedding ugly logic into your code about what type of user should be served a feature, you can handle most of that within LaunchDarkly's UI. For example, we have some restaurant groups that have 100's of locations. Rather than setting values at the location or <gasp> user-level, I can simply set a flag at the restaurant group level and that automatically propagates to the user level. Sure, you could build this hierarchy into your homegrown tool, but I guarantee doing that in a very flexible manner won't be a slam dunk, and you'll soon find that you will want to change it and have to invest more time on something that may not be core to your business.
- custom rollout -- you can roll out by % of users or user properties. For instance I can roll out to 10% of users that are in Rhode Island, US. Did you really add this to your homegrown FF service? I hope you didn't tell your VCs that if so...
- centralized feature flag service -- we leverage across dozens of components, and use feature flags centrally on our cloud servers and on 1000's of agents installed inside our customers' locations (and on hardware that we don't own). Since LaunchDarkly uses server-sent events (SSE) this introduces no additional latency in making network calls to a central server. Changes to users and flags are sent streaming, seamlessly, behind the scenes, with minimal network impact. This key infrastructure decision that the team at LaunchDarkly made has given us confidence of using feature flags wherever we could feasibly imagine them without having any adverse impact to us or our customers. Oh, did you remember to build that into your homegrown FF solution?
- multivariate flags -- LaunchDarkly supports multivariate flags. For example, we throttle data transport for some of our remote agents. We can control this centrally so that certain agents only send back certain amounts of data (e.g. 1MB, 5MB, 10MB per batch of data sent). This actually extends the concept of FeatureFlags -- there are discrete variations within a single 'flag'. Don't forget to build that into your homegrown solution too.
- feature flag chaining -- you can specify a prerequisite for a feature flag. For example, for a rebranding of an entire website, you could have individual feature flags for each page, and then have a 'big bang' flag that does the entire thing at once. This allows you to test and manage 'micro features' but then turn on the 'major feature' easily.
- they have thought a lot about the technical debt that feature flags can introduce. How often have you added a feature flag only to not use it again or forget to remove it? It can make your code base ugly in a hurry. LaunchDarkly has analytics that help you identify usage paths as well as 'flag the flag' so you can be more conscious about what is a temporary vs permanent flag. This might not sound like much, but it prevents questions later about 'can we remove this flag' without causing impact. You've future proofed your homegrown solution to grow into this too, no doubt.
- organizational visibility and integrations with third party services -- LaunchDarkly has many integrations already built into it. For example, we use it to post to our #product channel in slack, so we can see when a new feature becomes available and when someone changes an existing flag/adds users to it. This has made communication about features much less time intensive because anyone can see the current state of a feature. You didn't really build a Slack integration or create a bot just for your homegrown feature flag service, did you?
If you think you'll eventually be in the place where what I've spoken about would be helpful, I would strongly suggest you consider a service like LaunchDarkly early in your development so that you can focus your energy on the right things and take advantage of the hard work that LD is putting into their service.
We have mobile (iOS/Android) SDKs, so you can use LD for mobile apps as well, but the backend SDKs are primarily meant for SaaS-style server processes (where the SDK token can be guarded from users, and the process serves many different users, some of whom will get different flag values).
We have gotten some requests for desktop SDK support, which would work more like the mobile or JS clients work, since there is a single user at a time, and the key used to get flag values must not leak any proprietary business info or details about other users. This is on our roadmap, so the funding will help here :)
It's not on our immediate roadmap, but we have had some demand for it. We think we might go about it by building an Erlang SDK. Hopefully a light wrapper on that would work well from the Elixir side.
For the sake of high quality idiomatic code. I'd recommend creating the Erlang library and a separate Elixir one. But to be honest, I'm just so pleased to hear your even thinking if either of them :-D
Not to take away from the fact that this is a very useful tool, but every time I have wanted this, I have built it myself. My general strategy is to store flags in the user's data and for A/B testing, randomly assign A% of users to have a certain flag set. Then somewhere in the admin dashboard I'll drop a set of switches/buttons for activating flags and running A/B testing scripts. Collecting test results is simply modifying the database query to only include users that had the flag set.
An external service for boolean flags seems a bit overkill to me.
Edit: I read the first comment, looked at the features, and read the case studies: I am guilty of vastly oversimplifying what this does. The value proposition is still not right for my particular business, but at my previous job at Big Company, we had a custom rollout dashboard that didn't have most of these features (targeting groups, percentage rollout), and it still saved us many hours of development time.
I saw this exact same sentiment when Optimizely was announced. Now, I cringe when people say they implemented their own A/B system (similar to what you wrote above). If you're running a business you probably need to spend your development efforts elsewhere.
In general, I'd say most tech businesses are aware of Optimizely and the value proposition is quite compelling. I certainly wouldn't try to save money on our Optimizely spend by building any part of their system myself. But if I could save the company $700/month (the price for 50k MAU, which doesn't even cover our requirements) by implementing something as simple as feature flags, I'd add it to our development roadmap. For smaller companies, ~$10k a year is not trivial.
However, I acknowledge that I may be vastly oversimplifying, and am not knowledgeable enough about LaunchDarkly to speak so brazenly about how I could do it myself. My main point is that as a small web development company, we have this feature in our builds already, and it only cost us a couple hours of development time to implement it.
There are a few open-source feature flag libraries out there, but what makes our service better (IMO) is mostly the cross-platform consistency (so a user will get the same value of a flag across different services, mobile, etc) and the ease with which non-technical people (eg product managers, etc) can adjust the targeting.
Of course, you should evaluate the options for your situation. For many use cases, an open source library will solve the problem just fine. For larger projects/teams, something like LaunchDarkly will be a better fit.
The biggest challenge in implementing this is handling scale. To be able to do feature flagging for the largest web products you need to be able to handle the scale of those products.
Any of us could build this ourselves in a few minutes for a local app, but to scale up to millions of users is very tough.
I'm working on that. Currently we are using LD but it's had some issues, enough that their CEO sent email and refunded some of our money.
As far as Scale is concerned we routinely see request timeout to LD and then suddenly a user loses a FF they previously had. Our cached value for the FF expires and then LD fails and then my customer is mad
Can you implement analytics on top of those flags, plus a nice UI for nontechnical employees to toggle them faster? If you don't need those, then you probably aren't in their target audience.
Disclaimer: no relation, just playing devil's advocate.
Common HN comments: "I could build/setup/run this myself"
It's a SAAS tool and you could build it? Well hot damn, you must be a programmer!
I'm stunned by how many refuse to learn from Gmail (mailserver), Dropbox (intra-company sharing), Slack (IRC), AWS (servers), the list goes on.
What a severe waste of your potential. Someone else has built a better version than you will ever build because they are focused 100% on this one specific problem. Even better - the cost of building, serving, and maintaining it are amortized across a larger user base with a small profit tacked on top.
If you can't use a SAAS because of privacy/security concerns, totally get it, that's how it goes some times. If you refuse to use one because of not-built-here syndrome, you're wasting company resources and making yourself less valuable along the way. Companies pay for solutions, not this-was-built-here tools.
For me, I think it's not so much NIH syndrome but resentment at pricing among analytics products that seems calibrated to extract as much value from the customer as possible, which they can do because there is so little direct competition and because the ROI is usually still positive.
It's a perfectly rational pricing strategy, of course, but it can sometimes feel like one of those economics experiments: "A colleague receives 100 dollars and is asked to split it between him and yourself. If you disagree with the split, neither of you gets anything. Your colleague proposes to give you 10 dollars and keep 90 himself. Do you choose to accept the 10 dollars, or would you prefer both of you get nothing?"
Also, your examples are about ancillary utilities, but let's say you're a data scientist and you decide to build your own A/B testing system instead of using one of the existing services out there. If you add up all the dev time and maintenance time the homegrown system will probably be way more expensive, but on the other hand building it might teach you a bunch of intangible skills that make you a better experimenter and that could be worth the expense if the company expects employees to stick around for a while.
Having said that, personally I still prefer to just pay for something. But I'm not at all stunned when other people do decide to roll their own tools.
Let's say you want to open a restaurant. Now sure, you could go out, start a farm, open a steel mill and a mine to make all your knives and cutlery yourself, cut down timber and construct the entire building yourself, make your own furniture, etc.
Really, you could do all of those things. And I think that's the state of the software world, about 10 years ago. Hell, maybe even today, compared to 10 years from now.
These days, we're rapidly moving to a world where probably 95% of what a typical company needs -- code hosting, scaled relational storage, execution grids a la Docker on ECS, load balancers, A/B testing, SMS infrastructure, feature flags, etc. can be bought off the shelf, for a fraction of what it would cost to do yourself, and at much higher quality than a single individual could do.
It's a natural outgrowth of the increase in developer salaries, in fact, the two go hand-in-hand: labor productivity is up a lot because of these tools, and the incentive to buy cheap off-the-shelf stuff is higher than ever, vs. paying for expensive, scarce, flighty developer labor.
I do wonder what this means about starting companies, though. I think software will increasingly move toward an industry where, like owning a Burger King, you have to raise capital (rather than just doing everything yourself) because so much of what you need has to be bought, and peoples' standards are so high. And yet, it is more optimal to have to raise capital, because anyone can get money; not everyone has the skills/talent/know-how to build complicated software. Buying is less risky than building.
No longer is it enough to home-roll a webserver: now you need to launch from day 1 with a native mobile app on multiple platforms, everything has to work on any size of display, it all has to be bug-free, high-design, autocomplete-text-boxed, perfection, from day 1.
This whole "it's getting so cheap to ship a v1" is malarkey, IMO.
It probably isn't targeted at those users. If you're a small shop and just need feature flags, this is probably overkill.
Recognize that there's a cost to pulling in 3rd party libraries, much less 3rd party services, into your project. It's not just the cost to integrate, but of figuring out failure modes. The offering can evolve to no longer serve your use case. They could deprecate the API, causing you to have to update current, and past, projects. Google can buy them and shut them down.
If you take this approach for everything someone has a SAAS for, you could spend more time responding to their changes than evolving your own product. Been there, done that - it isn't fun.
I think this may be generalizing a bit too much. Certainly I would use Twilio instead of building my own telephony system, or use AWS instead of setting up my own servers. But in this case I think the reason many developers here (myself included) have an episode of build-it-myself syndrome is the many millions in funding and exorbitant pricing for something that seems (but may not necessarily be) trivial to implement.
Obviously there is the benefit of having a team of people who are constantly making the system better and more robust. I think with ML there would be a serious value proposition here, e.g. allow a learning system to automatically roll out a new feature for you, and reverse the rollout if things appear to be going poorly. Even the worst case of built-it-myself syndrome would not engineer a custom feature flag system that far.
100% agree. I can't recount how many times I've worked with people who wanted to build their own tools or only rely on their own information. I once worked with a dev who refused to talk to people better at systems engineering when we started reaching scaling issues. His claim - this is too specialized. Having gotten into infrastructure engineering recently and spent time working on microservices based architectures I realize that this was more of a "I built a shitty system and I don't want to show it to people".
I think some of this reaction is because then someone from the non-engineering side of the business is like "let's use this" even though it's over-engineered for your use case (i.e. a one line if statement branching on a query parameters or something). Now I realize the platform they're providing is way more than that, and for advanced cases it would be totally valid (i.e. we only want to launch this for 10% of logged out North American users after Jan 1st).
For me this is the Shark-Jumping moment in these "feature-as-service" startups. In the production systems I've worked on this is literally 25 lines of code.
We’ve been so pleased w/ LD that we agreed to do a case study with them if you want to take a gander: https://launchdarkly.com/casestudies/lanetix.html
Also, we implemented most of LD in a 24-hour quarterly hackathon we hold at my company: https://engineering.lanetix.com/2016-02-22/how-lanetix-does-...
Hope this helps!