That's dangerous. I remember an incident in which a team accidentally emptied their production experiment config (which controls which features are on: during ramp-up as a whitelist of enabled users and/or a percentage in (0%, 100%); most experiments in the file had long since reached 100%). Suddenly all their experiments reverted to 0% and they were serving live traffic in a totally untested state. Some of these experiments had been on for literally years and now suddenly were off. The combination of experiments had never been tested. There had also been many changes to the code since most of these had last been disabled. As you might expect, this didn't go well.
(That incident could also be part of a long and sad cautionary tale about the dangers of global configuration files, btw.)
The best practice I've seen is for the configuration to specify an end date for each experiment. If it's still listed in that date, a ticket is automatically filed against the defined "owner" of the experiment. The ticket reminds the owner to finish the experiment, remove the conditionals from the code, do a release, wait until they're confident there will be no roll-back to prior to that release, and then remove the experiment from the config.
> For us, almost never.
That's dangerous. I remember an incident in which a team accidentally emptied their production experiment config (which controls which features are on: during ramp-up as a whitelist of enabled users and/or a percentage in (0%, 100%); most experiments in the file had long since reached 100%). Suddenly all their experiments reverted to 0% and they were serving live traffic in a totally untested state. Some of these experiments had been on for literally years and now suddenly were off. The combination of experiments had never been tested. There had also been many changes to the code since most of these had last been disabled. As you might expect, this didn't go well.
(That incident could also be part of a long and sad cautionary tale about the dangers of global configuration files, btw.)
The best practice I've seen is for the configuration to specify an end date for each experiment. If it's still listed in that date, a ticket is automatically filed against the defined "owner" of the experiment. The ticket reminds the owner to finish the experiment, remove the conditionals from the code, do a release, wait until they're confident there will be no roll-back to prior to that release, and then remove the experiment from the config.