Huh? Either I do not understand what you mean, or you do not understand probability (sorry).
My understanding is that you wanted to test whether a certain feature improves user satisfaction or not. Assuming that users have a 0.05=1/20 probability of liking each feature, by testing 20 features you can get at least one successful feature (because 0.05x20=1).
This is wrong in two ways.
First, the p-value is the probability of observing an effect at least as large as what you measured, assuming that there is actually no effect, i.e., the null hypothesis is true. However, and this is crucial, the p-value does not tell you the probability that the null hypothesis is false (or true)! Again, p-values are unrelated to the truth-ness of the null hypothesis (this is a very common misunderstanding). In fact, if you test 20 hypotheses at a confidence of 5%, you have a probability of (at most) 64% of incorrectly thinking that a feature is useful, while it is not.
Second, setting aside hypothesis testing and p-values, if each feature has a 5% probability of being liked and you test 20 features, you only have 64% probability of finding an useful feature. This is because, assuming that the success probabilities of those features are independent and identically distributed, the number of successful features has a binomial distribution [1]. If you wanted to be 95% confident of finding at least one useful feature, you would need to test at least 59 different features each month. What you computed (0.05x20=1) is the expected (average) number of useful features per month over the course of many months.
My understanding is that you wanted to test whether a certain feature improves user satisfaction or not. Assuming that users have a 0.05=1/20 probability of liking each feature, by testing 20 features you can get at least one successful feature (because 0.05x20=1).
This is wrong in two ways.
First, the p-value is the probability of observing an effect at least as large as what you measured, assuming that there is actually no effect, i.e., the null hypothesis is true. However, and this is crucial, the p-value does not tell you the probability that the null hypothesis is false (or true)! Again, p-values are unrelated to the truth-ness of the null hypothesis (this is a very common misunderstanding). In fact, if you test 20 hypotheses at a confidence of 5%, you have a probability of (at most) 64% of incorrectly thinking that a feature is useful, while it is not.
Second, setting aside hypothesis testing and p-values, if each feature has a 5% probability of being liked and you test 20 features, you only have 64% probability of finding an useful feature. This is because, assuming that the success probabilities of those features are independent and identically distributed, the number of successful features has a binomial distribution [1]. If you wanted to be 95% confident of finding at least one useful feature, you would need to test at least 59 different features each month. What you computed (0.05x20=1) is the expected (average) number of useful features per month over the course of many months.
[1] https://en.wikipedia.org/wiki/Binomial_distribution