Wanting to collect usage information and errors isn't evil-by-default. It's incomparably useful for troubleshooting and improving. Absolutely nothing works better, it's the best by a ridiculously large margin.
But yeah, terminals are very sensitive environments, opt-in should be a default even at launch.
> Absolutely nothing works better, it's the best by a ridiculously large margin.
Is this really the case? It seems that to find mistakes in software for various interaction patterns, truly exhaustive automated tests would likely work far better by various measures (coverage, reliability, reproducibility, reusability etc.) and at the same time do not have the extreme downside of privacy invasion. For example, see a section from the Age of Empires Post Mortem https://www.gamedeveloper.com/pc/the-game-developer-archives... :
"8. We didn’t take enough advantage of automated testing. In the final weeks of development, we set up the game to automatically play up to eight computers against each other. Additionally, a second computer containing the development platform and debugger could monitor each computer that took part. These games, while randomly generated, were logged so that if anything happened, we could reproduce the exact game over and over until we isolated the problem. The games themselves were allowed to run at an accelerated speed and were left running overnight. This was a great success and helped us in isolating very hard to reproduce problems. Our failure was in not doing this earlier in development; it could have saved us a great deal of time and effort. All of our future production plans now include automated testing from Day One."
No data on this but instinctively it seems, given alternatives, most people abandon some buggy software rather than patiently reporting problems and waiting for it to get better.
Yeah. User reporting has a very obvious and very strong survivorship bias. Plus the people who take the time to send in a report are a rather small niche, so you have pretty strong bias even if you exclude people who leave.
Always-on metrics are massively higher quality data. They don't collect the same kind of data in many cases, but they can reveal a lot of things that never get reported. They also don't suffer from the well-established pattern of people not accurately reporting their own behavior when asked / polled (stronger when asking about future behavior, but it applies in all cases).
In production, agreed. In beta, I’ll accept it. I feel that the term beta gets abused a lot, but in what I believe is it’s proper meaning, there are a lot of inherent factors both parties are agreeing to; increased risk of error and data loss, and debugging flags that generate more data for the singular purpose of improving the product. That’s exactly what should be in the privacy policy and explicitly stated upon install. Anything short of that puts me firmly in the hell-no category with you.
With you here on this. Telemetry is a really important concern and I get why people don't like it, but fundamentally the expectations on a beta product surely have to be different in that. The thing is still in development.
T
I'm with you on not sending data, but have you ever read user reports? IF you get any (most won't report) they likely won't have enough information to reproduce or fix.
It is evil by default. Paying beta testers, or giving them a free, opt-in version with telemetry is the ethical route. Being ridiculously, over the top clear about exactly what you snarf off the end user is the ethical route.
You are not entitled to access my machine, and that shouldn't be casually dismissed with "don't worry, we're not doing anything bad." You're creating potential vulnerabilities, and by implementing identifiable patterns, reducing the security of your users.
You shouldn't spy on people, and when you do, it's wrong. Remotely inspecting people's behavior is spying.
Your software doesn't need to phone home. It doesn't need automatic updates. You don't need to spy on people to develop good software. That's toxic nonsense.
Telemetry is just the cheapest and most convenient option for business owners, and not necessarily the best option when it comes to improving customer value and experiences.
Case studies, focus groups, surveys and interviews are great ways to determine usage patterns and problems with products and services. Of course, you'd need to pay users to participate in them, and then you need to pay expensive employees to conduct, collect and analyze the results. Spying is cheaper than doing any of that.
I agree both that telemetry is useful and that there's not necessarily a place for it in the tool I use to manage my workstation and hundreds of servers. Perhaps I'd opt in to a middle ground, that is collect telemetry locally into a support file I can review, evaluate, and potentially redact before submission.
> Absolutely nothing works better, it's the best by a ridiculously large margin.
Then why is the telemetry-encrusted modern Windows a usability fail, even compared to past versions of Windows which relied on extensive in-house user testing?
But yeah, terminals are very sensitive environments, opt-in should be a default even at launch.