- I’d have to do something expensive drastic like run a replication experiment. Given how p-values are defined and how we use them, there’s no other way to tell if you’re sitting on that branch of the multiverse of “the 5% chance it just happened due to random chance”
I've read some research papers on degradation of service experiments, rather than an replication experiment. Could you/have you do one of those, to test a feature here?
Question. I read this statement:
- I’d have to do something expensive drastic like run a replication experiment. Given how p-values are defined and how we use them, there’s no other way to tell if you’re sitting on that branch of the multiverse of “the 5% chance it just happened due to random chance”
I've read some research papers on degradation of service experiments, rather than an replication experiment. Could you/have you do one of those, to test a feature here?
This was outstanding. Recommended it to several product communities.