We do something similar in house where I work. Is it hard to onboard new customers? Since they make a special container for you they basically adopt your build system, which may be hard for them. Does this go beyond mutation testing?
With most of our customers, they don't actually have to change their build system at all. We take their normal CI products, plus a small amount of special configuration, and run them. This is actually a good thing because we're testing something very similar to what runs in production.
The kinds of state space exploration we do are a lot more general than mutation testing. Our current product does exploration by (1) varying the space of faults, packet delivery times, thread schedules, etc., and (2) driving a customer-provided pseudo-randomized workload. We have plans to make both these mechanisms much more expressive, powerful, and configurable; and we have longer-term plans to add entirely new kinds of testing to the same platform.
Disclosure: I'm one of the co-founders of Antithesis.
Thanks! Where does the "want" come from to compare your "got"s to? If the customer code handles errors gracefully, can you still surface undesirable behavior?
We provide a bunch of different ways for you to express your test properties. (1) A lot of stuff we can detect automatically, like crashes, OOMs, etc. (2) You should turn on assertions in all the code you send to us. (3) We can write generalized temporal regexes against your log messages or other output. (4) We offer SDKs that allow you to declare test properties inline in your code. (5) We can integrate with off-the-shelf sanitizers like ASAN, the Go race detector, etc.
