I disagree with him on bots, though. CCP has a highly evolved bot system which is tied into their build system, so that unit tests can include player behavior. This isn't directly relevant to load testing (although CCP does that with bots as well), but it certainly demonstrates why bots are worth the effort.
And if you've got a full-fledged QA department, bot writing is a good QA engineer task. This is also a decent way to give ambitious QA guys a coding task that doesn't have the potential to affect customer-facing game play.
Without violating NDAs I can say I've worked at MMO companies that found bots invaluable for load testing. Replays will definitely catch things that bots won't, but if you're thinking about how many players you can fit in a zone, etc., etc. bots are great.
Pat's speaking for himself here, of course, as evidenced by the fact that bots were crucial for load testing GW2 (they ran on Amazon EC2, even, so that they could scale dramatically high.)
I disagree with him on bots, though. CCP has a highly evolved bot system which is tied into their build system, so that unit tests can include player behavior. This isn't directly relevant to load testing (although CCP does that with bots as well), but it certainly demonstrates why bots are worth the effort.
And if you've got a full-fledged QA department, bot writing is a good QA engineer task. This is also a decent way to give ambitious QA guys a coding task that doesn't have the potential to affect customer-facing game play.
Without violating NDAs I can say I've worked at MMO companies that found bots invaluable for load testing. Replays will definitely catch things that bots won't, but if you're thinking about how many players you can fit in a zone, etc., etc. bots are great.