All I can figure is they're trying really, really hard to keep this very parallel on their side and also avoiding having to coordinate between nodes. It can't possibly be the reading of the robots.txt that's hard, so I think that statement has more to do with applying those policies to all the nodes—they must regard coordination between nodes to ensure the system as a whole isn't exceeding e.g. request rate maximums, to be "a fair bit of extra work".

Judging just from the linked post, the issue on which this was discussed, and this thread, it's feeling a lot like this proxy was some kind of proof-of-concept that escaped its cage and got elevated to production.