I turn off password access to production machines, so nobody accesses them without a known key anyway. Ansible has its own key installed on the servers and its account is allowed to sudo without a password.
So if I can break into your ci machine (or just get jenkins to run random commands on your prod server, which is probably easier), I then have sudo access to your prod server?
Using ansible from a local machine is fine, because you can make your devs type in passwords and etc, but I can't think of a secure way to do it with continuous integration.