How many people have reviewed Paramiko? In particular, how about that ecdsa patch[1] to Paramiko that you'll need to be accessing modern Ubuntu or Fedora (and before long, RHEL/CentOS). What about the python-ecdsa[2] (that paramiko's provisional support for modern Fedora and Ubuntu's default configs is based on)? This entry from its README seems pretty frightening:

    This library does not protect against timing attacks. 
    Do not allow attackers to measure how long it takes you 
    to generate a keypair or sign a message. This library
    depends upon a strong source of random numbers. Do not
    use it on a system where os.urandom() is weak.

I'm not saying Paramiko (or its patch sets) are insecure, just pointing out that the same arguments can be made against the libraries and code that Ansible is based on.

[1] - https://github.com/paramiko/paramiko/pull/152

[2] - https://github.com/warner/python-ecdsa