I 100% agree. But with ssh (ideally) the passphrase never leaves the computer you are physically touching.

With passwords you are sending it to remote computers where it could be compromised. Thus the standard practice of forcing all users to change their passwords when a server is compromised.