Yeah, they can't accomplish much as adversaries. Except maybe make you worry that they're out there somewhere being better than you. And you can never prove otherwise.
This is really key: humans think about everything in finite, human lifetimes. We have no backups, archives nothing - when we die knowledge and experience vanishes.
This wouldn't be true for an AI. Death would be optional.
If we're still bound by special relativity (since you said no FTL), there is nothing outside of your light cone.