SSNs are identifiers and when knowledge of SSN gets you some privileges, it's problem with whatever gives you these privileges. Card numbers are on the other hand essentially explicitly designed to be used as authentication data and at the same time printed in big letters on the card and known by all parties involved in transaction.
