There's also the problem that many jurisdictions have 2+ elections each year. A federal poll tax would create quite a bit of room for local poll taxes that would be that much more abusive.