I have the same concern. I find it's even more troublesome with the flat/simplistic design aesthetic: it just takes way longer for my brain to differentiate one light-gray-on-white abstract symbol from another than it does to pattern match word shapes.
I've even turned on whatever Gmail lab puts the words back into the buttons.
My issue is that no-one should really have to 'remember' them.
When I first started using github it was straightforward as I only had to learn what the words meant (i.e 'fork', pull-request, etc). Now a new user is supposed to learn those concepts and their associated symbols. The cognitive overhead just increased.
I've even turned on whatever Gmail lab puts the words back into the buttons.
That said, Github looks to be mitigating this issue by having full text on the root screen for the repo. Compare the sidebar here: https://f.cloud.github.com/assets/1354/660756/cc8cad9c-d714-... and here: https://f.cloud.github.com/assets/1354/660769/fe4a1a0e-d714-...
Maybe that will train my brain with positional data a little better than Gmail managed to.