I've heard that in a couple of places, it's difficult to track down a specific source. It looks like that particular fact isn't on Wikipedia anymore, although it does say he learned English from a young age.
Looks like the truth may be somewhat less strict than what I had remembered. Nonetheless, not only was he an Anglophone, he was also an Anglophile, something most of those sources reinforce.
Source for being raised bilingual "he was not even aware that English and Spanish were separate languages until later in his childhood" (cache) http://cc.bingj.com/cache.aspx?q=borges+first+language&d=498...
Source for English as first langue: http://kirjasto.sci.fi/jlborges.htm
"English practically as a first language" http://www.thedailybeast.com/articles/2014/10/24/borges-had-...
Looks like the truth may be somewhat less strict than what I had remembered. Nonetheless, not only was he an Anglophone, he was also an Anglophile, something most of those sources reinforce.