I've been doing this for a few years, along with a reasonable minimum font size. I find it really tough to go back to a smorgasbord of fonts and sizes, with some of them often unreadable. Controlling the fonts via Firefox is awesome.

The biggest downside to this is where sites (e.g. github) try using private-range unicode codepoints for graphical icons, not realizing that users often have other fonts set. I really discourage site authors for hacks and stick to using graphics for icons.