When I first tried this, years ago, I found it too awkward. I eventually settled on using my thumbs for five, and my fingers for single digits (like a Japanese abacus).

This feels reasonably comfortable to me, is easy to use, and covers a sufficient range of numbers for my needs.