Address book access is a permission just like anything else (internet access, for example), so it can fly under the radar. Android has been trying to make things like "send text messages" stand out a tad more than "write to local storage", but they may need to go even further.
This is my personal favorite app permission that you can request: http://developer.android.com/reference/android/Manifest.perm...