> Or are you worried that someone at Google knows your voice?

Google already knows who you are, now they know a bit more about you, including what you talk about, and of course your voice which probably can be used to locate you with all those "smart" speakers around.