It's completely true! I'm probably pretty good at programming because my father had connections to a gov't lab and got me an internship. I also really wanted to get good at math... had my parents or myself known that most math professors/grad students will talk to anyone about anything within their subject for as long as they'll listen, they might've driven me to the local college and brought me to some prof's office. Nobody I knew had any idea of that though! So yes it's privilege to have all these connections... but even knowing you can just walk in and ask is a privilege. Privilege is really subtle.
Alternatively, my family was fairly fundamentally religious. I was taught at an early age to distrust science and math. I eventually fell in love with programming, but that wasn't until my 20s. It's taken a lot of hard work to retrain my childhood teachings.