This was true for me. I absolutely hated math in middle and high school. I never studied for it and got mediocre grades. I had a few teachers who really had no enthusiasm or drive to make it interesting. Fortunately, I started taking some community college classes my last couple of years, where I really fell in love with linear algebra. I wound up being a math major in undergrad and a geophysics PhD.