As a counterpoint I've been studying for leisure by doing exercises in math textbooks, and with advanced undergrad/grad topics my experience with ChatGPT and Gemini is that they fail miserably where they try basic proof techniques that they think work but don't.
Trying to marshal their help is a puzzle in figuring out where they made leaps in logic, pointing those out, to exhaustion until they acknowledge their lack of ability and give up.
It's really reminiscent of talking to overconfident and eager to please youth not realizing that they are tackling something beyond their ability.
It's a mistake to try to get these models to do math. Unless you explicitly have them solve problems in code they aren't going to work, they literally don't have the capability. They are usually ok to explain concepts though.
yeah my approach so far has been: try 3.5 until i am disappointed in the results (hasn't failed me yet), once it does, pay for the upgraded version and see how that goes (totally will to, but 3.5 has really been great so far, and i was blown away when i asked about 1 equation that didn't seem right, and it did realize it's mistake and corrected it...i don't know...just seems more than adequate so far. i realize it's not perfect...i just haven't been let down sufficiently yet).
Trying to marshal their help is a puzzle in figuring out where they made leaps in logic, pointing those out, to exhaustion until they acknowledge their lack of ability and give up.
It's really reminiscent of talking to overconfident and eager to please youth not realizing that they are tackling something beyond their ability.