Hacker News new | past | comments | ask | show | jobs | submit login

> You may think of these symbols as "Latin" because they're how people writing in Latin script happen to write mathematical expressions

No need for scare-quotes, Latin script is a proper noun and a technical term with precise meaning wrt text encoding - not "what I think."

> the exact same symbols are also used by Mandarin speakers, as well as in numerous other scripts. Writing math in Chinese

Which unicode code points do the Mandarin speakers and "numerous other scripts" use to write "12 + 89"? Could it be the very same code points as Latin script, which then are tokenized to the same vectors that the LLMs learn to associate more with English text rather than CJK in the latent space?

> i.e. easily distinguishable from their English descendants.

You're making broad assumptions about the tokenization design here that do not apply universally.






Precisely because the exact same codepoints are used for digits and mathematical symbols, there's nothing script-specific about them and their linguistic association is determined by the training data mixture. A model trained predominantly on text scraped from Chinese websites would learn to associate them more with Mandarin than English in the latent space, since that would be the context where they most often appear.



Consider applying for YC's Summer 2025 batch! Applications are open till May 13

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: