I've always wondered how stop tokens fit in here. Does the LLM generate a probability for "stop" in addition to every other token in the space? Or is stopping handled heuristically by the outer loop that generates the output tokens sequentially?
The API docs talk about letting you specify your own stop token (like "<!-->") but I don't think "token" is meant in the same sense here.
Yes, the model has something like an EOF token which it emits for the output to end. It is part of the probability distribution that the model predicts.
The API docs talk about letting you specify your own stop token (like "<!-->") but I don't think "token" is meant in the same sense here.