But if the server is allowed to vary X, it can basically act like different servers to each client, and can then when given a token check for which server would have been valid. The solution I got from the other replies is to make sure that the server uses the same X for everyone by verifying it as a client.
It appears that the public side of X is sent as the first part of the handshake, without any login info yet, and can be verified as part of B, thus a varying X would be easy to detect... I think.