The Yubikey would only work if there was an absolute guarantee that a hacked kernel/drivers would not be able to access the memory.
The way I can see it working is if there is a private key on the device, inaccessible to the host hardware, and the crypto stuff is done on the device - so the Yubikey was effectively the client. Auth service sends challenge to the browser which sends it to the driver which asks the yubikey wtf, the yubikey responds to the challenge, and the response is sent to the browser and back to the host.
But this would all fall down if there was even the slightest chink and your host hardware could be modified to access/save the keys on all of the Yubikeys when they are plugged in.
The way I can see it working is if there is a private key on the device, inaccessible to the host hardware, and the crypto stuff is done on the device - so the Yubikey was effectively the client. Auth service sends challenge to the browser which sends it to the driver which asks the yubikey wtf, the yubikey responds to the challenge, and the response is sent to the browser and back to the host.
But this would all fall down if there was even the slightest chink and your host hardware could be modified to access/save the keys on all of the Yubikeys when they are plugged in.