Then I'm afraid it's not possible. The reason the input switch takes so long is because of the HDID negotiation.
A video mixer acts as the source and sink for the output and the inputs respectively, where a HDMI switch will just physically disconnect and connect the output port to a different input port, meaning the HDID negotiation has to be re-done every time.
It still seems the right hardware, upstream or downstream depending on which direction you're viewing things from, could keep the connection alive with the HDID pre-negotiated. Like, assume the hardware doesn't change between switching. If you're the TV's firmware and have the CPU power to, why renegotiate when switching inputs. Solve for the default case and have it standing by and running hot. it's not like the panel is going to change in the interim.