It could be that pgvector isn't good enough for serious use. It would be great to see some benchmarks. The one you site sounds like there's enough reason to try something more specialized.

Postgres is quite good as a JSON store btw.