pykoi is an open-source python library for ML scientists. pykoi makes it easier to collect data for LLMs, to use that data for finetuning, and to compare models to each other (e.g. your model pre- and post- finetuning, or your model vs openai vs claude). The library comes from pain points we experienced in LLM development:

1. Collecting feedback data from users isn't as easy as it could be. (The current process usually involves sharing excel files of annotated responses back-and-forth, offering no insight into how users actually engage with your models).

2. RLHF remains complicated to carry out. By complicated, we mean requires a lot of steps, hundreds of configs, lengthy setups, etc.

3. Comparing models to each other as they're used (that is, independent from academic metrics) is full of friction. The current approach: spin up a model, ask questions, write them down. Repeat for other models then compare.

At a high-level, we think that the active learning process should be closed-loop: data collection, fine tuning, and inference all feed from the same system. This library is our first step in that direction.

The project is still very early but we hope that some if it is useful. Note, we're fully open-source, and actively adding features!

Website: https://www.cambioml.com/pykoi GitHub: https://github.com/CambioML/pykoi

We would love your feedback!