Bocoel

Bocoel (BoCoEL) is a Python library that uses Bayesian optimization to drastically reduce the cost of evaluating large language models. Instead of running expensive LLM inference on every example in a benchmark, Bocoel encodes the corpus into reusable embeddings, then applies Gaussian processes and acquisition functions to iteratively select the most informative samples. This yields accurate evaluation metrics with orders-of-magnitude fewer LLM calls. It is designed for ML researchers and engineers who need to benchmark LLMs on large datasets without blowing their compute budget. Bocoel follows a modular pipeline: embed corpus entries (much faster than LLM inference), use Bayesian optimization to select which queries to evaluate, retrieve relevant entries from the embedding index, and run the LLM on just that small subset. It supports both corpus-to-model and model-to-corpus evaluation, so you can assess how well a model covers a dataset or how well a dataset probes a model. The library integrates with Hugging Face Transformers and Datasets, and works with GPT-2, Pythia, and LLaMA model families out of the box. It also offers advanced embedding enhancements like N-sphere representation and whitening of the latent space to improve evaluation quality. All of this is packaged in a modular design covering embedder, index, optimizer, and manager components, with API reference and getting-started examples included. A key differentiator is the application of Bayesian optimization to evaluation itself—not just model tuning—which is a novel angle in the NLP benchmarking space. However, the project was archived on September 14, 2025, with the creator moving on to other work. This means it's best treated as a reference implementation for academic exploration or forking, not as a dependency for production systems. It's free, open-source under BSD-3, and installable via pip.