Bocoel

Bocoel is a research-born Python library that drastically reduces the cost of evaluating large language models by using Bayesian optimization to select a small, representative subset of a benchmark corpus. Instead of running expensive LLM inference on every example, it encodes the corpus into embeddings, then iteratively picks the most informative samples via Gaussian processes and acquisition functions. The result is an accurate evaluation metric achieved with orders-of-magnitude fewer LLM calls. Targeted at ML researchers and engineers who need to benchmark LLMs quickly without sacrificing fidelity, Bocoel integrates seamlessly with Hugging Face Transformers and Datasets. It supports models like GPT-2, Pythia, and LLaMA, and provides a modular architecture with components for embeddings, storage, indexing, and optimization. What sets Bocoel apart is its pioneering use of Bayesian optimization for evaluation — not just model tuning. It can evaluate a corpus against a model or a model against a corpus in a budget-conscious way. The library also offers advanced representation options like N-sphere and whitening to improve embedding quality. Bocoel is free and open-source under BSD-3 license. However, as of September 2025, the project has been archived and is no longer actively maintained. This makes it best suited for academic exploration or as a reference implementation rather than production deployments.