Skip to content

Welcome to ChemBench🔗

ChemBench is a Python package for building and running benchmarks of large language models and multimodal models such as vision-language models.

ChemBench was developed as a comprehensive benchmarking suite for the performance of LLMs in chemistry (see our paper) but has since then been extended to support multimodal models as well (see our paper). ChemBench is designed to be modular and extensible, allowing users to easily add new datasets, models, and evaluation metrics.

Features🔗

  • Standardized format for benchmark tasks based on the Big Bench format
  • Support for LLMs, VLLMs and other models via a unified interface: For many models, LiteLLM is used as a backend, others can be added easily
  • Scientifically sound parsing and evaluation: ChemBench goes beyond simple MCQ evaluation, supports floating point answers (also in scientific notation). We also support different extraction techniques, including LLM-based extraction.
  • Leaderboard generation: ChemBench generates reports that can easily be used to create HuggingFace leaderboard entries. Check the ChemBench Leaderboard for an example.
  • Extensible: ChemBench is designed to be easily extensible. Prompts, models, extraction, and evaluation methods can be easily added and customized via a pipeline system.
  • Easy to use: ChemBench is designed to be easy to use. We provide a simple you can use to get started testing your models with just a few lines of code.
  • Caching and parallelization: ChemBench can cache results and can parallelize inference to speed up benchmarking.
  • Refusal detection: ChemBench can detect when a model refuses to answer a question, handle it, and report it in the final results.

Table of Contents

ChemBench is open-source software licensed under the MIT license. Its development is led by the Lab for AI for Materials (LamaLab) at the University of Jena and the Helmholtz Institute for Polymers in Energy Applications Jena.