Metadata-Version: 2.1
Name: evalplus
Version: 0.1.1
Summary: "EvalPlus for rigourous evaluation of LLM-synthesized code"
Home-page: https://github.com/evalplus/evalplus
License: Apache-2.0
Platform: any
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: Apache Software License
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE

# `EvalPlus(📖) => 📚`

<p align="center">
    <a href="#-Quick-Start">🔥Quick Start</a> •
    <a href="#-Papers">📜Papers</a> •
    <a href="#-Useful-tools">🔨Useful tools</a> •
    <a href="#-Development">👷Development</a> •
    <a href="#-Acknowledgement">🙏Acknowledgement</a>
</p>

> **Warning**
> <div align="center">
> <b>
> 🚨 Evaluating LLM-generated code over datasets with "3 test-cases" is **NOT** enough! 🚨
> </b>
> </div>

To address this, we started the EvalPlus project -- a rigourous evaluation framework for LLM4Code that:

+ ✨ improves code benchmarks by adding up to thousands of new tests! (81x new tests for HumanEval!)
+ ✨ crafts a set [utility tools](#useful-tools) to sanitize, visualize and inspect LLM-generated code and evaluation results!
+ ✨ accelerates LLM4Code research by open-sourcing [LLM-generated samples](https://github.com/evalplus/evalplus/releases/tag/v0.1.0) for 14+ models -- no need to re-run the expensive benchmarks!

![](./gallary/overview.png)

## 🔥 Quick Start

To get started, please first setup the environment:

```bash
pip install evalplus --upgrade
```

...Or you can try out the latest developing version:


```bash
pip install "git+https://github.com/evalplus/evalplus.git" --upgrade
```

<details><summary>🤔 Want to use local GitHub repo? <i>:: click to expand ::</i></summary>
<div>

```bash
git clone https://github.com/evalplus/evalplus.git
cd evalplus
export PYTHONPATH=$PYTHONPATH:$(pwd)
pip install -r requirements.txt
```

</div>
</details>

### HumanEval+

The usage is just like the original HumanEval where you just need to implement the `generate_one_completion` function!

```python
from evalplus.data import get_human_eval_plus, write_jsonl

problems = get_human_eval_plus()

num_samples_per_task = 200
samples = [
    dict(task_id=task_id, completion=generate_one_completion(problems[task_id]["prompt"]))
    for task_id in problems
    for _ in range(num_samples_per_task)
]
write_jsonl("samples.jsonl", samples)
```

<details><summary>🤔 What is in a `problem`? <i>:: click to expand ::</i></summary>
<div>

* "task_id" is the identifier string for the task
* "entry_point": name of the function
* "prompt" is the function signature with docstring
+ "canonical_solution" is the ground-truth implementation (re-implemented to fix bugs in HumanEval)
+ "base_input" is the test inputs in original HumanEval
+ "plus_input" is the test inputs brought by EvalPlus

</div>
</details>

To evaluate the samples:

```bash
evalplus.evaluate --dataset humaneval --samples samples.jsonl
```

<details><summary>🤔 Want to use local GitHub repo? <i>:: click to expand ::</i></summary>
<div>

```bash
python evalplus/evaluate.py --dataset humaneval --samples samples.jsonl
```

</div>
</details>

<details><summary>⌨️ More command-line flags<i> :: click to expand ::</i></summary>
<div>

* `--parallel`: by default half of the cores
* `--base-only` (store_ture): only run base HumanEval tests
* `--i-just-wanna-run`: force a re-run

</div>
</details>

### MBPP+ (TBD)


## 📜 Papers

Read our [**paper**](https://arxiv.org/abs/2305.01210) for more detailed findings!

```bibtex
@article{evalplus,
  title={Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation},
  author={Jiawei Liu and Chunqiu Steven Xia and Yuyao Wang and Lingming Zhang},
  journal={arXiv preprint arXiv:2305.01210},
  year={2023},
}
```

## 🔨 Useful tools

To use these tools, please first install the repository from GitHub:

```bash
git clone https://github.com/evalplus/evalplus.git
cd evalplus
pip install -r requirements-tools.txt
```

### Syntax checker for LLM-generated code

Check LLM-produced code and answer the following questions:

1. Is the generation entirely done for all samples / all problems in the dataset?
2. Are LLM-generated code compilable? (if no, something could be wrong and you'd better check)

```shell
python tools/checker.py --folder /path/to/[model]-[??]b_temp_[??] --dataset humaneval
```

### Post code sanitizer

LLM-generated code may contain some syntax errors.
But some of them can be easily fixable by doing simple post-processing.
This tool will make the LLM-generated code more clean/compilable by doing certain post-processing such as trimming with more magical EOFs and some garbage non-code tokens.

```shell
python tools/sanitize.py --eof --folder /path/to/vicuna-[??]b_temp_[??]
# Sanitized code will be produced to `/path/to/vicuna-[??]b_temp_[??]-sanitized`
```

### Render `pass@k` results to `rich` and LaTeX tables

```shell
python tools/render.py --type /path/to/[model]-[??]b # NOTE: no `_temp_[??]`
```

![](./gallary/render.gif)

### Perform test input generation from scratch (TBD)


## 👷 Development

Before you start:

```bash
pip install pre-commit
pre-commit install
export PYTHONPATH=$PYTHONPATH:$(pwd)
```

### Name convention

- `evalplus` is the package name.
- `${DATASET}_plus` is the name of dataset applied with `evalplus`.


## 🙏 Acknowledgement

- [HumanEval](https://github.com/openai/human-eval)
