Metadata-Version: 2.1
Name: yival
Version: 0.1.0a9
Summary: YiVal is an open-source project designed to revolutionize the way developers and researchers evaluate and refine AI models.
License: Apachev2
Author: Tao Feng
Author-email: taofeng.nju@gmail.com
Requires-Python: >=3.10,<3.13
Classifier: License :: Other/Proprietary License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Provides-Extra: trainers
Requires-Dist: aiohttp-socks (>=0.8.0,<0.9.0)
Requires-Dist: alpaca-eval (>=0.3.1,<0.4.0)
Requires-Dist: bert-score (>=0.3.13,<0.4.0)
Requires-Dist: bitsandbytes (>=0.41.1,<0.42.0) ; extra == "trainers"
Requires-Dist: dash (>=2.11.1,<3.0.0)
Requires-Dist: dash-bootstrap-components (>=1.4.2,<2.0.0)
Requires-Dist: dash-dangerously-set-inner-html (>=0.0.2,<0.0.3)
Requires-Dist: faiss-cpu (>=1.7.4,<2.0.0)
Requires-Dist: fuzzywuzzy (>=0.18.0,<0.19.0)
Requires-Dist: hydra-core (>=1.3.2,<2.0.0)
Requires-Dist: inquirer (>=3.1.3,<4.0.0)
Requires-Dist: langchain (>=0.0.275,<0.0.276)
Requires-Dist: litellm (==0.1.607)
Requires-Dist: numpy (>=1.25.2,<2.0.0)
Requires-Dist: omegaconf (>=2.3.0,<3.0.0)
Requires-Dist: openai (==0.27.10)
Requires-Dist: pandas (>=2.0.3,<3.0.0)
Requires-Dist: pandas-stubs (>=2.1.1.230928,<3.0.0.0)
Requires-Dist: peft (>=0.5.0,<0.6.0) ; extra == "trainers"
Requires-Dist: pillow (==9.4.0)
Requires-Dist: psutil (>=5.9.5,<6.0.0)
Requires-Dist: pydantic (>=2.1.1,<3.0.0)
Requires-Dist: pydoc-markdown (>=4.8.2,<5.0.0)
Requires-Dist: pyngrok (>=6.0.0,<7.0.0)
Requires-Dist: python-levenshtein (>=0.21.1,<0.22.0)
Requires-Dist: replicate (>=0.12.0,<0.13.0)
Requires-Dist: rich (>=13.4.2,<14.0.0)
Requires-Dist: rouge (>=1.0.1,<2.0.0)
Requires-Dist: scipy (>=1.11.3,<2.0.0) ; extra == "trainers"
Requires-Dist: streamlit (>=1.27.2,<2.0.0)
Requires-Dist: tenacity (>=8.2.3,<9.0.0)
Requires-Dist: tiktoken (>=0.4.0,<0.5.0)
Requires-Dist: torch (>=2.0.0,!=2.0.1,!=2.1.0)
Requires-Dist: tqdm (>=4.65.0,<5.0.0)
Requires-Dist: transformers (>=4.33.0,<5.0.0)
Requires-Dist: trl (>=0.7.1,<0.8.0) ; extra == "trainers"
Requires-Dist: types-pillow (>=10.0.0.2,<11.0.0.0)
Requires-Dist: types-pyyaml (>=6.0.12.11,<7.0.0.0)
Requires-Dist: types-requests (>=2.31.0.2,<3.0.0.0)
Requires-Dist: types-tqdm (>=4.65.0.2,<5.0.0.0)
Description-Content-Type: text/markdown

<!-- markdownlint-disable MD033 -->
<!-- markdownlint-disable MD041 -->

<p align="center">
    <h1 align="center">
        <img src="https://github.com/YiVal/YiVal/assets/1544154/b0c681e7-7474-4b87-9c69-fde6e0e47401"
         alt="YiVal Logo" width="100"
        height="100" style="vertical-align: middle;">
        YiVal
    </h1>
     <p align="center">⚡ Build any Generative AI application with evaluation
    and enhancement ⚡</p>
</p>

<!-- markdownlint-disable-next-line MD013 -->
👉 Follow
us: [![Twitter](https://img.shields.io/twitter/url/https/twitter.com/YiValai.svg?style=social&label=Follow%20%40YiVal)](https://twitter.com/yivalloveaigc) |
[![Discord](https://dcbadge.vercel.app/api/server/HnUWVW4kth?compact=true&style=flat)](https://discord.gg/HnUWVW4kth)

[![Downloads](https://static.pepy.tech/badge/YiVal/month)](https://pepy.tech/project/YiVal)
[![License: MIT](https://img.shields.io/badge/License-Apache2.0-yellow.svg)](https://opensource.org/licenses/MIT)
[![GitHub star chart](https://img.shields.io/github/stars/YiVal/YiVal?style=social)](https://star-history.com/#YiVal/YiVal)
[![Open Issues](https://img.shields.io/github/issues-raw/YiVal/YiVal)](https://github.com/YiVal/YiVal/issues)

## What is YiVal?

**YiVal: Your Auto-tuning Assistant for GenAI Applications**  
YiVal is a state-of-the-art tool designed to streamline the tuning process for
your GenAI app prompts and **ANY** configs in the loop. With YiVal, manual adjustments
are a thing of the past. This **data-driven and evaluation-centric** approach ensures
optimal prompts, precise RAG configurations, and fine-tuned model parameters.
Empower your applications to achieve
**enhanced results, reduce latency, and minimize inference costs**
effortlessly with YiVal!

**Problems YiVal trying to tackle:**

1. Prompt Development Challenge: "I can't create a better prompt. A score of 60
    for my current prompt isn't helpful at all🤔."
2. Fine-tuning Difficulty: "I don't know how to fine-tune; the terminology and
   numerous fine-tune algorithms are overwhelming😵."
3. Confidence and Scalability: "I learned tutorials to build agents from Langchain
   and LlamaIndex, but am I doing it right? Will the bot burn through my money
    when I launch? Will users like my GenAI app🤯?"
4. Models and Data Drift: "Models and data keep changing; I worry a well-performing
   GenAI app now may fail later😰."
5. Relevant Metrics and Evaluators: "Which metrics and evaluators should I focus
   on for my use case📊?"

[Check out our quickstart guide!][1]

[1]: https://github.com/YiVal/YiVal/blob/master/demo/tutorial_notebook/tutorial.md

<img src="https://github.com/YiVal/YiVal/assets/1544154/dba5acd9-995c-45fd-9d08-c7cf198a77ad">

## Installation

### Prerequisites

- **Python Version**: Ensure you have `Python 3.10` or later installed.
- **OpenAI API Key**: Obtain an API key from OpenAI. Once you have the key, set
  it as an environment variable named `OPENAI_API_KEY`.

### Installation Methods

#### Using pip (Recommended for Users)

Install the `yival` package directly using pip:

```bash
pip install yival
```

#### Development Setup Using Poetry

If you're looking to contribute or set up a development environment:

1. **Install Poetry**: If you haven't already, [install Poetry](https://python-poetry.org/docs/#installation).

2. **Clone the Repository**:

    ```bash
    git clone https://github.com/YiVal/YiVal.git
    cd YiVal
    ```

3. **Setup with Poetry**: Initialize the Python virtual environment and install
   dependencies using Poetry. Make sure to run the below cmd in `/YiVal` directory:

    ```bash
    poetry install --sync
    ```

## Trying Out YiVal

After setting up, you can quickly get started with YiVal by generating datasets
of random tech startup business names.

### Steps to Run Your First YiVal Program

1. **Navigate to the yival Directory**:

    ```bash
    cd /YiVal/src/yival
    ```

2. **Set OpenAI API Key**: Replace `$YOUR_OPENAI_API_KEY` with your
   actual OpenAI API key.

    ```bash
    export OPENAI_API_KEY=$YOUR_OPENAI_API_KEY
    ```

3. **Define YiVal Configuration**:
   Create a configuration file named `config_data_generation.yml` for automated
   test dataset generation with the following content:

    ```yaml
    description: Generate test data
    dataset:
      data_generators:
        openai_prompt_data_generator:
          chunk_size: 100000
          diversify: true
          model_name: gpt-4
          input_function:
            description: # Description of the function
              Given a tech startup business, generate a corresponding landing
              page headline
            name: headline_generation_for_business
            parameters:
              tech_startup_business: str # Parameter name and type
          number_of_examples: 3
          output_csv_path: generated_examples.csv
      source_type: machine_generated
    ```

4. **Execute YiVal**:
   Run the following command from within the `/YiVal/src/yival` directory:

    ```bash
    yival run config_data_generation.yml
    ```

5. **Check the Generated Dataset**:
   The generated test dataset will be stored in `generated_examples.csv`.

## Demo

[Demo Video](https://user-images.githubusercontent.com/68423614/275689372-c3d3497c-1dc4-48bd-860b-a773f11db9f7.mp4)

| Use Case Demo | Supported Features                                                                                                                                                                                                                                                                                                                                                                                                                                                             | Colab Link |
|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------|
| 🐯 Craft your AI story with ChatGPT and MidJourney | **Multi-modal** support: Design an AI-powered narrative using YiVal's multi-modal support of simultaneous text and images. It supports native and seamless [Reinforcement Learning from Human Feedback](https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback)(RLHF) and [Reinforcement Learning from AI Feedback](https://arxiv.org/abs/2309.00267)(RLAIF). Please watch the video above for this use case.                                                |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1QTS3iMwQ5bOEgKDn5DrlmZ6V0rh5APdA?usp=sharing) |
| 🌟 Evaluate performance of multiple LLMs with your own Q&A test dataset | Conveniently **evaluate and compare** performance of your model of choice against 100+ models, thanks to [LiteLLM](https://github.com/BerriAI/litellm). Analyze model performance benchmarks tailored to your **customized test data** or use case.                                                                                                                                                                                                                            |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1cSjsEKNwDq8W4O2oeRI93vH-BhYH5JXj?usp=sharing) |
| 🔥 Startup Company Headline Generation Bot | Streamline generation of headlines for your startup with automated test data **creation**, prompt **crafting**, results **evaluation**, and performance **enhancement** via GPT-4.                                                                                                                                                                                                                                                                                             |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1EYyz5NKW6xDOLZt2r9CoahY_dTGgjW0B?usp=sharing) |
| 🧳 Build a Customized Travel Guide Bot | Leverage **automated prompts** inspired by the travel **community's** most popular suggestions, such as those from [awesome-chatgpt-prompts](https://github.com/f/awesome-chatgpt-prompts).                                                                                                                                                                                                                                                                                    |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/16EthcqowKlGRg6BkELwwq0-_OHy91F3z?usp=sharing) |
| 📖 Build a Cheaper Translator: Use GPT-3.5 to teach Llama2 to create a translator with lower inference cost | Using [Replicate](https://replicate.com/docs/guides/fine-tune-a-language-model) and GPT-3.5's test data, you can **fine-tune** Llama2's translation bot. Benefit from 18x savings while experiencing only a 6% performance decrease.                                                                                                                                                                                                                                           |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1312neVGsrNXfzYsYlgVzdfvN52oJmDo_?usp=sharing) |
| 🤖️ Chat with Your Favorite Characters - Dantan Ji from Till the End of the Moon | Bring your favorite characters to life through automated prompt creation and **character script retrieval**.                                                                                                                                                                                                                                                                                                                                                                   |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1T8KLej36p-DBEQFSLG7n4e1QZb6k2mLb?usp=sharing) |
| 🔍Evaluate guardrails's performance in generating Python(.py) outputs | [Guardrails](https://github.com/ShreyaR/guardrails): where are my guardrails? 😭 <br>Yival: I am here. ⭐️<br><br>The **integrated evaluation** [experiment](https://github.com/ShreyaR/guardrails/issues/345) is carried out with 80 LeetCode problems in csv, using guardrail and using only GPT-4. The accuracy drops from 0.625 to 0.55 with guardrail, latency increases by 44%, and cost increases by 140%. Guardrail still has a long way to go from demo to production. | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](<https://colab.research.google.com/drive/1QgRQmFmC_L07Ler4vbq_vcCNm_OHmJL_#scrollTo=KnmaSTEc13Rg>) |

## Contribution Guidelines

If you want to contribute to YiVal, be sure to review
the [contribution guidelines](https://yival.github.io/YiVal/contributing/).
We use [GitHub issues](https://github.com/YiVal/YiVal/issues) for tracking
requests and bugs.
Please join [YiVal's discord channel](https://discord.gg/HnUWVW4kth) for
general questions and discussion.
Join our collaborative community where your unique expertise as researchers and
software engineers is highly valued! Contribute to our project and be a part of
an innovative space where every line of code and research insight actively
fuels advancements in technology, fostering a future that is intelligently
connected and universally accessible.

## Contributors

<a href="https://github.com/YiVal/YiVal/graphs/contributors">
  <img src="https://contrib.rocks/image?repo=YiVal/YiVal" />
</a>  
<p align="center">
<br>
🌟 YiVal welcomes your contributions! 🌟<p align="center">
🥳 Thanks so much to all of our amazing contributors 🥳</p>

## Paper / Algorithm Implementation

| **Paper**                                                                                  | **Author**                                                                                                                                                                                                                                                  | **Topics**                        | **YiVal Contributor**                      | **Data Generator**                                                                                                                | **Variation Generator**                                                                                                                                | **Evaluator**                                                                                                                                                                                                                            | **Selector**                                                                                          | **Enhancer**                                                                                                                                                   | **Config**                                                                                                     |
|--------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------|--------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------|
| [Large Language Models Are Human-Level Prompt Engineers](https://arxiv.org/abs/2211.01910) | [Yongchao Zhou](https://arxiv.org/search/cs?searchtype=author&query=Zhou,+Y), [Andrei Ioan Muresanu](https://arxiv.org/search/cs?searchtype=author&query=Muresanu,+A+I), [Ziwen Han](https://arxiv.org/search/cs?searchtype=author&query=Han,+Z)            | YiVal Evolver, Auto-Prompting     | [@Tao Feng](https://github.com/oliverfeng) | [OpenAIPromptDataGenerator](https://github.com/YiVal/YiVal/blob/master/src/yival/data_generators/openai_prompt_data_generator.py) | [OpenAIPromptVariationGenerator](https://github.com/YiVal/YiVal/blob/master/src/yival/variation_generators/openai_prompt_based_variation_generator.py) | [OpenAIPromptEvaluator](https://github.com/YiVal/YiVal/blob/master/src/yival/evaluators/openai_prompt_based_evaluator.py), [OpenAIEloEvaluator](https://github.com/YiVal/YiVal/blob/master/src/yival/evaluators/openai_elo_evaluator.py) | [AHPSelector](https://github.com/YiVal/YiVal/blob/master/src/yival/result_selectors/ahp_selection.py) | [OpenAIPromptBasedCombinationEnhancer](https://github.com/YiVal/YiVal/blob/master/src/yival/enhancers/openai_prompt_based_combination_enhancer.py) | [config](https://github.com/YiVal/YiVal/blob/master/demo/configs/headline_generation_enhance.yml)              |
| [BERTScore: Evaluating Text Generation with BERT](https://arxiv.org/abs/1904.09675)        | [Tianyi Zhang](https://arxiv.org/search/cs?searchtype=author&query=Zhang,+T), [Varsha Kishore](https://arxiv.org/search/cs?searchtype=author&query=Kishore,+V), [Felix Wu](https://arxiv.org/search/cs?searchtype=author&query=Wu,+F)                       | YiVal Evaluator, bertscore, rouge | [@crazycth](https://github.com/crazycth)   | -                                                                                                                                 | -                                                                                                                                                      | [BertScoreEvaluator](https://github.com/YiVal/YiVal/blob/master/src/yival/evaluators/bertscore_evaluator.py)                                                                                                                             | -                                                                                                     | -                                                                                                                                                              | -                                                                                                              |
| [AlpacaEval](https://github.com/tatsu-lab/alpaca_eval)                                     | [Xuechen Li](https://arxiv.org/search/cs?searchtype=author&query=Xuechen%20Li), [Tianyi Zhang](https://arxiv.org/search/cs?searchtype=author&query=Tianyi%20Zhang), [Yann Dubois](https://arxiv.org/search/cs?searchtype=author&query=Yann%20Dubois) et. al | YiVal Evaluator                   | [@Tao Feng](https://github.com/oliverfeng) | -                                                                                                                                 | -                                                                                                                                                      | [AlpacaEvalEvaluator](https://github.com/YiVal/YiVal/blob/master/src/yival/evaluators/alpaca_eval_evaluator.py)                                                                                                                          | -                                                                                                     | -                                                                                                                                                              | [config](https://github.com/YiVal/YiVal/blob/master/demo/configs/alpaca_eval.yml)                              |
 [Chain of Density](https://arxiv.org/pdf/2309.04269.pdf)                                   | [Griffin Adams](https://arxiv.org/search/?query=Griffin+Adam) [Alexander R. Fabbri](https://arxiv.org/search/?query=Alexander+R.+Fabbri) et. al                                                                                                             | Prompt Engineering                | [@Tao Feng](https://github.com/oliverfeng) | -                                                                                                                                 | [ChainOfDensityGenerator](https://github.com/YiVal/YiVal/blob/master/src/yival/variation_generators/chain_of_density_prompt.py)                        | -                                                                                                                                                                                                                                        | -                                                                                                     | -                                                                                                                                                              | [config](https://github.com/YiVal/YiVal/blob/master/demo/configs/summary_config.yml)                           |
 [Large Language Models as Optimizers](https://arxiv.org/abs/2309.03409)                    | [Chengrun Yang](https://arxiv.org/search/cs?searchtype=author&query=Yang,+C) [Xuezhi Wang](https://arxiv.org/search/cs?searchtype=author&query=Wang,+X) et. al                                                                                              | Prompt Engineering                | [@crazycth](https://github.com/crazycth)   | -                                                                                                                                 | -                                                                                                                                                      | -                                                                                                                                                                                                                                        | -                                                                                                     | [optimize_by_prompt_enhancer](https://github.com/YiVal/YiVal/blob/opro_implement/src/yival/enhancers/optimize_by_prompt_improver.py)               | [config](https://github.com/YiVal/YiVal/blob/opro_implement/demo/configs/headline_generation_improve.yml#L174) | 123213 | [Chengrun Yang](https://arxiv.org/search/cs?searchtype=author&query=Yang,+C) [Xuezhi Wang](https://arxiv.org/search/cs?searchtype=author&query=Wang,+X) et. al | Prompt Engineering |[@crazycth](https://github.com/crazycth) |-  | - |-  |- |[optimize_by_prompt_improver](https://github.com/YiVal/YiVal/blob/opro_implement/src/yival/combination_improvers/optimize_by_prompt_improver.py)  | [config](https://github.com/YiVal/YiVal/blob/opro_implement/demo/configs/headline_generation_improve.yml#L174)|
 [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685)     | [Edward J. Hu](https://arxiv.org/search/cs?searchtype=author&query=Hu,+E+J) [Yelong Shen](https://arxiv.org/search/cs?searchtype=author&query=Shen,+Y) et. al                                                                                               | LLM Finetune                      | [@crazycth](https://github.com/crazycth)   | -                                                                                                                                 | -                                                                                                                                                      | -                                                                                                                                                                                                                                        | -                                                                                                     | [sft_trainer](https://github.com/YiVal/YiVal/blob/add_finetune_module/src/yival/finetune/sft_trainer.py#L40)                                                   | [config](https://github.com/YiVal/YiVal/blob/add_finetune_module/src/yival/schemas/trainer_configs.py#L48)     |

<!--  -->

