# -*- coding: utf-8 -*-
from setuptools import setup

packages = \
['openrlbenchmark']

package_data = \
{'': ['*']}

install_requires = \
['expt>=0.4.1,<0.5.0',
 'importlib-metadata>=5.1.0,<6.0.0',
 'multiprocess>=0.70.13,<0.71.0',
 'pip>=22.1.2,<23.0.0',
 'rich<12.0',
 'seaborn>=0.12.1,<0.13.0',
 'tabulate>=0.9.0,<0.10.0',
 'tqdm>=4.65.0,<5.0.0',
 'tueplots>=0.0.4,<0.0.5',
 'wandb>=0.13.7,<0.14.0']

setup_kwargs = {
    'name': 'openrlbenchmark',
    'version': '0.1.1b3',
    'description': '',
    'long_description': '# Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learning\n\n\n[<img src="https://img.shields.io/badge/license-MIT-blue">](https://github.com/vwxyzjn/cleanrl)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n[![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/)\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openrlbenchmark/openrlbenchmark/blob/master/README.ipynb)\n[<img src="https://dcbadge.vercel.app/api/server/GsmqhDcea5?style=flat">](https://discord.gg/GsmqhDcea5)\n\n\n\nOpen RL Benchmark is a comprehensive collection of tracked experiments for RL. It aims to make it easier for RL practitioners to pull and compare all kinds of metrics from reputable RL libraries like Stable-baselines3, Tianshou, CleanRL, and others.\n\n* 💾 [GitHub Repo](https://github.com/openrlbenchmark/openrlbenchmark): source code and more docs.\n* 📜 [Design docs](https://docs.google.com/document/d/1cDI_AMr2QVmkC53dCHFMYwGJtLC8V4p6KdL2wnYPaiI/edit?usp=sharing): our motivation and vision.\n* 🔗 [Open RL Benchmark reports](https://wandb.ai/openrlbenchmark/openrlbenchmark/reportlist):  W&B reports with tracked Atari, MuJoCo experiments from SB3, CleanRL, and others.\n\n\n\n## Installation\nYou can install it via pip or the dev setup.\n\n### Pip install\n\n```shell\npip install openrlbenchmark --upgrade\n```\n\n### Dev Setup\n\nPrerequisites:\n* Python >=3.7.1,<3.10 (not yet 3.10)\n* [Poetry 1.2.1+](https://python-poetry.org)\n\n```shell\ngit clone https://github.com/openrlbenchmark/openrlbenchmark.git\ncd openrlbenchmark\npoetry install\n```\n\n\n## Get started\n\nOpen RL Benchmark provides an RLops CLI to pull and compare metrics from Weights and Biases. The following example shows how to compare the performance of SB3\'s ppo, a2c, ddpg, ppo_lstm, sac, td3, ppo, trpo, CleanRL\'s sac on `HalfCheetahBulletEnv-v0`.\n\n```shell\npython -m openrlbenchmark.rlops \\\n    --filters \'?we=openrlbenchmark&wpn=sb3&ceik=env&cen=algo&metric=rollout/ep_rew_mean\' \\\n        \'a2c\' \\\n        \'ddpg\' \\\n        \'ppo_lstm?cl=PPO w/ LSTM\' \\\n        \'sac\' \\\n        \'td3\' \\\n        \'ppo\' \\\n        \'trpo\' \\\n    --filters \'?we=openrlbenchmark&wpn=cleanrl&ceik=env_id&cen=exp_name&metric=charts/episodic_return\' \\\n        \'sac_continuous_action?tag=rlops-pilot&cl=SAC\' \\\n    --env-ids HalfCheetahBulletEnv-v0 \\\n    --ncols 1 \\\n    --ncols-legend 2 \\\n    --xlabel \'Training Steps\' \\\n    --ylabel \'Episodic Return\' \\\n    --output-filename compare \\\n    --report\n```\n\nHere, we created multiple filters. The first string in the first filter is `\'?we=openrlbenchmark&wpn=sb3&ceik=env&cen=algo&metric=rollout/ep_rew_mean\'`, which is a query string that specifies the following:\n\n* `we`: the W&B entity name\n* `wpn`: the W&B project name\n* `ceik`: the custom key for the environment id\n* `cen`: the custom key for the experiment name\n* `metric`: the metric we are interested in\n\nSo we are fetching metrics from [https://wandb.ai/openrlbenchmark/sb3](https://wandb.ai/openrlbenchmark/sb3). The environment id is stored in the `env` key, and the experiment name is stored in the `algo` key. The metric we are interested in is `rollout/ep_rew_mean`.\n\nSimilarly, we are fetching metrics from [https://wandb.ai/openrlbenchmark/cleanrl](https://wandb.ai/openrlbenchmark/cleanrl). The environment id is stored in the `env_id` key, and the experiment name is stored in the `exp_name` key. The metric we are interested in is `charts/episodic_return`. You can also customize the legend with the `cl` query string, such as `ppo_lstm?cl=PPO w/ LSTM`.\n\nThe labels of the figure can be customized with the `--xlabel` and `--ylabel` flags. The `--ncols` flag specifies the number of columns in the figure. The `--ncols-legend` flag specifies the number of columns in the legend. The `--output-filename` flag specifies the filename of the output figure\n\nThe command above generates the following plot:\n\n|    cleanrl vs. Stable Baselines 3   |    cleanrl vs. Stable Baselines 3 (Time)   |\n|:----------------------------------:|:----------------------------------------:|\n|  ![](static/cleanrl_vs_sb3.png)   |   ![](static/cleanrl_vs_sb3-time.png)   |\n\n\n\nThe `--report` tag also generates a [wandb report](https://wandb.ai/costa-huang/cleanrl/reports/Regression-Report-sac_continuous_action--VmlldzozMTY4NDQ3)\n\n\nThe command also generates a `compare.png`, a `compare.md`, and a `compare.csv` in the current directory.\n\n**Learning curves:** the `compare.png` shows the learning curves which subsamples 10000 data points and and interpolate. The curves are smoothed by a rolling average with a window size 100 and their shaded region represents the standard deviation.\n\n**Result table:** the `compare.md` and `compare.csv` shows the average episodic return of the last 100 episodes. For each random seed $i$ (we have 3 random seeds for each set of experiments), we calculate the average episodic return of the last 100 training episodes as $a_i$. We then average the $a_i$\'s over all random seeds to get the final average episodic return and report its standard deviation. This type of evaluation is known as an implicit evaluation method ([Machado et al., 2017](https://arxiv.org/pdf/1709.06009.pdf)) which aligns better with the general goal of RL which is continual learning. This method also detects issues with catastrophic forgetting compared to the evaluation method that evaluates the best model.\n\n\n> **Warning**\n> You may get slightly different curves every time you run the commands. This is because we sample 500 data points from the track experiments to save bandwidth. You can, however, get all of the data points by running the command with `--scan-history` which will always give you the same curves but will take longer to run. For generating an accurate `compare.md`, you should always use `--scan-history`.\n\n\n\n## Currently supported libraries\n\n* [CleanRL](https://wandb.ai/openrlbenchmark/cleanrl)\n    * `ceik`: `env_id`\n    * `cen`: `exp_name` (e.g., `sac_continuous_action`, `ppo_continuous_action`, `ppo_atari`)\n    * `metric`: `charts/episodic_return`\n* [Stable-baselines3](https://wandb.ai/openrlbenchmark/sb3)\n    * `ceik`: `env`\n    * `cen`: `algo` (e.g., `sac`, `ppo`, `a2c`)\n    * `metric`: `rollout/ep_rew_mean` or `eval/mean_reward`\n* [ikostrikov/jaxrl](https://wandb.ai/openrlbenchmark/jaxrl)\n    * `ceik`: `env_name`\n    * `cen`: `algo` (e.g., `sac`)\n    * `metric`: `training/return` or `evaluation/average_returns`\n* [baselines](https://wandb.ai/openrlbenchmark/baselines)\n    * `ceik`: `env`\n    * `cen`: `alg` (e.g., `ppo2`)\n    * `metric`: `charts/episodic_return` or `eprewmean`\n* [sbx](https://wandb.ai/openrlbenchmark/sbx)\n    * `ceik`: `env`\n    * `cen`: `alg` (e.g., `sac`, `tqc`)\n    * `metric`: `rollout/ep_rew_mean` or `eval/mean_reward`\n* [Tianshou](https://wandb.ai/tianshou/atari.benchmark)\n    * `ceik`: `task`\n    * `cen`: `algo_name` (e.g., `ppo`, `iqn`)\n    * `metric`: `test/reward`\n\nThe following libraries have some recorded experiments:\n\n* [openai/phasic-policy-gradient](https://wandb.ai/openrlbenchmark/phasic-policy-gradient) (has some metrics)\n    * `ceik`: `env_name`\n    * `cen`: `arch` (`shared`)\n    * `metric`: `charts/episodic_return`\n* [sfujim/TD3](https://wandb.ai/openrlbenchmark/sfujim-TD3) (has some metrics)\n    * `ceik`: `env`\n    * `cen`: `policy` (e.g., `TD3`)\n    * `metric`: `charts/episodic_return`\n\n\n## More examples\n\n### Compare CleanRL\'s PPO with `openai/baselines`\'s PPO2 on Atari games:\n\n```shell\npython -m openrlbenchmark.rlops \\\n    --filters \'?we=openrlbenchmark&wpn=baselines&ceik=env&cen=exp_name&metric=charts/episodic_return\' \'baselines-ppo2-cnn\' \\\n    --filters \'?we=openrlbenchmark&wpn=envpool-atari&ceik=env_id&cen=exp_name&metric=charts/avg_episodic_return\' \'ppo_atari_envpool_xla_jax_truncation\' \\\n    --env-ids Alien-v5 Amidar-v5 Assault-v5 Asterix-v5 Asteroids-v5 Atlantis-v5 BankHeist-v5 BattleZone-v5 BeamRider-v5 Berzerk-v5 Bowling-v5 Boxing-v5 Breakout-v5 Centipede-v5 ChopperCommand-v5 CrazyClimber-v5 Defender-v5 DemonAttack-v5 DoubleDunk-v5 Enduro-v5 FishingDerby-v5 Freeway-v5 Frostbite-v5 Gopher-v5 Gravitar-v5 Hero-v5 IceHockey-v5 PrivateEye-v5 Qbert-v5 Riverraid-v5 RoadRunner-v5 Robotank-v5 Seaquest-v5 Skiing-v5 Solaris-v5 SpaceInvaders-v5 StarGunner-v5 Surround-v5 Tennis-v5 TimePilot-v5 Tutankham-v5 UpNDown-v5 Venture-v5 VideoPinball-v5 WizardOfWor-v5 YarsRevenge-v5 Zaxxon-v5 Jamesbond-v5 Kangaroo-v5 Krull-v5 KungFuMaster-v5 MontezumaRevenge-v5 MsPacman-v5 NameThisGame-v5 Phoenix-v5 Pitfall-v5 Pong-v5 \\\n    --check-empty-runs False \\\n    --ncols 5 \\\n    --ncols-legend 2 \\\n    --output-filename static/cleanrl_vs_baselines \\\n    --scan-history\n```\n\n![](static/cleanrl_vs_baselines.png)\n\n### Compare CleanRL\'s PPO with `openai/baselines`\'s PPO2 and `jaxrl`\'s SAC on Mujoco:\n\n```shell\npython -m openrlbenchmark.rlops \\\n    --filters \'?we=openrlbenchmark&wpn=baselines&ceik=env&cen=exp_name&metric=charts/episodic_return\' \'baselines-ppo2-mlp\' \\\n    --filters \'?we=openrlbenchmark&wpn=cleanrl&ceik=env_id&cen=exp_name&metric=charts/episodic_return\' \'ppo_continuous_action?tag=v1.0.0-27-gde3f410\' \\\n    --filters \'?we=openrlbenchmark&wpn=jaxrl&ceik=env_name&cen=algo&metric=training/return\' \'sac\' \\\n    --env-ids HalfCheetah-v2 Walker2d-v2 Hopper-v2 InvertedPendulum-v2 Humanoid-v2 Pusher-v2 \\\n    --check-empty-runs False \\\n    --ncols 3 \\\n    --ncols-legend 3 \\\n    --output-filename static/baselines_vs_cleanrl_vs_jaxrl \\\n    --scan-history\n```\n![](static/baselines_vs_cleanrl_vs_jaxrl.png)\n\n\n```shell\npython -m openrlbenchmark.rlops \\\n    --filters \'?we=openrlbenchmark&wpn=baselines&ceik=env&cen=exp_name&metric=charts/episodic_return\' \'baselines-ppo2-cnn\' \\\n    --filters \'?we=openrlbenchmark&wpn=envpool-atari&ceik=env_id&cen=exp_name&metric=charts/avg_episodic_return\' \'ppo_atari_envpool_xla_jax_truncation\' \\\n    --filters \'?we=openrlbenchmark&wpn=cleanrl&ceik=env_id&cen=exp_name&metric=charts/avg_episodic_return\' \'ppo_atari_envpool_xla_jax_scan?tag=pr-328\' \\\n    --env-ids BeamRider-v5 Breakout-v5 Pong-v5 \\\n    --check-empty-runs False \\\n    --ncols 3 \\\n    --ncols-legend 2 \\\n    --output-filename compare \\\n    --scan-history\n```\n\n\n### Compare Tianshou\'s algorithms with `openai/baselines`\'s PPO2 on Atari:\n\n```shell\npython -m openrlbenchmark.rlops \\\n    --filters \'?we=tianshou&wpn=atari.benchmark&ceik=task&cen=algo_name&metric=test/reward\' \'iqn\' \'ppo\' \'rainbow\' \'fqf\' \'c51\' \'dqn\' \'qrdqn\' \\\n    --filters \'?we=openrlbenchmark&wpn=baselines&ceik=env&cen=exp_name&metric=charts/episodic_return\' \'baselines-ppo2-cnn\' \\\n    --env-ids BreakoutNoFrameskip-v4 SpaceInvadersNoFrameskip-v4 SeaquestNoFrameskip-v4 MsPacmanNoFrameskip-v4 EnduroNoFrameskip-v4 PongNoFrameskip-v4 QbertNoFrameskip-v4 \\\n    --check-empty-runs False \\\n    --ncols 4 \\\n    --ncols-legend 4 \\\n    --output-filename static/baselines_vs_tianshou --scan-history\n```\n![](static/baselines_vs_tianshou.png)\n\n\n### Compare CleanRL\'s PPG and PPO with `openai/phasic-policy-gradient`\'s PPG on procgen:\n\n```shell\npython -m openrlbenchmark.rlops \\\n    --filters \'?we=openrlbenchmark&wpn=phasic-policy-gradient&ceik=env_name&cen=arch&metric=charts/episodic_return\' \'shared\' \\\n    --filters \'?we=openrlbenchmark&wpn=cleanrl&ceik=env_id&cen=exp_name&metric=charts/episodic_return\' \'ppo_procgen?tag=v1.0.0b1-4-g4ea73d9\' \'ppg_procgen?tag=v1.0.0b1-4-g4ea73d9\' \\\n    --env-ids starpilot bossfight bigfish \\\n    --check-empty-runs False \\\n    --ncols 3 \\\n    --ncols-legend 3 \\\n    --output-filename static/ppg_vs_cleanrl \\\n    --scan-history\n```\n\n![](static/ppg_vs_cleanrl.png)\n\n\n### Compare CleanRL\'s TD3 with `sfujim/TD3`\'s TD3 on Mujoco:\n\n```shell\npython -m openrlbenchmark.rlops \\\n    --filters \'?we=openrlbenchmark&wpn=sfujim-TD3&ceik=env&cen=policy&metric=charts/episodic_return\' \'TD3\' \\\n    --filters \'?we=openrlbenchmark&wpn=cleanrl&ceik=env_id&cen=exp_name&metric=charts/episodic_return\' \'td3_continuous_action_jax?tag=pr-285\' \'ddpg_continuous_action_jax?tag=pr-298\' \\\n    --env-ids HalfCheetah-v2 Walker2d-v2 Hopper-v2 \\\n    --check-empty-runs False \\\n    --ncols 3 \\\n    --ncols-legend 3 \\\n    --output-filename static/td3_vs_cleanrl \\\n    --scan-history\n```\n![](static/td3_vs_cleanrl.png)\n\n\n### Compare CleanRL\'s PPO + JAX + EnvPool\'s XLA with `openai/baselines`\'s Atari wrappers vs [Machado\'s recommendetation for Atari](https://arxiv.org/abs/1709.06009) (e.g., sticky action with probability 0.25). The machado\'s experiments ran for 10M steps, which corresponds to 40M frames.\n\n```shell\npython -m openrlbenchmark.rlops \\\n    --filters \'?we=openrlbenchmark&wpn=envpool-atari&ceik=env_id&cen=exp_name&metric=charts/avg_episodic_return\' \'ppo_atari_envpool_xla_jax_truncation_machado_10M\' \'ppo_atari_envpool_xla_jax_truncation\'  \\\n    --env-ids Alien-v5 Amidar-v5 Assault-v5 Asterix-v5 Asteroids-v5 Atlantis-v5 BankHeist-v5 BattleZone-v5 BeamRider-v5 Berzerk-v5 Bowling-v5 Boxing-v5 Breakout-v5 Centipede-v5 ChopperCommand-v5 CrazyClimber-v5 Defender-v5 DemonAttack-v5 DoubleDunk-v5 Enduro-v5 FishingDerby-v5 Freeway-v5 Frostbite-v5 Gopher-v5 Gravitar-v5 Hero-v5 IceHockey-v5 PrivateEye-v5 Qbert-v5 Riverraid-v5 RoadRunner-v5 Robotank-v5 Seaquest-v5 Skiing-v5 Solaris-v5 SpaceInvaders-v5 StarGunner-v5 Surround-v5 Tennis-v5 TimePilot-v5 Tutankham-v5 UpNDown-v5 Venture-v5 VideoPinball-v5 WizardOfWor-v5 YarsRevenge-v5 Zaxxon-v5 Jamesbond-v5 Kangaroo-v5 Krull-v5 KungFuMaster-v5 MontezumaRevenge-v5 MsPacman-v5 NameThisGame-v5 Phoenix-v5 Pitfall-v5 Pong-v5 \\\n    --check-empty-runs False \\\n    --ncols 5 \\\n    --ncols-legend 2 \\\n    --output-filename static/machado_10M \\\n    --scan-history\n```\n![](static/machado_10M.png)\n\n\n### Compare CleanRL\'s PPO + JAX + EnvPool\'s XLA with `openai/baselines`\'s Atari wrappers vs [Machado\'s recommendetation for Atari](https://arxiv.org/abs/1709.06009) (e.g., sticky action with probability 0.25). The machado\'s experiments ran for 50M steps, which corresponds to 200M frames.\n\n```shell\npython -m openrlbenchmark.rlops \\\n    --filters \'?we=openrlbenchmark&wpn=envpool-atari&ceik=env_id&cen=exp_name&metric=charts/avg_episodic_return\' \'ppo_atari_envpool_xla_jax_truncation_machado\' \'ppo_atari_envpool_xla_jax_truncation\'  \\\n    --env-ids Alien-v5 Amidar-v5 Assault-v5 Asterix-v5 Asteroids-v5 Atlantis-v5 BankHeist-v5 BattleZone-v5 BeamRider-v5 Berzerk-v5 Bowling-v5 Boxing-v5 Breakout-v5 Centipede-v5 ChopperCommand-v5 CrazyClimber-v5 Defender-v5 DemonAttack-v5 DoubleDunk-v5 Enduro-v5 FishingDerby-v5 Freeway-v5 Frostbite-v5 Gopher-v5 Gravitar-v5 Hero-v5 IceHockey-v5 PrivateEye-v5 Qbert-v5 Riverraid-v5 RoadRunner-v5 Robotank-v5 Seaquest-v5 Skiing-v5 Solaris-v5 SpaceInvaders-v5 StarGunner-v5 Surround-v5 Tennis-v5 TimePilot-v5 Tutankham-v5 UpNDown-v5 Venture-v5 VideoPinball-v5 WizardOfWor-v5 YarsRevenge-v5 Zaxxon-v5 Jamesbond-v5 Kangaroo-v5 Krull-v5 KungFuMaster-v5 MontezumaRevenge-v5 MsPacman-v5 NameThisGame-v5 Phoenix-v5 Pitfall-v5 Pong-v5 \\\n    --check-empty-runs False \\\n    --ncols 5 \\\n    --ncols-legend 2 \\\n    --output-filename static/machado_50M \\\n    --scan-history\n```\n![](static/machado_50M.png)\n\n### Calculate human normalized scores\n\n```shell\npython -m openrlbenchmark.hns --files static/cleanrl_vs_baselines.csv static/machado_10M.csv static/machado_50M.csv \n```\n\n```\nbaselines-ppo2-cnn ({})\n┣━━ median hns: 0.7959851540635047\n┣━━ mean hns: 4.54588939893709\nppo_atari_envpool_xla_jax_truncation ({})\n┣━━ median hns: 0.9783505154639175\n┣━━ mean hns: 6.841083973256849\nppo_atari_envpool_xla_jax_truncation_machado_10M ({})\n┣━━ median hns: 0.7347972972972973\n┣━━ mean hns: 2.919095857954249\nppo_atari_envpool_xla_jax_truncation ({\'metric\': [\'charts/avg_episodic_return\']})\n┣━━ median hns: 0.9783505154639175\n┣━━ mean hns: 6.841083973256849\nppo_atari_envpool_xla_jax_truncation_machado ({\'metric\': [\'charts/avg_episodic_return\']})\n┣━━ median hns: 1.5679929625118418\n┣━━ mean hns: 8.352308370550299\n```\n\n## What\'s going on right now?\n\nThis is a project we are slowly working on. There is no specific timeline or roadmap, but if you want to get involved. Feel free to reach out to me or open an issue. We are looking for volunteers to help us with the following:\n\n* Add experiments from other libraries\n* Run more experiments from currently supported libraries\n* Documentation and designing standards\n* Download the tensorboard metrics from the tracked experiments and load them locally to save time\n\n\n\n\n```shell\npip install --upgrade openrlbenchmark\npython -m openrlbenchmark.rlops \\\n    --filters \'?we=openrlbenchmark&wpn=cleanrl&ceik=env_id&cen=exp_name&metric=charts/episodic_return\' \'dqn_atari?tag=latest\' \'c51_atari?tag=latest\' \\\n    --filters \'?we=openrlbenchmark&wpn=baselines&ceik=env&cen=exp_name&metric=charts/episodic_return\' \'baselines-ppo2-cnn\' \\\n    --filters \'?we=openrlbenchmark&wpn=envpool-atari&ceik=env_id&cen=exp_name&metric=charts/avg_episodic_return\' \'ppo_atari_envpool_xla_jax_truncation?user=costa-huang\' \\\n    --env-ids BeamRider-v5 Breakout-v5 Pong-v5 \\\n    --ncols 3 \\\n    --ncols-legend 2 \\\n    --output-filename compare \\\n    --scan-history \\\n    --report \\\n    --check-empty-runs False \n```\n',
    'author': 'Costa Huang',
    'author_email': 'costa.huang@outlook.com',
    'maintainer': 'None',
    'maintainer_email': 'None',
    'url': 'None',
    'packages': packages,
    'package_data': package_data,
    'install_requires': install_requires,
    'python_requires': '>=3.7.1,<4.0.0',
}


setup(**setup_kwargs)
