[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-70795":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":17,"stars7d":18,"stars30d":19,"stars90d":16,"forks30d":16,"starsTrendScore":20,"compositeScore":21,"rankGlobal":10,"rankLanguage":10,"license":22,"archived":23,"fork":23,"defaultBranch":24,"hasWiki":25,"hasPages":23,"topics":26,"createdAt":10,"pushedAt":10,"updatedAt":46,"readmeContent":47,"aiSummary":48,"trendingCount":16,"starSnapshotCount":16,"syncStatus":49,"lastSyncTime":50,"discoverSource":51},70795,"tianshou","thu-ml\u002Ftianshou","thu-ml","An elegant PyTorch deep reinforcement learning library.","https:\u002F\u002Ftianshou.org",null,"Python",10790,1319,96,131,0,10,25,115,30,111.86,"MIT License",false,"master",true,[27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45],"a2c","atari","bcq","cql","ddpg","double-dqn","dqn","drl","imitation-learning","mujoco","npg","policy-gradient","ppo","pytorch","rl","sac","td3","transferlab","trpo","2026-06-12 04:00:57","\u003Cdiv align=\"center\">\n \u003Ca href=\"http:\u002F\u002Ftianshou.readthedocs.io\">\u003Cimg width=\"300px\" height=\"auto\" src=\"https:\u002F\u002Fgithub.com\u002Fthu-ml\u002Ftianshou\u002Fraw\u002Fmaster\u002Fdocs\u002F_static\u002Fimages\u002Ftianshou-logo.png\">\u003C\u002Fa>\n\u003C\u002Fdiv>\n\n---\n\n[![PyPI](https:\u002F\u002Fimg.shields.io\u002Fpypi\u002Fv\u002Ftianshou)](https:\u002F\u002Fpypi.org\u002Fproject\u002Ftianshou\u002F) [![Read the Docs](https:\u002F\u002Freadthedocs.org\u002Fprojects\u002Ftianshou\u002Fbadge\u002F?version=master)](https:\u002F\u002Ftianshou.org\u002Fen\u002Fmaster\u002F) [![Pytest](https:\u002F\u002Fgithub.com\u002Fthu-ml\u002Ftianshou\u002Factions\u002Fworkflows\u002Fpytest.yml\u002Fbadge.svg)](https:\u002F\u002Fgithub.com\u002Fthu-ml\u002Ftianshou\u002Factions) [![codecov](https:\u002F\u002Fimg.shields.io\u002Fcodecov\u002Fc\u002Fgh\u002Fthu-ml\u002Ftianshou)](https:\u002F\u002Fcodecov.io\u002Fgh\u002Fthu-ml\u002Ftianshou) [![GitHub issues](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Fissues\u002Fthu-ml\u002Ftianshou)](https:\u002F\u002Fgithub.com\u002Fthu-ml\u002Ftianshou\u002Fissues) [![GitHub stars](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Fstars\u002Fthu-ml\u002Ftianshou)](https:\u002F\u002Fgithub.com\u002Fthu-ml\u002Ftianshou\u002Fstargazers) [![GitHub forks](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Fforks\u002Fthu-ml\u002Ftianshou)](https:\u002F\u002Fgithub.com\u002Fthu-ml\u002Ftianshou\u002Fnetwork) [![GitHub license](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Flicense\u002Fthu-ml\u002Ftianshou)](https:\u002F\u002Fgithub.com\u002Fthu-ml\u002Ftianshou\u002Fblob\u002Fmaster\u002FLICENSE)\n\n> [!NOTE]\n> **Tianshou version 2 is here!** \n> \n> We have released the new major version of Tianshou on PyPI. \n> Version 2 is a complete overhaul of the software design of the procedural API, in which\n> * we establish a clear separation between learning algorithms and policies (via the separate abstractions `Algorithm` and `Policy`).\n> * we provide more well-defined, more usable interfaces with extensive documentation of all algorithm and trainer parameters,\n> renaming some parameters to make their names more consistent and intuitive.\n> * the class hierarchy is fully revised, establishing a clear separation between on-policy, off-policy and offline algorithms\n> at the type level and ensuring that all inheritance relationships are meaningful.\n> \n> Because of the extent of the changes, this version is not backwards compatible with previous versions of Tianshou.\n> For migration information, please see the [change log](CHANGELOG.md). \n\n**Tianshou** ([天授](https:\u002F\u002Fbaike.baidu.com\u002Fitem\u002F%E5%A4%A9%E6%8E%88)) is a reinforcement learning (RL) library based on pure PyTorch and [Gymnasium](http:\u002F\u002Fgithub.com\u002FFarama-Foundation\u002FGymnasium). Tianshou's main features at a glance are:\n\n1. Modular low-level interfaces for algorithm developers (RL researchers) that are both flexible, hackable and type-safe.\n1. Convenient high-level interfaces for applications of RL (training an implemented algorithm on a custom environment).\n1. Large scope: online (on- and off-policy) and offline RL, experimental support for multi-agent RL (MARL), experimental support for model-based RL, and more\n\nUnlike other reinforcement learning libraries, which may have complex codebases,\nunfriendly high-level APIs, or are not optimized for speed, Tianshou provides a high-performance, modularized framework\nand user-friendly interfaces for building deep reinforcement learning agents. One more aspect that sets Tianshou apart is its\ngenerality: it supports online and offline RL, multi-agent RL, and model-based algorithms.\n\nTianshou aims at enabling concise implementations, both for researchers and practitioners, without sacrificing flexibility.\n\nSupported algorithms include:\n\n- [Deep Q-Network (DQN)](https:\u002F\u002Fstorage.googleapis.com\u002Fdeepmind-media\u002Fdqn\u002FDQNNaturePaper.pdf)\n- [Double DQN](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1509.06461.pdf)\n- [Dueling DQN](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1511.06581.pdf)\n- [Branching DQN](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1711.08946.pdf)\n- [Categorical DQN (C51)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1707.06887.pdf)\n- [Rainbow DQN (Rainbow)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1710.02298.pdf)\n- [Quantile Regression DQN (QRDQN)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1710.10044.pdf)\n- [Implicit Quantile Network (IQN)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1806.06923.pdf)\n- [Fully-parameterized Quantile Function (FQF)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1911.02140.pdf)\n- [Policy Gradient (PG)](https:\u002F\u002Fpapers.nips.cc\u002Fpaper\u002F1713-policy-gradient-methods-for-reinforcement-learning-with-function-approximation.pdf)\n- [Natural Policy Gradient (NPG)](https:\u002F\u002Fproceedings.neurips.cc\u002Fpaper\u002F2001\u002Ffile\u002F4b86abe48d358ecf194c56c69108433e-Paper.pdf)\n- [Advantage Actor-Critic (A2C)](https:\u002F\u002Fopenai.com\u002Fblog\u002Fbaselines-acktr-a2c\u002F)\n- [Trust Region Policy Optimization (TRPO)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1502.05477.pdf)\n- [Proximal Policy Optimization (PPO)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1707.06347.pdf)\n- [Deep Deterministic Policy Gradient (DDPG)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1509.02971.pdf)\n- [Twin Delayed DDPG (TD3)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1802.09477.pdf)\n- [Soft Actor-Critic (SAC)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1812.05905.pdf)\n- [Randomized Ensembled Double Q-Learning (REDQ)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F2101.05982.pdf)\n- [Discrete Soft Actor-Critic (SAC-Discrete)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1910.07207.pdf)\n- [Vanilla Imitation Learning](https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FApprenticeship_learning)\n- [Batch-Constrained deep Q-Learning (BCQ)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1812.02900.pdf)\n- [Conservative Q-Learning (CQL)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F2006.04779.pdf)\n- [Twin Delayed DDPG with Behavior Cloning (TD3+BC)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F2106.06860.pdf)\n- [Discrete Batch-Constrained deep Q-Learning (BCQ-Discrete)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1910.01708.pdf)\n- [Discrete Conservative Q-Learning (CQL-Discrete)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F2006.04779.pdf)\n- [Discrete Critic Regularized Regression (CRR-Discrete)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F2006.15134.pdf)\n- [Generative Adversarial Imitation Learning (GAIL)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1606.03476.pdf)\n- [Prioritized Experience Replay (PER)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1511.05952.pdf)\n- [Generalized Advantage Estimator (GAE)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1506.02438.pdf)\n- [Posterior Sampling Reinforcement Learning (PSRL)](https:\u002F\u002Fwww.ece.uvic.ca\u002F~bctill\u002Fpapers\u002Flearning\u002FStrens_2000.pdf)\n- [Intrinsic Curiosity Module (ICM)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1705.05363.pdf)\n- [Hindsight Experience Replay (HER)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1707.01495.pdf)\n\nOther noteworthy features:\n\n- Elegant framework with dual APIs:\n - Tianshou's high-level API maximizes ease of use for application development while still retaining a high degree\n of flexibility.\n - The fundamental procedural API provides a maximum of flexibility for algorithm development without being\n overly verbose.\n- State-of-the-art results in [MuJoCo benchmarks](https:\u002F\u002Fgithub.com\u002Fthu-ml\u002Ftianshou\u002Ftree\u002Fmaster\u002Fexamples\u002Fmujoco) for REINFORCE\u002FA2C\u002FTRPO\u002FPPO\u002FDDPG\u002FTD3\u002FSAC algorithms\n- Support for vectorized environments (synchronous or asynchronous) for all algorithms (see [usage](https:\u002F\u002Ftianshou.readthedocs.io\u002Fen\u002Fmaster\u002F01_tutorials\u002F07_cheatsheet.html#parallel-sampling))\n- Support for super-fast vectorized environments based on [EnvPool](https:\u002F\u002Fgithub.com\u002Fsail-sg\u002Fenvpool\u002F) for all algorithms (see [usage](https:\u002F\u002Ftianshou.readthedocs.io\u002Fen\u002Fmaster\u002F01_tutorials\u002F07_cheatsheet.html#envpool-integration))\n- Support for recurrent state representations in actor networks and critic networks (RNN-style training for POMDPs) (see [usage](https:\u002F\u002Ftianshou.readthedocs.io\u002Fen\u002Fmaster\u002F01_tutorials\u002F07_cheatsheet.html#rnn-style-training))\n- Support any type of environment state\u002Faction (e.g. a dict, a self-defined class, ...) [Usage](https:\u002F\u002Ftianshou.readthedocs.io\u002Fen\u002Fmaster\u002F01_tutorials\u002F07_cheatsheet.html#user-defined-environment-and-different-state-representation)\n- Support for customized training processes (see [usage](https:\u002F\u002Ftianshou.readthedocs.io\u002Fen\u002Fmaster\u002F01_tutorials\u002F07_cheatsheet.html#customize-training-process))\n- Support n-step returns estimation and prioritized experience replay for all Q-learning based algorithms; GAE, nstep and PER are highly optimized thanks to numba's just-in-time compilation and vectorized numpy operations\n- Support for multi-agent RL (see [usage](https:\u002F\u002Ftianshou.readthedocs.io\u002Fen\u002Fmaster\u002F01_tutorials\u002F07_cheatsheet.html#multi-agent-reinforcement-learning))\n- Support for logging based on both [TensorBoard](https:\u002F\u002Fwww.tensorflow.org\u002Ftensorboard) and [W&B](https:\u002F\u002Fwandb.ai\u002F)\n- Support for multi-GPU training (see [usage](https:\u002F\u002Ftianshou.readthedocs.io\u002Fen\u002Fmaster\u002F01_tutorials\u002F07_cheatsheet.html#multi-gpu))\n- Comprehensive documentation, PEP8 code-style checking, type checking and thorough [tests](https:\u002F\u002Fgithub.com\u002Fthu-ml\u002Ftianshou\u002Factions)\n\nIn Chinese, Tianshou means divinely ordained, being derived to the gift of being born.\nTianshou is a reinforcement learning platform, and the nature of RL is not learn from humans.\nSo taking \"Tianshou\" means that there is no teacher to learn from, but rather to learn by oneself through constant interaction with the environment.\n\n“天授”意指上天所授，引申为与生具有的天赋。天授是强化学习平台，而强化学习算法并不是向人类学习的，所以取“天授”意思是没有老师来教，而是自己通过跟环境不断交互来进行学习。\n\n## Installation\n\nTianshou is currently hosted on [PyPI](https:\u002F\u002Fpypi.org\u002Fproject\u002Ftianshou\u002F) and [conda-forge](https:\u002F\u002Fgithub.com\u002Fconda-forge\u002Ftianshou-feedstock). It requires Python >= 3.11.\n\nFor installing the most recent version of Tianshou, the best way is clone the repository and install it with [poetry](https:\u002F\u002Fpython-poetry.org\u002F)\n(which you need to install on your system first)\n\n```bash\ngit clone git@github.com:thu-ml\u002Ftianshou.git\ncd tianshou\npoetry install\n```\n\nYou can also install the dev requirements by adding `--with dev` or the extras\nfor say mujoco and acceleration by [envpool](https:\u002F\u002Fgithub.com\u002Fsail-sg\u002Fenvpool)\nby adding `--extras \"mujoco envpool\"`\n\nIf you wish to install multiple extras, ensure that you include them in a single command. Sequential calls to `poetry install --extras xxx` will overwrite prior installations, leaving only the last specified extras installed.\nOr you may install all the following extras by adding `--all-extras`.\n\nAvailable extras are:\n\n- `atari` (for Atari environments)\n- `box2d` (for Box2D environments)\n- `classic_control` (for classic control (discrete) environments)\n- `mujoco` (for MuJoCo environments)\n- `mujoco-py` (for legacy mujoco-py environments[^1])\n- `pybullet` (for pybullet environments)\n- `robotics` (for gymnasium-robotics environments)\n- `vizdoom` (for ViZDoom environments)\n- `envpool` (for [envpool](https:\u002F\u002Fgithub.com\u002Fsail-sg\u002Fenvpool\u002F) integration)\n- `argparse` (in order to be able to run the high level API examples)\n\n[^1]:\n `mujoco-py` is a legacy package and is not recommended for new projects.\n It is only included for compatibility with older projects.\n Also note that there may be compatibility issues with macOS newer than\n Monterey.\n\nOtherwise, you can install the latest release from PyPI (currently\nfar behind the master) with the following command:\n\n```bash\n$ pip install tianshou\n```\n\nIf you are using Anaconda or Miniconda, you can install Tianshou from conda-forge:\n\n```bash\n$ conda install tianshou -c conda-forge\n```\n\nAlternatively to the poetry install, you can also install the latest source version through GitHub:\n\n```bash\n$ pip install git+https:\u002F\u002Fgithub.com\u002Fthu-ml\u002Ftianshou.git@master --upgrade\n```\n\nFinally, you may check the installation via your Python console as follows:\n\n```python\nimport tianshou\nprint(tianshou.__version__)\n```\n\nIf no errors are reported, you have successfully installed Tianshou.\n\n## Documentation\n\nFind example scripts in the [test\u002F]( https:\u002F\u002Fgithub.com\u002Fthu-ml\u002Ftianshou\u002Fblob\u002Fmaster\u002Ftest) and [examples\u002F](https:\u002F\u002Fgithub.com\u002Fthu-ml\u002Ftianshou\u002Fblob\u002Fmaster\u002Fexamples) folders.\n\nTutorials and API documentation are hosted on [tianshou.readthedocs.io](https:\u002F\u002Ftianshou.readthedocs.io\u002F).\n\n## Why Tianshou?\n\n### Comprehensive Functionality\n\n### High Software Engineering Standards\n\n| RL Platform | Documentation | Code Coverage | Type Hints | Last Update |\n| ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------ | ----------------------------------------------------------------------------------------------------------------- |\n| [Stable-Baselines3](https:\u002F\u002Fgithub.com\u002FDLR-RM\u002Fstable-baselines3) | [![Documentation Status](https:\u002F\u002Freadthedocs.org\u002Fprojects\u002Fstable-baselines\u002Fbadge\u002F?version=master)](https:\u002F\u002Fstable-baselines3.readthedocs.io\u002Fen\u002Fmaster\u002F?badge=master) | [![coverage report](https:\u002F\u002Fgitlab.com\u002Faraffin\u002Fstable-baselines3\u002Fbadges\u002Fmaster\u002Fcoverage.svg)](https:\u002F\u002Fgitlab.com\u002Faraffin\u002Fstable-baselines3\u002F-\u002Fcommits\u002Fmaster) | :heavy_check_mark: | ![GitHub last commit](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Flast-commit\u002FDLR-RM\u002Fstable-baselines3?label=last%20update) |\n| [Ray\u002FRLlib](https:\u002F\u002Fgithub.com\u002Fray-project\u002Fray\u002Ftree\u002Fmaster\u002Frllib\u002F) | [![](https:\u002F\u002Freadthedocs.org\u002Fprojects\u002Fray\u002Fbadge\u002F?version=master)](http:\u002F\u002Fdocs.ray.io\u002Fen\u002Fmaster\u002Frllib.html) | :heavy_minus_sign:\u003Csup>(1)\u003C\u002Fsup> | :heavy_check_mark: | ![GitHub last commit](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Flast-commit\u002Fray-project\u002Fray?label=last%20update) |\n| [SpinningUp](https:\u002F\u002Fgithub.com\u002Fopenai\u002Fspinningup) | [![](https:\u002F\u002Fimg.shields.io\u002Freadthedocs\u002Fspinningup)](https:\u002F\u002Fspinningup.openai.com\u002F) | :x: | :x: | ![GitHub last commit](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Flast-commit\u002Fopenai\u002Fspinningup?label=last%20update) |\n| [Dopamine](https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fdopamine) | [![](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Fdocs-passing-green)](https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fdopamine\u002Ftree\u002Fmaster\u002Fdocs) | :x: | :x: | ![GitHub last commit](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Flast-commit\u002Fgoogle\u002Fdopamine?label=last%20update) |\n| [ACME](https:\u002F\u002Fgithub.com\u002Fdeepmind\u002Facme) | [![](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Fdocs-passing-green)](https:\u002F\u002Fgithub.com\u002Fdeepmind\u002Facme\u002Fblob\u002Fmaster\u002Fdocs\u002Findex.md) | :heavy_minus_sign:\u003Csup>(1)\u003C\u002Fsup> | :heavy_check_mark: | ![GitHub last commit](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Flast-commit\u002Fdeepmind\u002Facme?label=last%20update) |\n| [Sample Factory](https:\u002F\u002Fgithub.com\u002Falex-petrenko\u002Fsample-factory) | [:heavy_minus_sign:](https:\u002F\u002Farxiv.org\u002Fabs\u002F2006.11751) | [![codecov](https:\u002F\u002Fcodecov.io\u002Fgh\u002Falex-petrenko\u002Fsample-factory\u002Fbranch\u002Fmaster\u002Fgraph\u002Fbadge.svg)](https:\u002F\u002Fcodecov.io\u002Fgh\u002Falex-petrenko\u002Fsample-factory) | :x: | ![GitHub last commit](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Flast-commit\u002Falex-petrenko\u002Fsample-factory?label=last%20update) |\n| | | | | |\n| [Tianshou](https:\u002F\u002Fgithub.com\u002Fthu-ml\u002Ftianshou) | [![Read the Docs](https:\u002F\u002Fimg.shields.io\u002Freadthedocs\u002Ftianshou)](https:\u002F\u002Ftianshou.readthedocs.io\u002Fen\u002Fmaster) | [![codecov](https:\u002F\u002Fimg.shields.io\u002Fcodecov\u002Fc\u002Fgh\u002Fthu-ml\u002Ftianshou)](https:\u002F\u002Fcodecov.io\u002Fgh\u002Fthu-ml\u002Ftianshou) | :heavy_check_mark: | ![GitHub last commit](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Flast-commit\u002Fthu-ml\u002Ftianshou?label=last%20update) |\n\n\u003Csup>(1): it has continuous integration but the coverage rate is not available\u003C\u002Fsup>\n\n### Reproducible, High-Quality Results\n\nTianshou is rigorously tested. In contrast to other RL platforms, **our tests include the full agent training procedure for all of the implemented algorithms**. Our tests would fail once if any of the agents failed to achieve a consistent level of performance on limited epochs.\nOur tests thus ensure reproducibility.\nCheck out the [GitHub Actions](https:\u002F\u002Fgithub.com\u002Fthu-ml\u002Ftianshou\u002Factions) page for more detail.\n\nAtari and MuJoCo benchmark results can be found in the [examples\u002Fatari\u002F](examples\u002Fatari\u002F) and [examples\u002Fmujoco\u002F](examples\u002Fmujoco\u002F) folders respectively. **Our MuJoCo results reach or exceed the level of performance of most existing benchmarks.**\n\n### Algorithm Abstraction\n\nReinforcement learning algorithms are build on abstractions for\n\n- on-policy algorithms (`OnPolicyAlgorithm`),\n- off-policy algorithms (`OffPolicyAlgorithm`), and\n- offline algorithms (`OfflineAlgorithm`),\n\nall of which clearly separate the core algorithm from the training process and the respective environment interactions.\n\nIn each case, the implementation of an algorithm necessarily involves only the implementation of methods for\n\n- pre-processing a batch of data, augmenting it with necessary information\u002Fsufficient statistics for learning (`_preprocess_batch`),\n- updating model parameters based on an augmented batch of data (`_update_with_batch`).\n\nThe implementation of these methods suffices for a new algorithm to be applicable within Tianshou,\nmaking experimentation with new approaches particularly straightforward.\n\n## Quick Start\n\nTianshou provides two API levels:\n\n- the high-level interface, which provides ease of use for end users seeking to run deep reinforcement learning applications\n- the procedural interface, which provides a maximum of control, especially for very advanced users and developers of reinforcement learning algorithms.\n\nIn the following, let us consider an example application using the _CartPole_ gymnasium environment.\nWe shall apply the deep Q-network (DQN) learning algorithm using both APIs.\n\n### High-Level API\n\nIn the high-level API, the basis for an RL experiment is an `ExperimentBuilder`\nwith which we can build the experiment we then seek to run.\nSince we want to use DQN, we use the specialization `DQNExperimentBuilder`.\n\nThe high-level API provides largely declarative semantics, i.e. the code is\nalmost exclusively concerned with configuration that controls what to do\n(rather than how to do it).\n\n```python\nfrom tianshou.highlevel.config import OffPolicyTrainingConfig\nfrom tianshou.highlevel.env import (\n EnvFactoryRegistered,\n VectorEnvType,\n)\nfrom tianshou.highlevel.experiment import DQNExperimentBuilder, ExperimentConfig\nfrom tianshou.highlevel.params.algorithm_params import DQNParams\nfrom tianshou.highlevel.trainer import (\n EpochStopCallbackRewardThreshold,\n)\n\nexperiment = (\n DQNExperimentBuilder(\n EnvFactoryRegistered(\n task=\"CartPole-v1\",\n venv_type=VectorEnvType.DUMMY,\n training_seed=0,\n test_seed=10,\n ),\n ExperimentConfig(\n persistence_enabled=False,\n watch=True,\n watch_render=1 \u002F 35,\n watch_num_episodes=100,\n ),\n OffPolicyTrainingConfig(\n max_epochs=10,\n epoch_num_steps=10000,\n batch_size=64,\n num_training_envs=10,\n num_test_envs=100,\n buffer_size=20000,\n collection_step_num_env_steps=10,\n update_step_num_gradient_steps_per_sample=1 \u002F 10,\n ),\n )\n .with_dqn_params(\n DQNParams(\n lr=1e-3,\n gamma=0.9,\n n_step_return_horizon=3,\n target_update_freq=320,\n eps_training=0.3,\n eps_inference=0.0,\n ),\n )\n .with_model_factory_default(hidden_sizes=(64, 64))\n .with_epoch_stop_callback(EpochStopCallbackRewardThreshold(195))\n .build()\n)\nexperiment.run()\n```\n\nThe experiment builder takes three arguments:\n\n- the environment factory for the creation of environments. In this case,\n we use an existing factory implementation for gymnasium environments.\n- the experiment configuration, which controls persistence and the overall\n experiment flow. In this case, we have configured that we want to observe\n the agent's behavior after it is trained (`watch=True`) for a number of\n episodes (`watch_num_episodes=100`). We have disabled persistence, because\n we do not want to save training logs, the agent or its configuration for\n future use.\n- the training configuration, which controls fundamental training parameters,\n such as the total number of epochs we run the experiment for (`num_epochs=10`) \n and the number of environment steps each epoch shall consist of\n (`epoch_num_steps=10000`).\n Every epoch consists of a series of data collection (rollout) steps and\n training steps.\n The parameter `collection_step_num_env_steps` controls the amount of data that is\n collected in each collection step and after each collection step, we\n perform a training step, applying a gradient-based update based on a sample\n of data (`batch_size=64`) taken from the buffer of data that has been\n collected. For further details, see the documentation of configuration class.\n\nWe then proceed to configure some of the parameters of the DQN algorithm itself:\nFor instance, we control the epsilon parameter for exploration.\nWe want to use random exploration during rollouts for training (`eps_training`),\nbut we don't when evaluating the agent's performance in the test environments\n(`eps_inference`).\nFurthermore, we configure model parameters of the network for the Q function,\nparametrising the number of hidden layers of the default MLP factory.\n\nFind the script in [examples\u002Fdiscrete\u002Fdiscrete_dqn_hl.py](examples\u002Fdiscrete\u002Fdiscrete_dqn_hl.py).\nHere's a run (with the training time cut short):\n\n\u003Cp align=\"center\" style=\"text-algin:center\">\n \u003Cimg src=\"docs\u002F_static\u002Fimages\u002Fdiscrete_dqn_hl.gif\">\n\u003C\u002Fp>\n\nFind many further applications of the high-level API in the `examples\u002F` folder;\nlook for scripts ending with `_hl.py`.\nNote that most of these examples require the extra `argparse`\n(install it by adding `--extras argparse` when invoking poetry).\n\n### Procedural API\n\nLet us now consider an analogous example in the procedural API.\nFind the full script in [examples\u002Fdiscrete\u002Fdiscrete_dqn.py](https:\u002F\u002Fgithub.com\u002Fthu-ml\u002Ftianshou\u002Fblob\u002Fmaster\u002Fexamples\u002Fdiscrete\u002Fdiscrete_dqn.py).\n\nFirst, import the relevant packages:\n\n```python\nimport gymnasium as gym\nimport tianshou as ts\nfrom tianshou.algorithm.modelfree.dqn import DiscreteQLearningPolicy\nfrom tianshou.algorithm.optim import AdamOptimizerFactory\nfrom tianshou.data import CollectStats\nfrom tianshou.trainer import OffPolicyTrainerParams\nfrom tianshou.utils.net.common import Net\nfrom tianshou.utils.space_info import SpaceInfo\nfrom torch.utils.tensorboard import SummaryWriter\n```\n\nDefine hyper-parameters:\n\n```python\ntask = 'CartPole-v1'\nlr, epoch, batch_size = 1e-3, 10, 64\nnum_training_envs, num_test_envs = 10, 100\ngamma, n_step, target_freq = 0.9, 3, 320\nbuffer_size = 20000\neps_train, eps_test = 0.1, 0.05\nepoch_num_steps, collection_step_num_env_steps = 10000, 10\n```\n\nInitialize the logger:\n\n```python\nlogger = ts.utils.TensorboardLogger(SummaryWriter('log\u002Fdqn'))\n```\n\nCreate the environments:\n\n```python\n# You can also try SubprocVectorEnv, which will use parallelization\ntraining_envs = ts.env.DummyVectorEnv([lambda: gym.make(task) for _ in range(num_training_envs)])\ntest_envs = ts.env.DummyVectorEnv([lambda: gym.make(task) for _ in range(num_test_envs)])\n```\n\nCreate the network, policy, and algorithm:\n\n```python\n# Create the network\n# Note: You can easily define other networks.\n# See https:\u002F\u002Ftianshou.readthedocs.io\u002Fen\u002Fmaster\u002F01_tutorials\u002F00_dqn.html#build-the-network\nenv = gym.make(task, render_mode=\"human\")\nassert isinstance(env.action_space, gym.spaces.Discrete)\nspace_info = SpaceInfo.from_env(env)\nstate_shape = space_info.observation_info.obs_shape\naction_shape = space_info.action_info.action_shape\nnet = Net(state_shape=state_shape, action_shape=action_shape, hidden_sizes=[128, 128, 128])\noptim = AdamOptimizerFactory(lr=lr)\n\n# Create the policy\npolicy = DiscreteQLearningPolicy(\n model=net,\n action_space=env.action_space,\n eps_training=eps_train,\n eps_inference=eps_test\n)\n\n# Create the algorithm with the policy and optimizer factory\nalgorithm = DQN(\n policy=policy,\n optim=AdamOptimizerFactory(lr=lr),\n gamma=gamma,\n n_step_return_horizon=n_step,\n target_update_freq=target_freq\n)\n```\n\nSet up the collectors:\n\n```python\ntraining_collector = ts.data.Collector[CollectStats](\n algorithm,\n training_envs,\n ts.data.VectorReplayBuffer(buffer_size, num_training_envs),\n exploration_noise=True,\n)\ntest_collector = ts.data.Collector[CollectStats](\n algorithm,\n test_envs,\n exploration_noise=True,\n) \n```\n\nLet's train the model using the algorithm:\n\n```python\nresult = algorithm.run_training(\n OffPolicyTrainerParams(\n training_collector=training_collector,\n test_collector=test_collector,\n max_epochs=epoch,\n epoch_num_steps=epoch_num_steps,\n collection_step_num_env_steps=collection_step_num_env_steps,\n test_step_num_episodes=num_test_envs,\n batch_size=batch_size,\n update_step_num_gradient_steps_per_sample=1 \u002F collection_step_num_env_steps,\n stop_fn=lambda mean_rewards: mean_rewards >= env.spec.reward_threshold,\n logger=logger,\n test_in_training=True,\n )\n)\nprint(f\"Finished training in {result.timing.total_time} seconds\")\n```\n\nThis is how you could manually save\u002Fload the trained policy (it's exactly the same as loading a `torch.nn.module`):\n\n```python\ntorch.save(policy.state_dict(), 'dqn.pth')\npolicy.load_state_dict(torch.load('dqn.pth'))\n```\n\nNow let's watch the agent with 35 FPS:\n\n```python\ncollector = ts.data.Collector(policy, env, exploration_noise=True)\ncollector.collect(n_episode=1, render=1 \u002F 35)\n```\n\nInspect the data saved in TensorBoard:\n\n```bash\n$ tensorboard --logdir log\u002Fdqn\n```\n\nPlease read the [documentation](https:\u002F\u002Ftianshou.readthedocs.io) for advanced usage.\n\n## Contributing\n\nTianshou is still under development.\nFurther algorithms and features are continuously being added, and we always welcome contributions to help make Tianshou better.\nIf you would like to contribute, please check out [this link](CONTRIBUTING.md).\n\n## Citing Tianshou\n\nIf you find Tianshou useful, please cite it in your publications.\n\n```latex\n@article{tianshou,\n author = {Jiayi Weng and Huayu Chen and Dong Yan and Kaichao You and Alexis Duburcq and Minghao Zhang and Yi Su and Hang Su and Jun Zhu},\n title = {Tianshou: A Highly Modularized Deep Reinforcement Learning Library},\n journal = {Journal of Machine Learning Research},\n year = {2022},\n volume = {23},\n number = {267},\n pages = {1--6},\n url = {http:\u002F\u002Fjmlr.org\u002Fpapers\u002Fv23\u002F21-1127.html}\n}\n```\n\n## Acknowledgments\n\nTianshou is supported by [appliedAI Institute for Europe](https:\u002F\u002Fwww.appliedai-institute.de\u002Fen\u002F),\na non-profit organization committed to providing long-term support and development.\n\nTianshou was previously a reinforcement learning platform based on TensorFlow. You can check out the branch [`priv`](https:\u002F\u002Fgithub.com\u002Fthu-ml\u002Ftianshou\u002Ftree\u002Fpriv) for more detail. Many thanks to [Haosheng Zou](https:\u002F\u002Fgithub.com\u002FHaoshengZou)'s pioneering work for Tianshou before version 0.1.1.\n\nWe would like to thank [TSAIL](http:\u002F\u002Fml.cs.tsinghua.edu.cn\u002F) and [Institute for Artificial Intelligence, Tsinghua University](http:\u002F\u002Fml.cs.tsinghua.edu.cn\u002Fthuai\u002F) for providing such an excellent AI research platform.\n","Tianshou 是一个基于 PyTorch 的深度强化学习库。它提供了灵活且类型安全的低级接口供算法开发者使用，同时也为应用开发者准备了便捷的高级接口来训练自定义环境中的已有算法。该库支持在线（包括on-policy和off-policy）和离线RL，并实验性地支持多智能体RL、基于模型的RL等。Tianshou 以其高性能、模块化设计以及友好的用户界面而著称，非常适合需要快速原型开发或研究新算法的研究人员及工程师在各种RL应用场景中使用。",2,"2026-06-11 03:34:13","high_star"]