[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-71086":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":15,"stars7d":17,"stars30d":18,"stars90d":16,"forks30d":16,"starsTrendScore":19,"compositeScore":20,"rankGlobal":10,"rankLanguage":10,"license":21,"archived":22,"fork":22,"defaultBranch":23,"hasWiki":22,"hasPages":22,"topics":24,"createdAt":10,"pushedAt":10,"updatedAt":44,"readmeContent":45,"aiSummary":46,"trendingCount":16,"starSnapshotCount":16,"syncStatus":47,"lastSyncTime":48,"discoverSource":49},71086,"nvitop","XuehaiPan\u002Fnvitop","XuehaiPan","An interactive NVIDIA-GPU process viewer and beyond, the one-stop solution for GPU process management.","https:\u002F\u002Fnvitop.readthedocs.io",null,"Python",6951,235,28,12,0,21,63,36,99.92,"Apache License 2.0",false,"main",[25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43],"command-line-tool","console","cuda","curses","gpu","gpu-monitoring","grafana","grafana-dashboard","htop","monitoring","monitoring-tool","nvidia","nvidia-smi","nvml","process-monitoring","prometheus","prometheus-exporter","resource-monitor","top","2026-06-12 04:00:58","# nvitop\n\n\u003C!-- markdownlint-disable html -->\n\n![Python 3.8+](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FPython-3.8%2B-brightgreen)\n[![PyPI](https:\u002F\u002Fimg.shields.io\u002Fpypi\u002Fv\u002Fnvitop?label=pypi&logo=pypi)](https:\u002F\u002Fpypi.org\u002Fproject\u002Fnvitop)\n[![conda-forge](https:\u002F\u002Fimg.shields.io\u002Fconda\u002Fvn\u002Fconda-forge\u002Fnvitop?label=conda&logo=condaforge)](https:\u002F\u002Fanaconda.org\u002Fconda-forge\u002Fnvitop)\n[![Documentation Status](https:\u002F\u002Fimg.shields.io\u002Freadthedocs\u002Fnvitop?label=docs&logo=readthedocs)](https:\u002F\u002Fnvitop.readthedocs.io)\n[![Downloads](https:\u002F\u002Fstatic.pepy.tech\u002Fpersonalized-badge\u002Fnvitop?period=total&left_color=gray&right_color=blue&left_text=downloads)](https:\u002F\u002Fpepy.tech\u002Fproject\u002Fnvitop)\n[![GitHub Repo Stars](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Fstars\u002FXuehaiPan\u002Fnvitop?label=stars&logo=github&color=brightgreen)](https:\u002F\u002Fgithub.com\u002FXuehaiPan\u002Fnvitop\u002Fstargazers)\n[![License](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Flicense\u002FXuehaiPan\u002Fnvitop?label=license&logo=data:image\u002Fsvg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAyNCAyNCIgd2lkdGg9IjI0IiBoZWlnaHQ9IjI0IiBmaWxsPSIjZmZmZmZmIj48cGF0aCBmaWxsLXJ1bGU9ImV2ZW5vZGQiIGQ9Ik0xMi43NSAyLjc1YS43NS43NSAwIDAwLTEuNSAwVjQuNUg5LjI3NmExLjc1IDEuNzUgMCAwMC0uOTg1LjMwM0w2LjU5NiA1Ljk1N0EuMjUuMjUgMCAwMTYuNDU1IDZIMi4zNTNhLjc1Ljc1IDAgMTAwIDEuNUgzLjkzTC41NjMgMTUuMThhLjc2Mi43NjIgMCAwMC4yMS44OGMuMDguMDY0LjE2MS4xMjUuMzA5LjIyMS4xODYuMTIxLjQ1Mi4yNzguNzkyLjQzMy42OC4zMTEgMS42NjIuNjIgMi44NzYuNjJhNi45MTkgNi45MTkgMCAwMDIuODc2LS42MmMuMzQtLjE1NS42MDYtLjMxMi43OTItLjQzMy4xNS0uMDk3LjIzLS4xNTguMzEtLjIyM2EuNzUuNzUgMCAwMC4yMDktLjg3OEw1LjU2OSA3LjVoLjg4NmMuMzUxIDAgLjY5NC0uMTA2Ljk4NC0uMzAzbDEuNjk2LTEuMTU0QS4yNS4yNSAwIDAxOS4yNzUgNmgxLjk3NXYxNC41SDYuNzYzYS43NS43NSAwIDAwMCAxLjVoMTAuNDc0YS43NS43NSAwIDAwMC0xLjVIMTIuNzVWNmgxLjk3NGMuMDUgMCAuMS4wMTUuMTQuMDQzbDEuNjk3IDEuMTU0Yy4yOS4xOTcuNjMzLjMwMy45ODQuMzAzaC44ODZsLTMuMzY4IDcuNjhhLjc1Ljc1IDAgMDAuMjMuODk2Yy4wMTIuMDA5IDAgMCAuMDAyIDBhMy4xNTQgMy4xNTQgMCAwMC4zMS4yMDZjLjE4NS4xMTIuNDUuMjU2Ljc5LjRhNy4zNDMgNy4zNDMgMCAwMDIuODU1LjU2OCA3LjM0MyA3LjM0MyAwIDAwMi44NTYtLjU2OWMuMzM4LS4xNDMuNjA0LS4yODcuNzktLjM5OWEzLjUgMy41IDAgMDAuMzEtLjIwNi43NS43NSAwIDAwLjIzLS44OTZMMjAuMDcgNy41aDEuNTc4YS43NS43NSAwIDAwMC0xLjVoLTQuMTAyYS4yNS4yNSAwIDAxLS4xNC0uMDQzbC0xLjY5Ny0xLjE1NGExLjc1IDEuNzUgMCAwMC0uOTg0LS4zMDNIMTIuNzVWMi43NXpNMi4xOTMgMTUuMTk4YTUuNDE4IDUuNDE4IDAgMDAyLjU1Ny42MzUgNS40MTggNS40MTggMCAwMDIuNTU3LS42MzVMNC43NSA5LjM2OGwtMi41NTcgNS44M3ptMTQuNTEtLjAyNGMuMDgyLjA0LjE3NC4wODMuMjc1LjEyNi41My4yMjMgMS4zMDUuNDUgMi4yNzIuNDVhNS44NDYgNS44NDYgMCAwMDIuNTQ3LS41NzZMMTkuMjUgOS4zNjdsLTIuNTQ3IDUuODA3eiI+PC9wYXRoPjwvc3ZnPgo=)](#license)\n\nAn interactive NVIDIA-GPU process viewer and beyond, the one-stop solution for GPU process management. The full API references host at \u003Chttps:\u002F\u002Fnvitop.readthedocs.io>.\n\n\u003Cp align=\"center\">\n  \u003Cimg width=\"100%\" src=\"https:\u002F\u002Fuser-images.githubusercontent.com\u002F16078332\u002F171005261-1aad126e-dc27-4ed3-a89b-7f9c1c998bf7.png\" alt=\"Monitor\">\n  \u003Cbr\u002F>\n  Monitor mode of \u003Ccode>nvitop\u003C\u002Fcode>.\n  \u003Cbr\u002F>\n  (TERM: GNOME Terminal \u002F OS: Ubuntu 16.04 LTS (over SSH) \u002F Locale: \u003Ccode>en_US.UTF-8\u003C\u002Fcode>)\n\u003C\u002Fp>\n\n\u003Cp align=\"center\">\n  \u003Ca href=\".\u002Fnvitop-exporter\">\n    \u003Cimg width=\"100%\" src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002Fe4867e64-2ca9-45bc-b524-929053f9673d\" alt=\"Grafana Dashboard\">\n  \u003C\u002Fa>\n  \u003Cbr\u002F>\n  A Grafana dashboard built on top of \u003Ccode>nvitop-exporter\u003C\u002Fcode>.\n\u003C\u002Fp>\n\n### Table of Contents  \u003C!-- omit in toc --> \u003C!-- markdownlint-disable heading-increment -->\n\n- [Features](#features)\n- [Requirements](#requirements)\n- [Installation](#installation)\n- [Usage](#usage)\n  - [Device and Process Status](#device-and-process-status)\n  - [Resource Monitor](#resource-monitor)\n    - [For Docker Users](#for-docker-users)\n    - [For SSH Users](#for-ssh-users)\n    - [Command Line Options and Environment Variables](#command-line-options-and-environment-variables)\n    - [Keybindings for Monitor Mode](#keybindings-for-monitor-mode)\n  - [CUDA Visible Devices Selection Tool](#cuda-visible-devices-selection-tool)\n  - [More than a Monitor](#more-than-a-monitor)\n    - [Quick Start](#quick-start)\n    - [Status Snapshot](#status-snapshot)\n    - [Resource Metric Collector](#resource-metric-collector)\n    - [Low-level APIs](#low-level-apis)\n      - [Device](#device)\n      - [Process](#process)\n      - [Host (inherited from psutil)](#host-inherited-from-psutil)\n- [Screenshots](#screenshots)\n- [Changelog](#changelog)\n- [License](#license)\n  - [Copyright Notice](#copyright-notice)\n\n------\n\n`nvitop` is an interactive NVIDIA device and process monitoring tool. It has a colorful and informative interface that continuously updates the status of the devices and processes. As a resource monitor, it includes many features and options, such as tree-view, environment variable viewing, process filtering, process metrics monitoring, etc. Beyond that, the package also ships a [CUDA device selection tool `nvisel`](#cuda-visible-devices-selection-tool) for deep learning researchers. It also provides handy APIs that allow developers to write their own monitoring tools. Please refer to section [More than a Monitor](#more-than-a-monitor) and the full API references at \u003Chttps:\u002F\u002Fnvitop.readthedocs.io> for more information.\n\n\u003Cp align=\"center\">\n  \u003Cimg width=\"100%\" src=\"https:\u002F\u002Fuser-images.githubusercontent.com\u002F16078332\u002F202362811-34f2c01d-97c8-49d2-b19b-0d7da648f2d5.png\" alt=\"Filter\">\n  \u003Cbr\u002F>\n  Process filtering and a more colorful interface.\n\u003C\u002Fp>\n\n\u003Cp align=\"center\">\n  \u003Cimg width=\"100%\" src=\"https:\u002F\u002Fuser-images.githubusercontent.com\u002F16078332\u002F202362686-859bf4ad-6237-46ca-b2f7-f547d2f63213.png\" alt=\"Comparison\">\n  \u003Cbr\u002F>\n  Compare to \u003Ccode>nvidia-smi\u003C\u002Fcode>.\n\u003C\u002Fp>\n\n------\n\n## Features\n\n- **Informative and fancy output**: show more information than `nvidia-smi` with colorized fancy box drawing.\n- **Monitor mode**: can run as a resource monitor, rather than print the results only once.\n  - bar charts and history graphs\n  - process sorting\n  - process filtering\n  - send signals to processes with a keystroke\n  - tree-view screen for GPU processes and their parent processes\n  - environment variable screen\n  - help screen\n  - mouse support\n- **Interactive**: responsive for user input (from keyboard and\u002For mouse) in monitor mode. (vs. [gpustat](https:\u002F\u002Fgithub.com\u002Fwookayin\u002Fgpustat) & [py3nvml](https:\u002F\u002Fgithub.com\u002Ffbcotter\u002Fpy3nvml))\n- **Efficient**:\n  - query device status using [*NVML Python bindings*](https:\u002F\u002Fpypi.org\u002Fproject\u002Fnvidia-ml-py) directly, instead of parsing the output of `nvidia-smi`. (vs. [nvidia-htop](https:\u002F\u002Fgithub.com\u002Fpeci1\u002Fnvidia-htop))\n  - support sparse query and cache results with `TTLCache` from [cachetools](https:\u002F\u002Fgithub.com\u002Ftkem\u002Fcachetools). (vs. [gpustat](https:\u002F\u002Fgithub.com\u002Fwookayin\u002Fgpustat))\n  - display information using the `curses` library rather than `print` with ANSI escape codes. (vs. [py3nvml](https:\u002F\u002Fgithub.com\u002Ffbcotter\u002Fpy3nvml))\n  - asynchronously gather information using multi-threading and correspond to user input much faster. (vs. [nvtop](https:\u002F\u002Fgithub.com\u002FSyllo\u002Fnvtop))\n- **Portable**: work on both Linux and Windows.\n  - get host process information using the cross-platform library [psutil](https:\u002F\u002Fgithub.com\u002Fgiampaolo\u002Fpsutil) instead of calling `ps -p \u003Cpid>` in a subprocess. (vs. [nvidia-htop](https:\u002F\u002Fgithub.com\u002Fpeci1\u002Fnvidia-htop) & [py3nvml](https:\u002F\u002Fgithub.com\u002Ffbcotter\u002Fpy3nvml))\n  - written in pure Python, easy to install with `pip`. (vs. [nvtop](https:\u002F\u002Fgithub.com\u002FSyllo\u002Fnvtop))\n- **Integrable**: easy to integrate into other applications, more than monitoring. (vs. [nvidia-htop](https:\u002F\u002Fgithub.com\u002Fpeci1\u002Fnvidia-htop) & [nvtop](https:\u002F\u002Fgithub.com\u002FSyllo\u002Fnvtop))\n\n\u003Cp align=\"center\">\n  \u003Cimg width=\"100%\" src=\"https:\u002F\u002Fuser-images.githubusercontent.com\u002F16078332\u002F129374533-fe06c01a-630d-4994-b54b-821cccd0d33c.png\" alt=\"Windows\">\n  \u003Cbr\u002F>\n  \u003Ccode>nvitop\u003C\u002Fcode> supports Windows!\n  \u003Cbr\u002F>\n  (SHELL: PowerShell \u002F TERM: Windows Terminal \u002F OS: Windows 10 \u002F Locale: \u003Ccode>en-US\u003C\u002Fcode>)\n\u003C\u002Fp>\n\n------\n\n## Requirements\n\n- Python 3.8+\n- NVIDIA Management Library (NVML)\n- nvidia-ml-py\n- psutil\n- curses\u003Csup>[*](#curses)\u003C\u002Fsup> (with `libncursesw`)\n\n**NOTE:** The [NVIDIA Management Library (*NVML*)](https:\u002F\u002Fdeveloper.nvidia.com\u002Fnvidia-management-library-nvml) is a C-based programmatic interface for monitoring and managing various states. The runtime version of the NVML library ships with the NVIDIA display driver (available at [Download Drivers | NVIDIA](https:\u002F\u002Fwww.nvidia.com\u002FDownload\u002Findex.aspx)), or can be downloaded as part of the NVIDIA CUDA Toolkit (available at [CUDA Toolkit | NVIDIA Developer](https:\u002F\u002Fdeveloper.nvidia.com\u002Fcuda-downloads)). The lists of OS platforms and NVIDIA-GPUs supported by the NVML library can be found in the [NVML API Reference](https:\u002F\u002Fdocs.nvidia.com\u002Fdeploy\u002Fnvml-api\u002Fnvml-api-reference.html).\n\nThis repository contains a Bash script to install\u002Fupgrade the NVIDIA drivers for Ubuntu Linux. For example:\n\n```bash\ngit clone --depth=1 https:\u002F\u002Fgithub.com\u002FXuehaiPan\u002Fnvitop.git && cd nvitop\n\n# Change to tty3 console (required for desktop users with GUI (tty2))\n# Optional for SSH users\nsudo chvt 3  # or use keyboard shortcut: Ctrl-LeftAlt-F3\n\nbash install-nvidia-driver.sh --package=nvidia-driver-595  # install the R595 driver from ppa:graphics-drivers\nbash install-nvidia-driver.sh --latest                     # install the latest driver from ppa:graphics-drivers\nbash install-nvidia-driver.sh --latest --open              # install the latest open-kernel-module driver\n```\n\n\u003Cp align=\"center\">\n  \u003Cimg width=\"100%\" src=\"https:\u002F\u002Fuser-images.githubusercontent.com\u002F16078332\u002F174480112-e9a35edc-8f42-438e-a103-1d0ce998b381.png\" alt=\"install-nvidia-driver\">\n  \u003Cbr\u002F>\n  NVIDIA driver installer for Ubuntu Linux.\n\u003C\u002Fp>\n\nRun `bash install-nvidia-driver.sh --help` for more information.\n\n\u003Ca name=\"curses\">*\u003C\u002Fa> The `curses` library is a built-in module of Python on Unix-like systems, and it is supported by a third-party package called `windows-curses` on Windows using PDCurses. Inconsistent behavior of `nvitop` may occur on different terminal emulators on Windows, such as missing mouse support.\n\n------\n\n## Installation\n\n**It is highly recommended to install `nvitop` in an isolated virtual environment.** Simple installation and run via [`uvx`](https:\u002F\u002Fdocs.astral.sh\u002Fuv\u002Fguides\u002Ftools) (a.k.a. `uv tool run`) or [`pipx`](https:\u002F\u002Fpypa.github.io\u002Fpipx):\n\n```bash\nuvx nvitop\n# or\npipx run nvitop\n```\n\nYou can also set this command as an alias in your shell startup file, e.g.:\n\n```bash\n# For Bash\necho 'alias nvitop=\"uvx nvitop\"' >> ~\u002F.bashrc\n\n# For Zsh\necho 'alias nvitop=\"uvx nvitop\"' >> ~\u002F.zshrc\n\n# For Fish\nmkdir -p ~\u002F.config\u002Ffish\necho 'alias nvitop=\"uvx nvitop\"' >> ~\u002F.config\u002Ffish\u002Fconfig.fish\n\n# For PowerShell\nNew-Item -Path (Split-Path -Parent -Path $PROFILE.CurrentUserAllHosts) -ItemType Directory -Force\n'Function nvitop { uvx nvitop @Args }' >> $PROFILE.CurrentUserAllHosts\n```\n\nor\n\n```bash\n# For Bash\necho 'alias nvitop=\"pipx run nvitop\"' >> ~\u002F.bashrc\n\n# For Zsh\necho 'alias nvitop=\"pipx run nvitop\"' >> ~\u002F.zshrc\n\n# For Fish\nmkdir -p ~\u002F.config\u002Ffish\necho 'alias nvitop=\"pipx run nvitop\"' >> ~\u002F.config\u002Ffish\u002Fconfig.fish\n\n# For PowerShell\nNew-Item -Path (Split-Path -Parent -Path $PROFILE.CurrentUserAllHosts) -ItemType Directory -Force\n'Function nvitop { pipx run nvitop @Args }' >> $PROFILE.CurrentUserAllHosts\n```\n\nInstall from PyPI ([![PyPI](https:\u002F\u002Fimg.shields.io\u002Fpypi\u002Fv\u002Fnvitop?label=pypi&logo=pypi)](https:\u002F\u002Fpypi.org\u002Fproject\u002Fnvitop)):\n\n```bash\npip3 install --upgrade nvitop\n```\n\nInstall from conda-forge ([![conda-forge](https:\u002F\u002Fimg.shields.io\u002Fconda\u002Fv\u002Fconda-forge\u002Fnvitop?logo=condaforge)](https:\u002F\u002Fanaconda.org\u002Fconda-forge\u002Fnvitop)):\n\n```bash\nconda install -c conda-forge nvitop\n```\n\nInstall the latest version from GitHub (![Commit Count](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Fcommits-since\u002FXuehaiPan\u002Fnvitop\u002Fv1.7.0)):\n\n```bash\npip3 install --upgrade pip setuptools\npip3 install git+https:\u002F\u002Fgithub.com\u002FXuehaiPan\u002Fnvitop.git\n```\n\nOr, clone this repo and install manually:\n\n```bash\ngit clone --depth=1 https:\u002F\u002Fgithub.com\u002FXuehaiPan\u002Fnvitop.git && cd nvitop\npip3 install .\n```\n\n**NOTE:** If you encounter the *\"nvitop: command not found\"* error after installation, please check whether you have added the Python console script path (e.g., `\"${HOME}\u002F.local\u002Fbin\"`) to your `PATH` environment variable. Alternatively, you can use `python3 -m nvitop`.\n\n\u003Cp align=\"center\">\n  \u003Cimg width=\"100%\" src=\"https:\u002F\u002Fuser-images.githubusercontent.com\u002F16078332\u002F178963038-a5cd4eb5-02a8-4456-966f-d5ff04eb44d8.png\" alt=\"MIG Device Support\">\n  \u003Cbr\u002F>\n  MIG Device Support.\n  \u003Cbr\u002F>\n\u003C\u002Fp>\n\n------\n\n## Usage\n\n### Device and Process Status\n\nQuery the device and process status. The output is similar to `nvidia-smi`, but has been enriched and colorized.\n\n```bash\n# Query the status of all devices\n$ nvitop -1  # or use `python3 -m nvitop -1`\n\n# Specify query devices (by integer indices)\n$ nvitop -1 -o 0 1  # only show \u003CGPU 0> and \u003CGPU 1>\n\n# Only show devices in `CUDA_VISIBLE_DEVICES` (by integer indices or UUID strings)\n$ nvitop -1 -ov\n\n# Only show GPU processes with the compute context (type: 'C' or 'C+G')\n$ nvitop -1 -c\n```\n\nWhen the `-1` switch is on, the result will be displayed **ONLY ONCE** (same as the default behavior of `nvidia-smi`). This is much faster and has lower resource usage. See [Command Line Options](#command-line-options-and-environment-variables) for more command options.\n\nThere is also a CLI tool called `nvisel` that ships with the `nvitop` PyPI package. See [CUDA Visible Devices Selection Tool](#cuda-visible-devices-selection-tool) for more information.\n\n### Resource Monitor\n\nRun as a resource monitor:\n\n```bash\n# Monitor mode (when the display mode is omitted, `NVITOP_MONITOR_MODE` will be used)\n$ nvitop  # or use `python3 -m nvitop`\n\n# Automatically configure the display mode according to the terminal size\n$ nvitop -m auto     # shortcut: `a` key\n\n# Arbitrarily display as `full` mode\n$ nvitop -m full     # shortcut: `f` key\n\n# Arbitrarily display as `compact` mode\n$ nvitop -m compact  # shortcut: `c` key\n\n# Specify query devices (by integer indices)\n$ nvitop -o 0 1  # only show \u003CGPU 0> and \u003CGPU 1>\n\n# Only show devices in `CUDA_VISIBLE_DEVICES` (by integer indices or UUID strings)\n$ nvitop -ov\n\n# Only show GPU processes with the compute context (type: 'C' or 'C+G')\n$ nvitop -c\n\n# Use ASCII characters only\n$ nvitop -U  # useful for terminals without Unicode support\n\n# For light terminals\n$ nvitop --light\n\n# For spectrum-like bar charts (requires the terminal supports 256-color)\n$ nvitop --colorful\n```\n\nYou can configure the default monitor mode with the `NVITOP_MONITOR_MODE` environment variable (default `auto` if not set). See [Command Line Options and Environment Variables](#command-line-options-and-environment-variables) for more command options.\n\nIn monitor mode, you can use \u003Ckbd>Ctrl-c\u003C\u002Fkbd> \u002F \u003Ckbd>T\u003C\u002Fkbd> \u002F \u003Ckbd>K\u003C\u002Fkbd> keys to interrupt \u002F terminate \u002F kill a process. And it's recommended to *terminate* or *kill* a process in the **tree-view screen** (shortcut: \u003Ckbd>t\u003C\u002Fkbd>). For normal users, `nvitop` will shallow other users' processes (in low-intensity colors). For **system administrators**, you can use `sudo nvitop` to terminate other users' processes.\n\nTo run `nvitop` as a viewer only and disable all process-mutating shortcuts, pass `--readonly` (or set `NVITOP_MONITOR_MODE=\"readonly\"`). The signal keys above become no-ops, the on-screen \"Press ^C(INT)\u002FT(TERM)\u002FK(KILL) to send signals\" hint is hidden, and the corresponding rows in the help screen are dimmed. Use this when sharing a session over SSH, demoing on a multi-tenant box, or wrapping `nvitop` in a non-admin alias.\n\nAlso, to enter the process metrics screen, select a process and then press the \u003Ckbd>Enter\u003C\u002Fkbd> \u002F \u003Ckbd>Return\u003C\u002Fkbd> key . `nvitop` dynamically displays the process metrics with live graphs.\n\n\u003Cp align=\"center\">\n  \u003Cimg width=\"100%\" src=\"https:\u002F\u002Fuser-images.githubusercontent.com\u002F16078332\u002F192108815-37c03705-be44-47d4-9908-6d05175db230.png\" alt=\"Process Metrics Screen\">\n  \u003Cbr\u002F>\n  Watch metrics for a specific process (shortcut: \u003Ckbd>Enter\u003C\u002Fkbd> \u002F \u003Ckbd>Return\u003C\u002Fkbd>).\n\u003C\u002Fp>\n\nPress \u003Ckbd>h\u003C\u002Fkbd> for help or \u003Ckbd>q\u003C\u002Fkbd> to return to the terminal. See [Keybindings for Monitor Mode](#keybindings-for-monitor-mode) for more shortcuts.\n\n\u003Cp align=\"center\">\n  \u003Cimg width=\"100%\" src=\"https:\u002F\u002Fuser-images.githubusercontent.com\u002F16078332\u002F192108664-61f1983c-6f62-48e6-87c5-29633d9c409e.png\" alt=\"Help Screen\">\n  \u003Cbr\u002F>\n  \u003Ccode>nvitop\u003C\u002Fcode> comes with a help screen (shortcut: \u003Ckbd>h\u003C\u002Fkbd>).\n\u003C\u002Fp>\n\n#### For Docker Users\n\nBuild and run the Docker image with [nvidia-container-toolkit](https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fnvidia-container-toolkit):\n\n```bash\ndocker run -it --rm --runtime=nvidia --gpus=all --pid=host ghcr.io\u002Fxuehaipan\u002Fnvitop:latest\n```\n\n**NOTE:** Don't forget to add the `--pid=host` option when running the container.\n\nIf you only need to set up the Grafana dashboard, you can start a dashboard at [`http:\u002F\u002Flocalhost:3000`](http:\u002F\u002Flocalhost:3000) with the following command:\n\n```bash\ndocker compose --project-directory=nvitop-exporter\u002Fgrafana up --build --detach\n```\n\nSee [`nvitop-exporter`](.\u002Fnvitop-exporter\u002FREADME.md) for more details.\n\n#### For SSH Users\n\nRun `nvitop` directly on the SSH session instead of a login shell:\n\n```bash\nssh user@host -t nvitop                 # installed by `sudo pip3 install ...`\nssh user@host -t '~\u002F.local\u002Fbin\u002Fnvitop'  # installed by `pip3 install --user ...`\n```\n\n**NOTE:** Users need to add the `-t` option to allocate a pseudo-terminal over the SSH session for monitor mode.\n\n#### Command Line Options and Environment Variables\n\nType `nvitop --help` for more command options:\n\n```text\nusage: nvitop [--help] [--version] [--once | --monitor [{auto,full,compact}]]\n              [--interval SEC] [--no-unicode] [--readonly] [--colorful]\n              [--force-color] [--light] [--gpu-util-thresh th1 th2]\n              [--mem-util-thresh th1 th2] [--only INDEX [INDEX ...]]\n              [--only-visible] [--compute] [--only-compute] [--graphics]\n              [--only-graphics] [--user [USERNAME ...]] [--pid PID [PID ...]]\n\nAn interactive NVIDIA-GPU process viewer.\n\noptions:\n  --help, -h            Show this help message and exit.\n  --version, -V         Show nvitop's version number and exit.\n  --once, -1            Report query data only once.\n  --monitor, -m [{auto,full,compact}]\n                        Run as a resource monitor. Continuously report query data and handle user inputs.\n                        If the argument is omitted, the value from `NVITOP_MONITOR_MODE` will be used.\n                        (default fallback mode: auto)\n  --interval SEC        Process status update interval in seconds. (default: 2)\n  --no-unicode, --ascii, -U\n                        Use ASCII characters only, which is useful for terminals without Unicode support.\n  --readonly            Disable all system and process changing features (e.g., terminating processes).\n                        Set variable `NVITOP_MONITOR_MODE=\"readonly\"` for convenience.\n\ncoloring:\n  --colorful            Use gradient colors to get spectrum-like bar charts.\n                        Set variable `NVITOP_MONITOR_MODE=\"colorful\"` for convenience.\n                        This option is only available when the terminal supports 256 colors.\n                        You may need to set environment variable `TERM=\"xterm-256color\"`. Note that the\n                        terminal multiplexer, such as `tmux`, may override the `TERM` variable.\n  --force-color         Force colorize even when `stdout` is not a TTY terminal.\n  --light               Tweak visual results for light theme terminals in monitor mode.\n                        Set variable `NVITOP_MONITOR_MODE=\"light\"` on light terminals for convenience.\n  --gpu-util-thresh th1 th2\n                        Thresholds of GPU utilization to determine the load intensity.\n                        Coloring rules: light \u003C th1 % \u003C= moderate \u003C th2 % \u003C= heavy.\n                        ( 1 \u003C= th1 \u003C th2 \u003C= 99, defaults: 10 75 )\n  --mem-util-thresh th1 th2\n                        Thresholds of GPU memory percent to determine the load intensity.\n                        Coloring rules: light \u003C th1 % \u003C= moderate \u003C th2 % \u003C= heavy.\n                        ( 1 \u003C= th1 \u003C th2 \u003C= 99, defaults: 10 80 )\n\ndevice filtering:\n  --only, -o INDEX [INDEX ...]\n                        Only show the specified devices, suppress option `--only-visible`.\n  --only-visible, -ov   Only show devices in the `CUDA_VISIBLE_DEVICES` environment variable.\n\nprocess filtering:\n  --compute, -c         Only show GPU processes with the compute context. (type: 'C' or 'C+G')\n  --only-compute, -C    Only show GPU processes exactly with the compute context. (type: 'C' only)\n  --graphics, -g        Only show GPU processes with the graphics context. (type: 'G' or 'C+G')\n  --only-graphics, -G   Only show GPU processes exactly with the graphics context. (type: 'G' only)\n  --user, -u [USERNAME ...]\n                        Only show processes of the given users (or `$USER` for no argument).\n  --pid, -p PID [PID ...]\n                        Only show processes of the given PIDs.\n```\n\n`nvitop` can accept the following environment variables for monitor mode:\n\n| Name                                   | Description                                         | Valid Values                                                                                                                | Default Value     |\n| -------------------------------------- | --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | ----------------- |\n| `NVITOP_MONITOR_MODE`                  | The default display mode (a comma-separated string) | `auto` \u002F `full` \u002F `compact`\u003Cbr>`plain` \u002F `colorful`\u003Cbr>`dark` \u002F `light`\u003Cbr>`readonly` (disables process-mutating shortcuts) | `auto,plain,dark` |\n| `NVITOP_GPU_UTILIZATION_THRESHOLDS`    | Thresholds of GPU utilization                       | `10,75` , `1,99`, ...                                                                                                       | `10,75`           |\n| `NVITOP_MEMORY_UTILIZATION_THRESHOLDS` | Thresholds of GPU memory percent                    | `10,80` , `1,99`, ...                                                                                                       | `10,80`           |\n| `LOGLEVEL`                             | Log level for log messages                          | `DEBUG` , `INFO`, `WARNING`, ...                                                                                            | `WARNING`         |\n\nFor example:\n\n```bash\n# Replace the following export statements if you are not using Bash \u002F Zsh\nexport NVITOP_MONITOR_MODE=\"full,light\"\n\n# Full monitor mode with light terminal tweaks\nnvitop\n```\n\nFor convenience, you can add these environment variables to your shell startup file, e.g.:\n\n```bash\n# For Bash\necho 'export NVITOP_MONITOR_MODE=\"full\"' >> ~\u002F.bashrc\n\n# For Zsh\necho 'export NVITOP_MONITOR_MODE=\"full\"' >> ~\u002F.zshrc\n\n# For Fish\necho 'set -gx NVITOP_MONITOR_MODE \"full\"' >> ~\u002F.config\u002Ffish\u002Fconfig.fish\n\n# For PowerShell\n'$Env:NVITOP_MONITOR_MODE = \"full\"' >> $PROFILE.CurrentUserAllHosts\n```\n\n#### Keybindings for Monitor Mode\n\n|                                                                        Key | Binding                                                                                    |\n| -------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------- |\n|                                                                        `q` | Quit and return to the terminal.                                                           |\n|                                                                  `h` \u002F `?` | Go to the help screen.                                                                     |\n|                                                            `a` \u002F `f` \u002F `c` | Change the display mode to *auto* \u002F *full* \u002F *compact*.                                    |\n|                                                     `r` \u002F `\u003CC-r>` \u002F `\u003CF5>` | Force refresh the window.                                                                  |\n|                                                                            |                                                                                            |\n| `\u003CUp>` \u002F `\u003CDown>`\u003Cbr>`\u003CA-k>` \u002F `\u003CA-j>`\u003Cbr>`\u003CTab>` \u002F `\u003CS-Tab>`\u003Cbr>`\u003CWheel>` | Select and highlight a process.                                                            |\n|                   `\u003CLeft>` \u002F `\u003CRight>`\u003Cbr>`\u003CA-h>` \u002F `\u003CA-l>`\u003Cbr>`\u003CS-Wheel>` | Scroll the host information of processes.                                                  |\n|                                                                   `\u003CHome>` | Select the first process.                                                                  |\n|                                                                    `\u003CEnd>` | Select the last process.                                                                   |\n|                                                             `\u003CC-a>`\u003Cbr>`^` | Scroll left to the beginning of the process entry (i.e. beginning of line).                |\n|                                                             `\u003CC-e>`\u003Cbr>`$` | Scroll right to the end of the process entry (i.e. end of line).                           |\n|              `\u003CPageUp>` \u002F `\u003CPageDown>`\u003Cbr\u002F> `\u003CA-K>` \u002F `\u003CA-J>`\u003Cbr>`[` \u002F `]` | scroll entire screen (for large amounts of processes).                                     |\n|                                                                            |                                                                                            |\n|                                                                  `\u003CSpace>` | Tag\u002Funtag current process.                                                                 |\n|                                                                    `\u003CEsc>` | Clear process selection.                                                                   |\n|                                                             `\u003CC-c>`\u003Cbr>`I` | Send `signal.SIGINT` to the selected process (interrupt). *(disabled under `--readonly`)*  |\n|                                                                        `T` | Send `signal.SIGTERM` to the selected process (terminate). *(disabled under `--readonly`)* |\n|                                                                        `K` | Send `signal.SIGKILL` to the selected process (kill). *(disabled under `--readonly`)*      |\n|                                                                            |                                                                                            |\n|                                                                        `e` | Show process environment.                                                                  |\n|                                                                        `t` | Toggle tree-view screen.                                                                   |\n|                                                                  `\u003CEnter>` | Show process metrics.                                                                      |\n|                                                                            |                                                                                            |\n|                                                                  `,` \u002F `.` | Select the sort column.                                                                    |\n|                                                                        `\u002F` | Reverse the sort order.                                                                    |\n|                                                                `on` (`oN`) | Sort processes in the natural order, i.e., in ascending (descending) order of `GPU`.       |\n|                                                                `ou` (`oU`) | Sort processes by `USER` in ascending (descending) order.                                  |\n|                                                                `op` (`oP`) | Sort processes by `PID` in descending (ascending) order.                                   |\n|                                                                `og` (`oG`) | Sort processes by `GPU-MEM` in descending (ascending) order.                               |\n|                                                                `os` (`oS`) | Sort processes by `%SM` in descending (ascending) order.                                   |\n|                                                                `oc` (`oC`) | Sort processes by `%CPU` in descending (ascending) order.                                  |\n|                                                                `om` (`oM`) | Sort processes by `%MEM` in descending (ascending) order.                                  |\n|                                                                `ot` (`oT`) | Sort processes by `TIME` in descending (ascending) order.                                  |\n\n**HINT:** It's recommended to terminate or kill a process in the tree-view screen (shortcut: \u003Ckbd>t\u003C\u002Fkbd>).\n\n------\n\n### CUDA Visible Devices Selection Tool\n\nAutomatically select `CUDA_VISIBLE_DEVICES` from the given criteria. Example usage of the CLI tool:\n\n```console\n# All devices but sorted\n$ nvisel       # or use `python3 -m nvitop.select`\n6,5,4,3,2,1,0,7,8\n\n# A simple example to select 4 devices\n$ nvisel -n 4  # or use `python3 -m nvitop.select -n 4`\n6,5,4,3\n\n# Select available devices that satisfy the given constraints\n$ nvisel --min-count 2 --max-count 3 --min-free-memory 5GiB --max-gpu-utilization 60\n6,5,4\n\n# Set `CUDA_VISIBLE_DEVICES` environment variable using `nvisel`\n$ export CUDA_DEVICE_ORDER=\"PCI_BUS_ID\" CUDA_VISIBLE_DEVICES=\"$(nvisel -c 1 -f 10GiB)\"\nCUDA_VISIBLE_DEVICES=\"6,5,4,3,2,1,0\"\n\n# Use UUID strings in `CUDA_VISIBLE_DEVICES` environment variable\n$ export CUDA_VISIBLE_DEVICES=\"$(nvisel -O uuid -c 2 -f 5000M)\"\nCUDA_VISIBLE_DEVICES=\"GPU-849d5a8d-610e-eeea-1fd4-81ff44a23794,GPU-18ef14e9-dec6-1d7e-1284-3010c6ce98b1,GPU-96de99c9-d68f-84c8-424c-7c75e59cc0a0,GPU-2428d171-8684-5b64-830c-435cd972ec4a,GPU-6d2a57c9-7783-44bb-9f53-13f36282830a,GPU-f8e5a624-2c7e-417c-e647-b764d26d4733,GPU-f9ca790e-683e-3d56-00ba-8f654e977e02\"\n\n# Pipe output to other shell utilities\n$ nvisel --newline -O uuid -C 6 -f 8GiB\nGPU-849d5a8d-610e-eeea-1fd4-81ff44a23794\nGPU-18ef14e9-dec6-1d7e-1284-3010c6ce98b1\nGPU-96de99c9-d68f-84c8-424c-7c75e59cc0a0\nGPU-2428d171-8684-5b64-830c-435cd972ec4a\nGPU-6d2a57c9-7783-44bb-9f53-13f36282830a\nGPU-f8e5a624-2c7e-417c-e647-b764d26d4733\n$ nvisel -0 -O uuid -c 2 -f 4GiB | xargs -0 -I {} nvidia-smi --id={} --query-gpu=index,memory.free --format=csv\nCUDA_VISIBLE_DEVICES=\"GPU-849d5a8d-610e-eeea-1fd4-81ff44a23794,GPU-18ef14e9-dec6-1d7e-1284-3010c6ce98b1,GPU-96de99c9-d68f-84c8-424c-7c75e59cc0a0,GPU-2428d171-8684-5b64-830c-435cd972ec4a,GPU-6d2a57c9-7783-44bb-9f53-13f36282830a,GPU-f8e5a624-2c7e-417c-e647-b764d26d4733,GPU-f9ca790e-683e-3d56-00ba-8f654e977e02\"\nindex, memory.free [MiB]\n6, 11018 MiB\nindex, memory.free [MiB]\n5, 11018 MiB\nindex, memory.free [MiB]\n4, 11018 MiB\nindex, memory.free [MiB]\n3, 11018 MiB\nindex, memory.free [MiB]\n2, 11018 MiB\nindex, memory.free [MiB]\n1, 11018 MiB\nindex, memory.free [MiB]\n0, 11018 MiB\n\n# Normalize the `CUDA_VISIBLE_DEVICES` environment variable (e.g. convert UUIDs to indices or get full UUIDs for an abbreviated form)\n$ nvisel -i \"GPU-18ef14e9,GPU-849d5a8d\" -S\n5,6\n$ nvisel -i \"GPU-18ef14e9,GPU-849d5a8d\" -S -O uuid --newline\nGPU-18ef14e9-dec6-1d7e-1284-3010c6ce98b1\nGPU-849d5a8d-610e-eeea-1fd4-81ff44a23794\n```\n\nYou can also integrate `nvisel` into your training script like this:\n\n```python\n# Put this at the top of the Python script\nimport os\nfrom nvitop import select_devices\n\nos.environ['CUDA_VISIBLE_DEVICES'] = ','.join(\n    select_devices(format='uuid', min_count=4, min_free_memory='8GiB')\n)\n```\n\nType `nvisel --help` for more command options:\n\n```text\nusage: nvisel [--help] [--version]\n              [--inherit [CUDA_VISIBLE_DEVICES]] [--account-as-free [USERNAME ...]]\n              [--min-count N] [--max-count N] [--count N]\n              [--min-free-memory SIZE] [--min-total-memory SIZE]\n              [--max-gpu-utilization RATE] [--max-memory-utilization RATE]\n              [--tolerance TOL]\n              [--format FORMAT] [--sep SEP | --newline | --null] [--no-sort]\n\nCUDA visible devices selection tool.\n\noptions:\n  --help, -h            Show this help message and exit.\n  --version, -V         Show nvisel's version number and exit.\n\nconstraints:\n  --inherit [CUDA_VISIBLE_DEVICES], -i [CUDA_VISIBLE_DEVICES]\n                        Inherit the given `CUDA_VISIBLE_DEVICES`. If the argument is omitted, use the\n                        value from the environment. This means selecting a subset of the currently\n                        CUDA-visible devices.\n  --account-as-free [USERNAME ...]\n                        Account the used GPU memory of the given users as free memory.\n                        If this option is specified but without argument, `$USER` will be used.\n  --min-count N, -c N   Minimum number of devices to select. (default: 0)\n                        The tool will fail (exit non-zero) if the requested resource is not available.\n  --max-count N, -C N   Maximum number of devices to select. (default: all devices)\n  --count N, -n N       Overriding both `--min-count N` and `--max-count N`.\n  --min-free-memory SIZE, -f SIZE\n                        Minimum free memory of devices to select. (example value: 4GiB)\n                        If this constraint is given, check against all devices.\n  --min-total-memory SIZE, -t SIZE\n                        Minimum total memory of devices to select. (example value: 10GiB)\n                        If this constraint is given, check against all devices.\n  --max-gpu-utilization RATE, -G RATE\n                        Maximum GPU utilization rate of devices to select. (example value: 30)\n                        If this constraint is given, check against all devices.\n  --max-memory-utilization RATE, -M RATE\n                        Maximum memory bandwidth utilization rate of devices to select. (example value: 50)\n                        If this constraint is given, check against all devices.\n  --tolerance TOL, --tol TOL\n                        The constraints tolerance (in percentage). (default: 0, i.e., strict)\n                        This option can loose the constraints if the requested resource is not available.\n                        For example, set `--tolerance=20` will accept a device with only 4GiB of free\n                        memory when set `--min-free-memory=5GiB`.\n\nformatting:\n  --format FORMAT, -O FORMAT\n                        The output format of the selected device identifiers. (default: index)\n                        If any MIG device found, the output format will be fallback to `uuid`.\n  --sep SEP, --separator SEP, -s SEP\n                        Separator for the output. (default: ',')\n  --newline             Use newline character as separator for the output, equivalent to `--sep=$'\\n'`.\n  --null, -0            Use null character ('\\x00') as separator for the output. This option corresponds\n                        to the `-0` option of `xargs`.\n  --no-sort, -S         Do not sort the device by memory usage and GPU utilization.\n```\n\n------\n\n### More than a Monitor\n\n`nvitop` can be easily integrated into other applications. You can use `nvitop` to make your own monitoring tools. The full API references host at \u003Chttps:\u002F\u002Fnvitop.readthedocs.io>. Runnable reference scripts live in [`examples\u002F`](.\u002Fexamples\u002F).\n\n#### Quick Start\n\nA minimal script to monitor the GPU devices based on APIs from `nvitop`:\n\n```python\nfrom nvitop import Device\n\ndevices = Device.all()  # or `Device.cuda.all()` to use CUDA ordinal instead\nfor device in devices:\n    processes = device.processes()  # type: Dict[int, GpuProcess]\n    sorted_pids = sorted(processes.keys())\n\n    print(device)\n    print(f'  - Fan speed:       {device.fan_speed()}%')\n    print(f'  - Temperature:     {device.temperature()}C')\n    print(f'  - GPU utilization: {device.gpu_utilization()}%')\n    print(f'  - Total memory:    {device.memory_total_human()}')\n    print(f'  - Used memory:     {device.memory_used_human()}')\n    print(f'  - Free memory:     {device.memory_free_human()}')\n    print(f'  - Processes ({len(processes)}): {sorted_pids}')\n    for pid in sorted_pids:\n        print(f'    - {processes[pid]}')\n    print('-' * 120)\n```\n\nAnother more advanced approach with coloring:\n\n```python\nimport time\n\nfrom nvitop import Device, GpuProcess, NA, colored\n\nprint(colored(time.strftime('%a %b %d %H:%M:%S %Y'), color='red', attrs=('bold',)))\n\ndevices = Device.cuda.all()  # or `Device.all()` to use NVML ordinal instead\nseparator = False\nfor device in devices:\n    processes = device.processes()  # type: Dict[int, GpuProcess]\n\n    print(colored(str(device), color='green', attrs=('bold',)))\n    print(colored('  - Fan speed:       ', color='blue', attrs=('bold',)) + f'{device.fan_speed()}%')\n    print(colored('  - Temperature:     ', color='blue', attrs=('bold',)) + f'{device.temperature()}C')\n    print(colored('  - GPU utilization: ', color='blue', attrs=('bold',)) + f'{device.gpu_utilization()}%')\n    print(colored('  - Total memory:    ', color='blue', attrs=('bold',)) + f'{device.memory_total_human()}')\n    print(colored('  - Used memory:     ', color='blue', attrs=('bold',)) + f'{device.memory_used_human()}')\n    print(colored('  - Free memory:     ', color='blue', attrs=('bold',)) + f'{device.memory_free_human()}')\n    if len(processes) > 0:\n        processes = GpuProcess.take_snapshots(processes.values(), failsafe=True)\n        processes.sort(key=lambda process: (process.username, process.pid))\n\n        print(colored(f'  - Processes ({len(processes)}):', color='blue', attrs=('bold',)))\n        fmt = '    {pid:\u003C5}  {username:\u003C8} {cpu:>5}  {host_memory:>8} {time:>8}  {gpu_memory:>8}  {sm:>3}  {command:\u003C}'.format\n        print(colored(fmt(pid='PID', username='USERNAME',\n                          cpu='CPU%', host_memory='HOST-MEM', time='TIME',\n                          gpu_memory='GPU-MEM', sm='SM%',\n                          command='COMMAND'),\n                      attrs=('bold',)))\n        for snapshot in processes:\n            print(fmt(pid=snapshot.pid,\n                      username=snapshot.username[:7] + ('+' if len(snapshot.username) > 8 else snapshot.username[7:8]),\n                      cpu=snapshot.cpu_percent, host_memory=snapshot.host_memory_human,\n                      time=snapshot.running_time_human,\n                      gpu_memory=(snapshot.gpu_memory_human if snapshot.gpu_memory_human is not NA else 'WDDM:N\u002FA'),\n                      sm=snapshot.gpu_sm_utilization,\n                      command=snapshot.command))\n    else:\n        print(colored('  - No Running Processes', attrs=('bold',)))\n\n    if separator:\n        print('-' * 120)\n    separator = True\n```\n\n\u003Cp align=\"center\">\n  \u003Cimg width=\"100%\" src=\"https:\u002F\u002Fuser-images.githubusercontent.com\u002F16078332\u002F177041142-fe988d58-6a97-4559-84fd-b51204cf9231.png\" alt=\"Demo\">\n  \u003Cbr\u002F>\n  An example monitoring script built with APIs from \u003Ccode>nvitop\u003C\u002Fcode>.\n\u003C\u002Fp>\n\n------\n\n#### Status Snapshot\n\n`nvitop` provides a helper function [`take_snapshots`](https:\u002F\u002Fnvitop.readthedocs.io\u002Fen\u002Flatest\u002Fapi\u002Fcollector.html#nvitop.take_snapshots) to retrieve the status of both GPU devices and GPU processes at once. You can type `help(nvitop.take_snapshots)` in Python REPL for detailed documentation.\n\n```python\nIn [1]: from nvitop import take_snapshots, Device\n   ...: import os\n   ...: os.environ['CUDA_DEVICE_ORDER'] = 'PCI_BUS_ID'\n   ...: os.environ['CUDA_VISIBLE_DEVICES'] = '1,0'  # comma-separated integers or UUID strings\n\nIn [2]: take_snapshots()  # equivalent to `take_snapshots(Device.all())`\nOut[2]:\nSnapshotResult(\n    devices=[\n        DeviceSnapshot(\n            real=Device(index=0, ...),\n            ...\n        ),\n        ...\n    ],\n    gpu_processes=[\n        GpuProcessSnapshot(\n            real=GpuProcess(pid=xxxxxx, device=Device(index=0, ...), ...),\n            ...\n        ),\n        ...\n    ]\n)\n\nIn [3]: device_snapshots, gpu_process_snapshots = take_snapshots(Device.all())  # type: Tuple[List[DeviceSnapshot], List[GpuProcessSnapshot]]\n\nIn [4]: device_snapshots, _ = take_snapshots(gpu_processes=False)  # ignore process snapshots\n\nIn [5]: take_snapshots(Device.cuda.all())  # use CUDA device enumeration\nOut[5]:\nSnapshotResult(\n    devices=[\n        CudaDeviceSnapshot(\n            real=CudaDevice(cuda_index=0, nvml_index=1, ...),\n            ...\n        ),\n        CudaDeviceSnapshot(\n            real=CudaDevice(cuda_index=1, nvml_index=0, ...),\n            ...\n        ),\n    ],\n    gpu_processes=[\n        GpuProcessSnapshot(\n            real=GpuProcess(pid=xxxxxx, device=CudaDevice(cuda_index=0, ...), ...),\n            ...\n        ),\n        ...\n    ]\n)\n\nIn [6]: take_snapshots(Device.cuda(1))  # \u003CCUDA 1> only\nOut[6]:\nSnapshotResult(\n    devices=[\n        CudaDeviceSnapshot(\n            real=CudaDevice(cuda_index=1, nvml_index=0, ...),\n            ...\n        )\n    ],\n    gpu_processes=[\n        GpuProcessSnapshot(\n            real=GpuProcess(pid=xxxxxx, device=CudaDevice(cuda_index=1, ...), ...),\n            ...\n        ),\n        ...\n    ]\n)\n```\n\nPlease refer to section [Low-level APIs](#low-level-apis) for more information.\n\n------\n\n#### Resource Metric Collector\n\n[`ResourceMetricCollector`](https:\u002F\u002Fnvitop.readthedocs.io\u002Fen\u002Flatest\u002Fapi\u002Fcollector.html#nvitop.ResourceMetricCollector) is a class that collects resource metrics for host, GPUs and processes running on the GPUs. All metrics will be collected in an asynchronous manner. You can type `help(nvitop.ResourceMetricCollector)` in Python REPL for detailed documentation.\n\n```python\nIn [1]: from nvitop import ResourceMetricCollector, Device\n   ...: import os\n   ...: os.environ['CUDA_DEVICE_ORDER'] = 'PCI_BUS_ID'\n   ...: os.environ['CUDA_VISIBLE_DEVICES'] = '3,2,1,0'  # comma-separated integers or UUID strings\n\nIn [2]: collector = ResourceMetricCollector()                                   # log all devices and descendant processes of the current process on the GPUs\nIn [3]: collector = ResourceMetricCollector(root_pids={1})                      # log all devices and all GPU processes\nIn [4]: collector = ResourceMetricCollector(devices=Device(0), root_pids={1})   # log \u003CGPU 0> and all GPU processes on \u003CGPU 0>\nIn [5]: collector = ResourceMetricCollector(devices=Device.cuda.all())          # use the CUDA ordinal\n\nIn [6]: with collector(tag='\u003Ctag>'):\n   ...:     # Do something\n   ...:     collector.collect()  # -> Dict[str, float]\n# key -> '\u003Ctag>\u002F\u003Cscope>\u002F\u003Cmetric (unit)>\u002F\u003Cmean\u002Fmin\u002Fmax>'\n{\n    '\u003Ctag>\u002Fhost\u002Fcpu_percent (%)\u002Fmean': 8.967849777683456,\n    '\u003Ctag>\u002Fhost\u002Fcpu_percent (%)\u002Fmin': 6.1,\n    '\u003Ctag>\u002Fhost\u002Fcpu_percent (%)\u002Fmax': 28.1,\n    ...,\n    '\u003Ctag>\u002Fhost\u002Fmemory_percent (%)\u002Fmean': 21.5,\n    '\u003Ctag>\u002Fhost\u002Fswap_percent (%)\u002Fmean': 0.3,\n    '\u003Ctag>\u002Fhost\u002Fmemory_used (GiB)\u002Fmean': 91.0136418208109,\n    '\u003Ctag>\u002Fhost\u002Fload_average (%) (1 min)\u002Fmean': 10.251427386878328,\n    '\u003Ctag>\u002Fhost\u002Fload_average (%) (5 min)\u002Fmean': 10.072539414569503,\n    '\u003Ctag>\u002Fhost\u002Fload_average (%) (15 min)\u002Fmean': 11.91126970422139,\n    ...,\n    '\u003Ctag>\u002Fcuda:0 (gpu:3)\u002Fmemory_used (MiB)\u002Fmean': 3.875,\n    '\u003Ctag>\u002Fcuda:0 (gpu:3)\u002Fmemory_free (MiB)\u002Fmean': 11015.562499999998,\n    '\u003Ctag>\u002Fcuda:0 (gpu:3)\u002Fmemory_total (MiB)\u002Fmean': 11019.437500000002,\n    '\u003Ctag>\u002Fcuda:0 (gpu:3)\u002Fmemory_percent (%)\u002Fmean': 0.0,\n    '\u003Ctag>\u002Fcuda:0 (gpu:3)\u002Fgpu_utilization (%)\u002Fmean': 0.0,\n    '\u003Ctag>\u002Fcuda:0 (gpu:3)\u002Fmemory_utilization (%)\u002Fmean': 0.0,\n    '\u003Ctag>\u002Fcuda:0 (gpu:3)\u002Ffan_speed (%)\u002Fmean': 22.0,\n    '\u003Ctag>\u002Fcuda:0 (gpu:3)\u002Ftemperature (C)\u002Fmean': 25.0,\n    '\u003Ctag>\u002Fcuda:0 (gpu:3)\u002Fpower_usage (W)\u002Fmean': 19.11166264116916,\n    ...,\n    '\u003Ctag>\u002Fcuda:1 (gpu:2)\u002Fmemory_used (MiB)\u002Fmean': 8878.875,\n    ...,\n    '\u003Ctag>\u002Fcuda:2 (gpu:1)\u002Fmemory_used (MiB)\u002Fmean': 8182.875,\n    ...,\n    '\u003Ctag>\u002Fcuda:3 (gpu:0)\u002Fmemory_used (MiB)\u002Fmean': 9286.875,\n    ...,\n    '\u003Ctag>\u002Fpid:12345\u002Fhost\u002Fcpu_percent (%)\u002Fmean': 151.34342772112265,\n    '\u003Ctag>\u002Fpid:12345\u002Fhost\u002Fhost_memory (MiB)\u002Fmean': 44749.72373447514,\n    '\u003Ctag>\u002Fpid:12345\u002Fhost\u002Fhost_memory_percent (%)\u002Fmean': 8.675082352111717,\n    '\u003Ctag>\u002Fpid:12345\u002Fhost\u002Frunning_time (min)': 336.23803206741576,\n    '\u003Ctag>\u002Fpid:12345\u002Fcuda:1 (gpu:4)\u002Fgpu_memory (MiB)\u002Fmean': 8861.0,\n    '\u003Ctag>\u002Fpid:12345\u002Fcuda:1 (gpu:4)\u002Fgpu_memory_percent (%)\u002Fmean': 80.4,\n    '\u003Ctag>\u002Fpid:12345\u002Fcuda:1 (gpu:4)\u002Fgpu_memory_utilization (%)\u002Fmean': 6.711118172407917,\n    '\u003Ctag>\u002Fpid:12345\u002Fcuda:1 (gpu:4)\u002Fgpu_sm_utilization (%)\u002Fmean': 48.23283397736476,\n    ...,\n    '\u003Ctag>\u002Fduration (s)': 7.247399162035435,\n    '\u003Ctag>\u002Ftimestamp': 1655909466.9981883\n}\n```\n\nThe results can be easily logged into [TensorBoard](https:\u002F\u002Fgithub.com\u002Ftensorflow\u002Ftensorboard) or a CSV file. For example:\n\n```python\nimport os\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.utils.tensorboard import SummaryWriter\n\nfrom nvitop import CudaDevice, ResourceMetricCollector\n\n\ndef add_scalar_dict(writer, main_tag, tag_scalar_dict, global_step=None, walltime=None):\n    for tag, scalar in tag_scalar_dict.items():\n        writer.add_scalar(f'{main_tag}\u002F{tag}', scalar, global_step=global_step, walltime=walltime)\n\n\n# Build networks and prepare datasets\n...\n\n# Logger and status collector\nwriter = SummaryWriter()\ncollector = ResourceMetricCollector(devices=CudaDevice.all(),  # log all visible CUDA devices and use the CUDA ordinal\n                                    root_pids={os.getpid()},   # only log the descendant processes of the current process\n                                    interval=1.0)              # snapshot interval for background daemon thread\n\n# Start training\nglobal_step = 0\nfor epoch in range(num_epoch):\n    with collector(tag='train'):\n        for batch in train_dataset:\n            with collector(tag='batch'):\n                metrics = train(net, batch)\n                global_step += 1\n                add_scalar_dict(writer, 'train', metrics, global_step=global_step)\n                add_scalar_dict(writer, 'resources',      # tag='resources\u002Ftrain\u002Fbatch\u002F...'\n                                collector.collect(),\n                                global_step=global_step)\n\n        add_scalar_dict(writer, 'resources',              # tag='resources\u002Ftrain\u002F...'\n                        collector.collect(),\n                        global_step=epoch)\n\n    with collector(tag='validate'):\n        metrics = validate(net, validation_dataset)\n        add_scalar_dict(writer, 'validate', metrics, global_step=epoch)\n        add_scalar_dict(writer, 'resources',              # tag='resources\u002Fvalidate\u002F...'\n                        collector.collect(),\n                        global_step=epoch)\n```\n\nAnother example for logging into a CSV file:\n\n```python\nimport datetime\nimport time\n\nimport pandas as pd\n\nfrom nvitop import ResourceMetricCollector\n\ncollector = ResourceMetricCollector(root_pids={1}, interval=2.0)  # log all devices and all GPU processes\ndf = pd.DataFrame()\n\nwith collector(tag='resources'):\n    for _ in range(60):\n        # Do something\n        time.sleep(60)\n\n        metrics = collector.collect()\n        df_metrics = pd.DataFrame.from_records(metrics, index=[len(df)])\n        df = pd.concat([df, df_metrics], ignore_index=True)\n        # Flush to CSV file ...\n\ndf.insert(0, 'time', df['resources\u002Ftimestamp'].map(datetime.datetime.fromtimestamp))\ndf.to_csv('results.csv', index=False)\n```\n\nYou can also daemonize the collector in the background using [`collect_in_background`](https:\u002F\u002Fnvitop.readthedocs.io\u002Fen\u002Flatest\u002Fapi\u002Fcollector.html#nvitop.collect_in_background) or [`ResourceMetricCollector.daemonize`](https:\u002F\u002Fnvitop.readthedocs.io\u002Fen\u002Flatest\u002Fapi\u002Fcollector.html#nvitop.ResourceMetricCollector.daemonize) with callback functions.\n\n```python\nfrom nvitop import Device, ResourceMetricCollector, collect_in_background\n\nlogger = ...\n\ndef on_collect(metrics):  # will be called periodically\n    if logger.is_closed():  # closed manually by user\n        return False\n    logger.log(metrics)\n    return True\n\ndef on_stop(collector):  # will be called only once at stop\n    if not logger.is_closed():\n        logger.close()  # cleanup\n\n# Record metrics to the logger in the background every 5 seconds.\n# It will collect 5-second mean\u002Fmin\u002Fmax for each metric.\ncollect_in_background(\n    on_collect,\n    ResourceMetricCollector(Device.cuda.all()),\n    interval=5.0,\n    on_stop=on_stop,\n)\n```\n\nor simply:\n\n```python\nResourceMetricCollector(Device.cuda.all()).daemonize(\n    on_collect,\n    interval=5.0,\n    on_stop=on_stop,\n)\n```\n\n------\n\n#### Low-level APIs\n\nThe full API references can be found at \u003Chttps:\u002F\u002Fnvitop.readthedocs.io>.\n\n##### Device\n\nThe [device module](https:\u002F\u002Fnvitop.readthedocs.io\u002Fen\u002Flatest\u002Fapi\u002Fdevice.html) provides:\n\n\u003Ctable class=\"autosummary longtable docutils align-default\">\n  \u003Ccolgroup>\n    \u003Ccol style=\"width: 10%\" \u002F>\n    \u003Ccol style=\"width: 90%\" \u002F>\n  \u003C\u002Fcolgroup>\n  \u003Ctbody>\n    \u003Ctr class=\"row-odd\">\n      \u003Ctd>\u003Cp>\u003Ca href=\"https:\u002F\u002Fnvitop.readthedocs.io\u002Fen\u002Flatest\u002Fapi\u002Fdevice.html#nvitop.Device\" title=\"nvitop.Device\">\u003Ccode class=\"xref py py-obj docutils literal notranslate\">\u003Cspan class=\"pre\">Device\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fa>([index, uuid, bus_id])\u003C\u002Fp>\u003C\u002Ftd>\n      \u003Ctd>\u003Cp>Live class of the GPU devices, different from the device snapshots.\u003C\u002Fp>\u003C\u002Ftd>\n    \u003C\u002Ftr>\n    \u003Ctr class=\"row-even\">\n      \u003Ctd>\u003Cp>\u003Ca href=\"https:\u002F\u002Fnvitop.readthedocs.io\u002Fen\u002Flatest\u002Fapi\u002Fdevice.html#nvitop.PhysicalDevice\" title=\"nvitop.PhysicalDevice\">\u003Ccode class=\"xref py py-obj docutils literal notranslate\">\u003Cspan class=\"pre\">PhysicalDevice\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fa>([index, uuid, bus_id])\u003C\u002Fp>\u003C\u002Ftd>\n      \u003Ctd>\u003Cp>Class for physical devices.\u003C\u002Fp>\u003C\u002Ftd>\n    \u003C\u002Ftr>\n    \u003Ctr class=\"row-odd\">\n      \u003Ctd>\u003Cp>\u003Ca href=\"https:\u002F\u002Fnvitop.readthedocs.io\u002Fen\u002Flatest\u002Fapi\u002Fdevice.html#nvitop.MigDevice\" title=\"nvitop.MigDevice\">\u003Ccode class=\"xref py py-obj docutils literal notranslate\">\u003Cspan class=\"pre\">MigDevice\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fa>([index, uuid, bus_id])\u003C\u002Fp>\u003C\u002Ftd>\n      \u003Ctd>\u003Cp>Class for MIG devices.\u003C\u002Fp>\u003C\u002Ftd>\n    \u003C\u002Ftr>\n    \u003Ctr class=\"row-even\">\n      \u003Ctd>\u003Cp>\u003Ca href=\"https:\u002F\u002Fnvitop.readthedocs.io\u002Fen\u002Flatest\u002Fapi\u002Fdevice.html#nvitop.CudaDevice\" title=\"nvitop.CudaDevice\">\u003Ccode class=\"xref py py-obj docutils literal notranslate\">\u003Cspan class=\"pre\">CudaDevice\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fa>([cuda_index, nvml_index, uuid])\u003C\u002Fp>\u003C\u002Ftd>\n      \u003Ctd>\u003Cp>Class for devices enumerated over the CUDA ordinal.\u003C\u002Fp>\u003C\u002Ftd>\n    \u003C\u002Ftr>\n    \u003Ctr class=\"row-odd\">\n      \u003Ctd>\u003Cp>\u003Ca href=\"https:\u002F\u002Fnvitop.readthedocs.io\u002Fen\u002Flatest\u002Fapi\u002Fdevice.html#nvitop.CudaMigDevice\" title=\"nvitop.CudaMigDevice\">\u003Ccode class=\"xref py py-obj docutils literal notranslate\">\u003Cspan class=\"pre\">CudaMigDevice\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fa>([cuda_index, nvml_index, uuid])\u003C\u002Fp>\u003C\u002Ftd>\n      \u003Ctd>\u003Cp>Class for CUDA devices that are MIG devices.\u003C\u002Fp>\u003C\u002Ftd>\n    \u003C\u002Ftr>\n    \u003Ctr class=\"row-even\">\n      \u003Ctd>\u003Cp>\u003Ca href=\"https:\u002F\u002Fnvitop.readthedocs.io\u002Fen\u002Flatest\u002Fapi\u002Fdevice.html#nvitop.parse_cuda_visible_devices\" title=\"nvitop.parse_cuda_visible_devices\">\u003Ccode class=\"xref py py-obj docutils literal notranslate\">\u003Cspan class=\"pre\">parse_cuda_visible_devices\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fa>([...])\u003C\u002Fp>\u003C\u002Ftd>\n      \u003Ctd>\u003Cp>Parse the given \u003Ccode class=\"docutils literal notranslate\">\u003Cspan class=\"pre\">CUDA_VISIBLE_DEVICES\u003C\u002Fspan>\u003C\u002Fcode> value into a list of NVML device indices.\u003C\u002Fp>\u003C\u002Ftd>\n    \u003C\u002Ftr>\n    \u003Ctr class=\"row-odd\">\n      \u003Ctd>\u003Cp>\u003Ca href=\"https:\u002F\u002Fnvitop.readthedocs.io\u002Fen\u002Flatest\u002Fapi\u002Fdevice.html#nvitop.normalize_cuda_visible_devices\" title=\"nvitop.normalize_cuda_visible_devices\">\u003Ccode class=\"xref py py-obj docutils literal notranslate\">\u003Cspan class=\"pre\">normalize_cuda_visible_devices\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fa>([...])\u003C\u002Fp>\u003C\u002Ftd>\n      \u003Ctd>\u003Cp>Parse the given \u003Ccode class=\"docutils literal notranslate\">\u003Cspan class=\"pre\">CUDA_VISIBLE_DEVICES\u003C\u002Fspan>\u003C\u002Fcode> value and convert it into a comma-separated string of UUIDs.\u003C\u002Fp>\u003C\u002Ftd>\n    \u003C\u002Ftr>\n  \u003C\u002Ftbody>\n\u003C\u002Ftable>\n\n```python\nIn [1]: from nvitop import (\n   ...:     host,\n   ...:     Device, PhysicalDevice, CudaDevice,\n   ...:     parse_cuda_visible_devices, normalize_cuda_visible_devices\n   ...:     HostProcess, GpuProcess,\n   ...:     NA,\n   ...: )\n   ...: import os\n   ...: os.environ['CUDA_DEVICE_ORDER'] = 'PCI_BUS_ID'\n   ...: os.environ['CUDA_VISIBLE_DEVICES'] = '9,8,7,6'  # comma-separated integers or UUID strings\n\nIn [2]: Device.driver_version()\nOut[2]: '525.60.11'\n\nIn [3]: Device.cuda_driver_version()  # the maximum CUDA version supported by the driver (can be different from the CUDA Runtime version)\nOut[3]: '12.0'\n\nIn [4]: Device.cuda_runtime_version()  # the CUDA Runtime version\nOut[4]: '11.8'\n\nIn [5]: Device.count()\nOut[5]: 10\n\nIn [6]: CudaDevice.count()  # or `Device.cuda.count()`\nOut[6]: 4\n\nIn [7]: all_devices      = Device.all()                 # all devices on board (physical device)\n   ...: nvidia0, nvidia1 = Device.from_indices([0, 1])  # from physical device indices\n   ...: all_devices\nOut[7]: [\n    PhysicalDevice(index=0, name=\"GeForce RTX 2080 Ti\", total_memory=11019MiB),\n    PhysicalDevice(index=1, name=\"GeForce RTX 2080 Ti\", total_memory=11019MiB),\n    PhysicalDevice(index=2, name=\"GeForce RTX 2080 Ti\", total_memory=11019MiB),\n    PhysicalDevice(index=3, name=\"GeForce RTX 2080 Ti\", total_memory=11019MiB),\n    PhysicalDevice(index=4, name=\"GeForce RTX 2080 Ti\", total_memory=11019MiB),\n    PhysicalDevice(index=5, name=\"GeForce RTX 2080 Ti\", total_memory=11019MiB),\n    PhysicalDevice(index=6, name=\"GeForce RTX 2080 Ti\", total_memory=11019MiB),\n    PhysicalDevice(index=7, name=\"GeForce RTX 2080 Ti\", total_memory=11019MiB),\n    PhysicalDevice(index=8, name=\"GeForce RTX 2080 Ti\", total_memory=11019MiB),\n    PhysicalDevice(index=9, name=\"GeForce RTX 2080 Ti\", total_memory=11019MiB)\n]\n\nIn [8]: # NOTE: The function results might be different between calls when the `CUDA_VISIBLE_DEVICES` environment variable has been modified\n   ...: cuda_visible_devices = Device.from_cuda_visible_devices()  # from the `CUDA_VISIBLE_DEVICES` environment variable\n   ...: cuda0, cuda1         = Device.from_cuda_indices([0, 1])    # from CUDA device indices (might be different from physical device indices if `CUDA_VISIBLE_DEVICES` is set)\n   ...: cuda_visible_devices = CudaDevice.all()                    # shortcut to `Device.from_cuda_visible_devices()`\n   ...: cuda_visible_devices = Device.cuda.all()                   # `Device.cuda` is aliased to `CudaDevice`\n   ...: cuda_visible_devices\nOut[8]: [\n    CudaDevice(cuda_index=0, nvml_index=9, name=\"NVIDIA GeForce RTX 2080 Ti\", total_memory=11019MiB),\n    CudaDevice(cuda_index=1, nvml_index=8, name=\"NVIDIA GeForce RTX 2080 Ti\", total_memory=11019MiB),\n    CudaDevice(cuda_index=2, nvml_index=7, name=\"NVIDIA GeForce RTX 2080 Ti\", total_memory=11019MiB),\n    CudaDevice(cuda_index=3, nvml_index=6, name=\"NVIDIA GeForce RTX 2080 Ti\", total_memory=11019MiB)\n]\n\nIn [9]: nvidia0 = Device(0)  # from device index (or `Device(index=0)`)\n   ...: nvidia0\nOut[9]: PhysicalDevice(index=0, name=\"GeForce RTX 2080 Ti\", total_memory=11019MiB)\n\nIn [10]: nvidia1 = Device(uuid='GPU-01234567-89ab-cdef-0123-456789abcdef')  # from UUID string (or just `Device('GPU-xxxxxxxx-...')`)\n    ...: nvidia2 = Device(bus_id='00000000:06:00.0')                        # from PCI bus ID\n    ...: nvidia1\nOut[10]: PhysicalDevice(index=1, name=\"GeForce RTX 2080 Ti\", total_memory=11019MiB)\n\nIn [11]: cuda0 = CudaDevice(0)                        # from CUDA device index (equivalent to `CudaDevice(cuda_index=0)`)\n    ...: cuda1 = CudaDevice(nvml_index=8)             # from physical device index\n    ...: cuda3 = CudaDevice(uuid='GPU-xxxxxxxx-...')  # from UUID string\n    ...: cuda4 = Device.cuda(4)                       # `Device.cuda` is aliased to `CudaDevice`\n    ...: cuda0\nOut[11]:\nCudaDevice(cuda_index=0, nvml_index=9, name=\"NVIDIA GeForce RTX 2080 Ti\", total_memory=11019MiB)\n\nIn [12]: nvidia0.memory_used()  # in bytes\nOut[12]: 9293398016\n\nIn [13]: nvidia0.memory_used_human()\nOut[13]: '8862MiB'\n\nIn [14]: nvidia0.gpu_utilization()  # in percentage\nOut[14]: 5\n\nIn [15]: nvidia0.processes()  # type: Dict[int, GpuProcess]\nOut[15]: {\n    52059: GpuProcess(pid=52059, gpu_memory=7885MiB, type=C, device=PhysicalDevice(index=0, name=\"GeForce RTX 2080 Ti\", total_memory=11019MiB), host=HostProcess(pid=52059, name='ipython3', status='sleeping', started='14:31:22')),\n    53002: GpuProcess(pid=53002, gpu_memory=967MiB, type=C, device=PhysicalDevice(index=0, name=\"GeForce RTX 2080 Ti\", total_memory=11019MiB), host=HostProcess(pid=53002, name='python', status='running', started='14:31:59'))\n}\n\nIn [16]: nvidia1_snapshot = nvidia1.as_snapshot()\n    ...: nvidia1_snapshot\nOut[16]: PhysicalDeviceSnapshot(\n    real=PhysicalDevice(index=1, name=\"GeForce RTX 2080 Ti\", total_memory=11019MiB),\n    bus_id='00000000:05:00.0',\n    compute_mode='Default',\n    clock_infos=ClockInfos(graphics=1815, sm=1815, memory=6800, video=1680),  # in MHz\n    clock_speed_infos=ClockSpeedInfos(current=ClockInfos(graphics=1815, sm=1815, memory=6800, video=1680), max=ClockInfos(graphics=2100, sm=2100, memory=7000, video=1950)),  # in MHz\n    cuda_compute_capability=(7, 5),\n    current_driver_model='N\u002FA',\n    decoder_utilization=0,              # in percentage\n    display_active='Disabled',\n    display_mode='Disabled',\n    encoder_utilization=0,              # in percentage\n    fan_speed=22,                       # in percentage\n    gpu_utilization=17,                 # in percentage (NOTE: this is the utilization rate of SMs, i.e. GPU percent)\n    index=1,\n    max_clock_infos=ClockInfos(graphics=2100, sm=2100, memory=7000, video=1950),  # in MHz\n    memory_clock=6800,                  # in MHz\n    memory_free=10462232576,            # in bytes\n    memory_free_human='9977MiB',\n    memory_info=MemoryInfo(total=11554717696, free=10462232576, used=1092485120)  # in bytes\n    memory_percent=9.5,                 # in percentage (NOTE: this is the percentage of used GPU memory)\n    memory_total=11554717696,           # in bytes\n    memory_total_human='11019MiB',\n    memory_usage='1041MiB \u002F 11019MiB',\n    memory_used=1092485120,             # in bytes\n    memory_used_human='1041MiB',\n    memory_utilization=7,               # in percentage (NOTE: this is the utilization rate of GPU memory bandwidth)\n    mig_mode='N\u002FA',\n    name='GeForce RTX 2080 Ti',\n    pcie_rx_throughput=1000,            # in KiB\u002Fs\n    pcie_rx_throughput_human='1000KiB\u002Fs',\n    pcie_throughput=ThroughputInfo(tx=1000, rx=1000),  # in KiB\u002Fs\n    pcie_tx_throughput=1000,            # in KiB\u002Fs\n    pcie_tx_throughput_human='1000KiB\u002Fs',\n    performance_state='P2',\n    persistence_mode='Disabled',\n    power_limit=250000,                 # in milliwatts (mW)\n    power_status='66W \u002F 250W',          # in watts (W)\n    power_usage=66051,                  # in milliwatts (mW)\n    sm_clock=1815,                      # in MHz\n    temperature=39,                     # in Celsius\n    total_volatile_uncorrected_ecc_errors='N\u002FA',\n    utilization_rates=UtilizationRates(gpu=17, memory=7, encoder=0, decoder=0),  # in percentage\n    uuid='GPU-01234567-89ab-cdef-0123-456789abcdef',\n)\n\nIn [17]: nvidia1_snapshot.memory_percent  # snapshot uses properties instead of function calls\nOut[17]: 9.5\n\nIn [18]: nvidia1_snapshot['memory_info']  # snapshot also supports `__getitem__` by string\nOut[18]: MemoryInfo(total=11554717696, free=10462232576, used=1092485120)\n\nIn [19]: nvidia1_snapshot.bar1_memory_info  # snapshot will automatically retrieve not presented attributes from `real`\nOut[19]: MemoryInfo(total=268435456, free=257622016, used=10813440)\n```\n\n**NOTE:** Some entry values may be `'N\u002FA'` (type: [`NaType`](https:\u002F\u002Fnvitop.readthedocs.io\u002Fen\u002Flatest\u002Findex.html#nvitop.NaType), a subclass of `str`) when the corresponding resources are not applicable. The [`NA`](https:\u002F\u002Fnvitop.readthedocs.io\u002Fen\u002Flatest\u002Findex.html#nvitop.NA) value supports arithmetic operations. It acts like `math.nan: float`.\n\n```python\n>>> from nvitop import NA\n>>> NA\n'N\u002FA'\n\n>>> 'memory usage: {}'.format(NA)  # NA is an instance of `str`\n'memory usage: N\u002FA'\n>>> NA.lower()                     # NA is an instance of `str`\n'n\u002Fa'\n>>> NA.ljust(5)                    # NA is an instance of `str`\n'N\u002FA  '\n>>> NA + 'str'                     # string contamination if the operand is a string\n'N\u002FAstr'\n\n>>> float(NA)                      # explicit conversion to float (`math.nan`)\nnan\n>>> NA + 1                         # auto-casting to float if the operand is a number\nnan\n>>> NA * 1024                      # auto-casting to float if the operand is a number\nnan\n>>> NA \u002F (1024 * 1024)             # auto-casting to float if the operand is a number\nnan\n```\n\nYou can use `entry != 'N\u002FA'` conditions to avoid exceptions. It's safe to use `float(entry)` for numbers while `NaType` will be converted to `math.nan`. For example:\n\n```python\nmemory_used: Union[int, NaType] = device.memory_used()            # memory usage in bytes or `'N\u002FA'`\nmemory_used_in_mib: float       = float(memory_used) \u002F (1 \u003C\u003C 20)  # memory usage in Mebibytes (MiB) or `math.nan`\n```\n\nIt's safe to compare `NaType` with numbers, but `NaType` is always larger than any number:\n\n```python\ndevices_by_used_memory = sorted(Device.all(), key=Device.memory_used, reverse=True)  # it's safe to compare `'N\u002FA'` with numbers\ndevices_by_free_memory = sorted(Device.all(), key=Device.memory_free, reverse=True)  # please add `memory_free != 'N\u002FA'` checks if sort in descending order here\n```\n\nSee [`nvitop.NaType`](https:\u002F\u002Fnvitop.readthedocs.io\u002Fen\u002Flatest\u002Fapis\u002Findex.html#nvitop.NaType) documentation for more details.\n\n##### Process\n\nThe [process module](https:\u002F\u002Fnvitop.readthedocs.io\u002Fen\u002Flatest\u002Fapi\u002Fprocess.html) provides:\n\n\u003Ctable class=\"autosummary longtable docutils align-default\">\n  \u003Ccolgroup>\n    \u003Ccol style=\"width: 10%\" \u002F>\n    \u003Ccol style=\"width: 90%\" \u002F>\n  \u003C\u002Fcolgroup>\n  \u003Ctbody>\n    \u003Ctr class=\"row-odd\">\n      \u003Ctd>\u003Cp>\u003Ca href=\"https:\u002F\u002Fnvitop.readthedocs.io\u002Fen\u002Flatest\u002Fapi\u002Fprocess.html#nvitop.HostProcess\" title=\"nvitop.HostProcess\">\u003Ccode class=\"xref py py-obj docutils literal notranslate\">\u003Cspan class=\"pre\">HostProcess\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fa>([pid])\u003C\u002Fp>\u003C\u002Ftd>\n      \u003Ctd>\u003Cp>Represents an OS process with the given PID.\u003C\u002Fp>\u003C\u002Ftd>\n    \u003C\u002Ftr>\n    \u003Ctr class=\"row-even\">\n      \u003Ctd>\u003Cp>\u003Ca href=\"https:\u002F\u002Fnvitop.readthedocs.io\u002Fen\u002Flatest\u002Fapi\u002Fprocess.html#nvitop.GpuProcess\" title=\"nvitop.GpuProcess\">\u003Ccode class=\"xref py py-obj docutils literal notranslate\">\u003Cspan class=\"pre\">GpuProcess\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fa>(pid, device[, gpu_memory, ...])\u003C\u002Fp>\u003C\u002Ftd>\n      \u003Ctd>\u003Cp>Represents a process with the given PID running on the given GPU device.\u003C\u002Fp>\u003C\u002Ftd>\n    \u003C\u002Ftr>\n    \u003Ctr class=\"row-odd\">\n      \u003Ctd>\u003Cp>\u003Ca href=\"https:\u002F\u002Fnvitop.readthedocs.io\u002Fen\u002Flatest\u002Fapi\u002Fprocess.html#nvitop.command_join\" title=\"nvitop.command_join\">\u003Ccode class=\"xref py py-obj docutils literal notranslate\">\u003Cspan class=\"pre\">command_join\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fa>(cmdline)\u003C\u002Fp>\u003C\u002Ftd>\n      \u003Ctd>\u003Cp>Returns a shell-escaped string from command line arguments.\u003C\u002Fp>\u003C\u002Ftd>\n    \u003C\u002Ftr>\n  \u003C\u002Ftbody>\n\u003C\u002Ftable>\n\n```python\nIn [20]: processes = nvidia1.processes()  # type: Dict[int, GpuProcess]\n    ...: processes\nOut[20]: {\n    23266: GpuProcess(pid=23266, gpu_memory=1031MiB, type=C, device=Device(index=1, name=\"GeForce RTX 2080 Ti\", total_memory=11019MiB), host=HostProcess(pid=23266, name='python3', status='running', started='2021-05-10 21:02:40'))\n}\n\nIn [21]: process = processes[23266]\n    ...: process\nOut[21]: GpuProcess(pid=23266, gpu_memory=1031MiB, type=C, device=Device(index=1, name=\"GeForce RTX 2080 Ti\", total_memory=11019MiB), host=HostProcess(pid=23266, name='python3', status='running', started='2021-05-10 21:02:40'))\n\nIn [22]: process.status()  # GpuProcess will automatically inherit attributes from GpuProcess.host\nOut[22]: 'running'\n\nIn [23]: process.cmdline()  # type: List[str]\nOut[23]: ['python3', 'rllib_train.py']\n\nIn [24]: process.command()  # type: str\nOut[24]: 'python3 rllib_train.py'\n\nIn [25]: process.cwd()  # GpuProcess will automatically inherit attributes from GpuProcess.host\nOut[25]: '\u002Fhome\u002Fxxxxxx\u002FProjects\u002Fxxxxxx'\n\nIn [26]: process.gpu_memory_human()\nOut[26]: '1031MiB'\n\nIn [27]: process.as_snapshot()\nOut[27]: GpuProcessSnapshot(\n    real=GpuProcess(pid=23266, gpu_memory=1031MiB, type=C, device=PhysicalDevice(index=1, name=\"GeForce RTX 2080 Ti\", total_memory=11019MiB), host=HostProcess(pid=23266, name='python3', status='running', started='2021-05-10 21:02:40')),\n    cmdline=['python3', 'rllib_train.py'],\n    command='python3 rllib_train.py',\n    compute_instance_id='N\u002FA',\n    cpu_percent=98.5,                       # in percentage\n    device=PhysicalDevice(index=1, name=\"GeForce RTX 2080 Ti\", total_memory=11019MiB),\n    gpu_encoder_utilization=0,              # in percentage\n    gpu_decoder_utilization=0,              # in percentage\n    gpu_instance_id='N\u002FA',\n    gpu_memory=1081081856,                  # in bytes\n    gpu_memory_human='1031MiB',\n    gpu_memory_percent=9.4,                 # in percentage (NOTE: this is the percentage of used GPU memory)\n    gpu_memory_utilization=5,               # in percentage (NOTE: this is the utilization rate of GPU memory bandwidth)\n    gpu_sm_utilization=0,                   # in percentage (NOTE: this is the utilization rate of SMs, i.e. GPU percent)\n    host=HostProcessSnapshot(\n        real=HostProcess(pid=","nvitop 是一个交互式的 NVIDIA GPU 进程查看器，提供一站式的 GPU 进程管理解决方案。它支持实时监控 GPU 使用情况、进程状态及资源占用等信息，并具备命令行工具特性，适用于需要精细化管理 GPU 资源的场景。该项目基于 Python 开发，兼容 CUDA 和 NVML API，能够与 Grafana 和 Prometheus 等流行监控系统集成，方便用户进行可视化展示和报警设置。无论是科研人员调试深度学习模型还是系统管理员优化服务器资源分配，nvitop 都能提供有力支持。",2,"2026-06-11 03:35:49","high_star"]