[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-80146":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":15,"subscribersCount":15,"size":15,"stars1d":15,"stars7d":16,"stars30d":16,"stars90d":15,"forks30d":15,"starsTrendScore":15,"compositeScore":17,"rankGlobal":10,"rankLanguage":10,"license":18,"archived":19,"fork":19,"defaultBranch":20,"hasWiki":19,"hasPages":19,"topics":21,"createdAt":10,"pushedAt":10,"updatedAt":27,"readmeContent":28,"aiSummary":29,"trendingCount":15,"starSnapshotCount":15,"syncStatus":16,"lastSyncTime":30,"discoverSource":31},80146,"surface-watch","Nextron-Labs\u002Fsurface-watch","Nextron-Labs","surface-watch monitors the authorized external attack surface of an organization over time","",null,"Python",55,4,1,0,2,43.3,"GNU General Public License v3.0",false,"main",[22,23,24,25,26],"attack","monitoring","network","scanning","security-tools","2026-06-12 04:01:26","\u003Cp align=\"center\">\n  \u003Cimg src=\"docs\u002Fsurface-watch.png\" alt=\"surface-watch logo\">\n\u003C\u002Fp>\n\n# surface-watch\n\n`surface-watch` monitors the authorized external attack surface of an organization over time.\n\nIt builds scan scope from known FQDNs and IPs plus automatic discovery for configured root domains, resolves candidate hosts, scans externally reachable ports with `nmap`, stores historical results in SQLite, detects meaningful changes between scans, and sends grouped webhook notifications to Slack, Microsoft Teams, or Discord.\n\n## What the tool does\n\n- Discovers scan targets from configured domains, explicit hosts, and explicit IPs.\n- Resolves DNS data for `A`, `AAAA`, `CNAME`, `MX`, `NS`, and `SRV` records.\n- Scans externally reachable TCP ports with `nmap`.\n- Stores full scan history, not just the latest result.\n- Compares the current scan against the previous successful baseline.\n- Detects host, IP, port, scan-status, and basic service changes.\n- Sends grouped webhook notifications based on configurable change rules and severity.\n\n## What the tool does not do\n\n- It does not provide a web UI in v1. The tool is CLI-first.\n- It does not include exploit modules.\n- It does not attempt to bypass rate limits, IDS, or firewall protections.\n- It does not perform aggressive probing beyond DNS resolution and `nmap`-based scanning.\n- It does not run its own custom port scanner.\n- It does not enable UDP scanning by default. UDP is disabled by default because it is slow and noisy.\n\n## Legal \u002F Authorization Warning\n\nUse this tool only for domains, hosts, and IPs that you own or are explicitly authorized to scan.\n\n`surface-watch` is intended for authorized external exposure monitoring of owned or approved assets. Do not use it against third-party systems without permission.\n\n## Installation\n\n1. Create a virtual environment.\n2. Install the package.\n3. Ensure `nmap` is available in `PATH`.\n\n```bash\npython3 -m venv .venv\nsource .venv\u002Fbin\u002Factivate\npip install -e .\n```\n\n## Requirements\n\n- Python `3.11+`\n- `nmap` installed and available in `PATH`\n\nIf an AI agent is helping with first-time deployment or system setup, start with [AGENTS.md](AGENTS.md). It tells the agent which questions to ask about scope, scheduling, passive discovery, webhook setup, baselining, and validation.\n\nExample on macOS with Homebrew:\n\n```bash\nbrew install nmap\n```\n\nExample on Debian\u002FUbuntu:\n\n```bash\nsudo apt-get install nmap\n```\n\n## Quick Start\n\n1. Create a config and initialize the database.\n2. Edit the config to match your authorized scope.\n3. Run discovery first and review the discovered host list.\n4. Add exclusions for anything that is not authorized for scanning.\n5. Run a baseline scan.\n6. Run the next scan later to detect changes.\n\n```bash\nsurface-watch init\n$EDITOR config.yaml\nsurface-watch discover\nsurface-watch scan\nsurface-watch list-scans\nsurface-watch show-changes --scan-id 2\n```\n\nIf your config file is not `.\u002Fconfig.yaml`, pass `--config` either before or after the\nsubcommand, for example `surface-watch --config \u002Fopt\u002Fsurface-watch\u002Fconfig.yaml list-scans`\nor `surface-watch list-scans --config \u002Fopt\u002Fsurface-watch\u002Fconfig.yaml`.\n\n## Configuration Example\n\nAn example file is included at [config\u002Fexample.yaml](config\u002Fexample.yaml).\n\nTypical configuration:\n\n```yaml\nproject:\n  name: \"Example External Surface Watch\"\n  database_path: \".\u002Fsurface-watch.sqlite3\"\n  log_level: \"INFO\"\n\nscope:\n  domains:\n    - \"example.com\"\n  explicit_hosts:\n    - \"vpn.example.com\"\n  explicit_ips:\n    - \"203.0.113.10\"\n  excluded_hosts:\n    - \"do-not-scan.example.com\"\n  excluded_ips:\n    - \"203.0.113.99\"\n\ndiscovery:\n  enabled: true\n  passive_sources:\n    enabled: false\n    dnsdumpster:\n      enabled: false\n      api_key_env: \"DNSDUMPSTER_API_KEY\"\n      min_interval_seconds: 2.0\n      max_pages: 1\n      restrict_to_domain_suffix: true\n    chaos:\n      enabled: false\n      api_key_env: \"PDCP_API_KEY\"\n    otx:\n      enabled: false\n      api_key_env: \"OTX_API_KEY\"\n\nscanning:\n  enabled: true\n  nmap_path: \"nmap\"\n  scan_mode: \"full_tcp\"\n  ports:\n    tcp: \"1-65535\"\n    udp: \"\"\n\nnotifications:\n  enabled: true\n  minimum_severity: \"medium\"\n  providers:\n    slack:\n      enabled: true\n      webhook_url_env: \"SLACK_WEBHOOK_URL\"\n```\n\n## How Discovery Works\n\n`surface-watch` has two ways to define scan targets:\n\n1. Manual scope definition with explicit FQDNs and IP addresses.\n2. Automatic discovery for configured root domains.\n\nManual scope definition is useful when you already know the important assets. Automatic discovery is essential in most real-world use cases because most teams do not have a complete, current inventory of every externally visible hostname under their domains.\n\nNormal DNS lookups rarely show the full external picture. Many hosts are only visible through passive DNS, historical DNS data, certificate transparency, and other external intelligence sources. Discovery is therefore one of the central parts of `surface-watch`, not a minor add-on.\n\nCurrent discovery inputs:\n\n- configured domains\n- explicit hosts\n- explicit IPs\n- optional MX, NS, and SRV-derived hosts\n- optional passive discovery providers: DNSDumpster, ProjectDiscovery Chaos, and AlienVault \u002F LevelBlue OTX\n\nPassive discovery is especially important because the default goal is not to guess hostnames with noisy brute force. DNS brute forcing is intentionally not the preferred default path because it is noisy, expensive, incomplete, and can lead to rate limiting or blocking. Passive intelligence sources provide broader coverage with less direct probing.\n\nPassive provider behavior:\n\n- Passive providers are optional and can be enabled independently.\n- Multiple passive providers can be enabled in the same run.\n- Each enabled provider is queried for each configured root domain.\n- Passive results are treated as candidate hostnames, not confirmed live assets.\n- Candidate hostnames are normalized before merge, including wildcard cleanup such as `*.app.example.com`.\n- Out-of-scope hostnames are dropped.\n- Results from all providers are merged and deduplicated before scanning.\n- A hostname reported by multiple providers is scanned only once per unique resolved target.\n- Source attribution is preserved so you can still see which provider or providers reported a hostname.\n- Exclusions are applied after discovery completes and before scanning starts.\n- Unresolvable passive candidates are not scanned.\n- One passive provider failing does not discard results from the other successful providers.\n\n## Passive Discovery Providers\n\n`surface-watch` supports three optional passive discovery providers. All are disabled by default and require an API key.\n\n- `DNSDumpster`: DNS and attack-surface research service for finding domain-related DNS records, infrastructure, and related hosts. Website: [dnsdumpster.com](https:\u002F\u002Fdnsdumpster.com\u002F). Account and API setup: [dnsdumpster.com\u002Fdeveloper](https:\u002F\u002Fdnsdumpster.com\u002Fdeveloper\u002F). Environment variable: `DNSDUMPSTER_API_KEY`.\n- `ProjectDiscovery Chaos`: Internet-scale DNS dataset and API for subdomain discovery from ProjectDiscovery. Website: [chaos.projectdiscovery.io](https:\u002F\u002Fchaos.projectdiscovery.io\u002F). Account and API-key setup: [cloud.projectdiscovery.io](https:\u002F\u002Fcloud.projectdiscovery.io\u002F) and [Chaos API key docs](https:\u002F\u002Fchaos.projectdiscovery.io\u002Fdocs\u002Fapi-key). Environment variable: `PDCP_API_KEY`.\n- `AlienVault \u002F LevelBlue OTX`: Open threat-intelligence community and API that can contribute passive DNS and hostname data useful for broader hostname discovery. Website: [LevelBlue Open Threat Exchange](https:\u002F\u002Flevelblue.com\u002Fopen-threat-exchange) and [OTX portal](https:\u002F\u002Fotx.alienvault.com\u002F). Account setup: [OTX sign-up](https:\u002F\u002Fotx.alienvault.com\u002Faccounts\u002Fsignup\u002F). Environment variable: `OTX_API_KEY`.\n\n`surface-watch` does not log API key values.\n\nThe tool does not assume one hostname maps to one IP, or one IP maps to one hostname. Discovery builds a candidate set first, then scanning operates only on the final unique in-scope target set.\n\n## First Discovery Run and Scope Review\n\nTreat the first automated discovery run as a scope-building and scope-cleanup step.\n\nPassive discovery can find hostnames under your domain that point to systems operated by third parties, SaaS platforms, CDNs, email providers, documentation platforms, status page providers, hosting providers, customer portals, or other external services.\n\nCommon examples:\n\n- `documentation.example.com` on a hosted documentation platform\n- `status.example.com` on a status page provider\n- `shop.example.com` on an e-commerce provider\n- `support.example.com` on a ticketing or SaaS platform\n- `blog.example.com` on an external CMS\n- `assets.example.com` on a CDN\n- mail-related hosts operated by an email provider\n\nThose systems may carry your domain name, but they may not be owned, operated, or security-managed by you.\n\nImportant review rules:\n\n- Passive discovery results are candidate assets, not automatically confirmed assets.\n- Passive records may be stale, wrong, incomplete, or no longer active.\n- Some discovered hosts may not be in your operational responsibility.\n- A third-party provider may already have its own security monitoring and scanning restrictions.\n- The provider's terms of service may prohibit port scanning or automated scanning.\n- Review the discovered host list after the first run.\n- Add exclusions for any host or IP that should not be scanned.\n- Enable regular recurring scans only after the scope has been reviewed and cleaned up.\n- You are responsible for ensuring that every scanned system is authorized for scanning.\n\n## How Scanning Works\n\nScanning is performed by calling external `nmap` via `subprocess`.\n\nFor TCP scans, the command is built around:\n\n- `-Pn`\n- `--open`\n- `-p \u003Cconfigured ports>`\n- `-6` automatically for IPv6 targets\n- `-sS` when running as root on Linux\u002FmacOS, otherwise `-sT`\n- `-sV` when service detection is enabled\n- `--version-intensity \u003Cvalue>`\n- `-T\u003Ctemplate>`\n- `--host-timeout \u003Cvalue>`\n- `-oX -`\n\n`surface-watch` parses `nmap` XML output into internal structured models before storing or diffing results.\n\nIf one host fails, the run continues for the remaining hosts. A run is only marked fully failed when all host scans fail.\nHosts that appear to hit `--host-timeout`, or that yield an invalid zero-host result such as\nan IPv6 target scanned without `-6`, are downgraded to `failed` or `partial` instead of being\nstored as a clean success.\n\n## Why Full TCP Scanning Is the Default\n\nThe default TCP range is `1-65535`.\n\nThis is intentional:\n\n- a limited port preset can miss newly exposed services\n- the tool is meant to maintain a reliable external baseline over time\n- meaningful exposure changes often happen on non-standard ports\n\nThe goal is not to be aggressive. The goal is to maintain a stable, repeatable baseline of externally exposed TCP services and detect meaningful drift.\n\nUDP scanning is disabled by default because it is slow and noisy.\n\n## How Scan History Is Stored\n\nHistory is stored in SQLite. Each run creates a scan record.\n\nThe database stores:\n\n- `scans`: run metadata and final status\n- `targets`: discovered targets for that run\n- `host_results`: per-host or per-IP scan outcome\n- `port_findings`: open port and service data\n- `changes`: detected differences for that run\n\nThis makes it possible to compare scans later, inspect previous baselines, and review what changed over time.\n\nUseful inspection commands:\n\n```bash\nsurface-watch list-scans --config config.yaml\nsurface-watch show-targets --config config.yaml --scan-id 2\nsurface-watch show-ports --config config.yaml --host vpn.example.com --scan-id 2\nsurface-watch show-scan --config config.yaml --scan-id 2\nsurface-watch show-changes --config config.yaml --scan-id 2\n```\n\n## How Change Detection Works\n\nThe default comparison baseline is the previous successful scan with the same stored config hash.\n\nDetection is based on normalized internal tuples:\n\n- host identity: `hostname + ip` where available\n- port identity: `ip + protocol + port`\n- service identity: `ip + protocol + port + service\u002Fproduct\u002Fversion`\n\nImportant behavior:\n\n- failed scans are stored\n- partial scans are stored\n- comparisons default to the previous successful scan with a matching config hash\n- empty and missing service fields are treated consistently to reduce false noise\n- missing version strings do not automatically produce version-change events\n\n## Supported Change Types\n\nImplemented in v1:\n\n- `new_host`\n- `disappeared_host`\n- `new_ip_for_host`\n- `removed_ip_from_host`\n- `host_scan_failed`\n- `host_scan_recovered`\n- `new_open_port`\n- `closed_port`\n- `service_changed`\n- `product_changed`\n- `version_changed`\n- `product_version_changed`\n\nDefined as possible future categories:\n\n- `cname_changed`\n- `mx_changed`\n- `ns_changed`\n- `srv_record_changed`\n- `host_timeout`\n- `host_unreachable`\n- `host_reachable_again`\n- `port_state_changed`\n- `new_protocol_on_host`\n- `banner_changed`\n- `tls_detected`\n- `tls_removed`\n- `certificate_subject_changed`\n- `certificate_issuer_changed`\n- `certificate_expiry_changed`\n- `http_title_changed`\n- `http_server_header_changed`\n- `http_redirect_changed`\n- `risky_port_exposed`\n- `admin_port_exposed`\n- `database_port_exposed`\n- `remote_access_port_exposed`\n\n## Notification Setup for Slack, Teams, and Discord\n\nWebhook URLs are not stored directly in config by default. The config references environment variable names instead.\n\nExample:\n\n```bash\nexport SLACK_WEBHOOK_URL=\"https:\u002F\u002Fhooks.slack.com\u002Fservices\u002F...\"\nexport TEAMS_WEBHOOK_URL=\"https:\u002F\u002F...\"\nexport DISCORD_WEBHOOK_URL=\"https:\u002F\u002Fdiscord.com\u002Fapi\u002Fwebhooks\u002F...\"\n```\n\nThen enable the provider in `config.yaml`.\n\nNotifications are:\n\n- filtered by `change_detection.notify_on`\n- filtered by `notifications.minimum_severity`\n- grouped into one message per scan\n- skipped when no change qualifies\n\n`risk_policy` can raise severity for specific ports such as `3389`, `5900`, `9200`, `27017`, `3306`, and similar exposures. This makes notifications more useful than treating every newly open port equally.\n\nService and version changes can be noisy and are therefore not notification-worthy by default.\n\nUse the built-in test command:\n\n```bash\nsurface-watch test-notification --config config.yaml\n```\n\n## Running from cron\n\nExample cron entry:\n\n```cron\nMAILTO=\"\"\n15 * * * * cd \u002Fopt\u002Fsurface-watch && \u002Fopt\u002Fsurface-watch\u002F.venv\u002Fbin\u002Fsurface-watch scan --config \u002Fopt\u002Fsurface-watch\u002Fconfig.yaml >> \u002Fvar\u002Flog\u002Fsurface-watch.log 2>&1\n```\n\nUse a dedicated virtual environment and absolute paths in cron jobs.\n\n## Running from systemd timer\n\nExample service unit:\n\n```ini\n[Unit]\nDescription=surface-watch scan\n\n[Service]\nType=oneshot\nWorkingDirectory=\u002Fopt\u002Fsurface-watch\nEnvironmentFile=\u002Fopt\u002Fsurface-watch\u002Fsurface-watch.env\nExecStart=\u002Fopt\u002Fsurface-watch\u002F.venv\u002Fbin\u002Fsurface-watch scan --config \u002Fopt\u002Fsurface-watch\u002Fconfig.yaml\n```\n\nExample timer unit:\n\n```ini\n[Unit]\nDescription=Run surface-watch every hour\n\n[Timer]\nOnCalendar=hourly\nPersistent=true\n\n[Install]\nWantedBy=timers.target\n```\n\n## Troubleshooting\n\n`nmap not found`\n\n- install `nmap`\n- confirm `nmap` is in `PATH`\n- or set `scanning.nmap_path` explicitly\n\nNo notifications sent\n\n- confirm the provider is enabled in config\n- confirm the referenced environment variable is set\n- run `surface-watch test-notification --config config.yaml`\n\nDNSDumpster passive discovery returns no results\n\n- confirm `discovery.passive_sources.enabled: true`\n- confirm `discovery.passive_sources.dnsdumpster.enabled: true`\n- confirm the `DNSDUMPSTER_API_KEY` environment variable is set\n- remember that free-tier use is rate-limited to one request every two seconds and limited in returned records\n- if you need external MX or NS hosts, check whether `restrict_to_domain_suffix` is filtering them on purpose\n\nChaos passive discovery returns no results\n\n- confirm `discovery.passive_sources.enabled: true`\n- confirm `discovery.passive_sources.chaos.enabled: true`\n- confirm the `PDCP_API_KEY` environment variable is set\n\nOTX passive discovery returns no results\n\n- confirm `discovery.passive_sources.enabled: true`\n- confirm `discovery.passive_sources.otx.enabled: true`\n- confirm the `OTX_API_KEY` environment variable is set\n\nPassive discovery found unexpected hosts\n\n- this can happen with historical passive DNS or third-party hosted subdomains\n- review the discovered host list before enabling regular scans\n- add exclusions for anything not owned or not authorized for scanning\n\nUnexpected service-change noise\n\n- this can happen with `nmap -sV` fingerprints\n- keep service-related notifications disabled unless they matter to your workflow\n\nNo previous scan to compare against\n\n- the first successful run becomes the baseline\n- change detection starts on later runs\n\nPermission differences for `-sS`\n\n- SYN scan is used only when the effective user is root on Linux\u002FmacOS\n- otherwise `surface-watch` falls back to `-sT`\n\n## Development \u002F Tests\n\nInstall dev dependencies:\n\n```bash\npython3 -m venv .venv\nsource .venv\u002Fbin\u002Factivate\npip install -e '.[dev]'\n```\n\nRun checks:\n\n```bash\nruff check .\npytest\n```\n\nUseful commands:\n\n```bash\nsurface-watch discover --config config.yaml\nsurface-watch scan --config config.yaml\nsurface-watch diff --config config.yaml --scan-id 2 --previous-scan-id 1\nsurface-watch list-scans --config config.yaml\nsurface-watch show-targets --config config.yaml --scan-id 2\nsurface-watch show-ports --config config.yaml --host vpn.example.com --scan-id 2\nsurface-watch show-scan --config config.yaml --scan-id 2\nsurface-watch show-changes --config config.yaml --scan-id 2\n```\n","surface-watch 是一个用于持续监控组织授权外部攻击面的工具。它通过已知的完全限定域名（FQDNs）和IP地址以及自动发现配置的根域名来构建扫描范围，解析候选主机，使用nmap扫描可从外部访问的TCP端口，并将历史结果存储在SQLite数据库中。此外，该工具能够检测扫描间有意义的变化，并通过Slack、Microsoft Teams或Discord发送分组的Webhook通知。适用于需要定期检查其数字资产暴露情况的企业安全团队，以确保及时发现并响应潜在的安全威胁。","2026-06-11 03:59:26","CREATED_QUERY"]