[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-83834":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":9,"language":10,"languages":9,"totalLinesOfCode":9,"stars":11,"forks":12,"watchers":13,"openIssues":14,"contributorsCount":14,"subscribersCount":14,"size":14,"stars1d":14,"stars7d":15,"stars30d":15,"stars90d":14,"forks30d":14,"starsTrendScore":16,"compositeScore":17,"rankGlobal":9,"rankLanguage":9,"license":18,"archived":19,"fork":19,"defaultBranch":20,"hasWiki":19,"hasPages":19,"topics":21,"createdAt":9,"pushedAt":9,"updatedAt":22,"readmeContent":23,"aiSummary":9,"trendingCount":14,"starSnapshotCount":14,"syncStatus":24,"lastSyncTime":25,"discoverSource":26},83834,"scoptix","Omnitarium\u002Fscoptix","Omnitarium","Open-source passive reconnaissance and attack surface exploration tool that leverages VirusTotal and the Wayback Machine to discover subdomains, URLs, archived web assets, and potential exposure findings.",null,"TypeScript",91,22,60,0,21,10,56.69,"Apache License 2.0",false,"main",[],"2026-06-12 04:01:42","# SCOPTIX\n\nSCOPTIX is a passive reconnaissance and attack surface exploration tool that helps analysts identify exposed content, potentially sensitive information, and application endpoints that may warrant further investigation. It aggregates subdomains, URLs, IP addresses, and archived web assets from external data sources to support security analysis and exposure discovery.\n\nData is currently sourced from **VirusTotal** and the Internet Archive's **Wayback Machine**.\n\n![Dashboard — target overview and scan history](.\u002Fimg\u002F1-dashboard.png)\n\n-----\n\n## Key Features\n\n* **Asset Discovery:** Discover subdomains, URLs, IP addresses, and archived web assets from multiple external data sources. IP resolutions come from VirusTotal passive DNS (hostname ↔ IP history for the apex and discovered subdomains). Each scan keeps an observed IP list for that run; the target view aggregates the same addresses across all scans with hostname timelines and historical resolution detail.\n* **Exposure Discovery:** Identify potentially exposed credentials, API keys, tokens, cloud secrets, and configuration artifacts across discovered assets using customizable detection rules.\n* **Asset Categorization:** Automatically group discovered URLs into extension categories you define in Settings—create categories and map pathname suffixes (e.g. `.pdf`, `.js`, `.zip`) so assets are organized for review.\n* **Deep Scan:** Optionally download JavaScript files when starting a scan and analyze their contents against your detection rules, in addition to the URL-string checks applied to every URL.\n* **Endpoint Discovery:** Explore parameters, application endpoints, authentication-related resources, and other security-relevant application assets.\n* **Scan Comparison:** Track changes across scans and quickly identify newly discovered subdomains, URLs, IP addresses, archived assets, and exposure findings.\n\n![Scan summary — compare results across scans](.\u002Fimg\u002F2-scan-summary.png)\n\n![Historical hostnames — passive DNS timeline per IP](.\u002Fimg\u002F3-historicalhostnames.png)\n\n![All findings — exposure and detection results](.\u002Fimg\u002F4-all-findings.png)\n\n-----\n\n## Real-World Examples\n\nSCOPTIX was built around methodologies demonstrated by **Urwah Atiyat (OrwaGodFather)** in the following presentations:\n\n* Art of VirusTotal Hacking – https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=Xosa-1o-01M\n* Essence of Recon in Bug Bounty and Pentesting – https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=CJnXjWXXB1Y\n\nReal-world examples discussed in these presentations include:\n\n* Identifying exposed origin infrastructure behind WAF.\n* Discovering publicly accessible sensitive documents, including identity records, passports, and other personal information.\n* Finding forgotten backup archives (such as `backup.7z`) exposing source code, credentials, configuration files, or other sensitive internal information.\n* Identifying password reset URLs that remain valid beyond their intended lifetime, potentially leading to account compromise.\n\n-----\n\n## Typical Workflow\n\n1. Discover subdomains, URLs, and IP resolution history from external data sources.\n2. Review identified assets (including per-IP hostname history on the target) and archived content.\n3. Analyze URLs and content for exposed credentials, secrets, and sensitive files.\n4. Investigate application endpoints and other security-relevant findings.\n5. Compare results across scans to identify newly discovered exposures.\n\n-----\n\n## Important Notice\n\n- **Not for production:** This tool focuses on functionality over hardened security. Use it exclusively in isolated, trusted environments.\n- **No built-in authentication:** Anyone with network access can view findings and trigger scans. Do NOT expose SCOPTIX to the public internet without your own access controls (e.g., VPN, reverse proxy).\n- **Third-party APIs and data:** VirusTotal and the Internet Archive impose their own terms, rate limits, and acceptable-use policies. This repository orchestrates queries and stores results locally; it is not a redistribution of upstream datasets.\n\n-----\n\n## Prerequisites\n\n- Node.js (LTS recommended) and npm\n- Git (to clone the repository)\n- **Either** Docker (recommended for Postgres + Redis) **or** your own PostgreSQL and Redis instances\n- One or more [VirusTotal](https:\u002F\u002Fwww.virustotal.com\u002F) API keys (required for VT-powered discovery)\n\n**Tested platform:** Ubuntu 26.04 (Docker and local dev workflows above). Other Linux distributions and macOS may work but are not routinely verified.\n\nOptional: SOCKS proxy (if outbound API or deep-fetch traffic must route through a proxy)\n\n> **Note on VirusTotal API Keys:**\n>\n> Community (Free) VirusTotal API keys are fully supported, although throughput is limited by VirusTotal's public API quotas (≈4 requests\u002Fminute, ≈500 requests\u002Fday, and ≈15,500 requests\u002Fmonth per key). If faster scans are needed, additional API keys can be configured, and SCOPTIX will automatically distribute requests across them. For example:\n>\n> * 1 API key ≈ 4 requests\u002Fminute\n> * 4 API keys ≈ 16 requests\u002Fminute\n\n## Getting started\n\nChoose one of two workflows:\n\n| Workflow | Best for | Command |\n|----------|----------|---------|\n| **Local dev (hot reload)** | Changing UI\u002Fworker code | `npm run dev:local` |\n| **Full Docker** | Try production-like stack without Node on host | `bash docker-start.sh` |\n\nBoth use the same `.env` file with **localhost** URLs for Postgres and Redis when developing on your machine. Docker Compose maps containers to `localhost:5432` and `localhost:6379`.\n\n### 1. Clone and install\n\n```bash\ngit clone \u003Cyour-clone-url>\ncd scoptix\nnpm install\n```\n\n### 2. Configure environment\n\n```bash\ncp .env.example .env\n```\n\nThe helper scripts can create `.env` automatically on first run (`scripts\u002Fensure-env.sh`).\n\nEdit `.env` and set at minimum:\n\n| Variable | Required | Purpose |\n|----------|----------|---------|\n| `DATABASE_URL` | Yes | PostgreSQL connection string (`postgresql:\u002F\u002Frecon:recon@localhost:5432\u002Frecon?schema=public` with Docker infra) |\n| `REDIS_URL` | Yes | Redis connection string (`redis:\u002F\u002F127.0.0.1:6379` with Docker infra) |\n| `APP_ENCRYPTION_KEY` | Recommended | 32-byte key (base64 or hex) for encrypting stored API keys |\n| `VT_SEED_API_KEY` | Optional | Seed a VirusTotal key during `npm run db:seed` (dev convenience) |\n\nGenerate an encryption key (Linux\u002FmacOS):\n\n```bash\nopenssl rand -base64 32\n```\n\nIf `APP_ENCRYPTION_KEY` is omitted, the app can auto-provision a key in the database on first use—but setting it explicitly is recommended so keys remain decryptable across deployments.\n\n### 3. Start services\n\n#### Option A — Local development (recommended)\n\nOne command starts Postgres + Redis in Docker, runs migrations on first use, then the Next.js dev server and scan worker with hot reload:\n\n```bash\nnpm run dev:local\n```\n\nOr step by step:\n\n```bash\nnpm run docker:infra          # Postgres + Redis only\nnpm run docker:wait           # wait until both are healthy\nnpm run setup                 # first time: migrate + seed\nnpm run dev:all               # dev server + worker\n```\n\nEquivalent shell wrapper: `bash docker-start-infra.sh` then `npm run dev:all`.\n\n#### Option B — Full stack in Docker\n\nBuilds and runs the app, worker, database setup, Postgres, and Redis in containers (no hot reload):\n\n```bash\nbash docker-start.sh\n```\n\nOpen [http:\u002F\u002Flocalhost:3000](http:\u002F\u002Flocalhost:3000).\n\n- Check status: `.\u002Fdocker-status.sh`\n- Logs: `npm run docker:logs`\n- Stop: `bash docker-stop.sh` or `npm run docker:down`\n\n#### Option C — Manual Postgres \u002F Redis\n\nIf you already run Postgres and Redis (not via this repo’s Compose file), point `DATABASE_URL` and `REDIS_URL` in `.env` at your instances, then:\n\n```bash\nnpm run setup\nnpm run dev:all\n```\n\n### 4. First scan\n\n1. Open **Settings → API & network** and add at least one VirusTotal API key (unless seeded).\n2. Confirm **VirusTotal** (and optionally **Wayback Machine**) are enabled under scan engines.\n3. Go to **Scans**, enter a domain, optionally enable deep scan, and start.\n\n## Docker & permissions\n\nDevelopment and Docker helper scripts in this repo were validated on **Ubuntu 26.04**.\n\n- **Postgres data** is stored in a named Docker volume (`scoptix_pg`), not a host bind mount—so you avoid UID\u002FGID permission issues on `.\u002Fdata` folders.\n- **Linux:** add your user to the `docker` group so you do not need `sudo` for Compose: `sudo usermod -aG docker $USER`, then log out and back in.\n- **Ports:** defaults are `5432` (Postgres) and `6379` (Redis). Override with `POSTGRES_PORT` \u002F `REDIS_PORT` in `.env` if those ports are already in use.\n\n## Useful npm scripts\n\n| Script | Purpose |\n|--------|---------|\n| `npm run dev:local` | Docker infra + migrate (if needed) + `dev:all` (one-shot local dev) |\n| `npm run docker:infra` | Start only Postgres + Redis |\n| `npm run docker:infra:setup` | Infra + wait + `setup` |\n| `npm run docker:up` \u002F `docker:down` | Full stack in Docker |\n| `npm run dev` | Next.js dev server |\n| `npm run worker` | BullMQ scan worker (required for scans to run) |\n| `npm run dev:all` | Dev server + worker via `concurrently` |\n| `npm run build` \u002F `npm run start` | Production build and server |\n| `npm run lint` | ESLint |\n| `npm run db:migrate` | Apply Prisma migrations |\n| `npm run db:push` | Push schema without migration files (dev shortcut) |\n| `npm run db:seed` | Seed extension rules and default settings |\n| `npm run setup` | `db:migrate` + `db:seed` |\n\n### Database migrations\n\nThe schema ships as a single Prisma migration: `prisma\u002Fmigrations\u002F0001_init`. Fresh installs (`npm run setup`, `bash docker-start.sh`) apply that file automatically.\n\n## Scan pipeline (overview)\n\nWhen both engines are enabled for a root-domain scan, the worker roughly follows:\n\n1. **VirusTotal — apex:** domain report, URL harvest, and passive DNS resolutions for the root domain.\n2. **VirusTotal — subdomains:** BFS expansion up to `maxSubdomains` (URLs and passive DNS per hostname).\n3. **VirusTotal — IP resolutions:** persist observed IPs for the scan and merge hostname↔IP sightings into the target’s global IP directory.\n4. **Wayback — apex:** CDX URL list for the root domain.\n5. **Wayback — subdomains:** CDX per discovered subdomain (rate-limited).\n6. **Consolidate:** dedupe URLs, assign extension categories, update target caches.\n7. **Analysis:** regex scan on URL strings; optional deep fetch + body scan for selected categories.\n\nSubdomain-only scans skip full apex expansion but still run enabled engines against the input hostname.\n\n## Contributing\n\nImprovements, bug reports, and security feedback are welcome as issues or pull requests. Please respect VirusTotal and Internet Archive terms of use when testing against live APIs.\n",2,"2026-06-11 04:11:36","CREATED_QUERY"]