[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-94452":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":13,"contributorsCount":13,"subscribersCount":13,"size":13,"stars1d":13,"stars7d":13,"stars30d":15,"stars90d":13,"forks30d":13,"starsTrendScore":13,"compositeScore":16,"rankGlobal":10,"rankLanguage":10,"license":17,"archived":18,"fork":18,"defaultBranch":19,"hasWiki":20,"hasPages":18,"topics":21,"createdAt":10,"pushedAt":10,"updatedAt":27,"readmeContent":28,"aiSummary":29,"trendingCount":13,"starSnapshotCount":13,"syncStatus":30,"lastSyncTime":31,"discoverSource":32},94452,"deident-wasm","ikuV\u002Fdeident-wasm","ikuV","Privacy transformation engine for structured datasets — pseudonymization and risk-assessed anonymization for CSV files, driven by a declarative YAML policy.","",null,"Rust",142,0,1,24,42.4,"Apache License 2.0",false,"main",true,[22,23,24,25,26],"anonymity","anonymization","encryption","privacy","pseudonymization","2026-08-24 04:01:22","# deident\n\n**Privacy transformation engine for structured datasets** — pseudonymization and\nrisk-assessed anonymization for CSV, JSONL, Parquet and DICOM, driven by a\ndeclarative YAML policy.\n\n```\n$ deident anonymize patients.csv --policy patients.yaml --out anon.csv --report report.json\nAnonymize complete: 12 row(s) in, 12 row(s) out (dataset 'patients-demo')\n  direct identifiers: patient_id (removed), full_name (removed), email (removed)\n  quasi-identifiers [age, zip, admission_date]: 5 equivalence class(es), min size 1, 1 unique row(s) (8.3%)\n  output: anon.csv\n  report: report.json\n```\n\n## Two modes, two very different promises\n\n| | `pseudonymize` | `anonymize` |\n|---|---|---|\n| Reversible? | **Yes** — with the key material | **No** — values are removed or generalized |\n| Output is still personal data? | **Yes.** Treat it as such. | Reduced risk, **not** zero risk |\n| What happens to direct identifiers | replaced with deterministic tokens | removed (or redacted) |\n| What happens to quasi-identifiers | kept unchanged | generalized\u002Fsuppressed per policy |\n| Typical use | joinable test\u002Fanalytics data, debugging with real structure | sharing data with reduced re-identification risk |\n\n> ⚠️ **No guarantees.** This tool performs *risk-assessed* anonymization: it reduces\n> re-identification risk and measures residual risk signals, but it cannot certify\n> anonymity — that always depends on external data and context it cannot observe.\n> Pseudonymized output remains personal data and is reversible by anyone holding the\n> key material, so protect keys separately from outputs.\n\n## What it does\n\n- **Formats** — CSV, JSONL\u002FNDJSON and Parquet, inferred from the file\n  extension. Input and output formats are independent, so a job converts while\n  it transforms. See [Formats](#formats).\n- **Column rules** — classify every column (`direct_identifier`,\n  `quasi_identifier`, `sensitive`, `utility`) and pick a strategy: tokenize,\n  remove, redact, bucket, truncate dates, keep a prefix. See\n  [Policy reference](#policy-reference).\n- **Content patterns** — find identifiers *inside* values (an IBAN in a\n  free-text note) with **16 built-in detectors** (email, IBAN, card, SSN, phone,\n  IP, URL, API key, passport, plate, IFSC, date of birth, plus heuristic name \u002F\n  address \u002F organization \u002F medical-term matchers) or your own regex, then detect,\n  redact, tokenize or replace them with structurally valid fakes. **Eight of the\n  sixteen are validated** by checksum or structural parse (mod-97, Luhn, real\n  calendar dates, IP parsing), so a loose pattern does not cost you false\n  positives. See [Built-in detectors](#built-in-detectors).\n- **Chained datasets** — process several files as one export with shared token\n  scoping, so foreign keys still join after pseudonymization. See\n  [Chained datasets](#chained-datasets).\n- **Sandboxed execution** — each job can run in its own WebAssembly sandbox\n  (Wasmtime + WASI): fresh store per job, one preopened directory, no network,\n  memory\u002FCPU\u002Ftime limits. See [Sandboxed execution](#sandboxed-execution).\n- **Encrypted mapping vault** — optionally record original→token mappings\n  under XChaCha20-Poly1305 for authorized re-identification. See\n  [Mapping vault and reversal](#mapping-vault-and-reversal).\n- **Risk report** — row counts, per-identifier actions, pattern findings and\n  equivalence-class statistics. See [Risk report](#risk-report).\n- **DICOM** — metadata de-identification of medical imaging instances, with\n  consistent UID remapping across a study. See [DICOM](#dicom).\n- **Policy lints** — warn about risky-but-valid policies before a job runs.\n  See [Policy lints](#policy-lints).\n- **Audit log** — append-only JSONL, metadata only. See\n  [Audit log](#audit-log).\n\nStatus: MVP, but a complete one — every feature above is implemented and\ntested. Streaming for very large datasets, richer vault workflows and more\nformats are on the [Roadmap](#roadmap).\n\n## Installation\n\nRequires a Rust toolchain (1.96+).\n\n```bash\ngit clone \u003Cthis-repo> && cd deident-wasm\n\n# install the `deident` binary onto your PATH\ncargo install --path crates\u002Fcli\n\n# to also sandbox jobs, build the guest module (see Sandboxed execution)\nrustup target add wasm32-wasip1\ncargo build -p deident-worker --target wasm32-wasip1 --release\n\n# — or just build and use it from target\u002Frelease\u002F\ncargo build --release\n.\u002Ftarget\u002Frelease\u002Fdeident --help\n```\n\n## Quick start\n\nThe repo ships a demo dataset and policy:\n\n```bash\n# reversible: tokenize direct identifiers, keep everything else\ndeident pseudonymize examples\u002Fdata\u002Fpatients.csv \\\n  --policy examples\u002Fpolicies\u002Fpatients.yaml \\\n  --out pseudo.csv\n\n# irreversible: remove\u002Fgeneralize identifiers, write a risk report\ndeident anonymize examples\u002Fdata\u002Fpatients.csv \\\n  --policy examples\u002Fpolicies\u002Fpatients.yaml \\\n  --out anon.csv \\\n  --report report.json\n```\n\nTo run against your own data you need a policy file that lists **every column**\nof your dataset — see [Policy reference](#policy-reference). `deident lint` will\ntell you if it looks risky.\n\n## CLI reference\n\n```\ndeident \u003CCOMMAND> [OPTIONS]\n```\n\n| Command | Description |\n|---|---|\n| `pseudonymize \u003CINPUT>...` | Reversibly tokenize direct identifiers (deterministic per dataset\u002Fpolicy) |\n| `anonymize \u003CINPUT>...` | Irreversibly remove\u002Fgeneralize identifiers and produce a risk report |\n| `chain \u003CMANIFEST> --mode \u003CMODE>` | Run several datasets as one chained export ([Chained datasets](#chained-datasets)) |\n| `lint \u003CPOLICY>` | Report risky-but-valid policy configurations ([Policy lints](#policy-lints)) |\n| `vault export \u003CVAULT> --policy \u003CFILE>` | Decrypt a mapping vault to CSV ([Mapping vault](#mapping-vault-and-reversal)) |\n| `reverse \u003CINPUT> --vault \u003CFILE> --policy \u003CFILE> --out \u003CFILE>` | Re-identify tokenized values using a vault |\n| `dicom \u003CINPUT> --policy \u003CFILE> --out \u003CPATH>` | De-identify DICOM instance metadata, file or directory ([DICOM](#dicom)) |\n| `help [COMMAND]` | Print help |\n\nOptions for `pseudonymize` \u002F `anonymize`:\n\n| Option | Required | Description |\n|---|---|---|\n| `\u003CINPUT>...` | yes | Input file(s); format inferred from the extension (`.csv`, `.jsonl`\u002F`.ndjson`, `.parquet`). Several inputs run concurrently, one sandbox each ([Parallel execution](#parallel-execution)) |\n| `--policy \u003CFILE>` | yes | Policy YAML describing field classes and strategies |\n| `--out \u003CPATH>` | yes | Output file; its extension selects the output format. With several inputs, a **directory** |\n| `--report \u003CFILE>` | no | Write the JSON risk report here |\n| `--vault \u003CFILE>` | no | Write the encrypted mapping vault here (only if the job produces reversible values) |\n| `--split \u003CN>` | no | Split one dataset across `N` sandboxes and merge the results ([Parallel execution](#parallel-execution)) |\n| `--jobs \u003CN>` | no | Maximum sandboxes running at once (default: cores, capped at 8) |\n| `--no-lint` | no | Skip the pre-flight policy lint |\n| `--deny-lints` | no | Refuse to run when a warning-level lint fires |\n\nEngine options (also accepted by `chain`):\n\n| Option | Default | Description |\n|---|---|---|\n| `--engine \u003CENGINE>` | `auto` | `auto` sandboxes when a worker module is available and falls back in-process with a warning; `wasm` requires the sandbox; `native` runs in-process |\n| `--worker \u003CFILE>` | discovery | Compiled worker module (see discovery order below) |\n| `--max-memory-mib \u003CN>` | `256` | Guest memory limit in MiB (sandbox only) |\n| `--timeout-secs \u003CN>` | `30` | Job wall-clock timeout in seconds (sandbox only) |\n| `--fuel \u003CN>` | scaled | Fixed CPU budget in Wasmtime fuel units; the default scales with input size |\n| `--no-fuel` | off | Disable fuel metering (the wall-clock timeout still applies) |\n| `--audit-log \u003CFILE>` | off | Append one JSONL audit record per job ([Audit log](#audit-log)) |\n\n`lint` also accepts `--mode \u003CMODE>` (restrict to lints relevant for one mode),\n`--json`, and `--deny` (exit non-zero on any warning).\n\nGlobal: `-h, --help`, and `-v` \u002F `-V` \u002F `--version`.\n\nEnvironment:\n\n| Variable | Purpose |\n|---|---|\n| `DEIDENT_KEY` (or whatever the policy's `key.env` names) | Secret for key derivation |\n| `DEIDENT_WORKER_WASM` | Path to the worker module for sandboxed execution |\n| `RUST_LOG` | Log verbosity on stderr, e.g. `RUST_LOG=debug` (default `info`) |\n\nExit codes: `0` success, non-zero on any failure (bad policy, unreadable input,\nunlisted column, missing key, a failed chain job, `--deny-lints` with warnings).\nHuman summary goes to stdout, logs and lint warnings to stderr.\n\n## Formats\n\nThe format of each file is inferred from its extension, independently for input\nand output — so a job can convert while it transforms:\n\n| Extension | Format | Notes |\n|---|---|---|\n| `.csv` | CSV | Header row required. A leading UTF-8 BOM (Excel writes one) is stripped, so the first column still matches its policy field |\n| `.jsonl`, `.ndjson` | JSON Lines | One **flat** object per line. The first record defines the columns; later records may omit keys (treated as empty) but not add new ones. Nested objects\u002Farrays are rejected rather than silently flattened. Numbers and booleans keep their JSON type when their value is unchanged; generalized values (`\"30-39\"`) become strings; empty becomes `null` |\n| `.parquet`, `.pq` | Apache Parquet | Column types are re-inferred from the transformed values, so untouched numeric columns stay `Int64`\u002F`Float64` while generalized ones become `Utf8`. Not available inside the sandbox (see below) |\n\n```bash\n# read Parquet, write JSONL, transforming on the way through\ndeident anonymize events.parquet --policy p.yaml --out events.jsonl\n```\n\nBoth Parquet directions hold the table in memory (its footer-based layout makes\ntrue streaming impractical); CSV and JSONL stream row by row.\n\n**Column names are matched exactly.** A policy field that matches no column is\ninert — if it named a direct identifier, that identifier would be copied through in\nthe clear. Two things guard against it: the default `on_unlisted: error` fails the\njob on any column the policy does not cover, and an unmatched *field* is reported\nas a warning that names a case- or whitespace-only near miss when one exists\n(`patient_id` vs `Patient_ID`).\n\n**Outputs are published, not written in place.** Each job writes to a temporary\nsibling file and moves it into place only after the transformation has run to\ncompletion, so a job that fails halfway leaves no truncated file for a downstream\nconsumer to mistake for a finished dataset. The same applies to the mapping vault:\na failed job publishes neither.\n\n## Sandboxed execution\n\n`--engine wasm` (and `auto`, the default, when a worker module is available)\nruns each job inside its own WebAssembly sandbox instead of in-process. The\nexact same transformation code runs either way — the core crate compiles into\nboth — and outputs are byte-identical, verified by tests. The sandbox adds an\nisolation layer around the parsing\u002Ftransformation logic:\n\n- **Fresh instance per job** — a new Wasmtime store and WASI context every\n  time; no state survives from one job to the next.\n- **One directory, nothing else** — the guest sees a single preopened job\n  workspace containing a *copy* of the input; your real filesystem paths never\n  reach it. Attempts to read outside (absolute paths, `..` escapes) fail.\n- **No network** — the WASI context simply has no socket capability.\n- **Minimal environment** — only the one key variable a pseudonymize policy\n  names is passed through, and only if set.\n- **Resource limits** — guest memory (`--max-memory-mib`), a wall-clock\n  timeout enforced by epoch interruption (`--timeout-secs`), and a CPU budget\n  in Wasmtime fuel units. The fuel budget **scales with input size** by\n  default (a fixed budget would either starve large jobs or be meaningless for\n  small ones); override with `--fuel \u003CN>` or turn it off with `--no-fuel`.\n\nSandboxing *reduces* the blast radius of malformed inputs and future untrusted\nplugins; it is a mitigation, not an absolute security boundary.\n\n**Format caveat:** the sandbox build deliberately excludes Parquet — the arrow\nstack inflates the guest module from ~1.9 MB to ~7.4 MB, which Wasmtime then\nhas to JIT-compile for every job. CSV and JSONL work in the sandbox; Parquet\njobs run in-process (`--engine auto` switches automatically and says so).\n\nBuild the worker module once, then use it:\n\n```bash\nrustup target add wasm32-wasip1\ncargo build -p deident-worker --target wasm32-wasip1 --release\n\ndeident anonymize input.csv --policy p.yaml --out out.csv --engine wasm\n```\n\nThe worker module is found in this order: `--worker \u003CFILE>`, then\n`$DEIDENT_WORKER_WASM`, then `deident-worker.wasm` next to the `deident`\nbinary, then the local cargo build under `target\u002Fwasm32-wasip1\u002F`. For\ndeployment, copy `deident-worker.wasm` next to the installed binary.\n\n## Parallel execution\n\nTwo independent axes, both giving every job its own sandbox with its own `Store`,\nWASI context and resource limits. The guest module is compiled once and shared;\nnothing else is.\n\n**Several datasets at once** — pass more than one input. `--out` then names a\ndirectory and each output keeps its input's file name:\n\n```bash\ndeident pseudonymize patients.csv visits.csv labs.jsonl \\\n  --policy policy.yaml --out .\u002Fout --jobs 3\n```\n\nResults are reported in the order the inputs were given, whatever order they\nfinished in. One dataset failing does not stop the others; the command exits\nnon-zero and names the input that failed.\n\nBecause `--report` and `--vault` each name a single file, they are refused with\nseveral inputs — the per-dataset artifacts would overwrite each other. Use\n[`deident chain`](#chained-datasets) when you want per-dataset reports, or run the\ndatasets separately.\n\n**One large dataset across N sandboxes** — `--split`:\n\n```bash\ndeident pseudonymize huge.csv --policy policy.yaml --out out.csv \\\n  --report risk.json --split 8 --jobs 8\n```\n\nThe input is divided by rows into `N` staged chunks (each CSV chunk repeats the\nheader, so it is a valid file in its own right), each chunk runs in its own\nsandbox, and the outputs are concatenated in order. The result is **byte-identical**\nto an unsplit run — tokens are a deterministic function of the key and the value,\nso chunks agree without coordinating. On a 200k-row file, 8 sandboxes cut wall\nclock roughly 3x.\n\nConstraints:\n\n- **Line-oriented formats only** (`.csv`, `.jsonl`\u002F`.ndjson`). Parquet is columnar\n  with a footer, so a byte range of it is not a valid file.\n- **Not with `--vault`.** Each chunk would write its own vault with overlapping\n  entries, and merging encrypted mapping files is not implemented.\n- `--split` applies to a single dataset. With several inputs each one already gets\n  its own sandbox.\n- Fewer rows than chunks, or `--split 1`, silently runs as one job.\n\n### How a split report is merged\n\nRow counts and pattern-match counts are additive, so they are summed. The\n**equivalence-class statistics are not**, and summing them would be wrong in a way\nthat matters:\n\n> A quasi-identifier combination appearing once in chunk A and once in chunk B is\n> one class of size two — not two classes of size one. Summing per-chunk figures\n> would report far more unique rows than the dataset contains, i.e. it would\n> *overstate* re-identification risk, and someone widening their buckets in\n> response would be chasing an artefact of the chunking.\n\nSo they are not merged at all. Once the chunk outputs are concatenated, the host\nrecomputes them over the **whole** output using the same code path a single-job run\nuses. That costs one extra pass and makes the figures host-attested rather than\nassembled from fragments. A split run's report carries a warning saying so, and its\n`unique_rows`, `equivalence_classes` and `k_thresholds` match an unsplit run\nexactly.\n\n## Policy reference\n\nA policy is a YAML file that classifies every column and configures how each mode\ntreats it. Complete annotated example:\n\n```yaml\nversion: 1                  # required; only 1 is supported\ndataset: patients-demo      # required; scopes key derivation (see Key management)\n\nkey:                        # required for pseudonymize, ignored by anonymize\n  env: DEIDENT_KEY          # name of the env var holding the secret (preferred)\n  inline: \"demo-secret\"     # fallback secret — demos\u002Ftests only, always warned\n\non_unlisted: error          # what to do with CSV columns not listed below:\n                            #   error  – fail the job (default, deny-by-default)\n                            #   keep   – pass through unchanged + warning\n                            #   remove – drop the column + warning\n\nfields:\n  - name: patient_id                # column name, must match the CSV header\n    class: direct_identifier        # see field classes below\n    pseudonymize:                   # optional, pseudonymize mode only\n      prefix: \"pid_\"                # cosmetic token prefix\n\n  - name: email\n    class: direct_identifier        # no config needed: tokenized in pseudonymize\n                                    # mode, removed in anonymize mode by default\n\n  - name: age\n    class: quasi_identifier\n    anonymize:                      # anonymize-mode strategy (see below)\n      strategy: bucket\n      width: 10\n\n  - name: zip\n    class: quasi_identifier\n    anonymize:\n      strategy: keep_prefix\n      chars: 3\n      pad: \"*\"                      # optional, default '*'\n\n  - name: admission_date\n    class: quasi_identifier\n    anonymize:\n      strategy: date_truncate\n      granularity: year             # year | year_month\n\n  - name: diagnosis\n    class: sensitive\n\n  - name: notes\n    class: utility\n\npatterns:                           # content-pattern rules, see below\n  - name: iban\n    builtin: iban\n    fields: [notes]\n    action: redact\n```\n\nUnknown YAML keys anywhere in the policy are rejected (typos fail fast).\n\n### Field classes\n\n| Class | Meaning | Pseudonymize mode | Anonymize mode |\n|---|---|---|---|\n| `direct_identifier` | Identifies a person on its own (name, email, ID number) | tokenized | removed (default) or the configured strategy |\n| `quasi_identifier` | Identifying in combination (age, zip, dates) | kept unchanged | configured strategy; kept + **warning** if none |\n| `sensitive` | Sensitive payload (diagnosis, salary) | kept unchanged | kept, unless a strategy is configured |\n| `utility` | Analytic utility only | kept unchanged | kept, unless a strategy is configured |\n\n### Anonymization strategies\n\nSet under a field's `anonymize:` block; applied only in `anonymize` mode.\n\n| `strategy` | Parameters | Example |\n|---|---|---|\n| `remove` | — | column is dropped entirely |\n| `redact` | `replacement` (default `\"REDACTED\"`) | `Alice` → `REDACTED` |\n| `bucket` | `width` (positive integer) | width 10: `34` → `30-39`, `-3` → `-10--1`; floats floored |\n| `date_truncate` | `granularity`: `year` \\| `year_month` | `2024-03-14` → `2024` or `2024-03`; ISO dates\u002Ftimestamps only |\n| `keep_prefix` | `chars`, `pad` (default `*`) | chars 3: `81549` → `815**` |\n\nValues that don't fit their strategy (e.g. `bucket` on `n\u002Fa`, `date_truncate` on\n`14.03.2024`) are suppressed to `*` and counted in the report warnings — a single\nbad cell never fails the job. Empty cells always pass through empty.\n\n### Pseudonymization options\n\nSet under a field's `pseudonymize:` block; applied only in `pseudonymize` mode and\nonly to `direct_identifier` fields (which are tokenized with or without this block).\n\n| Key | Description |\n|---|---|\n| `prefix` | Cosmetic prefix prepended to the token, e.g. `pid_` → `pid_21134bb99aee85cb...` |\n| `domain` | Identity domain the token is derived in; defaults to the column name. Give differently named columns in different files (e.g. `patient_id` and `patient_ref`) the same domain so the same value yields the same token — foreign keys survive (see [Chained datasets](#chained-datasets)) |\n\n### Content-pattern rules\n\nColumn-level rules can't reach identifiers hiding *inside* values — an IBAN in a\nfree-text `notes` column, an email in a comment. `patterns:` rules scan cell\ncontent and run in **both modes**, after the column-level transform:\n\n```yaml\npatterns:\n  - name: iban            # rule name; also the default redaction label\n    builtin: iban         # or a custom regex — exactly one of the two:\n    # regex: '\\b[A-Z]{2}[0-9]{2}[A-Z0-9]{11,30}\\b'\n    fields: [notes]       # columns to scan; omit = every column in the output\n    action: redact        # detect | redact | token | mock\n    # replacement: \"[IBAN]\"   # redact only; default \"[\u003CNAME>]\"\n    # prefix: \"ib_\"           # token only\n    # mock: iban              # mock only; defaults to the `builtin` shape\n```\n\n| `action` | Effect |\n|---|---|\n| `detect` | Only count matches for the risk report; values stay in the output (**a warning is recorded**) |\n| `redact` | Replace each match with a fixed label (default `[IBAN]`-style) |\n| `token` | Replace each match with a deterministic keyed token — same IBAN, same token, so joins\u002Fgrouping survive. Requires a key source |\n| `mock` | Replace each match with a deterministic, **structurally valid** fake of the same shape (see below). Requires a key source |\n\nBoth `token` and `mock` are **reversible with the key material**: in anonymize\nmode the report and the lints flag the affected output as pseudonymous rather\nthan anonymous.\n\n#### Format-preserving mocks\n\n`action: mock` is for downstream systems that validate their input and would\nchoke on `[IBAN]` or a hex token. Mocks are derived from the same keyed hash as\ntokens, so they are deterministic and stable — the same input always yields the\nsame mock, and joins on the mocked value keep working.\n\n| Shape | What is preserved | What is generated |\n|---|---|---|\n| `iban` | Country code and length | Correct mod-97 (ISO 7064) check digits, so validators accept it |\n| `credit_card` | Length and separators | Valid Luhn check digit, forced into the `999x` test IIN range so it cannot collide with a real issuer |\n| `phone` | Punctuation and digit count | New digits, leading digit kept non-zero |\n| `email` | Nothing of the original | A random local part at `example.com` (RFC 2606 documentation domain) |\n\nThe shape comes from `builtin:`, or set `mock:` explicitly when mocking a\ncustom `regex:` rule. A mock is a **pseudonym with a prettier shape**, not\nanonymization: it is recorded in the mapping vault exactly like a token, and\nanyone with the key can recompute it.\n\n> ⚠️ **Mocks collide, and sooner than you would guess.** Preserving a format\n> bounds the value space: a 9-digit phone number has only 10^9 possible mocks, so\n> by the birthday bound two different numbers start sharing a mock at around\n> **31,000 distinct values** — a small dataset. Email mocks (26^10) and IBANs are\n> far roomier; phone and short card shapes are the risky ones.\n>\n> When it happens, two identities share one value in the output and the mapping\n> stops being invertible. The tool does not paper over it:\n>\n> - the risk report names the pattern and counts the colliding values, at\n>   transformation time rather than months later;\n> - `deident reverse` **refuses** an ambiguous value, leaving it in place and\n>   exiting non-zero, rather than restoring a value that may belong to someone\n>   else.\n>\n> Use `action: token` (128-bit, no practical collisions) wherever the mapping has\n> to stay reversible. Mocks are for feeding format-validating systems, not for\n> round-tripping identities.\n\nDropped and tokenized columns are never scanned (nothing left to find).\n\n### Built-in detectors\n\nSixteen detectors, grouped by **how much a match can be trusted**. That grouping\nis the important part: it stops a heuristic guess from being mistaken for a\nverified identifier.\n\n**Eight of the sixteen are validated** beyond their pattern — the match must also\npass a checksum or a structural parse before it counts.\n\n| Detector | Example | Class | Validated by |\n|---|---|---|---|\n| `email` | `user@example.com` | precise | ✅ RFC 5321 structure: single `@`, length limits, dotted domain, alphabetic TLD |\n| `iban` | `DE89 3704 0044 0532 0130 00` | precise | ✅ mod-97 (ISO 7064) check digits |\n| `credit_card` | `4111 1111 1111 1111` | precise | ✅ Luhn **plus** card length (13–19) and issuer prefix (2–6) |\n| `ip_address` | `192.168.1.1`, `2001:db8::1` | precise | ✅ parsed as a real address (`std::net::IpAddr`) |\n| `url` | `https:\u002F\u002Finternal.company.com` | precise | ✅ known scheme, plausible host, no whitespace |\n| `api_key` | `AKIA…`, `sk-proj-…`, `github_pat_…`, `xoxb-…`, `glpat-…` | precise | — opaque by design |\n| `ifsc` | `HDFC0001234 000123456789` | precise | — no checksum exists |\n| `ssn` | `123-45-6789` | moderate | ✅ US allocation rules (area ≠ 000\u002F666\u002F9xx, group ≠ 00, serial ≠ 0000) |\n| `date_of_birth` | `15\u002F03\u002F1990`, `March 15, 1990` | moderate | ✅ a real calendar date (rejects `31\u002F02`, leap years honoured) |\n| `phone` | `+1-555-0123`, `+91 98765 43210` | moderate | ✅ E.164 limits: 7–15 digits, no `+0` country code |\n| `license_plate` | `MH 12 AB 1234` | moderate | — no check digit |\n| `passport` | `J1234567` | moderate | — no check digit |\n| `address` | `123 MG Road, Pune 411001` | **heuristic** | — not verifiable |\n| `organization` | `Apollo Hospital`, `HDFC Bank` | **heuristic** | — not verifiable |\n| `medical_term` | `diabetes`, `cardiac arrest` | **heuristic** | — gazetteer membership only |\n| `person_name` | `Dr. Priya Sharma`, `John Smith` | **heuristic** | — not verifiable |\n\nRows are listed in **execution order**, which is load-bearing: rules run in\nsequence over the same value, so specific detectors must precede greedy ones.\n`ssn` and `date_of_birth` come before `phone` (which otherwise swallows both),\nand `person_name` runs last because its bare-capitalised-pair alternative\notherwise claims `Apollo Hospital` and `Cardiac Arrest`.\n\nThe eight unvalidated ones are honest gaps, not oversights: a passport number and\na licence plate carry no check digit, an API key is opaque, and a heuristic is a\nheuristic. A test asserts the validated set is *exactly* those eight, so the table\ncannot drift into claiming verification the code does not perform.\n\n- **precise** — distinctive syntax, and five of the seven are validated. Safe to\n  redact unattended.\n- **moderate** — a recognisable shape that innocent data also has; three of the\n  five are validated. Expect some false positives; read the report.\n- **heuristic** — a stand-in for named entity recognition, which this tool does\n  **not** have. These are title-based patterns, suffix lists and a small\n  gazetteer. They produce false positives *and* miss real entities. They default\n  to `detect` in presets, every report says so, and setting one to modify data\n  triggers the `heuristic-pattern-modifies-data` lint. Treat them as \"show me\n  where to look\", never as \"this text is now clean\".\n\n#### Validation\n\nValidated detectors apply their check to **every match**, so a loose regex buys\nrecall without paying for it in false positives — a long order number matches the\ncard shape but fails Luhn, so it is not reported as a card.\n\nRejected matches are **left untouched and counted**, and the report names the\ncheck that rejected them:\n\n> `pattern 'credit_card' rejected 1 match(es) that had the right shape but failed\n> Luhn + card length\u002Fprefix validation, and left them unchanged. Set validate: none\n> to treat them as identifiers anyway (at the cost of false positives)`\n\nThat matters for test data: invented card numbers usually fail Luhn, so\n`validate: none` is the right choice when you want every card-shaped string\nflagged regardless. Override per rule with `validate: none` (or name a different\nvalidator) on any `patterns` entry.\n\n**Validators are deliberately conservative** — they reject only what is definitely\nnot the thing. Being too strict produces false *negatives*, a real identifier\npassing through silently, which is worse than a false positive a human dismisses.\nTwo consequences worth knowing:\n\n- Ambiguous dates are accepted under **either** reading, because `03\u002F04\u002F1990` is\n  a real date as both DD\u002FMM and MM\u002FDD and guessing wrong would drop a genuine one.\n- `date_of_birth` checks only that the date *exists*, not that it is a plausible\n  birth date. Rejecting future dates would drop appointment dates that a user\n  wants removed.\n\nTwo verification opportunities are deliberately left open. Indian **licence-plate\nstate codes** would be real validation, but would silently narrow the detector to\none country and break the European formats its pattern also matches — that\nbelongs in per-locale pattern packs. **GitHub tokens carry a CRC32 checksum** in\ntheir final characters, which is checkable but vendor-specific.\n\n#### Presets\n\nRather than listing sixteen rules, enable a whole class:\n\n```yaml\npresets:\n  - { preset: precise,   action: redact }   # checksum-verified: act on them\n  - { preset: moderate,  action: redact }   # read the report afterwards\n  - { preset: heuristic, action: detect }   # report only, for human review\n```\n\n`preset: all` covers everything. An explicit `patterns` entry always wins over a\npreset of the same detector name, so you can enable a class and still tune one\nmember of it. A complete example ships in\n[examples\u002Fpolicies\u002Fdetect-all.yaml](examples\u002Fpolicies\u002Fdetect-all.yaml).\n\nRules run in sequence over the same value, so two rules using the same detector\nwould mean the first one's replacement hides the second's matches — the\n`duplicate-builtin-detector` lint catches that.\n\n## Key management\n\nPseudonym tokens are 128-bit BLAKE3 keyed hashes. The key is derived from your\nsecret **and the policy's `dataset` name**, and the hash input includes the column\nname. Consequences:\n\n- **Same secret + same policy ⇒ same tokens.** Runs are repeatable, and repeated\n  exports of the same dataset stay joinable on their tokens.\n- The same value produces **different tokens in different columns and different\n  datasets** — tokens can't be used to link across datasets by accident.\n- Without the secret, tokens cannot be reversed or recomputed. **Whoever has the\n  secret can re-identify.** Store it in a secret manager, never next to the output.\n\nProvide the secret via the environment variable named in `key.env`:\n\n```bash\nexport DEIDENT_KEY=\"$(your-secret-manager get deident-prod)\"\ndeident pseudonymize ...\n```\n\n`key.inline` embeds the secret in the policy file — useful for demos and tests,\nunsafe for production. Every run using it records a warning in the report and\ntriggers the `inline-key` lint.\n\n### Resolution is fail-closed\n\nA policy may declare both `env` and `inline`. If the named environment variable is\n**unset or empty**, the run **fails** rather than quietly using the inline value:\n\n```yaml\nkey:\n  env: DEIDENT_KEY\n  inline: \"dev-only-secret-do-not-use-in-production\"\n```\n\n```\n$ deident pseudonymize in.csv --policy above.yaml --out out.csv\nerror: key error: environment variable 'DEIDENT_KEY' is unset or empty. The policy\nalso carries an inline key, but falling back to it silently would tokenize\nproduction data under a development secret — export the variable, or set\n`key.allow_inline_fallback: true` if that is genuinely what you want\n```\n\nA forgotten `export` would otherwise produce output that *looks* correctly\npseudonymized but is reversible by anyone holding the policy file, and would not\njoin with earlier exports. Opt in explicitly if you want the old behavior:\n\n```yaml\nkey:\n  env: DEIDENT_KEY\n  inline: \"dev-only-secret-do-not-use-in-production\"\n  allow_inline_fallback: true   # demos and tests only\n```\n\nThe fallback then happens and is recorded as a warning in the report — identically\nwhether the job ran in-process or in the sandbox.\n\n### Strength floor\n\nSecrets shorter than 32 bytes are rejected. Longer secrets that look like\npassphrases (few distinct byte values) are accepted with a warning; prefer\n`openssl rand -hex 32`.\n\nKey material is **purpose-separated**: the vault encryption key is derived from\nthe same secret under a different KDF context, so the vault key cannot forge\ntokens and the token key cannot decrypt a vault.\n\n## Mapping vault and reversal\n\nTokens are recomputable from the key alone, so a vault is optional. It exists\nfor the workflow where an authorized party needs to reverse *specific* values\nwithout holding a re-derivation pipeline — and it is the only way to reverse\nmocks and pattern matches conveniently.\n\n```bash\n# 1. pseudonymize, recording the mappings\ndeident pseudonymize patients.csv --policy p.yaml --out pseudo.csv --vault vault.jsonl\n\n# 2. later, with authorization: inspect the mappings\ndeident vault export vault.jsonl --policy p.yaml --out mappings.csv\n\n# 3. or reverse a whole file in place\ndeident reverse pseudo.csv --vault vault.jsonl --policy p.yaml --out restored.csv\n```\n\nHow it is protected:\n\n- Every entry is encrypted with **XChaCha20-Poly1305** under a key derived from\n  your secret and the dataset name. The file's header (format, version,\n  dataset) stays readable; the mappings do not.\n- Nonces are **synthetic** — derived from the key and the plaintext rather than\n  randomly. That keeps vault files reproducible and makes appends safe against\n  nonce reuse. The trade-off is that identical entries produce identical\n  ciphertext, revealing that two lines map the same value; for a deterministic\n  mapping table that equality is inherent to the design.\n- The AEAD tag is verified on read, so a wrong key or a tampered file **fails\n  loudly** instead of decrypting to garbage.\n\nA vault is written only when the job actually produces reversible values\n(pseudonymize mode, or `token`\u002F`mock` patterns); otherwise the report says so\nand no file is created.\n\n> ⚠️ **A vault is a re-identification table.** It is as sensitive as the\n> original data. Store it separately from the output, under stricter access\n> control, and treat `vault export` and `reverse` as privileged operations —\n> their output contains original personal data again.\n\n## DICOM\n\nMedical imaging instances are not tabular — a DICOM object is a nested,\ntag-keyed attribute tree with typed value representations, sequences, a separate\nfile-meta header and a pixel payload. So DICOM gets its own policy dialect and\nits own command, while reusing the same key derivation, tokenization, mocks,\nmapping vault and audit log.\n\n```bash\n# a single instance\ndeident dicom study\u002Fimage-001.dcm --policy dicom.yaml --out deid\u002Fimage-001.dcm\n\n# or a whole directory tree, recursively, with one shared identity scope\ndeident dicom study\u002F --policy dicom.yaml --out deid\u002F --report deid.json --vault vault.jsonl\n```\n\n### Scope — read this first\n\n> ⚠️ **This is not DICOM PS3.15 Annex E conformance.** It implements a *curated\n> core* of the Basic Application Level Confidentiality Profile plus structural\n> rules, and every report says so. If you need certified conformance you must\n> extend the policy's tag list and validate it against your own data.\n>\n> ⚠️ **Burned-in pixel PHI is detected and flagged, never removed.** Ultrasound\n> frames, secondary captures and scanned documents routinely render patient\n> details into the image itself. Cleaning that requires OCR and cannot be made\n> reliable, so this tool refuses to claim it. Every run prints the caveat and\n> reports a `pixel_risk` level with its reasoning.\n\n### How coverage works\n\nThree layers, highest precedence first:\n\n1. **Explicit `tags:` rules** in your policy.\n2. **The selected profile** (`basic` — the curated Annex E core).\n3. **Structural rules** that catch whole *classes* of attribute rather than\n   named instances: every person-name (`PN`) attribute, every identity UID,\n   every private attribute (odd group — unknown vendor semantics), and the\n   curve\u002Foverlay groups.\n\nThat third layer is deliberate. Transcribing ~500 Annex E rows from memory would\nbe error-prone, and a missed row means PHI survives. Rules keyed on VR and tag\nstructure fail *safe* — they remove what they don't recognise — and the curated\ntable then handles the well-known core exactly.\n\n### Actions\n\n| Action | Annex E | Effect |\n|---|---|---|\n| `remove` | `X` | Delete the attribute |\n| `empty` | `Z` | Keep the attribute, zero-length |\n| `replace` | `D` | Fixed literal (`value:`) |\n| `pseudonymize` | `D` | Deterministic keyed pseudonym; `mock: person_name` produces a readable `Family^Given` instead of a hex token |\n| `uid` | `U` | New UID, **consistently remapped** — the same original UID becomes the same replacement in every instance of the study |\n| `date_shift` | — | Shift by a deterministic per-subject offset, so intervals survive |\n| `date_truncate` | — | Truncate to year or year-month (padded to stay a valid `DA`) |\n| `clean_text` | `C` | Run the policy's [content-pattern rules](#content-pattern-rules) over the text |\n| `keep` | `K` | Leave untouched |\n\nTags are addressed by standard keyword (`PatientName`) or numerically\n(`(0010,0010)`). Replacement UIDs use the `2.25.\u003Cdecimal>` arc that DICOM PS3.5\nreserves for UUID-derived OIDs, so no registered organisational root is needed.\n\nA complete annotated example ships in\n[examples\u002Fpolicies\u002Fdicom-basic.yaml](examples\u002Fpolicies\u002Fdicom-basic.yaml).\n\n### What survives, and why\n\n`PatientSex` and `PatientAge` are **kept** by the basic profile because they are\nclinically load-bearing — but they are quasi-identifiers, and the report says so.\nFormat-identifying UIDs (`SOPClassUID`, `TransferSyntaxUID`) are never remapped;\ndoing so would make the file unreadable. Pixel data and image geometry pass\nthrough untouched.\n\nUID remapping intentionally **breaks references from outside the processed set**\n— a PACS or a report citing the original UIDs will no longer resolve.\n\n### Test data\n\nPublic DICOM collections (TCIA, pydicom-data, GDCM) are *already*\nde-identified, which makes them unable to demonstrate that a de-identifier\nworks — there is no PHI left to remove. So the crate generates its own fixtures\nwith identifiers planted in known attributes, including one nested inside a\nsequence and one in a private block:\n\n```bash\ncargo run -p deident-dicom --example gen_fixtures -- .\u002Fstudy 3\n```\n\nThe test suite runs against these and asserts at the **byte level** that no\nplanted identifier survives anywhere in the output file.\n\nDICOM jobs run in-process: the wasm guest does not carry the DICOM parser (the\nsame module-size trade-off as Parquet). Since DICOM parsers are historically a\nCVE-rich surface, sandboxing this path is on the roadmap.\n\n## Policy lints\n\nA policy can be perfectly valid and still not do what its author intended — a\nquasi-identifier with no generalization, a secret pasted into the file,\ndeny-by-default switched off. `deident lint` reports those:\n\n```bash\ndeident lint examples\u002Fpolicies\u002Fpatients.yaml --mode anonymize\ndeident lint policy.yaml --json          # machine-readable\ndeident lint policy.yaml --deny          # exit non-zero on any warning\n```\n\nLints also run automatically before every job (warnings to stderr). Use\n`--no-lint` to skip them, or `--deny-lints` to refuse to run when a warning\nfires — useful in CI.\n\nTwo levels: **warning** (likely a privacy problem) and **advice** (legitimate\nin many setups). Current rules include: `inline-key`, `missing-key-source`,\n`unlisted-columns-kept`, `unlisted-columns-removed`, `qi-without-strategy`,\n`direct-identifier-partially-kept`, `ineffective-bucket`,\n`free-text-without-patterns`, `no-direct-identifiers`, `no-quasi-identifiers`,\n`detect-only-pattern`, `reversible-pattern-in-anonymize`.\n\nLints are heuristics, not a compliance check — a clean lint run does not mean a\npolicy is adequate for your data.\n\n## Audit log\n\n`--audit-log \u003CFILE>` appends one JSON object per job:\n\n```json\n{\"timestamp\":\"2026-08-04T09:12:33Z\",\"job_id\":\"…\",\"mode\":\"anonymize\",\"engine\":\"wasm\",\n \"report_provenance\":\"host-attested\",\"dataset\":\"patients-demo\",\"policy_hash\":\"9f2c…\",\n \"input_path\":\"in.csv\",\"output_path\":\"out.csv\",\n \"status\":\"succeeded\",\"rows_read\":12,\"rows_written\":12,\"warnings\":1,\"error\":null,\n \"limits\":{\"max_memory_bytes\":268435456,\"timeout_ms\":30000,\"fuel\":2000000000}}\n```\n\nIt is deliberately **metadata only** — no cell values — so it can be retained\nand shipped to a SIEM without inheriting the sensitivity of the data it\ndescribes. Specifically:\n\n- `policy_hash` is a BLAKE3 fingerprint of the policy with the `key` block\n  removed, so an auditor can prove which policy produced an output and the\n  fingerprint does not commit to an inline secret.\n- `error` is capped at 300 characters and flattened to one line. Failure text is\n  assembled from whatever went wrong, so it can quote a policy value or a column\n  name from the input; the log keeps a bounded summary while the operator still\n  sees the full message on stderr.\n- `report_provenance` records who authored the risk figures — `host-attested`\n  means the host computed or verified them. A compromised worker could otherwise\n  report clean counts over untransformed data, and a consumer needs to know which\n  it is holding.\n\nRecords are written for failed jobs too, and it works identically for native,\nsandboxed, split and chained runs.\n\n## Chained datasets\n\nReal exports are rarely one file: `patients.csv` plus `visits.csv` that\nreferences it. A chain manifest runs them as one unit so **foreign keys survive\npseudonymization**:\n\n```yaml\n# hospital.yaml — paths are resolved relative to this file\nversion: 1\nname: hospital-demo\n# Optional overrides forced onto every job policy:\n# dataset: hospital-export    # one token scope for all files\n# key: { env: DEIDENT_KEY }   # one key source for all files\njobs:\n  - name: patients\n    input: ..\u002Fdata\u002Fpatients.csv\n    policy: ..\u002Fpolicies\u002Fpatients.yaml\n    output: out\u002Fpatients.csv\n    report: out\u002Fpatients-report.json   # optional per-job report\n  - name: visits\n    input: ..\u002Fdata\u002Fvisits.csv\n    policy: ..\u002Fpolicies\u002Fvisits.yaml\n    output: out\u002Fvisits.csv\n    vault: out\u002Fvisits-vault.jsonl      # optional per-job vault\n```\n\n```bash\ndeident chain hospital.yaml --mode pseudonymize --report out\u002Fchain-report.json\n```\n\nCross-file linkage needs two things:\n\n1. **Same token scope** — all policies share the same `dataset` (and secret),\n   or the manifest forces one via its `dataset:`\u002F`key:` overrides. Diverging\n   scopes in pseudonymize mode are flagged as a chain warning, because they\n   silently break joins.\n2. **Same identity domain** — tokens are namespaced by column name by default,\n   so `patient_id` (patients.csv) and `patient_ref` (visits.csv) would *not*\n   match. Declare the shared domain on the referencing column:\n\n   ```yaml\n   - name: patient_ref\n     class: direct_identifier\n     pseudonymize:\n       prefix: \"pid_\"\n       domain: patient_id    # ← same namespace as patients.csv's patient_id\n   ```\n\nJobs run sequentially and the chain stops at the first failure (remaining jobs\nare not run; the combined report says so). Exit code is non-zero unless every\njob succeeded. `--engine wasm` gives each job of the chain its own fresh\nsandbox. A complete working example ships in\n[examples\u002Fchains\u002Fhospital.yaml](examples\u002Fchains\u002Fhospital.yaml).\n\n## Risk report\n\n`--report \u003CFILE>` writes a JSON document (also available for `pseudonymize`):\n\n```json\n{\n  \"dataset\": \"patients-demo\",\n  \"mode\": \"anonymize\",\n  \"rows_read\": 12,\n  \"rows_written\": 12,\n  \"direct_identifiers\": [\n    { \"field\": \"patient_id\", \"action\": \"removed\" },\n    { \"field\": \"full_name\", \"action\": \"removed\" },\n    { \"field\": \"email\", \"action\": \"removed\" }\n  ],\n  \"quasi_identifiers\": {\n    \"fields\": [\"age\", \"zip\", \"admission_date\"],\n    \"equivalence_classes\": 5,\n    \"min_class_size\": 1,\n    \"max_class_size\": 5,\n    \"mean_class_size\": 2.4,\n    \"unique_rows\": 1,\n    \"unique_row_ratio\": 0.0833,\n    \"k_thresholds\": [\n      { \"k\": 2, \"rows_at_or_above\": 11, \"ratio\": 0.9167 },\n      { \"k\": 5, \"rows_at_or_above\": 5, \"ratio\": 0.4167 },\n      { \"k\": 10, \"rows_at_or_above\": 0, \"ratio\": 0.0 }\n    ]\n  },\n  \"patterns\": [\n    { \"pattern\": \"iban\", \"field\": \"notes\", \"matches\": 1, \"action\": \"redacted\" }\n  ],\n  \"warnings\": [],\n  \"limitations\": [ \"This report supports a risk assessment; it does not certify or guarantee anonymization.\", \"...\" ]\n}\n```\n\n`patterns` lists content-pattern matches per rule and column with the action\ntaken (`detected` \u002F `redacted` \u002F `tokenized`). `deident chain --report` writes a\ncombined chain report instead: chain name, completion flag, chain-level warnings\nand each job's outcome with its embedded `RiskReport`.\n\nHow to read the `quasi_identifiers` block: rows are grouped by their combination of\n(transformed) quasi-identifier values — each distinct combination is an\n*equivalence class*. Small classes mean higher re-identification risk:\n\n- `min_class_size` — the k in \"k-anonymity style\" terms; 1 means at least one row\n  is unique on its quasi-identifiers.\n- `unique_rows` \u002F `unique_row_ratio` — rows that are one-of-a-kind. These are the\n  riskiest rows; consider coarser generalization if this isn't near zero.\n- `k_thresholds` — share of rows living in classes of at least size k (2, 5, 10).\n\n`warnings` surfaces anything that needs human attention: inline key usage,\nquasi-identifiers without a strategy, suppressed values, unlisted-but-kept columns.\nThe `limitations` block is embedded in every report by design.\n\n## Security model & non-goals\n\n- Anonymization here is **risk-assessed, never guaranteed**. The report measures\n  what it can; residual risk always remains and depends on context.\n- Pseudonymized data **remains personal data** under most privacy regimes (e.g.\n  GDPR). Reversal requires only the key material — protect it separately.\n- Deny-by-default policy handling: unlisted columns and unknown policy keys fail\n  the job unless explicitly relaxed.\n- With `--engine wasm`, each job runs in a fresh WebAssembly sandbox with a\n  preopened job directory as its only filesystem capability, no network, and\n  per-job memory\u002Ftime limits (see [Sandboxed execution](#sandboxed-execution)).\n  Sandboxing *reduces* the blast radius of risky parsing logic and future\n  untrusted plugins; it is a mitigation, not an absolute boundary, and no\n  escape-proof claims are made.\n- The mapping vault is **re-identification material**, encrypted at rest but as\n  sensitive as the source data. `vault export` and `reverse` are privileged\n  operations that reproduce personal data.\n- `token` and `mock` pattern actions produce **pseudonymous, not anonymous**\n  values, even in anonymize mode. Mocks additionally *look* real, which is the\n  point and also the hazard — the report and lints call this out.\n- The audit log is metadata-only by design; it records what happened, never the\n  data it happened to.\n- Risk figures returned from a sandboxed job are **host-attested**: the host\n  re-derives what it owns and verifies what it can cheaply check, because a\n  compromised guest could otherwise report clean counts over untransformed data.\n  The `report_provenance` field states which regime produced a given report.\n- Secrets must be at least 32 bytes, and a policy declaring both `env` and\n  `inline` **fails closed** when the variable is unset — see\n  [Key management](#key-management).\n- A job that fails partway publishes neither its output nor its vault, so a\n  truncated artifact cannot be mistaken for a complete one.\n- Policy lints are heuristics that catch common mistakes. A clean lint run is\n  not a compliance statement.\n- Non-goals: differential privacy, synthetic data generation, free-text\u002FNLP\n  de-identification, and legal certification of any output.\n\n## Example datasets\n\nThe repo ships a 12-row demo (`examples\u002Fdata\u002Fpatients.csv`) for reading at a\nglance, and a generator for datasets large enough that the statistics mean\nsomething:\n\n```bash\ncargo run -p deident-core --example gen_dataset -- examples\u002Fdata 1000\n\ndeident anonymize examples\u002Fdata\u002Fclinic-patients.csv \\\n  --policy examples\u002Fpolicies\u002Fclinic.yaml --out anon.csv --report risk.json\n```\n\n| File | Contents |\n|---|---|\n| `clinic-patients.csv` | direct identifiers, quasi-identifiers, free text carrying every entity type the detectors know |\n| `clinic-visits.csv` | foreign key into patients under a *different* column name — for chained runs and `pseudonymize.domain` |\n| `clinic-labs.jsonl` | a JSONL table with a zero-padded code and a real float, so the JSONL path is exercised |\n| `clinic-messy.csv` | the same shape with real-world damage (see below) |\n\nTwo things make this more useful than simply being big:\n\n**The quasi-identifier distribution is engineered.** Ages, ZIPs and dates are\ndrawn so most rows land in large equivalence classes while a deliberate minority\nare unique. On 1,000 rows that yields ~300 classes with ~16% unique rows — a\nnumber you can reason about. Uniformly random data would make every row unique,\nwhich makes the report look alarming and teaches nothing.\n\n**`clinic-messy.csv` contains what real exports actually contain:** a UTF-8 BOM\nand mixed-case headers (both silently make an exact-match policy field inert),\nempty cells, dates in `14.03.2024` order that no ISO parser accepts, card-shaped\nnumbers that fail Luhn, zero-padded identifiers that naive type inference\ncorrupts, `1.2.3.4` version strings that look like IPv4, and non-ASCII names.\nPoint a policy at it to see how the tool behaves when the input misbehaves.\n\nOutput is deterministic — fixed seed, counter-based PRNG — so regenerating gives\nbyte-identical files and tests stay reproducible. Pass a larger row count to\nscale up; nothing about the generator is limited to 1,000.\n\n> These are synthetic records with deliberately planted identifiers. Do not mix\n> them with real data.\n\n## Versioning\n\nCurrent version: **0.2.0**. See [CHANGELOG.md](CHANGELOG.md) for what changed.\n\nFour compatibility surfaces move independently, and the crate version is the\nleast consequential of them:\n\n| Surface | Where | Breaking means |\n|---|---|---|\n| Crate version | `Cargo.toml` | Rust API changes |\n| Policy schema | `version:` in a policy | An existing policy stops loading |\n| Vault format | vault header `version` | An existing vault stops decrypting |\n| **Token derivation** | not yet versioned | **Every previously issued token changes value** |\n\nThe last row is the one to watch. Tokens are a keyed hash of a domain and a value,\nso any change to the hash input produces different tokens for the same input —\njoins against earlier exports break and **nothing errors**. Changing token\nderivation therefore requires a major version bump and a migration note, even if\nthe Rust API is untouched.\n\nEvery report and audit record carries `tool_version`, so an artifact can be traced\nto the build that produced it. That matters because detection patterns and default\nprofiles change between versions: \"no identifiers found\" only means something\nalongside the version that looked.\n\n## Project layout\n\n| Crate | Purpose |\n|---|---|\n| `crates\u002Fcli` | `deident` binary — command-line UX |\n| `crates\u002Fcore` | Policy schema, transforms, job engine, risk reports |\n| `crates\u002Fhost` | Execution engines: in-process native and per-job Wasmtime sandbox |\n| `crates\u002Fworker` | Wasm guest that executes one job inside its sandbox |\n| `crates\u002Fdicom` | DICOM policy, profile and de-identification engine |\n| `crates\u002Ftypes` | Shared request\u002Fresponse\u002Freport models |\n\n```bash\ncargo test --workspace        # unit + integration tests (includes the feature matrix)\ncargo clippy --workspace --all-targets\ncargo test -p deident-cli --test matrix   # just the feature-combination matrix\n```\n\nCI (GitHub Actions) — two workflows with no overlapping work:\n\n- `rust.yml` — build, clippy (`-D warnings`) and the test suite on every\n  push\u002FPR to main. It skips the feature matrix, which the second workflow\n  owns.\n- `feature-matrix.yml` — every mode × engine × single\u002Fchain combination\n  against the sample dataset, triggered by changes under `examples\u002F` or\n  `crates\u002F` (plus a manual \"Run workflow\" button). The matrix test recomputes\n  its expectations from the data itself — determinism, native\u002Fwasm\n  byte-parity, identifier survival, pattern counts, chain linkage — so editing\n  the sample dataset automatically re-validates every feature against it.\n  The full-feature policy it uses is\n  [examples\u002Fpolicies\u002Fpatients-full.yaml](examples\u002Fpolicies\u002Fpatients-full.yaml).\n\nNote that CI runs the latest stable Rust, which may lint more strictly than an\nolder local toolchain; run clippy with `-D warnings` locally to match it.\n\n## Roadmap\n\nEverything on the original roadmap is now implemented. What's next, roughly in\norder of value:\n\n- **Streaming at scale** — Parquet and the equivalence-class statistics hold\n  data in memory. `--split` divides a dataset across sandboxes and lowers peak\n  memory per job, but chunked row-group processing and a spill-to-disk class map\n  would lift the ceiling properly.\n- **Split with a vault** — merging per-chunk encrypted mapping files, so\n  `--split` and `--vault` can be combined.\n- **Parquet in the sandbox** — currently excluded to keep the guest module small;\n  it is also what stops `--split` from accepting Parquet.\n- **Broader DICOM coverage** — extend the tag table toward full Annex E, add the\n  profile options (Retain Longitudinal Temporal, Retain Patient Characteristics,\n  Retain Safe Private), and sandbox the DICOM parser.\n- **Burned-in pixel detection** — OCR-assisted flagging of PHI rendered into\n  image pixels. Detection only; cleaning would remain a claim we refuse to make.\n- **Ship the worker with releases** — embed or bundle `deident-worker.wasm`\n  next to the binary so `auto` always sandboxes instead of falling back.\n- **k-anonymity enforcement** — today the report *measures* small equivalence\n  classes; a `min_class_size: k` policy option could suppress or coarsen rows\n  until the threshold is met, and fail the job if it cannot be.\n- **Richer pattern library** — national ID formats, addresses, dates in free\n  text, and a `--dry-run` scan mode that reports findings without writing\n  output.\n- **Vault key rotation and re-tokenization** — re-derive tokens under a new\n  secret while preserving joins, using the vault as the bridge.\n- **Column-level pattern strategies per class** — e.g. apply a pattern set to\n  every `utility` column automatically instead of naming columns.\n- **Policy authoring help** — `deident init \u003Cinput>` to scaffold a policy from\n  a dataset's header with class guesses from column names and content sniffing.\n- **Differential-privacy noise for aggregates** — out of scope for row-level\n  output, but useful if the tool grows a summary-export mode.\n","ikuV\u002Fdeident-wasm 是一个面向结构化数据的隐私转换引擎，支持对 CSV、JSONL、Parquet 和 DICOM 文件执行声明式策略驱动的假名化（pseudonymization）与风险评估型匿名化（anonymization）。核心功能包括基于 YAML 策略的列级分类与处理（如移除、泛化、令牌化）、16 种内置内容模式检测器（含校验逻辑的 IBAN\u002FSSN\u002F信用卡等）、跨文件链式处理以保持外键可连接性，以及 WebAssembly 沙箱化执行保障隔离安全。适用于医疗、金融等需合规处理敏感结构化数据的场景，如测试环境脱敏、分析数据共享和审计报告生成。",2,"2026-08-09 02:30:03","CREATED_QUERY"]