[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-81720":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":9,"language":10,"languages":9,"totalLinesOfCode":9,"stars":11,"forks":12,"watchers":13,"openIssues":14,"contributorsCount":12,"subscribersCount":12,"size":12,"stars1d":12,"stars7d":13,"stars30d":13,"stars90d":12,"forks30d":12,"starsTrendScore":12,"compositeScore":15,"rankGlobal":9,"rankLanguage":9,"license":16,"archived":17,"fork":17,"defaultBranch":18,"hasWiki":17,"hasPages":19,"topics":20,"createdAt":9,"pushedAt":9,"updatedAt":21,"readmeContent":22,"aiSummary":23,"trendingCount":12,"starSnapshotCount":12,"syncStatus":24,"lastSyncTime":25,"discoverSource":26},81720,"leoflow","neochaotic\u002Fleoflow","neochaotic","GitOps-first, container-native workflow orchestrator in Go (Airflow-UI compatible).",null,"Go",38,0,1,105,40.6,"Apache License 2.0",false,"main",true,[],"2026-06-12 04:01:35","\u003Cp align=\"center\">\n  \u003Cimg src=\"docs\u002Fassets\u002Ficon-256.png\" alt=\"Leoflow\" width=\"140\">\n\u003C\u002Fp>\n\n\u003Ch1 align=\"center\">Leoflow\u003C\u002Fh1>\n\n\u003Cp align=\"center\">\n  \u003Cem>The workflow orchestrator that ate Apache Airflow's lunch.\u003Cbr>\n  Same UI. Same vocabulary. Ten times the speed. Zero of the pain.\u003C\u002Fem>\n\u003C\u002Fp>\n\n[![License: Apache 2.0](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FLicense-Apache_2.0-blue.svg)](https:\u002F\u002Fopensource.org\u002Flicenses\u002FApache-2.0)\n[![Go Report Card](https:\u002F\u002Fgoreportcard.com\u002Fbadge\u002Fgithub.com\u002Fneochaotic\u002Fleoflow?v=2)](https:\u002F\u002Fgoreportcard.com\u002Freport\u002Fgithub.com\u002Fneochaotic\u002Fleoflow)\n[![CI](https:\u002F\u002Fgithub.com\u002Fneochaotic\u002Fleoflow\u002Factions\u002Fworkflows\u002Fci.yaml\u002Fbadge.svg)](https:\u002F\u002Fgithub.com\u002Fneochaotic\u002Fleoflow\u002Factions\u002Fworkflows\u002Fci.yaml)\n[![Security](https:\u002F\u002Fgithub.com\u002Fneochaotic\u002Fleoflow\u002Factions\u002Fworkflows\u002Fsecurity.yaml\u002Fbadge.svg)](https:\u002F\u002Fgithub.com\u002Fneochaotic\u002Fleoflow\u002Factions\u002Fworkflows\u002Fsecurity.yaml)\n[![OpenSSF Scorecard](https:\u002F\u002Fapi.securityscorecards.dev\u002Fprojects\u002Fgithub.com\u002Fneochaotic\u002Fleoflow\u002Fbadge)](https:\u002F\u002Fsecurityscorecards.dev\u002Fviewer\u002F?uri=github.com\u002Fneochaotic\u002Fleoflow)\n\n> The OpenSSF Best Practices badge will be added once the project is registered at [bestpractices.dev](https:\u002F\u002Fwww.bestpractices.dev) (post-v0.1.0 target, per ADR 0014).\n\n---\n\n## 📚 Documentation\n\n**Full docs → \u003Chttps:\u002F\u002Fneochaotic.github.io\u002Fleoflow\u002F>** (DAG authoring, deploy, API reference, architecture).\n\n| | |\n|---|---|\n| [Operating modes](docs\u002Foperating-modes.md) | Demo · Dev · Production (coming soon) |\n| [DAG authoring](docs\u002Fdag-authoring.md) | write a DAG; the dev → deploy lifecycle |\n| [CI\u002FCD & deploy examples](docs\u002Fdeploy.md) | GitHub Actions · GitLab · Cloud Build\u002FRun · generic |\n| [HTTP API (Scalar)](docs\u002Fapi-reference.md) · [Go packages](docs\u002Fgo-api.md) | API references |\n| [Concepts & glossary](docs\u002Fconcepts.md) · [Architecture](docs\u002Farchitecture.md) | the model & the *why* |\n\n---\n\n## ⚡ Install\n\n```bash\ncurl -fsSL https:\u002F\u002Fraw.githubusercontent.com\u002Fneochaotic\u002Fleoflow\u002Fmain\u002Finstall.sh | sh\n```\n\nInstalls the binaries and runs `leoflow setup` (ensures Python, provisions the\nparser, creates your workspace) — **no sudo, no system Python, no package\nmanager**. Docker is optional and only unlocks the higher tiers. Linux + macOS,\namd64 + arm64 (Windows via WSL2). See **[Installation](docs\u002Finstallation.md)**.\n\n> **Pre-alpha:** builds carry a ~90-day expiry and refuse to run past it (this\n> version is not durable). Re-run the installer for a fresh build.\n\n---\n\n## The Five Wounds Apache Airflow Will Not Heal\n\nAirflow is the most widely deployed workflow orchestrator on earth. It is also the one that bleeds the most in production. Every data engineer recognizes these wounds:\n\n- **The scheduler that stalls.** Three seconds between tasks. Ten when the cluster is busy. Pipelines that should run in two minutes take twenty.\n- **The triggerer that suffocates.** Above five hundred concurrent sensors, the Python asyncio loop chokes. Sensors stop firing. SLAs miss.\n- **The DAG file that re-parses itself to death.** Every scheduler loop opens every `.py` file in `\u002Fdags`. CPU spikes for nothing. Memory grows. Restarts become a ritual.\n- **The worker that leaks until it dies.** Long-running Celery workers accumulate file descriptors, database connections, half-loaded modules. OOMKilled at three in the morning. Always.\n- **The dependency hell that has no door.** `pandas==1.0` for the legacy DAG, `pandas==2.0` for the new one. One Airflow image. Pick a side. Cry either way.\n\nLeoflow was built to close these five wounds, on day one, by construction.\n\n## How It Closes Them\n\nLeoflow does not invent a new execution model. Pod-per-task is the right pattern, and Airflow's `KubernetesExecutor` proved it years ago. What Leoflow does is **strip out the Python overhead from every layer of the orchestration stack**:\n\n| Wound | Airflow today | Leoflow |\n|---|---|---|\n| Scheduler latency | 3-10 seconds per decision | **\u003C200 ms** — native Go, zero GIL |\n| Sensor concurrency | ~500 (asyncio Triggerer) | **100,000+** — each sensor is a 2 KB goroutine |\n| DAG parsing cost | Re-parsed every scheduler loop | **Zero** — DAG is pre-compiled to immutable JSON |\n| Worker lifecycle | Long-lived, leak-prone | **Ephemeral pod per task** — spawn, run, die |\n| Worker image size | 1.5 GB+ Airflow base | **200 MB typical** — each DAG is its own slim image |\n| Dependency isolation | Workaround via `KubernetesPodOperator` | **Native** — every DAG is a container |\n| Cold start | 15-45 s | **2-5 s target** — agent is a 15 MB static binary |\n| Observability | Retrofitted with effort | **Native** — Prometheus + OpenTelemetry + structured logs from commit one |\n\nThis is not marketing. This is what falls out of replacing a Python control plane with Go and embracing the container as the unit of isolation.\n\n## What Leoflow Is\n\nLeoflow is a **GitOps-first, container-native workflow orchestrator** written in Go. Each phrase carries weight:\n\n- **GitOps-first.** Your DAG is a versioned artifact (`dag.json` + container image), not live source code. CI builds it. The registry stores it. Rollback is a tag change.\n- **Container-native.** Each DAG is its own container image, with its own dependencies, its own Python version, its own everything. Built automatically from a one-page `leoflow.yaml` — you never touch Docker unless you want to.\n- **Airflow-UI compatible.** The MVP runs the unmodified Apache Airflow 3.2.x UI. Your team's muscle memory survives the migration. No new tool to learn.\n- **Go performance, Go discipline.** Static binary. No GIL. Goroutines for concurrency. Test-driven from the first commit. Go Report Card A+ enforced in CI.\n\n## What It Looks Like to Use\n\nA complete Leoflow DAG project. No Dockerfile. No `requirements.txt`. No CI plumbing to invent.\n\n```yaml\n# leoflow.yaml\ndag_id: etl_vendas\npython_version: \"3.11\"\ndependencies:\n  - pandas==2.1.0\n  - requests==2.31.0\n```\n\n```python\n# dag.py\nfrom leoflow import DAG, task\n\n@task\ndef fetch():\n    import requests\n    return requests.get(\"https:\u002F\u002Fapi.example.com\u002Forders\").json()\n\n@task\ndef transform(orders):\n    return [{\"id\": o[\"id\"], \"value\": o[\"amount\"] * 1.1} for o in orders]\n\nwith DAG(\"etl_vendas\", schedule=\"0 5 * * *\") as dag:\n    raw = fetch()\n    processed = transform(raw)\n```\n\n```bash\nleoflow compile .              # generates Dockerfile, builds image, produces dag.json\nleoflow push .\u002Fdag.json        # registers with the control plane\n```\n\nThat is the entire developer surface. The CLI builds the image against an official base (`leoflow\u002Fpython-runtime:3.11`), pushes to your registry, and registers a versioned DAG. The Airflow UI shows it at the next refresh.\n\n## Architecture\n\nA DAG is compiled into an **immutable artifact** (a `dag.json` spec plus a\ncontainer image) and pushed to the control plane. A Go **control plane**\nschedules it and, for each task, dispatches an ephemeral **worker pod** whose\n`leoflow-agent` runs the user code and reports back over gRPC. Postgres holds\nmetadata; Redis holds XCom values and live-log fan-out.\n\n```mermaid\nflowchart LR\n    subgraph dev[\"Author \u002F CI\"]\n        src[\"leoflow.yaml · dag.py · Dockerfile\"]\n    end\n\n    subgraph cp[\"Control plane — Go\"]\n        direction TB\n        api[\"HTTP API \u002Fapi\u002Fv2\u003Cbr\u002F>JWT · RBAC · multi-tenant\"]\n        sched[\"Scheduler\u003Cbr\u002F>state machine · cron\u003Cbr\u002F>leader election · retries\"]\n        asvc[\"Agent gRPC service\u003Cbr\u002F>task spec · state · XCom · logs\"]\n        api --- sched --- asvc\n    end\n\n    src -->|\"leoflow compile \u002F push\"| api\n    sched --- pg[(\"Postgres\u003Cbr\u002F>metadata\")]\n    asvc --- redis[(\"Redis\u003Cbr\u002F>XCom · log tail\")]\n\n    sched -->|\"dispatch: one pod per task\"| pod\n    subgraph k8s[\"Kubernetes\"]\n        pod[\"Worker pod = your DAG image\u003Cbr\u002F>leoflow-agent ⇄ your Python \u002F Bash\"]\n    end\n    pod -->|\"gRPC: register · fetch spec · push XCom · stream logs · report state\"| asvc\n\n    classDef store fill:#1f2937,stroke:#4b5563,color:#e5e7eb;\n    class pg,redis store;\n```\n\nShort-lived `http_api` tasks skip the pod and run inline as goroutines in the\ncontrol plane (capped); everything else runs pod-per-task. Read\n[the ADRs](docs\u002Fadr\u002F) for the reasoning behind every decision.\n\n## Status\n\n🚧 **Pre-alpha, under active development. Not production-ready.**\n\n**Implemented today (Phases 1–4):**\n\n- **CLI + parser** — `leoflow init \u002F validate \u002F compile \u002F push \u002F runs trigger \u002F runs status \u002F auth create-token`; the Python DAG parser; `compile --build \u002F --push` builds and pushes the DAG image (out-of-process).\n- **Control plane** — Airflow-compatible `\u002Fapi\u002Fv2` API, JWT auth + RBAC + multi-tenant, the scheduler state machine with cron scheduling, Postgres advisory-lock leader election, **task retries**, embedded Scalar API docs, and Prometheus + OpenTelemetry observability.\n- **Execution** — real pod-per-task execution via the `leoflow-agent` over gRPC (Kubernetes, ADR 0015), plus inline `http_api` goroutines for short calls; orphaned-pod reconciliation and completed-pod garbage collection.\n- **Data flow** — XCom on Redis (256 KB limit, TTL, optional schema validation) passed between tasks; log shipping to disk with a read API and live tailing over Redis pub\u002Fsub.\n\n**Not yet implemented:** the Airflow 3.2.x UI integration (Phase 5); the Helm chart, load tests, and S3\u002FGCS log sinks (Phase 6). Tracked refinements live in the [issue tracker](https:\u002F\u002Fgithub.com\u002Fneochaotic\u002Fleoflow\u002Fissues).\n\n## Features in the MVP\n\n**Shipping in v0.1.0:**\n\n- Python, Bash, and HTTP API operators\n- DAG-as-Image model with automatic image build via `leoflow.yaml`\n- Hybrid DAG authoring: Python source parsed at compile time, or declarative YAML\n- XCom on Redis with 256 KB limit, TTL, and optional schema validation\n- Apache Airflow 3.2.x UI compatibility (no fork required)\n- JWT authentication, RBAC, multi-tenant schema (OIDC-ready)\n- Kubernetes-native execution (no worker pool to manage)\n- Local development on Kubernetes (k3d\u002Fkind) or a dev-only subprocess executor (ADR 0015)\n- Trigger rules: `all_success`, `all_failed`, `all_done`, `one_success`, `one_failed`\n- Clear task instance to re-run failed tasks\n- Leader election via Postgres advisory locks\n- OpenSSF Best Practices compliance, signed releases (cosign), supply chain scanning (govulncheck + Trivy + CodeQL)\n\n**On the post-MVP roadmap:**\n\n- Optimized backfill (parallel execution with throttling)\n- UI scaling for 10,000+ DAGs (caching, server-side pagination)\n- Dynamic task mapping\n- OIDC authentication (Google, Azure AD, Keycloak, Okta)\n- Mark success\u002Ffailed manually\n- Custom UI (replacing the Airflow UI)\n- Deferrable tasks (efficient dispatch + long-poll pattern, native Go implementation without a separate Triggerer process — see [ADR 0016](docs\u002Fadr\u002F0016-deferrable-tasks.md))\n\n## Getting Started\n\n### Try it with the UI (one command)\n\nThe full stack — Postgres, Redis, and the control plane with the **embedded\nAirflow 3.2.1 UI** — runs from a single Compose profile:\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002Fneochaotic\u002Fleoflow\ncd leoflow\ndocker compose --profile demo up --build\n```\n\nThen open **http:\u002F\u002Flocalhost:8080** and log in as **`admin@leoflow.local` \u002F `admin`**.\nThe server applies migrations, seeds the admin user, and serves the API (`\u002Fapi\u002Fv2`),\nthe internal UI API (`\u002Fui\u002F*`), and the React UI from one process (ADR 0017). Stop\nwith `docker compose --profile demo down` (add `-v` to wipe data).\n\n> The pinned Airflow UI is a tactical MVP choice; a purpose-built Leoflow UI is\n> the long-term direction (ADR 0018). See `docs\u002Fui-compatibility.md`.\n\n### Local development\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002Fneochaotic\u002Fleoflow\ncd leoflow\nmake setup            # Go tools, Python parser, pre-commit hook\nmake build            # builds bin\u002Fleoflow, bin\u002Fleoflow-server, bin\u002Fleoflow-agent\n\n# Start Postgres + Redis (Docker) and apply migrations\nmake dev-up           # docker compose up --wait + migrate-up; `make dev-down` to stop\n\n# Run the control plane (bootstraps a default admin user)\nLEOFLOW_AUTH_JWT_SECRET=dev LEOFLOW_BOOTSTRAP_PASSWORD=admin123 .\u002Fbin\u002Fleoflow-server &\n# API docs (Scalar) at http:\u002F\u002Flocalhost:8080\u002Fdocs ; metrics at http:\u002F\u002Flocalhost:9090\u002Fmetrics\n\n# Author, compile, and register a DAG\n.\u002Fbin\u002Fleoflow init my-dag\n.\u002Fbin\u002Fleoflow compile my-dag --image my-dag:dev -o my-dag\u002Fdag.json\nTOKEN=$(.\u002Fbin\u002Fleoflow auth create-token --username admin@leoflow.local --password admin123)\n.\u002Fbin\u002Fleoflow push my-dag\u002Fdag.json --token \"$TOKEN\"\n```\n\n> **Two dev environments.** `make dev-up` runs Postgres + Redis as plain Docker containers on the host for a fast inner loop (control plane on the host). Full in-cluster execution — control plane and dependencies on a local Kubernetes cluster (k3d\u002Fkind) via the Helm chart, mirroring production and exercising real task pods — arrives with the e2e work in a later phase. Task execution is on Kubernetes only (ADR 0015); the host containers are dev dependencies, not the execution path.\n\n> The Airflow 3.2.1 UI ships embedded in the server and is served at `\u002F` (Phase 5; see the one-command demo above and `docs\u002Fui-compatibility.md`). The Scalar API reference is at `\u002Fdocs`. The Helm chart and load tests are the remaining Phase 6 work.\n\n## Honest Comparison\n\nWe have no patience for marketing fiction. Here is where Leoflow sits in the landscape:\n\n| | Airflow | Argo Workflows | Prefect | Dagster | **Leoflow** |\n|---|---|---|---|---|---|\n| Language of control plane | Python | Go | Python | Python | **Go** |\n| Pod-per-task model | Optional (KubernetesExecutor) | Yes | Optional | Optional | **Yes, only mode** |\n| Dependency isolation per DAG | Workaround | Manual | Partial | Partial | **Native** |\n| UI familiar to Airflow users | Yes | No | No | No | **Yes (Airflow UI)** |\n| GitOps-first DAG model | No | Yes | Partial | Partial | **Yes** |\n| Scheduler in Go (no GIL) | No | Yes | No | No | **Yes** |\n| Native observability | Add-on | Partial | Partial | Yes | **Built-in** |\n| Mental model | Celery-era | K8s-native | Python-native | Software-defined assets | **K8s-native + Airflow vocabulary** |\n\nWe borrow from Argo Workflows (container-native), from Prefect (modern developer experience), and from Airflow (the UI and vocabulary). We do not pretend we invented any of those. We just put them together in a way nobody had.\n\n## Documentation\n\n- [Architecture overview](docs\u002Farchitecture.md)\n- [Architecture Decision Records (14 ADRs)](docs\u002Fadr\u002F) — every major decision, with its reasoning\n- [API reference](docs\u002Fapi\u002F) — OpenAPI spec, also rendered as interactive docs at `\u002Fdocs` in the running server\n- [Developer guide](docs\u002Fdeveloper-guide.md) — writing your first DAG, migration from Airflow\n- [Operator guide](docs\u002Foperator-guide.md) — production deployment, monitoring, backup\n- [Security policy](SECURITY.md) — how to report vulnerabilities\n\n## Engineering Discipline\n\nLeoflow holds itself to a higher bar than most open source projects, because workflow orchestrators must be boring and reliable to be useful:\n\n- **Strict TDD** — every line of production code is preceded by a failing test ([ADR 0011](docs\u002Fadr\u002F0011-tdd-strict.md))\n- **Go Report Card A+** — enforced in CI from the first commit ([ADR 0012](docs\u002Fadr\u002F0012-code-quality-standards.md))\n- **GoDocs on every exported identifier** — no exceptions\n- **Supply chain security from day one** — govulncheck, gosec, Trivy, CodeQL, Scorecard, signed releases ([ADR 0014](docs\u002Fadr\u002F0014-supply-chain-security.md))\n- **Per-phase coverage floors** — rising from 70% to 85% across the MVP phases\n- **Native observability** — Prometheus, OpenTelemetry, structured logs from commit one ([ADR 0010](docs\u002Fadr\u002F0010-observability.md))\n\nIf you contribute, read the [CONTRIBUTING guide](CONTRIBUTING.md) first.\n\n## License\n\nApache License 2.0. See [LICENSE](LICENSE).\n\n## Acknowledgements\n\nLeoflow stands on the shoulders of Apache Airflow. The team behind Airflow defined the vocabulary, proved the architecture, and built the UI that Leoflow reuses without modification in the MVP. This project would not exist without their work, and we credit them at every layer of our documentation.\n\nWe also studied the source of Argo Workflows, Prefect, and Dagster carefully. Each made decisions worth borrowing, and we did.\n\n---\n\n> **Star the repo if you have ever waited five seconds for an Airflow task to start.**\n> **Watch the repo if you want to be notified when the MVP ships.**\n> **Open an issue if you have a chronic Airflow pain we have not addressed yet — there is still time to fix it before v0.1.0.**\n","Leoflow 是一个基于GitOps理念、容器原生的工作流编排器，兼容Airflow UI。该项目使用Go语言开发，旨在提供比Apache Airflow更快的执行速度和更简洁的操作体验。其核心功能包括支持DAG定义与管理、多环境部署（如开发、测试及生产）、以及通过HTTP API进行交互等，并且在设计上特别注重性能优化，比如任务调度延迟低、并发处理能力强等。Leoflow适合需要高效可靠地管理和运行复杂工作流程的企业或团队使用，在数据处理管道构建、CI\u002FCD集成等方面尤为适用。",2,"2026-06-11 04:06:06","CREATED_QUERY"]