From f340ff941ff33299ca4b7bd56d07b0816d8a3695 Mon Sep 17 00:00:00 2001 From: Aleksei Kudriashov Date: Sat, 20 Jun 2026 17:24:22 +0200 Subject: [PATCH] Add codespaces skill for belief-map architecture discovery Initial import of the codespaces skill: query .belief_map.sexp for module boundaries, dependencies, blast radius, and the minimal files to read before non-trivial code changes. Includes the belief-map and infra-topology builders, the belief-search query engine and its tests, and reference docs for the S-expression notation and query patterns. --- .agents/designs/infra-topology-builders.md | 288 ++ .gitignore | 11 + LOG.md | 9 + SKILL.md | 254 ++ references/scheme-queries.md | 88 + references/sexp-notation.md | 176 ++ scripts/belief_search.py | 2256 ++++++++++++++ scripts/build_belief_map.py | 3182 ++++++++++++++++++++ scripts/build_infra_topology.py | 853 ++++++ scripts/git_descendants.py | 124 + tests/test_belief_search.py | 152 + 11 files changed, 7393 insertions(+) create mode 100644 .agents/designs/infra-topology-builders.md create mode 100644 .gitignore create mode 100644 LOG.md create mode 100644 SKILL.md create mode 100644 references/scheme-queries.md create mode 100644 references/sexp-notation.md create mode 100644 scripts/belief_search.py create mode 100755 scripts/build_belief_map.py create mode 100644 scripts/build_infra_topology.py create mode 100755 scripts/git_descendants.py create mode 100644 tests/test_belief_search.py diff --git a/.agents/designs/infra-topology-builders.md b/.agents/designs/infra-topology-builders.md new file mode 100644 index 0000000..f7ef7e5 --- /dev/null +++ b/.agents/designs/infra-topology-builders.md @@ -0,0 +1,288 @@ +Infrastructure Topology Builders for Belief Map +================================================= + +Context and motivation +---------------------- + +The belief map currently models **source-code** relationships: imports, entity references, data flows, and interface implementations. It has no visibility into **infrastructure topology** -- which services exist, how they connect at the network level, what databases/queues/caches they depend on, and how they are deployed. + +This creates a blind spot: when an agent investigates a cross-service issue or plans a deployment change, the belief map shows code dependencies but not the runtime service graph. The data exists in three IaC formats already present in the codebase: + +- **Kustomize** (`.k8s/base/kustomization.yaml`): 20+ services in the mono workspace, each with deployment manifests declaring env vars, service endpoints, and resource dependencies (postgres, rabbitmq, minio, vault) +- **Helm charts** (`charts/*/Chart.yaml`): avionica streaming services (sara-mock, transformer, ergoss-push, nats, prometheus, dashboard) +- **Terraform** (`infra/terraform/*.tf`): infrastructure prerequisites (namespaces, secrets, PVCs, configmaps) with module dependencies + +Goals: +- Extract service topology from all three IaC formats and emit sexp edges into the belief map +- Detect service-to-service dependencies: env var references, service DNS names, port mappings +- Detect service-to-infrastructure dependencies: databases, message queues, object stores, vaults +- Output is the same `.belief_map.sexp` format, appended or merged by the main builder + +Non-goals for first implementation (v1): +- No runtime service discovery (no kubectl, no cluster queries) +- No Tilt/docker-compose parsing (separate effort) +- No ingress/route topology (just service-to-service and service-to-infra) +- No multi-cluster or multi-namespace topology +- No cost or resource quota analysis + + +Implementation considerations +------------------------------ + +**Pure static analysis.** All three parsers read YAML/HCL files from disk. No cluster access needed. + +**Additive to existing graph.** Infrastructure edges are a new layer on top of the source-code graph. They use new edge types (`k8s-service`, `k8s-depends`, `infra-resource`) that don't conflict with existing `imports`/`calls-api`/`data-flow` edges. + +**Node identity.** Infrastructure nodes use the service name from k8s metadata (e.g., `app-service`, `drive-postgres`, `nats`) prefixed with `infra/` to distinguish from source-code module IDs. Example: `infra/app-service`, `infra/drive-postgres`. + +**Minimal dependencies.** Kustomize and Helm parsers use only PyYAML (already available). Terraform parser uses `hcl2` Python library or falls back to regex if unavailable. + +**Invocation.** Each script is standalone but designed to be called from `build_belief_map.py` as an optional post-pass: + +```bash +python3 scripts/build_infra_topology.py [--kustomize PATH] [--helm PATH] [--terraform PATH] +``` + +Or integrated into the main builder via `--infra` flag. + + +High-level behavior +------------------- + +### Discovery + +1. Scanner walks the workspace looking for: + - `kustomization.yaml` / `kustomization.yml` files (kustomize) + - `Chart.yaml` files (helm) + - `*.tf` files in directories containing `main.tf` (terraform) + +2. For each discovered IaC root, the appropriate parser runs. + +3. All parsers emit the same intermediate format: a list of `InfraNode` and `InfraEdge` dataclasses. + +4. A merge step deduplicates nodes by service name and writes sexp output. + +### Kustomize parser + +Reads each kustomization.yaml and its referenced resources: + +1. Parse `kustomization.yaml` -> extract `resources` list +2. For each resource directory, parse `deployment.yaml` / `statefulset.yaml`: + - Extract service name from `metadata.name` or `app.kubernetes.io/name` label + - Extract container image -> maps service to codebase repo + - Extract `env` and `envFrom` -> detect service endpoint references: + - `*_HOST`, `*_URL`, `*_SERVICE_HOST` -> depends-on target service + - `DATABASE_URL`, `POSTGRES_*` -> depends-on database + - `RABBITMQ_*`, `AMQP_*`, `NATS_*` -> depends-on message queue + - `REDIS_*` -> depends-on cache + - `S3_*`, `MINIO_*` -> depends-on object store +3. Parse `service.yaml` -> extract ports and service type + +Output edges: +```scheme +(infra-node infra/app-service :kind deployment :image upgraide/app-service :port 3001 :component api) +(infra-node infra/app-postgres :kind deployment :image postgres:16 :port 5432 :component database) +(k8s-depends infra/app-service infra/app-postgres :via env :transport tcp) +(k8s-depends infra/app-service infra/drive-service :via env :transport http) +(k8s-service infra/app-service app-service :namespace upgraide :port 3001) +``` + +### Helm parser + +Reads Chart.yaml and values.yaml: + +1. Parse `Chart.yaml` -> extract chart name, version, dependencies +2. Parse `values.yaml` -> extract: + - `image.repository` / `image.tag` -> container image + - `env.*` -> service endpoint references (same heuristics as kustomize) + - `service.port` -> exposed port +3. Parse `Chart.yaml` `dependencies` -> direct chart-to-chart edges + +Output edges: +```scheme +(infra-node infra/sara-mock :kind helm-chart :image sara-mock :port 8080 :chart sara-mock) +(helm-depends infra/sara-mock infra/nats :via chart-dependency) +``` + +### Terraform parser + +Reads `*.tf` files: + +1. Find all `module` blocks -> extract module name and source +2. Find all `resource` blocks -> extract resource type and name +3. Build dependency graph from: + - `module.X.output_name` references in other modules + - `depends_on` explicit declarations + - Variable references across modules + +Output edges: +```scheme +(infra-node infra/tf-namespace :kind terraform :resource kubernetes_namespace) +(infra-node infra/tf-secrets :kind terraform :resource module) +(tf-depends infra/tf-secrets infra/tf-namespace :via module-reference) +``` + + +S-expression output format +--------------------------- + +New fact types (appended to `.belief_map.sexp`): + +```scheme +; Infrastructure nodes +(infra-node :kind :image :port :component