apoloz/codespaces
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add codespaces skill for belief-map architecture discovery

Browse Source 
Initial import of the codespaces skill: query .belief_map.sexp for module
boundaries, dependencies, blast radius, and the minimal files to read before
non-trivial code changes.

Includes the belief-map and infra-topology builders, the belief-search query
engine and its tests, and reference docs for the S-expression notation and
query patterns.
This commit is contained in:
Aleksei Kudriashov
2026-06-20 17:24:22 +02:00
commit f340ff941f
11 changed files with 7393 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

288
.agents/designs/infra-topology-builders.md Normal file
View File

@@ -0,0 +1,288 @@
Infrastructure Topology Builders for Belief Map
=================================================
Context and motivation
----------------------
The belief map currently models **source-code** relationships: imports, entity references, data flows, and interface implementations. It has no visibility into **infrastructure topology** -- which services exist, how they connect at the network level, what databases/queues/caches they depend on, and how they are deployed.
This creates a blind spot: when an agent investigates a cross-service issue or plans a deployment change, the belief map shows code dependencies but not the runtime service graph. The data exists in three IaC formats already present in the codebase:
- **Kustomize** (`.k8s/base/kustomization.yaml`): 20+ services in the mono workspace, each with deployment manifests declaring env vars, service endpoints, and resource dependencies (postgres, rabbitmq, minio, vault)
- **Helm charts** (`charts/*/Chart.yaml`): avionica streaming services (sara-mock, transformer, ergoss-push, nats, prometheus, dashboard)
- **Terraform** (`infra/terraform/*.tf`): infrastructure prerequisites (namespaces, secrets, PVCs, configmaps) with module dependencies
Goals:
- Extract service topology from all three IaC formats and emit sexp edges into the belief map
- Detect service-to-service dependencies: env var references, service DNS names, port mappings
- Detect service-to-infrastructure dependencies: databases, message queues, object stores, vaults
- Output is the same `.belief_map.sexp` format, appended or merged by the main builder
Non-goals for first implementation (v1):
- No runtime service discovery (no kubectl, no cluster queries)
- No Tilt/docker-compose parsing (separate effort)
- No ingress/route topology (just service-to-service and service-to-infra)
- No multi-cluster or multi-namespace topology
- No cost or resource quota analysis
Implementation considerations
------------------------------
**Pure static analysis.** All three parsers read YAML/HCL files from disk. No cluster access needed.
**Additive to existing graph.** Infrastructure edges are a new layer on top of the source-code graph. They use new edge types (`k8s-service`, `k8s-depends`, `infra-resource`) that don't conflict with existing `imports`/`calls-api`/`data-flow` edges.
**Node identity.** Infrastructure nodes use the service name from k8s metadata (e.g., `app-service`, `drive-postgres`, `nats`) prefixed with `infra/` to distinguish from source-code module IDs. Example: `infra/app-service`, `infra/drive-postgres`.
**Minimal dependencies.** Kustomize and Helm parsers use only PyYAML (already available). Terraform parser uses `hcl2` Python library or falls back to regex if unavailable.
**Invocation.** Each script is standalone but designed to be called from `build_belief_map.py` as an optional post-pass:
```bash
python3 scripts/build_infra_topology.py [--kustomize PATH] [--helm PATH] [--terraform PATH]
```
Or integrated into the main builder via `--infra` flag.
High-level behavior
-------------------
### Discovery
1. Scanner walks the workspace looking for:
- `kustomization.yaml` / `kustomization.yml` files (kustomize)
- `Chart.yaml` files (helm)
- `*.tf` files in directories containing `main.tf` (terraform)
2. For each discovered IaC root, the appropriate parser runs.
3. All parsers emit the same intermediate format: a list of `InfraNode` and `InfraEdge` dataclasses.
4. A merge step deduplicates nodes by service name and writes sexp output.
### Kustomize parser
Reads each kustomization.yaml and its referenced resources:
1. Parse `kustomization.yaml` -> extract `resources` list
2. For each resource directory, parse `deployment.yaml` / `statefulset.yaml`:
- Extract service name from `metadata.name` or `app.kubernetes.io/name` label
- Extract container image -> maps service to codebase repo
- Extract `env` and `envFrom` -> detect service endpoint references:
- `*_HOST`, `*_URL`, `*_SERVICE_HOST` -> depends-on target service
- `DATABASE_URL`, `POSTGRES_*` -> depends-on database
- `RABBITMQ_*`, `AMQP_*`, `NATS_*` -> depends-on message queue
- `REDIS_*` -> depends-on cache
- `S3_*`, `MINIO_*` -> depends-on object store
3. Parse `service.yaml` -> extract ports and service type
Output edges:
```scheme
(infra-node infra/app-service :kind deployment :image upgraide/app-service :port 3001 :component api)
(infra-node infra/app-postgres :kind deployment :image postgres:16 :port 5432 :component database)
(k8s-depends infra/app-service infra/app-postgres :via env :transport tcp)
(k8s-depends infra/app-service infra/drive-service :via env :transport http)
(k8s-service infra/app-service app-service :namespace upgraide :port 3001)
```
### Helm parser
Reads Chart.yaml and values.yaml:
1. Parse `Chart.yaml` -> extract chart name, version, dependencies
2. Parse `values.yaml` -> extract:
- `image.repository` / `image.tag` -> container image
- `env.*` -> service endpoint references (same heuristics as kustomize)
- `service.port` -> exposed port
3. Parse `Chart.yaml` `dependencies` -> direct chart-to-chart edges
Output edges:
```scheme
(infra-node infra/sara-mock :kind helm-chart :image sara-mock :port 8080 :chart sara-mock)
(helm-depends infra/sara-mock infra/nats :via chart-dependency)
```
### Terraform parser
Reads `*.tf` files:
1. Find all `module` blocks -> extract module name and source
2. Find all `resource` blocks -> extract resource type and name
3. Build dependency graph from:
- `module.X.output_name` references in other modules
- `depends_on` explicit declarations
- Variable references across modules
Output edges:
```scheme
(infra-node infra/tf-namespace :kind terraform :resource kubernetes_namespace)
(infra-node infra/tf-secrets :kind terraform :resource module)
(tf-depends infra/tf-secrets infra/tf-namespace :via module-reference)
```
S-expression output format
---------------------------
New fact types (appended to `.belief_map.sexp`):
```scheme
; Infrastructure nodes
(infra-node <id> :kind <deployment|statefulset|helm-chart|terraform> :image <image> :port <port> :component <label>)
; Kustomize service dependencies
(k8s-depends <src-service> <tgt-service> :via <env|dns|volume> :transport <http|tcp|amqp|grpc>)
; Kustomize service declarations
(k8s-service <service-id> <dns-name> :namespace <ns> :port <port>)
; Helm chart dependencies
(helm-depends <src-chart> <tgt-chart> :via <chart-dependency|values-ref>)
; Terraform module dependencies
(tf-depends <src-module> <tgt-module> :via <module-reference|depends-on|variable>)
; Cross-layer: maps infra service to source-code repo
(infra-maps infra/app-service app-service :via image-name)
```
The `infra-maps` edge connects infrastructure nodes to source-code nodes, enabling queries like:
```scheme
(deps "infra/app-service" 2) ; what infra does app-service depend on?
(rdeps "infra/drive-postgres" 1) ; which services use this database?
```
Error handling and UX
---------------------
- Missing YAML files: skip with `; warning: skipped <path>: file not found` comment in output
- Malformed YAML: skip resource, emit `; error: parse failed <path>: <reason>`
- Missing PyYAML: exit with clear error message and install instructions
- Missing hcl2 (for terraform): fall back to regex-based parsing, emit `; warning: terraform parsed with regex fallback`
- No IaC files found: emit `; info: no infrastructure files found` and exit cleanly
Update cadence / Lifecycle
--------------------------
Infrastructure topology changes rarely compared to source code. The scripts should:
- Run only when `--infra` flag is passed to `build_belief_map.py`
- OR when the user explicitly runs `build_infra_topology.py`
- Cache results alongside `.belief_map_cache.json` using mtime-based invalidation on IaC files
Future-proofing
---------------
- **Tilt / docker-compose**: same intermediate format, different parser. Add `--tilt` and `--compose` flags.
- **Ingress topology**: parse `ingress.yaml` / `Ingress` resources for external routing
- **Multi-namespace**: extend `infra-node` with `:namespace` field
- **Service mesh**: parse Istio/Linkerd VirtualService/DestinationRule for traffic routing edges
- **GitOps**: parse ArgoCD Application manifests for deployment pipeline edges
- **Runtime enrichment**: optional `--live` flag to query kubectl for actual pod status
Implementation outline
----------------------
### Phase 1: Core infrastructure (shared)
1. Define `InfraNode` and `InfraEdge` dataclasses in `build_infra_topology.py`
2. Define env-var heuristic patterns for service/infra detection
3. Implement sexp writer for new fact types
4. Add `--infra` flag to `build_belief_map.py` that calls the topology builder post-pass
### Phase 2: Kustomize parser
1. Walk directories for `kustomization.yaml`
2. Parse resource references -> find deployment/service YAML
3. Extract service names, images, ports, env vars
4. Apply env-var heuristics to detect dependencies
5. Emit `infra-node`, `k8s-depends`, `k8s-service` facts
### Phase 3: Helm parser
1. Walk directories for `Chart.yaml`
2. Parse Chart.yaml dependencies
3. Parse values.yaml for image/port/env configuration
4. Parse templates/*.yaml for env var references (handle Go template syntax)
5. Emit `infra-node`, `helm-depends` facts
### Phase 4: Terraform parser
1. Walk directories for `main.tf`
2. Parse module blocks and resource blocks (via hcl2 or regex)
3. Build dependency graph from cross-module references
4. Emit `infra-node`, `tf-depends` facts
### Phase 5: Cross-layer mapping
1. For each `infra-node` with `:image`, match image name against known repo names
2. Emit `infra-maps` edges connecting infrastructure to source-code graph
3. Update `belief_search.py` to parse new edge types
### Phase 6: Integration
1. Add new edge types to `belief_search.py` edge parser
2. Update `sexp-notation.md` reference with new fact types
3. Update SKILL.md with infrastructure query examples
Testing approach
----------------
**Unit tests:**
- Kustomize parser: fixture kustomization.yaml + deployment.yaml -> expected InfraNode/InfraEdge list
- Helm parser: fixture Chart.yaml + values.yaml -> expected output
- Terraform parser: fixture main.tf with modules -> expected dependency graph
- Env-var heuristics: input env var name/value -> expected dependency type
**Integration tests:**
- Run kustomize parser against `mono/app-service/.k8s/base/` -> verify app-service depends on postgres
- Run helm parser against `nextnet-avsync-streaming/charts/` -> verify chart dependencies
- Run terraform parser against `avsync-stream-lab/infra/terraform/` -> verify module dependencies
**Manual verification:**
- Run `build_infra_topology.py` on mono workspace
- Query `belief_search.py rdeps infra/drive-postgres 1` -> verify all drive consumers listed
- Query `belief_search.py deps infra/app-service 2` -> verify database + queue dependencies shown
Acceptance criteria
-------------------
1. Given a workspace with `.k8s/base/` directories, when running `build_infra_topology.py --kustomize .`, then `infra-node` facts are emitted for each deployment with correct service name, image, and port.
2. Given a deployment.yaml with `DATABASE_URL` env var, when parsing, then a `k8s-depends` edge is emitted to the postgres service with `:transport tcp`.
3. Given a kustomization.yaml with `resources: [app-service, app-web, postgres]`, when parsing, then all three services appear as `infra-node` facts.
4. Given a Helm Chart.yaml with `dependencies`, when parsing, then `helm-depends` edges are emitted for each dependency.
5. Given Terraform modules with cross-references (`module.namespace.name`), when parsing, then `tf-depends` edges reflect the dependency order.
6. Given an `infra-node` with `:image upgraide/app-service`, when cross-layer mapping runs, then an `infra-maps` edge connects `infra/app-service` to the source-code `app-service` node.
7. Given `--infra` flag passed to `build_belief_map.py`, when building completes, then infrastructure facts appear in `.belief_map.sexp` alongside source-code facts.
8. Given no IaC files in the workspace, when running the topology builder, then it exits cleanly with an informational message and no errors.
9. `belief_search.py` correctly loads and queries `infra-node`, `k8s-depends`, `helm-depends`, `tf-depends`, and `infra-maps` edges.
Estimated scope
---------------
| Component | Lines | Effort |
|-----------|-------|--------|
| Shared infra types + sexp writer | ~80 | Small |
| Kustomize parser | ~200 | Medium |
| Helm parser | ~150 | Medium |
| Terraform parser | ~120 | Medium |
| Cross-layer mapping | ~40 | Small |
| belief_search.py integration | ~10 | Trivial |
| Tests | ~150 | Medium |
| **Total** | **~750** | ~2 days |

11
.gitignore vendored Normal file
View File

@@ -0,0 +1,11 @@
# macOS
.DS_Store
# Python
__pycache__/
*.py[cod]
.ruff_cache/
# Generated belief-map artifacts (produced by scripts/build_belief_map.py)
.belief_map.sexp
.belief_map_cache.json

9
LOG.md Normal file
View File

@@ -0,0 +1,9 @@
## 2026-06-19
### Changes
- Updated the codespaces skill workflow so agents are directed to query the belief map before reading source and to use `quick` as a first-pass `analyze` shortcut.
- Changed `belief_search.py quick` to run full `analyze` output instead of boundary-only output, and made it resolve legacy path-map aliases found in raw belief-map hits.
- Added focused unit coverage for quick analyze output, alias resolution from raw hits, boundary files-only output, and no-match suggestions.
### Motivation
- Session-log review showed that agents usually used `search` and `analyze` after activation, but often queried too late and did not discover shortcut or boundary commands.

254
SKILL.md Normal file
View File

@@ -0,0 +1,254 @@
---
name: codespaces
description: >
Query `.belief_map.sexp` before non-trivial code changes to find module boundaries,
dependencies, blast radius, and the minimal files to read. Use for architecture
discovery, impact analysis, and boundary checks.
---
# Codespaces -- Belief Map Architecture Discovery
Query `.belief_map.sexp` to discover modules, boundaries, and dependencies before reading source files. Never read code blindly -- query the graph first.
## Scripts
This skill bundles Python scripts. Run them from the skill's base directory:
- `scripts/belief_search.py` -- query the belief map graph
- `scripts/build_belief_map.py` -- generate the belief map from source
- `scripts/build_infra_topology.py` -- extract Kustomize, Helm, and Terraform topology facts
- `scripts/git_descendants.py` -- find all commits descending from HEAD (or a given ref)
Never use project-local copies of these scripts -- they may be outdated and output JSON instead of sexp.
## Prerequisite
Locate the belief map. If absent, build it:
```bash
ls .belief_map.sexp belief_map.sexp 2>/dev/null
# Build: pip3 install tree-sitter tree-sitter-typescript tree-sitter-python
python3 scripts/build_belief_map.py # incremental ~5s
python3 scripts/build_belief_map.py --full # full rebuild ~30s
```
## Mandatory Workflow: Query Before Code
For non-trivial code work, run a belief-map query before `rg`, `find`, `sed`,
`cat`, `nl`, opening directories, or reading source files. The skill only helps
when it is used as the first scoping step, not as confirmation after source has
already been inspected.
**Fast first pass when the task has a clear keyword:**
```bash
python3 scripts/belief_search.py quick "keyword"
```
`quick` searches for the first matching module and immediately runs `analyze`.
Use it to get boundary files, dependencies, reverse dependencies, layers, and
violations in one step. If `quick` returns the wrong module or no match, use the
explicit `search` -> `analyze` flow below.
**Step 1 -- Find the module ID:**
```bash
python3 scripts/belief_search.py search "keyword"
python3 scripts/belief_search.py entity EntityName
```
Use `search` or `entity` to locate the module ID. Module IDs are **full relative paths** like `pki-service/src/modules/ca/ca.service`, not short codes like `ps34`. The belief map may contain short path-map IDs internally, but they are resolved automatically -- always pass human-readable path-like strings. Do NOT guess or invent module IDs -- always confirm with `search` first.
**Step 2 -- Analyze the module (do this IMMEDIATELY, not optionally):**
```bash
python3 scripts/belief_search.py analyze <module-id>
```
`analyze` is the PRIMARY command. It returns the full context: boundary files, dependencies, reverse dependencies, architecture layer, and violations. Always run it before reading any source.
**Step 3 -- Read only the boundary files listed in the analyze output.**
**Step 4 -- Do NOT read entire directories.** If the boundary shows 2-3 relevant files, read only those files. Ignore everything else in the directory.
## Command Reference
| Command | Purpose | Example |
|---|---|---|
| `search "<pattern>"` | Regex grep on graph (find module IDs) | `search "operative.*service"` |
| `analyze <id>` | Full module analysis (PRIMARY command) | `analyze drive/modules/drive/drive_schema` |
| `quick <keyword>` | Search + analyze first match (shortcut) | `quick operatives` |
| `boundary <id>` | Files to read for a change | `boundary operatives.service` |
| `boundary <id> --files-only` | File paths only (compact) | `boundary operatives.service --files-only` |
| `entity <Name>` | Find definitions + references | `entity DrivePath` |
| `deps <id> [depth]` | Outgoing dependency tree | `deps operatives.service 3` |
| `rdeps <id> [depth]` | Blast radius (who depends on this) | `rdeps drive_schema 2` |
| `flow <id> <fn>` | Trace call/data flow | `flow drive_api fetchFiles` |
| `module <id>` | Single module with edges | `module operatives.controller` |
| `boundaries [id\|all]` | Check architecture violations | `boundaries all` |
| `invariants [id\|all]` | Check naming convention violations | `invariants all` |
| `layers` | Show all modules by layer | `layers` |
| `find_function <name>` | Find function/method definitions | `find_function createOrder` |
| `find_type <name>` | Find type/class/ifc/enum definitions | `find_type DriveDbCommitResponse` |
| `find_callchain <src> <tgt> [depth]` | Trace call path between functions | `find_callchain createOrder processPayment` |
| `find_callers <fn> [depth]` | Who calls this function | `find_callers handleRequest 3` |
| `find_calls <fn> [depth]` | What does this function call | `find_calls createOrder 2` |
| `grep_functions <pattern>` | Search function bodies in source | `grep_functions "validate.*input"` |
| `diff_functions [ref]` | Changed functions in git diff | `diff_functions HEAD~3` |
| `query '<sexp>'` | Composable Scheme query | `query '(count (boundary "X"))'` |
| `repl` | Interactive Scheme REPL | `repl` |
## Quick Reference: rg Queries
When you need fast, targeted lookups without invoking the Python script, query the belief map directly with `rg`. The bundled builder writes `.belief_map.sexp`; if a repo still uses the legacy `belief_map.sexp`, substitute that filename in the commands below:
```bash
# All facts about a module
rg 'drive/modules/drive_db/api ' .belief_map.sexp
# Who imports a module
rg 'imports .* drive/modules/commons/result' .belief_map.sexp
# What does a module import
rg 'imports drive/modules/drive_db/api' .belief_map.sexp
# All references to an entity
rg 'DriveDbCommitResponse' .belief_map.sexp
# All classes in a repo
rg '^\(cls drive/' .belief_map.sexp
# Data flow edges
rg 'data-flow' .belief_map.sexp
```
These one-liners are useful for quick checks. For structured output and full analysis, use the Python script commands.
## Output Notation
All output is S-expression facts, one per line. See [references/sexp-notation.md](references/sexp-notation.md) for the full format.
Key patterns:
```scheme
(boundary <mod> :lang py :file <path> :purpose "<desc>")
(boundary-dep <mod> <dep> :lang ts :file <path> :relation imports)
(boundary-rdep <mod> <rdep> :lang tsx :file <path> :relation imports)
(boundary-summary <mod> :total 10 :deps 7 :rdeps 2)
(entity-def cls <mod> <Name> <line> :lang py :file <path>)
(violation <src> <tgt> :src-layer domain :tgt-layer api :via imports :lang ts "<reason>")
(result <module-or-path>)
(result-count 42)
```
**`:lang` values**: `py` = Python (snake_case), `ts` = TypeScript (camelCase), `tsx` = TSX (PascalCase).
## Scheme Query Language
Compose queries for complex analysis. See [references/scheme-queries.md](references/scheme-queries.md) for full reference.
**Essentials:**
```scheme
(boundary "id") ; target + deps + rdeps (change boundary)
(entity "Name") ; modules defining entity
(deps "id" depth) ; outgoing dependencies
(rdeps "id" depth) ; reverse dependencies
(find-function "name") ; modules with matching function defs
(find-type "name") ; modules with matching type defs
(find-callers "fn" depth) ; modules calling a function
(find-calls "fn" depth) ; modules a function calls
(intersect A B) ; set intersection
(filter SET "pattern") ; regex filter
(files SET) ; resolve to file paths
(count SET) ; count members
(violations) ; modules with boundary violations
(layer "id") ; architecture layer of a module
```
## Analysis Plan Pattern
Before modifying code, compose a plan from queries:
```scheme
; 1. Scope
(boundary "operatives.service")
(count (boundary "operatives.service"))
; 2. Architecture constraints
(violations "operatives.service")
(layer "operatives.service")
; 3. Cross-language boundaries
(filter (refs-to "DrivePath") "app-service")
; 4. Narrow to actionable files
(filter (boundary "operatives.service") "dto")
(intersect (deps "controller" 2) (deps "service" 2))
; 5. After changes: verify
(violations "operatives.service")
```
## Trigger Examples
Use this skill before reading source for:
- Root-cause investigations that mention services, APIs, jobs, events, UI flows, or failing behavior.
- Cross-service or cross-module questions, including "how does X call Y?" and "where is this stored?"
- Redmine clarification when the answer depends on code behavior.
- Architecture reviews, impact analysis, public contract changes, or blast-radius checks.
- Reviews of non-trivial staged changes when dependency direction or module ownership matters.
Do not force it for commit-only tasks, git-log summaries, manual demo/server runs, pure browser testing, or session-log analysis.
## Error Recovery
If a command returns `(error no-match ...)`:
1. Do NOT retry with the same ID or guess another ID
2. Run `search` with a broader keyword to find the correct module ID
3. Use the `:suggestions` field if present -- it lists similar module IDs
```bash
# Wrong: guessing IDs
python3 scripts/belief_search.py analyze ps34 # fails: ps34 is a short path-map ID
python3 scripts/belief_search.py analyze ca.service # fails: incomplete path
# Right: search first, then use the full path from results
python3 scripts/belief_search.py search "ca.*service"
# Output shows: pki-service/src/modules/ca/ca.service
python3 scripts/belief_search.py analyze pki-service/src/modules/ca/ca.service
```
## Anti-patterns
- **Guessing module IDs.** Module IDs are full relative paths (e.g., `pki-service/src/modules/ca/ca.service`). Short codes like `ps34` are internal path-map IDs and must not be used directly. Always `search` first.
- **Build but never query.** Building the belief map and then never running `search`, `analyze`, or `boundary` provides zero value.
- **Querying after reading source.** The first source-code read should come from `boundary-file` lines in `analyze` output.
- **Reading `.belief_map.sexp` as a whole file.** Use `search`, `analyze`, or `rg` to query specific facts.
- **Skipping the belief map for cross-module tasks.** Always query the graph to understand boundaries before reading source files.
- **Reading all files in a directory.** When the boundary shows 2-3 files, read only those.
- **Using project-local scripts.** Always use this skill's bundled scripts. Project copies may output JSON instead of sexp.
## Rebuilding the Graph
```bash
python3 scripts/build_belief_map.py # incremental (~5s)
python3 scripts/build_belief_map.py --full # full rebuild (~30s)
python3 scripts/build_belief_map.py ./drive # single sub-repo
python3 scripts/build_belief_map.py --lsp # with LSP call hierarchy (~1-5min)
```
Rebuild after structural changes (add/remove files, change imports, add classes).
## Infrastructure Topology
Extract service topology from Kustomize, Helm, and Terraform files. This script requires `PyYAML` (`pip3 install pyyaml`).
```bash
python3 scripts/build_infra_topology.py # auto-detect all IaC
python3 scripts/build_infra_topology.py --kustomize . # kustomize only
python3 scripts/build_infra_topology.py --helm ./charts # helm charts only
python3 scripts/build_infra_topology.py --terraform ./infra # terraform only
python3 scripts/build_infra_topology.py --append .belief_map.sexp # merge into graph
```
Emits `infra-node`, `k8s-depends`, `k8s-service`, `helm-depends`, `tf-depends`, and `infra-maps` edges. Query with:
```bash
rg 'infra-node' .belief_map.sexp # all services
rg 'k8s-depends' .belief_map.sexp # service dependencies
rg 'infra-node.*:component database' .belief_map.sexp # all databases
rg 'infra-maps.*app-service' .belief_map.sexp # infra -> source code mapping
```

88
references/scheme-queries.md Normal file
View File

@@ -0,0 +1,88 @@
# Scheme Query Language Reference
Use `query '<expr>'` for one-shot or `repl` for interactive mode.
## Primitives (return module sets)
```scheme
(resolve "pattern") ; find module IDs matching pattern (suffix > substring)
(boundary "id") ; target + imports + dependents (change boundary)
(deps "id" depth) ; outgoing dependencies (default depth 2)
(rdeps "id" depth) ; reverse dependencies (default depth 1)
(imports "id") ; direct import targets only
(dependents "id") ; direct dependents only
(entity "Name") ; modules defining an entity
(refs-to "Name") ; modules referencing an entity
```
## Set Operations
```scheme
(intersect A B) ; set intersection
(union A B ...) ; set union (variadic)
(diff A B) ; A minus B
(filter SET "pattern") ; keep members matching regex (case-insensitive)
```
## Boundary Analysis
```scheme
(layer "id") ; architecture layer: domain/api/service/infra/shared/test/ui/config/other
(layer-of SET "layer") ; filter set to modules in a specific layer
(violations) ; all modules with boundary violations
(violations "id") ; violations for a specific module's scope
```
## Function & Type Search (return module-set)
```scheme
(find-function "name") ; modules containing matching function definitions
(find-type "name") ; modules containing matching type/class/ifc/enum defs
(find-callers "fn" depth) ; modules that call a function (default depth 2)
(find-calls "fn" depth) ; modules that a function calls (default depth 2)
(diff-functions "ref") ; print changed functions (side-effect, returns empty)
```
## Output Transforms
```scheme
(files SET) ; resolve module IDs to file paths
(count SET) ; count set members (returns int)
(analyze "id") ; run full analysis (prints sexp, returns empty set)
(search "pattern") ; grep .belief_map.sexp (prints matches, returns empty set)
```
## Composition Examples
```scheme
; Files shared between two modules
(intersect (deps "controller" 2) (deps "service" 2))
; Only DTOs in a boundary
(filter (boundary "operatives.service") "dto")
; How many files in the blast radius?
(count (deps "drive_schema" 3))
; Which TS modules reference a Python entity?
(filter (refs-to "DrivePath") "app-service")
; Domain modules that violate boundaries
(violations "all")
; Infra files in the boundary
(layer-of (boundary "operatives.service") "infra")
; Files to read (resolved paths)
(files (filter (boundary "X") "service"))
```
## Result Format
```scheme
(result <module-or-path>) ; one per set member
(result-empty) ; empty set
(result-count 42) ; from (count ...)
```
String results (from `layer`) print as `(result <value>)`.

176
references/sexp-notation.md Normal file
View File

@@ -0,0 +1,176 @@
# S-expression Output Notation
Every output line is a self-contained fact. Each fact that references a module carries `:lang` (py/ts/tsx).
**Module ID**: relative path from workspace root, no extension, no `/index`. Example: `drive/modules/drive_db/api/drive_db_api.ts` -> `drive/modules/drive_db/api/drive_db_api`.
## Raw Graph Facts (in .belief_map.sexp)
```scheme
(node <id> <lang> "<purpose>" :naming <convention> :pkg <repo>)
(fn <mod> <name> <line> :eid <uuid>)
(cls <mod> <name> <line> :eid <uuid> (:bases A) (:deco D) (:methods m1 m2))
(ifc <mod> <name> <line> :eid <uuid> (:methods m1 m2))
(typ <mod> <name> <line> :eid <uuid>)
(enm <mod> <name> <line> :eid <uuid>)
(imports <src> <tgt> [:via-base] [:lsp])
(calls-api <src> <tgt> :via-ifc [:lsp])
(data-flow <src> <tgt> [:validated])
(refs <src> <tgt>::<Entity> [:lsp])
(calls <src>::<fn> <tgt>::<fn> :lines L1 L2 :lsp)
```
## boundary Output
```scheme
(boundary <mod> :lang py :file <path> :purpose "<desc>")
(cls <mod> <Name> <line> :lang py (:bases A) (:deco D) (:methods m1 m2))
(boundary-dep <mod> <dep> :lang ts :file <path> :relation imports)
(boundary-rdep <mod> <rdep> :lang py :file <path> :relation imports)
(boundary-summary <mod> :total 10 :deps 7 :rdeps 2)
```
## analyze Output
```scheme
(analyze <mod> :lang py "<purpose>" :file <path>)
(cls <mod> <Name> <line> :lang py (:bases A) (:deco D) (:methods m1 m2))
(analyze-import <mod> <tgt> :lang ts :file <path>)
(analyze-dataflow <mod> <tgt> :lang py :validated)
(refs <src> <tgt>::<Entity>)
(analyze-dependent <mod> <src> :lang tsx :via imports refs :file <path>)
(dep <root> <tgt> :lang py :via imports :depth 1)
(rdep <root> <src> :lang tsx :via imports :depth 1)
(key-entity <mod> <Name> :lang py :kind cls :line 32 :refs 7 :modules 4)
(key-entity-ref <Name> <ref-mod> :lang ts)
(boundary-file <mod> <file-path>)
(layer <mod> <dep-name> :layer domain)
(analyze-summary <mod> :entities 31 :imports 1 :dataflows 0 :refs 1 :dependents 50 :boundary-files 52)
```
## entity Output
```scheme
(entity-def cls <mod> <Name> <line> :lang py :file <path>)
(entity-def ifc <mod> <Name> <line> :lang ts :file <path>)
(refs <src-mod> <tgt-mod>::<Name>)
```
## deps / rdeps Output
```scheme
(deps-root <mod> :lang py)
(dep <root> <tgt> :lang ts :via imports :depth 1)
(dep <root> <tgt> :lang py :via data-flow :depth 2 :cycle)
(rdeps-root <mod> :lang ts)
(rdep <root> <src> :lang tsx :via imports :depth 1)
```
## flow Output
```scheme
(flow <src>::<fn> <tgt>::<fn> :lang py :via calls :depth 0)
(flow <src>::<fn> <tgt> :lang ts :via imports :depth 1 :module-level)
```
## boundaries Output
```scheme
(violation <src> <tgt> :src-layer domain :tgt-layer api :via imports :lang ts "<reason>")
(boundaries-summary :violations 2 :checked 3587)
(boundaries-ok :checked 3587)
```
## invariants Output
```scheme
(invariant-violation <mod> :expected snake_case :actual PascalCase :lang py :pkg <repo>)
(invariants-summary :violations 12 :checked 3587)
(invariants-ok :checked 3587)
```
## layers Output
```scheme
(layer-group api :count 144)
(layer-member api <mod> :lang ts)
```
## find_function / find_type Output
```scheme
(fn-def <mod>::<name> :line 42 :lang ts :file <path> :kind function)
(fn-def <mod>::<cls>.<method> :line 42 :lang py :file <path> :kind method)
(type-def cls <mod>::<Name> :line 10 :lang ts :file <path> (:bases A) (:methods m1 m2))
(type-def ifc <mod>::<Name> :line 10 :lang ts :file <path>)
```
## find_callers / find_calls Output
```scheme
(callers-target <mod>::<fn> :line 42 :lang ts :file <path>)
(caller <src>::<fn> -> <tgt>::<fn> :lang ts :file <path> :lines L1 L2 :lsp)
(caller-module <src> -> <tgt> :via imports :depth 1 :lang ts :file <path>)
(calls-source <mod>::<fn> :line 42 :lang ts :file <path>)
(calls-target <src>::<fn> -> <tgt>::<fn> :lang ts :file <path>)
(calls-module <src> -> <tgt> :via imports :depth 1 :lang ts :file <path>)
```
## find_callchain Output
```scheme
(callchain-step 0 <mod>::<fn> :lang ts :file <path>)
(callchain-step 1 <mod>::<fn> :lang py :file <path>)
(callchain-summary :length 3 :hops 2)
(callchain-module-step 0 <mod> :lang ts :file <path>)
(callchain-module-summary :length 2 :hops 1)
(callchain-none "srcFn" -> "tgtFn" :max-depth 5)
```
## grep_functions Output
```scheme
(grep-fn <mod>::<fn> :line 55 :lang ts :file <path> :match "matched line text")
(grep-functions-summary :matches 12 :pattern "pattern")
(grep-functions-empty "pattern")
```
## diff_functions Output
```scheme
(diff-fn <mod>::<name> :kind fn :line 42 :lang ts :file <path>)
(diff-functions-summary :changed 5 :files 3 :ref "HEAD")
(diff-file <path> :hunks 2)
(diff-functions-no-match :files 3 :ref "HEAD")
(diff-functions-empty "HEAD")
```
## query / repl Output
```scheme
(result <module-or-path>)
(result-empty)
(result-count 42)
```
## :lang Values
| Value | Language | Naming | Example |
|---|---|---|---|
| `py` | Python | snake_case fns, PascalCase classes | `drive_path_to_entry`, `DrivePath` |
| `ts` | TypeScript | camelCase fns, PascalCase classes | `calculateTotal`, `OperativesService` |
| `tsx` | TSX/React | PascalCase components, camelCase hooks | `FileBrowser`, `useQuery` |
## Architecture Layers
| Layer | Path patterns | Role |
|---|---|---|
| `domain` | `/domain/`, `/entities/`, `_schema` | Pure business logic |
| `api` | `/api/`, `/routes/`, `/controllers/` | HTTP/RPC handlers |
| `service` | `/services/`, `/application/` | Business orchestration |
| `infra` | `/repositories/`, `/db/`, `/infra/` | Database, storage, messaging |
| `shared` | `/commons/`, `/utils/`, `/types/` | Cross-cutting utilities |
| `test` | `/tests/`, `.test.`, `.spec.` | Test files |
| `ui` | `/components/`, `/pages/`, `/hooks/` | Frontend UI |
| `config` | `/config/`, `/env/` | Configuration |
| `other` | (no match) | Unclassified |

2256
scripts/belief_search.py Normal file
View File

File diff suppressed because it is too large Load Diff

3182
scripts/build_belief_map.py Executable file
View File

File diff suppressed because it is too large Load Diff

853
scripts/build_infra_topology.py Normal file
View File

@@ -0,0 +1,853 @@
from __future__ import annotations
"""build_infra_topology.py -- scan Kustomize, Helm, and Terraform files and
emit infra topology facts in sexp format.
Usage:
python3 build_infra_topology.py [ROOT] [OPTIONS]
Options:
ROOT Workspace root (default: .)
--kustomize Scan for kustomize files (default: auto-detect)
--helm Scan for helm charts (default: auto-detect)
--terraform Scan for terraform files (default: auto-detect)
--append FILE Append output to existing sexp file
--stdout Print to stdout (default if no --append)
"""
import os
import re
import sys
from dataclasses import dataclass, field
from pathlib import Path
from typing import Optional
# ---------------------------------------------------------------------------
# PyYAML guard
# ---------------------------------------------------------------------------
try:
import yaml # type: ignore[import-untyped]
except ImportError:
print(
"Error: PyYAML is required. Install it with: pip install pyyaml",
file=sys.stderr,
)
sys.exit(1)
# ---------------------------------------------------------------------------
# Constants
# ---------------------------------------------------------------------------
SKIP_DIRS: frozenset[str] = frozenset(
{"node_modules", ".git", ".venv", "dist", "build", "coverage", "target", "__pycache__"}
)
KUSTOMIZE_FILENAMES: tuple[str, ...] = ("kustomization.yaml", "kustomization.yml")
HELM_CHART_FILENAME = "Chart.yaml"
TERRAFORM_MAIN_FILENAME = "main.tf"
# Env-var heuristic patterns: (compiled_regex, inferred_dep_kind, transport)
# The dep_kind is either a generic service hint or a named infrastructure type.
_ENV_HEURISTICS: list[tuple[re.Pattern[str], str, str]] = [
(re.compile(r"DATABASE_URL|POSTGRES_HOST|POSTGRES_URL|PG_HOST|PG_URL|PGHOST"), "postgres", "tcp"),
(re.compile(r"RABBITMQ_HOST|RABBITMQ_URL|AMQP_HOST|AMQP_URL|AMQP_URI"), "rabbitmq", "amqp"),
(re.compile(r"NATS_URL|NATS_HOST|NATS_URI"), "nats", "tcp"),
(re.compile(r"REDIS_HOST|REDIS_URL|REDIS_URI"), "redis", "tcp"),
(re.compile(r"S3_ENDPOINT|S3_HOST|MINIO_HOST|MINIO_URL|AWS_S3_ENDPOINT"), "minio", "http"),
(re.compile(r"VAULT_ADDR|VAULT_HOST|VAULT_URL"), "vault", "http"),
# Generic service references come last to avoid shadowing the above
(re.compile(r"(.+?)_HOST$|(.+?)_URL$|(.+?)_SERVICE_HOST$|(.+?)_ENDPOINT$"), "service", "http"),
]
# Module-reference pattern inside Terraform: module.<name>.<attr>
_TF_MODULE_REF_RE = re.compile(r"module\.(\w+)\.")
# Terraform block parsers (no hcl2 dependency)
_TF_MODULE_BLOCK_RE = re.compile(r'module\s+"(\w+)"\s*\{([^}]*)\}', re.DOTALL)
_TF_MODULE_SOURCE_RE = re.compile(r'source\s*=\s*"([^"]+)"')
_TF_RESOURCE_BLOCK_RE = re.compile(r'resource\s+"(\w+)"\s+"(\w+)"\s*\{')
# ---------------------------------------------------------------------------
# Domain types
# ---------------------------------------------------------------------------
@dataclass(frozen=True)
class InfraNode:
id: str # infra/<name>
kind: str # deployment | statefulset | helm-chart | terraform
image: str # container image or ""
port: int # primary port or 0
component: str # k8s app.kubernetes.io/component label or ""
source_file: str # path to the manifest
@dataclass(frozen=True)
class InfraEdge:
source: str # infra/<name>
target: str # infra/<name>
edge_type: str # k8s-depends | helm-depends | tf-depends | infra-maps | k8s-service
via: str # env | dns | chart-dependency | module-reference | image-name
transport: str # http | tcp | amqp | grpc | ""
# For k8s-service facts the target holds the dns-name and extra is namespace
extra: str = ""
@dataclass
class TopologyResult:
nodes: list[InfraNode] = field(default_factory=list)
edges: list[InfraEdge] = field(default_factory=list)
warnings: list[str] = field(default_factory=list)
# ---------------------------------------------------------------------------
# Pure utilities
# ---------------------------------------------------------------------------
def _infra_id(name: str) -> str:
"""Return the canonical infra/ prefixed node id for a service name."""
return f"infra/{name}"
def _safe_load_yaml(path: Path, warnings: list[str]) -> Optional[dict]:
"""Load a YAML file, appending a warning on failure and returning None."""
try:
with path.open("r", encoding="utf-8", errors="replace") as fh:
data = yaml.safe_load(fh)
if not isinstance(data, dict):
return None
return data
except FileNotFoundError:
warnings.append(f"; warning: skipped {path}: file not found")
return None
except yaml.YAMLError as exc:
warnings.append(f"; error: parse failed {path}: {exc}")
return None
def _walk_dirs(root: Path) -> list[Path]:
"""Yield all sub-directories under root, skipping SKIP_DIRS."""
results: list[Path] = []
for dirpath, dirnames, _ in os.walk(root):
dirnames[:] = [d for d in dirnames if d not in SKIP_DIRS]
results.append(Path(dirpath))
return results
def _infer_dep_from_env(env_name: str) -> Optional[tuple[str, str]]:
"""Return (dep_kind, transport) inferred from an env var name, or None."""
for pattern, dep_kind, transport in _ENV_HEURISTICS:
m = pattern.match(env_name)
if m:
if dep_kind == "service":
# Extract the service name from the prefix captured group
prefix = next((g for g in m.groups() if g), None)
if prefix:
# Convert SCREAMING_SNAKE to kebab-case service name
service_name = prefix.lower().replace("_", "-")
return (service_name, transport)
return None
return (dep_kind, transport)
return None
def _extract_env_deps(
env_list: list[dict],
source_id: str,
known_node_names: frozenset[str],
) -> list[InfraEdge]:
"""Return dependency edges inferred from a list of k8s env var dicts."""
edges: list[InfraEdge] = []
seen: set[str] = set()
for env in env_list:
name = env.get("name", "")
if not name:
continue
result = _infer_dep_from_env(name)
if result is None:
continue
dep_kind, transport = result
# Resolve target: prefer a known infra node by name, else use the dep_kind
target_name = dep_kind
# Check if any known node name contains the dep_kind as a substring
for node_name in known_node_names:
if dep_kind in node_name or node_name.endswith(dep_kind):
target_name = node_name
break
target_id = _infra_id(target_name)
key = f"{source_id}->{target_id}"
if key in seen:
continue
seen.add(key)
edges.append(
InfraEdge(
source=source_id,
target=target_id,
edge_type="k8s-depends",
via="env",
transport=transport,
)
)
return edges
# ---------------------------------------------------------------------------
# Kustomize parser
# ---------------------------------------------------------------------------
def _parse_kustomization_resources(
kustom_path: Path,
warnings: list[str],
) -> list[str]:
"""Return the list of resource paths declared in a kustomization.yaml."""
data = _safe_load_yaml(kustom_path, warnings)
if data is None:
return []
resources = data.get("resources", []) or []
return [str(r) for r in resources if r]
def _find_workload_manifest(resource_dir: Path) -> Optional[Path]:
"""Find the primary workload manifest inside a resource directory."""
candidates = (
"deployment.yaml",
"deployment.yml",
"statefulset.yaml",
"statefulset.yml",
)
for name in candidates:
p = resource_dir / name
if p.is_file():
return p
return None
def _find_service_manifest(resource_dir: Path) -> Optional[Path]:
"""Find the service manifest inside a resource directory."""
for name in ("service.yaml", "service.yml"):
p = resource_dir / name
if p.is_file():
return p
return None
def _parse_service_port(service_path: Path, warnings: list[str]) -> tuple[int, str]:
"""Return (port, namespace) from a Service manifest."""
data = _safe_load_yaml(service_path, warnings)
if data is None:
return (0, "")
namespace: str = (data.get("metadata") or {}).get("namespace", "")
ports: list[dict] = ((data.get("spec") or {}).get("ports") or [])
port: int = 0
if ports:
port = int(ports[0].get("port", 0))
return (port, namespace)
def _extract_container_env(container: dict) -> list[dict]:
"""Return the flat env list from a container spec (env only, not envFrom)."""
return container.get("env", []) or []
def parse_kustomize(root: Path) -> TopologyResult:
"""Scan root for kustomization.yaml files and extract infra topology."""
result = TopologyResult()
kustom_files: list[Path] = []
for d in _walk_dirs(root):
for fname in KUSTOMIZE_FILENAMES:
p = d / fname
if p.is_file():
kustom_files.append(p)
if not kustom_files:
return result
# First pass: collect all node names for dependency resolution
workload_data: list[tuple[Path, Path, dict]] = [] # (kustom_path, manifest, data)
for kustom_path in kustom_files:
kustom_dir = kustom_path.parent
for resource_str in _parse_kustomization_resources(kustom_path, result.warnings):
resource_dir = kustom_dir / resource_str
if not resource_dir.is_dir():
# Could be a direct file reference
continue
manifest = _find_workload_manifest(resource_dir)
if manifest is None:
continue
data = _safe_load_yaml(manifest, result.warnings)
if data and data.get("kind") in ("Deployment", "StatefulSet"):
workload_data.append((kustom_path, manifest, data))
# Collect known names for env heuristic resolution
known_names: set[str] = set()
for _, _, data in workload_data:
name = (data.get("metadata") or {}).get("name", "")
if name:
known_names.add(name)
known_names_frozen = frozenset(known_names)
# Second pass: emit nodes and edges
for kustom_path, manifest, data in workload_data:
kustom_dir = kustom_path.parent
k8s_kind = data.get("kind", "Deployment").lower()
metadata = data.get("metadata", {}) or {}
name: str = metadata.get("name", "")
if not name:
continue
labels = metadata.get("labels", {}) or {}
component: str = labels.get("app.kubernetes.io/component", "")
spec = data.get("spec", {}) or {}
template = spec.get("template", {}) or {}
pod_spec = template.get("spec", {}) or {}
containers: list[dict] = pod_spec.get("containers", []) or []
if not containers:
continue
first_container = containers[0]
image: str = first_container.get("image", "")
ports: list[dict] = first_container.get("ports", []) or []
port: int = int(ports[0].get("containerPort", 0)) if ports else 0
node_id = _infra_id(name)
result.nodes.append(
InfraNode(
id=node_id,
kind=k8s_kind,
image=image,
port=port,
component=component,
source_file=str(manifest.relative_to(root) if manifest.is_relative_to(root) else manifest),
)
)
# Service manifest for k8s-service fact
# Determine the resource dir from the manifest's parent
resource_dir = manifest.parent
svc_manifest = _find_service_manifest(resource_dir)
if svc_manifest:
svc_port, namespace = _parse_service_port(svc_manifest, result.warnings)
if svc_port == 0:
svc_port = port
result.edges.append(
InfraEdge(
source=node_id,
target=name,
edge_type="k8s-service",
via="service-manifest",
transport="",
extra=namespace,
)
)
# Env-var dependency edges
env_list = _extract_container_env(first_container)
dep_edges = _extract_env_deps(env_list, node_id, known_names_frozen)
result.edges.extend(dep_edges)
# infra-maps cross-layer edge (image name -> source code node)
if image:
image_base = image.split(":")[0].split("/")[-1]
if image_base:
result.edges.append(
InfraEdge(
source=node_id,
target=image_base,
edge_type="infra-maps",
via="image-name",
transport="",
)
)
return result
# ---------------------------------------------------------------------------
# Helm parser
# ---------------------------------------------------------------------------
def _parse_chart_yaml(chart_path: Path, warnings: list[str]) -> Optional[dict]:
"""Load and return Chart.yaml contents."""
return _safe_load_yaml(chart_path, warnings)
def _parse_values_yaml(values_path: Path, warnings: list[str]) -> dict:
"""Load values.yaml; return empty dict on failure."""
if not values_path.is_file():
return {}
data = _safe_load_yaml(values_path, warnings)
return data if isinstance(data, dict) else {}
def _extract_helm_image(values: dict) -> str:
"""Extract image string from values.yaml using common conventions."""
image_section = values.get("image", {}) or {}
if isinstance(image_section, dict):
repo = image_section.get("repository", "")
tag = image_section.get("tag", "")
if repo:
return f"{repo}:{tag}" if tag else repo
return ""
def _extract_helm_port(values: dict) -> int:
"""Extract primary service port from values.yaml."""
svc = values.get("service", {}) or {}
if isinstance(svc, dict):
return int(svc.get("port", 0))
return 0
def _extract_helm_env_deps(
values: dict,
source_id: str,
known_chart_names: frozenset[str],
) -> list[InfraEdge]:
"""Infer service dependencies from env entries in values.yaml."""
env_section = values.get("env", {}) or {}
if not isinstance(env_section, dict):
return []
env_list = [{"name": k, "value": str(v)} for k, v in env_section.items()]
return _extract_env_deps(env_list, source_id, known_chart_names)
def parse_helm(root: Path) -> TopologyResult:
"""Scan root for Chart.yaml files and extract Helm chart topology."""
result = TopologyResult()
chart_files: list[Path] = []
for d in _walk_dirs(root):
p = d / HELM_CHART_FILENAME
if p.is_file():
chart_files.append(p)
if not chart_files:
return result
# Collect chart names for cross-reference resolution
chart_names: dict[str, Path] = {}
chart_data_map: dict[Path, dict] = {}
for chart_path in chart_files:
data = _parse_chart_yaml(chart_path, result.warnings)
if data is None:
continue
chart_name: str = data.get("name", "")
if chart_name:
chart_names[chart_name] = chart_path
chart_data_map[chart_path] = data
known_chart_names = frozenset(chart_names.keys())
for chart_path, data in chart_data_map.items():
chart_dir = chart_path.parent
chart_name = data.get("name", "")
if not chart_name:
continue
values = _parse_values_yaml(chart_dir / "values.yaml", result.warnings)
image = _extract_helm_image(values)
port = _extract_helm_port(values)
node_id = _infra_id(chart_name)
result.nodes.append(
InfraNode(
id=node_id,
kind="helm-chart",
image=image,
port=port,
component="",
source_file=str(chart_path.relative_to(root) if chart_path.is_relative_to(root) else chart_path),
)
)
# Chart dependency edges
dependencies: list[dict] = data.get("dependencies", []) or []
for dep in dependencies:
dep_name: str = dep.get("name", "")
if not dep_name:
continue
target_id = _infra_id(dep_name)
result.edges.append(
InfraEdge(
source=node_id,
target=target_id,
edge_type="helm-depends",
via="chart-dependency",
transport="",
)
)
# Env-var dependency edges from values
env_edges = _extract_helm_env_deps(values, node_id, known_chart_names)
result.edges.extend(env_edges)
# infra-maps cross-layer edge
if image:
image_base = image.split(":")[0].split("/")[-1]
if image_base:
result.edges.append(
InfraEdge(
source=node_id,
target=image_base,
edge_type="infra-maps",
via="image-name",
transport="",
)
)
return result
# ---------------------------------------------------------------------------
# Terraform parser (regex-based, no hcl2 dependency)
# ---------------------------------------------------------------------------
def _read_tf_files(tf_dir: Path) -> str:
"""Concatenate all .tf files in a directory into one string."""
parts: list[str] = []
for p in sorted(tf_dir.glob("*.tf")):
try:
parts.append(p.read_text(encoding="utf-8", errors="replace"))
except OSError:
pass
return "\n".join(parts)
def _parse_tf_modules(content: str) -> dict[str, str]:
"""Return {module_name: source_path} from HCL content."""
modules: dict[str, str] = {}
for m in _TF_MODULE_BLOCK_RE.finditer(content):
mod_name = m.group(1)
body = m.group(2)
src_match = _TF_MODULE_SOURCE_RE.search(body)
source = src_match.group(1) if src_match else ""
modules[mod_name] = source
return modules
def _parse_tf_resources(content: str) -> list[tuple[str, str]]:
"""Return [(resource_type, resource_name)] from HCL content."""
return [
(m.group(1), m.group(2))
for m in _TF_RESOURCE_BLOCK_RE.finditer(content)
]
def _find_cross_module_refs(
content: str,
all_module_names: frozenset[str],
) -> set[str]:
"""Return set of module names referenced via module.<name>. syntax."""
refs: set[str] = set()
for m in _TF_MODULE_REF_RE.finditer(content):
ref_name = m.group(1)
if ref_name in all_module_names:
refs.add(ref_name)
return refs
def parse_terraform(root: Path) -> TopologyResult:
"""Scan root for Terraform directories and extract module topology."""
result = TopologyResult()
# Find all directories containing main.tf
tf_dirs: list[Path] = []
for d in _walk_dirs(root):
if (d / TERRAFORM_MAIN_FILENAME).is_file():
tf_dirs.append(d)
if not tf_dirs:
return result
result.warnings.append(
"; warning: terraform parsed with regex fallback (no hcl2 dependency)"
)
# Each tf_dir is an independent Terraform root; we treat module blocks as nodes
# and cross-module references as edges within the same root.
for tf_dir in tf_dirs:
content = _read_tf_files(tf_dir)
if not content.strip():
continue
modules = _parse_tf_modules(content)
resources = _parse_tf_resources(content)
all_module_names = frozenset(modules.keys())
rel_dir = str(tf_dir.relative_to(root) if tf_dir.is_relative_to(root) else tf_dir)
# If there are resource blocks, emit per-resource nodes
for res_type, res_name in resources:
node_name = f"tf-{res_name}"
node_id = _infra_id(node_name)
result.nodes.append(
InfraNode(
id=node_id,
kind="terraform",
image="",
port=0,
component=res_type,
source_file=rel_dir,
)
)
# Emit a node for each module block
for mod_name in modules:
node_name = f"tf-{mod_name}"
node_id = _infra_id(node_name)
result.nodes.append(
InfraNode(
id=node_id,
kind="terraform",
image="",
port=0,
component="module",
source_file=rel_dir,
)
)
# Find cross-module dependencies: for each module block, scan its body
# for references to other modules
for m in _TF_MODULE_BLOCK_RE.finditer(content):
src_mod_name = m.group(1)
body = m.group(2)
refs = _find_cross_module_refs(body, all_module_names - {src_mod_name})
src_id = _infra_id(f"tf-{src_mod_name}")
for ref in sorted(refs):
tgt_id = _infra_id(f"tf-{ref}")
result.edges.append(
InfraEdge(
source=src_id,
target=tgt_id,
edge_type="tf-depends",
via="module-reference",
transport="",
)
)
return result
# ---------------------------------------------------------------------------
# Sexp writer
# ---------------------------------------------------------------------------
def _quote(s: str) -> str:
"""Wrap s in double quotes, escaping inner quotes and backslashes."""
escaped = s.replace("\\", "\\\\").replace('"', '\\"')
return f'"{escaped}"'
def _needs_quoting(s: str) -> bool:
"""Return True if s contains whitespace or sexp-special characters."""
return bool(re.search(r'[\s"()\\]', s))
def _atom(s: str) -> str:
"""Return s as a sexp atom, quoting if necessary."""
if not s:
return '""'
return _quote(s) if _needs_quoting(s) else s
def render_sexp(nodes: list[InfraNode], edges: list[InfraEdge]) -> list[str]:
"""Return a list of sexp lines representing the infra topology."""
lines: list[str] = ["", "; --- infrastructure topology ---"]
# Deduplicate nodes by id, keeping first occurrence
seen_nodes: set[str] = set()
unique_nodes: list[InfraNode] = []
for node in nodes:
if node.id not in seen_nodes:
seen_nodes.add(node.id)
unique_nodes.append(node)
for node in sorted(unique_nodes, key=lambda n: n.id):
parts: list[str] = [f"(infra-node {node.id}"]
parts.append(f":kind {node.kind}")
if node.image:
parts.append(f":image {_atom(node.image)}")
if node.port:
parts.append(f":port {node.port}")
if node.component:
parts.append(f":component {_atom(node.component)}")
lines.append(" ".join(parts) + ")")
if unique_nodes:
lines.append("")
# Deduplicate edges by (source, target, edge_type, via)
seen_edges: set[tuple[str, str, str, str]] = set()
unique_edges: list[InfraEdge] = []
for edge in edges:
key = (edge.source, edge.target, edge.edge_type, edge.via)
if key not in seen_edges:
seen_edges.add(key)
unique_edges.append(edge)
for edge in sorted(unique_edges, key=lambda e: (e.edge_type, e.source, e.target)):
if edge.edge_type == "k8s-depends":
transport_str = f" :transport {edge.transport}" if edge.transport else ""
lines.append(
f"({edge.edge_type} {edge.source} {edge.target}"
f" :via {edge.via}{transport_str})"
)
elif edge.edge_type == "k8s-service":
ns_str = f" :namespace {_atom(edge.extra)}" if edge.extra else ""
# target holds the dns service name for k8s-service facts
lines.append(
f"(k8s-service {edge.source} {_atom(edge.target)}{ns_str})"
)
elif edge.edge_type == "helm-depends":
lines.append(
f"(helm-depends {edge.source} {edge.target} :via {edge.via})"
)
elif edge.edge_type == "tf-depends":
lines.append(
f"(tf-depends {edge.source} {edge.target} :via {edge.via})"
)
elif edge.edge_type == "infra-maps":
lines.append(
f"(infra-maps {edge.source} {_atom(edge.target)} :via {edge.via})"
)
return lines
# ---------------------------------------------------------------------------
# Merge results
# ---------------------------------------------------------------------------
def merge_results(results: list[TopologyResult]) -> TopologyResult:
"""Combine multiple TopologyResult instances into one."""
merged = TopologyResult()
for r in results:
merged.nodes.extend(r.nodes)
merged.edges.extend(r.edges)
merged.warnings.extend(r.warnings)
return merged
# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------
def _parse_args(argv: list[str]) -> tuple[Path, bool, bool, bool, Optional[Path], bool]:
"""Parse CLI arguments.
Returns (root, do_kustomize, do_helm, do_terraform, append_file, stdout_mode).
"""
root = Path(".")
do_kustomize: Optional[bool] = None
do_helm: Optional[bool] = None
do_terraform: Optional[bool] = None
append_file: Optional[Path] = None
stdout_mode = False
i = 0
while i < len(argv):
arg = argv[i]
if arg == "--kustomize":
do_kustomize = True
elif arg == "--helm":
do_helm = True
elif arg == "--terraform":
do_terraform = True
elif arg == "--stdout":
stdout_mode = True
elif arg == "--append":
i += 1
if i < len(argv):
append_file = Path(argv[i])
elif not arg.startswith("--"):
root = Path(arg)
i += 1
# Auto-detect mode: run all parsers unless explicitly requested
if do_kustomize is None and do_helm is None and do_terraform is None:
do_kustomize = True
do_helm = True
do_terraform = True
return (
root,
bool(do_kustomize),
bool(do_helm),
bool(do_terraform),
append_file,
stdout_mode or (append_file is None),
)
def main() -> None:
root, do_kustomize, do_helm, do_terraform, append_file, stdout_mode = _parse_args(
sys.argv[1:]
)
if not root.is_dir():
print(f"Error: root directory not found: {root}", file=sys.stderr)
sys.exit(1)
results: list[TopologyResult] = []
if do_kustomize:
results.append(parse_kustomize(root))
if do_helm:
results.append(parse_helm(root))
if do_terraform:
results.append(parse_terraform(root))
merged = merge_results(results)
# Emit warnings to stderr
for w in merged.warnings:
print(w, file=sys.stderr)
if not merged.nodes and not merged.edges:
print("; info: no infrastructure files found", file=sys.stderr)
if not stdout_mode and append_file is None:
return
# Still write the comment so callers know the script ran
output_lines = ["", "; --- infrastructure topology ---", "; info: no infrastructure files found"]
else:
output_lines = render_sexp(merged.nodes, merged.edges)
output_text = "\n".join(output_lines) + "\n"
if stdout_mode:
sys.stdout.write(output_text)
if append_file is not None:
try:
with append_file.open("a", encoding="utf-8") as fh:
fh.write(output_text)
except OSError as exc:
print(f"Error: could not append to {append_file}: {exc}", file=sys.stderr)
sys.exit(1)
node_count = len({n.id for n in merged.nodes})
edge_count = len(merged.edges)
print(
f"[infra-topology] Done: {node_count} nodes, {edge_count} edges",
file=sys.stderr,
)
if __name__ == "__main__":
main()

124
scripts/git_descendants.py Executable file
View File

@@ -0,0 +1,124 @@
#!/usr/bin/env python3
"""
Git Descendants Finder -- find all commits that descend from HEAD.
Builds a parent-child graph from git rev-list, then BFS-walks forward
from HEAD to find all descendant commits. Useful for understanding
what work was built on top of a given commit.
Usage::
git_descendants.py # descendants of HEAD
git_descendants.py <ref> # descendants of a specific commit/ref
"""
from __future__ import annotations
import subprocess
import sys
def run_git(cmd: str, cwd: str | None = None) -> list[str]:
"""Run a git command and return output lines."""
result = subprocess.run(cmd, shell=True, capture_output=True, text=True, cwd=cwd)
if result.returncode != 0:
print(f"Error running: {cmd}", file=sys.stderr)
print(result.stderr, file=sys.stderr)
sys.exit(1)
return result.stdout.strip().split("\n") if result.stdout.strip() else []
def get_commit_sha(ref: str = "HEAD", cwd: str | None = None) -> str:
"""Resolve a ref to a full SHA."""
result = subprocess.run(
["git", "rev-parse", ref], capture_output=True, text=True, cwd=cwd,
)
if result.returncode != 0:
print(f"Error: cannot resolve ref '{ref}'", file=sys.stderr)
sys.exit(1)
return result.stdout.strip()
def build_parent_child_map(
cwd: str | None = None,
) -> tuple[dict[str, str], dict[str, list[str]]]:
"""Build a map of commit -> leftmost parent and parent -> children."""
print("Building commit graph...", file=sys.stderr)
commits = run_git("git rev-list --all --reflog --parents", cwd=cwd)
parent_map: dict[str, str] = {}
children_map: dict[str, list[str]] = {}
for line in commits:
parts = line.split()
if not parts:
continue
commit = parts[0]
parents = parts[1:]
if parents:
leftmost_parent = parents[0]
parent_map[commit] = leftmost_parent
if leftmost_parent not in children_map:
children_map[leftmost_parent] = []
children_map[leftmost_parent].append(commit)
print(f"Found {len(parent_map)} commits with parents", file=sys.stderr)
return parent_map, children_map
def find_all_descendants(
commit: str, children_map: dict[str, list[str]],
) -> set[str]:
"""Find all descendants of a commit using BFS."""
descendants: set[str] = set()
queue = [commit]
while queue:
current = queue.pop(0)
if current in children_map:
for child in children_map[current]:
if child not in descendants:
descendants.add(child)
queue.append(child)
return descendants
def main() -> None:
ref = sys.argv[1] if len(sys.argv) > 1 else "HEAD"
full_sha = get_commit_sha(ref)
short_sha = subprocess.run(
["git", "rev-parse", "--short", full_sha],
capture_output=True, text=True,
).stdout.strip()
print(f"Finding descendants of {ref}: {short_sha}", file=sys.stderr)
print(f"Full SHA: {full_sha}", file=sys.stderr)
print(file=sys.stderr)
_parent_map, children_map = build_parent_child_map()
descendants = find_all_descendants(full_sha, children_map)
print(f"Found {len(descendants)} descendant commits", file=sys.stderr)
print(file=sys.stderr)
if descendants:
for commit in sorted(descendants):
commit_info = subprocess.run(
["git", "log", "-1", "--format=%h %s", commit],
capture_output=True, text=True,
).stdout.strip()
print(commit_info)
else:
print("No descendants found", file=sys.stderr)
if __name__ == "__main__":
main()

152
tests/test_belief_search.py Normal file
View File

@@ -0,0 +1,152 @@
import importlib.util
import io
import sys
import tempfile
import textwrap
import unittest
from contextlib import redirect_stdout
from pathlib import Path
SCRIPT_PATH = Path("/Users/alexeus/.codex/skills/codespaces/scripts/belief_search.py")
SPEC = importlib.util.spec_from_file_location("belief_search", SCRIPT_PATH)
belief_search = importlib.util.module_from_spec(SPEC)
assert SPEC.loader is not None
sys.modules[SPEC.name] = belief_search
SPEC.loader.exec_module(belief_search)
class BeliefSearchAliasTest(unittest.TestCase):
def _load_graph(self) -> tuple[object, str]:
tmpdir = tempfile.TemporaryDirectory()
self.addCleanup(tmpdir.cleanup)
sexp_path = Path(tmpdir.name) / ".belief_map.sexp"
sexp_path.write_text(
textwrap.dedent(
"""
(paths
(app-service
(app-service
(src
(workspace-operatives
(asp543 workspace-operative.service)
(asp544 workspace-operative.service.spec)
)
)
)
)
)
(node asp543 ts service)
(node asp544 ts test)
(cls asp543 WorkspaceOperativeService 16)
(imports asp544 asp543)
"""
).strip()
+ "\n",
encoding="utf-8",
)
graph = belief_search.BeliefGraph()
graph.load(str(sexp_path))
return graph, str(sexp_path)
def test_resolve_module_accepts_path_aliases(self) -> None:
"""/* REQ-CS-001: analyze must accept internal path-map aliases returned by older belief maps. */"""
graph, _sexp_path = self._load_graph()
self.assertEqual(
graph.resolve_module("asp543"),
["app-service/app-service/src/workspace-operatives/workspace-operative.service"],
)
output = io.StringIO()
with redirect_stdout(output):
belief_search.cmd_analyze(graph, "asp543")
self.assertIn(
"(analyze app-service/app-service/src/workspace-operatives/workspace-operative.service",
output.getvalue(),
)
def test_search_returns_resolved_module_ids(self) -> None:
"""/* REQ-CS-002: search must emit resolved module IDs instead of raw path-map aliases. */"""
graph, sexp_path = self._load_graph()
output = io.StringIO()
with redirect_stdout(output):
belief_search.cmd_search(graph, "workspace-operative.service", sexp_path)
lines = output.getvalue().strip().splitlines()
self.assertEqual(
lines,
[
"(result app-service/app-service/src/workspace-operatives/workspace-operative.service)",
"(result app-service/app-service/src/workspace-operatives/workspace-operative.service.spec)",
"(result-count 2)",
],
)
def test_quick_runs_analyze_for_first_resolved_module(self) -> None:
"""/* REQ-CS-003: quick must provide full analyze output for the first matching module. */"""
graph, sexp_path = self._load_graph()
output = io.StringIO()
with redirect_stdout(output):
belief_search.cmd_quick(graph, "workspace-operative.service", sexp_path)
text = output.getvalue()
self.assertIn(
"(analyze app-service/app-service/src/workspace-operatives/workspace-operative.service",
text,
)
self.assertIn(
"(boundary-file app-service/app-service/src/workspace-operatives/workspace-operative.service",
text,
)
self.assertNotIn("(boundary app-service/app-service/src/workspace-operatives/workspace-operative.service", text)
def test_quick_resolves_alias_from_raw_search_hits(self) -> None:
"""/* REQ-CS-004: quick must resolve legacy path-map aliases found in raw belief-map facts. */"""
graph, sexp_path = self._load_graph()
output = io.StringIO()
with redirect_stdout(output):
belief_search.cmd_quick(graph, "WorkspaceOperativeService", sexp_path)
self.assertIn(
"(analyze app-service/app-service/src/workspace-operatives/workspace-operative.service",
output.getvalue(),
)
def test_boundary_files_only_emits_minimal_file_list(self) -> None:
"""/* REQ-CS-005: boundary --files-only must emit the target and directly related files only. */"""
graph, _sexp_path = self._load_graph()
output = io.StringIO()
with redirect_stdout(output):
belief_search.cmd_boundary(graph, "workspace-operative.service", files_only=True)
self.assertEqual(
output.getvalue().strip().splitlines(),
[
"app-service/app-service/src/workspace-operatives/workspace-operative.service",
"app-service/app-service/src/workspace-operatives/workspace-operative.service.spec",
],
)
def test_no_match_includes_suggestions(self) -> None:
"""/* REQ-CS-006: no-match errors must surface actionable module suggestions. */"""
graph, _sexp_path = self._load_graph()
output = io.StringIO()
with redirect_stdout(output):
belief_search.cmd_analyze(graph, "workspace-operative.controller")
text = output.getvalue()
self.assertIn("(error no-match workspace-operative.controller :suggestions", text)
self.assertIn("workspace-operative.service", text)
if __name__ == "__main__":
unittest.main()