# Build Pipeline Elimination

**Date**: 2026-02-11
**Project ID**: build-pipeline-elimination
**Status**: COMPLETED
**Steps**: 15 TDD steps across 4 phases, all COMMIT/PASS

## Executive Summary

Eliminated the entire build pipeline from the nWave framework. The installer now reads
directly from source directories instead of building to an intermediate dist/ide/ output.
Net result: 68,158 lines removed across 133 files, with zero new production Python code.
The feature was entirely about removal and simplification.

## Problem Statement

The nWave framework had a build pipeline (tools/build.py, tools/core/build_ide_bundle.py,
6 processors, 3 utility modules) that transformed source files into dist/ide/ output before
installation. This pipeline existed to:

1. Inject YAML frontmatter into agent files (BUILD:INJECT markers)
2. Substitute template variables in command files
3. Inline embed data into agent content

By early 2026, all 22 agent source files already had native frontmatter, commands had no
template variables, and embed content had been superseded by skills. The build pipeline's
~2,700 lines of processing code produced output identical to its input. It was pure overhead:
added CI time, created a dist/ide dependency that complicated testing, and obscured the
source-of-truth relationship between nWave/ files and installed output.

## Solution Approach

Four-phase incremental removal with a safety-net-first strategy.

### Phase 1: Source-as-Truth (steps 01-01 to 01-03)

Made source files directly installable without breaking the existing build path. Added
YAML frontmatter to all 18 command source files and created a CI frontmatter lint
validator. After this phase, both installation paths (source-direct and build-then-install)
produced identical results.

### Phase 2: Switch Installer (steps 02-01 to 02-05)

Rewired installer plugins to read from nWave/ source directories following the existing
skills_plugin.py pattern. Removed build_framework(), check_source(), and dist/ide
references from install_nwave.py. Updated 27 test files that referenced dist/ide fixtures.
This was the point of no return -- the build pipeline became dead code.

### Phase 3: Delete Build Pipeline (steps 03-01 to 03-04)

Deleted 10+ build pipeline files (processors, utilities, build entry points). Updated
CI/CD workflow to replace build steps with frontmatter validation. Updated the release
packager to read from source. Removed build-related test files.

### Phase 4: Cleanup (steps 04-01 to 04-03)

Deleted 31 legacy agent files and 22 legacy command files containing BUILD:INJECT markers
and template variables. Migrated 27 embed data files to skills. Updated
framework-catalog.yaml. Cleaned up dist/ references and removed --force-rebuild flag.

## Key Decisions and Trade-offs

| Decision | Choice | Rejected Alternative | Rationale |
|----------|--------|---------------------|-----------|
| Incremental vs atomic | 4-phase incremental | Single atomic commit | 133 files across installer, CI/CD, tests. Too many simultaneous changes for safe rollback. |
| Safety net first | Phase 1 makes source installable before switching | Switch immediately | Both paths work after Phase 1. If anything breaks, revert one phase, not everything. |
| Flat-to-namespace copy | mkdir + copy individual files | copytree | Source (nWave/tasks/nw/) is flat. Target needs nw/ subdirectory. copytree would create wrong nesting. |
| Embed file handling | Audit, migrate to skills, then delete | Blind delete | Embed files were dual-use: build pipeline consumed them AND agents referenced them as knowledge base. Required migration path. |
| Legacy file retention | Delete entirely | Keep as archive | Legacy files contained BUILD:INJECT markers that only the deleted pipeline understood. No value without the pipeline. |
| config.json elimination | Implicit (no code reads it) | Explicit migration | Grep confirmed zero consumers. Generated by build at dist/ide/agents/nw/config.json, never read by installer. |

## Metrics

| Metric | Value |
|--------|-------|
| Lines deleted | 68,499 |
| Lines added | 341 |
| Net reduction | 68,158 lines |
| Files changed | 133 |
| TDD steps | 15 (all COMMIT/PASS) |
| New production Python code | 0 |
| RED_UNIT skipped | 8/15 (removal steps with no new logic) |
| Deliver Phase 3 (Refactoring) | 1/22 files needed work (dead test removal) |
| Deliver Phase 4 (Adversarial Review) | REJECTED then PASSED (3 doc defects fixed) |
| Deliver Phase 5 (Mutation Testing) | NOT_APPLICABLE (no new production logic) |
| Deliver Phase 6 (Integrity) | PASSED |

## Lessons Learned

### What Went Well

1. **Safety-net-first sequencing**: Phase 1 ensured source files were installable before
   switching. This meant the build pipeline could be restored at any point during Phase 2
   if problems surfaced. No emergency was needed, but the safety net was real.

2. **Following established patterns**: The skills_plugin.py pattern for direct source
   reading already existed. Reusing it for agents and commands plugins reduced design risk
   to near zero.

3. **Roadmap review caught a critical gap**: The reviewer identified that nWave/data/embed/
   was dual-use (build pipeline AND live agent references). Without the review, blind
   deletion would have broken nw-researcher and nw-data-engineer agents.

4. **Adversarial review caught documentation rot**: Three stale documentation artifacts
   (build pipeline guide, workflows README, historical docs) referenced the deleted
   pipeline. Caught and fixed before merge.

5. **Batch test fixture updates**: 27 test files all used the same dist/ide pattern.
   Systematic substitution in a single step (02-05) was efficient and low-risk.

### What Could Improve

1. **Step 03-04 was a no-op**: Build test files had already been removed in prior steps.
   The roadmap should have verified file existence before planning a deletion step.

2. **Execution log format inconsistency**: Early steps used pipe-delimited strings, later
   steps used structured YAML objects. The v2-to-v3 format migration happened mid-feature.
   Future features should start with v3.0 format.

3. **Legacy file count was underestimated**: Roadmap estimated 30 legacy agents, actual was
   31. Minor, but the quantitative baseline should have been exact.

## Impact on Developer Workflow

**Before**: `python tools/build.py` -> wait for build -> `python scripts/install/install_nwave.py`

**After**: `python scripts/install/install_nwave.py` (reads source directly)

- CI pipeline lost the build step, gaining ~30 seconds per run
- Source files in nWave/ are now the single source of truth -- what you edit is what gets installed
- No more dist/ide/ drift where build output could diverge from source
- Release packager reads from source, eliminating a build-before-release dependency
- Frontmatter validation replaced build validation, catching real issues (missing metadata)
  instead of checking that a build tool ran successfully

---

**Document Status**: COMPLETE
**Feature Status**: DELIVERED
