Field Notes
From the field
Architecture decisions, production systems, and technical leadership — notes from actually shipping AI infrastructure.
Empty Is Not Clean: Five Fail-Open Bugs in an AI Agent
One fail-open bug, five disguises: empty args, an empty log, a missing property, a sheared chain, an unset scope. When there's nothing to check, fail closed.
The Kernel Must Not Import Its Agents
We extracted a stateful watcher agent from a governance kernel and enforced a one-way agent-to-kernel dependency edge — the kernel must not import its agents.
Making a Fire-and-Forget Writer Safe Under Failure
Five properties an unattended capture hook needs before you turn it on: idempotent intake, durable outbox, backpressure, atomic receipts, consent gate.
Adversarial Review: The Six Lenses That Halted a Rollout
A six-lens adversarial review checked a team knowledge system against live state, broke three shipped assumptions, and gated 18 risks to halt the rollout.
Noise-Robust LLM-Judge Evals: Don't Sign a Coin Flip
An un-seeded LLM judge is a coin flip even at temperature 0. How we made signed eval verdicts replayable instead of attesting noise.
Gate the Statement, Not the Tool Name
When one MCP tool carries every SQL verb, allowlisting tool names is theater. The safety boundary has to read the statement — here's how that gate was built.
When LLM Output Lies Instead of Crashing
An LLM-output parsing bug silently understated a cost report via case-sensitivity. One-line normalizer fix plus 5 defensive hardening steps.
MCP Server Auth: The API Is the Real Boundary
Per-user tokens, a server-side write gate, and a separate access log — why an MCP server's client-side tool gate is UX, not a security boundary.
When --cap-drop ALL Broke the Gate Socket
Hardening a container hid a permission bug: --cap-drop ALL stripped CAP_DAC_OVERRIDE, and a gate socket silently stopped governing every tool call.
Green CI Proves Nothing: Why Your Tests Gate Zero Calls
CI dogfood for AI-agent governance went green while gating zero tool calls. Here's why a passing test proving nothing is worse than a red one.
The Wrong Product, Built Perfectly
A new site was scaffolded, deployed, and live with valid TLS in under an hour — then declared the wrong product. The decoupling made the reversal cheap.
Safety Model First: 16-Tool Ops MCP, One Day
Design a 7-point safety model before writing tools. How server-ops-mcp shipped 16 tools, 40 tests, and v0.1.0 in a single day.
Honest Perf Benchmarks for a Paid-API Compiler
Four PRs, three releases, and a benchmark suite that won't lie to you: seeded-RNG corpora, double-gated Claude scenarios, and skipped-but-recorded records.
Forty-Four Minutes Before First Pitch: An LLM Fallback Chain and a Live Probability Gauge in One Session
Groq quota died 44 minutes before a live broadcast. The fix was a 4-tier LLM fallback chain ending in a facts-only renderer. Then we shipped JChad's Challenge in the same session.
The Rubric Sits On Top Of The Spec: A Schema Validator Postmortem
Postmortem on a schema validator debacle: when a stricter enterprise rubric is built on top of a permissive open spec, the rubric must be additive, never the spec's floor.
Ending a Four-Month Silent Fail: Cross-Repo Triangulation on a Broken Gemini PR Workflow
When a Gemini PR review workflow silently fails for four months, your first-principles fix is the suspect. Triangulate against a working reference.
Enforcement Travels With the Code: Shipping @intentsolutions/audit-harness v0.1.0
Testing policy enforcement that lives in ~/.claude/ cannot travel with a fresh clone. audit-harness v0.1.0 moves the gates into each repo as a versioned dev dependency — npm, PyPI, crates, and bash install.sh.
Manifest System + Mutation Testing: Two Ways to Find Out What Actually Works
Epic 31-B lands the publish side of the bot-manifest protocol in claude-code-slack-channel. Epic 31 closes with an audit. Mutation tests and a cyclomatic gate make the test pyramid stop lying.
Four Releases in One Day: How the claude-code-slack-channel Security Sprint Actually Shipped
Epic 29-A, 30-A, 30-B, 32-B land in a single calendar day across v0.5.0 → v0.5.1 → v0.6.0 → v0.7.0 — a supervisor, a hash-chained audit journal, and a policy engine that never sees manifests.
LLM-as-Reducer and the Case for Killing the AI Label
Two AI product lessons from the Braves dashboard post-game expansion: use the LLM as a reducer over noisy community signal, and pull the AI label off the UI.
The 35x FLOPs Error That Peer Review Predicted
A 35x FLOPs correction in pre-filing patent artifacts validated the reviewers' unchecked-derivation warning. The day's other shipments show what systematizing against named failure classes looks like.
Repo-Resolver: Typed Errors and Monorepo Detection
A shared repo-resolver package shipped into claude-runtime — ADR to integration in seven PRs, with typed error classes and transparent fallback that avoided a flag day.
Knowledge OS: Render, Promote, and the Research Command
intentional-cognition-os ships Epics 8 and 9 — a Marp slide deck renderer, a 7-rule promotion engine with rollback, and the first research command. Three releases in one day: v0.3.0 through v0.4.0.
Resizable Columns and WCAG Contrast Fixes on the Braves Dashboard
Adding drag-to-resize panels and fixing WCAG AA contrast ratios on the Braves Booth dashboard — the kind of polish that separates prototypes from broadcast tools.
Knowledge OS Bootstrap: Seven Epics, Marketplace Security, and Production AI Fixes
Bootstrapping a local-first knowledge OS from standards documents to working retrieval in one day. Seven epics, six releases, 485 tests. Plus marketplace docs, Firestore security, and Braves AI fixes.
Legal Toolkit, Epic Planning, and a Canary That Stopped Singing
A 12-skill legal toolkit with 5 parallel agents, a 117-bead execution plan reviewed by 6 auditors, and a canary CI pipeline that silently failed for 3 days. Trust but verify across three projects.
Braves Booth v1.0.0: Player Drill-Down, Lineup Cache Bug, and Shipping 1.0
Shipping v1.0.0 of Braves Booth Intelligence — a real-time broadcast ops tool for Atlanta Braves radio. Player drill-down, a lineup cache bug found in live testing, and 1,684 lines of new code.
Braves Booth — Idle Recap, Dashboard Density, and AI Pitcher Narratives
Refactoring the Braves broadcast dashboard idle view, eliminating sub-10px fonts, and replacing duplicated AI narrative logic with a structured buildPitcherFacts function.
IntentCAD Viewer — Closing the DWG FastView Gap
Web workers, animated zoom, progress feedback, and color-coded selections. The changes that took IntentCAD's browser viewer from prototype to production-grade — benchmarked against DWG FastView.
Write Once, Publish Everywhere: Building Content Distribution Across Three Sites
Three sites now share content through automated pipelines: Hugo to Astro backfill, cross-repo plugin sync with repository_dispatch, and a CodeRabbit-caught no-op filter.
Usable, Not Just Functional: Entity Selection, Binary Eval, and 6 UX Fixes Across 4 Repos
40+ commits across 4 repos. CAD entity selection gets real bounding boxes, a binary eval framework bootstraps to v0.2.8, and the X triage plugin ships 6 UX features.
Knowledge Base from Zero to API: Lifecycle State Machines and Policy Engines
Building a team knowledge base with lifecycle state machines, policy engines, and git-exported articles — from empty repo to working API in one day.
Live Data Backbone: Cache Layers, Circuit Breakers, and SSE for Broadcast
Wiring MLB Stats API, GUMBO feeds, and Statcast into a fault-tolerant data layer with TTL caching, circuit breakers, and Server-Sent Events.
Zero to CI: Full-Stack Dashboard in One Session
11 commits from empty repo to green CI. Building the Braves Booth Intelligence dashboard with opinionated stack choices and a CAD agent side quest.
Engine to Product: Three Interfaces, One Codebase
Building CLI, REST API, and frontend wizard interfaces on top of a comparison engine. The jump from library to usable product in cad-dxf-agent.
February 2026 State of Affairs: 255 Commits, 12 Projects, and What I Shipped
A candid look at 18 days of engineering output: 255+ commits across 12 projects, from blockchain infrastructure to CAD agents to auth debugging.
IRSB Monorepo v1.0.0: Extracting Shared Packages and Unifying a Blockchain Platform
How I extracted @irsb/kms-signer and @irsb/types into shared packages, unified a blockchain platform under one monorepo, dealt with port conflicts, and split CI tests for 3x faster runs.
Designing a Local-First Resume Parser: Architecture Decisions for Edge AI
Documenting the architecture decisions for Resume Realtime - a planned Chrome extension for local resume parsing. This post covers the technical choices made before writing production code.
How to Get Your ADK Agent into Google's Official Community Showcase
A tactical breakdown of contributing to Google's agent-starter-pack repository and getting recognized as a production-grade ADK reference implementation.
Building an Idempotent Stripe Billing Enforcement Engine for Firestore
How we built a unified plan enforcement engine that handles duplicate webhooks, out-of-order events, and automatic drift correction for Stripe billing in a Firestore-backed SaaS application.
Fine-Tuning IAM1: Building a Hierarchical Multi-Agent System on Vertex AI
Deep dive into fine-tuning a Regional Manager AI agent (IAM1) with hierarchical multi-agent orchestration on Vertex AI Agent Engine. Real implementation of decision frameworks and specialist delegation.
Building a Production Multi-Agent AI System: BrightStream's 10-Agent Architecture on Vertex AI
The real journey from concept to production: building BrightStream's 10-agent positive news platform on Google Vertex AI Agent Engine. Multi-agent debate, parallel execution, and 89% cost reduction through architectural optimization.
Architecting a Production Multi-Agent AI Platform: Technical Leadership in Action
Architecting a Production Multi-Agent AI Platform: Technical Leadership in Action
The Coasean Singularity: How AI Agents Will Transform Digital Markets
Analysis of groundbreaking NBER research on how autonomous AI agents will reshape digital markets by reducing transaction costs to near-zero, enabling new market designs and regulatory challenges.
October 2025 State of Affairs: Five Production Platforms and What They Taught Me
A candid look at Intent Solutions' current projects: DiagnosticPro, Hustle, CostPlusDB, ClaudeCodePlugins, and what running five production platforms simultaneously teaches you about deployment velocity.
Intent Solutions Portfolio 2025: Five Production Platforms and the Architecture Behind 4-Day Deployments
Deep dive into Intent Solutions' five production platforms: DiagnosticPro, Hustle, CostPlusDB, ClaudeCodePlugins, and the technical architecture enabling concept-to-production deployment in days, not months.
Scaling AI Batch Processing: Enhancing 235 Plugins with Vertex AI Gemini on the Free Tier
Building an overnight batch processing system to enhance 235 Claude Code plugins using Vertex AI Gemini 2.0 Flash - complete with rate limiting, SQLite audit trails, and Turso disaster recovery. The full technical journey from conservative 90s delays to optimized 45s processing.
Scaling AI Systems: Production Batch Processing with Built-In Disaster Recovery
Scaling AI Systems: Production Batch Processing with Built-In Disaster Recovery
Building Production-Grade CI/CD: From Documentation Chaos to Automated Deployment
Building Production-Grade CI/CD: From Documentation Chaos to Automated Deployment
Deploying Next.js 15 to Google Cloud Run: From Zero to HTTPS in 2 Hours
Complete guide to deploying Next.js 15 with Docker to Google Cloud Run, configuring custom domains, SSL certificates, and global load balancing. Real troubleshooting included.
Making a Youth Sports App COPPA-Compliant: The Real Process From Question to Production
A complete walkthrough of implementing COPPA-compliant legal framework for a youth athlete tracking app, including the questions asked, solutions tried, failures encountered, and the final production release.
Building Production-Grade Testing Infrastructure: A Playwright + GitHub Actions Case Study
How I implemented enterprise-grade testing infrastructure with Playwright, GitHub Actions CI/CD, and automated quality gates for a production survey system - complete with debugging journey and lessons learned.
Hybrid AI Stack: Reduce AI API Costs by 60-80% with Intelligent Request Routing
Production-ready AI orchestration system that intelligently routes requests between local CPU-based models and cloud APIs to slash costs by 60-80%. Complete architecture, deployment guide, and ROI analysis.
Terraform for AI Infrastructure: Complete Learning Guide from Zero to Production
Comprehensive Terraform learning resource covering core concepts, best practices, state management, modules, and real-world examples for AI and cloud infrastructure deployment.
Self-Hosting n8n on Contabo VPS: Enterprise Automation with Zero Recurring Costs
Production-ready n8n deployment on a Contabo VPS — complete architecture, SSL via Caddy, Docker stack, and operational runbook for eliminating recurring SaaS fees.
Leading Complex System Onboarding: From Documentation to Infrastructure Access
How I approached onboarding a Senior Cybersecurity Engineer to a complex multi-project platform, building reusable frameworks and solving real infrastructure challenges.
Enterprise Documentation Transformation: Git-Native TaskWarrior Workflows for Platform Reorganization
How to systematically transform production platforms using git-native documentation workflows, TaskWarrior CLI integration, and enterprise-grade organizational patterns. A real case study from DiagnosticPro platform reorganization.
Enterprise Software Transformation: From Framework to Production Server - A Project Management Case Study
Enterprise Software Transformation: From Framework to Production Server - A Project Management Case Study
Transforming Complex N8N Workflows: From Analysis to Enterprise Tech Intelligence Platform
Transforming Complex N8N Workflows: From Analysis to Enterprise Tech Intelligence Platform
Building Multi-Platform Developer Tools: Scaling an Open-Source Project from 1 to 5 Platforms
Building Multi-Platform Developer Tools: Scaling an Open-Source Project from 1 to 5 Platforms
Building a 254-Table BigQuery Schema in 72 Hours
How we architected and deployed a massive 254-table BigQuery schema in just 72 hours, handling data from multiple sources at scale with 10,000 records/second throughput
Building DiagnosticPro: AI-Powered Vehicle Diagnostics Platform
How we built a professional-grade AI vehicle diagnostics platform in days, not months - featuring GPT-4, 254+ BigQuery tables, and multi-source data pipelines