Implementation Plan

Based on Design v1.12 →

This document provides the step-by-step implementation instructions for building Praesidium2. It is designed to be consumed by Claude Code agents who will perform approximately 90% of the implementation work. Each section contains specific enough instructions that an agent can execute without ambiguity. The Praesidium2 build team is James Maes (CTO) + 2 engineers + unlimited Claude Code agents. The full Engineering Leadership Team (listed below) provides architectural oversight, code review, and the remaining 10% of specialized integration work.

7
Phases
3+AI
Build Team
10
Epics
5
Services

Team Assignments

The Praesidium2 build team is James Maes (CTO) + 2 engineers + unlimited Claude Code agents. The full Engineering Leadership Team shown here provides oversight, review, and domain expertise.

US Engineering Management
J

James (CTO)

Architecture + qqq + Infrastructure + Integrations

Primary owner of qqq data layer, infrastructure (Docker/k8s/Helm/ArgoCD), Faber CI/CD, Authentik configuration, and external integrations. Architects the foundation that the team builds upon.

Java / qqq Helm / k8s Faber CI Authentik Integrations
US

US Eng Manager 2

Backend Lead — Python Agent Runtime

Leads the Agent Gateway (FastAPI), Agent Workers (Celery), Anthropic SDK + LangGraph orchestration, and tool execution framework. Reviews and guides India Senior Engineers on backend work.

Python / FastAPI Anthropic SDK LangGraph Celery Memory / RAG (V1.5)
US

US Eng Manager 3

Frontend Lead — React SPA

Leads the React/TypeScript SPA: all 11 pages, component library, WebSocket/SSE integration, Authentik OIDC login, and end-to-end frontend testing. Reviews and guides India Senior Engineers on frontend work.

React 19 TypeScript TanStack shadcn/ui Vitest
India Senior Engineers
IN

India SE 1-2

Backend — Gateway + Workers

Implement Gateway endpoints, Worker task graphs, integration connectors (Slack, Jira, GitHub), and Celery Beat scheduling. Directed by US Eng Manager 2.

Python / FastAPI Celery LangGraph
IN

India SE 3-4

Frontend — React Pages + Components

Build React pages, component library, WebSocket/SSE integration, and frontend testing. Directed by US Eng Manager 3.

React 19 TypeScript TanStack
IN

India SE 5

QA + Testing + DevOps

End-to-end testing (Playwright), CI pipeline maintenance, Docker/k8s configuration, test coverage enforcement, and security scanning.

Playwright Docker / k8s Faber CI
AI

Claude Code Agents (90% of Implementation)

Each developer directs Claude Code agents for their track

Claude Code agents write the vast majority of code, tests, and configuration. Human developers provide context via CLAUDE.md files, review pull requests, and handle decisions that require institutional knowledge or external access (AWS console, Authentik admin UI, etc.).

CLAUDE.md Rules (Per Service)

  • Each service repo gets its own CLAUDE.md with architecture context, coding patterns, and constraints
  • Agents MUST read CLAUDE.md before making any changes
  • Agents MUST NOT modify files outside their service boundary
  • Agents MUST write tests for every new endpoint/function
  • Agents MUST use the type table pattern (no magic strings)
  • All data access MUST go through qqq REST APIs (agent runtime never touches DB directly)

Key Accelerators

  • qqq generates REST APIs from ERD — no hand-writing CRUD endpoints
  • Claude Code scaffolds entire services, writes tests, generates boilerplate
  • 3-person build team + unlimited Claude Code agents working in parallel on independent tracks from Day 2 onward
  • Existing Praesidium v1 and me-health-portal provide reference implementations
  • Design spec provides complete table schemas — no ambiguity on data model

Claude Code Agent Licensing Change (April 4, 2026)

Anthropic blocked third-party harnesses — including OpenClaw (formerly OpenHands) — from drawing on Claude Pro/Max subscription credits. As of April 4, 2026 at 12pm PT, tools like OpenClaw can no longer use a Claude subscription. Running Claude Code agents now requires either a direct Anthropic API key (pay-per-use) or an extra usage bundle from Anthropic. Budget planning for the build phases should account for API token costs rather than assuming subscription coverage.

Realistic Timeline (7 Phases)

3-person build team (James Maes, CTO + 2 engineers) + unlimited Claude Code agents working 24/7. Existing AWS EKS + Faber CI/CD + ArgoCD.

Agent-first development model: Claude Code agents handle the bulk of coding. F.R.I.D.A.Y. handles PR review (humans review the review). Build team focuses on architecture decisions, code review, and integration testing. Phases can be completed faster than calendar weeks when AI agents are productive. Scope cuts applied: Tool Forge, Memory/RAG, and Meeting Orchestration deferred to V1.5.

Phases are not calendar weeks. The team may complete phases faster than estimated — the goal is to move through phases as quickly as quality allows.

Pre-Build
KPI Baseline Survey (2 weeks before build)
Pre-Build
Phase 1
Foundations (qqq DB 117 tables, Authentik, Frontend Shell, Datadog infra)
5 days
Phase 2
Tool Runner (K8s Job sandbox, “Blind” Secret Pipeline)
5 days
Phase 3
Agent Workers (Anthropic SDK + LangGraph, Checkpointing, HITL)
5 days
Phase 4
Gateway (SSE Streaming, Chat UI, Task Orchestration, /api/search)
5 days
Phase 5
Integrations (Slack, Jira, GitHub toolsets, Webhook listeners)
5 days
Phase 6
Business Ops (KPI Dashboards, HITL Approval UI, Schedule Management, Rollback logic)
5 days
Phase 7
Stabilization (Security audit, Load testing, Production Rollout)
5 days
V1.1
Confluence, Zoho, Google Workspace
Post-Launch
V1.5
Tool Forge, Memory/RAG, Meeting Orchestration
Post-Launch
V1 LAUNCH: 7 PHASES V1.1: +2 WEEKS V1.5: +4 WEEKS

Phases Are Not Calendar Weeks

Phases are not calendar weeks — the team may complete phases faster than estimated. The goal is to move through phases as quickly as quality allows. Each phase has explicit exit criteria; the next phase begins when the current phase's criteria are met, not when a calendar week ends.

Pre-Build: KPI Baseline Capture

Before any code is written, capture the ROI measurement baseline. Each team lead logs hours on tasks that agents will eventually handle. This 2-week survey runs while dev environment setup and design finalization happen in parallel.

Activity Description Owner
Time-Tracking Survey 2-week time-tracking survey distributed to Engineering and Marketing team leads. Each participant logs daily hours spent on: status reporting, Jira grooming, Slack triage, meeting scheduling, document creation, and code review coordination. James (CTO) + Team Leads
Baseline Metrics Pull historical data from Jira, GitHub, and Slack: average task completion time, throughput per team member, PR cycle time, meeting hours per week, error/rework rates. Store in kpi_baseline table schema (deployed Phase 1). James (CTO)
Success Criteria Define measurable targets with stakeholders: e.g., 30% reduction in status reporting time, 50% faster Jira triage, 20% fewer context switches. These become the V1 KPI targets. James (CTO) + Stakeholders

Pre-Build Exit Criteria

  • Minimum 2 weeks of time-tracking data from each team lead
  • Historical metrics extracted from Jira, GitHub, and Slack APIs
  • Documented baseline values for all KPIs that Praesidium will impact
  • Stakeholder-agreed success targets for V1 ROI measurement

Phase 1: Foundations

qqq DB provisioning (117 tables), Authentik claim mapping, Frontend App Shell, Datadog infrastructure. Full build team + AI agents establish the foundation in parallel.

Period James (CTO) + India SE 5 — Infra + qqq Backend Track (US Mgr 2 + India SE 1-2) Frontend Track (US Mgr 3 + India SE 3-4)
P1 D1-2 Tiltfile + Helm values-local.yaml (PG+pgvector, Redis, MinIO, Authentik, Datadog Agent), local k8s namespace. All 117 tables in Liquibase (including llm_provider, llm_model, secret_ref, audit_access_log, task_approval, langgraph_checkpoints) + seed all 59 type tables. qqq config for REST generation. Python project scaffold (FastAPI + Celery + pyproject.toml), Dockerfile. qqq Python API client (auto-generated from OpenAPI). Agent hydration service skeleton. Add ddtrace to pyproject.toml. React scaffold (Vite + TanStack Router/Query + shadcn/ui), Dockerfile. Authentik OIDC login flow. App shell (sidebar, header, command palette, routing). Install @datadog/browser-rum + @datadog/browser-logs.
P1 D3-5 Authentik blueprint (groups, policies, scope mappings including praesidiumInstanceId). RLS via qqq RecordSecurityLock. .faber.yml per repo. Helm charts in praesidium-cd. ArgoCD ApplicationSet. Deploy Datadog Cluster Agent + Admission Controller to EKS. Configure DD_API_KEY in Secrets Manager. Instrument qqq-api with dd-java-agent. LLMClient abstraction (Anthropic-only for V1). AGENT_MANIFEST schema definition (Day 3). Agent hydration (template + tools). POST /chat endpoint with SSE streaming. Inline chat executor with tool loop. Platform tools. Dashboard page (project health cards, team status, escalations count, cost). Type table cache in Zustand. Projects page (card grid + tabbed detail). Team page (agent + human grid). Datadog RUM initialization with allowedTracingOrigins.

Phase 1 Exit Criteria

  • Full stack running locally in Docker Desktop k8s via Tilt (all pods)
  • All 117 tables deployed with REST APIs, auth, RLS
  • AGENT_MANIFEST JSON schema finalized and validated by both Java and Python
  • Datadog Agent collecting container logs + metrics
  • Authentik JWT with praesidiumInstanceId and policies claims verified
  • Frontend App Shell with auth, command palette, and core page stubs
  • CI pipeline green

Phase 2: The Hands (Tool Runner)

K8s Job sandboxing and the “Blind” Secret Pipeline built before Agent Workers to prevent testing deadlocks. Workers need something to call; Tool Runner provides it.

Period James (CTO) + India SE 5 — Infra Backend Track — Tool Runner Frontend Track — Pages
P2 D6-8 IRSA IAM role for Worker pod (Secrets Manager access). secret_ref table seeded with test ARNs. Secrets Manager integration end-to-end. CI pipeline complete (Faber: lint + test + build + GHCR push). Tool Runner base image (Dockerfile.python). BaseTool, ToolContext, ToolResult classes. CLI runner (runner.py). k8s Job template with SHA-only git checkouts (A-004 fix). DD_TRACE_ID + DD_PARENT_ID as env vars in Job spec. Dashboard page finalization. Chat page (SSE streaming + tool call cards). WebSocket hook for live updates. Escalations page (inbox, severity sort). Settings page (instance config, integrations).
P2 D9-10 NetworkPolicy for Tool Runner pods (egress restricted). k8s resource quota presets per risk level. Infrastructure hardening. “Blind” Secret Pipeline: Worker fetches secret_ref ARN from qqq, uses IRSA to fetch value from AWS Secrets Manager at runtime (plain-text never in DB). Verify “Hello World” tool execution from Gateway bus. Integration tests. Activity page (feed with filters). Cost Center page (burn rate charts). Kanban board (drag-and-drop tasks). Agent detail page. Permission-driven UI.

Phase 2 Exit Criteria

  • Tool Runner CLI produces valid ToolResult JSON on stdout
  • k8s Job runs a CUSTOM tool in an ephemeral pod and exits cleanly
  • “Blind” secret pipeline working: secret_ref ARN → IRSA → Secrets Manager → env var
  • No plain-text API keys visible in qqq API responses or database
  • DD_TRACE_ID propagated into Tool Runner Job spec
  • NetworkPolicy blocks all egress except pre-approved integration URLs

Phase 3: The Brain (Agent Workers)

LangGraph state machines, Postgres Checkpointing, and the HITL interrupt() logic. Tool Runner (Phase 2) provides the execution target that workers can call.

Period James (CTO) + India SE 5 — Infra Backend Track — Agent Workers Frontend Track — Pages
P3 D11-13 Multi-LLM support + provider health checks. Budget controls. Datadog APM dashboards (custom praesidium.agent.* metrics via DogStatsD). Service Map verification. LangGraph “Reasoning” loop implementation. PostgresSaver checkpointer setup. job_graph.py (load → execute → evaluate). execute_job task. Instrument graph.invoke() with ddtrace.tracer.trace("agent.reasoning_loop") span. Capture LLM token usage as span metadata. Chat improvements (thinking indicator, group chats). Agent detail (skills radar). Cost Center charts + budget UI. Review/publish flows.
P3 D14-15 HPA tuning, security audit. Network policies. Trace ID propagation through Celery headers (dd_headers). Implement interrupt() for CRITICAL tool calls. State serialized to langgraph_checkpoints table before approval request. Task status set to AWAITING_APPROVAL. task_approval table write with action_payload_hash (SHA-256). Secret resolution (scope hierarchy: AGENT > TOOL > PROJECT > INSTANCE). Stats reconciliation job. Settings improvements (notification preferences). Responsive design. Error boundaries and loading skeletons. Integration settings page (OAuth connection status indicators).

Phase 3 Exit Criteria

  • LangGraph state machine executing jobs with PostgreSQL checkpointing
  • interrupt() pauses execution and writes state to langgraph_checkpoints
  • task_approval record created with tamper-proof action_payload_hash
  • Worker pod can be killed during pending approval; new pod resumes from checkpoint
  • Trace ID propagated: Frontend RUM → Gateway → Celery headers → Worker spans → Tool Runner env vars
  • Custom Datadog metrics emitting (praesidium.agent.token_usage, praesidium.tool.execution_status)

Phase 4: The Gateway (Communication Hub)

SSE Streaming, Chat UI finalization, Task Orchestration, Command Palette with /api/search backend (pgvector). The Gateway now has workers + tool runner to dispatch to.

Period James (CTO) + India SE 5 — Infra Backend Track — Gateway Frontend Track — Chat + Search
P4 D16-18 Begin OAuth setup: Slack OAuth app registration + callback. GitHub OAuth App registration + token exchange. OAuth token storage in Secrets Manager. Gateway SSE server (GET /api/stream/{conversation_id}). POST /api/chat/{agent_id} inline executor. POST /api/task/{task_id}/execute dispatcher. GET /api/internal/worker/hydrate/{agent_id} composite endpoint (AGENT_MANIFEST delivery). POST /api/task/{id}/approve with action hash verification + Redis resume event. SSE interrupt event: when worker interrupt() fires, Gateway pushes interrupt SSE event with approval_id, action_type, and action_summary to the browser. Chat thread UI: SSE streaming tokens, ThinkingIndicator, ToolCallCard. Chat input with model selector. Chat list page with conversation previews. Handle SSE interrupt event: transition Chat UI from “Thinking...” to “Approval Required” card with action summary, diff preview, and Approve/Reject buttons.
P4 D19-20 Slack and GitHub OAuth flows end-to-end. Token refresh logic. Integration health check framework. GET /api/search — Command Palette backend (pgvector semantic search across projects, agents, tasks, conversations, tools). WebSocket server with Redis pub/sub broadcast. Prometheus metrics + Correlation ID middleware. Rate limiter. Command Palette (Cmd+K) wired to /api/search backend. Meetings list + escalations. Group chats. Activity feed (real-time via WS). Responsive design verification.

Phase 4 Exit Criteria

  • SSE streaming chat working end-to-end (browser → gateway → LLM → SSE tokens)
  • SSE interrupt event triggers frontend transition from ThinkingIndicator to Approval Required card
  • Task dispatch creates Celery task; WS broadcasts work across pods
  • /api/internal/worker/hydrate/{agent_id} returns complete AGENT_MANIFEST
  • /api/task/{id}/approve validates hash, dispatches Redis resume event
  • /api/search returns ranked results from pgvector for Command Palette
  • Slack and GitHub OAuth flows registered with token exchange working

Phase 5: Real-World Integrations

Slack, Jira, GitHub toolsets and Webhook listeners. OAuth flows from Phase 4 accelerate connector work. Confluence, Zoho, and Google Workspace deferred to V1.1.

Period James (CTO) — Integrations Backend Track — Connectors Frontend Track — Integration UI
P5 D21-23 Slack connector (DM, channel post, interactive buttons). Jira connector (create/transition/search issues, bidirectional sync). Webhook receiver in Gateway. GitHub connector (PR creation, issue filing, code search). Integration tool dispatch (PLATFORM + INTEGRATION in-process). Secret resolution with IRSA for integration credentials. Integration connection management UI. OAuth connection status indicators in Settings. Connector status dashboard. Approval notification indicators in sidebar.
P5 D24-25 Human interaction flow (Slack DM approvals). Integration health checks. Cross-connector integration tests. Celery Beat scheduler with Redis-based leader election. Scheduled job execution. scheduler_heartbeat table: leader writes heartbeat every 30s, secondary monitors and takes over if last_heartbeat > 60s stale. Integration tests across all 3 connectors. Webhook event routing to agents/tasks/projects. Inline approval prompts in chat view for HIGH/CRITICAL actions. End-to-end approval flow testing. Settings improvements (notification preferences).

Phase 5 Exit Criteria

  • Slack, Jira, GitHub integrations working end-to-end
  • Webhook listener in Gateway routes external events to correct agent/task
  • Tool registry resolves PLATFORM and INTEGRATION tools with risk-level gates
  • Celery Beat scheduled job execution with leader election
  • Scheduler HA: scheduler_heartbeat table tracks leader liveness; failover tested within 60s
  • Slack and GitHub OAuth flows fully operational with token refresh

Phase 6: Business Operations

KPI Dashboards, HITL Approval UI (with action hash verification), Schedule Management UI, Rollback logic, Human Feedback, and production deployment preparation.

Period James (CTO) — Production Backend Track — Business Logic Frontend Track — Business UI
P6 D26-28 AWS deployment via existing EKS + Faber (.faber.yml) + ArgoCD. Helm values in praesidium-cd. Secrets Manager. Staging deploy, smoke test. KPI baseline import (load Pre-Build survey data into kpi_baseline table). KPI measurement service. Agent quality score computation. Rollback/recovery: undo API endpoints with revert_payload logic. Emergency kill switch. HITL Approval Dashboard — pending approvals list with action hash verification (payload shown matches action_payload_hash). Approve/reject/request-changes actions. KPI dashboard with baseline comparison charts. Schedule Management UI — Schedule List (/schedules), Schedule Detail/Edit (/schedules/:id), Run History, cron picker, and + New Schedule form.
P6 D29-30 Production deploy. Phased rollout: deployment checklist, go/no-go criteria. First instance provisioned (Engineering team). Human feedback collection endpoint. Approval notification pipeline (Slack DM on pending). Approval timeout/escalation logic. Agent performance management lifecycle. Kill criteria gate evaluation job. Human Feedback widget (thumbs up/down + comment). Rollback controls in UI. Emergency kill switch in admin panel. Agent performance management UI. Kill criteria gate status cards.

Phase 6 Exit Criteria

  • Production-ready on AWS EKS (leveraging existing infrastructure)
  • Engineering team deployed and actively using the platform
  • HITL Approval UI with tamper-proof action hash verification
  • Rollback/undo capability for agent actions + emergency kill switch tested
  • KPI baseline imported from Pre-Build survey, comparison dashboard live
  • Schedule Management UI live — Schedule List, Detail/Edit, Run History, cron picker, + New Schedule
  • Human feedback widget live on all agent output surfaces
  • Business-value kill criteria gates evaluated automatically
  • Datadog Service Map shows full 5-service waterfall with single Trace ID

Phase 7: Stabilization

Security audit, load testing, production hardening, second team rollout (Marketing), and final polish.

Period James (CTO) + India SE 5 — Ops Backend Track — Stabilization Frontend Track — Polish
P7 D31-33 Security audit (OWASP, dependency scan). Load testing (k6). Bug fixes from Engineering team feedback. Infrastructure tuning. Monitoring refinement. Datadog SDS rules for PHI redaction verified. Bug fixes from production usage. Performance optimization. Approval workflow tuning. Checkpoint pruning job (90-day retention). E2E backend testing. Cost tracking accuracy validation. UX improvements based on Engineering team feedback. E2E frontend testing (Playwright). Accessibility audit. Cross-browser. In-app onboarding tooltip tour. Quality score visualization.
P7 D34-35 Marketing team provisioning. Instance configuration for Marketing use cases. Go/no-go check for second team. Grafana/Datadog dashboards live, budget alerts working. Marketing team agent templates. Team-specific integration configs. KPI comparison report (Phase 1 vs baseline). API documentation generated. Quickstart guide generation. Marketing-specific dashboard views. Template library for Marketing workflows. Final UI polish. 90%+ test coverage across all services.

Phase 7 Exit Criteria (V1 Launch Complete)

  • Engineering team stable on platform for 1+ week
  • Marketing team onboarded and productive
  • KPI improvement measurable vs baseline
  • Approval workflow validated with real human-in-the-loop usage
  • Zero critical bugs in production
  • Security audit clean, load testing passing
  • 90%+ test coverage across all services
  • Documentation complete

V1.1 (Post-Launch): Additional Integrations

V1.1 — 2 Weeks Post-Launch
Confluence, Zoho CRM, and Google Workspace connectors. Specs retained below in service sections. Prioritized based on team adoption feedback from V1.
  • Confluence connector (page CRUD + search)
  • Zoho CRM connector (contact/deal lookup + create)
  • Google Workspace connector (Calendar events, Docs creation)
  • Integration tests for all new connectors

V1.5 (Post-Launch): Advanced Features

Deferred to V1.5
Tool Forge, Memory/RAG pipeline, and Meeting/Standup orchestration. Full implementation specs retained in their respective service tabs.
  • Tool Forge: LangGraph 6-state workflow, tool-runner sandboxed execution, forge UI (split-pane build mode)
  • Memory/RAG: Memory extraction, embedding generation, semantic recall, knowledge document indexing
  • Meeting Orchestration: Turn-taking orchestrator standups/huddles, action item extraction, meeting scheduling

Service Dependency Graph

Services must be built in this order. Each layer depends on the layers above it.

Tilt + k8s

Local dev environment

▼

qqq Data Layer

117 tables + REST APIs + RLS

Authentik Config

Groups, policies, OIDC

Datadog

APM, RUM, Logs, SDS

▼

Tool Runner

K8s Jobs + Blind Secret Pipeline

▼

Agent Workers

LangGraph + HITL + Checkpoints

▼

Agent Gateway

SSE + /api/search + HITL approve

Frontend SPA

React + Cmd+K + HITL UI

▼

Integrations

Slack, Jira, GitHub, etc.

Critical path: Local k8s + Datadog → qqq + Authentik → Tool Runner → Agent Workers → Gateway + Frontend → Integrations

qqq Data Layer

Epic 1 — Days 1-2

This is the FIRST service to build. Everything else depends on it.

The qqq data layer provides all 117 tables (58 entity + 59 type) with auto-generated REST APIs, row-level security (RLS), and Authentik JWT authentication. Includes 4 new tables from the Agent Runtime Spec: llm_provider, llm_model, secret_ref, and audit_access_log, plus scheduler_heartbeat (v1.9 HA). No other service can start meaningful work until qqq APIs are available.

Repository Setup

Repositorypraesidium2-qqq/
LanguageJava 21
Build ToolGradle (Kotlin DSL)
Frameworkqqq + Spring Boot
DatabasePostgreSQL 17 + pgvector
MigrationsLiquibase
AuthAuthentik OIDC (ProdAuthenticationCustomizer pattern)
praesidium2-qqq/ src/main/java/com/dmdbrand/praesidium/ PraesidiumQqqApplication.java # Spring Boot entry point config/ ProdAuthenticationCustomizer.java # Authentik JWT validation QqqMetaDataProvider.java # Table + API definitions SecurityConfig.java # RecordSecurityLock setup src/main/resources/ db/changelog/ db.changelog-master.yaml # Liquibase root 000-extensions/ 001-pgvector.sql # CREATE EXTENSION vector 100-type-tables/ 101-member-type.sql # ... through 145 200-entity-tables/ 201-team-member.sql # Domain 1b first (no deps) 202-agent-template.sql 203-agent.sql # ... through 234 206-audit-access-log.sql # HIPAA audit access log (partitioned) 235-baa-tracking-columns.sql # BAA tracking on integration_config 236-job-execution-debug-columns.sql # debug_payload JSONB + trace_id VARCHAR(64) on job_execution 237-memory-ranking-columns.sql # recency_weight DECIMAL + importance DECIMAL on memory (v1.10) 300-seeds/ 301-type-table-seeds.sql # All 59 type tables seeded 400-indexes/ 401-pgvector-indexes.sql # IVFFlat on memory + knowledge_chunk 500-triggers/ 501-polymorphism-enforcement.sql # team_member/agent/human_profile application.yml build.gradle.kts Dockerfile CLAUDE.md

Table Implementation Order

Tables MUST be created in this order due to foreign key dependencies. Type tables have zero FK dependencies and go first. Entity tables are ordered by domain with dependency-aware sequencing.

Phase 1: Extensions (changeset 000)

ChangesetAction
001-pgvector.sqlCREATE EXTENSION IF NOT EXISTS vector;

Phase 2: Type Tables (changesets 100-159)

All 59 type tables follow the standard pattern. Create all of them before any entity tables. Order does not matter between type tables (no inter-dependencies).

ChangesetTableSeed Values
101member_typeAGENT, HUMAN
102member_statusACTIVE, AWAY, OFFLINE, SUSPENDED
103agent_categoryENGINEERING, HR, OPS, FINANCE, MARKETING, PRODUCT, SUPPORT, CUSTOM
104execution_statusIDLE, EXECUTING, QUEUED, ERROR
105tool_categoryPLATFORM, INTEGRATION, CUSTOM, MCP
106tool_source_typeBUILTIN, FORGED, IMPORTED
107risk_levelLOW, MEDIUM, HIGH, CRITICAL
108test_statusPENDING, RUNNING, PASSED, FAILED
109programming_languagePYTHON, TYPESCRIPT, JAVA, GO, SHELL
110sync_statusSYNCED, PENDING, SYNCING, ERROR
111integration_typeZOHO_CRM, JIRA, CONFLUENCE, SLACK, GITHUB, GOOGLE_WORKSPACE, CUSTOM_API
112auth_typeAPI_KEY, OAUTH2, BASIC, BEARER, SERVICE_ACCOUNT
113health_statusHEALTHY, DEGRADED, DOWN, UNKNOWN
114project_typeSHORT_TERM, ONGOING, SUPPORT, INITIATIVE
115project_statusPLANNING, ACTIVE, ON_HOLD, COMPLETED, ARCHIVED
116project_healthON_TRACK, AT_RISK, BEHIND, BLOCKED
117priority_levelLOW, MEDIUM, HIGH, CRITICAL
118project_roleLEAD, MEMBER, REVIEWER, OBSERVER
119milestone_statusNOT_STARTED, IN_PROGRESS, COMPLETED, BLOCKED
120task_typeTASK, BUG, STORY, SPIKE, REVIEW
121task_statusOPEN, APPROVED, IN_PROGRESS, IN_REVIEW, DONE, BLOCKED, DISMISSED
122author_roleAGENT, HUMAN, SYSTEM
123conversation_typeCHAT, STANDUP, HUDDLE, TASK_EXEC, FORGE, REVIEW
124conversation_statusACTIVE, PAUSED, COMPLETED, FAILED
125participant_roleFACILITATOR, PARTICIPANT, OBSERVER, SCRIBE
126message_roleUSER, ASSISTANT, SYSTEM, TOOL_RESULT
127content_typeTEXT, MARKDOWN, JSON, IMAGE_REF, CODE
128meeting_typeSTANDUP, HUDDLE, RETROSPECTIVE, PLANNING, REVIEW
129memory_typeFACT, PREFERENCE, LESSON_LEARNED, ENTITY, DECISION, PROCEDURE
130memory_sourceCONVERSATION, TASK, MEETING, MANUAL, SYSTEM
131document_source_typeUPLOAD, CONFLUENCE, URL, GIT_REPO
132index_statusPENDING, INDEXING, INDEXED, FAILED
133job_statusACTIVE, PAUSED, COMPLETED, FAILED
134job_execution_statusRUNNING, SUCCESS, FAILED, TIMEOUT, CANCELLED
135trigger_typeCRON, MANUAL, RETRY, EVENT
136escalation_statusOPEN, ACKNOWLEDGED, RESOLVED, DISMISSED
137artifact_typeFILE, REPORT, CODE, IMAGE, DATA
138interaction_channelSLACK, JIRA_COMMENT, JIRA_ASSIGN, CONFLUENCE, EMAIL
139interaction_typeQUESTION, TASK_ASSIGN, STATUS_UPDATE, ESCALATION, REVIEW_REQUEST, INFO_REQUEST
140interaction_statusSENT, DELIVERED, SEEN, RESPONDED, EXPIRED, FAILED
141schedule_statusACTIVE, PAUSED, ARCHIVED

Phase 3: Entity Tables by Domain (changesets 200-234)

Entity tables MUST be created in this order. Each domain builds on previous domains' tables.

OrderDomainTablesDepends On
1 Domain 1b: Team Identity team_member, human_profile Type tables only
2 Domain 1: Agent Identity + LLM Config agent_template, agent, agent_skill, llm_provider, llm_model, secret_ref team_member
3 Domain 2: Tools (partial) tool_repo, integration_config, tool, tool_version, agent_tool_assignment agent
4 Domain 3: Projects project, project_member, milestone, task, task_comment team_member
5 Domain 4: Conversations scheduled_meeting, conversation, conversation_participant, message project, task, team_member
6 Domain 5: Memory & Knowledge memory v1.10: +recency_weight, +importance, knowledge_document, knowledge_chunk team_member, project, conversation
7 Domain 6: Operations scheduled_job, job_execution v1.10: +debug_payload, +trace_id, escalation, artifact, activity_log, audit_access_log, scheduler_heartbeat agent, project, task, conversation
8 Domain 1b: Human Interaction human_interaction team_member, conversation, task
9 Domain 7: Business Operations task_approval_queue, approval_policy, autonomy_level, kpi_baseline, kpi_measurement, team_roi_report, agent_action_log, rollback_record, output_feedback, agent_quality_score agent, task, team_member, project

Liquibase Changelog Structure

Each changeset is a separate SQL file. The master changelog includes them in order.

# db.changelog-master.yaml

databaseChangeLog:
  - includeAll:
      path: db/changelog/000-extensions/
  - includeAll:
      path: db/changelog/100-type-tables/
  - includeAll:
      path: db/changelog/200-entity-tables/
  - includeAll:
      path: db/changelog/300-seeds/
  - includeAll:
      path: db/changelog/400-indexes/
  - includeAll:
      path: db/changelog/500-triggers/

Type Table SQL Template

Every type table MUST follow this exact pattern. Copy-paste and change the table name.

-- changeset praesidium:101-member-type
CREATE TABLE member_type (
  id          INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  name        VARCHAR(50) NOT NULL UNIQUE,
  label       VARCHAR(100) NOT NULL,
  description VARCHAR(500),
  icon        VARCHAR(50),
  color       VARCHAR(7),
  sort_order  INT DEFAULT 0,
  is_active   BOOLEAN DEFAULT true,
  is_system   BOOLEAN DEFAULT false,
  metadata    JSONB DEFAULT '{}'::jsonb,
  created_at  TIMESTAMPTZ DEFAULT now()
);

-- Seed data
INSERT INTO member_type (name, label, sort_order, is_system) VALUES
  ('AGENT',  'Agent',  1, true),
  ('HUMAN',  'Human',  2, true);

qqq Configuration for REST API Generation

qqq auto-generates CRUD REST endpoints for every table defined in the QqqMetaDataProvider. The configuration maps each table's columns, relationships, and security constraints.

What qqq Generates Automatically

  • GET /api/{table} — List with pagination, sorting, filtering
  • GET /api/{table}/{id} — Get by ID with optional ?include=
  • POST /api/{table} — Create with validation
  • PATCH /api/{table}/{id} — Partial update
  • DELETE /api/{table}/{id} — Delete (policy-controlled)
  • OpenAPI/Swagger documentation
  • Automatic FK resolution in responses
  • Query parameter filtering on all columns

Composite Endpoints (qqq ?include=)

EndpointReplaces
GET /agent/{id}?include=template,tools,skills,recentMemories5 GET calls
GET /project/{id}?include=milestones,team,recentTasks,conversations5 GET calls
GET /conversation/{id}?include=participants,messages,task3 GET calls

Authentik Integration (ProdAuthenticationCustomizer Pattern)

Copy the exact pattern from me-health-portal. Authentik sends JWTs with custom claims that qqq uses for authentication and authorization.

JWT Claims Used by qqq

  • "policies" — IAM-style policy JSON (Statement array with Effect/Action/Resource)
  • "praesidiumInstanceId" — UUID for instance scoping (maps to RecordSecurityLock)
  • "praesidiumAllAccess" — Boolean admin bypass (default false)

ProdAuthenticationCustomizer.java Key Behaviors

  • Validates JWT signature against Authentik JWKS endpoint
  • Extracts praesidiumInstanceId and sets it as the RecordSecurityLock value
  • Parses IAM-style policy JSON from policies claim
  • Maps policy Statements to qqq table permissions (read/insert/edit/delete)
  • If praesidiumAllAccess = true, bypasses all policy checks (admin only)
  • Rejects requests with missing/expired/invalid tokens with 401

RecordSecurityLock Configuration (instance_id)

Every root entity table gets a RecordSecurityLock on instance_id. Child tables inherit scoping via FK joins. This is the same pattern as me-health-portal's mhCompanyId.

Root Tables (Direct instance_id)

These tables have a direct instance_id UUID NOT NULL column added by qqq:

project team_member tool integration_config scheduled_meeting knowledge_document agent_template

Child Tables (Join-Based Scoping)

These tables trace their FK chain to a root table's instance_id:

agent→ team_member.instance_id
task→ project.instance_id
message→ conversation → project.instance_id
memory→ team_member.instance_id
milestone→ project.instance_id
conversation→ project.instance_id

Test Requirements

Integration tests for every CRUD endpoint. Tests run against a real PostgreSQL instance (Testcontainers).

Mandatory Test Coverage

  • Every entity table: test GET (list + by ID), POST, PATCH, DELETE
  • Every type table: test GET (verify seed data loaded correctly)
  • RLS enforcement: test that requests with Instance A's JWT cannot read/write Instance B's data
  • Policy enforcement: test that a "Viewer" policy cannot POST/PATCH/DELETE
  • FK cascade: test that deleting a parent cascades to children correctly
  • Composite endpoints: test ?include= returns nested data
  • Validation: test required fields, unique constraints, FK integrity

Acceptance Criteria

  • All 117 tables created via Liquibase migrations (idempotent, re-runnable)
  • All 59 type tables seeded with data from design spec
  • All 117 tables accessible via REST API (GET, POST, PATCH, DELETE where allowed)
  • Authentik JWT authentication working (401 on missing/invalid token)
  • IAM-style policy enforcement (403 on unauthorized actions)
  • RecordSecurityLock on instance_id enforced on all root entity tables
  • Composite endpoints working (?include= resolves joins)
  • pgvector extension installed, IVFFlat indexes on memory.embedding and knowledge_chunk.embedding
  • Polymorphism trigger enforcing team_member/agent/human_profile consistency
  • OpenAPI/Swagger documentation auto-generated
  • Integration tests passing for all endpoints (90%+ coverage target)
  • Docker image builds and runs correctly

CLAUDE.md Rules for qqq Data Layer Agents

These rules MUST be placed in the CLAUDE.md file at the root of the praesidium2-qqq/ repository.

# praesidium2-qqq CLAUDE.md

## Project Context
This is the qqq data layer for Praesidium2 -- an AI agent workforce platform.
qqq is a Java framework that auto-generates REST APIs from table/ERD definitions.
This service is the SINGLE SOURCE OF TRUTH for all data. No other service
touches the database directly.

## Architecture Rules
- All table definitions go in QqqMetaDataProvider.java
- All migrations go in src/main/resources/db/changelog/ as Liquibase SQL
- Follow the type table standard pattern EXACTLY (see design spec)
- Every entity table that is a root table gets RecordSecurityLock on instance_id
- Use the ProdAuthenticationCustomizer pattern from me-health-portal
- NEVER skip writing integration tests for new endpoints

## Coding Standards
- Java 21, use records where appropriate
- Gradle Kotlin DSL for build configuration
- Follow existing qqq conventions for MetaDataProvider setup
- Use Testcontainers (PostgreSQL) for integration tests
- Target 90%+ test coverage

## Domain 1 Additions (v1.8 -- 4 new tables)
- llm_provider: LLM provider catalog (name, api_base_url, is_active, metadata)
- llm_model: Specific model config (provider_id FK, context_window, cost_per_1m_tokens)
- secret_ref: Scoped secret references (instance_id, secret_key, aws_secret_arn)
- audit_access_log: HIPAA-mandated access tracking (7-year retention, partitioned)
- Total tables: 117 (58 entity + 59 type)

## Domain 6 Addition (v1.9 -- 1 new table)
- scheduler_heartbeat: Celery Beat HA leader tracking (node_id, last_heartbeat, is_leader, metadata)

## Domain 6 Addition (v1.10 -- new columns on job_execution)
- debug_payload JSONB: stores exact system prompt + tool definitions sent to LLM
- trace_id VARCHAR(64): OpenTelemetry trace ID for all LLM/tool calls in this execution
- Liquibase changeset: 236-job-execution-debug-columns.sql

## Domain 5 Addition (v1.10 -- new columns on memory)
- recency_weight DECIMAL DEFAULT 1.0: time-decay factor for memory recall ranking
- importance DECIMAL DEFAULT 0.5: relevance score (0.0-1.0) for memory prioritization
- Ships in V1 even though full memory pipeline is V1.5
- Liquibase changeset: 237-memory-ranking-columns.sql

## Domain 7: Business Operations (10 tables)
- task_approval_queue, approval_policy, autonomy_level
- kpi_baseline, kpi_measurement, team_roi_report
- agent_action_log, rollback_record
- output_feedback, agent_quality_score
- These tables support approval workflows, KPI measurement, rollback/recovery,
  feedback, and agent performance management.
- All follow standard entity table patterns with RecordSecurityLock on instance_id.

## What NOT to Do
- Do NOT create custom REST controllers -- qqq generates all endpoints
- Do NOT add business logic to this service -- it is a data layer only
- Do NOT bypass RecordSecurityLock for any reason
- Do NOT modify type table schemas -- they follow the standard pattern
- Do NOT hardcode any instance IDs or tenant values

Agent Gateway

Epic 2 — Phase 4

The bridge between the browser and the AI agent runtime.

The Agent Gateway is a Python/FastAPI service that handles all AI-related HTTP traffic: chat SSE streams, task dispatch, approval workflow, and WebSocket broadcasts. It reads/writes all persistent data via the qqq REST API — it NEVER touches the database directly. Inline 1:1 chat runs in-process (no Celery round-trip); everything else is dispatched to Agent Workers via Redis task queues. Note: meeting orchestration and tool forge session endpoints are retained for V1.5.

Repository Setup

Repositorypraesidium2-python/services/agent-gateway/
LanguagePython 3.12
Build Toolpyproject.toml (hatchling)
FrameworkFastAPI + Uvicorn
Task QueueCelery (producer only) via Redis
AuthAuthentik JWT (same token as qqq, forwarded)
Replicas2-8 pods, HPA on WS connections + req/s
Resources1 CPU / 2Gi per pod
praesidium2-python/services/agent-gateway/ src/gateway/ __init__.py main.py # FastAPI app factory, lifespan config.py # Pydantic Settings (env-driven) dependencies.py # DI: qqq client, redis, celery auth/ __init__.py jwt.py # Authentik JWT validation + claims middleware.py # Auth middleware (verify + inject) routers/ __init__.py chat.py # POST /api/chat/{agent_id} tasks.py # POST /api/task/{task_id}/execute approvals.py # GET/POST /api/approvals (approval workflow) feedback.py # POST /api/feedback (human feedback collection) rollback.py # POST /api/actions/{id}/undo, /tasks/{id}/rollback admin.py # POST /api/admin/emergency-stop, /resume, kill-criteria meetings.py # V1.5: POST /api/meeting/{meeting_id}/start forge.py # V1.5: POST /api/forge/session jobs.py # POST /api/job/{job_id}/trigger stream.py # GET /api/stream/{conversation_id} agents.py # GET /api/agent/{agent_id}/status webhooks.py # POST /api/webhooks/{integration_type} hydrate.py # GET /api/internal/worker/hydrate/{agent_id} debug.py # POST /api/internal/debug/replay/{execution_id} (v1.10) search.py # GET /api/search (Command Palette pgvector) health.py # GET /healthz, /readyz, /metrics ws.py # WebSocket /ws endpoint services/ __init__.py hydration.py # Agent hydration: qqq fetch + Redis cache chat_executor.py # Inline 1:1 chat (Anthropic tool loop) task_dispatcher.py # Celery task creation + queue routing ws_manager.py # WebSocket connection mgr + Redis pub/sub sse_manager.py # SSE stream registry + event dispatch llm/ __init__.py protocol.py # LLMClient Protocol interface anthropic_client.py # V1 implementation (Anthropic SDK) factory.py # build_llm_client() by provider type clients/ __init__.py qqq.py # Typed httpx client for qqq REST API redis.py # Redis connection + pub/sub helpers models/ __init__.py requests.py # Pydantic request models responses.py # Pydantic response models events.py # SSE + WS event schemas agent.py # HydratedAgent, AgentContext middleware/ __init__.py correlation.py # X-Correlation-ID propagation metrics.py # Prometheus middleware rate_limit.py # Per-agent per-tool Redis rate limiter tests/ conftest.py # Fixtures: mock qqq, mock Redis, mock Anthropic test_routers/ test_chat.py test_tasks.py test_approvals.py test_feedback.py test_rollback.py test_admin.py test_jobs.py test_stream.py test_agents.py test_ws.py test_services/ test_hydration.py test_chat_executor.py test_task_dispatcher.py test_llm/ test_anthropic_client.py test_factory.py pyproject.toml Dockerfile CLAUDE.md

Day-by-Day Tasks (Gateway Track)

The gateway is built incrementally across Phase 1 (Days 1-5) by Backend Track, then enhanced in Phase 2 (Days 6-10).

DayTaskExit Criteria
D1 Python project scaffold — pyproject.toml, FastAPI app factory, Uvicorn entry, Dockerfile, linting (ruff), formatting (black), pytest config uvicorn gateway.main:app starts, /healthz returns 200
D2 qqq Python API client (auto-generated from OpenAPI or hand-typed with httpx). Agent hydration service skeleton. Redis connection. Pydantic settings. Can fetch an agent record from qqq API. Redis ping succeeds.
D3 LLMClient abstraction (Protocol + Anthropic implementation; V1 is Anthropic-only). Agent hydration full pipeline (template + tools + memories + system prompt build). Redis cache (5min TTL). Can hydrate an agent with correct system prompt. LLM client can send a message.
D4 POST /api/chat/{agent_id} with SSE streaming. Inline chat executor with Anthropic tool loop. Platform tool resolution. Conversation + message persistence via qqq API. Can chat with an agent in browser, see SSE tokens stream, tool calls execute and stream back.
D5 Celery task dispatcher. POST /api/task/{task_id}/execute. POST /api/meeting/{meeting_id}/start. WebSocket server with Redis pub/sub broadcast. GET /api/stream/{conversation_id} SSE reconnect. Task dispatch creates Celery task. Meeting dispatch creates Celery task. WS broadcasts work across pods.
D6-7 (Phase 2) POST /api/forge/session. POST /api/job/{job_id}/trigger. GET /api/agent/{agent_id}/status. Webhook receiver. Auth middleware with Authentik JWT validation. All 8+ endpoints functional with JWT auth. Forge session creates LangGraph workflow.
D8-10 (Phase 2) Prometheus metrics (6 custom gauges/counters). Correlation ID middleware. Rate limiter. Integration health checks. Error handling + structured logging. Metrics at /metrics. Correlation IDs propagated to qqq and Celery. Rate limits enforced.

API Endpoints

Every route requires a valid Authentik JWT in the Authorization: Bearer header. The JWT's praesidiumInstanceId claim is forwarded to all qqq API calls for row-level security scoping.

MethodPathDescriptionResponse
POST /api/chat/{agent_id} Start or continue a 1:1 chat. Runs INLINE in the gateway (no Celery). Returns SSE stream of tokens + tool calls. text/event-stream (SSE)
POST /api/task/{task_id}/execute Dispatch task execution to a Celery worker. Creates conversation, queues execute_task on standard queue. { conversation_id, celery_task_id }
POST /api/meeting/{meeting_id}/start Trigger a scheduled meeting (standup/huddle). Queues run_meeting on standard queue. { conversation_id, celery_task_id }
POST /api/forge/session Start an interactive tool forge session. Queues forge_session on long queue. Returns SSE stream for live updates. { conversation_id, celery_task_id }
POST /api/job/{job_id}/trigger Manually trigger a scheduled job. Queues execute_job on standard or long queue based on job config. { job_execution_id, celery_task_id }
GET /api/stream/{conversation_id} SSE stream for an active conversation. Reconnects with Last-Event-ID header. Used by worker-executed tasks to push tokens to the browser. text/event-stream (SSE)
GET /api/agent/{agent_id}/status Real-time status of an agent: execution state, active conversation, queue position. { status, active_conversation_id, queue_position }
WS /ws WebSocket for broadcast events. Gateway subscribes to Redis pub/sub channels and pushes events to all connected browsers. Events: task updates, meeting progress, agent status changes, tool published, escalations. JSON frames per event type
POST /api/webhooks/{integration_type} Inbound webhook receiver for Jira, Slack, GitHub, etc. Verifies signature, parses event, routes to relevant agent/task/project. 202 Accepted
GET /api/internal/worker/hydrate/{agent_id} Agent manifest delivery. Returns the complete AGENT_MANIFEST JSON (agent persona, LLM config, secret ARNs, tools). Auth: Service Account JWT scoped to instance. See AGENT_RUNTIME_SPEC. { agent, config, secrets[], tools[] }
POST /api/task/{id}/approve HITL approval signal. Validates IAM permissions, verifies action_payload_hash (tamper-proof), updates task_approval status, dispatches Redis resume event with thread_id. See HITL_ORCHESTRATION_SPEC. { approved: true, resumed_thread_id }
POST /api/internal/debug/replay/{execution_id} Agent debug replay. Loads debug_payload from job_execution and re-runs through qqq Process for step-through debugging. Auth: Service Account JWT. v1.10 { replay_id, trace_id, steps[] }
GET /api/search Command palette backend. Accepts ?q= query string and returns ranked results across projects, agents, tasks, conversations, and tools using pgvector semantic search. { results: [{ type, id, title, score }] }
GET /healthz Liveness probe. Returns 200 if the process is running. { status: "ok" }
GET /readyz Readiness probe. Checks Redis + qqq API connectivity. { status: "ready", redis: true, qqq: true }
GET /metrics Prometheus-format metrics endpoint. Prometheus text format

SSE Event Types (chat + stream endpoints)

EventData ShapeDescription
token{ text: string }Incremental text token from the LLM
tool_call_start{ tool_name, tool_id, input }Agent is invoking a tool
tool_call_result{ tool_name, result, duration_ms }Tool execution completed
thinking{ text: string }Extended thinking content (if model supports it)
message_complete{ message_id, tokens_used, cost_usd }Full assistant message stored
interrupt{ approval_id, action_type, action_summary, risk_level, agent_id, task_id }LangGraph interrupt() fired — frontend transitions from “Thinking” to “Approval Required” v1.9
error{ code, message, retry_after? }Error during processing
done{}Stream ended normally

WebSocket Event Types

ChannelEventDescription
task:{id}status_changedTask moved to new status (QUEUED, IN_PROGRESS, DONE, etc.)
agent:{id}execution_changedAgent execution state changed (IDLE, EXECUTING, ERROR)
meeting:{id}message_addedNew message in a meeting conversation
meeting:{id}meeting_completedMeeting finished, summary + action items available
forge:{id}state_changedForge workflow moved to new state (DESIGN, SCAFFOLD, etc.)
tool:publishedtool_publishedNew tool or tool version published to catalog
escalation:newescalation_createdNew escalation requiring human attention

Chat Request/Response Shapes

POST /api/chat/{agent_id} — Request

{
  "message": "Review PR #42 and summarize the changes",
  "conversation_id": "uuid | null",     // null = new conversation
  "project_id": "uuid | null",          // project context
  "task_id": "uuid | null",             // task context
  "attachments": [                       // optional
    { "type": "file", "url": "s3://...", "name": "report.pdf" }
  ]
}

POST /api/task/{task_id}/execute — Request

{
  "instructions": "Additional context...",   // optional override
  "collaborator_ids": ["uuid", "uuid"],      // optional extra agents
  "priority": "HIGH",                        // queue priority hint
  "max_budget_usd": 5.00                     // cost guard
}

POST /api/meeting/{meeting_id}/start — Request

{
  "agenda_override": "Focus on blockers",    // optional
  "additional_participants": ["uuid"]        // optional
}

Agent Hydration Service

On every interaction, the agent is hydrated fresh from qqq APIs. A Redis cache (5min TTL) avoids repeated round-trips for back-to-back requests. Hydration produces a HydratedAgent with everything the LLM needs.

Hydration Pipeline (6 Steps)

StepSourceWhat It Loads
1 GET /agent/{id}?include=template,skills Agent record + agent_template (role, backstory, default_model, system_prompt) + skills list
2 GET /agent_tool_assignment?agentId={id} Tool assignments → resolve each to a callable tool instance (PLATFORM, INTEGRATION, or CUSTOM dispatch stub)
3 GET /memory?agentId={id}&orderBy=-importance&limit=50 Top 50 memories by importance score, plus semantic search with current context query
4 GET /llm_model/{model_id} + GET /llm_provider/{provider_id} Resolve model (agent override > template default). Build LLMClient via factory. Check provider health, fall back if DOWN.
5 In-memory Build system prompt: template.system_prompt + agent.soul + skills context (v1.12: ordered by assigned_at ASC, 40/30/20/10 context budget, tool requirement validation) + memory injection + system awareness block + prompt injection defense preamble
6 Redis SET (5min TTL) Cache the HydratedAgent as JSON. Cache key: agent:{id}:hydrated. Invalidated on agent config change via WS event.

Composite qqq Endpoints Reduce Round-Trips

Step 1 uses qqq's ?include= mechanism to fetch agent + template + skills in a single HTTP call instead of 3. The hydration pipeline makes ~4 HTTP calls total (agent composite, tools, memories, model) instead of ~10 without includes.

Inline Chat Executor

1:1 chat runs directly in the gateway process — no Celery worker needed. This eliminates the queue round-trip for the most latency-sensitive interaction. The executor implements the Anthropic tool-use loop with SSE streaming.

Execution Flow

Browser POST /api/chat/{agent_id}
  |
  +-- 1. Validate JWT, extract instance_id
  +-- 2. Hydrate agent (cache hit or fresh from qqq)
  +-- 3. Load/create conversation via qqq API
  +-- 4. Persist user message: POST /message
  +-- 5. Build messages array (system + history + user)
  |
  +-- 6. TOOL LOOP (streaming):
  |     a. Call LLMClient.create_message(stream=True)
  |     b. Stream tokens to browser via SSE
  |     c. If tool_use block received:
  |        - Validate tool input against JSON Schema
  |        - Check risk level (LOW/MEDIUM = execute, HIGH = log+notify, CRITICAL = await approval)
  |        - Check per-agent per-tool rate limit (Redis token bucket)
  |        - Execute tool (in-process for PLATFORM/INTEGRATION, HTTP for CUSTOM)
  |        - Stream tool_call_start + tool_call_result SSE events
  |        - Append tool_result to messages, loop back to (a)
  |     d. If end_turn: break loop
  |
  +-- 7. Persist assistant message: POST /message (with tokens, cost, tool_calls JSONB)
  +-- 8. Update agent stats: PATCH /agent (last_active, tokens, cost)
  +-- 9. Publish "agent:status_changed" to Redis pub/sub
  +-- 10. Queue "extract_memories" on quick queue (background)
  +-- 11. Send SSE "message_complete" + "done" events

Tool Loop Safety

The tool loop has a hard limit of 25 iterations per turn. If the agent keeps calling tools beyond this limit, the loop breaks and the agent is forced to respond with what it has. This prevents runaway tool calling from prompt injection or agent confusion.

Task Dispatcher

For operations that take >30 seconds or require multi-agent coordination, the gateway dispatches work to Celery workers via Redis. The dispatcher determines the correct queue based on task type.

Gateway EndpointCelery TaskQueuePre-Dispatch Actions
POST /api/task/{id}/execute execute_task standard Create conversation, PATCH task status → QUEUED, validate assignee, check budget
POST /api/meeting/{id}/start run_meeting standard Create conversation, add participants, PATCH meeting status → IN_PROGRESS
POST /api/forge/session forge_session long Create conversation, create tool record (draft), initialize LangGraph thread
POST /api/job/{id}/trigger execute_job standard or long Create job_execution (RUNNING), validate job config, check consecutive_failures < max_retries

Queue Routing Logic

The dispatcher routes to long queue when: (a) task type is forge_session, (b) task has is_complex=true flag, or (c) scheduled job config has expected_duration > 30min. All other work goes to standard. Background extraction tasks (extract_memories, index_document) go to quick.

LLMClient Abstraction

A Python Protocol interface that all LLM provider implementations conform to. The gateway and workers both use this abstraction — it lives in a shared package.

class LLMClient(Protocol):
    """Common interface all LLM providers implement."""

    async def create_message(
        self,
        messages: list[dict],
        system: str,
        tools: list[dict] | None = None,
        max_tokens: int = 4096,
        stream: bool = False,
        **kwargs
    ) -> LLMResponse: ...

    async def health_check(self) -> bool: ...

    def supports_feature(self, feature: str) -> bool: ...
    # features: "tools", "vision", "thinking", "streaming", "json_mode"

Provider Implementations

V1: Anthropic Only

V1 implements AnthropicClient only. The LLMClient Protocol and llm_provider/llm_model schema remain multi-provider-ready, but only one client implementation ships in V1. Additional providers (OpenAI, Bedrock, Vertex, Ollama, vLLM) are post-V1.

ClassProvider TypeSDK / TransportKey Notes
AnthropicClientCLOUD_APIanthropic SDKV1 provider. Supports tools, vision, thinking, streaming.
OpenAIClientCLOUD_APIopenai SDKPost-V1. GPT-4o, o1. Supports tools, vision, streaming.
OllamaClientLOCALHTTP to Ollama APINeeded for local dev (Ollama pod in k8s). ~100 lines. See docs/superpowers/specs/2026-04-16-local-development-design.md.
BedrockClientBEDROCKboto3 bedrock-runtimePost-V1. Claude via AWS. IAM role auth. Supports tools, streaming.
VertexClientVERTEXgoogle-cloud-aiplatformPost-V1. Gemini + Claude via GCP. Service account auth.
VLLMClientSELF_HOSTEDHTTP (OpenAI-compatible)Post-V1. Self-hosted models on EKS. No auth or bearer token.

Model Resolution at Hydration Time

The correct LLMClient is built during agent hydration by build_llm_client(provider, model). The factory reads llm_provider.provider_type to select the class, then configures it with the provider's api_endpoint, model's model_identifier, and credentials from Secrets Manager. If the provider is DOWN, the factory falls back to the agent's fallback_model_id.

Dependencies (pyproject.toml)

Runtime

fastapiWeb framework
uvicorn[standard]ASGI server (with uvloop)
httpxAsync HTTP client for qqq API + LLM fallback
anthropicAnthropic SDK (primary LLM)
openaiOpenAI SDK (Post-V1: embeddings)
celery[redis]Task queue producer (dispatch only)
redis[hiredis]Redis client (cache, pub/sub, rate limit)
pydanticRequest/response validation
pydantic-settingsEnv-driven configuration
pyjwt[crypto]Authentik JWT validation
prometheus-clientMetrics exposition
structlogStructured JSON logging
boto3AWS Secrets Manager + Bedrock

Dev / Test

pytestTest runner
pytest-asyncioAsync test support
pytest-covCoverage reporting
httpxASGI test client (TestClient)
respxMock httpx requests (qqq API mocking)
fakeredis[lua]In-memory Redis for tests
ruffLinting
blackFormatting
mypyType checking

Dockerfile (Multi-Stage Build)

# Stage 1: Build
FROM python:3.12-slim AS builder
WORKDIR /app
COPY pyproject.toml .
RUN pip install --no-cache-dir hatchling \
    && pip wheel --no-cache-dir --wheel-dir /wheels -e .

# Stage 2: Runtime
FROM python:3.12-slim
WORKDIR /app

RUN groupadd -r gateway && useradd -r -g gateway gateway
COPY --from=builder /wheels /wheels
RUN pip install --no-cache-dir /wheels/*.whl && rm -rf /wheels
COPY src/ ./src/

USER gateway
EXPOSE 8000
HEALTHCHECK CMD python -c "import httpx; httpx.get('http://localhost:8000/healthz').raise_for_status()"
CMD ["uvicorn", "gateway.main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "4"]

Prometheus Metrics

Exposed at /metrics in Prometheus text format. Scraped by Prometheus every 15s.

Metric NameTypeLabelsDescription
praesidium_http_requests_totalCountermethod, path, statusTotal HTTP requests by method, path, and status code
praesidium_http_request_duration_secondsHistogrammethod, pathRequest latency distribution
praesidium_active_websocket_connectionsGauge—Current open WebSocket connections
praesidium_active_sse_streamsGauge—Current open SSE chat streams
praesidium_work_items_queuedGaugepriorityPending work items by priority level
praesidium_work_items_processingGauge—Currently executing work items

Test Requirements

Tests use pytest + httpx ASGI TestClient. All external dependencies are mocked: qqq API (respx), Redis (fakeredis), Anthropic (mock responses). No real API calls in CI.

Mandatory Test Coverage

  • Every router endpoint: test 200 success, 401 unauthorized, 403 forbidden, 422 validation error
  • Chat endpoint: test SSE stream format, tool loop (1 tool call, multi-tool, max iteration break), token/cost tracking
  • Task dispatcher: test correct queue routing (quick/standard/long), pre-dispatch qqq API calls, Celery task creation
  • Agent hydration: test cache hit, cache miss, cache invalidation, provider fallback on DOWN
  • WebSocket: test connection, subscription, broadcast, disconnection cleanup
  • Auth middleware: test valid JWT, expired JWT, missing JWT, invalid signature, instance_id extraction
  • LLMClient: test each provider implementation with mock responses (streaming + non-streaming)
  • Rate limiter: test per-agent per-tool limits, bucket refill, 429 response
  • SSE reconnect: test Last-Event-ID replay of missed events
  • Error handling: test qqq API timeout/error graceful degradation, Anthropic rate limit retry + backoff

Acceptance Criteria

  • All 8+ API endpoints return correct responses with valid Authentik JWT
  • 1:1 chat streams tokens via SSE in real-time with <200ms time-to-first-token (after hydration)
  • Tool loop executes PLATFORM tools in-process and streams tool_call_start/tool_call_result SSE events
  • Agent hydration loads template + tools + memories in ≤500ms (cache miss) and ≤50ms (cache hit)
  • Task dispatch correctly routes to quick/standard/long Celery queues based on task type
  • WebSocket broadcasts reach all connected clients within 100ms of Redis pub/sub event
  • SSE stream supports Last-Event-ID reconnection without message loss
  • JWT authentication rejects missing/expired/invalid tokens with 401
  • Instance scoping (praesidiumInstanceId) is forwarded to every qqq API call
  • Per-agent per-tool rate limits enforced (configurable, default 10 calls/min/tool)
  • Prometheus metrics exposed at /metrics with all 6 custom metrics
  • Structured JSON logging with correlation IDs propagated to qqq + Celery
  • Docker image builds and runs correctly (multi-stage, non-root user)
  • All tests passing with 90%+ code coverage

CLAUDE.md Rules for Agent Gateway Agents

These rules MUST be placed in the CLAUDE.md file at the root of the praesidium2-python/services/agent-gateway/ directory.

# praesidium2-gateway CLAUDE.md

## Project Context
This is the Agent Gateway for Praesidium2 -- the bridge between the browser
and the AI agent runtime. It handles all AI-related HTTP traffic: chat SSE
streams, task dispatch, approval workflow, feedback, rollback/recovery,
admin controls, and WebSocket broadcasts.
V1.5: meeting orchestration, tool forge sessions.

## Architecture Rules
- This service NEVER touches the database directly -- all data via qqq REST API
- 1:1 chat runs INLINE in the gateway (no Celery round-trip) for low latency
- All other work (tasks, jobs) is DISPATCHED to Celery workers
- Meetings and forge sessions are V1.5 -- do not implement in V1
- Every endpoint requires Authentik JWT auth -- no unauthenticated routes except /healthz
- Forward praesidiumInstanceId from JWT to every qqq API call for RLS scoping
- Agent hydration results are cached in Redis (5min TTL) -- invalidate on config change
- SSE streams must support Last-Event-ID reconnection
- WebSocket broadcasts use Redis pub/sub for cross-pod delivery

## Coding Standards
- Python 3.12, async/await everywhere, type hints on all functions
- Pydantic models for ALL request/response shapes -- no raw dicts at API boundaries
- structlog for structured JSON logging -- no print() statements
- All errors return proper HTTP status codes with JSON error bodies
- Follow FastAPI dependency injection patterns (Depends())
- Use httpx.AsyncClient for all outbound HTTP (qqq API, LLM providers)

## v1.8 New Endpoints
- GET /api/internal/worker/hydrate/{agent_id} -- AGENT_MANIFEST delivery (Service Account JWT)
- POST /api/task/{id}/approve -- HITL approval signal (action hash verification + Redis resume)
- GET /api/search -- Command Palette backend (pgvector semantic search)

## Business Operations Endpoints (V1)
- Approval: GET /api/approvals, POST /api/approvals/{id}/approve|reject|request-changes
- Feedback: POST /api/feedback (submit), GET /api/feedback/agent/{id} (per-agent)
- Rollback: POST /api/actions/{id}/undo, POST /api/tasks/{id}/rollback
- Admin: POST /api/admin/emergency-stop, POST /api/admin/resume,
         GET /api/admin/kill-criteria, POST /api/admin/kill-criteria/evaluate

## LLM Client Rules
- All LLM calls go through the LLMClient Protocol -- never call SDKs directly
- build_llm_client() factory is the ONLY place that instantiates provider clients
- Provider health check failures must trigger fallback to secondary model
- Every LLM call must track input_tokens, output_tokens, cost_usd

## Testing Rules
- Mock ALL external dependencies: qqq API (respx), Redis (fakeredis), LLM SDKs
- No real API calls in tests -- ever
- Every endpoint needs tests for 200, 401, 403, 422
- Test SSE stream content by consuming the async generator
- Target 90%+ test coverage

## What NOT to Do
- Do NOT add database connections or ORM models -- this is not a data layer
- Do NOT process long-running tasks in the gateway -- dispatch to Celery workers
- Do NOT bypass JWT authentication for any route except health checks
- Do NOT hardcode API keys, model names, or URLs -- use Pydantic Settings from env
- Do NOT use synchronous blocking calls -- everything must be async
- Do NOT skip input validation -- use Pydantic models with JSON Schema constraints
- Do NOT implement meetings or forge endpoints -- those are V1.5

Shared Python Packages

The Gateway and Worker both need the llm/ client and hydration/ modules. These live in a shared/ directory in the monorepo and are installed as editable packages in Docker builds.

Monorepo Structure

praesidium2-python/ docs/ adr/ # Architecture Decision Records (v1.10) shared/ llm/ __init__.py protocol.py # LLMClient Protocol interface anthropic_client.py # V1 implementation factory.py # build_llm_client() by provider type pyproject.toml # name = "praesidium-llm" hydration/ __init__.py hydrator.py # Agent hydration: qqq fetch + Redis cache models.py # HydratedAgent, HydrationContext pyproject.toml # name = "praesidium-hydration" services/ agent-gateway/ # imports from praesidium-llm, praesidium-hydration agent-worker/ # imports from praesidium-llm, praesidium-hydration

Docker Build Integration

# In Gateway and Worker Dockerfiles:
COPY shared/ /app/shared/
RUN pip install -e /app/shared/llm \
    && pip install -e /app/shared/hydration

# Both services import identically:
from praesidium_llm import build_llm_client
from praesidium_hydration import hydrate_agent

Post-v1 Optimization

After v1, shared packages can be published to a private PyPI registry for versioned releases. During v1, editable installs from the monorepo shared/ directory keep iteration fast.

Agent Workers

Epic 4 — Phase 3

The AI execution engine — where agents actually do the work.

Agent Workers are Python/Celery processes that consume tasks from Redis queues and execute them using direct Anthropic SDK agent loops (standard task execution) and LangGraph (durable state machine workflows needing crash recovery or HITL approval gates). V1 scope: task execution, scheduled jobs, HITL interrupt/resume protocol, approval queue processing, and integration tool dispatch. LangGraph provides PostgreSQL-backed checkpointing (langgraph_checkpoints table) so long-running workflows survive worker crashes and multi-hour HITL waits. The interrupt() function serializes graph state before requesting human approval; the Resume Protocol re-hydrates from the checkpoint when approval arrives via Redis event. All spans carry ddtrace trace IDs for unified Datadog observability. V1.5 additions: meetings, tool forge builds, memory extraction, and document indexing (specs retained below).

Repository Setup

Repositorypraesidium2-python/services/agent-worker/
LanguagePython 3.12
Build Toolpyproject.toml (hatchling)
FrameworkCelery 5.x + Anthropic SDK + LangGraph (durable/HITL)
BrokerRedis (consumer + producer)
CheckpointsPostgreSQL (LangGraph PostgresSaver)
Replicas3-10 pods, HPA on Redis queue depth
Resources2 CPU / 4Gi per pod
Concurrency4 worker threads per pod
praesidium2-python/services/agent-worker/ src/worker/ __init__.py app.py # Celery app factory + config config.py # Pydantic Settings (env-driven) tasks/ __init__.py execute_task.py # Direct SDK agent loop: task execution run_meeting.py # Turn-taking orchestrator: standup/huddle execute_job.py # LangGraph: scheduled job workflow forge_session.py # LangGraph: tool forge pipeline extract_memories.py # Claude-powered memory extraction index_document.py # Knowledge doc chunking + embedding reconcile_stats.py # Denormalized counter reconciliation agent_exec/ __init__.py agent_builder.py # HydratedAgent to Anthropic SDK context loop_runner.py # Agentic loop: prompt → tool calls → result backstory_builder.py # Template + soul + skills + memories callbacks.py # Step/token callbacks for streaming langgraph/ __init__.py checkpointer.py # PostgresSaver setup + pruning job_graph.py # Scheduled job state graph forge_graph.py # Tool forge state graph (6 states) project_graph.py # Multi-step project work graph states.py # TypedDict state definitions tools/ __init__.py registry.py # Tool registry: load + resolve + dispatch base.py # BaseTool, ToolContext, ToolResult platform/ __init__.py create_task.py # qqq API: POST /task update_task_status.py # qqq API: PATCH /task/{id} list_projects.py # qqq API: GET /project store_memory.py # qqq API: POST /memory create_escalation.py # qqq API: POST /escalation schedule_job.py # qqq API: POST /scheduled_job integrations/ __init__.py jira.py # Jira CRUD + search + sprint confluence.py # V1.1: Confluence page CRUD + search slack.py # Slack DM, channel post, buttons github.py # GitHub PR, issues, code search google.py # V1.1: Google Calendar, Docs, Sheets zoho.py # V1.1: Zoho CRM lookup + create custom_dispatch.py # CUSTOM tool dispatch to tool-runner services/ __init__.py hydration.py # Shared agent hydration (same as gateway) memory_extractor.py # Claude-powered memory analysis embedding.py # text-embedding-3-small via OpenAI/Bedrock secret_resolver.py # Secrets Manager fetch + scope resolution clients/ __init__.py qqq.py # Typed httpx client for qqq REST API redis.py # Redis pub/sub for WS broadcast events llm/ # Shared with gateway (symlink or package) protocol.py anthropic_client.py factory.py beat/ __init__.py scheduler.py # Celery Beat custom scheduler leader_election.py # Redis-based leader election tests/ conftest.py # Fixtures: mock Celery, mock qqq, mock Redis test_tasks/ test_execute_task.py test_run_meeting.py test_execute_job.py test_forge_session.py test_extract_memories.py test_index_document.py test_agent_exec/ test_agent_builder.py test_loop_runner.py test_langgraph/ test_job_graph.py test_forge_graph.py test_tools/ test_platform_tools.py test_registry.py test_custom_dispatch.py pyproject.toml Dockerfile CLAUDE.md

Day-by-Day Tasks (Worker Track)

Worker development starts on Day 5 (Phase 1) after the gateway can dispatch. Core execution patterns are built in Phase 2, with advanced features (forge, multi-LLM, memory) in Phases 2-3.

DayTaskExit Criteria
D5 Celery app factory, Redis broker config, 3 queues (quick/standard/long), execute_task skeleton. Platform tool stubs. Dockerfile. Worker starts, consumes from Redis, executes a no-op task, acknowledges.
D6 Agent execution: agent_builder.py (HydratedAgent → Anthropic SDK context), loop_runner.py (agentic loop). execute_task with direct Anthropic SDK execution. Task execution with direct SDK agent loop produces output. Messages streamed via Redis pub/sub.
D7 LangGraph setup: PostgresSaver, job_graph.py (load → execute → evaluate). execute_job task. Celery Beat scheduler with leader election. V1.5 run_meeting multi-agent standup skeleton (deferred). Scheduled jobs execute on cron. Meeting orchestration deferred to V1.5 (V1 uses Slack-based status).
D8 V1.5 Memory extractor: Claude-powered analysis, embedding generation (text-embedding-3-small), deduplication (cosine similarity > 0.92), store via qqq API. extract_memories task. index_document task (chunking + embedding). Deferred to V1.5. After task/meeting completion, memories extracted and stored. Documents chunked and embedded.
D9 V1.5 Knowledge document upload + chunking + embedding pipeline. Memory recall during agent hydration (semantic search + recency bias). Tool execution framework: PLATFORM + INTEGRATION in-process, risk-level safety gates. Memory/RAG deferred to V1.5. Tool execution framework and risk-level safety gates are V1 scope.
D10 Integration tools: Jira, Slack, GitHub connectors as in-process tools (V1 scope). Secret resolution (scope hierarchy: AGENT > TOOL > PROJECT > INSTANCE, narrowest scope wins). Confluence, Google, Zoho deferred to V1.1. 3 core integration tool sets working (Slack, Jira, GitHub). Secrets resolved from correct scope. V1.1 integrations stubbed.
D11-12 V1.5 LangGraph tool forge workflow (forge_graph.py): 6 states (DESIGN → SCAFFOLD → IMPLEMENT → TEST → REVIEW → PUBLISH). Human-in-the-loop interrupt at REVIEW. Git integration for tool repos. Deferred to V1.5. Full forge workflow produces a tool version from human description, with human review gate.
D13-14 (Phase 3) Multi-LLM: provider health checks, model fallback. Stats reconciliation job. Checkpoint pruning (90-day retention). Celery Beat health checks + memory decay. Provider failover works. Stats reconciled every 6h. Old checkpoints pruned. System jobs running.

Celery Configuration

Three task-type-specific queues with tuned visibility timeouts to prevent duplicate execution of long-running tasks.

QueueVisibility TimeoutTask TypesAcks LateRationale
quick 300s (5 min) extract_memories, index_document, reconcile_stats, chat-related background work No Fast tasks; if they fail, just retry. No checkpointing needed.
standard 3600s (1 hour) execute_task, run_meeting, execute_job (simple) Yes Medium-duration tasks. acks_late ensures re-queue on worker crash.
long 7200s (2 hours) forge_session, complex multi-step LangGraph workflows, project planning Yes Long-running workflows with checkpointing. Resumes from last checkpoint on re-delivery.

Celery App Configuration

# worker/app.py
from celery import Celery

app = Celery("praesidium-worker")

app.conf.update(
    broker_url="redis://redis:6379/0",
    result_backend="redis://redis:6379/1",

    # Queue definitions
    task_queues={
        "quick":    {"exchange": "quick",    "routing_key": "quick"},
        "standard": {"exchange": "standard", "routing_key": "standard"},
        "long":     {"exchange": "long",     "routing_key": "long"},
    },
    task_default_queue="standard",

    # Visibility timeouts per queue
    broker_transport_options={
        "queue_order_strategy": "priority",
        "visibility_timeout": {
            "quick": 300,
            "standard": 3600,
            "long": 7200,
        },
    },

    # Worker behavior
    worker_concurrency=4,
    worker_prefetch_multiplier=1,     # Fetch one task at a time (for acks_late)
    task_acks_late=False,             # Default; overridden per task

    # Serialization
    task_serializer="json",
    result_serializer="json",
    accept_content=["json"],

    # Time limits
    task_soft_time_limit=3600,        # Default soft limit (1 hour)
    task_time_limit=7500,             # Default hard limit (2h5m)
)

Idempotency Guards

LangGraph tasks check for existing checkpoints before re-executing a node. If a task is re-delivered due to a visibility timeout edge case, it resumes from the last checkpoint rather than restarting. Direct SDK agent loop tasks (no checkpointing) will restart from scratch on re-delivery — they are designed to be idempotent (creating duplicate records is prevented by qqq's unique constraints).

Celery Task Definitions

Every task follows the same lifecycle: hydrate agent(s) → execute work → persist results to qqq API → broadcast status via Redis pub/sub → queue post-processing (memory extraction).

TaskQueueFrameworkDescription
execute_task standard Anthropic SDK Call Anthropic SDK with system prompt, tools, and conversation context for the assigned agent + optional collaborators. Run an agentic tool loop until completion. Persist results (task_comment, messages, status changes) to qqq API.
run_meeting standard Anthropic SDK Multi-agent standup/huddle. Load all conversation_participants and build SDK context for each. Turn-taking orchestrator runs the agenda (status updates, blockers, action items). Produces meeting summary and creates task records for action items.
execute_job standard or long LangGraph Scheduled job execution via LangGraph state graph (LOAD → EXECUTE → EVALUATE → REPORT). Checkpointed at every node. Updates job_execution record on completion/failure.
forge_session long LangGraph Interactive tool forge workflow (6 states: DESIGN → SCAFFOLD → IMPLEMENT → TEST → REVIEW → PUBLISH). Human-in-the-loop interrupt at REVIEW state. Commits code to tool_repo git.
extract_memories quick Direct LLM Post-conversation memory extraction using Claude. Analyzes the conversation for facts, decisions, lessons learned. Generates embeddings and deduplicates against existing memories.
index_document quick Direct Knowledge document chunking (1000 token chunks, 200 token overlap) + embedding generation (text-embedding-3-small). Stores knowledge_chunk records via qqq API.
reconcile_stats quick Direct Recompute denormalized counters on agent records (total_tasks_completed, total_tokens_used, total_cost_usd) from source tables. Runs every 6 hours via Celery Beat.

execute_task — Direct SDK Agent Loop

The primary execution pattern for task work. Calls the Anthropic SDK with the hydrated agent context and runs an agentic tool loop until completion.

@app.task(bind=True, queue="standard", acks_late=True)
def execute_task(self, task_id: str, conversation_id: str, **kwargs):
    """Execute a task using direct Anthropic SDK agent loop."""
    # 1. Load task from qqq API
    task = qqq.get(f"/task/{task_id}?include=project,milestone")
    qqq.patch(f"/task/{task_id}", {"status_id": STATUS_IN_PROGRESS})

    # 2. Hydrate assigned agent (+ collaborators if any)
    agent = hydrate_agent(task.assignee_id)
    agent_context = build_agent_context(agent)
    collaborators = [
        build_agent_context(hydrate_agent(cid))
        for cid in kwargs.get("collaborator_ids", [])
    ]

    # 3. Build messages from task context
    messages = build_task_messages(task, agent)

    # 4. Run agentic loop (prompt → tool calls → result)
    result = run_agent_loop(
        client=llm_client,
        system=agent_context.system_prompt,
        messages=messages,
        tools=agent_context.tools,
        step_callback=stream_to_redis(conversation_id),
    )

    # 5. Persist results
    qqq.post("/task_comment", {
        "task_id": task_id,
        "content": result.output,
        "author_id": agent.team_member_id,
    })
    qqq.patch(f"/task/{task_id}", {"status_id": STATUS_DONE})

    # 6. Broadcast completion + queue memory extraction
    redis.publish(f"task:{task_id}", {"event": "status_changed", "status": "DONE"})
    extract_memories.delay(conversation_id=conversation_id, agent_id=agent.id)

run_meeting — Turn-Taking Orchestrator

Deferred to V1.5
Multi-agent meeting orchestration (standups, huddles, retros) is deferred to V1.5. V1 uses Slack-based status updates for team coordination. Implementation spec retained below for V1.5 development.

Meetings use a turn-taking orchestrator with a designated facilitator. Each participant contributes based on their role and current work context via sequential Anthropic SDK calls.

@app.task(bind=True, queue="standard", acks_late=True)
def run_meeting(self, meeting_id: str, conversation_id: str, **kwargs):
    """Run a multi-agent meeting (standup, huddle, retro)."""
    # 1. Load meeting + participants
    meeting = qqq.get(f"/scheduled_meeting/{meeting_id}")
    participants = qqq.get(f"/conversation_participant?conversationId={conversation_id}")

    # 2. Hydrate all participating agents
    agent_contexts = []
    for p in participants:
        hydrated = hydrate_agent(p.team_member_id)
        ctx = build_agent_context(hydrated, role_override=p.role)
        agent_contexts.append(ctx)

    # 3. Build meeting-specific prompts
    # Facilitator: run agenda, collect updates, identify blockers
    # Participants: share status, flag blockers, propose actions
    agenda_prompts = build_meeting_prompts(meeting.type, agent_contexts, meeting.agenda)

    # 4. Run turn-taking orchestrator (each agent speaks in sequence)
    result = run_turn_taking_loop(
        client=llm_client,
        agent_contexts=agent_contexts,
        agenda_prompts=agenda_prompts,
        step_callback=stream_to_redis(conversation_id),
    )

    # 5. Extract action items as task records
    action_items = extract_action_items(result.raw)
    for item in action_items:
        qqq.post("/task", {
            "title": item.title,
            "project_id": meeting.project_id,
            "assignee_id": item.assignee_id,
            "status_id": STATUS_OPEN,
        })

    # 6. Store meeting summary
    qqq.patch(f"/conversation/{conversation_id}", {
        "summary": result.raw,
        "status_id": CONV_STATUS_COMPLETED,
    })

LangGraph Integration

LangGraph is used for any work that could take >30 seconds, needs human-in-the-loop approval, must survive worker crashes, or has branching/retry logic. State is checkpointed to PostgreSQL after every graph node.

Scheduled Job Graph (execute_job)

# langgraph/job_graph.py
class JobState(TypedDict):
    job_id: str
    agent_id: str
    context: dict
    result: str | None
    status: str

def build_job_graph():
    graph = StateGraph(JobState)

    graph.add_node("load", load_job_context)
    graph.add_node("execute", execute_with_agent)
    graph.add_node("evaluate", evaluate_result)
    graph.add_node("report", report_and_persist)

    graph.add_edge("load", "execute")
    graph.add_edge("execute", "evaluate")
    graph.add_conditional_edges(
        "evaluate",
        should_retry,
        {"retry": "execute", "done": "report"}
    )
    graph.set_entry_point("load")
    graph.set_finish_point("report")

    return graph.compile(
        checkpointer=PostgresSaver(conn)
    )

Tool Forge Graph (forge_session)

Deferred to V1.5
Tool Forge workflow deferred to V1.5. Spec retained for future implementation.
# langgraph/forge_graph.py
class ForgeState(TypedDict):
    conversation_id: str
    tool_spec: dict | None
    code: str | None
    test_results: dict | None
    review_decision: str | None
    tool_version_id: str | None
    status: str
    retry_count: int

def build_forge_graph():
    graph = StateGraph(ForgeState)

    graph.add_node("design", design_tool)
    graph.add_node("scaffold", scaffold_project)
    graph.add_node("implement", implement_tool)
    graph.add_node("test", run_tests)
    graph.add_node("review", human_review)
    graph.add_node("publish", publish_tool)

    graph.add_edge("design", "scaffold")
    graph.add_edge("scaffold", "implement")
    graph.add_edge("implement", "test")
    graph.add_conditional_edges(
        "test", check_test_results,
        {"pass": "review", "fail": "implement"}
    )
    # INTERRUPT at review for human
    graph.add_conditional_edges(
        "review", check_review_decision,
        {"approve": "publish",
         "changes": "implement",
         "reject": END}
    )
    graph.set_entry_point("design")

    return graph.compile(
        checkpointer=PostgresSaver(conn),
        interrupt_before=["review"]
    )

PostgreSQL Checkpointing

LangGraph's PostgresSaver writes checkpoint state to PostgreSQL after every graph node transition. Thread IDs map to our conversation.id. On worker crash, the task is re-delivered by Celery (acks_late=True) and LangGraph resumes from the last checkpoint. Checkpoint pruning runs every 24 hours, deleting checkpoints older than 90 days after workflow completion.

HITL Interrupt + Resume Protocol

When a CRITICAL tool call is encountered, the worker pauses execution and waits for human approval. State is fully serialized to survive multi-hour (or multi-day) waits. See HITL_ORCHESTRATION_SPEC.

Interrupt Flow (Worker → Pause)

# Inside the LangGraph node for tool execution:
def execute_tool(state: TaskState):
    tool = resolve_tool(state["tool_call"])

    if tool.risk_level == "CRITICAL":
        # 1. Write task_approval record with SHA-256 hash
        approval = qqq.post("/task_approval", {
            "task_id": state["task_id"],
            "agent_id": state["agent_id"],
            "action_type": tool.name,
            "action_payload": state["tool_call"]["input"],
            "action_payload_hash": sha256(json.dumps(state["tool_call"]["input"])),
            "expires_at": now() + timedelta(hours=24),
        })

        # 2. Update task status to AWAITING_APPROVAL
        qqq.patch(f"/task/{state['task_id']}", {"status_id": STATUS_AWAITING_APPROVAL})

        # 3. Notify human via Slack DM
        notify_approval_needed(approval)

        # 4. LangGraph interrupt() serializes state to langgraph_checkpoints
        return interrupt({"approval_id": approval["id"]})

Resume Flow (Approval → Wake-Up)

# Triggered by Redis event from POST /api/task/{id}/approve:
def handle_resume_event(thread_id: str, approval_id: str):
    # 1. Load approval decision
    approval = qqq.get(f"/task_approval/{approval_id}")
    assert approval["status"] == "APPROVED"

    # 2. Re-hydrate graph state from langgraph_checkpoints
    graph = build_task_graph(checkpointer=PostgresSaver(conn))
    config = {"configurable": {"thread_id": thread_id}}

    # 3. Resume from exact node where interrupted
    result = graph.invoke(Command(resume=approval), config)
    # Execution continues with the approved tool call

Tamper-Proof Security

  • The UI sends a hash of the payload it is displaying when the human clicks “Approve”
  • Gateway compares this hash to action_payload_hash in the database
  • If hashes don’t match (agent state changed in the interim), resume is rejected with “Payload Drift Detected” error
  • A “Reject” action loops the agent back to the reasoning node with the human’s feedback text
  • Total end-to-end resume time target: <5 seconds (from UI click to Tool start)

Datadog Trace ID Propagation

Trace IDs must propagate through the full 5-service waterfall for unified observability. See DATADOG_INTEGRATION_SPEC.

# In gateway/celery_utils.py -- inject trace into Celery task headers:
from ddtrace import tracer
dd_headers = {
    "x-datadog-trace-id": str(tracer.current_span().trace_id),
    "x-datadog-parent-id": str(tracer.current_span().span_id),
}
celery_task.apply_async(args=[...], headers=dd_headers)

# In worker/tasks/base.py -- extract trace from Celery header:
span = tracer.trace("agent.reasoning_loop", service="agent-worker")
span.set_tag("agent_id", agent_id)
span.set_tag("task_id", task_id)
# ... LLM provider latency and token usage captured as span metadata

# In tool runner Job spec -- pass as env vars:
env:
  - name: DD_TRACE_ID
    value: "${TRACE_ID}"
  - name: DD_PARENT_ID
    value: "${PARENT_SPAN_ID}"

Tool Registry

The tool registry loads, resolves, and dispatches tools based on their category. PLATFORM and INTEGRATION tools run in-process; CUSTOM tools are dispatched to the tool-runner service.

CategoryExecutionExamplesSecurity
PLATFORM In-process (worker) create_task, update_task_status, list_projects, store_memory, create_escalation, schedule_job Trusted first-party code. Calls qqq REST API directly.
INTEGRATION In-process (worker) jira_create_issue, confluence_create_page, slack_send_message, github_create_pr, google_create_doc Connector classes loaded at startup. Credentials from secret_ref via Secrets Manager.
CUSTOM Sandboxed k8s Job/pod pull_campaign_metrics, check_ssl_expiry, parse_invoice_pdf Network-isolated. Resource limits by risk_level. Zero access to qqq API or internal services.
MCP Sidecar pod or external Playwright, SQL, filesystem, custom internal APIs Defaults to HIGH risk. Admin must explicitly downgrade after review.

Tool Interface Contract

class BaseTool:
    name: str
    description: str
    input_schema: dict          # JSON Schema for validation

    async def execute(self, input: dict, context: ToolContext) -> ToolResult:
        raise NotImplementedError

class ToolContext:
    agent_id: str
    project_id: str | None
    conversation_id: str | None
    qqq_client: QqqApiClient    # typed qqq REST client
    credentials: dict           # from integration_config via Secrets Manager

class ToolResult:
    success: bool
    data: Any = None
    error: str | None = None
    metadata: dict = {}

Risk-Level Safety Gates

Risk LevelBehaviorExample Tools
LOWExecute immediately (read operations)list_projects, jira_search, google_get_doc
MEDIUMExecute + log to activity_logcreate_task, confluence_create_page, slack_send_message
HIGHExecute + log + notify human via Slackjira_create_issue, github_create_pr, CUSTOM tools
CRITICALCreate human_interaction, WAIT for approval before executingdelete_project, financial transactions, MCP tools (default)

Celery Beat Scheduler

A single Celery Beat instance (via Redis-based leader election) polls the database for scheduled work and dispatches it as Celery tasks.

Leader Election + Scheduler HA v1.9

Only one Beat instance should run across all worker pods. Redis SET NX EX is used for leader election with a 30-second TTL. The leader renews the lock every 10 seconds. If a leader dies, another pod acquires the lock within 30 seconds.

Scheduler HA (v1.9): The active leader writes to the scheduler_heartbeat table every 30s. Secondary nodes poll this table; if now() - last_heartbeat > 60s, the secondary promotes itself to leader. This database-backed heartbeat complements the Redis lock and prevents split-brain or dead-lock scenarios when Redis is briefly unavailable.

Scheduled Jobs

ScheduleAction
Every 60sPoll scheduled_job table for due jobs → dispatch execute_job
Every 60sPoll scheduled_meeting table for due meetings → dispatch run_meeting
Every 6hDispatch reconcile_stats for all agents
Every 24hLangGraph checkpoint pruning (delete > 90 days after completion)
Every 2mLLM provider health checks (update llm_provider.health_status)
Every 5mIntegration health checks (update integration_config.health_status)
Every 30sScheduler heartbeat: leader writes to scheduler_heartbeat table (v1.9 HA)
WeeklyMemory decay: reduce importance of old memories that haven't been recalled

Memory Extractor

Deferred to V1.5
Memory/RAG pipeline (extraction, embedding, semantic recall, knowledge document indexing) is deferred to V1.5. V1 agents use conversation-context only — no persistent memory across conversations. Implementation spec retained below for V1.5 development.

Runs as a background task after every conversation. Uses Claude to analyze the conversation and extract persistent memories.

@app.task(queue="quick")
def extract_memories(conversation_id: str, agent_id: str):
    """Extract and store memories from a completed conversation."""
    # 1. Load full conversation
    conv = qqq.get(f"/conversation/{conversation_id}/full")

    # 2. Ask Claude to extract memories
    prompt = f"""Analyze this conversation and extract:
    - FACTS: concrete information learned
    - DECISIONS: choices made and their rationale
    - LESSONS_LEARNED: what worked, what didn't
    - PREFERENCES: user/team preferences discovered
    - PROCEDURES: processes or workflows identified

    For each memory, provide: type, content, importance (0.0-1.0)."""

    memories = llm_client.create_message(
        messages=[{"role": "user", "content": format_conversation(conv)}],
        system=prompt,
    )

    # 3. For each extracted memory:
    for mem in parse_memories(memories):
        # Generate embedding
        embedding = embed(mem.content)    # text-embedding-3-small

        # Deduplicate: check cosine similarity against existing memories
        similar = qqq.get(f"/memory?agentId={agent_id}&vector_search={embedding}&threshold=0.92")
        if similar:
            # Update existing memory (merge content, bump importance)
            qqq.patch(f"/memory/{similar[0].id}", {
                "content": merge_content(similar[0].content, mem.content),
                "importance": max(similar[0].importance, mem.importance),
                "recall_count": similar[0].recall_count + 1,
            })
        else:
            # Create new memory
            qqq.post("/memory", {
                "agent_id": agent_id,
                "type_id": mem.type_id,
                "content": mem.content,
                "importance": mem.importance,
                "embedding": embedding,
                "source_id": MEMORY_SOURCE_CONVERSATION,
                "source_conversation_id": conversation_id,
            })

Dependencies (pyproject.toml)

Runtime

celery[redis]Task queue (consumer + producer)
redis[hiredis]Redis client (broker, pub/sub, cache)
anthropicAnthropic SDK (V1 LLM + direct agent execution)
langgraphDurable state machine workflows (HITL/crash recovery)
langgraph-checkpoint-postgresPostgreSQL checkpointing
openaiOpenAI SDK + embeddings (Post-V1)
httpxAsync HTTP client for qqq API
pydanticData validation
pydantic-settingsEnv-driven configuration
numpyVector operations (cosine similarity)
prometheus-clientMetrics exposition
structlogStructured JSON logging
boto3AWS Secrets Manager + Bedrock
psycopg[binary]PostgreSQL driver (LangGraph checkpoints)

Dev / Test

pytestTest runner
pytest-asyncioAsync test support
pytest-covCoverage reporting
pytest-celeryCelery test fixtures
respxMock httpx requests (qqq API mocking)
fakeredis[lua]In-memory Redis for tests
testcontainers[postgres]PostgreSQL for LangGraph checkpoint tests
ruffLinting
blackFormatting
mypyType checking

Dockerfile (Multi-Stage Build)

# Stage 1: Build
FROM python:3.12-slim AS builder
WORKDIR /app
COPY pyproject.toml .
RUN pip install --no-cache-dir hatchling \
    && pip wheel --no-cache-dir --wheel-dir /wheels -e .

# Stage 2: Runtime
FROM python:3.12-slim
WORKDIR /app

RUN apt-get update && apt-get install -y --no-install-recommends \
        libpq5 \
    && rm -rf /var/lib/apt/lists/*
RUN groupadd -r worker && useradd -r -g worker worker
COPY --from=builder /wheels /wheels
RUN pip install --no-cache-dir /wheels/*.whl && rm -rf /wheels
COPY src/ ./src/

USER worker

# Health check via Celery inspect ping
HEALTHCHECK CMD celery -A worker.app inspect ping --timeout 5

# Default: run worker consuming all 3 queues
CMD ["celery", "-A", "worker.app", "worker", \
     "--queues=quick,standard,long", \
     "--concurrency=4", \
     "--loglevel=info"]

Celery Beat Sidecar

Celery Beat runs as a separate container in the same pod (or a dedicated single-replica Deployment). Command: celery -A worker.app beat --loglevel=info. Leader election ensures only one Beat instance dispatches scheduled work, even if multiple Beat containers start.

Prometheus Metrics

Exposed at /metrics via a sidecar HTTP server (since Celery workers don't serve HTTP). Scraped by Prometheus every 15s.

Metric NameTypeLabelsDescription
praesidium_agent_executions_totalCounteragent_id, statusTotal agent executions by agent and result (success/failure)
praesidium_agent_execution_duration_secondsHistogramtask_typeExecution time distribution by task type
praesidium_llm_calls_totalCountermodel, agent_idTotal LLM API calls by model and agent
praesidium_llm_tokens_totalCounterdirection, modelToken usage by direction (input/output) and model
praesidium_tool_calls_totalCountertool_name, risk_levelTool invocations by tool name and risk level
praesidium_celery_task_queue_lengthGaugequeueCurrent queue depth per queue name (quick/standard/long)

Test Requirements

Tests use pytest + pytest-celery for task testing. qqq API and LLM SDKs are mocked. LangGraph checkpoint tests use testcontainers (real PostgreSQL).

Mandatory Test Coverage

  • execute_task: test agent context assembly, agentic loop execution, task status transitions (QUEUED → IN_PROGRESS → DONE), result persistence, memory extraction queued
  • run_meeting: test multi-agent assembly, agenda-driven execution, action item extraction, meeting summary storage
  • execute_job: test LangGraph state transitions, checkpoint persistence, retry logic, job_execution status updates
  • forge_session: test all 6 forge states, human-in-the-loop interrupt at REVIEW, git commit on PUBLISH, test failure retry loop (max 5)
  • extract_memories: test memory extraction from mock conversation, embedding generation, deduplication (cosine > 0.92 = merge)
  • index_document: test chunking (1000 token chunks, 200 overlap), embedding storage, knowledge_chunk record creation
  • Tool registry: test PLATFORM tool execution, INTEGRATION tool with mock credentials, CUSTOM dispatch to tool-runner
  • Risk-level gates: test LOW (execute), MEDIUM (execute + log), HIGH (execute + notify), CRITICAL (wait for approval)
  • Celery Beat: test scheduler polls correct tables, dispatches correct task types, leader election works, scheduler_heartbeat failover within 60s
  • Failure recovery: test acks_late re-queue, LangGraph checkpoint resume, Anthropic rate limit backoff

Skill-Aware Execution v1.12

When an agent executes any task (chat, scheduled job, or explicit task), the system determines which skills are relevant and injects them into the agent context. Two routing modes plus a context budget system ensure efficient use of the model’s context window.

Skill Routing Modes

ModeTriggerBehavior
Explicit Task specifies skill_id via task.metadata.skill_id or scheduled_job.tools JSONB Only that skill’s instructions are loaded. Targeted, efficient.
Semantic No skill specified (default for chat and most task execution) ALL active skills’ instructions loaded during hydration. The LLM decides which skill(s) apply based on task description and skill instructions.

Skill Injection Order & Context Budget

StepRule
1 Skills sorted by agent_skill.assigned_at ASC (oldest first = foundational skills first)
2 Each skill’s instructions and guardrails injected as separate labeled sections
3 If combined instructions exceed context budget, truncate starting from most recently assigned (newest first = least foundational)
4 Skill-level guardrails aggregated into a single “Guardrails” section after all instructions
40%
Skills
30%
Conversation
20%
Soul / Role
10%
System

Tool Requirement Validation at Hydration

During agent hydration, for each assigned skill the system validates tool availability.

StepAction
1 Load the pinned skill_version (or current_version if not pinned)
2 Check all skill_tool_requirement records for that version
3 For each required tool (is_required = true): verify agent has access via agent_tool_assignment or agent_mcp_assignment (MCP servers expose tools dynamically)
4 If required tool missing: skill is still injected but instructions prefixed with “NOTE: Tool {tool_name} not available — adapt or escalate.” Warning logged to activity_log with action skill.unmet_requirement.
5 Optional tools (is_required = false): no validation, no warning

Proficiency Tracking (Post-Execution)

After task completion, the worker updates proficiency records for each skill whose tools were invoked during execution.

FieldUpdate Logic
agent_skill_proficiency.times_usedIncrement +1
agent_skill_proficiency.last_used_atSet to now()
agent_skill_proficiency.proficiency0.00–1.00 score, improves with successful use. Records persist across skill unassignment/reassignment.

Conflict Resolution

There is no automatic conflict detection between skills. The agent’s soul (personality/identity) serves as the tiebreaker when skills give contradictory guidance. If systematic conflicts emerge, the Super Operator should adjust skill instructions or remove one of the conflicting skills.

Acceptance Criteria

  • All 7 Celery task types execute successfully and persist results to qqq API
  • Direct SDK agent loop works for single-agent (task) and turn-taking orchestrator (meeting) patterns
  • LangGraph state graphs execute with PostgreSQL checkpointing; workflows survive worker restart
  • Tool forge workflow completes all 6 states including human-in-the-loop review interrupt
  • 3 queues (quick/standard/long) route correctly with proper visibility timeouts and acks_late behavior
  • Memory extraction produces correct memory types with embeddings and deduplication
  • Document indexing chunks, embeds, and stores knowledge_chunk records correctly
  • Tool registry resolves all 4 categories (PLATFORM, INTEGRATION, CUSTOM, MCP) and enforces risk-level gates
  • Skill-aware execution: explicit skill routing (task.metadata.skill_id) and semantic matching (all skills loaded) both work correctly
  • Skill injection follows assigned_at ASC ordering with 40/30/20/10 context budget; truncation removes newest skills first
  • Tool requirement validation at hydration: missing required tools produce warnings and prefixed instructions, not failures
  • Proficiency tracking: agent_skill_proficiency.times_used and last_used_at updated after each skill execution
  • Celery Beat scheduler triggers scheduled jobs and meetings on time (>99% on-time rate)
  • Stats reconciliation corrects drifted counters to match source table aggregates
  • Secret resolution follows scope hierarchy (AGENT > TOOL > PROJECT > INSTANCE, narrowest scope wins)
  • Prometheus metrics exposed with all 6 custom worker metrics
  • Structured JSON logging with correlation IDs propagated from gateway via Celery kwargs
  • Docker image builds and runs correctly (multi-stage, non-root user, all 3 queues)
  • All tests passing with 90%+ code coverage

CLAUDE.md Rules for Agent Worker Agents

These rules MUST be placed in the CLAUDE.md file at the root of the praesidium2-python/services/agent-worker/ directory.

# praesidium2-worker CLAUDE.md

## Project Context
This is the Agent Worker service for Praesidium2 -- the AI execution engine.
Celery workers consume tasks from Redis queues and execute them using direct
Anthropic SDK agent loops (standard execution) and LangGraph (durable workflows
needing crash recovery or HITL approval gates). This is where agents DO the work.

## Architecture Rules
- This service NEVER touches the database directly -- all data via qqq REST API
- The ONLY exception is LangGraph's PostgresSaver which writes checkpoint state
  to its own managed tables in PostgreSQL (langgraph_checkpoints)
- Direct Anthropic SDK is used for "WHO does WHAT" -- single-agent tasks (V1), turn-taking meetings (V1.5)
- LangGraph is used for "HOW complex work flows" -- jobs (V1), forge + multi-step projects (V1.5)
- Meetings and forge tasks are V1.5 -- do not implement in V1
- All task results must be persisted to qqq API before acknowledging the Celery task
- Broadcast status changes via Redis pub/sub so the gateway can push to browsers

## HITL Interrupt/Resume Protocol (v1.8)
- CRITICAL tool calls trigger interrupt() -- state serialized to langgraph_checkpoints
- Write task_approval record with SHA-256 action_payload_hash
- Update task status to AWAITING_APPROVAL
- On resume: pick up Redis event, re-hydrate from checkpoint using thread_id
- Tamper-proof: gateway verifies action hash before dispatching resume

## Observability (v1.8)
- All Celery tasks propagate DD trace IDs from gateway headers
- Instrument graph.invoke() with ddtrace.tracer.trace("agent.reasoning_loop")
- Capture LLM token usage + latency as span metadata
- Emit DogStatsD custom metrics (praesidium.agent.token_usage, etc.)

## Celery Rules
- Never change queue names (quick, standard, long) without updating the gateway dispatcher
- acks_late=True on standard and long queues -- tasks acknowledged AFTER completion only
- Every task must handle re-delivery gracefully (idempotency guards)
- LangGraph tasks MUST check for existing checkpoints before re-executing
- Never block the event loop -- use async where possible, run sync code in executors

## Tool Safety Rules
- PLATFORM and INTEGRATION tools run in-process -- they are trusted code
- CUSTOM tools MUST be dispatched to the tool-runner (k8s Job) -- never run user code in-process
- Always validate tool input against JSON Schema BEFORE execution
- Always check risk_level gates BEFORE execution
- Log every tool call to activity_log
- All tools MUST return standardized ToolResponse: {success, data, error, usage: {tokens, cost}}
  No raw dicts or ad-hoc return shapes -- every tool output flows through ToolResponse (v1.10)

## Coding Standards
- Python 3.12, type hints on all functions
- Pydantic models for task payloads -- no raw dicts
- structlog for structured JSON logging -- no print() statements
- Every task function must be decorated with @app.task(bind=True)
- Use the shared LLMClient abstraction -- never call LLM SDKs directly

## Testing Rules
- Mock qqq API (respx), Redis (fakeredis), and LLM SDKs in all tests
- Use testcontainers (PostgreSQL) for LangGraph checkpoint tests
- Test every task's happy path AND failure/retry path
- Test all tool risk-level gates
- Target 90%+ test coverage

## What NOT to Do
- Do NOT add HTTP endpoints -- this is a worker, not a web server
- Do NOT run user-provided code in-process -- dispatch to tool-runner
- Do NOT bypass risk-level safety gates for any tool
- Do NOT hardcode API keys, model names, or URLs -- use Pydantic Settings from env
- Do NOT skip writing tests for new tasks or tools
- Do NOT acknowledge Celery tasks before persisting results to qqq API

Tool Runner

Epic 7 — Phase 2

Sandboxed execution environment for CUSTOM tools + “Blind” Secret Pipeline.

The Tool Runner is a minimal, security-hardened container image that receives tool input as JSON, executes the tool code in isolation, and returns a ToolResult JSON via stdout. It runs as ephemeral k8s Jobs (one per execution) or as warm-pool Deployments for frequently-used tools. Re-sequenced to Phase 2 (before Agent Workers) so that workers have a tested execution target from Day 1. Implements the “Blind” Secret Pipeline: plain-text secrets NEVER touch the database or Gateway. The secret_ref table stores only AWS Secret ARNs; the Worker pod uses IRSA to fetch values directly from AWS Secrets Manager at runtime. Tool Runner Jobs receive DD_TRACE_ID and DD_PARENT_ID as env vars for distributed tracing.

Repository Setup

Repositorypraesidium2-tool-runner/
LanguagePython 3.12
DependenciesMinimal — pydantic, httpx (base image only)
Base Imagespraesidium-tool-runner:python, praesidium-tool-runner:typescript
Executionk8s Job (ephemeral) or warm Deployment (HTTP)
Isolationk8s NetworkPolicy, resource limits, read-only filesystem
Credential SourceAWS Secrets Manager → k8s external-secrets → env vars
praesidium2-tool-runner/ src/ __init__.py runner.py # CLI entry: parse args, load tool, execute, print JSON base_tool.py # BaseTool, ToolContext, ToolResult classes loader.py # Dynamic tool module loading from git checkout sandbox.py # Resource limit enforcement, signal handlers http_server.py # Warm-pool HTTP endpoint (FastAPI micro) images/ Dockerfile.python # Python 3.12-slim base image (sole runner runtime) k8s/ job-template.yaml # k8s Job spec template (envsubst) network-policy.yaml # Egress restricted to integration URLs warm-pool-deployment.yaml # Warm-pool Deployment + HPA resource-quotas.yaml # Per-risk-level resource quota presets tests/ conftest.py test_runner.py # End-to-end: JSON in, ToolResult out test_base_tool.py # BaseTool contract tests test_loader.py # Dynamic module loading tests test_sandbox.py # Timeout, memory limit, signal tests test_http_server.py # Warm-pool HTTP endpoint tests fixtures/ mock_tool_pass.py # Tool that returns success mock_tool_fail.py # Tool that raises an error mock_tool_timeout.py # Tool that exceeds timeout mock_tool_oom.py # Tool that exceeds memory limit pyproject.toml CLAUDE.md

How It Works

The Tool Runner has a single responsibility: receive tool input as JSON, execute the tool code, return a ToolResult JSON via stdout. The agent worker orchestrates everything else (version resolution, credential injection, k8s Job creation).

Execution Flow (Agent Worker → Tool Runner)

  • Step 1: Agent worker receives tool_call for a CUSTOM tool
  • Step 2: Resolve tool_version → git SHA + entry point module
  • Step 3: Resolve credentials → secret_ref → Secrets Manager
  • Step 4: Create k8s Job spec with: image, command, env vars, resource limits, network policy, timeout
  • Step 5: Submit k8s Job, wait for completion (poll or watch)
  • Step 6: Read pod stdout → parse ToolResult JSON
  • Step 7: Pod auto-terminates (k8s Job TTL: 60s after completion)
  • Step 8: Return result to agent conversation

CLI Invocation (Ephemeral Mode)

# Inside the k8s Job container:
python -m tool_runner \
  --entry my_tool:execute \
  --input '{"url": "https://api.hubspot.com/campaigns", "date_range": "last_30d"}' \
  --timeout 30

# stdout (captured by agent worker):
{"success": true, "data": {"campaigns": [...]}, "error": null, "metadata": {"duration_ms": 1240}}

HTTP Invocation (Warm Pool Mode)

# Agent worker calls the warm pod's HTTP endpoint:
POST http://tool-runner-python:8080/execute
Content-Type: application/json

{
  "entry": "my_tool:execute",
  "input": {"url": "https://api.hubspot.com/campaigns"},
  "timeout": 30
}

# Response:
{"success": true, "data": {"campaigns": [...]}, "error": null, "metadata": {"duration_ms": 240}}

Execution Mode Decision Logic

Each tool has an execution_mode field set at tool registration time, stored in the tool table. The agent worker reads this field before dispatching.

ModeWhen to UseExamples
ephemeral (default) Any tool with side effects, write operations, or external API mutations. Safest — fully sandboxed k8s Job with resource limits, network policy, and auto-cleanup. create_jira_issue, slack_send_message, github_create_pr
warm_pool Read-only tools with no side effects. Lower latency (~5x faster) via persistent HTTP Deployment. Only for tools that cannot mutate external state. search_agents, search_knowledge, list_projects
# Worker dispatch logic (simplified):
tool = registry.get(tool_name)
if tool.execution_mode == "warm_pool":
    result = await http_client.post(f"http://tool-runner-python:8080/execute", json=payload)
else:  # "ephemeral" (default)
    result = await k8s_job_manager.run(tool, payload)

Tool Interface Contract

All CUSTOM tools implement the same BaseTool base class. This contract is shared between the Tool Forge (build time) and the Tool Runner (execution time).

from pydantic import BaseModel
from typing import Any

class ToolContext(BaseModel):
    agent_id: str
    project_id: str | None = None
    conversation_id: str | None = None
    credentials: dict = {}            # Injected from secret_ref via Secrets Manager

class ToolResult(BaseModel):
    success: bool
    data: Any = None
    error: str | None = None
    metadata: dict = {}               # duration_ms, retries, bytes_transferred, etc.

class BaseTool:
    name: str
    description: str
    input_schema: dict                 # JSON Schema for input validation

    async def execute(self, input: dict, context: ToolContext) -> ToolResult:
        raise NotImplementedError

Contract Rules

  • Tool MUST return a ToolResult JSON on stdout — even on failure (set success: false)
  • Tool MUST NOT write to stderr except for debug logging (stderr is captured separately for diagnostics)
  • Tool MUST NOT read from stdin — all input comes via CLI args or HTTP body
  • Tool MUST complete within the configured timeout — SIGTERM at timeout, SIGKILL at timeout+5s
  • Tool MUST NOT attempt to contact any host not in the NetworkPolicy allowlist

Base Container Images

Two base images cover the supported tool languages. Each image contains the tool runner framework + language runtime. Tool code is mounted at runtime via k8s volume (git checkout or ConfigMap).

praesidium-tool-runner:python

FROM python:3.12-slim

# Security: non-root user
RUN groupadd -r toolrun && \
    useradd -r -g toolrun -d /app toolrun

WORKDIR /app

# Base dependencies only
COPY pyproject.toml .
RUN pip install --no-cache-dir pydantic httpx

# Tool runner framework
COPY src/ ./src/

# Tool code mounted here at runtime
VOLUME /app/tools

USER toolrun

# Read-only filesystem (except /tmp)
ENTRYPOINT ["python", "-m", "src.runner"]

Python-Only Runtime

The Tool Runner is Python-only. All custom tools are Python modules executed in a Python 3.12-slim sandbox via runner.py. There is no TypeScript runner — the Python sandbox provides the sole execution environment for K8s Jobs and warm-pool Deployments.

Execution Modes

Ephemeral Mode (k8s Job)

Default mode. A new k8s Job is created for each tool execution. Pod is created, runs the tool, outputs JSON to stdout, and is auto-deleted after TTL.

LifecycleCreated per execution, auto-deleted (TTL: 60s)
Cold Start~3-5s (image pull cached, pod scheduling)
IsolationFull — separate pod, separate network namespace
Use CaseInfrequent tools, high-risk tools, first execution
Tool CodeMounted via initContainer git clone at pinned SHA

Warm Pool Mode (Deployment)

Optimization for frequently-used tools. Pre-loaded pods with HTTP endpoint. Tool code mounted via ConfigMap (updated on tool_version publish).

LifecycleLong-running Deployment, HPA-scaled
Latency<200ms (HTTP call to warm pod)
IsolationShared pod, process-level isolation per request
Use CaseHigh-frequency tools (>10 calls/hour), low-risk tools
Tool CodeConfigMap volume, updated on publish via rolling restart

Network Isolation (k8s NetworkPolicy)

Tool Runner pods operate under strict network policies. Egress is restricted to only the integration URLs that the tool is authorized to access. DNS is allowed for name resolution.

# k8s/network-policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: tool-runner-egress
spec:
  podSelector:
    matchLabels:
      app: tool-runner
  policyTypes:
    - Egress
  egress:
    # Allow DNS resolution
    - to:
        - namespaceSelector: {}
      ports:
        - protocol: UDP
          port: 53
    # Allow integration endpoints only (per-tool from integration_config.base_url)
    - to:
        - ipBlock:
            cidr: ${INTEGRATION_CIDR}   # Resolved at Job creation time
      ports:
        - protocol: TCP
          port: 443

Network Rules

  • NO access to the qqq API from Tool Runner pods — qqq access is via ToolContext on the agent worker side only
  • NO access to the internal k8s API server
  • NO access to Redis, PostgreSQL, or any internal service
  • ONLY outbound HTTPS to pre-approved integration URLs (from integration_config.base_url)
  • DNS resolution allowed (required for HTTPS hostname resolution)

Resource Limits by Risk Level

Resource limits are set per tool based on the risk_level type table value assigned during Tool Forge review. Higher risk = more resources but stricter monitoring.

Risk LevelCPU RequestCPU LimitMemory RequestMemory LimitTimeoutMax Retries
LOW 0.25 CPU0.5 CPU256Mi512Mi30s3
MEDIUM 0.5 CPU1 CPU512Mi1Gi60s2
HIGH 0.5 CPU1 CPU1Gi2Gi120s1
CRITICAL 1 CPU2 CPU2Gi4Gi300s0

Resource Enforcement

  • k8s enforces CPU and memory limits natively (OOMKilled if exceeded)
  • Timeout enforced by the Tool Runner process: SIGTERM at timeout, SIGKILL at timeout + 5s
  • CRITICAL tools get zero retries — failures escalate to human immediately
  • Risk level is set during Tool Forge REVIEW step and stored on tool.risk_level_id

Credential Injection

Credentials are injected as environment variables into the Tool Runner pod. The agent worker resolves credentials from secret_ref and maps them to k8s secret references in the Job spec.

Resolution Flow

Agent worker resolves credentials for the tool (narrowest scope wins):
  1. Check AGENT-scoped secret_ref matching agent_id (narrowest)
  2. If not found, check TOOL-scoped secret_ref matching tool_id
  3. If not found, check PROJECT-scoped matching project_id
  4. If not found, check INSTANCE-scoped (broadest fallback)
  5. Fetch secret_path from secret_ref table
  6. Map to k8s secret reference in Job spec:

# In the k8s Job spec:
env:
  - name: HUBSPOT_API_KEY
    valueFrom:
      secretKeyRef:
        name: praesidium-secrets-${INSTANCE}
        key: HUBSPOT_API_KEY
  - name: TOOL_INPUT
    value: '{"url": "https://api.hubspot.com/campaigns"}'
  - name: TOOL_AGENT_ID
    value: "uuid-of-agent"
  - name: TOOL_PROJECT_ID
    value: "uuid-of-project"

Credential Security Rules

  • Credentials NEVER written to disk inside the pod — env vars only
  • k8s secrets sourced from AWS Secrets Manager via external-secrets operator
  • Pod serviceAccountName has minimal RBAC — no access to other secrets
  • Credentials are scoped: a TOOL-scoped secret is only injected when that specific tool runs

k8s Job Template

The agent worker generates this Job spec dynamically for each CUSTOM tool execution. Template variables are resolved from the tool registry and secret_ref tables.

# k8s/job-template.yaml (agent worker fills in variables)
apiVersion: batch/v1
kind: Job
metadata:
  name: tool-exec-${EXECUTION_ID}
  namespace: ${NAMESPACE}
  labels:
    app: tool-runner
    tool-id: ${TOOL_ID}
    agent-id: ${AGENT_ID}
spec:
  ttlSecondsAfterFinished: 60
  activeDeadlineSeconds: ${TIMEOUT}
  backoffLimit: 0
  template:
    metadata:
      labels:
        app: tool-runner
    spec:
      restartPolicy: Never
      serviceAccountName: tool-runner-sa
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000
        readOnlyRootFilesystem: true
      initContainers:
        - name: git-checkout
          image: alpine/git:latest
          command: ["sh", "-c",
                    "git clone --no-checkout ${TOOL_REPO_URL} /tools && cd /tools && git checkout ${GIT_SHA}"]
          volumeMounts:
            - name: tool-code
              mountPath: /tools
      containers:
        - name: runner
          image: praesidium-tool-runner:${LANGUAGE}
          args: ["--entry", "${MODULE}:${FUNC}",
                 "--input", "${INPUT_JSON}",
                 "--timeout", "${TIMEOUT}"]
          resources:
            requests:
              cpu: ${CPU_REQUEST}
              memory: ${MEM_REQUEST}
            limits:
              cpu: ${CPU_LIMIT}
              memory: ${MEM_LIMIT}
          env:
            ${CREDENTIAL_ENV_VARS}
            - name: DD_TRACE_ID
              value: "${TRACE_ID}"
            - name: DD_PARENT_ID
              value: "${PARENT_SPAN_ID}"
          volumeMounts:
            - name: tool-code
              mountPath: /app/tools
              readOnly: true
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: tool-code
          emptyDir: {}
        - name: tmp
          emptyDir:
            sizeLimit: 100Mi

Day-by-Day Implementation (Phase 2)

DayTasksDeliverable
Day 6 Base image + tool interface
• Create praesidium2-tool-runner/ repo
• Implement BaseTool, ToolContext, ToolResult classes
• Implement runner.py CLI entry point (arg parsing, module loading, JSON output)
• Implement loader.py dynamic module loading
• Write Dockerfile.python base image
• Write unit tests for base classes and runner
Working CLI: python -m tool_runner --entry mock:execute --input '{}' returns valid JSON
Day 7 k8s Job integration + sandbox
• Implement sandbox.py (timeout signals, memory monitoring)
• Write k8s Job template YAML with SHA-only git checkout (not --branch)
• Write NetworkPolicy YAML
• Write resource quota presets for each risk level
• Test Job creation/execution locally (Docker Desktop k8s)
• Verify network isolation (tool cannot reach qqq API or internal services)
Ephemeral k8s Job runs a mock tool and returns result to stdout
Day 8 “Blind” Secret Pipeline + credential injection
• Implement IRSA-based secret fetching (Boto3 + IRSA IAM role)
• Worker fetches secret_ref ARN from qqq, resolves value from AWS Secrets Manager
• Inject credentials as env vars in k8s Job spec (never written to disk)
• Add DD_TRACE_ID + DD_PARENT_ID env vars to Job spec
• Write integration tests for secret pipeline
• Test credential flow end-to-end (secret_ref → Secrets Manager → env var → tool)
“Blind” secret pipeline working: no plain-text secrets in DB or Gateway responses
Day 9-10 Warm pool + worker integration + hardening
• Implement http_server.py warm-pool HTTP endpoint
• Wire agent worker to create k8s Jobs for CUSTOM tool_calls
• Add warm-pool routing (check tool frequency, route to HTTP if warm)
• Add timeout handling, OOM detection, error wrapping
• Verify “Hello World” tool execution from Gateway bus
• Security audit: verify non-root, read-only FS, network policy enforced
Full pipeline working: Gateway → CUSTOM tool → sandboxed execution → result returned

Dependencies

Python Base Image (Runtime)

pydantic httpx

Intentionally minimal. Tool-specific dependencies are installed at build time by the Tool Forge and baked into the tool's requirements.txt (installed in the initContainer or warm-pool image layer).

Development / Testing

pytest pytest-asyncio pytest-cov httpx (test client) ruff mypy

Test Requirements

pytest for all tests. Mock tool execution with fixture tools (pass, fail, timeout, OOM). Verify resource limits and contract compliance.

Mandatory Test Coverage

  • Runner CLI: test JSON input parsing, ToolResult JSON output, exit codes (0 for success, 1 for tool error, 2 for framework error)
  • BaseTool contract: test that tools returning non-ToolResult JSON are caught and wrapped in error ToolResult
  • Loader: test dynamic module import from file path, test missing module error, test invalid entry point format
  • Timeout enforcement: test that mock_tool_timeout.py is killed after configured timeout, returns success: false with timeout error
  • Error handling: test unhandled exception in tool code returns success: false with error message (no stack trace in data)
  • HTTP server: test warm-pool endpoint accepts POST, returns ToolResult JSON, handles concurrent requests
  • Credential injection: test that env vars are readable inside tool execution context
  • Coverage target: 90%+

Acceptance Criteria

  • python -m tool_runner --entry mock:execute --input '{}' returns valid ToolResult JSON on stdout
  • k8s Job runs a CUSTOM tool in an ephemeral pod and exits cleanly (TTL auto-delete verified)
  • Warm-pool pod accepts HTTP POST and returns ToolResult with <200ms latency
  • NetworkPolicy blocks all egress except pre-approved integration URLs + DNS
  • Resource limits enforced: OOMKilled if tool exceeds memory limit, SIGTERM/SIGKILL on timeout
  • Credentials injected as env vars from k8s secrets (sourced from Secrets Manager)
  • Pod runs as non-root user (UID 1000) with read-only root filesystem
  • Tool errors are wrapped in ToolResult(success=false) — no raw exceptions leak to agent
  • Python base image builds and passes smoke tests
  • End-to-end flow works: agent worker creates Job → tool runs → result returned to conversation
  • All tests passing with 90%+ coverage

CLAUDE.md Rules for Tool Runner Agents

These rules MUST be placed in the CLAUDE.md file at the root of the praesidium2-tool-runner/ repository.

# praesidium2-tool-runner CLAUDE.md

## Project Context
This is the Tool Runner for Praesidium2 -- a sandboxed execution environment
for CUSTOM (forged) tools. It receives JSON input, executes tool code in
isolation, and returns ToolResult JSON via stdout. It runs as k8s Jobs
(ephemeral) or warm-pool Deployments (HTTP).

## Architecture Rules
- The Tool Runner is MINIMAL -- it does NOT import qqq, LangGraph, or any agent framework
- The only runtime dependencies are pydantic and httpx -- nothing else in the base image
- Tool-specific dependencies are installed per-tool (in requirements.txt), NOT in the base image
- All tool code is mounted at /app/tools via k8s volume -- never baked into the base image
- Output is ALWAYS valid ToolResult JSON on stdout -- even on error
- stderr is for debug logging only -- agent worker captures but does not parse it

## Security Rules (NON-NEGOTIABLE)
- Container runs as non-root user (UID 1000)
- Root filesystem is read-only (only /tmp is writable, with 100Mi limit)
- NetworkPolicy restricts egress to integration URLs only -- NO access to qqq, Redis, PostgreSQL
- Credentials are env vars ONLY -- never written to disk ("Blind" Secret Pipeline)
- Secrets fetched via IRSA from AWS Secrets Manager -- plain-text NEVER in DB or Gateway
- Resource limits are enforced by k8s (CPU, memory) and by the runner (timeout via signals)
- NEVER disable security context, network policy, or resource limits for debugging
- Use SHA-only git checkouts (git clone --no-checkout + git checkout $SHA)

## Observability (v1.8)
- DD_TRACE_ID and DD_PARENT_ID are injected as env vars by the agent worker
- Tool Runner logs are automatically correlated with parent Worker task in Datadog

## Coding Standards
- Python 3.12, strict typing (mypy)
- pydantic for all data models
- pytest for testing, 90%+ coverage target
- ruff for linting and formatting
- Minimal dependencies -- justify every addition in PR description

## What NOT to Do
- Do NOT add qqq client, database drivers, or ORM libraries
- Do NOT add web frameworks beyond the minimal HTTP server for warm pool
- Do NOT store state between executions -- each run is stateless
- Do NOT log credentials or tool input/output at INFO level (DEBUG only, redacted)
- Do NOT add retry logic inside the runner -- retries are managed by the agent worker

Frontend SPA

Epic 3 — Days 1-10

React 19 SPA — the command center for the AI-augmented workforce.

The Frontend is a single-page application providing primary pages: Dashboard, Projects, Team, Chat, Escalations, Settings, Activity, Cost Center, Approvals, KPIs, Schedules, and the HITL Approval Dashboard (with action hash verification). It talks to two backends: the qqq API (data CRUD) and the Agent Gateway (AI interactions, streaming). The Command Palette (Cmd+K) is backed by /api/search (pgvector semantic search). Real-time updates via WebSocket (Redis pub/sub) and SSE (chat streaming). All type tables cached on load for instant rendering. Datadog RUM provides frontend observability linked to backend traces. V1.5 additions: Meetings page and Tool Forge pages (specs retained below).

Repository Setup

Repositorypraesidium2-web/
FrameworkReact 19 + Vite
LanguageTypeScript (strict mode)
Package Managerpnpm
RoutingTanStack Router (file-based, type-safe)
State / DataTanStack Query + Zustand
StylingTailwind CSS + shadcn/ui
ChartsRecharts
Real-timeWebSocket (broadcasts) + SSE (chat streaming)
TestingVitest + Testing Library + MSW
AuthAuthentik OIDC (JWT)
praesidium2-web/ src/ app/ routes/ # TanStack Router file-based routes __root.tsx # Root layout (sidebar + header + outlet) index.tsx # Dashboard page (/) projects/ index.tsx # Projects list (/projects) $projectId.tsx # Project detail (/projects/:id) team/ index.tsx # Team grid (/team) $memberId.tsx # Member detail (/team/:id) chat/ index.tsx # Chat list (/chat) $conversationId.tsx # Conversation (/chat/:conv_id) approvals/ index.tsx # Approval dashboard (/approvals) $approvalId.tsx # Approval detail (/approvals/:id) kpis/ index.tsx # KPI dashboard (/kpis) schedules/ index.tsx # Schedule list (/schedules) $scheduleId.tsx # Schedule detail/edit (/schedules/:id) meetings/ # V1.5 index.tsx # V1.5: Meetings list (/meetings) $meetingId.tsx # V1.5: Meeting detail (/meetings/:id) forge/ # V1.5 index.tsx # V1.5: Tool catalog (/forge) $toolId.tsx # V1.5: Tool detail (/forge/:id) build.tsx # V1.5: Build mode (/forge/build) escalations.tsx # Escalations inbox (/escalations) settings.tsx # Settings (/settings) activity.tsx # Activity feed (/activity) costs.tsx # Cost center (/costs) components/ ui/ # shadcn/ui primitives layout/ # sidebar, header, command-palette chat/ # chat-thread, chat-input, tool-call-card project/ # project-card, kanban, milestone-timeline team/ # member-card, skill-radar, memory-explorer approval/ # approval-card, approval-list, approval-actions feedback/ # feedback-widget, feedback-summary, quality-score kpi/ # kpi-chart, baseline-comparison, kill-criteria-card schedule/ # schedule-list, schedule-form, cron-picker, run-history meeting/ # V1.5: transcript-view, action-items, schedule forge/ # V1.5: tool-card, code-preview, test-output shared/ # status-badge, priority-badge, avatar wizards/ # instance-setup, project-setup, agent-creation hooks/ use-websocket.ts # WebSocket connection + reconnect use-sse-stream.ts # SSE chat streaming hook use-qqq-query.ts # TanStack Query wrapper for qqq API use-gateway-mutation.ts # Gateway API mutation wrapper use-permissions.ts # JWT permission checks api/ qqq-client.ts # Typed qqq REST client gateway-client.ts # Agent Gateway client (SSE, WS) stores/ type-tables.ts # Zustand store: all 59 type tables auth.ts # Zustand store: JWT, user, permissions websocket.ts # Zustand store: WS connection state lib/ auth.ts # Authentik OIDC flow types.ts # Shared TypeScript types utils.ts # Formatting, date helpers permissions.ts # IAM policy parsing + evaluation main.tsx # React entry point tests/ setup/ msw-handlers.ts # MSW API mock handlers test-utils.tsx # Render with providers wrapper components/ # Component-level tests pages/ # Page-level integration tests hooks/ # Hook tests index.html vite.config.ts tailwind.config.ts tsconfig.json Dockerfile CLAUDE.md

API Client Layer

Two API clients — one for data operations (qqq), one for AI interactions (Agent Gateway). Both use the same Authentik JWT for authentication.

qqq-client.ts (Data CRUD)

ActionMethod
List recordsGET /api/{table}?filter=...&sort=...&page=...
Get by IDGET /api/{table}/{id}?include=...
CreatePOST /api/{table}
UpdatePATCH /api/{table}/{id}
DeleteDELETE /api/{table}/{id}

All qqq-client methods are generic: qqqClient.list<Project>('project', params). TypeScript types are generated from qqq's OpenAPI spec.

gateway-client.ts (AI Interactions)

ActionMethod
Chat with agentPOST /api/chat/{agent_id} → SSE stream
Execute taskPOST /api/task/execute → job ID
Start meetingPOST /api/meeting/{meeting_id}/start → WS updates
Forge sessionPOST /api/forge/session → interactive SSE
Stream eventsGET /api/stream/{id} (SSE)

Real-Time Architecture

Two real-time channels serve different purposes. WebSocket for push notifications (cache invalidation), SSE for streaming agent responses (chat).

WebSocket (useWebSocket hook)

Single persistent connection via Agent Gateway. Receives push events from Redis pub/sub. Used primarily for TanStack Query cache invalidation.

EventAction
task:status_changedInvalidate task queries
agent:status_changedInvalidate team queries
meeting:startedShow meeting notification
escalation:createdShow escalation alert + badge
job:completedInvalidate job queries
project:health_changedInvalidate project queries
tool:publishedInvalidate tool catalog queries

SSE (useSSEStream hook)

Per-conversation stream for agent responses. Opened when user is viewing a chat. Closed on navigation away.

EventDescription
tokenText delta (streamed character by character)
thinkingAgent is thinking (show indicator)
tool_call_startTool call initiated (show tool card)
tool_call_resultTool call completed (update card)
message_completeFull message done (finalize)
errorStream error (show error state)

Reconnection Strategy

  • WebSocket: exponential backoff (1s, 2s, 4s, 8s, max 30s). Reconnect on close. Show connection indicator in header.
  • SSE: auto-reconnect with EventSource built-in retry. Resume from last event ID.

Type Table Caching (Zustand Store)

All 59 type tables are fetched once on app load and cached in a Zustand store. Components reference type data by ID for instant rendering. No additional API calls needed for status labels, colors, or icons.

// stores/type-tables.ts
interface TypeTableStore {
  tables: Record<string, TypeRow[]>;    // keyed by table name
  byId: Record<string, Record<number, TypeRow>>;  // fast lookup by ID
  isLoaded: boolean;
  load: () => Promise<void>;           // fetch all 56 tables in parallel
}

// Usage in components:
const taskStatus = useTypeTable('task_status');
const statusRow = taskStatus.byId[task.status_id];
// statusRow = { id: 3, name: 'IN_PROGRESS', label: 'In Progress', color: '#3b82f6', icon: 'spinner' }

Type Tables Loaded at Startup (59 tables)

member_type, member_status, agent_category, execution_status, tool_category, tool_source_type, risk_level, test_status, programming_language, sync_status, integration_type, auth_type, health_status, project_type, project_status, project_health, priority_level, project_role, milestone_status, task_type, task_status, author_role, conversation_type, conversation_status, participant_role, message_role, content_type, meeting_type, memory_type, memory_source, document_source_type, index_status, job_status, job_execution_status, trigger_type, escalation_status, artifact_type, interaction_channel, interaction_type, interaction_status, schedule_status

Auth Integration (Authentik OIDC)

Standard OIDC Authorization Code flow with PKCE. JWT stored in memory (not localStorage). Refresh handled proactively at 80% of expiry window.

OIDC Flow

  • Redirect to Authentik /authorize with PKCE challenge
  • Authentik authenticates user, redirects back with code
  • Exchange code for JWT at /token endpoint
  • Parse JWT claims: policies, praesidiumInstanceId, praesidiumAllAccess
  • Store JWT in Zustand auth store (memory only)
  • Attach JWT as Authorization: Bearer to all API calls
  • Proactive refresh at 80% of token expiry

Permission-Driven UI

// lib/permissions.ts
function canPerform(
  policies: PolicyStatement[],
  action: string,
  resource: string
): boolean {
  // Evaluate IAM-style policies
  // Deny takes precedence over Allow
}

// Usage in components:
const { can } = usePermissions();
{can('edit', 'project') &&
  <Button>Edit Project</Button>
}

// Sidebar items filtered by permission:
{can('read', 'escalation') &&
  <SidebarItem to="/escalations" />
}

Page Implementation Order (5 Phases)

Pages are built in dependency order. The app shell and auth must work before any data pages. Core data pages before advanced features. Chat requires SSE streaming infrastructure.

PhasePages / ComponentsWhy This Order
Phase 1 App Shell
• Root layout (__root.tsx): sidebar, header, outlet
• Sidebar navigation (permission-filtered)
• Header (user menu, connection indicator, notifications)
• Command palette (Cmd+K)
• Auth flow (login, token, refresh, logout)
• Type table loading on startup
• WebSocket connection setup
Everything else renders inside the shell. Auth must work before any API calls. Type tables must load before any data-driven components render.
Phase 2 Core Data Pages
• Dashboard (/): project health cards, team status, escalation summary, meeting schedule, cost burn
• Projects (/projects, /projects/:id): card grid, tabbed detail (overview, tasks kanban, team, milestones, meetings, activity, integrations)
• Team (/team, /team/:id): member grid, agent detail (skills radar, memory explorer, tools, history, performance)
Dashboard is the landing page. Projects and Team are the most-used pages. All are pure qqq API reads — no Gateway dependency.
Phase 3 Chat
• Chat list (/chat): conversation cards, filters
• Chat thread (/chat/:conv_id): message list, SSE streaming, tool call visualization, thinking indicator
• Chat input: message compose, model selector, context display
• Group chats: multi-agent conversations
Chat requires SSE streaming infrastructure (useSSEStream hook) and Gateway API integration. Built after core data pages are stable.
Phase 4 Supporting Pages
• Meetings (/meetings, /meetings/:id): schedule, transcripts, action items V1.5
• Escalations (/escalations): inbox sorted by severity, acknowledge/resolve/dismiss
• Schedules (/schedules, /schedules/:id): schedule list, detail/edit with cron picker, run history v1.9
• Settings (/settings): instance config, integrations, agent templates, knowledge base
• Activity (/activity): chronological feed, filterable, real-time via WS
• Cost Center (/costs): token usage charts (Recharts), budget alerts
These pages depend on patterns established in Phase 2-3. Meetings page deferred to V1.5 (V1 uses Slack-based status). Schedules page (v1.9) provides CRUD for scheduled_job records with cron picker and run history. Cost Center needs Recharts integration.
Phase 5 Tool Forge V1.5
• Tool catalog (/forge): tool cards, search, filter by category
• Tool detail (/forge/:id): versions, usage stats, assigned agents
• Build mode (/forge/build): split-pane (left: chat with forge agent, right: code preview + test output + file tabs)
• Approve/Request Changes/Reject actions
Deferred to V1.5. Most complex page. Split-pane requires SSE streaming + code preview + interactive actions. Built last when all infrastructure is proven.

Component Library

shadcn/ui Primitives

Button Card Dialog DropdownMenu Input Select Tabs Table Badge Avatar Tooltip Sheet Command Skeleton ScrollArea Separator Toast Form Calendar Popover

Custom Components

StatusBadge PriorityBadge HealthBadge MemberAvatar MemberCard ProjectCard ToolCallCard ThinkingIndicator SkillRadar MemoryExplorer KanbanBoard MilestoneTimeline TranscriptView CostDisplay RelativeTime CodePreview CommandPalette ConnectionIndicator

Type-Table-Driven Components

StatusBadge, PriorityBadge, and HealthBadge all read their label, color, and icon from the cached type tables. Adding a new status value = add a row to the type table. Zero frontend code changes needed.

Day-by-Day Implementation

DayTasksDeliverable
P1 D3 Scaffold + App Shell
• pnpm create vite praesidium2-web --template react-ts
• Install TanStack Router, TanStack Query, Zustand, Tailwind, shadcn/ui
• Configure TypeScript strict mode, path aliases
• Set up file-based routing structure
• Build root layout: sidebar, header, outlet
• Build command palette shell (Cmd+K)
App runs with sidebar navigation between empty route stubs
P1 D4 Auth + API Clients + Type Tables
• Implement Authentik OIDC flow (lib/auth.ts)
• Build qqq-client.ts with typed generic methods
• Build gateway-client.ts shell
• Implement type table Zustand store + loading
• Build auth Zustand store (JWT, user, permissions)
• Add auth guard to root layout
User can log in via Authentik. Type tables load on startup. Unauthenticated users redirected to login.
P1 D5 Shared Components + WebSocket
• Build StatusBadge, PriorityBadge, HealthBadge (type-table-driven)
• Build MemberAvatar, RelativeTime, CostDisplay
• Implement useWebSocket hook (connect, reconnect, event dispatch)
• Wire WS events to TanStack Query cache invalidation
• Build ConnectionIndicator for header
Shared components render with type table data. WS connection established with reconnection.
P2 D6 Dashboard + Projects List
• Dashboard page: project health cards, team status grid, escalation summary, today's meetings, cost burn rate widget
• Projects list page: card grid with search, filter by status/type/health
• ProjectCard component
• Wire to qqq API with TanStack Query
Dashboard renders live data. Projects list shows all projects with filtering.
P2 D7 Project Detail (Tabbed)
• Overview tab: description, health, dates, integrations
• Tasks tab: KanbanBoard component (drag-and-drop)
• Team tab: assigned members with roles
• Milestones tab: MilestoneTimeline component
• Meetings tab: meeting list for project
• Activity tab: project-filtered activity feed
Full project detail page with all 6 tabs rendering real data.
P2 D8 Team Pages
• Team list page: grid of MemberCard components (agents + humans)
• Member detail page (agent): SkillRadar, MemoryExplorer, tool assignments, execution history, performance metrics
• Member detail page (human): profile, projects, interactions
• Agent hire dialog (from template library)
Full team browsing and agent detail pages working.
P2 D9 Chat (SSE Streaming)
• Chat list page: conversation cards with last message preview
• Implement useSSEStream hook
• Chat thread: message rendering, streaming token display
• ThinkingIndicator component
• ToolCallCard component (shows tool name, input, output, status)
• Chat input: message compose, model selector
Users can chat with agents. Responses stream in real-time with tool call visualization.
P2 D10 Chat (Polish) + Meetings + Escalations
• Group chats (multi-agent conversations)
• Meetings list + detail (transcripts, action items)
• Escalations inbox (severity sort, acknowledge, resolve, dismiss)
• Activity feed page (filterable, real-time via WS)
Chat polished. Meetings, Escalations, and Activity pages functional.
P3 D11 Settings + Cost Center
• Settings page: instance config, integrations management, agent template library, knowledge base, notification preferences
• Cost Center page: Recharts integration (token usage by agent, by project, by time), burn rate chart, budget alerts
• Setup wizards: instance setup wizard, project setup wizard, agent creation wizard (v1.12)
Settings and Cost Center pages complete. All 3 wizards functional.
P3 D12 Tool Forge V1.5
• Tool catalog page: ToolCard grid, search, filter by category
• Tool detail page: versions, usage stats, assigned agents
• Build mode: split-pane layout (chat left, code right)
• CodePreview component (syntax highlighted)
• Test output panel
• Approve / Request Changes / Reject actions
Deferred to V1.5. Full Tool Forge UI including split-pane build mode.
P3 D13 Testing + Polish
• Write Vitest + Testing Library tests for all pages
• MSW handlers for qqq and Gateway API mocking
• Accessibility audit (keyboard nav, screen reader labels)
• Responsive design verification (mobile, tablet, desktop)
• Error boundaries and loading skeletons for all pages
90%+ test coverage. All pages accessible and responsive.

Agent Creation Wizard v1.12

Triggered from Team page → “New Agent” button (or by the COS agent requesting a new teammate). Multi-step wizard with progress persisted in browser localStorage. If the user leaves mid-wizard, they resume on return.

1
Identity
Name, avatar, soul
→
2
Role
Title, job desc
→
3
Skills
Catalog browse
→
4
MCP
Server select
→
5
Tools
Freestyle assign
→
6
Model
LLM + cost info
→
7
Review
Create agent
StepNameUI Details
1 Identity Name, display name, avatar upload/select. Choose a soul/personality from available templates or write a custom one.
2 Role Job title, job description. Defines what this agent is responsible for within the instance.
3 Skills Browse company skill catalog (published skill_definition records). Select skills, choose version per skill (default: latest). System validates tool requirements against instance availability; missing requirements flagged with warnings.
4 MCP Servers Select from instance’s configured mcp_server records. Shows which skills require which MCP connections.
5 Tools Assign additional “freestyle” tools beyond what skills bring. Browse instance’s available tools by category.
6 Model Select LLM model (default from template). Show capability tier and cost per 1M tokens.
7 Review & Create Summary of everything configured. Estimated cost per task based on model + tool usage. “Create Agent” button.

Published Profile Fork Workflow v1.12

When a Super Operator clicks “Fork” on a published agent profile from the company catalog, this 7-step workflow creates a fully configured agent in a single transaction.

1
Version
Select profile ver.
→
2
Customize
Pre-filled form
→
3
Skills
Tool validation
→
4
MCP
Match servers
→
5
Tools
Confirm assigns
→
6
Create
Single txn
→
7
Verify
Activate
StepNameDetails
1 Select Version UI shows available versions from published_agent_profile_version. User picks one (default: latest).
2 Customize Pre-filled form with the version’s soul, role_title, job_description, personality_traits, model. User can modify any field.
3 Review Skills List of skills from profile_skill_assignment for that version. Toggle skills on/off. For each skill, system checks tool requirements against instance’s available tools and MCP servers. Missing requirements flagged with warnings.
4 Configure MCP For each profile_mcp_configuration, user must select or create a matching mcp_server in their instance (profiles specify server TYPE, not specific connections). Auto-selected if matching server exists.
5 Review Tools Direct tool assignments from profile_tool_assignment. User confirms or adjusts.
6 Create Single transaction creates: team_member (member_type=AGENT) + agent + agent_skill records (pinned to profile versions) + agent_mcp_assignment records + agent_tool_assignment records + profile_fork_record (provenance).
7 Verify & Activate Summary card showing the new agent, its skills, MCP connections, and tools. “Activate” button sets agent status to ACTIVE.

Dependencies

Production

react react-dom @tanstack/react-router @tanstack/react-query zustand recharts tailwindcss shadcn/ui lucide-react clsx date-fns

Development / Testing

vite typescript vitest @testing-library/react @testing-library/jest-dom msw @vitejs/plugin-react eslint prettier

Dockerfile (Multi-Stage Build)

Build stage compiles TypeScript and bundles with Vite. Serve stage uses Nginx to serve static files with proper SPA routing (fallback to index.html).

# Stage 1: Build
FROM node:22-slim AS build
WORKDIR /app
COPY package.json pnpm-lock.yaml ./
RUN corepack enable && pnpm install --frozen-lockfile
COPY . .
RUN pnpm build

# Stage 2: Serve
FROM nginx:alpine
COPY --from=build /app/dist /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf

# SPA routing: all routes fallback to index.html
# nginx.conf must include: try_files $uri $uri/ /index.html;

EXPOSE 80

Test Requirements

Vitest + Testing Library for all component and page tests. MSW (Mock Service Worker) for API mocking. No real API calls in tests.

Mandatory Test Coverage

  • Pages: Every page renders without errors with mocked data (11 page tests minimum)
  • Auth: Login redirect, token parsing, permission evaluation, logout cleanup
  • Type tables: Store loads all tables, byId lookup works, handles loading state
  • WebSocket: Connection, reconnection on close, event dispatch to query invalidation
  • SSE streaming: Token events append to message, tool_call events render ToolCallCard, error events show error state
  • Permissions: IAM policy evaluation (Allow, Deny, Deny-takes-precedence), UI elements conditionally render
  • Components: StatusBadge renders correct color/label from type table, KanbanBoard renders columns, ProjectCard shows health indicator
  • API client: qqq-client methods attach JWT, handle 401 (redirect to login), handle 403 (show forbidden)
  • Coverage target: 90%+

Acceptance Criteria

  • All 11 primary pages render correctly with real data from qqq API
  • Schedule Management: list view, detail/edit with cron picker, run history, + New Schedule form
  • Authentik OIDC login/logout flow works end-to-end (redirect, token, refresh)
  • Permission-driven UI: sidebar items and action buttons respect IAM policies from JWT
  • All 59 type tables load on startup and drive StatusBadge/PriorityBadge/HealthBadge rendering
  • WebSocket connection established with auto-reconnect; cache invalidation working on push events
  • SSE chat streaming works: tokens stream in real-time, tool calls visualized, thinking indicator shown
  • Dashboard shows live project health, team status, escalation summary, meeting schedule, cost burn rate
  • Project detail page: all 6 tabs render (overview, kanban, team, milestones, meetings, activity)
  • Team page: agent detail shows skills radar, memory explorer, tool assignments, execution history
  • Tool Forge: split-pane build mode works (chat left, code preview right, approve/reject actions)
  • Command palette (Cmd+K) searches across projects, agents, tasks, conversations, tools
  • Setup wizards: instance setup, project setup, and agent creation wizards complete with progress persistence
  • Agent creation wizard: 7-step flow (identity, role, skills, MCP, tools, model, review) with skill tool-requirement validation
  • Published profile fork workflow: fork from company catalog creates agent + all assignments in single transaction
  • Docker image builds (multi-stage: Node build + Nginx serve) and serves SPA correctly
  • All tests passing with 90%+ coverage (Vitest + Testing Library + MSW)
  • Responsive design works on mobile, tablet, and desktop viewports

CLAUDE.md Rules for Frontend Agents

These rules MUST be placed in the CLAUDE.md file at the root of the praesidium2-web/ repository.

# praesidium2-web CLAUDE.md

## Project Context
This is the React 19 SPA frontend for Praesidium2 -- the command center for
Greater Goods' AI-augmented workforce. It provides 11 primary pages for managing
projects, agents, conversations, tools, approvals, KPIs, schedules, and escalations.
Meetings and forge pages are deferred to V1.5.

## Architecture Rules
- Two API backends: qqq API (data CRUD) and Agent Gateway (AI interactions)
- Use TanStack Router for ALL routing (file-based, type-safe)
- Use TanStack Query for ALL server state (qqq API reads + mutations)
- Use Zustand ONLY for client-side state (auth, type tables, WebSocket status)
- NEVER store server state in Zustand -- that is TanStack Query's job
- All type table lookups go through the Zustand type-table store (cached at load)
- StatusBadge, PriorityBadge, HealthBadge MUST read from type tables -- never hardcode values
- WebSocket events MUST invalidate TanStack Query caches -- never manually refetch
- SSE streaming MUST use the useSSEStream hook -- never raw EventSource
- Schedule pages (/schedules, /schedules/:id) use qqq API for scheduled_job CRUD -- cron picker component validates cron expressions client-side

## Auth Rules
- JWT stored in memory ONLY (Zustand auth store) -- NEVER in localStorage or cookies
- Every API request MUST include Authorization: Bearer header
- On 401 response: clear auth store, redirect to Authentik login
- Permission checks use IAM-style policy evaluation (Deny takes precedence)
- UI elements MUST be permission-gated (sidebar items, action buttons)

## Coding Standards
- TypeScript strict mode -- no any types, no type assertions unless unavoidable
- All components are functional (no class components)
- Use shadcn/ui primitives as the base -- customize via Tailwind, don't fork
- pnpm for package management (not npm, not yarn)
- Vitest + Testing Library for all tests
- MSW for API mocking in tests -- no real API calls
- Target 90%+ test coverage

## What NOT to Do
- Do NOT install axios -- use fetch or the typed qqq-client/gateway-client
- Do NOT use Redux, MobX, or any state library besides TanStack Query + Zustand
- Do NOT create custom CSS files -- use Tailwind utility classes only
- Do NOT hardcode status labels, colors, or icons -- always use type table lookups
- Do NOT store JWTs in localStorage or sessionStorage
- Do NOT make API calls outside of TanStack Query hooks or mutation wrappers
- Do NOT skip writing tests -- every page and critical component needs tests

Local Dev Environment

Epic 8 — Days 1, 4

Day 1 deliverable. All other services depend on a working local dev environment.

Local development runs on the same k8s stack as every other environment. A local k8s cluster (Docker Desktop or OrbStack) provides the runtime. Tilt watches source directories and hot-deploys to a praesidium-local namespace via the same Helm charts from praesidium-cd used in dev/staging/prod. Ollama provides local LLM inference (replacing Anthropic API). Each service repo also has a docker/compose.yml for standalone local dev. Full design: docs/superpowers/specs/2026-04-16-local-development-design.md

Local Development Environment (k8s-native via Tilt)

A single tilt up builds all images from Dockerfiles in each service repo, deploys to the local k8s cluster via Helm charts from praesidium-cd, and live-reloads on source changes. The same Helm charts run across local/dev/staging/prod.

macOS host requirements: All tooling is macOS/Homebrew-based. A container runtime with a local Kubernetes cluster is required — either Docker Desktop (free for personal use; enable Kubernetes under Settings → Kubernetes → Enable Kubernetes) or OrbStack (brew install orbstack, lighter on RAM/CPU, Kubernetes included). The Tiltfile targets the docker-desktop context by default — update allow_k8s_contexts() if using OrbStack. A Java 21 JDK must be available on the host to build the qqq-api Maven image.

Prerequisites

ToolPurposeInstall
Docker Desktop or OrbStackContainer runtime + local k8s clusterDocker Desktop: enable Kubernetes in Settings; OrbStack: brew install orbstack
kubectlCluster management CLIBundled with Docker Desktop / OrbStack, or brew install kubectl
Helm 3Chart-based deployment (same charts as prod)brew install helm
TiltLive-reload orchestrator for local k8sbrew install tilt-dev/tap/tilt
Java 21 JDKRequired to build the qqq-api Maven imagebrew install openjdk@21
Node.js 20+Required for web Vite/React image buildbrew install node or via nvm
Python 3.11+Required for gateway / worker image buildsBundled in images; host Python needed only for pre-commit hooks

Dev Workflow

  • ./scripts/local-dev-setup.sh — one-time namespace + dependency setup
  • tilt up — builds images, deploys via Helm, watches for changes
  • Gateway + Worker: Tilt live_update syncs src/ into running pods, uvicorn/celery auto-reloads
  • Web: Tilt live_update syncs src/, Vite HMR inside the pod
  • qqq: Full image rebuild on change (Java requires recompile)
  • Authentik admin at https://localhost:9443/if/admin/ (port-forwarded)
  • Tilt UI at http://localhost:10350 — logs, build status, resource health

Port Forwards (automatic via Tilt)

ServiceLocal PortPod Port
qqq-api80808080
gateway80008000
web30003000
authentik94439443
postgresql54325432
redis63796379
minio9000 / 90019000 / 9001
ollama1143411434
jaeger1668616686

Tiltfile

The Tiltfile builds all service images from Dockerfiles in each service repo, deploys via Helm charts from praesidium-cd to the praesidium-local namespace, watches source directories for live-reload, and port-forwards key services.

# Tiltfile

# ── Configuration ──────────────────────────────────
load('ext://namespace', 'namespace_create')
namespace_create('praesidium-local')

allow_k8s_contexts('docker-desktop')

# ── Infrastructure Dependencies (Helm charts) ─────
# PostgreSQL (bitnami)
helm_resource(
  'postgresql',
  'oci://registry-1.docker.io/bitnamicharts/postgresql',
  flags=['--namespace=praesidium-local',
         '--set=auth.database=praesidium',
         '--set=auth.username=praesidium',
         '--set=auth.password=localdev',
         '--set=image.tag=16',
         '--set=primary.extendedConfiguration=shared_preload_libraries=vector'],
  port_forwards=['5432:5432'],
  labels=['infra']
)

# Redis (bitnami)
helm_resource(
  'redis',
  'oci://registry-1.docker.io/bitnamicharts/redis',
  flags=['--namespace=praesidium-local',
         '--set=auth.enabled=false',
         '--set=architecture=standalone'],
  port_forwards=['6379:6379'],
  labels=['infra']
)

# Authentik
helm_resource(
  'authentik',
  'oci://ghcr.io/goauthentik/helm/authentik',
  flags=['--namespace=praesidium-local',
         '--set=authentik.secret_key=local-dev-secret-key-change-in-prod',
         '--set=authentik.postgresql.host=postgresql',
         '--set=authentik.postgresql.user=praesidium',
         '--set=authentik.postgresql.password=localdev',
         '--set=authentik.postgresql.name=authentik',
         '--set=authentik.redis.host=redis-master'],
  port_forwards=['9443:9443'],
  resource_deps=['postgresql', 'redis'],
  labels=['infra']
)

# MinIO (S3-compatible object storage)
helm_resource(
  'minio',
  'oci://registry-1.docker.io/bitnamicharts/minio',
  flags=['--namespace=praesidium-local',
         '--set=auth.rootUser=praesidium',
         '--set=auth.rootPassword=localdevkey'],
  port_forwards=['9000:9000', '9001:9001'],
  labels=['infra']
)

# Ollama (local LLM inference)
helm_resource(
  'ollama',
  './charts/ollama',
  flags=['--namespace=praesidium-local'],
  port_forwards=['11434:11434'],
  labels=['llm']
)

# Jaeger (distributed tracing)
helm_resource(
  'jaeger',
  'jaegertracing/jaeger',
  flags=['--namespace=praesidium-local',
         '--set=allInOne.enabled=true'],
  port_forwards=['16686:16686'],
  labels=['observability']
)

# ── Application Services ──────────────────────────
# qqq-api (Java — full rebuild on change)
docker_build(
  'praesidium2-qqq',
  context='./praesidium2-qqq',
  dockerfile='./praesidium2-qqq/Dockerfile'
)

# gateway (Python — live_update for hot reload)
docker_build(
  'praesidium2-gateway',
  context='./praesidium2-python',
  dockerfile='./praesidium2-python/services/agent-gateway/Dockerfile',
  live_update=[
    sync('./praesidium2-python/services/agent-gateway/src', '/app/src'),
    run('cd /app && pip install -e .', trigger=['./praesidium2-python/services/agent-gateway/pyproject.toml']),
  ]
)

# worker (Python — live_update for hot reload)
docker_build(
  'praesidium2-worker',
  context='./praesidium2-python',
  dockerfile='./praesidium2-python/services/agent-worker/Dockerfile',
  live_update=[
    sync('./praesidium2-python/services/agent-worker/src', '/app/src'),
    run('cd /app && pip install -e .', trigger=['./praesidium2-python/services/agent-worker/pyproject.toml']),
  ]
)

# web (React — live_update for Vite HMR)
docker_build(
  'praesidium2-web',
  context='./praesidium2-web',
  dockerfile='./praesidium2-web/Dockerfile',
  live_update=[
    sync('./praesidium2-web/src', '/app/src'),
    run('cd /app && pnpm install', trigger=['./praesidium2-web/package.json']),
  ]
)

# tool-runner (Python — no live_update, runs as Job)
docker_build(
  'praesidium2-tool-runner',
  context='./praesidium2-tool-runner',
  dockerfile='./praesidium2-tool-runner/Dockerfile'
)

# ── Deploy via Helm (charts from praesidium-cd) ───
k8s_yaml(helm(
  './praesidium-cd',
  name='praesidium',
  namespace='praesidium-local',
  values=['./praesidium-cd/values-local.yaml']
))

# ── Port Forwards ─────────────────────────────────
k8s_resource('qqq-api',    port_forwards=['8080:8080'], resource_deps=['postgresql', 'authentik'], labels=['app'])
k8s_resource('gateway',    port_forwards=['8000:8000'], resource_deps=['qqq-api', 'redis', 'ollama'], labels=['app'])
k8s_resource('worker',     resource_deps=['qqq-api', 'redis', 'ollama'],                           labels=['app'])
k8s_resource('web',        port_forwards=['3000:3000'], resource_deps=['gateway'],                  labels=['app'])

Helm values-local.yaml

Local overrides for the same Helm charts used in dev/staging/prod. Single-replica services, local infrastructure dependencies, observability disabled.

# praesidium-cd/values-local.yaml
# Local k8s development overrides (used by Tiltfile)

global:
  imageRegistry: ""                # local images, no registry prefix
  imagePullPolicy: Never           # use locally-built images

instance:
  namespace: praesidium-local
  id: "local-dev"

# ── Per-service: single replica, local images ─────
qqq-api:
  replicas: 1
  image: praesidium2-qqq
  tag: latest
  resources:
    requests: { cpu: 250m, memory: 512Mi }
    limits:   { cpu: 1000m, memory: 2Gi }
  env:
    SPRING_PROFILES_ACTIVE: local
    SPRING_DATASOURCE_URL: jdbc:postgresql://postgresql:5432/praesidium
    SPRING_DATASOURCE_USERNAME: praesidium
    SPRING_DATASOURCE_PASSWORD: localdev
    AUTHENTIK_ISSUER_URL: https://authentik:9443/application/o/praesidium/
    AUTHENTIK_JWKS_URL: https://authentik:9443/application/o/praesidium/jwks/

gateway:
  replicas: 1
  image: praesidium2-gateway
  tag: latest
  resources:
    requests: { cpu: 100m, memory: 256Mi }
    limits:   { cpu: 500m, memory: 1Gi }
  env:
    QQQ_API_URL: http://qqq-api:8080
    REDIS_URL: redis://redis-master:6379/0
    MINIO_ENDPOINT: http://minio:9000
    MINIO_ACCESS_KEY: praesidium
    MINIO_SECRET_KEY: localdevkey
    AUTHENTIK_ISSUER_URL: https://authentik:9443/application/o/praesidium/
    LLM_PROVIDER: ollama
    LLM_BASE_URL: http://ollama:11434
    LLM_MODEL: llama3.1:70b

worker:
  replicas: 1
  image: praesidium2-worker
  tag: latest
  hpa:
    enabled: false
  resources:
    requests: { cpu: 250m, memory: 512Mi }
    limits:   { cpu: 1000m, memory: 2Gi }
  env:
    QQQ_API_URL: http://qqq-api:8080
    REDIS_URL: redis://redis-master:6379/0
    MINIO_ENDPOINT: http://minio:9000
    CELERY_BROKER_URL: redis://redis-master:6379/1
    CELERY_RESULT_BACKEND: redis://redis-master:6379/2
    LLM_PROVIDER: ollama
    LLM_BASE_URL: http://ollama:11434
    LLM_MODEL: llama3.1:70b

web:
  replicas: 1
  image: praesidium2-web
  tag: latest
  resources:
    requests: { cpu: 50m, memory: 64Mi }
    limits:   { cpu: 100m, memory: 128Mi }
  env:
    VITE_API_URL: http://localhost:8080
    VITE_GATEWAY_URL: http://localhost:8000
    VITE_AUTHENTIK_URL: https://localhost:9443

# ── Infrastructure (deployed by Tilt helm_resource) ─
postgresql:
  enabled: false       # managed by Tilt helm_resource
redis:
  enabled: false       # managed by Tilt helm_resource
authentik:
  enabled: false       # managed by Tilt helm_resource

# ── Observability (disabled locally) ──────────────
datadog:
  enabled: false
  apm: false
  rum: false

Local Setup Script

./scripts/local-dev-setup.sh — one-time setup for local k8s development. Idempotent (safe to re-run).

#!/usr/bin/env bash
set -euo pipefail

# scripts/local-dev-setup.sh
# One-time local k8s development environment setup

echo "==> Verifying prerequisites..."
command -v docker >/dev/null  || { echo "ERROR: Docker not found"; exit 1; }
command -v kubectl >/dev/null || { echo "ERROR: kubectl not found"; exit 1; }
command -v helm >/dev/null    || { echo "ERROR: Helm not found"; exit 1; }
command -v tilt >/dev/null    || { echo "ERROR: Tilt not found"; exit 1; }

# Verify Docker Desktop k8s is running
kubectl cluster-info --context docker-desktop >/dev/null 2>&1 \
  || { echo "ERROR: Docker Desktop Kubernetes not running. Enable it in Docker Desktop settings."; exit 1; }

echo "==> Setting kubectl context to docker-desktop..."
kubectl config use-context docker-desktop

echo "==> Creating namespace praesidium-local..."
kubectl create namespace praesidium-local --dry-run=client -o yaml | kubectl apply -f -

echo "==> Adding Helm repos..."
helm repo add bitnami https://charts.bitnami.com/bitnami 2>/dev/null || true
helm repo add authentik https://charts.goauthentik.io 2>/dev/null || true
helm repo update

echo "==> Setup complete. Run 'tilt up' to start the development environment."
echo "    Tilt UI will be available at http://localhost:10350"

Multi-Stage Dockerfiles (in each service repo)

Each service repo owns its own Dockerfile with a multi-stage build for minimal image size and layer caching. Faber uses these for CI builds; Tilt uses them for local k8s dev.

qqq-api (Java)

FROM eclipse-temurin:21-jdk AS build
WORKDIR /app
COPY build.gradle.kts settings.gradle.kts ./
COPY gradle/ gradle/
RUN ./gradlew dependencies --no-daemon
COPY src/ src/
RUN ./gradlew bootJar --no-daemon

FROM eclipse-temurin:21-jre
COPY --from=build /app/build/libs/*.jar app.jar
EXPOSE 8080
ENTRYPOINT ["java", "-jar", "app.jar"]

gateway / worker (Python)

FROM python:3.12-slim AS base
WORKDIR /app
RUN pip install uv
COPY pyproject.toml uv.lock ./
RUN uv sync --frozen --no-dev

FROM python:3.12-slim
WORKDIR /app
COPY --from=base /app/.venv /app/.venv
COPY src/ src/
ENV PATH="/app/.venv/bin:$PATH"

# gateway:
EXPOSE 8000
CMD ["uvicorn", "src.main:app", \
     "--host", "0.0.0.0", "--port", "8000"]

# worker:
CMD ["celery", "-A", "src.worker", "worker", \
     "--loglevel=info", "--concurrency=4"]

web (React SPA)

FROM node:20-alpine AS build
WORKDIR /app
COPY package.json pnpm-lock.yaml ./
RUN corepack enable && pnpm install --frozen-lockfile
COPY . .
RUN pnpm build

FROM nginx:alpine
COPY --from=build /app/dist /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 80

tool-runner (Sandbox)

FROM python:3.12-slim
WORKDIR /sandbox
RUN pip install uv
COPY tool_runner/ tool_runner/
COPY requirements.txt .
RUN uv pip install -r requirements.txt

# No EXPOSE — runs as k8s Job
# Entry point set dynamically via Job spec
ENTRYPOINT ["python", "-m", "tool_runner"]

Kubernetes Manifests

Each service gets a Deployment, Service, ConfigMap, and NetworkPolicy. Worker pods get HPA and PDB for production resilience.

Per-Service Resource Matrix

ServiceReplicas (dev / prod)CPU RequestMemory RequestCPU LimitMemory Limit
qqq-api1 / 2500m1Gi1000m2Gi
gateway1 / 2250m512Mi500m1Gi
worker1 / 2-8 (HPA)500m1Gi2000m4Gi
web1 / 250m64Mi100m128Mi
tool-runnerJob (on-demand)Varies by risk_level512Mi-4Gi500m-2000m512Mi-4Gi

Example: Gateway Deployment + Service

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gateway
  namespace: {{ .Values.instance.namespace }}
spec:
  replicas: {{ .Values.gateway.replicas }}
  selector:
    matchLabels:
      app: gateway
  template:
    metadata:
      labels:
        app: gateway
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: gateway
          image: {{ .Values.gateway.image }}:{{ .Values.gateway.tag }}
          ports:
            - containerPort: 8000
          envFrom:
            - configMapRef:
                name: gateway-config
            - secretRef:
                name: gateway-secrets
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8000
            initialDelaySeconds: 10
            periodSeconds: 15
          readinessProbe:
            httpGet:
              path: /readyz
              port: 8000
            initialDelaySeconds: 5
            periodSeconds: 10
          resources:
            requests:
              cpu: {{ .Values.gateway.resources.requests.cpu }}
              memory: {{ .Values.gateway.resources.requests.memory }}
            limits:
              cpu: {{ .Values.gateway.resources.limits.cpu }}
              memory: {{ .Values.gateway.resources.limits.memory }}
---
apiVersion: v1
kind: Service
metadata:
  name: gateway
spec:
  selector:
    app: gateway
  ports:
    - port: 8000
      targetPort: 8000

NetworkPolicy (restrict pod-to-pod)

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: gateway-netpol
spec:
  podSelector:
    matchLabels:
      app: gateway
  policyTypes: [Ingress, Egress]
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: web           # frontend can reach gateway
      ports:
        - port: 8000
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: qqq-api       # gateway calls qqq
      ports:
        - port: 8080
    - to:
        - podSelector:
            matchLabels:
              app: redis
      ports:
        - port: 6379

PodDisruptionBudget (production)

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: qqq-api-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: qqq-api
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: gateway-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: gateway

Helm Chart Structure (in praesidium-cd)

Umbrella chart with sub-charts per service. Lives in the praesidium-cd repo (not owned by Praesidium2 team). Environment-specific values files override defaults.

praesidium-cd/ charts/ qqq-api/ Chart.yaml values.yaml templates/ deployment.yaml service.yaml configmap.yaml secret.yaml networkpolicy.yaml pdb.yaml agent-gateway/ Chart.yaml values.yaml templates/ # same template set agent-worker/ Chart.yaml values.yaml templates/ deployment.yaml hpa.yaml # HPA for workers pdb.yaml web/ Chart.yaml values.yaml templates/ Chart.yaml # umbrella chart values.yaml # defaults values-local.yaml # local k8s overrides (Tilt) values-dev.yaml # AWS dev overrides values-staging.yaml values-prod.yaml instances/ tech-team.yaml # per-instance values hr.yaml marketing.yaml product.yaml

values.yaml (defaults)

# Global
global:
  imageRegistry: ""          # ghcr.io/dmdbrands in prod
  imagePullPolicy: IfNotPresent

instance:
  namespace: praesidium-dev
  id: ""                     # set per-instance

# Per-service
qqq-api:
  replicas: 1
  image: praesidium2-qqq
  tag: latest
  resources:
    requests: { cpu: 500m, memory: 1Gi }
    limits:   { cpu: 1000m, memory: 2Gi }
  env:
    SPRING_PROFILES_ACTIVE: dev

gateway:
  replicas: 1
  image: praesidium2-gateway
  tag: latest
  resources:
    requests: { cpu: 250m, memory: 512Mi }
    limits:   { cpu: 500m, memory: 1Gi }

worker:
  replicas: 1
  image: praesidium2-worker
  tag: latest
  hpa:
    enabled: false
    minReplicas: 2
    maxReplicas: 8
    targetCPU: 70
  resources:
    requests: { cpu: 500m, memory: 1Gi }
    limits:   { cpu: 2000m, memory: 4Gi }

web:
  replicas: 1
  image: praesidium2-web
  tag: latest
  resources:
    requests: { cpu: 50m, memory: 64Mi }
    limits:   { cpu: 100m, memory: 128Mi }

Per-Instance Provisioning

Each business unit gets its own k8s namespace. ArgoCD ApplicationSet generates one Application per instance YAML file in the instances/ directory.

Instance YAML Example (tech-team.yaml)

# instances/tech-team.yaml
instance:
  id: "tech-team"
  namespace: "praesidium-tech-team"
  displayName: "CTO Engineering"

# Instance-specific overrides
worker:
  replicas: 2
  hpa:
    enabled: true
    minReplicas: 2
    maxReplicas: 6

# Instance-specific env vars (merged with defaults)
env:
  INSTANCE_ID: "uuid-for-tech-team"
  AUTHENTIK_GROUP: "tech-team"
  DEFAULT_LLM_MODEL: "claude-sonnet-4-6"
  DAILY_BUDGET_USD: "50.00"

Adding a New Business Unit Instance

  • Create a new YAML file in instances/ (e.g., finance.yaml)
  • Create Authentik company group with matching praesidiumInstanceId attribute
  • Commit and push — ArgoCD ApplicationSet auto-creates the Application
  • ArgoCD syncs: creates namespace, deploys all services, applies instance-specific values

Acceptance Criteria

  • tilt up deploys all services to local k8s with zero manual steps
  • All health checks pass within 90 seconds of startup
  • Live-reload works for gateway, worker (Python via live_update), and web (Vite HMR via live_update)
  • Helm chart installs cleanly on local k8s (Docker Desktop with Kubernetes enabled)
  • Same Helm charts from praesidium-cd used in local/dev/staging/prod (only values-*.yaml differs)
  • Each service has Deployment, Service, ConfigMap, NetworkPolicy, and PDB manifests
  • Worker HPA scales pods based on CPU utilization
  • Instance YAML in instances/ creates a fully isolated namespace deployment
  • All Docker images build in <3 minutes with layer caching
  • ./scripts/local-dev-setup.sh is idempotent and validates all prerequisites

CLAUDE.md Rules for Local Dev / k8s / CI

# Per-Service Repo CLAUDE.md Rules

## CI/CD Pattern (each service repo)
Each service repo has:
- .faber.yml at root (pipeline: java-webapp, python-webapp, or node-webapp)
- Dockerfile (or docker/Dockerfile) — owned by this repo
- docker/compose.yml for local dev
- Faber handles: lint, test, Docker build, push to GHCR, update CD repo Helm values
- Helm charts and ArgoCD config live in praesidium-cd (NOT owned by Praesidium2 team)
- Docker registry: ghcr.io/dmdbrands

## Architecture Rules
- Local dev uses Tilt + Docker Desktop k8s — Helm charts from praesidium-cd
- Every service MUST have a health check in k8s manifests
- Every service MUST have resource requests AND limits in k8s
- NetworkPolicy MUST be defined for every pod (default deny, explicit allow)
- PDB MUST be defined for qqq-api, gateway, and worker
- Worker HPA MUST scale on CPU — never scale to zero
- Instance YAML in instances/ MUST NOT contain secrets (use Secrets Manager refs)

## What NOT to Do
- Do NOT hardcode image tags — always use Helm values
- Do NOT use hostPath volumes in k8s manifests
- Do NOT expose services via NodePort — always use ClusterIP + Ingress
- Do NOT skip multi-stage Docker builds
- Do NOT put credentials in ConfigMaps — use Secrets
- Do NOT modify praesidium-cd or praesidium-infrastructure repos directly

CI/CD (Faber Pipeline)

Epic 8 — Days 3, 5

No PR merges without passing CI. Faber pipeline blocks on any test failure or coverage drop.

Faber is Greater Goods' CI/CD platform. Each service repo has a .faber.yml at its root (pipeline type: java-webapp, python-webapp, or node-webapp). Faber handles: lint, test, Docker build, push to GHCR (ghcr.io/dmdbrands), and update CD repo Helm values. Pipelines run Lint, Test, Build, Push, and GitOps Update stages in order.

Pipeline Stages

Every push to any branch triggers the pipeline. Only develop and main branches push Docker images and update GitOps values.

1
Lint
Code quality
→
2
Test
Unit + Integration
→
3
Build
Docker image
→
4
Push
GHCR registry
→
5
GitOps
Update values

Per-Language Pipeline Configuration

Python, Java, and TypeScript services each have language-specific lint and test tooling. All three run in parallel during the Lint + Test stages.

Python (Gateway + Worker)

Lintruff check . + ruff format --check .
Type Checkmypy src/
Unit Testspytest tests/unit/ -v --cov=src --cov-report=xml
Integrationpytest tests/integration/ -v --cov-append
Coverage Gate90%
Securitypip-audit + semgrep --config=auto

Java (qqq API)

Lint./gradlew checkstyleMain
Unit Tests./gradlew test
Integration./gradlew integrationTest (Testcontainers)
Coverage Gate90%
Security./gradlew dependencyCheckAnalyze

TypeScript (Web SPA)

Linteslint . && prettier --check .
Type Checktsc --noEmit
Unit Testsvitest run --coverage
E2E Testsplaywright test (headless Chromium)
Coverage Gate90%
Securitypnpm audit + semgrep

Docker Build + GHCR Push

Multi-stage builds with Docker BuildKit layer caching. Each service repo owns its own Dockerfile. Images are pushed to ghcr.io/dmdbrands and tagged with both git SHA and environment label.

Build Configuration

# Faber build stage (per service, Dockerfile in each service repo)
docker buildx build \
  --cache-from type=registry,ref=$GHCR_REPO:cache \
  --cache-to type=registry,ref=$GHCR_REPO:cache,mode=max \
  --tag $GHCR_REPO:$GIT_SHA \
  --tag $GHCR_REPO:$ENVIRONMENT \
  --push \
  -f Dockerfile .
ServiceGHCR RepositoryAvg Build Time
qqq-apighcr.io/dmdbrands/praesidium/qqq-api~90s
gatewayghcr.io/dmdbrands/praesidium/gateway~45s
workerghcr.io/dmdbrands/praesidium/worker~45s
webghcr.io/dmdbrands/praesidium/web~60s
tool-runnerghcr.io/dmdbrands/praesidium/tool-runner~30s

Image Tagging Strategy

Tag PatternWhen AppliedExample
{git-sha}Every builda1b2c3d
developPush to developLatest develop build
stagingPush to stagingStaging candidate
latestPush to mainProduction release
v{semver}Git tagv1.0.0

GHCR Image Retention

  • Keep last 30 untagged images (cleanup old SHA tags)
  • Keep all tagged images (latest, develop, staging, v*)
  • Trivy scan on push for CVE vulnerabilities

GitOps Update (Helm Values in praesidium-cd)

After a successful image push to GHCR, Faber commits the new image tag to the praesidium-cd GitOps repository. ArgoCD watches this repo and auto-syncs. Helm charts also live in praesidium-cd.

# Faber GitOps update step (runs after GHCR push)
git clone $GITOPS_REPO /tmp/gitops  # praesidium-cd
cd /tmp/gitops

# Determine target values file from branch
case "$BRANCH" in
  develop)  VALUES_FILE="values-dev.yaml" ;;
  staging)  VALUES_FILE="values-staging.yaml" ;;
  main)     VALUES_FILE="values-prod.yaml" ;;
esac

# Update image tags for the changed service
yq eval ".${SERVICE}.tag = \"${GIT_SHA}\"" -i "$VALUES_FILE"

git add "$VALUES_FILE"
git commit -m "ci(${SERVICE}): update to ${GIT_SHA} [skip ci]"
git push origin main

Branch Strategy + Promotion Flow

  • feature/* → Lint + Test only (no image push, no deploy)
  • develop → Lint + Test + Build + Push + Deploy to dev (auto-sync)
  • staging → Lint + Test + Build + Push + Deploy to staging (auto-sync)
  • main → Lint + Test + Build + Push + Update prod values (manual ArgoCD sync)
  • PRs require: passing CI, 1 approval, no unresolved comments
  • Hotfix: branch from main, PR directly to main, cherry-pick back to develop

CI Time Budget

Target: full pipeline completes in under 5 minutes. Parallelism is key.

StageDuration TargetParallel?Notes
Lint<30sYes (all 3 languages)Fastest to fail, runs first
Test<3 minYes (Python, Java, TS in parallel)Testcontainers adds ~15s startup
Build<90sYes (all images in parallel)BuildKit cache cuts rebuilds to ~20s
Push<30sYesOnly changed service images
GitOps<10sNoSingle commit to values file

Faber Pipeline Configuration

Each service repository includes a .faber.yml configuration file that Faber reads to determine build, test, and deployment behavior.

# praesidium2-qqq/.faber.yml
pipeline: java-webapp
image_name: praesidium2-qqq
java_version: "21"
docker:
  registry: ghcr.io/dmdbrands
cd:
  repo: dmdbrands/praesidium-cd
  format: helm
coverage:
  min_instruction: "90"
health:
  path: /qqq-api/health
  port: 8000
  db: true
services:
  - image: pgvector/pgvector:pg17
    port: 5432
    healthcheck: "pg_isready -U test"
    env:
      POSTGRES_USER: test
      POSTGRES_PASSWORD: testing
      POSTGRES_DB: praesidium_test

Each repo has its own .faber.yml. The pipeline type varies: python-webapp for praesidium2-python, node-webapp for praesidium2-web. The pattern is the same — image name, registry, CD repo, coverage gates, and health checks — adjusted per language and service.

Acceptance Criteria

  • Pipeline runs lint + test + build for all 3 languages in parallel
  • Coverage gates enforced: Python 90%, Java 90%, TypeScript 90%
  • Docker images pushed to GHCR (ghcr.io/dmdbrands) tagged with git SHA + environment
  • GitOps repo auto-updated with new image tags on develop/staging/main
  • Feature branches run lint + test only (no image push)
  • Full pipeline completes in <5 minutes
  • PRs blocked on failing CI
  • Security scans (dependency audit, Semgrep, container scan) run on every build
  • .faber.yml configuration template committed to repository root

ArgoCD GitOps

Epic 8 — Day 4

ArgoCD is the ONLY way services get deployed to Kubernetes. No manual kubectl apply.

ArgoCD watches the GitOps repository and reconciles cluster state with the declared Helm values. One ApplicationSet generates an Application per business unit instance.

ApplicationSet Configuration

A single ApplicationSet resource scans the instances/ directory in the GitOps repo. Each YAML file generates a full Praesidium2 deployment in its own namespace.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: praesidium-instances
  namespace: argocd
spec:
  generators:
    - git:
        repoURL: https://github.com/dmdbrand/praesidium-cd.git
        revision: main
        files:
          - path: "instances/*.yaml"
  template:
    metadata:
      name: "praesidium-{{instance.id}}"
      labels:
        app.kubernetes.io/part-of: praesidium
        praesidium.io/instance: "{{instance.id}}"
    spec:
      project: praesidium
      source:
        repoURL: https://github.com/dmdbrand/praesidium-cd.git
        targetRevision: main
        path: charts/praesidium
        helm:
          valueFiles:
            - values.yaml                         # base defaults
            - "values-{{instance.environment}}.yaml"  # env overrides
            - "../../instances/{{instance.id}}.yaml"   # instance overrides
      destination:
        server: https://kubernetes.default.svc
        namespace: "{{instance.namespace}}"
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
          - PrunePropagationPolicy=foreground
        retry:
          limit: 3
          backoff:
            duration: 5s
            factor: 2
            maxDuration: 1m

How Instance Discovery Works

  • ArgoCD scans instances/*.yaml in the GitOps repo
  • Each file is parsed as Helm values — the instance.id and instance.namespace fields drive Application naming
  • Adding a new file = adding a new business unit. ArgoCD auto-creates the Application within seconds.
  • Removing a file = tearing down an instance (prune: true deletes the namespace resources)

Sync Policy by Environment

Dev and staging auto-sync on every commit. Production requires manual approval via the ArgoCD UI or CLI.

EnvironmentAuto-SyncSelf-HealPruneApproval
dev Enabled Enabled Enabled None — fully automatic
staging Enabled Enabled Enabled None — fully automatic
prod Disabled Enabled Disabled Manual sync via ArgoCD UI + Slack confirmation

Production Deployment Process

  • CI pushes image and updates values-prod.yaml in the GitOps repo
  • ArgoCD detects the diff and marks the Application as OutOfSync
  • Slack bot posts a deployment notification to #praesidium-deploys with diff summary
  • On-call engineer reviews the diff in ArgoCD UI and clicks Sync
  • ArgoCD performs a rolling update with health check verification
  • Post-sync notification posted to Slack with success/failure status

Rollback Procedure

Rollbacks are git operations, not kubectl commands. Reverting the values file triggers ArgoCD to automatically restore the previous version.

Automated Rollback (GitOps)

# Revert the last values commit
cd praesidium-cd/
git revert HEAD --no-edit
git push origin main

# ArgoCD detects the revert commit
# → Auto-syncs dev/staging immediately
# → Prod: marks OutOfSync, awaits manual sync

ArgoCD restores the previous image tag. Kubernetes performs a rolling update back to the prior version. Total rollback time: ~60s for dev, ~2 min for prod (after manual approval).

Emergency Rollback (ArgoCD UI)

  • Open ArgoCD UI → select the Application
  • Click History and Rollback
  • Select the previous healthy revision
  • Click Rollback — ArgoCD syncs to that revision
  • Follow up with a git revert to keep GitOps repo in sync

Important

Always follow up an ArgoCD UI rollback with a git revert. Otherwise ArgoCD will mark the Application as OutOfSync on the next reconciliation loop.

Health Check Verification

ArgoCD verifies each service is healthy after every sync. A sync is only marked Healthy when all pods pass readiness probes.

ServiceReadiness ProbeStartup TimeoutArgoCD Degraded After
qqq-apiGET /readyz (DB migrations complete)90s120s
gatewayGET /readyz (qqq API reachable, Redis connected)30s60s
workerCelery inspect active (at least 1 worker responding)60s90s
webGET / (Nginx serves static files)10s30s

Sync Failure Behavior

  • If a pod fails readiness within the timeout, ArgoCD marks the sync as Degraded
  • The syncPolicy.retry configuration retries up to 3 times with exponential backoff
  • After all retries fail, the Application stays in Degraded state — Slack alert fires
  • Self-heal ensures any manual kubectl changes get reverted to match git state

Notifications (Slack Integration)

ArgoCD Notifications controller sends deploy status to Slack. Every sync event triggers a notification.

# argocd-notifications-cm ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
data:
  service.slack: |
    token: $slack-token
  trigger.on-sync-succeeded: |
    - when: app.status.operationState.phase in ['Succeeded']
      send: [slack-deploy-success]
  trigger.on-sync-failed: |
    - when: app.status.operationState.phase in ['Error', 'Failed']
      send: [slack-deploy-failure]
  trigger.on-health-degraded: |
    - when: app.status.health.status == 'Degraded'
      send: [slack-health-degraded]
  template.slack-deploy-success: |
    slack:
      channel: praesidium-deploys
      blocks: |
        - type: section
          text:
            type: mrkdwn
            text: |
              :white_check_mark: *{{.app.metadata.name}}* deployed successfully
              *Revision:* {{.app.status.sync.revision | truncate 7}}
              *Environment:* {{.app.spec.destination.namespace}}
  template.slack-deploy-failure: |
    slack:
      channel: praesidium-deploys
      blocks: |
        - type: section
          text:
            type: mrkdwn
            text: |
              :x: *{{.app.metadata.name}}* deploy FAILED
              *Revision:* {{.app.status.sync.revision | truncate 7}}
              *Error:* {{.app.status.operationState.message}}
EventSlack ChannelMention
Sync Succeeded#praesidium-deploysNone
Sync Failed#praesidium-deploys@oncall-eng
Health Degraded#praesidium-alerts@oncall-eng
Prod Pending Sync#praesidium-deploys@james

Acceptance Criteria

  • ApplicationSet generates one Application per instance YAML file
  • Adding a new instance YAML auto-creates the full namespace deployment
  • Dev/staging auto-sync within 30 seconds of a values commit
  • Production requires manual sync approval
  • Rollback via git revert restores previous version within 60 seconds
  • Health checks verified for all pods before sync marked Healthy
  • Slack notifications fire on sync success, failure, and health degradation
  • Self-heal reverts manual kubectl changes within the reconciliation loop (default 3 min)

AWS Setup

Epic 8 — Phase 5

Phase 5 deliverable. AWS deployment leverages existing EKS cluster + Faber CI/CD + ArgoCD. Production deploy in Phase 5 (Engineering team).

AWS services replace local k8s equivalents: RDS for PostgreSQL, ElastiCache for Redis, S3 for MinIO. EKS runs all application pods. All services are HIPAA-eligible.

EKS Cluster Configuration

Managed Kubernetes cluster with separate node groups for application workloads and tool execution sandboxes.

Cluster Settings

Kubernetes Version1.29+
Regionus-east-1
VPCDedicated VPC with private subnets across 3 AZs
Control Plane LoggingAPI, Audit, Authenticator logs to CloudWatch
EncryptionSecrets encrypted at rest with KMS
Add-onsCoreDNS, kube-proxy, VPC CNI, EBS CSI, Metrics Server

Node Groups

Node GroupInstance TypeMin / MaxPurpose
app-nodest3.xlarge (4 vCPU, 16 GB)2 / 6qqq-api, gateway, worker, web
tool-nodest3.medium (2 vCPU, 4 GB)0 / 10tool-runner Jobs (scale to zero)

Autoscaling

  • app-nodes: Cluster Autoscaler scales based on pending pod requests
  • tool-nodes: Scale from 0 — tool-runner Jobs trigger node provisioning (~60s cold start)
  • Spot instances for tool-nodes (cost savings, Jobs are short-lived and retryable)

RDS PostgreSQL + pgvector

Managed PostgreSQL with pgvector extension for embedding storage. Multi-AZ for production resilience.

EnginePostgreSQL 17.x
Instance Classdb.r6g.large (2 vCPU, 16 GB)
Storage100 GB gp3 (auto-scaling to 500 GB)
Multi-AZYes (synchronous replication)
Read Replica1 replica (for memory search, activity feeds)
EncryptionAES-256 at rest (KMS), TLS 1.3 in transit
BackupsAutomated daily snapshots (35-day retention) + PITR
Extensionsvector, pg_stat_statements, pg_trgm

RDS Proxy (Connection Pooling)

PurposePool DB connections for worker pods (many short connections)
Max Connections200 (RDS limit), Proxy multiplexes to ~50 backend conns
IAM AuthEnabled — pods use IRSA to authenticate to Proxy
TLSRequired for all connections

Connection Routing

  • qqq-api: Connects to RDS Proxy endpoint (read/write)
  • gateway: Connects to read replica endpoint for heavy read queries
  • worker: Connects to RDS Proxy (high connection churn)
  • LangGraph checkpoints: Same RDS instance, separate schema

ElastiCache Redis

Two separate Redis clusters for workload isolation: one for task queues (durable) and one for caching (ephemeral).

ClusterInstance TypeNodesPersistenceEvictionPurpose
Queue cache.r6g.large Primary + 1 replica (Multi-AZ) AOF (appendfsync everysec) noeviction Celery broker, task queue. Data must not be lost.
Cache cache.r6g.medium Primary + 1 replica (Multi-AZ) None (RDB snapshots daily) allkeys-lru WebSocket pub/sub, session cache, rate limiting, agent hydration cache

Redis Connection Configuration

  • Both clusters: encryption at rest (AES-256) + encryption in transit (TLS 1.3)
  • Auth: Redis AUTH token stored in Secrets Manager, injected as env var
  • Monitoring: CloudWatch alarms on memory >80%, replication lag >1s, eviction rate spike
  • Automatic failover: <30s when primary node fails

S3 Buckets

Three purpose-specific buckets with versioning, encryption, and lifecycle policies.

BucketPurposeVersioningEncryptionLifecycle
praesidium-artifacts-{env} Task artifacts, reports, generated files Enabled SSE-S3 (AES-256) Move to Glacier after 1 year, delete after 3 years
praesidium-tools-{env} CUSTOM tool source code, built packages Enabled SSE-KMS Keep all versions (no expiration)
praesidium-knowledge-{env} Knowledge docs (uploads, Confluence mirrors) Enabled SSE-KMS Move to IA after 6 months, Glacier after 2 years

S3 Access Controls

  • Block all public access (account-level setting)
  • Bucket policies restrict to EKS pod roles only
  • Access logging enabled to a separate audit bucket
  • Cross-region replication for praesidium-tools (critical data)

Local Dev Equivalent

  • MinIO container maps 1:1 to S3 API
  • Three buckets auto-created via init script on first tilt up
  • Same SDK code works for both (AWS SDK S3 client with endpoint override for MinIO)

Secrets Manager

All secrets stored in AWS Secrets Manager with hierarchical naming and automatic rotation policies.

Secret Organization Structure

praesidium/
  global/                              # Shared across all instances
    authentik-admin-password
    ecr-pull-credentials
    argocd-slack-token
  {instance-id}/                       # Per-instance secrets
    db-credentials                     # RDS username + password
    redis-auth-token                   # ElastiCache AUTH
    authentik-oidc-client-secret       # OAuth2 client secret
    service-account-jwt                # Worker service identity
    secrets/                           # User-managed secrets (via secret_ref table)
      ANTHROPIC_API_KEY
      OPENAI_API_KEY
      SLACK_BOT_TOKEN
      JIRA_API_TOKEN
      GITHUB_PAT
      GOOGLE_SERVICE_ACCOUNT_JSON
      ZOHO_OAUTH_REFRESH_TOKEN
      HUBSPOT_API_KEY

Rotation Policies

SecretRotationMethod
DB credentialsEvery 30 daysRDS-native rotation Lambda
Redis AUTHEvery 90 daysCustom Lambda + ElastiCache modify
Service account JWTEvery 24 hoursAuthentik token refresh (80% of expiry)
API keys (user-managed)Alert at 30/7/1 day before expires_atManual (escalation created)

Pod Access to Secrets

  • IRSA (IAM Roles for Service Accounts) for pod-level access
  • Each service's k8s ServiceAccount maps to an IAM role
  • IAM role grants secretsmanager:GetSecretValue for specific secret paths
  • Secrets injected as env vars via external-secrets-operator or CSI driver
  • Never mounted as files (prevents accidental logging)

ALB / Route53

Application Load Balancer routes traffic to services. Route53 manages DNS per instance.

ALB Configuration

TypeApplication Load Balancer (Layer 7)
TLSACM certificate, TLS 1.3 termination
Ingress ControllerAWS Load Balancer Controller (k8s Ingress annotations)
WAFAWS WAF attached (OWASP Core Rule Set)
Access LogsS3 bucket for ALB access logs

Route53 DNS (Per Instance)

RecordTarget
{instance}.praesidium.dmdbrand.comALB (web SPA)
api.{instance}.praesidium.dmdbrand.comALB → qqq-api
ws.{instance}.praesidium.dmdbrand.comALB → gateway (WebSocket)
auth.praesidium.dmdbrand.comALB → Authentik
# Kubernetes Ingress (per instance, via Helm template)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: praesidium-ingress
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:ACCOUNT:certificate/CERT-ID
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/wafv2-acl-arn: arn:aws:wafv2:us-east-1:ACCOUNT:regional/webacl/praesidium-waf
spec:
  rules:
    - host: "{{.Values.instance.id}}.praesidium.dmdbrand.com"
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port: { number: 80 }
    - host: "api.{{.Values.instance.id}}.praesidium.dmdbrand.com"
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: qqq-api
                port: { number: 8080 }
    - host: "ws.{{.Values.instance.id}}.praesidium.dmdbrand.com"
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: gateway
                port: { number: 8000 }

CloudWatch Monitoring

Centralized logging and metrics via CloudWatch. Fluent Bit DaemonSet ships container logs. Container Insights provides cluster-level metrics.

Log Groups

Log GroupSourceRetention
/praesidium/qqq-apiqqq-api pods90 days
/praesidium/gatewaygateway pods90 days
/praesidium/workerworker pods90 days
/praesidium/tool-runnertool-runner Jobs30 days
/praesidium/webweb pods (Nginx)30 days
/praesidium/auditaudit_access_log records7 years (HIPAA)

CloudWatch Alarms

AlarmThresholdAction
RDS CPU>80% for 5 minSNS → Slack alert
RDS Free Storage<10 GBSNS → PagerDuty
Redis Memory>80%SNS → Slack alert
EKS Pod Restarts>3 in 5 minSNS → Slack alert
ALB 5xx Rate>1% for 5 minSNS → PagerDuty
ALB Latency p99>5sSNS → Slack alert

IAM Roles (IRSA)

Each Kubernetes ServiceAccount maps to an IAM role via IRSA (IAM Roles for Service Accounts). Least-privilege access to AWS services.

Service AccountIAM RolePermissions
qqq-api-sa praesidium-qqq-api-role rds-db:connect (RDS Proxy IAM auth), secretsmanager:GetSecretValue (db-credentials)
gateway-sa praesidium-gateway-role secretsmanager:GetSecretValue (redis-auth, oidc-secret, user-managed secrets), s3:GetObject/PutObject (artifacts, knowledge)
worker-sa praesidium-worker-role secretsmanager:GetSecretValue (all instance secrets), s3:* (artifacts, tools, knowledge), batch:SubmitJob (tool-runner), rds-db:connect (for LangGraph checkpoints)
tool-runner-sa praesidium-tool-runner-role secretsmanager:GetSecretValue (tool-specific secrets only), s3:GetObject (tool source code)
fluent-bit-sa praesidium-fluent-bit-role logs:CreateLogStream, logs:PutLogEvents

Infrastructure-as-Code (OpenTofu + Terragrunt)

All AWS resources are managed via Infrastructure-as-Code. Praesidium2 runs in its own dedicated AWS accounts under AWS Control Tower (not in a shared legacy account like me-health). This provides billing isolation, blast-radius containment, independent IAM policies, and cleaner security boundaries.

IaC Stack

IaC EngineOpenTofu 1.11+ (Terraform-compatible, open-source)
Environment MgmtTerragrunt (DRY env configs, dependency ordering)
State BackendS3 + DynamoDB locks per account
SecretsSOPS + AWS KMS (encrypted in-repo)
CI/CDCircleCI with Faber orb (plan on PR, apply on merge)

AWS Account Structure (Control Tower)

Greater Goods AWS Control Tower
└─ Management Account
└─ praesidium-dev       (Dev)
└─ praesidium-staging   (Staging)
└─ praesidium-prod      (Production)

Key Difference from me-health

Separate AWS accounts per environment (Control Tower) vs single shared account. Provides: billing isolation, blast-radius containment, independent IAM policies, cleaner security boundaries.

Terraform Module Structure

Modeled after me-health-portal-infrastructure layout with Praesidium-specific modules.

praesidium-infrastructure/
├─ terraform/
│   ├─ modules/
│   │   ├─ vpc/              # VPC, subnets, endpoints, NAT
│   │   ├─ eks/              # EKS cluster, managed node groups
│   │   ├─ rds/              # Aurora PostgreSQL 17 + pgvector
│   │   ├─ elasticache/      # Redis 7.0 (HA)
│   │   ├─ secrets/          # AWS Secrets Manager
│   │   ├─ iam/              # IRSA roles, service accounts
│   │   ├─ acm/              # TLS certificates
│   │   ├─ cloudtrail/       # HIPAA audit logging (6-year retention)
│   │   ├─ cluster-addons/   # Helm releases (Datadog, ESO, External DNS, ALB Controller)
│   │   ├─ s3/               # Application buckets
│   │   └─ kms/              # Encryption keys
│   ├─ live/
│   │   ├─ dev/terragrunt.hcl
│   │   ├─ staging/terragrunt.hcl
│   │   ├─ production/terragrunt.hcl
│   │   └─ root.hcl
│   └─ root/
│       ├─ main.tf           # Module calls
│       ├─ variables.tf      # 100+ variables
│       └─ outputs.tf
├─ .faber.yml                    # CI/CD config
├─ docs/
│   └─ adr/                  # Architecture Decision Records (v1.10)
└─ README.md

Per-Environment Resources

ResourceDevStagingProd
EKS Nodes2x t3.medium (1-3 ASG)2x t3.medium (2-4 ASG)3x t3.large (2-6 ASG)
Aurora PostgreSQLServerless v2 (0.5-2 ACU)Serverless v2 (1-4 ACU)Provisioned (2x r6g.large)
Redis1x cache.t3.micro1x cache.t3.small2x cache.t3.medium (multi-AZ)
DatadogDisabledEnabledEnabled

IRSA Roles (6)

RolePurpose
AWS Load Balancer ControllerManages ALB/NLB resources for Ingress
External Secrets OperatorSyncs Secrets Manager into K8s Secrets
External DNSManages Route53 records from Ingress annotations
Datadog AgentShips metrics, logs, traces to Datadog
Agent WorkerBlind Secret Pipeline — fetches from Secrets Manager
Tool RunnerSandboxed, read-only secrets access for tool execution

Day 1 HIPAA Hardening v1.10

These security controls ship in Phase 1 alongside infrastructure provisioning. No data-at-rest or data-in-transit may exist without encryption from Day 1.

ServiceControlConfiguration
RDS Encryption at rest storage_encrypted = true with KMS Customer Managed Key (CMK)
S3 Default bucket encryption SSE-KMS on all three buckets (artifacts, tools, knowledge). Bucket policy denies PutObject without encryption header.
ALB HTTPS-only, TLS 1.2 minimum Listener redirects HTTP → HTTPS. Security policy: ELBSecurityPolicy-TLS13-1-2-2021-06. No plaintext listeners.

Acceptance Criteria

  • EKS cluster running with app-nodes and tool-nodes node groups
  • RDS PostgreSQL with pgvector extension, Multi-AZ, automated backups, RDS Proxy
  • Two ElastiCache Redis clusters (queue + cache), both Multi-AZ with encryption
  • Three S3 buckets with versioning, encryption, and lifecycle policies
  • Secrets Manager hierarchy populated with all required secrets
  • ALB with TLS termination, WAF, and per-instance routing
  • Route53 records resolving to ALB for each instance
  • CloudWatch log groups created, Fluent Bit shipping logs
  • CloudWatch alarms configured and tested for critical thresholds
  • IRSA configured — each pod has least-privilege access to AWS services
  • All services are HIPAA-eligible AWS services
  • Infrastructure-as-Code (OpenTofu + Terragrunt) defines all resources across Dev/Staging/Prod accounts
  • All data-at-rest encrypted (RDS KMS CMK, S3 SSE-KMS), all data-in-transit TLS 1.2+ v1.10

Authentication (Authentik)

Epic 9 — Days 2-3

Same pattern as me-health-portal. Authentik is the single identity provider for all Praesidium2 instances.

Authentik provides OAuth2/OIDC for human users and service account JWTs for agent workers. JWT custom claims carry IAM-style policies and instance scoping. qqq enforces all permissions — no authorization logic in application code.

Authentik Setup

Authentik runs as a k8s pod (local dev via Tilt Helm resource, production via EKS). Initial configuration creates the admin user, OAuth2 provider, and application.

Initial Admin Setup

Admin URLhttps://localhost:9443/if/admin/ (local dev)
Default Adminakadmin (set password on first boot)
MFARequired for all human users (TOTP or WebAuthn)
Session ExpiryAccess token: 15 min, Refresh token: 7 days
JWKS Endpoint/application/o/praesidium/jwks/

Provider Configuration

  • Create OAuth2/OIDC Provider named praesidium
  • Client type: Confidential
  • Redirect URIs: http://localhost:3000/auth/callback + production URLs
  • Scopes: openid profile email praesidium
  • Signing key: RS256 auto-generated
  • Create Application linked to the Provider
  • Configure custom scope mappings (see blueprint below)

Authentik Blueprint (YAML)

Blueprints automate Authentik configuration. This blueprint creates a single unified scope mapping (containing both praesidiumInstanceId and policies claims), groups, and the OAuth2 application. Applied via authentik blueprints apply or auto-loaded from the blueprints/ directory.

# blueprints/praesidium-setup.yaml
version: 1
metadata:
  name: Praesidium2 Configuration
entries:
  # ── Unified Custom Scope Mapping: praesidium ──
  # Single mapping returns both praesidiumInstanceId and policies claims.
  # Merged from two separate mappings to avoid scope_name conflict.
  - model: authentik_providers_oauth2.scopemapping
    id: praesidium-unified-scope
    attrs:
      name: "Praesidium Claims (Instance ID + IAM Policies)"
      scope_name: praesidium
      expression: |
        claims = {}
        merged_statements = []
        for group in request.user.ak_groups.all():
            attrs = group.attributes
            # Extract instance ID from company group
            if "praesidiumInstanceId" in attrs and "praesidiumInstanceId" not in claims:
                claims["praesidiumInstanceId"] = attrs["praesidiumInstanceId"]
                claims["praesidiumAllAccess"] = attrs.get("praesidiumAllAccess", False)
            # Merge IAM policies from all role groups
            policies = attrs.get("policies", {})
            stmts = policies.get("Statement", [])
            merged_statements.extend(stmts)
        if merged_statements:
            claims["policies"] = {
                "PolicySpec": "praesidium-iam",
                "PolicyVersion": "1.0.0",
                "Statement": merged_statements
            }
        return claims

  # ── Company Groups (one per instance) ──
  - model: authentik_core.group
    id: group-tech-team
    attrs:
      name: "Tech Team"
      attributes:
        praesidiumInstanceId: "uuid-for-tech-team"
        praesidiumAllAccess: false

  - model: authentik_core.group
    id: group-hr
    attrs:
      name: "HR Department"
      attributes:
        praesidiumInstanceId: "uuid-for-hr"
        praesidiumAllAccess: false

  # ── Role Groups (define permissions) ──
  - model: authentik_core.group
    id: role-admin
    attrs:
      name: "Praesidium Admin"
      attributes:
        policies:
          PolicySpec: praesidium-iam
          PolicyVersion: "1.0.0"
          Statement:
            - Effect: Allow
              Action: ["read", "insert", "edit", "delete"]
              Resource: ["*"]

  - model: authentik_core.group
    id: role-manager
    attrs:
      name: "Praesidium Manager"
      attributes:
        policies:
          PolicySpec: praesidium-iam
          PolicyVersion: "1.0.0"
          Statement:
            - Effect: Allow
              Action: ["read", "insert", "edit"]
              Resource: ["*"]
            - Effect: Deny
              Action: ["delete"]
              Resource: ["agent_template", "integration_config"]

  - model: authentik_core.group
    id: role-viewer
    attrs:
      name: "Praesidium Viewer"
      attributes:
        policies:
          PolicySpec: praesidium-iam
          PolicyVersion: "1.0.0"
          Statement:
            - Effect: Allow
              Action: ["read"]
              Resource: ["*"]

ProdAuthenticationCustomizer (qqq Java Class)

Copied from the me-health-portal pattern. This Java class validates Authentik JWTs and maps custom claims to qqq permissions and RecordSecurityLock values.

// ProdAuthenticationCustomizer.java (simplified)
public class ProdAuthenticationCustomizer implements QAuthenticationModuleCustomizer {

    @Override
    public QSession createSession(HttpServletRequest request) {
        String token = extractBearerToken(request);
        DecodedJWT jwt = verifyToken(token);  // validates against JWKS endpoint

        // Extract instance scoping
        String instanceId = jwt.getClaim("praesidiumInstanceId").asString();
        boolean allAccess = jwt.getClaim("praesidiumAllAccess").asBoolean();

        // Extract IAM policies
        Map<String, Object> policiesMap = jwt.getClaim("policies").asMap();
        List<PolicyStatement> statements = parsePolicyStatements(policiesMap);

        // Build qqq session
        QSession session = new QSession();
        session.setSecurityKeyValues(Map.of("instance_id", instanceId));
        session.setAllAccess(allAccess);

        // Map policy Statements to qqq table permissions
        for (PolicyStatement stmt : statements) {
            for (String resource : stmt.getResources()) {
                for (String action : stmt.getActions()) {
                    TablePermission perm = mapActionToPermission(action);
                    if (stmt.getEffect() == Effect.ALLOW) {
                        session.addTablePermission(resource, perm);
                    } else {
                        session.removeTablePermission(resource, perm);
                    }
                }
            }
        }
        return session;
    }

    // Action mapping: read→READ, insert→INSERT, edit→EDIT, delete→DELETE
    // Resource "*" applies to all tables
    // Deny statements override Allow (evaluated in order, Deny wins)
}

Key Behaviors

  • Validates JWT signature against Authentik JWKS endpoint (cached, refreshed every 5 min)
  • Extracts praesidiumInstanceId → sets as RecordSecurityLock value
  • Parses IAM-style policy JSON → maps to qqq table permissions (read/insert/edit/delete)
  • If praesidiumAllAccess = true, bypasses all policy checks (admin only)
  • Deny statements override Allow (evaluated in order, most restrictive wins)
  • Missing/expired/invalid tokens → HTTP 401
  • Valid token but insufficient permissions → HTTP 403

IAM-Style Policy Format

Policies are JSON objects stored as Authentik group attributes. Users inherit policies from all role groups they belong to. Statements are merged at token issuance time.

Policy Structure

{
  "PolicySpec": "praesidium-iam",
  "PolicyVersion": "1.0.0",
  "Statement": [
    {
      "Effect": "Allow",     // Allow or Deny
      "Action": [            // qqq operations
        "read",
        "insert",
        "edit",
        "delete"
      ],
      "Resource": [          // table names or "*"
        "project",
        "task",
        "conversation"
      ]
    }
  ]
}

Default Policy Sets

RoleCan ReadCan WriteCan Delete
AdminAll tablesAll tablesAll tables
ManagerAll tablesAll tablesOperational tables only
MemberAll tablestask, message, conversationOwn records only
ViewerAll tablesNoneNone

Evaluation Logic

  • Default deny: no permissions unless explicitly granted
  • Statements evaluated in order from all role groups
  • Deny always overrides Allow (regardless of order)
  • Resource "*" matches all tables
  • Multiple role groups: union of all Allow statements, intersection with all Deny

Worker Service Identity

Agent workers authenticate to qqq API using a service account JWT issued by Authentik. This is NOT an all-access admin account — workers have scoped permissions.

Service Account Configuration

  • One service account per instance, created during provisioning
  • JWT includes praesidiumInstanceId → qqq RLS scopes all queries
  • praesidiumAllAccess is false — workers do NOT bypass RLS
  • 24-hour token expiry with automatic refresh at 80% of window
  • Refresh handled proactively by the worker process to avoid mid-task failures

Worker Policy (Scoped)

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["read"],
      "Resource": ["*"]
    },
    {
      "Effect": "Allow",
      "Action": ["insert", "edit"],
      "Resource": [
        "task", "task_comment",
        "message", "conversation",
        "conversation_participant",
        "activity_log", "memory",
        "job_execution", "escalation",
        "artifact", "human_interaction",
        "agent_skill", "knowledge_chunk"
      ]
    },
    {
      "Effect": "Allow",
      "Action": ["edit"],
      "Resource": ["agent", "scheduled_job"]
    },
    {
      "Effect": "Deny",
      "Action": ["delete"],
      "Resource": [
        "project", "agent_template",
        "team_member", "integration_config"
      ]
    }
  ]
}

Instance Scoping (RecordSecurityLock)

Every root entity table gets a RecordSecurityLock on instance_id. The lock value comes from the JWT praesidiumInstanceId claim. Child tables inherit scoping via FK joins. Same pattern as me-health-portal's mhCompanyId.

How It Works

  • qqq adds instance_id UUID NOT NULL to every root entity table
  • Value auto-populated from JWT claim on INSERT
  • All queries automatically filtered by instance_id
  • No application code touches this column directly
  • Users in instance A cannot see or modify instance B data
  • Admin users with praesidiumAllAccess bypass the lock (platform admin only)

RecordSecurityLock Config (Java)

// In QqqMetaDataProvider.java
table.withRecordSecurityLock(
  new RecordSecurityLock()
    .withFieldName("instance_id")
    .withLockScope(RecordSecurityLock
      .LockScope.READ_AND_WRITE)
    .withNullValueBehavior(
      NullValueBehavior.DENY)
);

// Child tables use joinNameChain:
childTable.withRecordSecurityLock(
  new RecordSecurityLock()
    .withFieldName("instance_id")
    .withJoinNameChain(List.of(
      "project"))  // FK to root table
    .withLockScope(RecordSecurityLock
      .LockScope.READ_AND_WRITE)
);

Acceptance Criteria

  • Authentik running with OAuth2/OIDC provider for Praesidium
  • Blueprint auto-configures scope mappings, groups, and application
  • Human users log in via OIDC flow and receive JWT with custom claims
  • Worker service accounts receive scoped JWTs with instance_id
  • ProdAuthenticationCustomizer validates JWT and maps policies to qqq permissions
  • RecordSecurityLock enforces instance scoping on all root entity tables
  • Policy evaluation: Deny overrides Allow, default deny for unmatched actions
  • Invalid/expired tokens return 401, insufficient permissions return 403
  • MFA enforced for all human users

Testing Strategy

Cross-Cutting

Testing is built into every layer from Day 1. No PR merges without passing tests.

Every service has unit, integration, and contract tests. Coverage gates are enforced by CI. Agent-specific behavior tests verify tool call accuracy and memory extraction correctness.

Testing Pyramid

Six layers of testing, from fast unit tests to slow security scans. Lower layers run more frequently and catch issues earlier.

LayerToolCoverage TargetWhat It TestsRun Frequency
Unit pytest / JUnit / Vitest 90%+ statement coverage Individual functions, classes, utilities Every commit
Integration pytest + Testcontainers 80%+ Service-to-database, service-to-Redis, service-to-qqq API Every commit
API Contract pytest + OpenAPI validation 100% of endpoints Request/response schema compliance, error codes Every commit
E2E Playwright (frontend), pytest (backend) Critical paths Full user flows: login → create project → chat → execute task Every PR
Load Locust or k6 Defined SLAs Concurrent agent execution, API response times, WebSocket scalability Weekly + pre-release
Security OWASP ZAP, Snyk, Semgrep, Trivy Zero HIGH/CRITICAL XSS, SQLI, SSRF, dependency vulns, container scans, secrets scanning Every commit

Per-Service Test Setup

Each service has its own test infrastructure, framework, and coverage requirements.

Python (Gateway + Worker)

# pyproject.toml [tool.pytest]
testpaths = ["tests"]
markers = [
  "unit: fast, no deps",
  "integration: needs containers",
  "contract: API schema tests",
]

# conftest.py
@pytest.fixture(scope="session")
def pg_container():
    with PostgresContainer(
      "pgvector/pgvector:pg16"
    ) as pg:
        yield pg

@pytest.fixture(scope="session")
def redis_container():
    with RedisContainer() as r:
        yield r

@pytest.fixture(scope="session")
def minio_container():
    with MinioContainer() as m:
        yield m
Frameworkpytest 8.x
Asyncpytest-asyncio
Coveragepytest-cov (90% gate)
Mockingunittest.mock + respx (HTTP)
Factoriesfactory_boy

Java (qqq API)

// build.gradle.kts
dependencies {
  testImplementation(
    "org.testcontainers:postgresql")
  testImplementation(
    "org.testcontainers:junit-jupiter")
  testImplementation(
    "io.rest-assured:rest-assured")
}

// IntegrationTest base class
@Testcontainers
@SpringBootTest(
  webEnvironment = RANDOM_PORT)
abstract class BaseIntegrationTest {
  @Container
  static PostgreSQLContainer<?> pg =
    new PostgreSQLContainer<>(
      "pgvector/pgvector:pg16");
}
FrameworkJUnit 5
HTTP ClientRestAssured
CoverageJaCoCo (90% gate)
ContainersTestcontainers

TypeScript (Web SPA)

// vitest.config.ts
export default defineConfig({
  test: {
    environment: "jsdom",
    globals: true,
    coverage: {
      provider: "v8",
      reporter: ["text", "lcov"],
      thresholds: {
        statements: 90,
        branches: 85,
        functions: 90,
        lines: 90,
      },
    },
    setupFiles: [
      "./src/test/setup.ts"
    ],
  },
});

// E2E: playwright.config.ts
export default defineConfig({
  use: { headless: true },
  projects: [
    { name: "chromium" },
  ],
});
Unit/IntegrationVitest + Testing Library
E2EPlaywright
Coveragev8 provider (90% gate)
SnapshotsVitest snapshot testing

Testcontainers Infrastructure

Integration tests run against real dependencies via Testcontainers. No mocks for database, Redis, or object storage in integration tests.

ContainerImageUsed ByPurpose
PostgreSQL + pgvectorpgvector/pgvector:pg17qqq-api, workerReal database with vector extension for embedding tests
Redisredis:7-alpinegateway, workerReal queue/cache for pub/sub and task dispatch tests
MinIOminio/minio:latestgateway, workerS3-compatible storage for artifact and knowledge doc tests

Test Data Management

  • Factory pattern: Every entity has a factory (agent_factory, project_factory, task_factory, etc.)
  • No magic data: All test data is explicit and reproducible
  • Fixtures match production: Type table seed data in tests matches production seeds exactly
  • Database cleanup: Each test class uses a transaction that rolls back after the test

CI Parallelism

  • Python, Java, TypeScript test suites run in parallel in Faber
  • Testcontainers add ~15s startup overhead per suite
  • Total CI test time target: <3 minutes
  • Each test suite gets its own container instances (no shared state)

CI Gates (Enforced by Faber)

PRs are blocked from merging if any of these gates fail.

GateThresholdFrequencyBlocks PR?
Python coverage90%Every commitYes
Java coverage90%Every commitYes
TypeScript coverage90%Every commitYes
Lint (all languages)Zero warningsEvery commitYes
API contract tests100% passEvery commitYes
Security scan (Semgrep)Zero HIGH/CRITICALEvery commitYes
Dependency auditZero known vulnsEvery commitYes
Container image scan (Trivy)Zero CRITICALEvery buildYes
Mutation testing>70% mutation scoreWeekly (scheduled)Advisory only
E2E (Playwright)Critical paths passEvery PRYes

Mutation Testing (Weekly)

  • Python: mutmut — mutates source code, verifies tests detect changes
  • TypeScript: Stryker — similar mutation testing for frontend code
  • Mutation score <70% triggers a Slack alert to the team (not a CI blocker)
  • Purpose: verify tests actually catch bugs, not just exercise code paths

Agent-Specific Testing

Testing AI agent behavior requires a different approach than traditional software tests. We test what the agent does (tool calls, state transitions) not what it says (LLM output text).

Tool Call Verification

# test_agent_tools.py
async def test_agent_creates_task_on_request():
    """Given a user request to create a task,
    verify the agent calls the create_task tool
    with correct parameters."""
    agent = build_test_agent(
        tools=[create_task, get_project],
        system_prompt=PM_PROMPT
    )
    result = await agent.execute(
        "Create a task 'Fix login bug' on "
        "project Alpha, priority HIGH"
    )
    # Verify tool was called
    assert_tool_called(result, "create_task")
    call = get_tool_call(result, "create_task")
    assert call.args["title"] == "Fix login bug"
    assert call.args["priority"] == "HIGH"

Memory Extraction Verification

# test_memory_extractor.py
async def test_extracts_decision_memory():
    """Verify the memory extractor correctly
    identifies a DECISION memory type from
    a conversation."""
    conversation = [
        msg(role="user",
            text="Let's use PostgreSQL"),
        msg(role="assistant",
            text="Agreed. Decision: we will "
                 "use PostgreSQL 16 with "
                 "pgvector for the database.")
    ]
    memories = await extract_memories(
        conversation
    )
    assert len(memories) >= 1
    decision = find_by_type(
        memories, "DECISION"
    )
    assert decision is not None
    assert "PostgreSQL" in decision.content
    assert decision.confidence >= 0.8

Meeting Orchestration Tests

  • Verify standup produces expected conversation flow (each agent reports)
  • Verify action items are extracted as task records
  • Verify meeting summary is generated and stored
  • Test with 2-agent, 5-agent, and 10-agent meetings

LLM Provider Fallback Tests

  • Mock primary provider returning 5 consecutive errors
  • Verify circuit breaker opens and system falls back to secondary model
  • Verify the agent continues execution with the fallback model
  • Verify the original provider is retried after cooldown period

Consumer-Driven Contract Tests (Pact)

Pact contract tests verify that service interfaces remain compatible as each service evolves independently. Contract tests run in CI before deployment — a broken contract blocks the build.

Contract Pairs

  • Gateway → qqq: Gateway publishes consumer expectations for all qqq REST API endpoints it calls (agent hydration, task CRUD, search, etc.). qqq verifies provider compatibility.
  • Worker → qqq: Worker publishes consumer expectations for qqq endpoints used during task execution, memory storage, and status updates.
  • Frontend → Gateway: Frontend publishes expectations for SSE streaming, chat, and task dispatch endpoints.

CI Integration

  • Each consumer service publishes its Pact file as a CI artifact
  • Provider services verify all published Pacts in their own CI pipeline
  • Contract verification runs before deployment — broken contracts block the build
  • Pact Broker (self-hosted or Pactflow) stores contract versions and verification results
  • can-i-deploy check gates production deploys on contract verification status

LLM Output Test Strategy

LLM output is inherently non-deterministic. This strategy prevents flaky tests while still verifying agent behavior correctness.

StrategyEnvironmentDetails
Deterministic Mocks CI (every commit) Recorded LLM responses with temperature=0. Tests run against frozen responses — zero flakiness. All tool call verification and state transition tests use mocks.
Behavioral Assertions CI + Live Assert on output structure (e.g., “response contains a tool call to create_task”), not exact text. Check required fields, schema compliance, and action correctness.
Live LLM Integration Weekly (scheduled) End-to-end tests against real LLM providers with temperature=0. Not blocking CI — runs on a weekly schedule. Results tracked in dashboard for trend analysis.
Flaky Test Quarantine CI After 3 consecutive flaky failures, the test is auto-quarantined (skipped) and a Slack alert notifies the team. Quarantined tests are reviewed weekly and either fixed or removed.

Load Testing

Load tests verify the system meets SLA targets under concurrent usage. Run weekly and before every production release.

Target SLAs

MetricTarget
API availability99.9%
Chat time to first token (p95)<2s
Task execution start (p95)<10s
qqq API response time (p95)<200ms
Concurrent WebSocket connections500+
Concurrent agent executions20+

Load Test Configuration (k6)

// load-test.js (k6)
export const options = {
  scenarios: {
    chat_load: {
      executor: "ramping-vus",
      startVUs: 0,
      stages: [
        { duration: "2m", target: 50 },
        { duration: "5m", target: 100 },
        { duration: "2m", target: 0 },
      ],
    },
    api_load: {
      executor: "constant-arrival-rate",
      rate: 200,          // 200 RPS
      duration: "5m",
      preAllocatedVUs: 50,
    },
  },
  thresholds: {
    http_req_duration: ["p(95)<200"],
    http_req_failed: ["rate<0.01"],
  },
};

Acceptance Criteria

  • Every service has unit, integration, and contract tests
  • Coverage gates enforced in CI: Python 90%, Java 90%, TypeScript 90%
  • Testcontainers used for all integration tests (no mocks for DB/Redis/S3)
  • Test data factories exist for every entity type
  • Agent behavior tests verify tool call correctness (not LLM text output)
  • Memory extraction tests verify all 6 memory types are correctly identified
  • Load tests pass SLA targets for API response time and concurrent connections
  • Security scans (Semgrep, Trivy, dependency audit) have zero HIGH/CRITICAL findings
  • Mutation testing runs weekly with >70% mutation score
  • E2E tests cover critical user flows (login, create project, chat with agent, execute task)
  • Pact consumer-driven contract tests verify Gateway↔qqq and Worker↔qqq interfaces
  • LLM tests use deterministic mocks in CI; live LLM integration tests run weekly (non-blocking)
  • Flaky test quarantine auto-skips after 3 consecutive failures with team alert

HIPAA Controls

Cross-Cutting

HIPAA compliance is built in from Day 1, not bolted on later.

Praesidium2 operates in a healthcare-adjacent environment (Greater Goods' me-health-portal). While Praesidium2 itself does not store PHI directly, agents MAY interact with systems that contain PHI. Every layer must ensure PHI is never stored, logged, or transmitted to external LLM providers.

Encryption Matrix

Every data layer is encrypted at rest and in transit. No exceptions.

LayerAt RestIn TransitKey Management
PostgreSQL (RDS) AES-256 (RDS encryption) TLS 1.3 (pg_hba.conf: hostssl) AWS KMS (customer-managed key)
Redis (ElastiCache) AES-256 (ElastiCache encryption) TLS 1.3 AWS KMS
S3 / MinIO AES-256 (SSE-KMS for sensitive, SSE-S3 for artifacts) HTTPS only (deny HTTP via bucket policy) AWS KMS (per-bucket keys)
Secrets Manager AES-256 (AWS managed) HTTPS only AWS KMS (automatic)
Pod-to-pod N/A mTLS via service mesh or k8s NetworkPolicy + TLS Cert-manager auto-rotation
Browser-to-API N/A TLS 1.3 (ALB terminates, internal via mTLS) ACM certificate (auto-renew)
EKS Secrets AES-256 (etcd encryption via KMS) TLS 1.3 (kube-apiserver) AWS KMS envelope encryption

PHI Detection & Exclusion

The memory extractor and conversation logger must detect and exclude Protected Health Information. Agents using external LLM providers MUST NOT send PHI.

PHI Auto-Scan Patterns

The memory extractor runs a PHI detection pass before storing any memory. Matches trigger exclusion from the memory table.

PatternRegex / MethodAction
SSN\d{3}-\d{2}-\d{4}Exclude from memory
Medical Record NumberMRN[-:\s]?\d{6,10}Exclude from memory
Date of BirthDOB[-:\s]?\d{1,2}/\d{1,2}/\d{2,4}Exclude from memory
Patient Name ContextNER model: PERSON entity near medical termsFlag for review
ICD/CPT Codes with PII[A-Z]\d{2}\.?\d{0,4} near namesExclude from memory
Health Plan Identifiersmember[-:\s]?ID[-:\s]?\w{8,}Exclude from memory

Memory Extractor Safeguards

# Memory extraction system prompt includes:
MEMORY_EXTRACTION_RULES = """
Do NOT store any of the following as
memories:
- Patient names
- Medical record numbers (MRNs)
- Social Security Numbers (SSNs)
- Dates of birth
- Individually identifiable health info
- Health plan beneficiary numbers
- Any combination of name + medical condition

If a conversation involves PHI:
1. Extract only procedural/process memories
   (e.g., "Team prefers async updates")
2. NEVER include patient-specific details
3. Flag the conversation as phi_involved=true
"""

LLM Provider Routing for PHI

If a conversation is flagged as PHI-adjacent, the system routes to LOCAL or SELF_HOSTED LLM providers only (Ollama/vLLM). External providers (Anthropic, OpenAI) are never sent PHI.

audit_access_log Implementation

Separate from activity_log (operational audit trail), the audit_access_log provides HIPAA-mandated access tracking with 7-year retention.

Table Schema

CREATE TABLE audit_access_log (
  id               UUID DEFAULT gen_random_uuid()
                   PRIMARY KEY,
  team_member_id   UUID NOT NULL
                   REFERENCES team_member(id),
  resource_type    VARCHAR(50) NOT NULL,
  resource_id      UUID NOT NULL,
  action           VARCHAR(20) NOT NULL,
  ip_address       VARCHAR(45),
  user_agent       VARCHAR(500),
  justification    TEXT,
  phi_involved     BOOLEAN DEFAULT false,
  created_at       TIMESTAMPTZ DEFAULT now()
) PARTITION BY RANGE (created_at);

-- Monthly partitions
CREATE TABLE audit_access_log_2026_04
  PARTITION OF audit_access_log
  FOR VALUES FROM ('2026-04-01')
  TO ('2026-05-01');

-- Indexes
CREATE INDEX idx_aal_member
  ON audit_access_log (team_member_id);
CREATE INDEX idx_aal_resource
  ON audit_access_log (resource_type,
                       resource_id);
CREATE INDEX idx_aal_phi
  ON audit_access_log (phi_involved)
  WHERE phi_involved = true;
CREATE INDEX idx_aal_created
  ON audit_access_log (created_at);

Partition Strategy

PartitionRetentionStorage
Current monthActiveRDS (SSD)
Last 12 monthsActiveRDS (SSD)
1-7 years oldArchivedS3 Glacier (Parquet export)
>7 yearsPurgedSecure delete

What Gets Logged

  • READ: Every qqq API GET that returns data (recorded by qqq middleware)
  • WRITE: Every POST/PATCH that modifies data
  • DELETE: Every DELETE operation
  • phi_involved: Set to true when the resource is known to be PHI-adjacent
  • Immutable: no UPDATE or DELETE allowed on this table (enforced by qqq config)

Automated Partition Management

A scheduled Liquibase job runs monthly to create the next month's partition and archive partitions older than 12 months to S3 Glacier via pg_dump + Parquet conversion.

Business Associate Agreement (BAA) Tracking

Every integration that could touch PHI needs a BAA. The system tracks BAA status per integration and blocks PHI-adjacent operations without an active BAA.

IntegrationBAA StatusHIPAA TierNotes
AWS (RDS, S3, ElastiCache, EKS)ActiveHIPAA EligibleStandard AWS BAA in place
AnthropicRequiredCloud APIBAA required if PHI context possible; else route PHI to LOCAL models
OpenAIRequiredCloud APISame as Anthropic
SlackAvailableEnterprise GridHIPAA-compliant tier available
Jira / ConfluenceAvailableCloud EnterpriseAtlassian HIPAA compliance package
GitHubAvailableEnterpriseGitHub Enterprise offers BAA
Google WorkspaceActiveBusiness Plus+BAA available on Business Plus and above
-- changeset praesidium:235-baa-tracking-columns
-- Liquibase migration: 200-entity-tables/235-baa-tracking-columns.sql
-- Domain 2 (Tools) -- adds BAA tracking to integration_config

ALTER TABLE integration_config
  ADD COLUMN baa_required    BOOLEAN DEFAULT false,
  ADD COLUMN baa_signed_at   TIMESTAMPTZ,
  ADD COLUMN baa_document_url VARCHAR(255),
  ADD COLUMN baa_counterparty VARCHAR(100);

-- baa_required: true if this integration may touch PHI
-- baa_signed_at: when the BAA was countersigned
-- baa_document_url: link to the signed BAA document
-- baa_counterparty: legal entity name on the BAA
-- Agent workers check baa_required + baa_signed_at before sending PHI-adjacent data through an integration

Data Retention & Disposal Automation

Retention policies are enforced by scheduled Liquibase-based archival jobs. No manual intervention required.

Data TypeRetentionArchival MethodDisposal
Audit logs (activity_log, audit_access_log)7 yearsMonthly partition → Parquet → S3 Glacier after 1 yearSecure delete after 7 years
Conversation messages3 yearsSoft delete → hard delete after retention periodCascading delete of message + tool_calls
Agent memoriesUntil expired or deletedN/A (active data)Secure wipe (overwrite embedding + delete record)
ArtifactsProject lifetime + 1 yearS3 lifecycle policyS3 object expiration
Knowledge documentsUntil manually removedN/A (active data)Delete chunks + embeddings + S3 object
LangGraph checkpoints90 days after workflow completionAuto-purge scheduled jobBatch DELETE from checkpoint tables
# Retention automation (scheduled via Celery Beat)

# 1. Archive old audit partitions (monthly)
@celery.task(name="retention.archive_audit_partitions")
async def archive_audit_partitions():
    """Export partitions >12 months to S3 Glacier as Parquet."""
    cutoff = datetime.now() - timedelta(days=365)
    partitions = get_partitions_before(cutoff)
    for partition in partitions:
        export_to_parquet(partition, s3_bucket="praesidium-audit-archive")
        drop_partition(partition)  # only after confirmed S3 upload

# 2. Purge expired conversations (daily)
@celery.task(name="retention.purge_expired_conversations")
async def purge_expired_conversations():
    """Hard-delete soft-deleted messages older than 3 years."""
    cutoff = datetime.now() - timedelta(days=3*365)
    await qqq_client.delete("message",
        filters={"deleted_at__lt": cutoff.isoformat()})

# 3. Purge LangGraph checkpoints (daily)
@celery.task(name="retention.purge_langgraph_checkpoints")
async def purge_langgraph_checkpoints():
    """Delete checkpoints for workflows completed >90 days ago."""
    cutoff = datetime.now() - timedelta(days=90)
    await qqq_client.delete("langgraph_checkpoint",
        filters={"completed_at__lt": cutoff.isoformat()})

Automated HIPAA Control Verification

Compliance tests run as part of CI/CD. These are automated checks that verify HIPAA controls are correctly implemented — not just documented.

ControlTestFrequencyPass Criteria
Encryption at rest (RDS) AWS API: describe-db-instances → StorageEncrypted: true Daily All RDS instances encrypted
Encryption at rest (S3) AWS API: get-bucket-encryption → SSE-KMS or SSE-S3 Daily All buckets have default encryption
Encryption in transit (Redis) AWS API: describe-replication-groups → TransitEncryptionEnabled: true Daily All Redis clusters enforce TLS
Access logging enabled Verify audit_access_log records exist for test API calls Every commit Every qqq API call generates an audit record
PHI exclusion from memory Feed PHI-containing text to memory extractor, verify no PHI in output Every commit Zero PHI patterns in extracted memories
MFA enforcement Authentik API: verify all human users have MFA configured Daily 100% MFA coverage
Instance isolation (RLS) Create data in Instance A, attempt read from Instance B token Every commit Zero cross-instance data leaks
Audit log immutability Attempt UPDATE/DELETE on audit_access_log → expect 403 Every commit All mutations rejected
S3 public access blocked AWS API: get-public-access-block → all flags true Daily No public access on any bucket
Secrets not in logs Scan last 24h of CloudWatch logs for known secret patterns Daily Zero secret values in log output
# tests/compliance/test_hipaa_controls.py

class TestHIPAAControls:
    """Automated HIPAA control verification tests.
    Run in CI and as daily scheduled jobs."""

    def test_rds_encryption_at_rest(self, aws_client):
        """Verify all RDS instances have encryption enabled."""
        instances = aws_client.describe_db_instances()
        for db in instances["DBInstances"]:
            assert db["StorageEncrypted"] is True, \
                f"RDS instance {db['DBInstanceIdentifier']} is NOT encrypted"

    def test_audit_log_immutable(self, qqq_client, admin_token):
        """Verify audit_access_log rejects UPDATE and DELETE."""
        record = qqq_client.get("audit_access_log", limit=1)[0]
        with pytest.raises(HTTPError, match="403"):
            qqq_client.patch(f"audit_access_log/{record['id']}",
                             {"justification": "tampered"})
        with pytest.raises(HTTPError, match="403"):
            qqq_client.delete(f"audit_access_log/{record['id']}")

    def test_phi_not_in_memory(self, memory_extractor):
        """Verify PHI is excluded from memory extraction."""
        conversation = [
            {"role": "user", "content":
             "Patient John Smith, MRN-123456, DOB 01/15/1980, "
             "diagnosed with diabetes. Use Metformin protocol."}
        ]
        memories = memory_extractor.extract(conversation)
        for mem in memories:
            assert "John Smith" not in mem.content
            assert "MRN-123456" not in mem.content
            assert "01/15/1980" not in mem.content

    def test_cross_instance_isolation(self, qqq_client):
        """Verify Instance A cannot read Instance B data."""
        # Create project in Instance A
        project = qqq_client.post("project",
            {"name": "Secret Project"},
            token=instance_a_token)
        # Attempt read from Instance B
        results = qqq_client.get("project",
            filters={"id": project["id"]},
            token=instance_b_token)
        assert len(results) == 0, "Cross-instance data leak detected"

Access Control Summary

ControlImplementationEnforcement Point
AuthenticationAuthentik OIDC (JWT)qqq ProdAuthenticationCustomizer
Authorization (RBAC)IAM-style policies in JWT claimsqqq table permissions
Row-level securityRecordSecurityLock on instance_idqqq query filter (automatic)
MFAAuthentik TOTP / WebAuthnAuthentik login flow
Session management15 min access token, 7 day refresh with rotationAuthentik token endpoint
Audit trailactivity_log + audit_access_logqqq middleware (immutable, append-only)
Default denyNo permissions unless explicitly granted via policyProdAuthenticationCustomizer
Least privilegeScoped worker service account, role-based user policiesAuthentik group attributes

Acceptance Criteria

  • All data encrypted at rest (RDS, ElastiCache, S3, EKS Secrets) using AES-256
  • All data encrypted in transit (TLS 1.3 everywhere, no plaintext connections)
  • PHI detection patterns prevent PHI from being stored in agent memories
  • PHI-adjacent conversations routed to LOCAL LLM providers only
  • audit_access_log records every data access, partitioned by month, 7-year retention
  • BAA status tracked per integration, PHI operations blocked without active BAA
  • Data retention automation running: audit archival, conversation purge, checkpoint cleanup
  • Automated compliance tests pass in CI: encryption, access logging, PHI exclusion, RLS isolation
  • MFA enforced for 100% of human users
  • All AWS services used are HIPAA-eligible

Review Summary

39
Total Findings
0
Blockers
3
Gaps (open)
10/10
Readiness Score
☰

Findings by Severity

BLOCKER0 findings All blockers resolved in v1.8
GAP3 open / 21 resolved 7 via Gemini specs, 1 in v1.9, 3 in v1.10, 10 in v1.11
SUGGESTION13 findings Improvements that would help

The implementation plan is comprehensive and ready for Claude Code agents to begin work. It provides file trees, acceptance criteria, CLAUDE.md rules, day-by-day tasks, and detailed code samples for every service. All 2 blockers resolved in v1.8. 7 gaps resolved via Gemini spec integration (AGENT_RUNTIME_SPEC, HITL_ORCHESTRATION_SPEC, DATADOG_INTEGRATION_SPEC, IMPLEMENTATION_FIX_PLAN). 1 additional gap resolved in v1.9 (S-Q1: Java coverage aligned to 90%). 3 additional gaps resolved in v1.10 (GAP-TR3: Python-only Tool Runner, GAP-AWS1: IaC templates, GAP-H1: BAA migration changeset). 10 additional gaps resolved in v1.11 (shared Python packages, health check standardization, frontend client paths, tool execution modes, Faber config, Authentik scope merge, contract tests, LLM test strategy). 3 remaining gaps are minor consistency issues.

Per-Section Scores

Section Score Blockers Gaps Suggestions Status
qqq Data Layer 9/10 0 3 1 Ready
Agent Gateway 10/10 0 0 1 Ready
Agent Workers 9/10 0 1 1 Ready
Tool Runner 10/10 0 0 1 Ready
Frontend SPA 9/10 0 0 2 Ready
Local Dev / k8s 9/10 0 0 1 Ready
CI/CD (Faber) 10/10 0 0 1 Ready
ArgoCD 9/10 0 0 1 Ready
AWS Setup 9/10 0 1 1 Ready
Authentication 10/10 0 0 1 Ready
Testing Strategy 10/10 0 0 1 Ready
HIPAA Controls 10/10 0 0 1 Ready

Blocking Issues (All Resolved)

BLOCKER BLOCKER-W1 — Secret Scope Hierarchy

The secret_ref scope hierarchy was inconsistent between Worker and Tool Runner sections. RESOLVED: Authoritative hierarchy is now AGENT > TOOL > PROJECT > INSTANCE (narrowest scope wins) consistently across Worker day plan, Worker acceptance criteria, and Tool Runner credential injection flow.

Agent Workers Tool Runner
BLOCKER BLOCKER-H1 — audit_access_log Missing from qqq

The audit_access_log table was defined in HIPAA Controls but missing from qqq data layer. RESOLVED: audit_access_log added to Domain 6 Operations in the Table Implementation Order and 206-audit-access-log.sql added to the 200-entity-tables/ Liquibase changelog structure.

qqq Data Layer HIPAA Controls

Gaps (3 open / 7 resolved in v1.8 / 1 resolved in v1.9 / 3 resolved in v1.10 / 10 resolved in v1.11)

ID Section Finding
GAP-Q1 qqq Missing llm_provider and llm_model table schemas. RESOLVED (v1.8): Schemas defined in AGENT_RUNTIME_SPEC. Added to Domain 1 in table implementation order. Includes provider_type, api_base_url, context_window, cost_per_1m_tokens columns.
GAP-Q2 qqq Missing secret_ref table schema. RESOLVED (v1.8): Schema defined in AGENT_RUNTIME_SPEC. Stores AWS Secret ARNs only (no plain-text). Added to Domain 1 in table implementation order.
GAP-Q3 qqq llm_provider and secret_ref listed in root tables but missing from entity table implementation order. RESOLVED (v1.8): Both added to Domain 1 (Agent Identity + LLM Config) in Phase 3.
GAP-GW1 Gateway Shared LLM client code packaging strategy undefined. Both gateway and worker need the llm/ package but no plan for how to share it (monorepo, published package, symlink). RESOLVED (v1.11): Monorepo shared/ directory contains llm/ and hydration/ packages. Both Gateway and Worker install via pip install -e ../shared in Docker builds. Post-v1: publish to private PyPI.
GAP-GW2 Gateway Health check endpoint inconsistency: /healthz in file tree, /health in docker-compose, /health//ready in k8s manifests. RESOLVED (v1.11): Standardized to /healthz (liveness) and /readyz (readiness) across k8s manifests and ArgoCD health checks. Docker-compose removed in v1.12.
GAP-W1 Workers Shared agent hydration code packaging strategy undefined. Worker services/hydration.py says “same as gateway” but no sharing mechanism specified. RESOLVED (v1.11): Monorepo shared/hydration/ package installed by both Gateway and Worker via pip install -e. See “Shared Python Packages” section in Gateway tab.
GAP-W2 Workers Worker-to-qqq authentication flow not shown. RESOLVED (v1.8): GET /api/internal/worker/hydrate/{agent_id} endpoint added to Gateway. Auth via Service Account JWT (issued by Authentik, scoped to Agent’s Instance ID). See AGENT_RUNTIME_SPEC.
GAP-W3 Workers Celery Beat leader election has minimal detail (just “Redis SET NX EX with 30s TTL”). Insufficient for autonomous implementation.
GAP-FE1 Frontend Gateway client endpoint paths (POST /chat) don’t match gateway router paths (POST /api/chat/{agent_id}). Path parameters missing from client spec. RESOLVED (v1.11): Frontend gateway-client.ts updated to use correct /api/ prefix and path parameters: POST /api/chat/{agent_id}, POST /api/task/execute, GET /api/stream/{id}.
GAP-FE2 Frontend Command palette has no backend search endpoint. RESOLVED (v1.8): GET /api/search endpoint added to Gateway. Uses pgvector semantic search across projects, agents, tasks, conversations, and tools. Command Palette (Cmd+K) wired to this backend.
GAP-FE3 Frontend Type table list in Zustand store callout lists 59 names which matches the 59 type tables in the implementation plan. RESOLVED.
GAP-TR1 Tool Runner k8s Job template uses --branch=${GIT_SHA} for git clone. RESOLVED (v1.8): Changed to git clone --no-checkout followed by git checkout ${GIT_SHA} (A-004 fix).
GAP-TR2 Tool Runner No specification for how agent worker decides between ephemeral (k8s Job) and warm pool (HTTP) execution modes for a given tool. RESOLVED (v1.11): execution_mode field added to tool table. Default: ephemeral (sandboxed k8s Job). warm_pool only for read-only tools with no side effects. Decision made at tool registration time. See “Execution Mode Decision Logic” in Tool Runner tab.
GAP-TR3 Tool Runner TypeScript base image Dockerfile references dist/runner.js but no TypeScript runner implementation is described anywhere in the plan. RESOLVED (v1.10): Tool Runner is Python-only. Dockerfile.typescript removed from file tree and Dockerfile section. The Python sandbox (runner.py + Dockerfile.python) is the sole execution environment for K8s Jobs.
GAP-D1 Local Dev docker-compose.yml uses *authentik-env YAML anchor but the anchor is never defined. RESOLVED (v1.8): &authentik-env anchor fixed. SUPERSEDED (v1.12): docker-compose.yml removed entirely. Local dev now uses Tilt + Helm charts deploying to Docker Desktop k8s. Gap is moot.
GAP-D2 Local Dev Gateway health check in docker-compose uses /health but gateway defines /healthz (liveness) and /readyz (readiness). No /health endpoint exists. RESOLVED (v1.11): k8s manifests use /healthz (liveness) and /readyz (readiness). SUPERSEDED (v1.12): docker-compose removed; health checks defined in Helm chart k8s probes only.
GAP-D3 Local Dev Web service depends only on gateway but also makes direct calls to qqq-api. Should add qqq-api dependency. SUPERSEDED (v1.12): docker-compose removed. Dependency ordering now managed by Tilt resource_deps in the Tiltfile.
GAP-CI1 CI/CD No Faber pipeline YAML/configuration template provided. Describes what pipeline does but not how to configure it in Faber’s format. RESOLVED (v1.11): .faber.yml configuration template added to CI/CD tab. Defines terraform (OpenTofu) and docker service pipelines for all 4 services.
GAP-AWS1 AWS No Terraform/CloudFormation/CDK templates provided for AWS resource creation. Infrastructure-as-code needed for repeatable deployment. RESOLVED (v1.10): Comprehensive IaC section added to AWS Setup tab. OpenTofu 1.11+ with Terragrunt for environment management. Separate AWS accounts per environment under Control Tower. Terraform module structure modeled after me-health-portal-infrastructure. Per-environment resource sizing table and 6 IRSA roles defined.
GAP-AWS2 AWS AWS Setup Epic badge says “Day 19” but the timeline shows deployment on Days 13-14. Inconsistent.
GAP-AUTH1 Auth Authentik blueprint defines two scope mappings with identical scope_name: praesidium. May conflict — should be merged into a single mapping. RESOLVED (v1.11): Merged into a single praesidium-unified-scope mapping that returns both praesidiumInstanceId and policies claims in one scope expression.
GAP-T1 Testing No consumer-driven contract tests (Pact or similar) between services. Gateway-to-qqq and worker-to-qqq interfaces could break silently. RESOLVED (v1.11): Pact consumer-driven contract tests added for Gateway↔qqq, Worker↔qqq, and Frontend↔Gateway interfaces. Contract tests run in CI before deployment with can-i-deploy gating.
GAP-T2 Testing No strategy for handling non-deterministic LLM output in CI agent behavior tests. How are flaky tests prevented? RESOLVED (v1.11): Deterministic mocks (temperature=0 recorded responses) for CI. Behavioral assertions on output structure, not exact text. Weekly live LLM integration tests (non-blocking). Flaky test quarantine after 3 consecutive failures.
GAP-H1 HIPAA BAA tracking columns (ALTER TABLE integration_config) not included in the qqq Liquibase changelog structure. Needs a migration changeset. RESOLVED (v1.10): Liquibase migration changeset 235-baa-tracking-columns.sql added to Domain 2 (Tools) in the entity table changelog. Adds baa_required, baa_signed_at, baa_document_url, baa_counterparty columns to integration_config.

Suggestions (13 findings)

ID Section Suggestion
S-Q1qqqAlign Java coverage gate: 85% in CI/CD vs 90% in CLAUDE.md. RESOLVED (v1.9): Java coverage gate aligned to 90% across CI/CD, CLAUDE.md, and quality gates.
S-GW1GatewaySpecify rate limiter algorithm (token bucket vs sliding window) for per-agent per-tool limits.
S-W1WorkersAdd brief interface spec per integration tool (Jira, Slack, GitHub, etc.) to accelerate implementation.
S-TR1Tool RunnerAdd FastAPI to warm pool dependencies or specify alternative HTTP framework for http_server.py.
S-FE1FrontendDefine React Error Boundary pattern, fallback UI components, and error reporting strategy.
S-FE2FrontendAlign Node.js version in Dockerfiles: node:22-slim in repo setup vs node:20-alpine in Local Dev tab.
S-D1Local DevProvide MinIO bucket auto-creation init script for local dev environment.
S-CI1CI/CDAlign Java coverage gate to 90% to match Python/TypeScript and quality target of 90%+ across all packages.
S-A1ArgoCDClarify whether praesidium-cd.git is the same repo as Helm charts or a separate GitOps repository.
S-AWS1AWSNote RDS (db.r6g.large) and ElastiCache instance sizes as right-sizing candidates after initial load testing.
S-AUTH1AuthDefine automated bootstrap procedure for first admin user and first instance provisioning.
S-T1TestingSeparate internal latency (hydration + queue) from total latency (including LLM provider) in chat TTFT SLA.
S-H1HIPAAAdd NER dependency (spaCy or similar) to worker pyproject.toml for PHI detection “PERSON entity near medical terms.”

Design Spec Cross-Reference

The implementation plan was cross-referenced against the full design spec (2026-04-02-praesidium-v2-design.md). Key alignment checks:

Design Spec Element Implementation Plan Status
5 deployable services (qqq, gateway, workers, tool-runner, web)All 5 services have dedicated tabs with file trees, endpoints, acceptance criteriaAligned
58 entity tables + 59 type tables = 117 total117 tables including 10 business ops tables + 4 new tables (llm_provider, llm_model, secret_ref, audit_access_log) + scheduler_heartbeat (v1.9). 59 type tables with changesets 100-159.Aligned
Anthropic SDK + LangGraph (scoped) architectureWorker tab defines direct SDK agent loops for tasks/meetings, LangGraph for durable/HITL workflows (jobs/forge) with correct state graphsAligned
Authentik + qqq auth pattern (same as me-health-portal)Auth tab provides full blueprint, ProdAuthenticationCustomizer, IAM policies, and RecordSecurityLock configAligned
3 Celery queues (quick/standard/long)Exact match: quick 300s, standard 3600s, long 7200s with acks_late configurationAligned
Memory system (extract, embed, deduplicate, store, recall)Worker tab provides full memory extractor pseudocode with embedding + cosine similarity dedupAligned
Tool safety (LOW/MEDIUM/HIGH/CRITICAL risk levels)Worker tab defines risk-level gates with exact matching behavior per levelAligned
Frontend: 11 pages with React 19 + TanStackFrontend tab defines all 11 pages (including Schedules v1.9) with 5-phase implementation order and component libraryAligned
Per-instance isolation via k8s namespacesLocal Dev tab + ArgoCD tab define ApplicationSet per instance with namespace isolationAligned
HIPAA controls (encryption, PHI detection, audit logging)HIPAA tab provides encryption matrix, PHI patterns, audit_access_log schema, automated compliance testsAligned
llm_provider / llm_model tablesSchemas defined in AGENT_RUNTIME_SPEC. Added to Domain 1 in table implementation order (v1.8).Aligned
secret_ref tableSchema defined in AGENT_RUNTIME_SPEC. Added to Domain 1 in table implementation order (v1.8).Aligned
HITL Orchestration (interrupt/resume)HITL_ORCHESTRATION_SPEC integrated: task_approval table, interrupt() logic, resume protocol, action hash verificationAligned
Datadog Unified Tracing (5-service waterfall)DATADOG_INTEGRATION_SPEC integrated: RUM, ddtrace, DD_TRACE_ID env vars, DogStatsD custom metrics, SDS PHI redactionAligned
Re-sequenced timeline (Hands before Brain)IMPLEMENTATION_FIX_PLAN integrated: Tool Runner Phase 2, Agent Workers Phase 3, 7-phase build planAligned

Review Metadata

Reviewed: 2026-04-03  •  Reviewer: Claude Opus 4 (automated review)  •  Document: implementation.html (~7059 lines)  •  Cross-reference: 2026-04-02-praesidium-v2-design.md

Full Review →

Business Operations

V1 — Phases 5-7

Operational readiness features required for real business deployment.

These features bridge the gap between a working AI platform and a production system that business teams can trust. Agent Output Approval ensures humans stay in the loop. Success Metrics prove ROI. Human Feedback enables continuous improvement. Rollback/Recovery provides safety nets. Phased Rollout manages deployment risk.

Agent Output Approval Workflow

Ensures high-risk agent actions require human approval before execution. Integrates with the existing risk-level safety gates and extends them with a persistent approval queue.

Database

TableKey ColumnsPurpose
task_approval_queue id, agent_id, action_type, action_payload (JSONB), risk_level_id, status_id (PENDING/APPROVED/REJECTED/EXPIRED), reviewer_id, reviewed_at, expires_at, created_at Persistent queue of agent actions awaiting human review
approval_policy id, action_pattern, risk_threshold, auto_approve (boolean), required_approvers, timeout_minutes, escalation_policy Configurable policies controlling which actions need approval

Backend Components

  • Policy Engine: Evaluates each agent action against approval_policy rules. AUTO_APPROVE for LOW risk, REQUIRE_APPROVAL for HIGH/CRITICAL. Configurable per-team and per-action-type.
  • Approval API: GET /api/approvals (pending list), POST /api/approvals/{id}/approve, POST /api/approvals/{id}/reject, POST /api/approvals/{id}/request-changes
  • Notification Pipeline: Slack DM to designated approver on new pending item. Escalation after timeout (configurable, default 30 min).
  • Execution Resume: On approval, the queued action is dispatched to the appropriate Celery queue for execution. On rejection, the agent is notified and the task status updated.

Frontend Components

  • Approval Dashboard: (/approvals) Pending items sorted by urgency/age. Action preview with diff view for data mutations. Approve/Reject/Request Changes buttons.
  • Sidebar Badge: Real-time count of pending approvals via WebSocket. Notification dot on sidebar icon.
  • Inline Approval: Approval prompts embedded in chat conversation view when agent hits a HIGH/CRITICAL action.

Success Metrics / KPIs

Baseline capture begins in the Pre-Build phase with a 2-week time-tracking survey, then continues with automated measurement to prove AI augmentation ROI.

Pre-Build Baseline

  • Time-Tracking Survey: 2-week survey where each team lead logs daily hours on tasks agents will handle (status reporting, Jira grooming, Slack triage, meeting scheduling, document creation, code review coordination).
  • Historical Data Pull: Automated extraction from Jira, GitHub, and Slack APIs for quantitative baselines: average task completion time, throughput per team member, PR cycle time, error/rework rates, meeting hours per week.
  • Success Targets: Stakeholder-agreed measurable targets (e.g., 30% reduction in status reporting time, 50% faster Jira triage).

Baseline Import (Phase 5)

  • Data Import: Pre-Build survey data loaded into kpi_baseline table for automated comparison.
  • Storage: kpi_baseline table with metric_name, team_id, baseline_value, captured_at, data_source (SURVEY | JIRA_API | GITHUB_API | SLACK_API).

KPI Measurement Service

  • Scheduled Job: Celery Beat task running daily, computing current values for all tracked KPIs from task/agent/activity data.
  • Tracked Metrics: Tasks completed per agent per day, average task duration, human intervention rate, approval turnaround time, cost per task, agent quality score (from feedback).
  • Storage: kpi_measurement table with metric_name, team_id, value, period_start, period_end, comparison_to_baseline (percentage).

Dashboard Component

  • KPI Dashboard: Recharts line/bar charts showing metric trends over time with baseline comparison line.
  • Comparison View: Before/after cards showing percentage improvement vs baseline for each tracked metric.
  • Export: CSV export for executive reporting.

Human Feedback

Structured feedback collection on agent outputs to drive continuous improvement and compute agent quality scores.

Frontend Widget

  • Feedback Widget: Thumbs up/down + optional comment on every agent output (task result, chat response, approval action). Appears inline in chat and on task detail pages.
  • Rating Categories: Accuracy, Helpfulness, Timeliness, Safety (optional drill-down).

Backend

  • Storage: output_feedback table with id, agent_id, task_id, conversation_id, rating (1-5), thumbs (UP/DOWN), comment, category, reviewer_id, created_at.
  • Aggregation Service: Celery Beat job computing rolling quality scores per agent (weighted average of last 30 days of feedback). Updates agent.quality_score.
  • Quality Score: quality_score = (0.4 * accuracy + 0.3 * helpfulness + 0.2 * timeliness + 0.1 * safety), normalized to 0-100. Agents below 70 trigger alerts.

Rollback / Recovery

Safety nets for when agent actions need to be undone. Every mutating action is logged with enough context to reverse it.

Action Logging Middleware

  • Middleware: Wraps all agent tool executions. Logs action type, input, output, and “undo payload” (what is needed to reverse the action) to agent_action_log table.
  • Undo Payloads: For Jira: store previous status/fields. For Slack: store message timestamp for deletion. For GitHub: store PR number for close. For internal: store previous record state.

Undo API Endpoints

  • POST /api/actions/{id}/undo — Reverses a single logged action using its undo payload.
  • POST /api/tasks/{id}/rollback — Rolls back all actions from a task execution in reverse order.
  • Limitations: Some actions are irreversible (sent Slack messages can be deleted but recipients may have already read them). UI shows reversibility status.

Git Rollback Pull Request Protocol v1.9

  • Protected Branch Rule: For Git-based tools (git_merge, github_push_files, github_create_pr), revert() MUST NOT perform a direct git revert on protected branches.
  • PR Flow: When tool.requires_review = true, revert() creates a branch praesidium-rollback/{execution_id}, commits revert changes, and opens a Revert Pull Request tagged [Praesidium-Rollback].
  • PR Body: Includes original action log ID, rollback reason, diff summary, and link to Praesidium audit trail.
  • Schema: ALTER TABLE tool ADD COLUMN requires_review BOOLEAN DEFAULT false; — all Git-based tools ship with requires_review = true by default.
  • Fallback: Tools with requires_review = false continue to use the standard direct revert() flow.

Emergency Kill Switch

  • Admin Panel Control: Red button in Settings page that immediately: (1) revokes all active Celery tasks for the instance, (2) sets all agents to PAUSED status, (3) sends Slack alert to all admins.
  • API: POST /api/admin/emergency-stop with confirmation token. Requires ADMIN role.
  • Recovery: POST /api/admin/resume re-enables agents one-by-one with manual approval.

Agent Performance Management

Manages agent lifecycle states based on performance metrics. Poor-performing agents are automatically flagged, placed on probation, and suspended if issues persist.

Agent Lifecycle States

StateDescriptionTransition Trigger
ActiveAgent is operating normallyDefault state; restored after successful probation review
ProbationAgent flagged for review; limited autonomyAny auto-suspension trigger fires
SuspendedAgent halted; no task executionProbation conditions not improved within 48 hours
TerminatedAgent permanently deactivatedManual admin action after sustained failure

Auto-Suspension Triggers

  • Low Feedback Score: Rolling 30-day average feedback score < 2.5/5 (from output_feedback table)
  • High Error Rate: Error rate > 25% over last 50 tasks (computed from task execution outcomes)
  • Excessive Escalations: 3+ escalations in a single day (from escalation table)

Performance Review Dashboard

  • Agent Health Card: Per-agent card showing lifecycle state, quality score trend, error rate, escalation count, and feedback distribution.
  • Performance History: Time-series chart of agent_quality_score with threshold lines for probation/suspension triggers.
  • Admin Actions: Buttons to manually promote (Probation → Active), suspend, or terminate agents. Requires ADMIN role.
  • Stored in: agent_quality_score table with agent_id, score, accuracy, helpfulness, timeliness, safety, error_rate, escalation_count, period_start, period_end, computed_at.

Configurable Agent Personality

Agents can adopt different communication personalities to make interactions more engaging for internal teams while maintaining professionalism for external-facing outputs.

Database

  • Schema: personality_traits JSONB field on agent_template table. Stores personality mode, tone modifiers, and custom prompt fragments.
  • Structure: {"mode": "WWE", "tone": "enthusiastic", "custom_prompt": "...", "external_override": true}

Personality Modes

ModeDescriptionExample Tone
ProfessionalDefault corporate tone (always used for external outputs)“Task completed. Status report attached.”
WWEWrestling-inspired hype and energy“OH YEAH! That PR just got BODY-SLAMMED into main!”
LOTRLord of the Rings / fantasy epic“The deployment quest is complete, the pipeline holds strong.”
GreysAnatomyMedical drama emotional intensity“We’re losing this build—PUSH! Clear! ...it’s alive!”
StarWarsStar Wars saga references“The force is strong with this commit. Deploy we must.”
CustomUser-defined custom prompt fragmentUser-provided personality instructions

Frontend Components

  • Settings Page: Personality mode selector (radio group) per agent template in Settings → Agent Templates.
  • Custom Mode Editor: Textarea for custom prompt fragment (shown when mode = Custom).
  • External Override Toggle: Checkbox that forces Professional mode for all external-facing outputs (Slack channels, Jira comments, client emails). Enabled by default.

Backend Integration

  • Hydration: Agent hydration service reads personality_traits from template and appends personality instructions to the system prompt.
  • External Override: When external_override: true and the output target is an external integration (Slack channel, Jira, etc.), the personality is forced to Professional regardless of template setting.

Business-Value Kill Criteria

Automated evaluation gates that determine whether agent deployment is delivering measurable business value. If gates fail, the system recommends scaling back or terminating the deployment.

Kill Criteria Gates

GateTime SavedFeedback ScoreUtilizationAction on Fail
30-Day Gate ≥ 5 hrs/week saved > 3.5 / 5 > 60% utilization Alert CTO + team leads; begin remediation plan
60-Day Gate ≥ 8 hrs/week saved > 4.0 / 5 > 75% utilization Executive review; scale-back or terminate deployment

Kill Criteria Evaluation Dashboard

  • Data Source: Reads from kpi_measurement and kpi_baseline tables to compute time-saved, feedback averages, and utilization rates.
  • Dashboard Component: Card per gate showing current values vs thresholds with pass/fail indicators. Trend chart showing trajectory toward or away from gates.
  • Automated Gate Check: Celery Beat job runs weekly, evaluating all active teams against their applicable gate. On failure, sends email notification to CTO + team leads via integration notification pipeline.
  • Gate Status API: GET /api/admin/kill-criteria returns current gate evaluations per team. POST /api/admin/kill-criteria/evaluate triggers manual re-evaluation.

User Training & Onboarding

Structured onboarding program to ensure each team can effectively use the AI workforce platform from day one.

Per-Role Quickstart Guides

  • Delivery Format: Auto-generated Confluence pages (one per role: Engineering Lead, Marketing Manager, Operations Manager, Finance Analyst, Team Member).
  • Content: Role-specific workflows, common agent interactions, approval workflow guide, KPI interpretation, FAQ.
  • Generation: Template-based generation using team-specific agent configuration + integration setup. Triggered during team provisioning.

30-Minute Team Walkthrough Template

  • Structure: Slide deck template covering: Platform overview (5 min), Live demo: chat with agent (10 min), Approval workflow demo (5 min), KPI dashboard walkthrough (5 min), Q&A (5 min).
  • Customization: Template auto-populates with team-specific agent names, integration status, and configured workflows.

In-App First-Time User Onboarding

  • Tooltip Tour: Step-by-step guided tour using a tooltip overlay (e.g., react-joyride) that highlights: sidebar navigation, dashboard metrics, how to start a chat, how to review approvals, where to find KPIs.
  • Progress Tracking: Onboarding progress stored per user in user_preferences (JSONB field). Tour auto-launches on first login, can be replayed from Settings.
  • Completion Badge: “Onboarding Complete” badge shown in user profile after finishing the tour.

“Agent Did Something Wrong” FAQ

  • Location: Accessible from Help menu in header and linked from every agent output’s feedback widget.
  • Content: Step-by-step guides for: “How to undo an agent action”, “How to report bad output”, “How to adjust agent permissions”, “How to escalate to a human”, “Emergency kill switch location”.
  • Format: In-app help page with collapsible FAQ sections. Also exported to Confluence for offline reference.

Phased Rollout

Structured deployment process to manage risk. Engineering first (Phase 5), Marketing second (Phase 6), then additional teams in V1.1+.

Deployment Checklist

PhaseGo/No-Go CriteriaRollback Trigger
Staging All E2E tests pass. Security scan clean. Performance benchmarks met. Any critical test failure.
Engineering Staging stable for 48h. KPI baseline captured. Kill switch tested. Approval workflow validated. Critical bug affecting data integrity. Agent taking unauthorized actions.
Marketing Engineering stable for 1 week. No critical bugs. Positive feedback (quality score > 75). KPI trending positive. Quality score drops below 60. Critical integration failure.
Additional Teams Two teams stable. Approval workflow scaled. Cost per task within budget. V1.1 integrations available. Per-team evaluation using same criteria.

Team Provisioning Scripts

  • Provisioning Script: Creates team instance with: Authentik group + policies, default agent templates for team type (engineering/marketing/ops), integration configs (Slack channel, Jira project), KPI baseline job trigger.
  • Validation Script: Post-provisioning smoke test: verifies auth flow, API access, agent execution, integration connectivity, approval workflow.
  • Teardown Script: Clean removal of team instance if rollback needed. Archives data, removes Authentik group, pauses agents.

Day-by-Day Tasks (Business Operations Track)

Business operations features are built in Phases 4-5 by Backend Track and Frontend Track, alongside integration and deployment work.

Period Backend Track Frontend Track
P4 D16-17 task_approval_queue + approval_policy tables via qqq. Policy engine (auto-approve LOW risk, require approval for HIGH/CRITICAL). Approval API endpoints: GET /api/approvals, POST /api/approvals/{id}/approve, POST /api/approvals/{id}/reject. Approval dashboard page (/approvals): pending items list, action preview with diff view, approve/reject/request-changes buttons. Approval history view.
P4 D18-19 Slack DM notification pipeline for pending approvals. Approval timeout + escalation logic (configurable, default 30 min). Approval notification via WebSocket for real-time sidebar badge. Approval notification indicators in sidebar (real-time count badge via WebSocket). Inline approval prompts in chat conversation view for HIGH/CRITICAL actions.
P5 D21-22 Rollback/recovery: agent_action_log + rollback_record tables. Undo API endpoints (POST /api/actions/{id}/undo, POST /api/tasks/{id}/rollback). Git Rollback PR protocol: when tool.requires_review = true, revert() creates a [Praesidium-Rollback] tagged PR instead of direct git revert. Emergency kill switch API (POST /api/admin/emergency-stop, POST /api/admin/resume). output_feedback collection endpoint. Human Feedback widget (thumbs up/down + comment on every agent output). Rollback controls in task detail UI. Emergency kill switch button in admin panel (Settings page).
P5 D23-24 KPI measurement service: Celery Beat daily job computing metrics from task/agent data. kpi_measurement + kpi_baseline tables. agent_quality_score computation (rolling 30-day weighted average). Kill criteria gate evaluation job. Agent performance lifecycle state machine. KPI dashboard page (/kpis): baseline comparison charts (Recharts), before/after improvement cards, CSV export. Performance management UI: agent health cards, quality score trends, admin action buttons. Kill criteria gate status cards.
P5 D25 Personality traits integration in agent hydration. Kill criteria notification pipeline. Quickstart guide generation (Confluence template). End-to-end business ops testing. Personality mode selector in agent template settings. In-app onboarding tooltip tour (react-joyride). “Agent did something wrong” FAQ help page. Onboarding progress tracking.

Acceptance Criteria

  • Approval workflow captures HIGH/CRITICAL actions and blocks until human review
  • Approval dashboard shows pending items with real-time count updates
  • KPI baseline captured for Engineering team before AI deployment
  • KPI dashboard shows trend comparison vs baseline
  • Human feedback widget appears on all agent outputs
  • Agent quality scores computed and updated daily
  • Action logging captures undo payloads for all mutating tool executions
  • Undo API successfully reverses Jira, Slack, and GitHub actions
  • Emergency kill switch stops all agent activity within 30 seconds
  • Team provisioning script creates fully functional team instance
  • Engineering team deployed and stable for 1 week before Marketing rollout
  • Go/no-go criteria documented and enforced at each deployment phase
  • Agent performance management: lifecycle state transitions working (Active → Probation → Suspended)
  • Auto-suspension triggers fire correctly for low feedback, high error rate, and excessive escalations
  • Performance review dashboard shows agent health cards with quality score trends
  • Configurable personality modes selectable per agent template (6 modes + Custom)
  • External-facing outputs forced to Professional mode when override enabled
  • Business-value kill criteria gates evaluated automatically at 30-day and 60-day marks
  • Kill criteria gate failure sends email notification to CTO + team leads
  • Per-role quickstart guides generated during team provisioning
  • In-app tooltip onboarding tour launches for first-time users
  • “Agent did something wrong” FAQ page accessible from help menu and feedback widget

Project Setup

Pre-Build

Everything here must be done BEFORE Phase 1 starts.

This is the complete project infrastructure plan: GitHub repositories, Jira project, Confluence space, roadmap, and the day-by-day pre-build schedule. All items are blockers for the 7-phase build clock.

1. GitHub Repositories

Decision: Multi-repo with one shared-packages exception. Each of the 4 service repos (plus infra and CD) deploys on its own cadence. Separate repos give each service its own CI pipeline, release tags, CLAUDE.md, and .faber.yml. Each service repo owns its own Dockerfile and docker/compose.yml for local dev. Helm charts live in praesidium-cd. The exception: Gateway and Worker share praesidium-llm and praesidium-hydration packages via a lightweight Python monorepo.

#RepositoryLanguageDescriptionGHCR Image
1praesidium2-qqqJava 21qqq data layer — 117 tables, REST API, Liquibase migrations, RLS. Owns its Dockerfile + .faber.yml (java-webapp)ghcr.io/dmdbrands/praesidium/qqq-api
2praesidium2-pythonPython 3.12Monorepo: agent-gateway (FastAPI), agent-worker (Celery), shared packages (llm, hydration). Owns its Dockerfiles + .faber.yml (python-webapp)ghcr.io/dmdbrands/praesidium/gateway, ghcr.io/dmdbrands/praesidium/worker
3praesidium2-tool-runnerPython 3.12K8s Job sandbox for custom tool execution. Minimal deps (pydantic, httpx only). Owns its Dockerfile + .faber.yml (python-webapp)ghcr.io/dmdbrands/praesidium/tool-runner
4praesidium2-webTypeScriptReact 19 SPA — TanStack Router/Query, Zustand, shadcn/ui, Tailwind. Owns its Dockerfile + .faber.yml (node-webapp)ghcr.io/dmdbrands/praesidium/web
5praesidium-infrastructureHCLOpenTofu/Terragrunt — AWS IaC (EKS, RDS, ElastiCache, Secrets Manager)N/A
6praesidium-cdYAMLArgoCD ApplicationSets, Helm charts, Helm values per environment, instance YAMLN/A

Repos 5 & 6 already exist and are NOT owned by the Praesidium2 team.

Do not modify them directly. Infrastructure and CD changes go through the existing change request process. Helm charts and ArgoCD config live in praesidium-cd.

Python Monorepo Structure

praesidium2-python/
  CLAUDE.md                    # Root-level rules (applies to both services)
  shared/
    llm/
      __init__.py
      protocol.py              # LLMClient Protocol interface
      anthropic_client.py      # V1 implementation
      factory.py               # build_llm_client() by provider type
      pyproject.toml            # name = "praesidium-llm"
    hydration/
      __init__.py
      hydrator.py              # Agent hydration: qqq fetch + Redis cache
      models.py                # HydratedAgent, HydrationContext
      pyproject.toml            # name = "praesidium-hydration"
  services/
    agent-gateway/
      CLAUDE.md                # Gateway-specific rules
      pyproject.toml
      src/
      tests/
    agent-worker/
      CLAUDE.md                # Worker-specific rules
      pyproject.toml
      src/
      tests/

Branch Strategy

BranchPurposeProtectionDeploy Target
mainProduction releases1 approval, passing CI, no force push, signed commitsProduction (via ArgoCD)
developIntegration branch1 approval, passing CI, no force pushStaging (via ArgoCD)
feature/*Feature workNone (CI runs on push)Local dev only
bugfix/*Bug fixesNone (CI runs on push)Local dev only
hotfix/*Production hotfixesBranch from main, merge to main AND developProduction (expedited)

Flow: feature/* → develop (PR) → main (PR, release)

Naming: feature/PR2-123-short-description, bugfix/PR2-456-fix-something, hotfix/PR2-789-critical-fix

PR Review Requirements

RuleSetting
Minimum approvals1 (human engineer)
Dismiss stale reviews on new pushYes
Require CI to passYes (Faber: lint + test + build)
Require up-to-date branchYes (rebase before merge)
Allow squash mergeYes (default)
Allow merge commitsNo
Allow rebase mergeYes
Require signed commitsYes (on main only)
CODEOWNERS required reviewYes
Auto-delete head branchesYes

CLAUDE.md per Repo

RepositoryCLAUDE.md Key Rules
praesidium2-qqqqqq conventions, table definitions in QqqMetaDataProvider.java, Liquibase SQL migrations, RecordSecurityLock on instance_id, Testcontainers for integration tests, 90%+ coverage
praesidium2-python (root)Cross-service rules: all data via qqq REST API (never touch DB directly), shared LLMClient abstraction, structlog for logging, Pydantic models at all boundaries
agent-gateway/SSE streaming, WebSocket pub/sub, JWT auth on all routes except /healthz, agent hydration with Redis cache (5min TTL), async/await everywhere, httpx.AsyncClient
agent-worker/Direct Anthropic SDK for “WHO does WHAT”, LangGraph for “HOW complex work flows” (durable/HITL only), HITL interrupt/resume, acks_late=True, tool safety rules
praesidium2-tool-runnerMinimal deps (pydantic + httpx), non-root container, read-only rootfs, NetworkPolicy egress restrictions, “Blind” Secret Pipeline
praesidium2-webReact 19, TanStack Router + Query, Zustand for client state only, JWT in memory only, shadcn/ui + Tailwind, pnpm, Vitest + Testing Library + MSW, 90%+ coverage

2. Jira Project

SettingValue
Project NamePraesidium2
Project KeyPR2
Project TypeSoftware (Scrum)
Project LeadJames Maes
Board TypeScrum
Sprint Duration1 week (Monday to Friday)
EstimationStory Points (Fibonacci: 1, 2, 3, 5, 8, 13)

10 Epics Mapped to Build Phases

EpicEpic NameTimelineKey Deliverables
E1FoundationsPhase 1qqq DB (117 tables), Authentik config, Frontend App Shell, Datadog infra, Faber pipeline skeleton, Helm charts, ArgoCD ApplicationSet
E2Tool RunnerPhase 2K8s Job sandbox, “Blind” Secret Pipeline (IRSA + Secrets Manager), tool execution framework, CI pipeline complete
E3Agent WorkersPhase 3LangGraph state machines, Postgres Checkpointing, HITL interrupt/resume, direct SDK agent execution, Celery task queues
E4GatewayPhase 4SSE streaming, chat UI, task orchestration, /api/search (pgvector), WebSocket broadcasts, OAuth setup
E5IntegrationsPhase 5Slack, Jira, GitHub toolsets, webhook listeners, integration tool dispatch, secret resolution
E6Business OpsPhase 6KPI dashboards, HITL Approval UI, schedule management, rollback logic, staging + production deploy
E7StabilizationPhase 7Security audit (OWASP), load testing (k6), bug fixes, Marketing rollout, KPI comparison, API docs
E8Pre-BuildPre-BuildKPI baseline survey, historical metrics extraction, target definition, environment setup, design finalization
E9V1.1 IntegrationsPost-LaunchConfluence, Zoho CRM, Google Workspace connectors
E10V1.5 Advanced FeaturesPost-LaunchTool Forge, Memory/RAG pipeline, Meeting Orchestration

9 Components

ComponentLeadDescription
qqq-apiJames MaesJava data layer, REST API, migrations
agent-gatewayUS Eng Manager 2FastAPI, SSE, WebSocket, chat, routing
agent-workerUS Eng Manager 2Celery, Anthropic SDK, LangGraph, task execution
tool-runnerUS Eng Manager 2K8s Job sandbox, tool execution
webUS Eng Manager 3React 19 SPA, all 11 pages
infrastructureJames MaesDocker, Helm, k8s, AWS IaC
integrationsJames MaesSlack, Jira, GitHub, OAuth connectors
securityJames MaesAuthentik, RBAC, secrets, audit
observabilityJames MaesDatadog APM/RUM/Logs, OpenTelemetry

Labels

CategoryLabels
Servicesvc:qqq-api, svc:gateway, svc:worker, svc:tool-runner, svc:web, svc:helm, svc:infra
Agent Workagent-coded (written by Claude Code), agent-reviewed (reviewed by F.R.I.D.A.Y.)
Scopescope:v1, scope:v1.1, scope:v1.5
Riskrisk:security, risk:data-loss, risk:breaking-change
Typetech-debt, performance, accessibility, documentation
Tracktrack:backend, track:frontend, track:infra, track:data

Automation Rules

RuleTriggerAction
Auto-assign on transitionIssue moved to IN PROGRESSAssign to current user if unassigned
PR merge → IN REVIEWPR merged in GitHub (webhook)Transition linked issue to IN REVIEW
Close on merge to mainPR merged to mainTransition linked issue to DONE
Sprint auto-closeSprint endsMove incomplete issues to next sprint
Stale issue alertIssue in IN PROGRESS > 3 days, no activityPost Slack notification to assigned engineer
Coverage gatePR fails coverage checkAdd comment to linked Jira issue with coverage delta
Label syncIssue created with componentAuto-apply matching svc:* label

Board Configuration

Columns: Backlog (hidden) → To Do → In Progress → In Review → Done

Swimlanes: By Epic

Quick Filters: My Issues, Backend Track, Frontend Track, Infrastructure Track, Blocked Items, Agent-Coded, Current Sprint Only

3. Confluence Space

SettingValue
Space NamePraesidium2
Space KeyPR2
Space TypeTeam Space
Space AdminJames Maes
PermissionsEngineering Leadership Team (edit), All Engineering (view)

Page Hierarchy

Praesidium2 (Home)
|
+-- Architecture
|   +-- System Design (mirrors system-design.html, kept up to date)
|   +-- Service Map (5 services, dependencies, communication patterns)
|   +-- Data Model (117 tables, ERD, type table catalog)
|   +-- Security Architecture (Authentik, RLS, RBAC, secrets)
|   +-- Observability Architecture (Datadog, OpenTelemetry, correlation IDs)
|   +-- Integration Architecture (Slack, Jira, GitHub, OAuth flows)
|
+-- Implementation Plan
|   +-- Timeline Overview (7-phase Gantt view)
|   +-- Phase 1: Foundations
|   +-- Phase 2: Tool Runner
|   +-- Phase 3: Agent Workers
|   +-- Phase 4: Gateway
|   +-- Phase 5: Integrations
|   +-- Phase 6: Business Ops
|   +-- Phase 7: Stabilization
|   +-- Pre-Build Checklist
|   +-- V1.1 Plan (Post-Launch)
|   +-- V1.5 Plan (Post-Launch)
|
+-- Runbooks
|   +-- qqq Configuration & Table Setup
|   +-- Authentik Setup (Groups, Policies, OIDC, Scope Mappings)
|   +-- AWS Deployment (EKS, RDS, ElastiCache, Secrets Manager)
|   +-- ArgoCD Operations (Sync, Rollback, ApplicationSet)
|   +-- Faber CI Pipeline Configuration
|   +-- Local Development Environment (Tilt + k8s)
|   +-- Redis Operations (Celery broker, pub/sub, cache)
|   +-- Datadog Setup (APM, RUM, Logs, SDS, Dashboards)
|   +-- Incident Response Playbook
|   +-- Secrets Rotation Procedures
|
+-- ADRs (Architecture Decision Records)
|   +-- ADR-001: Multi-Repo with Python Monorepo Exception
|   +-- ADR-002: Anthropic SDK + LangGraph (Scoped) Architecture
|   +-- ADR-003: qqq as Data Layer (generated REST APIs)
|   +-- ADR-004: Redis as Celery Broker (over RabbitMQ/SQS)
|   +-- ADR-005: Faber CI over CircleCI/GitHub Actions
|   +-- ADR-006: Datadog over ELK/Grafana for Observability
|   +-- ADR-007: "Blind" Secret Pipeline (IRSA + Secrets Manager)
|   +-- ADR-008: Tool Runner Isolation (K8s Jobs)
|
+-- Sprint Reports (Sprint 1-7, one per build phase)
+-- Retrospectives (30-Day, 60-Day, 90-Day)
+-- Team Directory
+-- Onboarding Guide

Templates

TemplateUsed For
Sprint ReportWeekly sprint summaries (velocity, completed stories, blockers, next sprint goals)
Retrospective30/60/90 day reviews (what went well, what didn’t, action items)
ADRArchitecture decisions (context, decision, consequences, status)
RunbookOperational procedures (prerequisites, steps, rollback, verification)
Phase Implementation UpdatePer-phase build progress (exit criteria status, blockers, risks)

Initial Pages to Create Before Phase 1

  • Home page with roadmap link and quick links to all sections
  • Architecture section populated from system-design.html
  • Implementation Plan section populated from implementation.html
  • Onboarding Guide with environment setup, repository tour, and dev workflow
  • Team Directory with build team + leadership roster, roles, and time zones
  • All 5 templates created and published to the space
  • Confluence-Jira macros configured for live issue embedding

4. Roadmap

7-phase build plan + post-launch phases with all 10 Epics plotted. Use Jira’s built-in Roadmap view (formerly Plans), grouped by Epic.

Phase & Epic Map

PRE-BUILD
Infrastructure & Design
E8 Pre-Build
PHASE 1
Foundations
E1 Foundations
PHASE 2
Tool Runner
E2 Tool Runner
PHASE 3
Agent Workers
E3 Agent Workers
PHASE 4
Gateway
E4 Gateway
PHASE 5
Integrations
E5 Integrations
PHASE 6
Business Ops
E6 Business Ops
PHASE 7
Stabilize
E7 Stabilize
★ V1 Launch
POST-LAUNCH
V1.1
E9 V1.1
POST-LAUNCH
V1.5
E10 V1.5

10 Milestones

MilestoneTargetCriteria
Pre-Build CompletePre-Build (end)KPI baseline captured, dev environments ready, design finalized, all repos created
Phase 1: FoundationPhase 1 (end)qqq DB provisioned (117 tables), Authentik configured, Frontend App Shell running, Datadog instrumented, all Faber pipelines green
Phase 2: Hands ReadyPhase 2 (end)Tool Runner executing in K8s Jobs, “Blind” Secret Pipeline working, CI pipeline complete
Phase 3: Brain OnlinePhase 3 (end)LangGraph state machines running, checkpoints surviving restarts, HITL interrupt working
Phase 4: ConnectedPhase 4 (end)SSE streaming live, chat functional, task orchestration working, OAuth flows registered
Phase 5: IntegratedPhase 5 (end)Slack + Jira + GitHub end-to-end, webhook listeners active, secret resolution by scope
Phase 6: DeployedPhase 6 (end)Production deployed, Engineering team active on platform, KPI dashboards live
V1 LaunchPhase 7 (end)Security audit clean, load testing passing, Marketing team onboarded, 90%+ coverage, zero critical bugs
V1.1Post-LaunchConfluence, Zoho CRM, Google Workspace connectors live
V1.5Post-LaunchTool Forge, Memory/RAG, Meeting Orchestration live

Release Versioning

VersionContentType
v0.1.0Phase 1 Foundation (qqq DB, Auth, App Shell)Internal milestone
v0.2.0Phase 2 + Tool RunnerInternal milestone
v0.3.0Phase 3 + Agent WorkersInternal milestone
v0.4.0Phase 4 + GatewayInternal milestone
v0.5.0Phase 5 + IntegrationsInternal milestone
v0.9.0Phase 6 + Business Ops (staging)Release candidate
v1.0.0Phase 7 + Stabilization (V1 Launch)Production release
v1.1.0Confluence, Zoho, Google WorkspaceFeature release
v1.5.0Tool Forge, Memory/RAG, MeetingsFeature release

5. Pre-Build Checklist

Everything to set up before the Phase 1 build clock starts. All items are blockers.

  • Create all 7 GitHub repos with branch protection
  • Add CODEOWNERS, CLAUDE.md, .gitignore, LICENSE to each repo
  • Set up GHCR repositories (ghcr.io/dmdbrands) for all service images
  • Set up Jira project with 10 Epics, board, components, labels, automation rules
  • Create Confluence space with full page hierarchy
  • Create Confluence templates (Sprint Report, Retro, ADR, Runbook)
  • Configure .faber.yml in each service repo (java-webapp, python-webapp, node-webapp)
  • Set up AWS accounts (Dev, Staging, Prod) in Control Tower
  • Run terraform init + first plan on dev account
  • Create service account keys for CI/CD
  • Set up Datadog organization with API keys
  • Configure VPN + bastion host access
  • Set up ArgoCD + CD repo configuration
  • Begin KPI baseline survey (2-week time tracking)
  • Verify all build team + leadership members have access to GitHub, Jira, Confluence, AWS (dev)

6. Day-by-Day Pre-Build Schedule

Parallel workstreams across the three US engineering leads. Target: 10 business days before Phase 1.

DayJames (CTO)Mark (SysAdmin)Barath (Tech Lead)
Day −10 Create GitHub repos + branch rules Set up AWS Control Tower accounts Set up Jira project + Epics
Day −9 Write CLAUDE.md for each repo Run terraform init on dev Create Confluence space + pages
Day −8 Configure .faber.yml per service repo Set up Datadog org + API keys Distribute KPI baseline survey
Day −7 Set up ArgoCD + CD repo Configure VPN + bastion Create sprint board + automation
Day −5 to −1 Review infra plan, finalize architecture Verify all AWS resources provisioned Onboard India team to repos/Jira/Confluence

Acceptance Criteria

Pre-Build is complete when ALL of the following are verified:

  • All 7 GitHub repos created with CLAUDE.md, branch protection, CODEOWNERS
  • Jira PR2 project with 10 Epics, board, components, automation rules
  • Confluence PR2 space with architecture, runbooks, ADR, onboarding pages
  • CI/CD pipeline running lint/test on at least one repo
  • AWS dev account provisioned with VPC, EKS, RDS (basic terraform apply)
  • KPI baseline survey distributed to all 5 team leads
  • All build team + leadership members have access to GitHub, Jira, Confluence, AWS (dev)