Implementation Plan
Based on Design v1.12 →This document provides the step-by-step implementation instructions for building Praesidium2. It is designed to be consumed by Claude Code agents who will perform approximately 90% of the implementation work. Each section contains specific enough instructions that an agent can execute without ambiguity. The Praesidium2 build team is James Maes (CTO) + 2 engineers + unlimited Claude Code agents. The full Engineering Leadership Team (listed below) provides architectural oversight, code review, and the remaining 10% of specialized integration work.
Team Assignments
The Praesidium2 build team is James Maes (CTO) + 2 engineers + unlimited Claude Code agents. The full Engineering Leadership Team shown here provides oversight, review, and domain expertise.
James (CTO)
Architecture + qqq + Infrastructure + Integrations
Primary owner of qqq data layer, infrastructure (Docker/k8s/Helm/ArgoCD), Faber CI/CD, Authentik configuration, and external integrations. Architects the foundation that the team builds upon.
US Eng Manager 2
Backend Lead — Python Agent Runtime
Leads the Agent Gateway (FastAPI), Agent Workers (Celery), Anthropic SDK + LangGraph orchestration, and tool execution framework. Reviews and guides India Senior Engineers on backend work.
US Eng Manager 3
Frontend Lead — React SPA
Leads the React/TypeScript SPA: all 11 pages, component library, WebSocket/SSE integration, Authentik OIDC login, and end-to-end frontend testing. Reviews and guides India Senior Engineers on frontend work.
India SE 1-2
Backend — Gateway + Workers
Implement Gateway endpoints, Worker task graphs, integration connectors (Slack, Jira, GitHub), and Celery Beat scheduling. Directed by US Eng Manager 2.
India SE 3-4
Frontend — React Pages + Components
Build React pages, component library, WebSocket/SSE integration, and frontend testing. Directed by US Eng Manager 3.
India SE 5
QA + Testing + DevOps
End-to-end testing (Playwright), CI pipeline maintenance, Docker/k8s configuration, test coverage enforcement, and security scanning.
Claude Code Agents (90% of Implementation)
Each developer directs Claude Code agents for their track
Claude Code agents write the vast majority of code, tests, and configuration. Human developers provide context via CLAUDE.md files, review pull requests, and handle decisions that require institutional knowledge or external access (AWS console, Authentik admin UI, etc.).
CLAUDE.md Rules (Per Service)
- Each service repo gets its own
CLAUDE.mdwith architecture context, coding patterns, and constraints - Agents MUST read
CLAUDE.mdbefore making any changes - Agents MUST NOT modify files outside their service boundary
- Agents MUST write tests for every new endpoint/function
- Agents MUST use the type table pattern (no magic strings)
- All data access MUST go through qqq REST APIs (agent runtime never touches DB directly)
Key Accelerators
- qqq generates REST APIs from ERD — no hand-writing CRUD endpoints
- Claude Code scaffolds entire services, writes tests, generates boilerplate
- 3-person build team + unlimited Claude Code agents working in parallel on independent tracks from Day 2 onward
- Existing Praesidium v1 and me-health-portal provide reference implementations
- Design spec provides complete table schemas — no ambiguity on data model
Claude Code Agent Licensing Change (April 4, 2026)
Anthropic blocked third-party harnesses — including OpenClaw (formerly OpenHands) — from drawing on Claude Pro/Max subscription credits. As of April 4, 2026 at 12pm PT, tools like OpenClaw can no longer use a Claude subscription. Running Claude Code agents now requires either a direct Anthropic API key (pay-per-use) or an extra usage bundle from Anthropic. Budget planning for the build phases should account for API token costs rather than assuming subscription coverage.
Realistic Timeline (7 Phases)
3-person build team (James Maes, CTO + 2 engineers) + unlimited Claude Code agents working 24/7. Existing AWS EKS + Faber CI/CD + ArgoCD.
Agent-first development model: Claude Code agents handle the bulk of coding. F.R.I.D.A.Y. handles PR review (humans review the review). Build team focuses on architecture decisions, code review, and integration testing. Phases can be completed faster than calendar weeks when AI agents are productive. Scope cuts applied: Tool Forge, Memory/RAG, and Meeting Orchestration deferred to V1.5.
Phases are not calendar weeks. The team may complete phases faster than estimated — the goal is to move through phases as quickly as quality allows.
Phases Are Not Calendar Weeks
Phases are not calendar weeks — the team may complete phases faster than estimated. The goal is to move through phases as quickly as quality allows. Each phase has explicit exit criteria; the next phase begins when the current phase's criteria are met, not when a calendar week ends.
Pre-Build: KPI Baseline Capture
Before any code is written, capture the ROI measurement baseline. Each team lead logs hours on tasks that agents will eventually handle. This 2-week survey runs while dev environment setup and design finalization happen in parallel.
| Activity | Description | Owner |
|---|---|---|
| Time-Tracking Survey | 2-week time-tracking survey distributed to Engineering and Marketing team leads. Each participant logs daily hours spent on: status reporting, Jira grooming, Slack triage, meeting scheduling, document creation, and code review coordination. | James (CTO) + Team Leads |
| Baseline Metrics | Pull historical data from Jira, GitHub, and Slack: average task completion time, throughput per team member, PR cycle time, meeting hours per week, error/rework rates. Store in kpi_baseline table schema (deployed Phase 1). |
James (CTO) |
| Success Criteria | Define measurable targets with stakeholders: e.g., 30% reduction in status reporting time, 50% faster Jira triage, 20% fewer context switches. These become the V1 KPI targets. | James (CTO) + Stakeholders |
Pre-Build Exit Criteria
- Minimum 2 weeks of time-tracking data from each team lead
- Historical metrics extracted from Jira, GitHub, and Slack APIs
- Documented baseline values for all KPIs that Praesidium will impact
- Stakeholder-agreed success targets for V1 ROI measurement
Phase 1: Foundations
qqq DB provisioning (117 tables), Authentik claim mapping, Frontend App Shell, Datadog infrastructure. Full build team + AI agents establish the foundation in parallel.
| Period | James (CTO) + India SE 5 — Infra + qqq | Backend Track (US Mgr 2 + India SE 1-2) | Frontend Track (US Mgr 3 + India SE 3-4) |
|---|---|---|---|
| P1 D1-2 | Tiltfile + Helm values-local.yaml (PG+pgvector, Redis, MinIO, Authentik, Datadog Agent), local k8s namespace. All 117 tables in Liquibase (including llm_provider, llm_model, secret_ref, audit_access_log, task_approval, langgraph_checkpoints) + seed all 59 type tables. qqq config for REST generation. |
Python project scaffold (FastAPI + Celery + pyproject.toml), Dockerfile. qqq Python API client (auto-generated from OpenAPI). Agent hydration service skeleton. Add ddtrace to pyproject.toml. |
React scaffold (Vite + TanStack Router/Query + shadcn/ui), Dockerfile. Authentik OIDC login flow. App shell (sidebar, header, command palette, routing). Install @datadog/browser-rum + @datadog/browser-logs. |
| P1 D3-5 | Authentik blueprint (groups, policies, scope mappings including praesidiumInstanceId). RLS via qqq RecordSecurityLock. .faber.yml per repo. Helm charts in praesidium-cd. ArgoCD ApplicationSet. Deploy Datadog Cluster Agent + Admission Controller to EKS. Configure DD_API_KEY in Secrets Manager. Instrument qqq-api with dd-java-agent. |
LLMClient abstraction (Anthropic-only for V1). AGENT_MANIFEST schema definition (Day 3). Agent hydration (template + tools). POST /chat endpoint with SSE streaming. Inline chat executor with tool loop. Platform tools. | Dashboard page (project health cards, team status, escalations count, cost). Type table cache in Zustand. Projects page (card grid + tabbed detail). Team page (agent + human grid). Datadog RUM initialization with allowedTracingOrigins. |
Phase 1 Exit Criteria
- Full stack running locally in Docker Desktop k8s via Tilt (all pods)
- All 117 tables deployed with REST APIs, auth, RLS
- AGENT_MANIFEST JSON schema finalized and validated by both Java and Python
- Datadog Agent collecting container logs + metrics
- Authentik JWT with
praesidiumInstanceIdandpoliciesclaims verified - Frontend App Shell with auth, command palette, and core page stubs
- CI pipeline green
Phase 2: The Hands (Tool Runner)
K8s Job sandboxing and the “Blind” Secret Pipeline built before Agent Workers to prevent testing deadlocks. Workers need something to call; Tool Runner provides it.
| Period | James (CTO) + India SE 5 — Infra | Backend Track — Tool Runner | Frontend Track — Pages |
|---|---|---|---|
| P2 D6-8 | IRSA IAM role for Worker pod (Secrets Manager access). secret_ref table seeded with test ARNs. Secrets Manager integration end-to-end. CI pipeline complete (Faber: lint + test + build + GHCR push). |
Tool Runner base image (Dockerfile.python). BaseTool, ToolContext, ToolResult classes. CLI runner (runner.py). k8s Job template with SHA-only git checkouts (A-004 fix). DD_TRACE_ID + DD_PARENT_ID as env vars in Job spec. |
Dashboard page finalization. Chat page (SSE streaming + tool call cards). WebSocket hook for live updates. Escalations page (inbox, severity sort). Settings page (instance config, integrations). |
| P2 D9-10 | NetworkPolicy for Tool Runner pods (egress restricted). k8s resource quota presets per risk level. Infrastructure hardening. | “Blind” Secret Pipeline: Worker fetches secret_ref ARN from qqq, uses IRSA to fetch value from AWS Secrets Manager at runtime (plain-text never in DB). Verify “Hello World” tool execution from Gateway bus. Integration tests. |
Activity page (feed with filters). Cost Center page (burn rate charts). Kanban board (drag-and-drop tasks). Agent detail page. Permission-driven UI. |
Phase 2 Exit Criteria
- Tool Runner CLI produces valid
ToolResultJSON on stdout - k8s Job runs a CUSTOM tool in an ephemeral pod and exits cleanly
- “Blind” secret pipeline working:
secret_refARN → IRSA → Secrets Manager → env var - No plain-text API keys visible in qqq API responses or database
DD_TRACE_IDpropagated into Tool Runner Job spec- NetworkPolicy blocks all egress except pre-approved integration URLs
Phase 3: The Brain (Agent Workers)
LangGraph state machines, Postgres Checkpointing, and the HITL interrupt() logic. Tool Runner (Phase 2) provides the execution target that workers can call.
| Period | James (CTO) + India SE 5 — Infra | Backend Track — Agent Workers | Frontend Track — Pages |
|---|---|---|---|
| P3 D11-13 | Multi-LLM support + provider health checks. Budget controls. Datadog APM dashboards (custom praesidium.agent.* metrics via DogStatsD). Service Map verification. |
LangGraph “Reasoning” loop implementation. PostgresSaver checkpointer setup. job_graph.py (load → execute → evaluate). execute_job task. Instrument graph.invoke() with ddtrace.tracer.trace("agent.reasoning_loop") span. Capture LLM token usage as span metadata. |
Chat improvements (thinking indicator, group chats). Agent detail (skills radar). Cost Center charts + budget UI. Review/publish flows. |
| P3 D14-15 | HPA tuning, security audit. Network policies. Trace ID propagation through Celery headers (dd_headers). |
Implement interrupt() for CRITICAL tool calls. State serialized to langgraph_checkpoints table before approval request. Task status set to AWAITING_APPROVAL. task_approval table write with action_payload_hash (SHA-256). Secret resolution (scope hierarchy: AGENT > TOOL > PROJECT > INSTANCE). Stats reconciliation job. |
Settings improvements (notification preferences). Responsive design. Error boundaries and loading skeletons. Integration settings page (OAuth connection status indicators). |
Phase 3 Exit Criteria
- LangGraph state machine executing jobs with PostgreSQL checkpointing
interrupt()pauses execution and writes state tolanggraph_checkpointstask_approvalrecord created with tamper-proofaction_payload_hash- Worker pod can be killed during pending approval; new pod resumes from checkpoint
- Trace ID propagated: Frontend RUM → Gateway → Celery headers → Worker spans → Tool Runner env vars
- Custom Datadog metrics emitting (
praesidium.agent.token_usage,praesidium.tool.execution_status)
Phase 4: The Gateway (Communication Hub)
SSE Streaming, Chat UI finalization, Task Orchestration, Command Palette with /api/search backend (pgvector). The Gateway now has workers + tool runner to dispatch to.
| Period | James (CTO) + India SE 5 — Infra | Backend Track — Gateway | Frontend Track — Chat + Search |
|---|---|---|---|
| P4 D16-18 | Begin OAuth setup: Slack OAuth app registration + callback. GitHub OAuth App registration + token exchange. OAuth token storage in Secrets Manager. | Gateway SSE server (GET /api/stream/{conversation_id}). POST /api/chat/{agent_id} inline executor. POST /api/task/{task_id}/execute dispatcher. GET /api/internal/worker/hydrate/{agent_id} composite endpoint (AGENT_MANIFEST delivery). POST /api/task/{id}/approve with action hash verification + Redis resume event. SSE interrupt event: when worker interrupt() fires, Gateway pushes interrupt SSE event with approval_id, action_type, and action_summary to the browser. |
Chat thread UI: SSE streaming tokens, ThinkingIndicator, ToolCallCard. Chat input with model selector. Chat list page with conversation previews. Handle SSE interrupt event: transition Chat UI from “Thinking...” to “Approval Required” card with action summary, diff preview, and Approve/Reject buttons. |
| P4 D19-20 | Slack and GitHub OAuth flows end-to-end. Token refresh logic. Integration health check framework. | GET /api/search — Command Palette backend (pgvector semantic search across projects, agents, tasks, conversations, tools). WebSocket server with Redis pub/sub broadcast. Prometheus metrics + Correlation ID middleware. Rate limiter. |
Command Palette (Cmd+K) wired to /api/search backend. Meetings list + escalations. Group chats. Activity feed (real-time via WS). Responsive design verification. |
Phase 4 Exit Criteria
- SSE streaming chat working end-to-end (browser → gateway → LLM → SSE tokens)
- SSE
interruptevent triggers frontend transition from ThinkingIndicator to Approval Required card - Task dispatch creates Celery task; WS broadcasts work across pods
/api/internal/worker/hydrate/{agent_id}returns complete AGENT_MANIFEST/api/task/{id}/approvevalidates hash, dispatches Redis resume event/api/searchreturns ranked results from pgvector for Command Palette- Slack and GitHub OAuth flows registered with token exchange working
Phase 5: Real-World Integrations
Slack, Jira, GitHub toolsets and Webhook listeners. OAuth flows from Phase 4 accelerate connector work. Confluence, Zoho, and Google Workspace deferred to V1.1.
| Period | James (CTO) — Integrations | Backend Track — Connectors | Frontend Track — Integration UI |
|---|---|---|---|
| P5 D21-23 | Slack connector (DM, channel post, interactive buttons). Jira connector (create/transition/search issues, bidirectional sync). Webhook receiver in Gateway. | GitHub connector (PR creation, issue filing, code search). Integration tool dispatch (PLATFORM + INTEGRATION in-process). Secret resolution with IRSA for integration credentials. | Integration connection management UI. OAuth connection status indicators in Settings. Connector status dashboard. Approval notification indicators in sidebar. |
| P5 D24-25 | Human interaction flow (Slack DM approvals). Integration health checks. Cross-connector integration tests. | Celery Beat scheduler with Redis-based leader election. Scheduled job execution. scheduler_heartbeat table: leader writes heartbeat every 30s, secondary monitors and takes over if last_heartbeat > 60s stale. Integration tests across all 3 connectors. Webhook event routing to agents/tasks/projects. |
Inline approval prompts in chat view for HIGH/CRITICAL actions. End-to-end approval flow testing. Settings improvements (notification preferences). |
Phase 5 Exit Criteria
- Slack, Jira, GitHub integrations working end-to-end
- Webhook listener in Gateway routes external events to correct agent/task
- Tool registry resolves PLATFORM and INTEGRATION tools with risk-level gates
- Celery Beat scheduled job execution with leader election
- Scheduler HA:
scheduler_heartbeattable tracks leader liveness; failover tested within 60s - Slack and GitHub OAuth flows fully operational with token refresh
Phase 6: Business Operations
KPI Dashboards, HITL Approval UI (with action hash verification), Schedule Management UI, Rollback logic, Human Feedback, and production deployment preparation.
| Period | James (CTO) — Production | Backend Track — Business Logic | Frontend Track — Business UI |
|---|---|---|---|
| P6 D26-28 | AWS deployment via existing EKS + Faber (.faber.yml) + ArgoCD. Helm values in praesidium-cd. Secrets Manager. Staging deploy, smoke test. |
KPI baseline import (load Pre-Build survey data into kpi_baseline table). KPI measurement service. Agent quality score computation. Rollback/recovery: undo API endpoints with revert_payload logic. Emergency kill switch. |
HITL Approval Dashboard — pending approvals list with action hash verification (payload shown matches action_payload_hash). Approve/reject/request-changes actions. KPI dashboard with baseline comparison charts. Schedule Management UI — Schedule List (/schedules), Schedule Detail/Edit (/schedules/:id), Run History, cron picker, and + New Schedule form. |
| P6 D29-30 | Production deploy. Phased rollout: deployment checklist, go/no-go criteria. First instance provisioned (Engineering team). | Human feedback collection endpoint. Approval notification pipeline (Slack DM on pending). Approval timeout/escalation logic. Agent performance management lifecycle. Kill criteria gate evaluation job. | Human Feedback widget (thumbs up/down + comment). Rollback controls in UI. Emergency kill switch in admin panel. Agent performance management UI. Kill criteria gate status cards. |
Phase 6 Exit Criteria
- Production-ready on AWS EKS (leveraging existing infrastructure)
- Engineering team deployed and actively using the platform
- HITL Approval UI with tamper-proof action hash verification
- Rollback/undo capability for agent actions + emergency kill switch tested
- KPI baseline imported from Pre-Build survey, comparison dashboard live
- Schedule Management UI live — Schedule List, Detail/Edit, Run History, cron picker, + New Schedule
- Human feedback widget live on all agent output surfaces
- Business-value kill criteria gates evaluated automatically
- Datadog Service Map shows full 5-service waterfall with single Trace ID
Phase 7: Stabilization
Security audit, load testing, production hardening, second team rollout (Marketing), and final polish.
| Period | James (CTO) + India SE 5 — Ops | Backend Track — Stabilization | Frontend Track — Polish |
|---|---|---|---|
| P7 D31-33 | Security audit (OWASP, dependency scan). Load testing (k6). Bug fixes from Engineering team feedback. Infrastructure tuning. Monitoring refinement. Datadog SDS rules for PHI redaction verified. | Bug fixes from production usage. Performance optimization. Approval workflow tuning. Checkpoint pruning job (90-day retention). E2E backend testing. Cost tracking accuracy validation. | UX improvements based on Engineering team feedback. E2E frontend testing (Playwright). Accessibility audit. Cross-browser. In-app onboarding tooltip tour. Quality score visualization. |
| P7 D34-35 | Marketing team provisioning. Instance configuration for Marketing use cases. Go/no-go check for second team. Grafana/Datadog dashboards live, budget alerts working. | Marketing team agent templates. Team-specific integration configs. KPI comparison report (Phase 1 vs baseline). API documentation generated. Quickstart guide generation. | Marketing-specific dashboard views. Template library for Marketing workflows. Final UI polish. 90%+ test coverage across all services. |
Phase 7 Exit Criteria (V1 Launch Complete)
- Engineering team stable on platform for 1+ week
- Marketing team onboarded and productive
- KPI improvement measurable vs baseline
- Approval workflow validated with real human-in-the-loop usage
- Zero critical bugs in production
- Security audit clean, load testing passing
- 90%+ test coverage across all services
- Documentation complete
V1.1 (Post-Launch): Additional Integrations
- Confluence connector (page CRUD + search)
- Zoho CRM connector (contact/deal lookup + create)
- Google Workspace connector (Calendar events, Docs creation)
- Integration tests for all new connectors
V1.5 (Post-Launch): Advanced Features
- Tool Forge: LangGraph 6-state workflow, tool-runner sandboxed execution, forge UI (split-pane build mode)
- Memory/RAG: Memory extraction, embedding generation, semantic recall, knowledge document indexing
- Meeting Orchestration: Turn-taking orchestrator standups/huddles, action item extraction, meeting scheduling
Service Dependency Graph
Services must be built in this order. Each layer depends on the layers above it.
Tilt + k8s
Local dev environment
qqq Data Layer
117 tables + REST APIs + RLS
Authentik Config
Groups, policies, OIDC
Datadog
APM, RUM, Logs, SDS
Tool Runner
K8s Jobs + Blind Secret Pipeline
Agent Workers
LangGraph + HITL + Checkpoints
Agent Gateway
SSE + /api/search + HITL approve
Frontend SPA
React + Cmd+K + HITL UI
Integrations
Slack, Jira, GitHub, etc.
Critical path: Local k8s + Datadog → qqq + Authentik → Tool Runner → Agent Workers → Gateway + Frontend → Integrations
qqq Data Layer
Epic 1 — Days 1-2This is the FIRST service to build. Everything else depends on it.
The qqq data layer provides all 117 tables (58 entity + 59 type) with auto-generated REST APIs, row-level security (RLS), and Authentik JWT authentication. Includes 4 new tables from the Agent Runtime Spec: llm_provider, llm_model, secret_ref, and audit_access_log, plus scheduler_heartbeat (v1.9 HA). No other service can start meaningful work until qqq APIs are available.
Repository Setup
| Repository | praesidium2-qqq/ |
| Language | Java 21 |
| Build Tool | Gradle (Kotlin DSL) |
| Framework | qqq + Spring Boot |
| Database | PostgreSQL 17 + pgvector |
| Migrations | Liquibase |
| Auth | Authentik OIDC (ProdAuthenticationCustomizer pattern) |
Table Implementation Order
Tables MUST be created in this order due to foreign key dependencies. Type tables have zero FK dependencies and go first. Entity tables are ordered by domain with dependency-aware sequencing.
Phase 1: Extensions (changeset 000)
| Changeset | Action |
|---|---|
001-pgvector.sql | CREATE EXTENSION IF NOT EXISTS vector; |
Phase 2: Type Tables (changesets 100-159)
All 59 type tables follow the standard pattern. Create all of them before any entity tables. Order does not matter between type tables (no inter-dependencies).
| Changeset | Table | Seed Values |
|---|---|---|
101 | member_type | AGENT, HUMAN |
102 | member_status | ACTIVE, AWAY, OFFLINE, SUSPENDED |
103 | agent_category | ENGINEERING, HR, OPS, FINANCE, MARKETING, PRODUCT, SUPPORT, CUSTOM |
104 | execution_status | IDLE, EXECUTING, QUEUED, ERROR |
105 | tool_category | PLATFORM, INTEGRATION, CUSTOM, MCP |
106 | tool_source_type | BUILTIN, FORGED, IMPORTED |
107 | risk_level | LOW, MEDIUM, HIGH, CRITICAL |
108 | test_status | PENDING, RUNNING, PASSED, FAILED |
109 | programming_language | PYTHON, TYPESCRIPT, JAVA, GO, SHELL |
110 | sync_status | SYNCED, PENDING, SYNCING, ERROR |
111 | integration_type | ZOHO_CRM, JIRA, CONFLUENCE, SLACK, GITHUB, GOOGLE_WORKSPACE, CUSTOM_API |
112 | auth_type | API_KEY, OAUTH2, BASIC, BEARER, SERVICE_ACCOUNT |
113 | health_status | HEALTHY, DEGRADED, DOWN, UNKNOWN |
114 | project_type | SHORT_TERM, ONGOING, SUPPORT, INITIATIVE |
115 | project_status | PLANNING, ACTIVE, ON_HOLD, COMPLETED, ARCHIVED |
116 | project_health | ON_TRACK, AT_RISK, BEHIND, BLOCKED |
117 | priority_level | LOW, MEDIUM, HIGH, CRITICAL |
118 | project_role | LEAD, MEMBER, REVIEWER, OBSERVER |
119 | milestone_status | NOT_STARTED, IN_PROGRESS, COMPLETED, BLOCKED |
120 | task_type | TASK, BUG, STORY, SPIKE, REVIEW |
121 | task_status | OPEN, APPROVED, IN_PROGRESS, IN_REVIEW, DONE, BLOCKED, DISMISSED |
122 | author_role | AGENT, HUMAN, SYSTEM |
123 | conversation_type | CHAT, STANDUP, HUDDLE, TASK_EXEC, FORGE, REVIEW |
124 | conversation_status | ACTIVE, PAUSED, COMPLETED, FAILED |
125 | participant_role | FACILITATOR, PARTICIPANT, OBSERVER, SCRIBE |
126 | message_role | USER, ASSISTANT, SYSTEM, TOOL_RESULT |
127 | content_type | TEXT, MARKDOWN, JSON, IMAGE_REF, CODE |
128 | meeting_type | STANDUP, HUDDLE, RETROSPECTIVE, PLANNING, REVIEW |
129 | memory_type | FACT, PREFERENCE, LESSON_LEARNED, ENTITY, DECISION, PROCEDURE |
130 | memory_source | CONVERSATION, TASK, MEETING, MANUAL, SYSTEM |
131 | document_source_type | UPLOAD, CONFLUENCE, URL, GIT_REPO |
132 | index_status | PENDING, INDEXING, INDEXED, FAILED |
133 | job_status | ACTIVE, PAUSED, COMPLETED, FAILED |
134 | job_execution_status | RUNNING, SUCCESS, FAILED, TIMEOUT, CANCELLED |
135 | trigger_type | CRON, MANUAL, RETRY, EVENT |
136 | escalation_status | OPEN, ACKNOWLEDGED, RESOLVED, DISMISSED |
137 | artifact_type | FILE, REPORT, CODE, IMAGE, DATA |
138 | interaction_channel | SLACK, JIRA_COMMENT, JIRA_ASSIGN, CONFLUENCE, EMAIL |
139 | interaction_type | QUESTION, TASK_ASSIGN, STATUS_UPDATE, ESCALATION, REVIEW_REQUEST, INFO_REQUEST |
140 | interaction_status | SENT, DELIVERED, SEEN, RESPONDED, EXPIRED, FAILED |
141 | schedule_status | ACTIVE, PAUSED, ARCHIVED |
Phase 3: Entity Tables by Domain (changesets 200-234)
Entity tables MUST be created in this order. Each domain builds on previous domains' tables.
| Order | Domain | Tables | Depends On |
|---|---|---|---|
| 1 | Domain 1b: Team Identity | team_member, human_profile |
Type tables only |
| 2 | Domain 1: Agent Identity + LLM Config | agent_template, agent, agent_skill, llm_provider, llm_model, secret_ref |
team_member |
| 3 | Domain 2: Tools (partial) | tool_repo, integration_config, tool, tool_version, agent_tool_assignment |
agent |
| 4 | Domain 3: Projects | project, project_member, milestone, task, task_comment |
team_member |
| 5 | Domain 4: Conversations | scheduled_meeting, conversation, conversation_participant, message |
project, task, team_member |
| 6 | Domain 5: Memory & Knowledge | memory v1.10: +recency_weight, +importance, knowledge_document, knowledge_chunk |
team_member, project, conversation |
| 7 | Domain 6: Operations | scheduled_job, job_execution v1.10: +debug_payload, +trace_id, escalation, artifact, activity_log, audit_access_log, scheduler_heartbeat |
agent, project, task, conversation |
| 8 | Domain 1b: Human Interaction | human_interaction |
team_member, conversation, task |
| 9 | Domain 7: Business Operations | task_approval_queue, approval_policy, autonomy_level, kpi_baseline, kpi_measurement, team_roi_report, agent_action_log, rollback_record, output_feedback, agent_quality_score |
agent, task, team_member, project |
Liquibase Changelog Structure
Each changeset is a separate SQL file. The master changelog includes them in order.
# db.changelog-master.yaml
databaseChangeLog:
- includeAll:
path: db/changelog/000-extensions/
- includeAll:
path: db/changelog/100-type-tables/
- includeAll:
path: db/changelog/200-entity-tables/
- includeAll:
path: db/changelog/300-seeds/
- includeAll:
path: db/changelog/400-indexes/
- includeAll:
path: db/changelog/500-triggers/
Type Table SQL Template
Every type table MUST follow this exact pattern. Copy-paste and change the table name.
-- changeset praesidium:101-member-type
CREATE TABLE member_type (
id INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
name VARCHAR(50) NOT NULL UNIQUE,
label VARCHAR(100) NOT NULL,
description VARCHAR(500),
icon VARCHAR(50),
color VARCHAR(7),
sort_order INT DEFAULT 0,
is_active BOOLEAN DEFAULT true,
is_system BOOLEAN DEFAULT false,
metadata JSONB DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ DEFAULT now()
);
-- Seed data
INSERT INTO member_type (name, label, sort_order, is_system) VALUES
('AGENT', 'Agent', 1, true),
('HUMAN', 'Human', 2, true);
qqq Configuration for REST API Generation
qqq auto-generates CRUD REST endpoints for every table defined in the QqqMetaDataProvider. The configuration maps each table's columns, relationships, and security constraints.
What qqq Generates Automatically
- GET /api/{table} — List with pagination, sorting, filtering
- GET /api/{table}/{id} — Get by ID with optional
?include= - POST /api/{table} — Create with validation
- PATCH /api/{table}/{id} — Partial update
- DELETE /api/{table}/{id} — Delete (policy-controlled)
- OpenAPI/Swagger documentation
- Automatic FK resolution in responses
- Query parameter filtering on all columns
Composite Endpoints (qqq ?include=)
| Endpoint | Replaces |
|---|---|
GET /agent/{id}?include=template,tools,skills,recentMemories | 5 GET calls |
GET /project/{id}?include=milestones,team,recentTasks,conversations | 5 GET calls |
GET /conversation/{id}?include=participants,messages,task | 3 GET calls |
Authentik Integration (ProdAuthenticationCustomizer Pattern)
Copy the exact pattern from me-health-portal. Authentik sends JWTs with custom claims that qqq uses for authentication and authorization.
JWT Claims Used by qqq
"policies"— IAM-style policy JSON (Statement array with Effect/Action/Resource)"praesidiumInstanceId"— UUID for instance scoping (maps to RecordSecurityLock)"praesidiumAllAccess"— Boolean admin bypass (default false)
ProdAuthenticationCustomizer.java Key Behaviors
- Validates JWT signature against Authentik JWKS endpoint
- Extracts
praesidiumInstanceIdand sets it as the RecordSecurityLock value - Parses IAM-style policy JSON from
policiesclaim - Maps policy Statements to qqq table permissions (read/insert/edit/delete)
- If
praesidiumAllAccess = true, bypasses all policy checks (admin only) - Rejects requests with missing/expired/invalid tokens with 401
RecordSecurityLock Configuration (instance_id)
Every root entity table gets a RecordSecurityLock on instance_id. Child tables inherit scoping via FK joins. This is the same pattern as me-health-portal's mhCompanyId.
Root Tables (Direct instance_id)
These tables have a direct instance_id UUID NOT NULL column added by qqq:
Child Tables (Join-Based Scoping)
These tables trace their FK chain to a root table's instance_id:
agent | → team_member.instance_id |
task | → project.instance_id |
message | → conversation → project.instance_id |
memory | → team_member.instance_id |
milestone | → project.instance_id |
conversation | → project.instance_id |
Test Requirements
Integration tests for every CRUD endpoint. Tests run against a real PostgreSQL instance (Testcontainers).
Mandatory Test Coverage
- Every entity table: test GET (list + by ID), POST, PATCH, DELETE
- Every type table: test GET (verify seed data loaded correctly)
- RLS enforcement: test that requests with Instance A's JWT cannot read/write Instance B's data
- Policy enforcement: test that a "Viewer" policy cannot POST/PATCH/DELETE
- FK cascade: test that deleting a parent cascades to children correctly
- Composite endpoints: test
?include=returns nested data - Validation: test required fields, unique constraints, FK integrity
Acceptance Criteria
- All 117 tables created via Liquibase migrations (idempotent, re-runnable)
- All 59 type tables seeded with data from design spec
- All 117 tables accessible via REST API (GET, POST, PATCH, DELETE where allowed)
- Authentik JWT authentication working (401 on missing/invalid token)
- IAM-style policy enforcement (403 on unauthorized actions)
- RecordSecurityLock on
instance_idenforced on all root entity tables - Composite endpoints working (
?include=resolves joins) - pgvector extension installed, IVFFlat indexes on memory.embedding and knowledge_chunk.embedding
- Polymorphism trigger enforcing team_member/agent/human_profile consistency
- OpenAPI/Swagger documentation auto-generated
- Integration tests passing for all endpoints (90%+ coverage target)
- Docker image builds and runs correctly
CLAUDE.md Rules for qqq Data Layer Agents
These rules MUST be placed in the CLAUDE.md file at the root of the praesidium2-qqq/ repository.
# praesidium2-qqq CLAUDE.md ## Project Context This is the qqq data layer for Praesidium2 -- an AI agent workforce platform. qqq is a Java framework that auto-generates REST APIs from table/ERD definitions. This service is the SINGLE SOURCE OF TRUTH for all data. No other service touches the database directly. ## Architecture Rules - All table definitions go in QqqMetaDataProvider.java - All migrations go in src/main/resources/db/changelog/ as Liquibase SQL - Follow the type table standard pattern EXACTLY (see design spec) - Every entity table that is a root table gets RecordSecurityLock on instance_id - Use the ProdAuthenticationCustomizer pattern from me-health-portal - NEVER skip writing integration tests for new endpoints ## Coding Standards - Java 21, use records where appropriate - Gradle Kotlin DSL for build configuration - Follow existing qqq conventions for MetaDataProvider setup - Use Testcontainers (PostgreSQL) for integration tests - Target 90%+ test coverage ## Domain 1 Additions (v1.8 -- 4 new tables) - llm_provider: LLM provider catalog (name, api_base_url, is_active, metadata) - llm_model: Specific model config (provider_id FK, context_window, cost_per_1m_tokens) - secret_ref: Scoped secret references (instance_id, secret_key, aws_secret_arn) - audit_access_log: HIPAA-mandated access tracking (7-year retention, partitioned) - Total tables: 117 (58 entity + 59 type) ## Domain 6 Addition (v1.9 -- 1 new table) - scheduler_heartbeat: Celery Beat HA leader tracking (node_id, last_heartbeat, is_leader, metadata) ## Domain 6 Addition (v1.10 -- new columns on job_execution) - debug_payload JSONB: stores exact system prompt + tool definitions sent to LLM - trace_id VARCHAR(64): OpenTelemetry trace ID for all LLM/tool calls in this execution - Liquibase changeset: 236-job-execution-debug-columns.sql ## Domain 5 Addition (v1.10 -- new columns on memory) - recency_weight DECIMAL DEFAULT 1.0: time-decay factor for memory recall ranking - importance DECIMAL DEFAULT 0.5: relevance score (0.0-1.0) for memory prioritization - Ships in V1 even though full memory pipeline is V1.5 - Liquibase changeset: 237-memory-ranking-columns.sql ## Domain 7: Business Operations (10 tables) - task_approval_queue, approval_policy, autonomy_level - kpi_baseline, kpi_measurement, team_roi_report - agent_action_log, rollback_record - output_feedback, agent_quality_score - These tables support approval workflows, KPI measurement, rollback/recovery, feedback, and agent performance management. - All follow standard entity table patterns with RecordSecurityLock on instance_id. ## What NOT to Do - Do NOT create custom REST controllers -- qqq generates all endpoints - Do NOT add business logic to this service -- it is a data layer only - Do NOT bypass RecordSecurityLock for any reason - Do NOT modify type table schemas -- they follow the standard pattern - Do NOT hardcode any instance IDs or tenant values
Agent Gateway
Epic 2 — Phase 4The bridge between the browser and the AI agent runtime.
The Agent Gateway is a Python/FastAPI service that handles all AI-related HTTP traffic: chat SSE streams, task dispatch, approval workflow, and WebSocket broadcasts. It reads/writes all persistent data via the qqq REST API — it NEVER touches the database directly. Inline 1:1 chat runs in-process (no Celery round-trip); everything else is dispatched to Agent Workers via Redis task queues. Note: meeting orchestration and tool forge session endpoints are retained for V1.5.
Repository Setup
| Repository | praesidium2-python/services/agent-gateway/ |
| Language | Python 3.12 |
| Build Tool | pyproject.toml (hatchling) |
| Framework | FastAPI + Uvicorn |
| Task Queue | Celery (producer only) via Redis |
| Auth | Authentik JWT (same token as qqq, forwarded) |
| Replicas | 2-8 pods, HPA on WS connections + req/s |
| Resources | 1 CPU / 2Gi per pod |
Day-by-Day Tasks (Gateway Track)
The gateway is built incrementally across Phase 1 (Days 1-5) by Backend Track, then enhanced in Phase 2 (Days 6-10).
| Day | Task | Exit Criteria |
|---|---|---|
| D1 | Python project scaffold — pyproject.toml, FastAPI app factory, Uvicorn entry, Dockerfile, linting (ruff), formatting (black), pytest config |
uvicorn gateway.main:app starts, /healthz returns 200 |
| D2 | qqq Python API client (auto-generated from OpenAPI or hand-typed with httpx). Agent hydration service skeleton. Redis connection. Pydantic settings. | Can fetch an agent record from qqq API. Redis ping succeeds. |
| D3 | LLMClient abstraction (Protocol + Anthropic implementation; V1 is Anthropic-only). Agent hydration full pipeline (template + tools + memories + system prompt build). Redis cache (5min TTL). | Can hydrate an agent with correct system prompt. LLM client can send a message. |
| D4 | POST /api/chat/{agent_id} with SSE streaming. Inline chat executor with Anthropic tool loop. Platform tool resolution. Conversation + message persistence via qqq API. |
Can chat with an agent in browser, see SSE tokens stream, tool calls execute and stream back. |
| D5 | Celery task dispatcher. POST /api/task/{task_id}/execute. POST /api/meeting/{meeting_id}/start. WebSocket server with Redis pub/sub broadcast. GET /api/stream/{conversation_id} SSE reconnect. |
Task dispatch creates Celery task. Meeting dispatch creates Celery task. WS broadcasts work across pods. |
| D6-7 | (Phase 2) POST /api/forge/session. POST /api/job/{job_id}/trigger. GET /api/agent/{agent_id}/status. Webhook receiver. Auth middleware with Authentik JWT validation. |
All 8+ endpoints functional with JWT auth. Forge session creates LangGraph workflow. |
| D8-10 | (Phase 2) Prometheus metrics (6 custom gauges/counters). Correlation ID middleware. Rate limiter. Integration health checks. Error handling + structured logging. | Metrics at /metrics. Correlation IDs propagated to qqq and Celery. Rate limits enforced. |
API Endpoints
Every route requires a valid Authentik JWT in the Authorization: Bearer header. The JWT's praesidiumInstanceId claim is forwarded to all qqq API calls for row-level security scoping.
| Method | Path | Description | Response |
|---|---|---|---|
| POST | /api/chat/{agent_id} |
Start or continue a 1:1 chat. Runs INLINE in the gateway (no Celery). Returns SSE stream of tokens + tool calls. | text/event-stream (SSE) |
| POST | /api/task/{task_id}/execute |
Dispatch task execution to a Celery worker. Creates conversation, queues execute_task on standard queue. |
{ conversation_id, celery_task_id } |
| POST | /api/meeting/{meeting_id}/start |
Trigger a scheduled meeting (standup/huddle). Queues run_meeting on standard queue. |
{ conversation_id, celery_task_id } |
| POST | /api/forge/session |
Start an interactive tool forge session. Queues forge_session on long queue. Returns SSE stream for live updates. |
{ conversation_id, celery_task_id } |
| POST | /api/job/{job_id}/trigger |
Manually trigger a scheduled job. Queues execute_job on standard or long queue based on job config. |
{ job_execution_id, celery_task_id } |
| GET | /api/stream/{conversation_id} |
SSE stream for an active conversation. Reconnects with Last-Event-ID header. Used by worker-executed tasks to push tokens to the browser. |
text/event-stream (SSE) |
| GET | /api/agent/{agent_id}/status |
Real-time status of an agent: execution state, active conversation, queue position. | { status, active_conversation_id, queue_position } |
| WS | /ws |
WebSocket for broadcast events. Gateway subscribes to Redis pub/sub channels and pushes events to all connected browsers. Events: task updates, meeting progress, agent status changes, tool published, escalations. | JSON frames per event type |
| POST | /api/webhooks/{integration_type} |
Inbound webhook receiver for Jira, Slack, GitHub, etc. Verifies signature, parses event, routes to relevant agent/task/project. | 202 Accepted |
| GET | /api/internal/worker/hydrate/{agent_id} |
Agent manifest delivery. Returns the complete AGENT_MANIFEST JSON (agent persona, LLM config, secret ARNs, tools). Auth: Service Account JWT scoped to instance. See AGENT_RUNTIME_SPEC. |
{ agent, config, secrets[], tools[] } |
| POST | /api/task/{id}/approve |
HITL approval signal. Validates IAM permissions, verifies action_payload_hash (tamper-proof), updates task_approval status, dispatches Redis resume event with thread_id. See HITL_ORCHESTRATION_SPEC. |
{ approved: true, resumed_thread_id } |
| POST | /api/internal/debug/replay/{execution_id} |
Agent debug replay. Loads debug_payload from job_execution and re-runs through qqq Process for step-through debugging. Auth: Service Account JWT. v1.10 |
{ replay_id, trace_id, steps[] } |
| GET | /api/search |
Command palette backend. Accepts ?q= query string and returns ranked results across projects, agents, tasks, conversations, and tools using pgvector semantic search. |
{ results: [{ type, id, title, score }] } |
| GET | /healthz |
Liveness probe. Returns 200 if the process is running. | { status: "ok" } |
| GET | /readyz |
Readiness probe. Checks Redis + qqq API connectivity. | { status: "ready", redis: true, qqq: true } |
| GET | /metrics |
Prometheus-format metrics endpoint. | Prometheus text format |
SSE Event Types (chat + stream endpoints)
| Event | Data Shape | Description |
|---|---|---|
token | { text: string } | Incremental text token from the LLM |
tool_call_start | { tool_name, tool_id, input } | Agent is invoking a tool |
tool_call_result | { tool_name, result, duration_ms } | Tool execution completed |
thinking | { text: string } | Extended thinking content (if model supports it) |
message_complete | { message_id, tokens_used, cost_usd } | Full assistant message stored |
interrupt | { approval_id, action_type, action_summary, risk_level, agent_id, task_id } | LangGraph interrupt() fired — frontend transitions from “Thinking” to “Approval Required” v1.9 |
error | { code, message, retry_after? } | Error during processing |
done | {} | Stream ended normally |
WebSocket Event Types
| Channel | Event | Description |
|---|---|---|
task:{id} | status_changed | Task moved to new status (QUEUED, IN_PROGRESS, DONE, etc.) |
agent:{id} | execution_changed | Agent execution state changed (IDLE, EXECUTING, ERROR) |
meeting:{id} | message_added | New message in a meeting conversation |
meeting:{id} | meeting_completed | Meeting finished, summary + action items available |
forge:{id} | state_changed | Forge workflow moved to new state (DESIGN, SCAFFOLD, etc.) |
tool:published | tool_published | New tool or tool version published to catalog |
escalation:new | escalation_created | New escalation requiring human attention |
Chat Request/Response Shapes
POST /api/chat/{agent_id} — Request
{
"message": "Review PR #42 and summarize the changes",
"conversation_id": "uuid | null", // null = new conversation
"project_id": "uuid | null", // project context
"task_id": "uuid | null", // task context
"attachments": [ // optional
{ "type": "file", "url": "s3://...", "name": "report.pdf" }
]
}
POST /api/task/{task_id}/execute — Request
{
"instructions": "Additional context...", // optional override
"collaborator_ids": ["uuid", "uuid"], // optional extra agents
"priority": "HIGH", // queue priority hint
"max_budget_usd": 5.00 // cost guard
}
POST /api/meeting/{meeting_id}/start — Request
{
"agenda_override": "Focus on blockers", // optional
"additional_participants": ["uuid"] // optional
}
Agent Hydration Service
On every interaction, the agent is hydrated fresh from qqq APIs. A Redis cache (5min TTL) avoids repeated round-trips for back-to-back requests. Hydration produces a HydratedAgent with everything the LLM needs.
Hydration Pipeline (6 Steps)
| Step | Source | What It Loads |
|---|---|---|
| 1 | GET /agent/{id}?include=template,skills |
Agent record + agent_template (role, backstory, default_model, system_prompt) + skills list |
| 2 | GET /agent_tool_assignment?agentId={id} |
Tool assignments → resolve each to a callable tool instance (PLATFORM, INTEGRATION, or CUSTOM dispatch stub) |
| 3 | GET /memory?agentId={id}&orderBy=-importance&limit=50 |
Top 50 memories by importance score, plus semantic search with current context query |
| 4 | GET /llm_model/{model_id} + GET /llm_provider/{provider_id} |
Resolve model (agent override > template default). Build LLMClient via factory. Check provider health, fall back if DOWN. |
| 5 | In-memory | Build system prompt: template.system_prompt + agent.soul + skills context (v1.12: ordered by assigned_at ASC, 40/30/20/10 context budget, tool requirement validation) + memory injection + system awareness block + prompt injection defense preamble |
| 6 | Redis SET (5min TTL) | Cache the HydratedAgent as JSON. Cache key: agent:{id}:hydrated. Invalidated on agent config change via WS event. |
Composite qqq Endpoints Reduce Round-Trips
Step 1 uses qqq's ?include= mechanism to fetch agent + template + skills in a single HTTP call instead of 3. The hydration pipeline makes ~4 HTTP calls total (agent composite, tools, memories, model) instead of ~10 without includes.
Inline Chat Executor
1:1 chat runs directly in the gateway process — no Celery worker needed. This eliminates the queue round-trip for the most latency-sensitive interaction. The executor implements the Anthropic tool-use loop with SSE streaming.
Execution Flow
Browser POST /api/chat/{agent_id}
|
+-- 1. Validate JWT, extract instance_id
+-- 2. Hydrate agent (cache hit or fresh from qqq)
+-- 3. Load/create conversation via qqq API
+-- 4. Persist user message: POST /message
+-- 5. Build messages array (system + history + user)
|
+-- 6. TOOL LOOP (streaming):
| a. Call LLMClient.create_message(stream=True)
| b. Stream tokens to browser via SSE
| c. If tool_use block received:
| - Validate tool input against JSON Schema
| - Check risk level (LOW/MEDIUM = execute, HIGH = log+notify, CRITICAL = await approval)
| - Check per-agent per-tool rate limit (Redis token bucket)
| - Execute tool (in-process for PLATFORM/INTEGRATION, HTTP for CUSTOM)
| - Stream tool_call_start + tool_call_result SSE events
| - Append tool_result to messages, loop back to (a)
| d. If end_turn: break loop
|
+-- 7. Persist assistant message: POST /message (with tokens, cost, tool_calls JSONB)
+-- 8. Update agent stats: PATCH /agent (last_active, tokens, cost)
+-- 9. Publish "agent:status_changed" to Redis pub/sub
+-- 10. Queue "extract_memories" on quick queue (background)
+-- 11. Send SSE "message_complete" + "done" events
Tool Loop Safety
The tool loop has a hard limit of 25 iterations per turn. If the agent keeps calling tools beyond this limit, the loop breaks and the agent is forced to respond with what it has. This prevents runaway tool calling from prompt injection or agent confusion.
Task Dispatcher
For operations that take >30 seconds or require multi-agent coordination, the gateway dispatches work to Celery workers via Redis. The dispatcher determines the correct queue based on task type.
| Gateway Endpoint | Celery Task | Queue | Pre-Dispatch Actions |
|---|---|---|---|
POST /api/task/{id}/execute |
execute_task |
standard |
Create conversation, PATCH task status → QUEUED, validate assignee, check budget |
POST /api/meeting/{id}/start |
run_meeting |
standard |
Create conversation, add participants, PATCH meeting status → IN_PROGRESS |
POST /api/forge/session |
forge_session |
long |
Create conversation, create tool record (draft), initialize LangGraph thread |
POST /api/job/{id}/trigger |
execute_job |
standard or long |
Create job_execution (RUNNING), validate job config, check consecutive_failures < max_retries |
Queue Routing Logic
The dispatcher routes to long queue when: (a) task type is forge_session, (b) task has is_complex=true flag, or (c) scheduled job config has expected_duration > 30min. All other work goes to standard. Background extraction tasks (extract_memories, index_document) go to quick.
LLMClient Abstraction
A Python Protocol interface that all LLM provider implementations conform to. The gateway and workers both use this abstraction — it lives in a shared package.
class LLMClient(Protocol):
"""Common interface all LLM providers implement."""
async def create_message(
self,
messages: list[dict],
system: str,
tools: list[dict] | None = None,
max_tokens: int = 4096,
stream: bool = False,
**kwargs
) -> LLMResponse: ...
async def health_check(self) -> bool: ...
def supports_feature(self, feature: str) -> bool: ...
# features: "tools", "vision", "thinking", "streaming", "json_mode"
Provider Implementations
V1: Anthropic Only
V1 implements AnthropicClient only. The LLMClient Protocol and llm_provider/llm_model schema remain multi-provider-ready, but only one client implementation ships in V1. Additional providers (OpenAI, Bedrock, Vertex, Ollama, vLLM) are post-V1.
| Class | Provider Type | SDK / Transport | Key Notes |
|---|---|---|---|
AnthropicClient | CLOUD_API | anthropic SDK | V1 provider. Supports tools, vision, thinking, streaming. |
OpenAIClient | CLOUD_API | openai SDK | Post-V1. GPT-4o, o1. Supports tools, vision, streaming. |
OllamaClient | LOCAL | HTTP to Ollama API | Needed for local dev (Ollama pod in k8s). ~100 lines. See docs/superpowers/specs/2026-04-16-local-development-design.md. |
BedrockClient | BEDROCK | boto3 bedrock-runtime | Post-V1. Claude via AWS. IAM role auth. Supports tools, streaming. |
VertexClient | VERTEX | google-cloud-aiplatform | Post-V1. Gemini + Claude via GCP. Service account auth. |
VLLMClient | SELF_HOSTED | HTTP (OpenAI-compatible) | Post-V1. Self-hosted models on EKS. No auth or bearer token. |
Model Resolution at Hydration Time
The correct LLMClient is built during agent hydration by build_llm_client(provider, model). The factory reads llm_provider.provider_type to select the class, then configures it with the provider's api_endpoint, model's model_identifier, and credentials from Secrets Manager. If the provider is DOWN, the factory falls back to the agent's fallback_model_id.
Dependencies (pyproject.toml)
Runtime
fastapi | Web framework |
uvicorn[standard] | ASGI server (with uvloop) |
httpx | Async HTTP client for qqq API + LLM fallback |
anthropic | Anthropic SDK (primary LLM) |
openai | OpenAI SDK (Post-V1: embeddings) |
celery[redis] | Task queue producer (dispatch only) |
redis[hiredis] | Redis client (cache, pub/sub, rate limit) |
pydantic | Request/response validation |
pydantic-settings | Env-driven configuration |
pyjwt[crypto] | Authentik JWT validation |
prometheus-client | Metrics exposition |
structlog | Structured JSON logging |
boto3 | AWS Secrets Manager + Bedrock |
Dev / Test
pytest | Test runner |
pytest-asyncio | Async test support |
pytest-cov | Coverage reporting |
httpx | ASGI test client (TestClient) |
respx | Mock httpx requests (qqq API mocking) |
fakeredis[lua] | In-memory Redis for tests |
ruff | Linting |
black | Formatting |
mypy | Type checking |
Dockerfile (Multi-Stage Build)
# Stage 1: Build
FROM python:3.12-slim AS builder
WORKDIR /app
COPY pyproject.toml .
RUN pip install --no-cache-dir hatchling \
&& pip wheel --no-cache-dir --wheel-dir /wheels -e .
# Stage 2: Runtime
FROM python:3.12-slim
WORKDIR /app
RUN groupadd -r gateway && useradd -r -g gateway gateway
COPY --from=builder /wheels /wheels
RUN pip install --no-cache-dir /wheels/*.whl && rm -rf /wheels
COPY src/ ./src/
USER gateway
EXPOSE 8000
HEALTHCHECK CMD python -c "import httpx; httpx.get('http://localhost:8000/healthz').raise_for_status()"
CMD ["uvicorn", "gateway.main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "4"]
Prometheus Metrics
Exposed at /metrics in Prometheus text format. Scraped by Prometheus every 15s.
| Metric Name | Type | Labels | Description |
|---|---|---|---|
praesidium_http_requests_total | Counter | method, path, status | Total HTTP requests by method, path, and status code |
praesidium_http_request_duration_seconds | Histogram | method, path | Request latency distribution |
praesidium_active_websocket_connections | Gauge | — | Current open WebSocket connections |
praesidium_active_sse_streams | Gauge | — | Current open SSE chat streams |
praesidium_work_items_queued | Gauge | priority | Pending work items by priority level |
praesidium_work_items_processing | Gauge | — | Currently executing work items |
Test Requirements
Tests use pytest + httpx ASGI TestClient. All external dependencies are mocked: qqq API (respx), Redis (fakeredis), Anthropic (mock responses). No real API calls in CI.
Mandatory Test Coverage
- Every router endpoint: test 200 success, 401 unauthorized, 403 forbidden, 422 validation error
- Chat endpoint: test SSE stream format, tool loop (1 tool call, multi-tool, max iteration break), token/cost tracking
- Task dispatcher: test correct queue routing (quick/standard/long), pre-dispatch qqq API calls, Celery task creation
- Agent hydration: test cache hit, cache miss, cache invalidation, provider fallback on DOWN
- WebSocket: test connection, subscription, broadcast, disconnection cleanup
- Auth middleware: test valid JWT, expired JWT, missing JWT, invalid signature, instance_id extraction
- LLMClient: test each provider implementation with mock responses (streaming + non-streaming)
- Rate limiter: test per-agent per-tool limits, bucket refill, 429 response
- SSE reconnect: test Last-Event-ID replay of missed events
- Error handling: test qqq API timeout/error graceful degradation, Anthropic rate limit retry + backoff
Acceptance Criteria
- All 8+ API endpoints return correct responses with valid Authentik JWT
- 1:1 chat streams tokens via SSE in real-time with <200ms time-to-first-token (after hydration)
- Tool loop executes PLATFORM tools in-process and streams tool_call_start/tool_call_result SSE events
- Agent hydration loads template + tools + memories in ≤500ms (cache miss) and ≤50ms (cache hit)
- Task dispatch correctly routes to quick/standard/long Celery queues based on task type
- WebSocket broadcasts reach all connected clients within 100ms of Redis pub/sub event
- SSE stream supports Last-Event-ID reconnection without message loss
- JWT authentication rejects missing/expired/invalid tokens with 401
- Instance scoping (praesidiumInstanceId) is forwarded to every qqq API call
- Per-agent per-tool rate limits enforced (configurable, default 10 calls/min/tool)
- Prometheus metrics exposed at /metrics with all 6 custom metrics
- Structured JSON logging with correlation IDs propagated to qqq + Celery
- Docker image builds and runs correctly (multi-stage, non-root user)
- All tests passing with 90%+ code coverage
CLAUDE.md Rules for Agent Gateway Agents
These rules MUST be placed in the CLAUDE.md file at the root of the praesidium2-python/services/agent-gateway/ directory.
# praesidium2-gateway CLAUDE.md
## Project Context
This is the Agent Gateway for Praesidium2 -- the bridge between the browser
and the AI agent runtime. It handles all AI-related HTTP traffic: chat SSE
streams, task dispatch, approval workflow, feedback, rollback/recovery,
admin controls, and WebSocket broadcasts.
V1.5: meeting orchestration, tool forge sessions.
## Architecture Rules
- This service NEVER touches the database directly -- all data via qqq REST API
- 1:1 chat runs INLINE in the gateway (no Celery round-trip) for low latency
- All other work (tasks, jobs) is DISPATCHED to Celery workers
- Meetings and forge sessions are V1.5 -- do not implement in V1
- Every endpoint requires Authentik JWT auth -- no unauthenticated routes except /healthz
- Forward praesidiumInstanceId from JWT to every qqq API call for RLS scoping
- Agent hydration results are cached in Redis (5min TTL) -- invalidate on config change
- SSE streams must support Last-Event-ID reconnection
- WebSocket broadcasts use Redis pub/sub for cross-pod delivery
## Coding Standards
- Python 3.12, async/await everywhere, type hints on all functions
- Pydantic models for ALL request/response shapes -- no raw dicts at API boundaries
- structlog for structured JSON logging -- no print() statements
- All errors return proper HTTP status codes with JSON error bodies
- Follow FastAPI dependency injection patterns (Depends())
- Use httpx.AsyncClient for all outbound HTTP (qqq API, LLM providers)
## v1.8 New Endpoints
- GET /api/internal/worker/hydrate/{agent_id} -- AGENT_MANIFEST delivery (Service Account JWT)
- POST /api/task/{id}/approve -- HITL approval signal (action hash verification + Redis resume)
- GET /api/search -- Command Palette backend (pgvector semantic search)
## Business Operations Endpoints (V1)
- Approval: GET /api/approvals, POST /api/approvals/{id}/approve|reject|request-changes
- Feedback: POST /api/feedback (submit), GET /api/feedback/agent/{id} (per-agent)
- Rollback: POST /api/actions/{id}/undo, POST /api/tasks/{id}/rollback
- Admin: POST /api/admin/emergency-stop, POST /api/admin/resume,
GET /api/admin/kill-criteria, POST /api/admin/kill-criteria/evaluate
## LLM Client Rules
- All LLM calls go through the LLMClient Protocol -- never call SDKs directly
- build_llm_client() factory is the ONLY place that instantiates provider clients
- Provider health check failures must trigger fallback to secondary model
- Every LLM call must track input_tokens, output_tokens, cost_usd
## Testing Rules
- Mock ALL external dependencies: qqq API (respx), Redis (fakeredis), LLM SDKs
- No real API calls in tests -- ever
- Every endpoint needs tests for 200, 401, 403, 422
- Test SSE stream content by consuming the async generator
- Target 90%+ test coverage
## What NOT to Do
- Do NOT add database connections or ORM models -- this is not a data layer
- Do NOT process long-running tasks in the gateway -- dispatch to Celery workers
- Do NOT bypass JWT authentication for any route except health checks
- Do NOT hardcode API keys, model names, or URLs -- use Pydantic Settings from env
- Do NOT use synchronous blocking calls -- everything must be async
- Do NOT skip input validation -- use Pydantic models with JSON Schema constraints
- Do NOT implement meetings or forge endpoints -- those are V1.5
Shared Python Packages
The Gateway and Worker both need the llm/ client and hydration/ modules. These live in a shared/ directory in the monorepo and are installed as editable packages in Docker builds.
Monorepo Structure
Docker Build Integration
# In Gateway and Worker Dockerfiles:
COPY shared/ /app/shared/
RUN pip install -e /app/shared/llm \
&& pip install -e /app/shared/hydration
# Both services import identically:
from praesidium_llm import build_llm_client
from praesidium_hydration import hydrate_agent
Post-v1 Optimization
After v1, shared packages can be published to a private PyPI registry for versioned releases. During v1, editable installs from the monorepo shared/ directory keep iteration fast.
Agent Workers
Epic 4 — Phase 3The AI execution engine — where agents actually do the work.
Agent Workers are Python/Celery processes that consume tasks from Redis queues and execute them using direct Anthropic SDK agent loops (standard task execution) and LangGraph (durable state machine workflows needing crash recovery or HITL approval gates). V1 scope: task execution, scheduled jobs, HITL interrupt/resume protocol, approval queue processing, and integration tool dispatch. LangGraph provides PostgreSQL-backed checkpointing (langgraph_checkpoints table) so long-running workflows survive worker crashes and multi-hour HITL waits. The interrupt() function serializes graph state before requesting human approval; the Resume Protocol re-hydrates from the checkpoint when approval arrives via Redis event. All spans carry ddtrace trace IDs for unified Datadog observability. V1.5 additions: meetings, tool forge builds, memory extraction, and document indexing (specs retained below).
Repository Setup
| Repository | praesidium2-python/services/agent-worker/ |
| Language | Python 3.12 |
| Build Tool | pyproject.toml (hatchling) |
| Framework | Celery 5.x + Anthropic SDK + LangGraph (durable/HITL) |
| Broker | Redis (consumer + producer) |
| Checkpoints | PostgreSQL (LangGraph PostgresSaver) |
| Replicas | 3-10 pods, HPA on Redis queue depth |
| Resources | 2 CPU / 4Gi per pod |
| Concurrency | 4 worker threads per pod |
Day-by-Day Tasks (Worker Track)
Worker development starts on Day 5 (Phase 1) after the gateway can dispatch. Core execution patterns are built in Phase 2, with advanced features (forge, multi-LLM, memory) in Phases 2-3.
| Day | Task | Exit Criteria |
|---|---|---|
| D5 | Celery app factory, Redis broker config, 3 queues (quick/standard/long), execute_task skeleton. Platform tool stubs. Dockerfile. |
Worker starts, consumes from Redis, executes a no-op task, acknowledges. |
| D6 | Agent execution: agent_builder.py (HydratedAgent → Anthropic SDK context), loop_runner.py (agentic loop). execute_task with direct Anthropic SDK execution. |
Task execution with direct SDK agent loop produces output. Messages streamed via Redis pub/sub. |
| D7 | LangGraph setup: PostgresSaver, job_graph.py (load → execute → evaluate). execute_job task. Celery Beat scheduler with leader election. V1.5 run_meeting multi-agent standup skeleton (deferred). |
Scheduled jobs execute on cron. Meeting orchestration deferred to V1.5 (V1 uses Slack-based status). |
| D8 | V1.5 Memory extractor: Claude-powered analysis, embedding generation (text-embedding-3-small), deduplication (cosine similarity > 0.92), store via qqq API. extract_memories task. index_document task (chunking + embedding). |
Deferred to V1.5. After task/meeting completion, memories extracted and stored. Documents chunked and embedded. |
| D9 | V1.5 Knowledge document upload + chunking + embedding pipeline. Memory recall during agent hydration (semantic search + recency bias). Tool execution framework: PLATFORM + INTEGRATION in-process, risk-level safety gates. | Memory/RAG deferred to V1.5. Tool execution framework and risk-level safety gates are V1 scope. |
| D10 | Integration tools: Jira, Slack, GitHub connectors as in-process tools (V1 scope). Secret resolution (scope hierarchy: AGENT > TOOL > PROJECT > INSTANCE, narrowest scope wins). Confluence, Google, Zoho deferred to V1.1. | 3 core integration tool sets working (Slack, Jira, GitHub). Secrets resolved from correct scope. V1.1 integrations stubbed. |
| D11-12 | V1.5 LangGraph tool forge workflow (forge_graph.py): 6 states (DESIGN → SCAFFOLD → IMPLEMENT → TEST → REVIEW → PUBLISH). Human-in-the-loop interrupt at REVIEW. Git integration for tool repos. |
Deferred to V1.5. Full forge workflow produces a tool version from human description, with human review gate. |
| D13-14 | (Phase 3) Multi-LLM: provider health checks, model fallback. Stats reconciliation job. Checkpoint pruning (90-day retention). Celery Beat health checks + memory decay. | Provider failover works. Stats reconciled every 6h. Old checkpoints pruned. System jobs running. |
Celery Configuration
Three task-type-specific queues with tuned visibility timeouts to prevent duplicate execution of long-running tasks.
| Queue | Visibility Timeout | Task Types | Acks Late | Rationale |
|---|---|---|---|---|
quick |
300s (5 min) | extract_memories, index_document, reconcile_stats, chat-related background work |
No | Fast tasks; if they fail, just retry. No checkpointing needed. |
standard |
3600s (1 hour) | execute_task, run_meeting, execute_job (simple) |
Yes | Medium-duration tasks. acks_late ensures re-queue on worker crash. |
long |
7200s (2 hours) | forge_session, complex multi-step LangGraph workflows, project planning |
Yes | Long-running workflows with checkpointing. Resumes from last checkpoint on re-delivery. |
Celery App Configuration
# worker/app.py
from celery import Celery
app = Celery("praesidium-worker")
app.conf.update(
broker_url="redis://redis:6379/0",
result_backend="redis://redis:6379/1",
# Queue definitions
task_queues={
"quick": {"exchange": "quick", "routing_key": "quick"},
"standard": {"exchange": "standard", "routing_key": "standard"},
"long": {"exchange": "long", "routing_key": "long"},
},
task_default_queue="standard",
# Visibility timeouts per queue
broker_transport_options={
"queue_order_strategy": "priority",
"visibility_timeout": {
"quick": 300,
"standard": 3600,
"long": 7200,
},
},
# Worker behavior
worker_concurrency=4,
worker_prefetch_multiplier=1, # Fetch one task at a time (for acks_late)
task_acks_late=False, # Default; overridden per task
# Serialization
task_serializer="json",
result_serializer="json",
accept_content=["json"],
# Time limits
task_soft_time_limit=3600, # Default soft limit (1 hour)
task_time_limit=7500, # Default hard limit (2h5m)
)
Idempotency Guards
LangGraph tasks check for existing checkpoints before re-executing a node. If a task is re-delivered due to a visibility timeout edge case, it resumes from the last checkpoint rather than restarting. Direct SDK agent loop tasks (no checkpointing) will restart from scratch on re-delivery — they are designed to be idempotent (creating duplicate records is prevented by qqq's unique constraints).
Celery Task Definitions
Every task follows the same lifecycle: hydrate agent(s) → execute work → persist results to qqq API → broadcast status via Redis pub/sub → queue post-processing (memory extraction).
| Task | Queue | Framework | Description |
|---|---|---|---|
execute_task |
standard |
Anthropic SDK | Call Anthropic SDK with system prompt, tools, and conversation context for the assigned agent + optional collaborators. Run an agentic tool loop until completion. Persist results (task_comment, messages, status changes) to qqq API. |
run_meeting |
standard |
Anthropic SDK | Multi-agent standup/huddle. Load all conversation_participants and build SDK context for each. Turn-taking orchestrator runs the agenda (status updates, blockers, action items). Produces meeting summary and creates task records for action items. |
execute_job |
standard or long |
LangGraph | Scheduled job execution via LangGraph state graph (LOAD → EXECUTE → EVALUATE → REPORT). Checkpointed at every node. Updates job_execution record on completion/failure. |
forge_session |
long |
LangGraph | Interactive tool forge workflow (6 states: DESIGN → SCAFFOLD → IMPLEMENT → TEST → REVIEW → PUBLISH). Human-in-the-loop interrupt at REVIEW state. Commits code to tool_repo git. |
extract_memories |
quick |
Direct LLM | Post-conversation memory extraction using Claude. Analyzes the conversation for facts, decisions, lessons learned. Generates embeddings and deduplicates against existing memories. |
index_document |
quick |
Direct | Knowledge document chunking (1000 token chunks, 200 token overlap) + embedding generation (text-embedding-3-small). Stores knowledge_chunk records via qqq API. |
reconcile_stats |
quick |
Direct | Recompute denormalized counters on agent records (total_tasks_completed, total_tokens_used, total_cost_usd) from source tables. Runs every 6 hours via Celery Beat. |
execute_task — Direct SDK Agent Loop
The primary execution pattern for task work. Calls the Anthropic SDK with the hydrated agent context and runs an agentic tool loop until completion.
@app.task(bind=True, queue="standard", acks_late=True)
def execute_task(self, task_id: str, conversation_id: str, **kwargs):
"""Execute a task using direct Anthropic SDK agent loop."""
# 1. Load task from qqq API
task = qqq.get(f"/task/{task_id}?include=project,milestone")
qqq.patch(f"/task/{task_id}", {"status_id": STATUS_IN_PROGRESS})
# 2. Hydrate assigned agent (+ collaborators if any)
agent = hydrate_agent(task.assignee_id)
agent_context = build_agent_context(agent)
collaborators = [
build_agent_context(hydrate_agent(cid))
for cid in kwargs.get("collaborator_ids", [])
]
# 3. Build messages from task context
messages = build_task_messages(task, agent)
# 4. Run agentic loop (prompt → tool calls → result)
result = run_agent_loop(
client=llm_client,
system=agent_context.system_prompt,
messages=messages,
tools=agent_context.tools,
step_callback=stream_to_redis(conversation_id),
)
# 5. Persist results
qqq.post("/task_comment", {
"task_id": task_id,
"content": result.output,
"author_id": agent.team_member_id,
})
qqq.patch(f"/task/{task_id}", {"status_id": STATUS_DONE})
# 6. Broadcast completion + queue memory extraction
redis.publish(f"task:{task_id}", {"event": "status_changed", "status": "DONE"})
extract_memories.delay(conversation_id=conversation_id, agent_id=agent.id)
run_meeting — Turn-Taking Orchestrator
Meetings use a turn-taking orchestrator with a designated facilitator. Each participant contributes based on their role and current work context via sequential Anthropic SDK calls.
@app.task(bind=True, queue="standard", acks_late=True)
def run_meeting(self, meeting_id: str, conversation_id: str, **kwargs):
"""Run a multi-agent meeting (standup, huddle, retro)."""
# 1. Load meeting + participants
meeting = qqq.get(f"/scheduled_meeting/{meeting_id}")
participants = qqq.get(f"/conversation_participant?conversationId={conversation_id}")
# 2. Hydrate all participating agents
agent_contexts = []
for p in participants:
hydrated = hydrate_agent(p.team_member_id)
ctx = build_agent_context(hydrated, role_override=p.role)
agent_contexts.append(ctx)
# 3. Build meeting-specific prompts
# Facilitator: run agenda, collect updates, identify blockers
# Participants: share status, flag blockers, propose actions
agenda_prompts = build_meeting_prompts(meeting.type, agent_contexts, meeting.agenda)
# 4. Run turn-taking orchestrator (each agent speaks in sequence)
result = run_turn_taking_loop(
client=llm_client,
agent_contexts=agent_contexts,
agenda_prompts=agenda_prompts,
step_callback=stream_to_redis(conversation_id),
)
# 5. Extract action items as task records
action_items = extract_action_items(result.raw)
for item in action_items:
qqq.post("/task", {
"title": item.title,
"project_id": meeting.project_id,
"assignee_id": item.assignee_id,
"status_id": STATUS_OPEN,
})
# 6. Store meeting summary
qqq.patch(f"/conversation/{conversation_id}", {
"summary": result.raw,
"status_id": CONV_STATUS_COMPLETED,
})
LangGraph Integration
LangGraph is used for any work that could take >30 seconds, needs human-in-the-loop approval, must survive worker crashes, or has branching/retry logic. State is checkpointed to PostgreSQL after every graph node.
Scheduled Job Graph (execute_job)
# langgraph/job_graph.py
class JobState(TypedDict):
job_id: str
agent_id: str
context: dict
result: str | None
status: str
def build_job_graph():
graph = StateGraph(JobState)
graph.add_node("load", load_job_context)
graph.add_node("execute", execute_with_agent)
graph.add_node("evaluate", evaluate_result)
graph.add_node("report", report_and_persist)
graph.add_edge("load", "execute")
graph.add_edge("execute", "evaluate")
graph.add_conditional_edges(
"evaluate",
should_retry,
{"retry": "execute", "done": "report"}
)
graph.set_entry_point("load")
graph.set_finish_point("report")
return graph.compile(
checkpointer=PostgresSaver(conn)
)
Tool Forge Graph (forge_session)
# langgraph/forge_graph.py
class ForgeState(TypedDict):
conversation_id: str
tool_spec: dict | None
code: str | None
test_results: dict | None
review_decision: str | None
tool_version_id: str | None
status: str
retry_count: int
def build_forge_graph():
graph = StateGraph(ForgeState)
graph.add_node("design", design_tool)
graph.add_node("scaffold", scaffold_project)
graph.add_node("implement", implement_tool)
graph.add_node("test", run_tests)
graph.add_node("review", human_review)
graph.add_node("publish", publish_tool)
graph.add_edge("design", "scaffold")
graph.add_edge("scaffold", "implement")
graph.add_edge("implement", "test")
graph.add_conditional_edges(
"test", check_test_results,
{"pass": "review", "fail": "implement"}
)
# INTERRUPT at review for human
graph.add_conditional_edges(
"review", check_review_decision,
{"approve": "publish",
"changes": "implement",
"reject": END}
)
graph.set_entry_point("design")
return graph.compile(
checkpointer=PostgresSaver(conn),
interrupt_before=["review"]
)
PostgreSQL Checkpointing
LangGraph's PostgresSaver writes checkpoint state to PostgreSQL after every graph node transition. Thread IDs map to our conversation.id. On worker crash, the task is re-delivered by Celery (acks_late=True) and LangGraph resumes from the last checkpoint. Checkpoint pruning runs every 24 hours, deleting checkpoints older than 90 days after workflow completion.
HITL Interrupt + Resume Protocol
When a CRITICAL tool call is encountered, the worker pauses execution and waits for human approval. State is fully serialized to survive multi-hour (or multi-day) waits. See HITL_ORCHESTRATION_SPEC.
Interrupt Flow (Worker → Pause)
# Inside the LangGraph node for tool execution:
def execute_tool(state: TaskState):
tool = resolve_tool(state["tool_call"])
if tool.risk_level == "CRITICAL":
# 1. Write task_approval record with SHA-256 hash
approval = qqq.post("/task_approval", {
"task_id": state["task_id"],
"agent_id": state["agent_id"],
"action_type": tool.name,
"action_payload": state["tool_call"]["input"],
"action_payload_hash": sha256(json.dumps(state["tool_call"]["input"])),
"expires_at": now() + timedelta(hours=24),
})
# 2. Update task status to AWAITING_APPROVAL
qqq.patch(f"/task/{state['task_id']}", {"status_id": STATUS_AWAITING_APPROVAL})
# 3. Notify human via Slack DM
notify_approval_needed(approval)
# 4. LangGraph interrupt() serializes state to langgraph_checkpoints
return interrupt({"approval_id": approval["id"]})
Resume Flow (Approval → Wake-Up)
# Triggered by Redis event from POST /api/task/{id}/approve:
def handle_resume_event(thread_id: str, approval_id: str):
# 1. Load approval decision
approval = qqq.get(f"/task_approval/{approval_id}")
assert approval["status"] == "APPROVED"
# 2. Re-hydrate graph state from langgraph_checkpoints
graph = build_task_graph(checkpointer=PostgresSaver(conn))
config = {"configurable": {"thread_id": thread_id}}
# 3. Resume from exact node where interrupted
result = graph.invoke(Command(resume=approval), config)
# Execution continues with the approved tool call
Tamper-Proof Security
- The UI sends a hash of the payload it is displaying when the human clicks “Approve”
- Gateway compares this hash to
action_payload_hashin the database - If hashes don’t match (agent state changed in the interim), resume is rejected with “Payload Drift Detected” error
- A “Reject” action loops the agent back to the reasoning node with the human’s feedback text
- Total end-to-end resume time target: <5 seconds (from UI click to Tool start)
Datadog Trace ID Propagation
Trace IDs must propagate through the full 5-service waterfall for unified observability. See DATADOG_INTEGRATION_SPEC.
# In gateway/celery_utils.py -- inject trace into Celery task headers:
from ddtrace import tracer
dd_headers = {
"x-datadog-trace-id": str(tracer.current_span().trace_id),
"x-datadog-parent-id": str(tracer.current_span().span_id),
}
celery_task.apply_async(args=[...], headers=dd_headers)
# In worker/tasks/base.py -- extract trace from Celery header:
span = tracer.trace("agent.reasoning_loop", service="agent-worker")
span.set_tag("agent_id", agent_id)
span.set_tag("task_id", task_id)
# ... LLM provider latency and token usage captured as span metadata
# In tool runner Job spec -- pass as env vars:
env:
- name: DD_TRACE_ID
value: "${TRACE_ID}"
- name: DD_PARENT_ID
value: "${PARENT_SPAN_ID}"
Tool Registry
The tool registry loads, resolves, and dispatches tools based on their category. PLATFORM and INTEGRATION tools run in-process; CUSTOM tools are dispatched to the tool-runner service.
| Category | Execution | Examples | Security |
|---|---|---|---|
| PLATFORM | In-process (worker) | create_task, update_task_status, list_projects, store_memory, create_escalation, schedule_job |
Trusted first-party code. Calls qqq REST API directly. |
| INTEGRATION | In-process (worker) | jira_create_issue, confluence_create_page, slack_send_message, github_create_pr, google_create_doc |
Connector classes loaded at startup. Credentials from secret_ref via Secrets Manager. |
| CUSTOM | Sandboxed k8s Job/pod | pull_campaign_metrics, check_ssl_expiry, parse_invoice_pdf |
Network-isolated. Resource limits by risk_level. Zero access to qqq API or internal services. |
| MCP | Sidecar pod or external | Playwright, SQL, filesystem, custom internal APIs | Defaults to HIGH risk. Admin must explicitly downgrade after review. |
Tool Interface Contract
class BaseTool:
name: str
description: str
input_schema: dict # JSON Schema for validation
async def execute(self, input: dict, context: ToolContext) -> ToolResult:
raise NotImplementedError
class ToolContext:
agent_id: str
project_id: str | None
conversation_id: str | None
qqq_client: QqqApiClient # typed qqq REST client
credentials: dict # from integration_config via Secrets Manager
class ToolResult:
success: bool
data: Any = None
error: str | None = None
metadata: dict = {}
Risk-Level Safety Gates
| Risk Level | Behavior | Example Tools |
|---|---|---|
| LOW | Execute immediately (read operations) | list_projects, jira_search, google_get_doc |
| MEDIUM | Execute + log to activity_log | create_task, confluence_create_page, slack_send_message |
| HIGH | Execute + log + notify human via Slack | jira_create_issue, github_create_pr, CUSTOM tools |
| CRITICAL | Create human_interaction, WAIT for approval before executing | delete_project, financial transactions, MCP tools (default) |
Celery Beat Scheduler
A single Celery Beat instance (via Redis-based leader election) polls the database for scheduled work and dispatches it as Celery tasks.
Leader Election + Scheduler HA v1.9
Only one Beat instance should run across all worker pods. Redis SET NX EX is used for leader election with a 30-second TTL. The leader renews the lock every 10 seconds. If a leader dies, another pod acquires the lock within 30 seconds.
Scheduler HA (v1.9): The active leader writes to the scheduler_heartbeat table every 30s. Secondary nodes poll this table; if now() - last_heartbeat > 60s, the secondary promotes itself to leader. This database-backed heartbeat complements the Redis lock and prevents split-brain or dead-lock scenarios when Redis is briefly unavailable.
Scheduled Jobs
| Schedule | Action |
|---|---|
| Every 60s | Poll scheduled_job table for due jobs → dispatch execute_job |
| Every 60s | Poll scheduled_meeting table for due meetings → dispatch run_meeting |
| Every 6h | Dispatch reconcile_stats for all agents |
| Every 24h | LangGraph checkpoint pruning (delete > 90 days after completion) |
| Every 2m | LLM provider health checks (update llm_provider.health_status) |
| Every 5m | Integration health checks (update integration_config.health_status) |
| Every 30s | Scheduler heartbeat: leader writes to scheduler_heartbeat table (v1.9 HA) |
| Weekly | Memory decay: reduce importance of old memories that haven't been recalled |
Memory Extractor
Runs as a background task after every conversation. Uses Claude to analyze the conversation and extract persistent memories.
@app.task(queue="quick")
def extract_memories(conversation_id: str, agent_id: str):
"""Extract and store memories from a completed conversation."""
# 1. Load full conversation
conv = qqq.get(f"/conversation/{conversation_id}/full")
# 2. Ask Claude to extract memories
prompt = f"""Analyze this conversation and extract:
- FACTS: concrete information learned
- DECISIONS: choices made and their rationale
- LESSONS_LEARNED: what worked, what didn't
- PREFERENCES: user/team preferences discovered
- PROCEDURES: processes or workflows identified
For each memory, provide: type, content, importance (0.0-1.0)."""
memories = llm_client.create_message(
messages=[{"role": "user", "content": format_conversation(conv)}],
system=prompt,
)
# 3. For each extracted memory:
for mem in parse_memories(memories):
# Generate embedding
embedding = embed(mem.content) # text-embedding-3-small
# Deduplicate: check cosine similarity against existing memories
similar = qqq.get(f"/memory?agentId={agent_id}&vector_search={embedding}&threshold=0.92")
if similar:
# Update existing memory (merge content, bump importance)
qqq.patch(f"/memory/{similar[0].id}", {
"content": merge_content(similar[0].content, mem.content),
"importance": max(similar[0].importance, mem.importance),
"recall_count": similar[0].recall_count + 1,
})
else:
# Create new memory
qqq.post("/memory", {
"agent_id": agent_id,
"type_id": mem.type_id,
"content": mem.content,
"importance": mem.importance,
"embedding": embedding,
"source_id": MEMORY_SOURCE_CONVERSATION,
"source_conversation_id": conversation_id,
})
Dependencies (pyproject.toml)
Runtime
celery[redis] | Task queue (consumer + producer) |
redis[hiredis] | Redis client (broker, pub/sub, cache) |
anthropic | Anthropic SDK (V1 LLM + direct agent execution) |
langgraph | Durable state machine workflows (HITL/crash recovery) |
langgraph-checkpoint-postgres | PostgreSQL checkpointing |
openai | OpenAI SDK + embeddings (Post-V1) |
httpx | Async HTTP client for qqq API |
pydantic | Data validation |
pydantic-settings | Env-driven configuration |
numpy | Vector operations (cosine similarity) |
prometheus-client | Metrics exposition |
structlog | Structured JSON logging |
boto3 | AWS Secrets Manager + Bedrock |
psycopg[binary] | PostgreSQL driver (LangGraph checkpoints) |
Dev / Test
pytest | Test runner |
pytest-asyncio | Async test support |
pytest-cov | Coverage reporting |
pytest-celery | Celery test fixtures |
respx | Mock httpx requests (qqq API mocking) |
fakeredis[lua] | In-memory Redis for tests |
testcontainers[postgres] | PostgreSQL for LangGraph checkpoint tests |
ruff | Linting |
black | Formatting |
mypy | Type checking |
Dockerfile (Multi-Stage Build)
# Stage 1: Build
FROM python:3.12-slim AS builder
WORKDIR /app
COPY pyproject.toml .
RUN pip install --no-cache-dir hatchling \
&& pip wheel --no-cache-dir --wheel-dir /wheels -e .
# Stage 2: Runtime
FROM python:3.12-slim
WORKDIR /app
RUN apt-get update && apt-get install -y --no-install-recommends \
libpq5 \
&& rm -rf /var/lib/apt/lists/*
RUN groupadd -r worker && useradd -r -g worker worker
COPY --from=builder /wheels /wheels
RUN pip install --no-cache-dir /wheels/*.whl && rm -rf /wheels
COPY src/ ./src/
USER worker
# Health check via Celery inspect ping
HEALTHCHECK CMD celery -A worker.app inspect ping --timeout 5
# Default: run worker consuming all 3 queues
CMD ["celery", "-A", "worker.app", "worker", \
"--queues=quick,standard,long", \
"--concurrency=4", \
"--loglevel=info"]
Celery Beat Sidecar
Celery Beat runs as a separate container in the same pod (or a dedicated single-replica Deployment). Command: celery -A worker.app beat --loglevel=info. Leader election ensures only one Beat instance dispatches scheduled work, even if multiple Beat containers start.
Prometheus Metrics
Exposed at /metrics via a sidecar HTTP server (since Celery workers don't serve HTTP). Scraped by Prometheus every 15s.
| Metric Name | Type | Labels | Description |
|---|---|---|---|
praesidium_agent_executions_total | Counter | agent_id, status | Total agent executions by agent and result (success/failure) |
praesidium_agent_execution_duration_seconds | Histogram | task_type | Execution time distribution by task type |
praesidium_llm_calls_total | Counter | model, agent_id | Total LLM API calls by model and agent |
praesidium_llm_tokens_total | Counter | direction, model | Token usage by direction (input/output) and model |
praesidium_tool_calls_total | Counter | tool_name, risk_level | Tool invocations by tool name and risk level |
praesidium_celery_task_queue_length | Gauge | queue | Current queue depth per queue name (quick/standard/long) |
Test Requirements
Tests use pytest + pytest-celery for task testing. qqq API and LLM SDKs are mocked. LangGraph checkpoint tests use testcontainers (real PostgreSQL).
Mandatory Test Coverage
execute_task: test agent context assembly, agentic loop execution, task status transitions (QUEUED → IN_PROGRESS → DONE), result persistence, memory extraction queuedrun_meeting: test multi-agent assembly, agenda-driven execution, action item extraction, meeting summary storageexecute_job: test LangGraph state transitions, checkpoint persistence, retry logic, job_execution status updatesforge_session: test all 6 forge states, human-in-the-loop interrupt at REVIEW, git commit on PUBLISH, test failure retry loop (max 5)extract_memories: test memory extraction from mock conversation, embedding generation, deduplication (cosine > 0.92 = merge)index_document: test chunking (1000 token chunks, 200 overlap), embedding storage, knowledge_chunk record creation- Tool registry: test PLATFORM tool execution, INTEGRATION tool with mock credentials, CUSTOM dispatch to tool-runner
- Risk-level gates: test LOW (execute), MEDIUM (execute + log), HIGH (execute + notify), CRITICAL (wait for approval)
- Celery Beat: test scheduler polls correct tables, dispatches correct task types, leader election works,
scheduler_heartbeatfailover within 60s - Failure recovery: test acks_late re-queue, LangGraph checkpoint resume, Anthropic rate limit backoff
Skill-Aware Execution v1.12
When an agent executes any task (chat, scheduled job, or explicit task), the system determines which skills are relevant and injects them into the agent context. Two routing modes plus a context budget system ensure efficient use of the model’s context window.
Skill Routing Modes
| Mode | Trigger | Behavior |
|---|---|---|
| Explicit | Task specifies skill_id via task.metadata.skill_id or scheduled_job.tools JSONB |
Only that skill’s instructions are loaded. Targeted, efficient. |
| Semantic | No skill specified (default for chat and most task execution) | ALL active skills’ instructions loaded during hydration. The LLM decides which skill(s) apply based on task description and skill instructions. |
Skill Injection Order & Context Budget
| Step | Rule |
|---|---|
| 1 | Skills sorted by agent_skill.assigned_at ASC (oldest first = foundational skills first) |
| 2 | Each skill’s instructions and guardrails injected as separate labeled sections |
| 3 | If combined instructions exceed context budget, truncate starting from most recently assigned (newest first = least foundational) |
| 4 | Skill-level guardrails aggregated into a single “Guardrails” section after all instructions |
Tool Requirement Validation at Hydration
During agent hydration, for each assigned skill the system validates tool availability.
| Step | Action |
|---|---|
| 1 | Load the pinned skill_version (or current_version if not pinned) |
| 2 | Check all skill_tool_requirement records for that version |
| 3 | For each required tool (is_required = true): verify agent has access via agent_tool_assignment or agent_mcp_assignment (MCP servers expose tools dynamically) |
| 4 | If required tool missing: skill is still injected but instructions prefixed with “NOTE: Tool {tool_name} not available — adapt or escalate.” Warning logged to activity_log with action skill.unmet_requirement. |
| 5 | Optional tools (is_required = false): no validation, no warning |
Proficiency Tracking (Post-Execution)
After task completion, the worker updates proficiency records for each skill whose tools were invoked during execution.
| Field | Update Logic |
|---|---|
agent_skill_proficiency.times_used | Increment +1 |
agent_skill_proficiency.last_used_at | Set to now() |
agent_skill_proficiency.proficiency | 0.00–1.00 score, improves with successful use. Records persist across skill unassignment/reassignment. |
Conflict Resolution
There is no automatic conflict detection between skills. The agent’s soul (personality/identity) serves as the tiebreaker when skills give contradictory guidance. If systematic conflicts emerge, the Super Operator should adjust skill instructions or remove one of the conflicting skills.
Acceptance Criteria
- All 7 Celery task types execute successfully and persist results to qqq API
- Direct SDK agent loop works for single-agent (task) and turn-taking orchestrator (meeting) patterns
- LangGraph state graphs execute with PostgreSQL checkpointing; workflows survive worker restart
- Tool forge workflow completes all 6 states including human-in-the-loop review interrupt
- 3 queues (quick/standard/long) route correctly with proper visibility timeouts and acks_late behavior
- Memory extraction produces correct memory types with embeddings and deduplication
- Document indexing chunks, embeds, and stores knowledge_chunk records correctly
- Tool registry resolves all 4 categories (PLATFORM, INTEGRATION, CUSTOM, MCP) and enforces risk-level gates
- Skill-aware execution: explicit skill routing (task.metadata.skill_id) and semantic matching (all skills loaded) both work correctly
- Skill injection follows assigned_at ASC ordering with 40/30/20/10 context budget; truncation removes newest skills first
- Tool requirement validation at hydration: missing required tools produce warnings and prefixed instructions, not failures
- Proficiency tracking: agent_skill_proficiency.times_used and last_used_at updated after each skill execution
- Celery Beat scheduler triggers scheduled jobs and meetings on time (>99% on-time rate)
- Stats reconciliation corrects drifted counters to match source table aggregates
- Secret resolution follows scope hierarchy (AGENT > TOOL > PROJECT > INSTANCE, narrowest scope wins)
- Prometheus metrics exposed with all 6 custom worker metrics
- Structured JSON logging with correlation IDs propagated from gateway via Celery kwargs
- Docker image builds and runs correctly (multi-stage, non-root user, all 3 queues)
- All tests passing with 90%+ code coverage
CLAUDE.md Rules for Agent Worker Agents
These rules MUST be placed in the CLAUDE.md file at the root of the praesidium2-python/services/agent-worker/ directory.
# praesidium2-worker CLAUDE.md
## Project Context
This is the Agent Worker service for Praesidium2 -- the AI execution engine.
Celery workers consume tasks from Redis queues and execute them using direct
Anthropic SDK agent loops (standard execution) and LangGraph (durable workflows
needing crash recovery or HITL approval gates). This is where agents DO the work.
## Architecture Rules
- This service NEVER touches the database directly -- all data via qqq REST API
- The ONLY exception is LangGraph's PostgresSaver which writes checkpoint state
to its own managed tables in PostgreSQL (langgraph_checkpoints)
- Direct Anthropic SDK is used for "WHO does WHAT" -- single-agent tasks (V1), turn-taking meetings (V1.5)
- LangGraph is used for "HOW complex work flows" -- jobs (V1), forge + multi-step projects (V1.5)
- Meetings and forge tasks are V1.5 -- do not implement in V1
- All task results must be persisted to qqq API before acknowledging the Celery task
- Broadcast status changes via Redis pub/sub so the gateway can push to browsers
## HITL Interrupt/Resume Protocol (v1.8)
- CRITICAL tool calls trigger interrupt() -- state serialized to langgraph_checkpoints
- Write task_approval record with SHA-256 action_payload_hash
- Update task status to AWAITING_APPROVAL
- On resume: pick up Redis event, re-hydrate from checkpoint using thread_id
- Tamper-proof: gateway verifies action hash before dispatching resume
## Observability (v1.8)
- All Celery tasks propagate DD trace IDs from gateway headers
- Instrument graph.invoke() with ddtrace.tracer.trace("agent.reasoning_loop")
- Capture LLM token usage + latency as span metadata
- Emit DogStatsD custom metrics (praesidium.agent.token_usage, etc.)
## Celery Rules
- Never change queue names (quick, standard, long) without updating the gateway dispatcher
- acks_late=True on standard and long queues -- tasks acknowledged AFTER completion only
- Every task must handle re-delivery gracefully (idempotency guards)
- LangGraph tasks MUST check for existing checkpoints before re-executing
- Never block the event loop -- use async where possible, run sync code in executors
## Tool Safety Rules
- PLATFORM and INTEGRATION tools run in-process -- they are trusted code
- CUSTOM tools MUST be dispatched to the tool-runner (k8s Job) -- never run user code in-process
- Always validate tool input against JSON Schema BEFORE execution
- Always check risk_level gates BEFORE execution
- Log every tool call to activity_log
- All tools MUST return standardized ToolResponse: {success, data, error, usage: {tokens, cost}}
No raw dicts or ad-hoc return shapes -- every tool output flows through ToolResponse (v1.10)
## Coding Standards
- Python 3.12, type hints on all functions
- Pydantic models for task payloads -- no raw dicts
- structlog for structured JSON logging -- no print() statements
- Every task function must be decorated with @app.task(bind=True)
- Use the shared LLMClient abstraction -- never call LLM SDKs directly
## Testing Rules
- Mock qqq API (respx), Redis (fakeredis), and LLM SDKs in all tests
- Use testcontainers (PostgreSQL) for LangGraph checkpoint tests
- Test every task's happy path AND failure/retry path
- Test all tool risk-level gates
- Target 90%+ test coverage
## What NOT to Do
- Do NOT add HTTP endpoints -- this is a worker, not a web server
- Do NOT run user-provided code in-process -- dispatch to tool-runner
- Do NOT bypass risk-level safety gates for any tool
- Do NOT hardcode API keys, model names, or URLs -- use Pydantic Settings from env
- Do NOT skip writing tests for new tasks or tools
- Do NOT acknowledge Celery tasks before persisting results to qqq API
Tool Runner
Epic 7 — Phase 2Sandboxed execution environment for CUSTOM tools + “Blind” Secret Pipeline.
The Tool Runner is a minimal, security-hardened container image that receives tool input as JSON, executes the tool code in isolation, and returns a ToolResult JSON via stdout. It runs as ephemeral k8s Jobs (one per execution) or as warm-pool Deployments for frequently-used tools. Re-sequenced to Phase 2 (before Agent Workers) so that workers have a tested execution target from Day 1. Implements the “Blind” Secret Pipeline: plain-text secrets NEVER touch the database or Gateway. The secret_ref table stores only AWS Secret ARNs; the Worker pod uses IRSA to fetch values directly from AWS Secrets Manager at runtime. Tool Runner Jobs receive DD_TRACE_ID and DD_PARENT_ID as env vars for distributed tracing.
Repository Setup
| Repository | praesidium2-tool-runner/ |
| Language | Python 3.12 |
| Dependencies | Minimal — pydantic, httpx (base image only) |
| Base Images | praesidium-tool-runner:python, praesidium-tool-runner:typescript |
| Execution | k8s Job (ephemeral) or warm Deployment (HTTP) |
| Isolation | k8s NetworkPolicy, resource limits, read-only filesystem |
| Credential Source | AWS Secrets Manager → k8s external-secrets → env vars |
How It Works
The Tool Runner has a single responsibility: receive tool input as JSON, execute the tool code, return a ToolResult JSON via stdout. The agent worker orchestrates everything else (version resolution, credential injection, k8s Job creation).
Execution Flow (Agent Worker → Tool Runner)
- Step 1: Agent worker receives
tool_callfor a CUSTOM tool - Step 2: Resolve
tool_version→ git SHA + entry point module - Step 3: Resolve credentials →
secret_ref→ Secrets Manager - Step 4: Create k8s Job spec with: image, command, env vars, resource limits, network policy, timeout
- Step 5: Submit k8s Job, wait for completion (poll or watch)
- Step 6: Read pod stdout → parse
ToolResultJSON - Step 7: Pod auto-terminates (k8s Job TTL: 60s after completion)
- Step 8: Return result to agent conversation
CLI Invocation (Ephemeral Mode)
# Inside the k8s Job container:
python -m tool_runner \
--entry my_tool:execute \
--input '{"url": "https://api.hubspot.com/campaigns", "date_range": "last_30d"}' \
--timeout 30
# stdout (captured by agent worker):
{"success": true, "data": {"campaigns": [...]}, "error": null, "metadata": {"duration_ms": 1240}}
HTTP Invocation (Warm Pool Mode)
# Agent worker calls the warm pod's HTTP endpoint:
POST http://tool-runner-python:8080/execute
Content-Type: application/json
{
"entry": "my_tool:execute",
"input": {"url": "https://api.hubspot.com/campaigns"},
"timeout": 30
}
# Response:
{"success": true, "data": {"campaigns": [...]}, "error": null, "metadata": {"duration_ms": 240}}
Execution Mode Decision Logic
Each tool has an execution_mode field set at tool registration time, stored in the tool table. The agent worker reads this field before dispatching.
| Mode | When to Use | Examples |
|---|---|---|
ephemeral (default) |
Any tool with side effects, write operations, or external API mutations. Safest — fully sandboxed k8s Job with resource limits, network policy, and auto-cleanup. | create_jira_issue, slack_send_message, github_create_pr |
warm_pool |
Read-only tools with no side effects. Lower latency (~5x faster) via persistent HTTP Deployment. Only for tools that cannot mutate external state. | search_agents, search_knowledge, list_projects |
# Worker dispatch logic (simplified):
tool = registry.get(tool_name)
if tool.execution_mode == "warm_pool":
result = await http_client.post(f"http://tool-runner-python:8080/execute", json=payload)
else: # "ephemeral" (default)
result = await k8s_job_manager.run(tool, payload)
Tool Interface Contract
All CUSTOM tools implement the same BaseTool base class. This contract is shared between the Tool Forge (build time) and the Tool Runner (execution time).
from pydantic import BaseModel
from typing import Any
class ToolContext(BaseModel):
agent_id: str
project_id: str | None = None
conversation_id: str | None = None
credentials: dict = {} # Injected from secret_ref via Secrets Manager
class ToolResult(BaseModel):
success: bool
data: Any = None
error: str | None = None
metadata: dict = {} # duration_ms, retries, bytes_transferred, etc.
class BaseTool:
name: str
description: str
input_schema: dict # JSON Schema for input validation
async def execute(self, input: dict, context: ToolContext) -> ToolResult:
raise NotImplementedError
Contract Rules
- Tool MUST return a
ToolResultJSON on stdout — even on failure (setsuccess: false) - Tool MUST NOT write to stderr except for debug logging (stderr is captured separately for diagnostics)
- Tool MUST NOT read from stdin — all input comes via CLI args or HTTP body
- Tool MUST complete within the configured timeout — SIGTERM at timeout, SIGKILL at timeout+5s
- Tool MUST NOT attempt to contact any host not in the NetworkPolicy allowlist
Base Container Images
Two base images cover the supported tool languages. Each image contains the tool runner framework + language runtime. Tool code is mounted at runtime via k8s volume (git checkout or ConfigMap).
praesidium-tool-runner:python
FROM python:3.12-slim
# Security: non-root user
RUN groupadd -r toolrun && \
useradd -r -g toolrun -d /app toolrun
WORKDIR /app
# Base dependencies only
COPY pyproject.toml .
RUN pip install --no-cache-dir pydantic httpx
# Tool runner framework
COPY src/ ./src/
# Tool code mounted here at runtime
VOLUME /app/tools
USER toolrun
# Read-only filesystem (except /tmp)
ENTRYPOINT ["python", "-m", "src.runner"]
Python-Only Runtime
The Tool Runner is Python-only. All custom tools are Python modules executed in a Python 3.12-slim sandbox via runner.py. There is no TypeScript runner — the Python sandbox provides the sole execution environment for K8s Jobs and warm-pool Deployments.
Execution Modes
Ephemeral Mode (k8s Job)
Default mode. A new k8s Job is created for each tool execution. Pod is created, runs the tool, outputs JSON to stdout, and is auto-deleted after TTL.
| Lifecycle | Created per execution, auto-deleted (TTL: 60s) |
| Cold Start | ~3-5s (image pull cached, pod scheduling) |
| Isolation | Full — separate pod, separate network namespace |
| Use Case | Infrequent tools, high-risk tools, first execution |
| Tool Code | Mounted via initContainer git clone at pinned SHA |
Warm Pool Mode (Deployment)
Optimization for frequently-used tools. Pre-loaded pods with HTTP endpoint. Tool code mounted via ConfigMap (updated on tool_version publish).
| Lifecycle | Long-running Deployment, HPA-scaled |
| Latency | <200ms (HTTP call to warm pod) |
| Isolation | Shared pod, process-level isolation per request |
| Use Case | High-frequency tools (>10 calls/hour), low-risk tools |
| Tool Code | ConfigMap volume, updated on publish via rolling restart |
Network Isolation (k8s NetworkPolicy)
Tool Runner pods operate under strict network policies. Egress is restricted to only the integration URLs that the tool is authorized to access. DNS is allowed for name resolution.
# k8s/network-policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: tool-runner-egress
spec:
podSelector:
matchLabels:
app: tool-runner
policyTypes:
- Egress
egress:
# Allow DNS resolution
- to:
- namespaceSelector: {}
ports:
- protocol: UDP
port: 53
# Allow integration endpoints only (per-tool from integration_config.base_url)
- to:
- ipBlock:
cidr: ${INTEGRATION_CIDR} # Resolved at Job creation time
ports:
- protocol: TCP
port: 443
Network Rules
- NO access to the qqq API from Tool Runner pods — qqq access is via
ToolContexton the agent worker side only - NO access to the internal k8s API server
- NO access to Redis, PostgreSQL, or any internal service
- ONLY outbound HTTPS to pre-approved integration URLs (from
integration_config.base_url) - DNS resolution allowed (required for HTTPS hostname resolution)
Resource Limits by Risk Level
Resource limits are set per tool based on the risk_level type table value assigned during Tool Forge review. Higher risk = more resources but stricter monitoring.
| Risk Level | CPU Request | CPU Limit | Memory Request | Memory Limit | Timeout | Max Retries |
|---|---|---|---|---|---|---|
| LOW | 0.25 CPU | 0.5 CPU | 256Mi | 512Mi | 30s | 3 |
| MEDIUM | 0.5 CPU | 1 CPU | 512Mi | 1Gi | 60s | 2 |
| HIGH | 0.5 CPU | 1 CPU | 1Gi | 2Gi | 120s | 1 |
| CRITICAL | 1 CPU | 2 CPU | 2Gi | 4Gi | 300s | 0 |
Resource Enforcement
- k8s enforces CPU and memory limits natively (OOMKilled if exceeded)
- Timeout enforced by the Tool Runner process: SIGTERM at timeout, SIGKILL at timeout + 5s
- CRITICAL tools get zero retries — failures escalate to human immediately
- Risk level is set during Tool Forge REVIEW step and stored on
tool.risk_level_id
Credential Injection
Credentials are injected as environment variables into the Tool Runner pod. The agent worker resolves credentials from secret_ref and maps them to k8s secret references in the Job spec.
Resolution Flow
Agent worker resolves credentials for the tool (narrowest scope wins):
1. Check AGENT-scoped secret_ref matching agent_id (narrowest)
2. If not found, check TOOL-scoped secret_ref matching tool_id
3. If not found, check PROJECT-scoped matching project_id
4. If not found, check INSTANCE-scoped (broadest fallback)
5. Fetch secret_path from secret_ref table
6. Map to k8s secret reference in Job spec:
# In the k8s Job spec:
env:
- name: HUBSPOT_API_KEY
valueFrom:
secretKeyRef:
name: praesidium-secrets-${INSTANCE}
key: HUBSPOT_API_KEY
- name: TOOL_INPUT
value: '{"url": "https://api.hubspot.com/campaigns"}'
- name: TOOL_AGENT_ID
value: "uuid-of-agent"
- name: TOOL_PROJECT_ID
value: "uuid-of-project"
Credential Security Rules
- Credentials NEVER written to disk inside the pod — env vars only
- k8s secrets sourced from AWS Secrets Manager via external-secrets operator
- Pod
serviceAccountNamehas minimal RBAC — no access to other secrets - Credentials are scoped: a TOOL-scoped secret is only injected when that specific tool runs
k8s Job Template
The agent worker generates this Job spec dynamically for each CUSTOM tool execution. Template variables are resolved from the tool registry and secret_ref tables.
# k8s/job-template.yaml (agent worker fills in variables)
apiVersion: batch/v1
kind: Job
metadata:
name: tool-exec-${EXECUTION_ID}
namespace: ${NAMESPACE}
labels:
app: tool-runner
tool-id: ${TOOL_ID}
agent-id: ${AGENT_ID}
spec:
ttlSecondsAfterFinished: 60
activeDeadlineSeconds: ${TIMEOUT}
backoffLimit: 0
template:
metadata:
labels:
app: tool-runner
spec:
restartPolicy: Never
serviceAccountName: tool-runner-sa
securityContext:
runAsNonRoot: true
runAsUser: 1000
fsGroup: 1000
readOnlyRootFilesystem: true
initContainers:
- name: git-checkout
image: alpine/git:latest
command: ["sh", "-c",
"git clone --no-checkout ${TOOL_REPO_URL} /tools && cd /tools && git checkout ${GIT_SHA}"]
volumeMounts:
- name: tool-code
mountPath: /tools
containers:
- name: runner
image: praesidium-tool-runner:${LANGUAGE}
args: ["--entry", "${MODULE}:${FUNC}",
"--input", "${INPUT_JSON}",
"--timeout", "${TIMEOUT}"]
resources:
requests:
cpu: ${CPU_REQUEST}
memory: ${MEM_REQUEST}
limits:
cpu: ${CPU_LIMIT}
memory: ${MEM_LIMIT}
env:
${CREDENTIAL_ENV_VARS}
- name: DD_TRACE_ID
value: "${TRACE_ID}"
- name: DD_PARENT_ID
value: "${PARENT_SPAN_ID}"
volumeMounts:
- name: tool-code
mountPath: /app/tools
readOnly: true
- name: tmp
mountPath: /tmp
volumes:
- name: tool-code
emptyDir: {}
- name: tmp
emptyDir:
sizeLimit: 100Mi
Day-by-Day Implementation (Phase 2)
| Day | Tasks | Deliverable |
|---|---|---|
| Day 6 |
Base image + tool interface • Create praesidium2-tool-runner/ repo• Implement BaseTool, ToolContext, ToolResult classes• Implement runner.py CLI entry point (arg parsing, module loading, JSON output)• Implement loader.py dynamic module loading• Write Dockerfile.python base image• Write unit tests for base classes and runner |
Working CLI: python -m tool_runner --entry mock:execute --input '{}' returns valid JSON |
| Day 7 |
k8s Job integration + sandbox • Implement sandbox.py (timeout signals, memory monitoring)• Write k8s Job template YAML with SHA-only git checkout (not --branch)• Write NetworkPolicy YAML • Write resource quota presets for each risk level • Test Job creation/execution locally (Docker Desktop k8s) • Verify network isolation (tool cannot reach qqq API or internal services) |
Ephemeral k8s Job runs a mock tool and returns result to stdout |
| Day 8 |
“Blind” Secret Pipeline + credential injection • Implement IRSA-based secret fetching (Boto3 + IRSA IAM role) • Worker fetches secret_ref ARN from qqq, resolves value from AWS Secrets Manager• Inject credentials as env vars in k8s Job spec (never written to disk) • Add DD_TRACE_ID + DD_PARENT_ID env vars to Job spec• Write integration tests for secret pipeline • Test credential flow end-to-end (secret_ref → Secrets Manager → env var → tool) |
“Blind” secret pipeline working: no plain-text secrets in DB or Gateway responses |
| Day 9-10 |
Warm pool + worker integration + hardening • Implement http_server.py warm-pool HTTP endpoint• Wire agent worker to create k8s Jobs for CUSTOM tool_calls • Add warm-pool routing (check tool frequency, route to HTTP if warm) • Add timeout handling, OOM detection, error wrapping • Verify “Hello World” tool execution from Gateway bus • Security audit: verify non-root, read-only FS, network policy enforced |
Full pipeline working: Gateway → CUSTOM tool → sandboxed execution → result returned |
Dependencies
Python Base Image (Runtime)
Intentionally minimal. Tool-specific dependencies are installed at build time by the Tool Forge and baked into the tool's requirements.txt (installed in the initContainer or warm-pool image layer).
Development / Testing
Test Requirements
pytest for all tests. Mock tool execution with fixture tools (pass, fail, timeout, OOM). Verify resource limits and contract compliance.
Mandatory Test Coverage
- Runner CLI: test JSON input parsing, ToolResult JSON output, exit codes (0 for success, 1 for tool error, 2 for framework error)
- BaseTool contract: test that tools returning non-ToolResult JSON are caught and wrapped in error ToolResult
- Loader: test dynamic module import from file path, test missing module error, test invalid entry point format
- Timeout enforcement: test that
mock_tool_timeout.pyis killed after configured timeout, returnssuccess: falsewith timeout error - Error handling: test unhandled exception in tool code returns
success: falsewith error message (no stack trace in data) - HTTP server: test warm-pool endpoint accepts POST, returns ToolResult JSON, handles concurrent requests
- Credential injection: test that env vars are readable inside tool execution context
- Coverage target: 90%+
Acceptance Criteria
python -m tool_runner --entry mock:execute --input '{}'returns validToolResultJSON on stdout- k8s Job runs a CUSTOM tool in an ephemeral pod and exits cleanly (TTL auto-delete verified)
- Warm-pool pod accepts HTTP POST and returns ToolResult with <200ms latency
- NetworkPolicy blocks all egress except pre-approved integration URLs + DNS
- Resource limits enforced: OOMKilled if tool exceeds memory limit, SIGTERM/SIGKILL on timeout
- Credentials injected as env vars from k8s secrets (sourced from Secrets Manager)
- Pod runs as non-root user (UID 1000) with read-only root filesystem
- Tool errors are wrapped in
ToolResult(success=false)— no raw exceptions leak to agent - Python base image builds and passes smoke tests
- End-to-end flow works: agent worker creates Job → tool runs → result returned to conversation
- All tests passing with 90%+ coverage
CLAUDE.md Rules for Tool Runner Agents
These rules MUST be placed in the CLAUDE.md file at the root of the praesidium2-tool-runner/ repository.
# praesidium2-tool-runner CLAUDE.md
## Project Context
This is the Tool Runner for Praesidium2 -- a sandboxed execution environment
for CUSTOM (forged) tools. It receives JSON input, executes tool code in
isolation, and returns ToolResult JSON via stdout. It runs as k8s Jobs
(ephemeral) or warm-pool Deployments (HTTP).
## Architecture Rules
- The Tool Runner is MINIMAL -- it does NOT import qqq, LangGraph, or any agent framework
- The only runtime dependencies are pydantic and httpx -- nothing else in the base image
- Tool-specific dependencies are installed per-tool (in requirements.txt), NOT in the base image
- All tool code is mounted at /app/tools via k8s volume -- never baked into the base image
- Output is ALWAYS valid ToolResult JSON on stdout -- even on error
- stderr is for debug logging only -- agent worker captures but does not parse it
## Security Rules (NON-NEGOTIABLE)
- Container runs as non-root user (UID 1000)
- Root filesystem is read-only (only /tmp is writable, with 100Mi limit)
- NetworkPolicy restricts egress to integration URLs only -- NO access to qqq, Redis, PostgreSQL
- Credentials are env vars ONLY -- never written to disk ("Blind" Secret Pipeline)
- Secrets fetched via IRSA from AWS Secrets Manager -- plain-text NEVER in DB or Gateway
- Resource limits are enforced by k8s (CPU, memory) and by the runner (timeout via signals)
- NEVER disable security context, network policy, or resource limits for debugging
- Use SHA-only git checkouts (git clone --no-checkout + git checkout $SHA)
## Observability (v1.8)
- DD_TRACE_ID and DD_PARENT_ID are injected as env vars by the agent worker
- Tool Runner logs are automatically correlated with parent Worker task in Datadog
## Coding Standards
- Python 3.12, strict typing (mypy)
- pydantic for all data models
- pytest for testing, 90%+ coverage target
- ruff for linting and formatting
- Minimal dependencies -- justify every addition in PR description
## What NOT to Do
- Do NOT add qqq client, database drivers, or ORM libraries
- Do NOT add web frameworks beyond the minimal HTTP server for warm pool
- Do NOT store state between executions -- each run is stateless
- Do NOT log credentials or tool input/output at INFO level (DEBUG only, redacted)
- Do NOT add retry logic inside the runner -- retries are managed by the agent worker
Frontend SPA
Epic 3 — Days 1-10React 19 SPA — the command center for the AI-augmented workforce.
The Frontend is a single-page application providing primary pages: Dashboard, Projects, Team, Chat, Escalations, Settings, Activity, Cost Center, Approvals, KPIs, Schedules, and the HITL Approval Dashboard (with action hash verification). It talks to two backends: the qqq API (data CRUD) and the Agent Gateway (AI interactions, streaming). The Command Palette (Cmd+K) is backed by /api/search (pgvector semantic search). Real-time updates via WebSocket (Redis pub/sub) and SSE (chat streaming). All type tables cached on load for instant rendering. Datadog RUM provides frontend observability linked to backend traces. V1.5 additions: Meetings page and Tool Forge pages (specs retained below).
Repository Setup
| Repository | praesidium2-web/ |
| Framework | React 19 + Vite |
| Language | TypeScript (strict mode) |
| Package Manager | pnpm |
| Routing | TanStack Router (file-based, type-safe) |
| State / Data | TanStack Query + Zustand |
| Styling | Tailwind CSS + shadcn/ui |
| Charts | Recharts |
| Real-time | WebSocket (broadcasts) + SSE (chat streaming) |
| Testing | Vitest + Testing Library + MSW |
| Auth | Authentik OIDC (JWT) |
API Client Layer
Two API clients — one for data operations (qqq), one for AI interactions (Agent Gateway). Both use the same Authentik JWT for authentication.
qqq-client.ts (Data CRUD)
| Action | Method |
|---|---|
| List records | GET /api/{table}?filter=...&sort=...&page=... |
| Get by ID | GET /api/{table}/{id}?include=... |
| Create | POST /api/{table} |
| Update | PATCH /api/{table}/{id} |
| Delete | DELETE /api/{table}/{id} |
All qqq-client methods are generic: qqqClient.list<Project>('project', params). TypeScript types are generated from qqq's OpenAPI spec.
gateway-client.ts (AI Interactions)
| Action | Method |
|---|---|
| Chat with agent | POST /api/chat/{agent_id} → SSE stream |
| Execute task | POST /api/task/execute → job ID |
| Start meeting | POST /api/meeting/{meeting_id}/start → WS updates |
| Forge session | POST /api/forge/session → interactive SSE |
| Stream events | GET /api/stream/{id} (SSE) |
Real-Time Architecture
Two real-time channels serve different purposes. WebSocket for push notifications (cache invalidation), SSE for streaming agent responses (chat).
WebSocket (useWebSocket hook)
Single persistent connection via Agent Gateway. Receives push events from Redis pub/sub. Used primarily for TanStack Query cache invalidation.
| Event | Action |
|---|---|
task:status_changed | Invalidate task queries |
agent:status_changed | Invalidate team queries |
meeting:started | Show meeting notification |
escalation:created | Show escalation alert + badge |
job:completed | Invalidate job queries |
project:health_changed | Invalidate project queries |
tool:published | Invalidate tool catalog queries |
SSE (useSSEStream hook)
Per-conversation stream for agent responses. Opened when user is viewing a chat. Closed on navigation away.
| Event | Description |
|---|---|
token | Text delta (streamed character by character) |
thinking | Agent is thinking (show indicator) |
tool_call_start | Tool call initiated (show tool card) |
tool_call_result | Tool call completed (update card) |
message_complete | Full message done (finalize) |
error | Stream error (show error state) |
Reconnection Strategy
- WebSocket: exponential backoff (1s, 2s, 4s, 8s, max 30s). Reconnect on close. Show connection indicator in header.
- SSE: auto-reconnect with
EventSourcebuilt-in retry. Resume from last event ID.
Type Table Caching (Zustand Store)
All 59 type tables are fetched once on app load and cached in a Zustand store. Components reference type data by ID for instant rendering. No additional API calls needed for status labels, colors, or icons.
// stores/type-tables.ts
interface TypeTableStore {
tables: Record<string, TypeRow[]>; // keyed by table name
byId: Record<string, Record<number, TypeRow>>; // fast lookup by ID
isLoaded: boolean;
load: () => Promise<void>; // fetch all 56 tables in parallel
}
// Usage in components:
const taskStatus = useTypeTable('task_status');
const statusRow = taskStatus.byId[task.status_id];
// statusRow = { id: 3, name: 'IN_PROGRESS', label: 'In Progress', color: '#3b82f6', icon: 'spinner' }
Type Tables Loaded at Startup (59 tables)
member_type, member_status, agent_category, execution_status, tool_category, tool_source_type, risk_level, test_status, programming_language, sync_status, integration_type, auth_type, health_status, project_type, project_status, project_health, priority_level, project_role, milestone_status, task_type, task_status, author_role, conversation_type, conversation_status, participant_role, message_role, content_type, meeting_type, memory_type, memory_source, document_source_type, index_status, job_status, job_execution_status, trigger_type, escalation_status, artifact_type, interaction_channel, interaction_type, interaction_status, schedule_status
Auth Integration (Authentik OIDC)
Standard OIDC Authorization Code flow with PKCE. JWT stored in memory (not localStorage). Refresh handled proactively at 80% of expiry window.
OIDC Flow
- Redirect to Authentik
/authorizewith PKCE challenge - Authentik authenticates user, redirects back with code
- Exchange code for JWT at
/tokenendpoint - Parse JWT claims:
policies,praesidiumInstanceId,praesidiumAllAccess - Store JWT in Zustand auth store (memory only)
- Attach JWT as
Authorization: Bearerto all API calls - Proactive refresh at 80% of token expiry
Permission-Driven UI
// lib/permissions.ts
function canPerform(
policies: PolicyStatement[],
action: string,
resource: string
): boolean {
// Evaluate IAM-style policies
// Deny takes precedence over Allow
}
// Usage in components:
const { can } = usePermissions();
{can('edit', 'project') &&
<Button>Edit Project</Button>
}
// Sidebar items filtered by permission:
{can('read', 'escalation') &&
<SidebarItem to="/escalations" />
}
Page Implementation Order (5 Phases)
Pages are built in dependency order. The app shell and auth must work before any data pages. Core data pages before advanced features. Chat requires SSE streaming infrastructure.
| Phase | Pages / Components | Why This Order |
|---|---|---|
| Phase 1 |
App Shell • Root layout ( __root.tsx): sidebar, header, outlet• Sidebar navigation (permission-filtered) • Header (user menu, connection indicator, notifications) • Command palette (Cmd+K) • Auth flow (login, token, refresh, logout) • Type table loading on startup • WebSocket connection setup |
Everything else renders inside the shell. Auth must work before any API calls. Type tables must load before any data-driven components render. |
| Phase 2 |
Core Data Pages • Dashboard ( /): project health cards, team status, escalation summary, meeting schedule, cost burn• Projects ( /projects, /projects/:id): card grid, tabbed detail (overview, tasks kanban, team, milestones, meetings, activity, integrations)• Team ( /team, /team/:id): member grid, agent detail (skills radar, memory explorer, tools, history, performance)
|
Dashboard is the landing page. Projects and Team are the most-used pages. All are pure qqq API reads — no Gateway dependency. |
| Phase 3 |
Chat • Chat list ( /chat): conversation cards, filters• Chat thread ( /chat/:conv_id): message list, SSE streaming, tool call visualization, thinking indicator• Chat input: message compose, model selector, context display • Group chats: multi-agent conversations |
Chat requires SSE streaming infrastructure (useSSEStream hook) and Gateway API integration. Built after core data pages are stable. |
| Phase 4 |
Supporting Pages • Meetings ( /meetings, /meetings/:id): schedule, transcripts, action items V1.5• Escalations ( /escalations): inbox sorted by severity, acknowledge/resolve/dismiss• Schedules ( /schedules, /schedules/:id): schedule list, detail/edit with cron picker, run history v1.9• Settings ( /settings): instance config, integrations, agent templates, knowledge base• Activity ( /activity): chronological feed, filterable, real-time via WS• Cost Center ( /costs): token usage charts (Recharts), budget alerts
|
These pages depend on patterns established in Phase 2-3. Meetings page deferred to V1.5 (V1 uses Slack-based status). Schedules page (v1.9) provides CRUD for scheduled_job records with cron picker and run history. Cost Center needs Recharts integration. |
| Phase 5 |
Tool Forge V1.5 • Tool catalog ( /forge): tool cards, search, filter by category• Tool detail ( /forge/:id): versions, usage stats, assigned agents• Build mode ( /forge/build): split-pane (left: chat with forge agent, right: code preview + test output + file tabs)• Approve/Request Changes/Reject actions |
Deferred to V1.5. Most complex page. Split-pane requires SSE streaming + code preview + interactive actions. Built last when all infrastructure is proven. |
Component Library
shadcn/ui Primitives
Custom Components
Type-Table-Driven Components
StatusBadge, PriorityBadge, and HealthBadge all read their label, color, and icon from the cached type tables. Adding a new status value = add a row to the type table. Zero frontend code changes needed.
Day-by-Day Implementation
| Day | Tasks | Deliverable |
|---|---|---|
| P1 D3 |
Scaffold + App Shell • pnpm create vite praesidium2-web --template react-ts• Install TanStack Router, TanStack Query, Zustand, Tailwind, shadcn/ui • Configure TypeScript strict mode, path aliases • Set up file-based routing structure • Build root layout: sidebar, header, outlet • Build command palette shell (Cmd+K) |
App runs with sidebar navigation between empty route stubs |
| P1 D4 |
Auth + API Clients + Type Tables • Implement Authentik OIDC flow (lib/auth.ts) • Build qqq-client.ts with typed generic methods • Build gateway-client.ts shell • Implement type table Zustand store + loading • Build auth Zustand store (JWT, user, permissions) • Add auth guard to root layout |
User can log in via Authentik. Type tables load on startup. Unauthenticated users redirected to login. |
| P1 D5 |
Shared Components + WebSocket • Build StatusBadge, PriorityBadge, HealthBadge (type-table-driven) • Build MemberAvatar, RelativeTime, CostDisplay • Implement useWebSocket hook (connect, reconnect, event dispatch) • Wire WS events to TanStack Query cache invalidation • Build ConnectionIndicator for header |
Shared components render with type table data. WS connection established with reconnection. |
| P2 D6 |
Dashboard + Projects List • Dashboard page: project health cards, team status grid, escalation summary, today's meetings, cost burn rate widget • Projects list page: card grid with search, filter by status/type/health • ProjectCard component • Wire to qqq API with TanStack Query |
Dashboard renders live data. Projects list shows all projects with filtering. |
| P2 D7 |
Project Detail (Tabbed) • Overview tab: description, health, dates, integrations • Tasks tab: KanbanBoard component (drag-and-drop) • Team tab: assigned members with roles • Milestones tab: MilestoneTimeline component • Meetings tab: meeting list for project • Activity tab: project-filtered activity feed |
Full project detail page with all 6 tabs rendering real data. |
| P2 D8 |
Team Pages • Team list page: grid of MemberCard components (agents + humans) • Member detail page (agent): SkillRadar, MemoryExplorer, tool assignments, execution history, performance metrics • Member detail page (human): profile, projects, interactions • Agent hire dialog (from template library) |
Full team browsing and agent detail pages working. |
| P2 D9 |
Chat (SSE Streaming) • Chat list page: conversation cards with last message preview • Implement useSSEStream hook • Chat thread: message rendering, streaming token display • ThinkingIndicator component • ToolCallCard component (shows tool name, input, output, status) • Chat input: message compose, model selector |
Users can chat with agents. Responses stream in real-time with tool call visualization. |
| P2 D10 |
Chat (Polish) + Meetings + Escalations • Group chats (multi-agent conversations) • Meetings list + detail (transcripts, action items) • Escalations inbox (severity sort, acknowledge, resolve, dismiss) • Activity feed page (filterable, real-time via WS) |
Chat polished. Meetings, Escalations, and Activity pages functional. |
| P3 D11 |
Settings + Cost Center • Settings page: instance config, integrations management, agent template library, knowledge base, notification preferences • Cost Center page: Recharts integration (token usage by agent, by project, by time), burn rate chart, budget alerts • Setup wizards: instance setup wizard, project setup wizard, agent creation wizard (v1.12) |
Settings and Cost Center pages complete. All 3 wizards functional. |
| P3 D12 |
Tool Forge V1.5 • Tool catalog page: ToolCard grid, search, filter by category • Tool detail page: versions, usage stats, assigned agents • Build mode: split-pane layout (chat left, code right) • CodePreview component (syntax highlighted) • Test output panel • Approve / Request Changes / Reject actions |
Deferred to V1.5. Full Tool Forge UI including split-pane build mode. |
| P3 D13 |
Testing + Polish • Write Vitest + Testing Library tests for all pages • MSW handlers for qqq and Gateway API mocking • Accessibility audit (keyboard nav, screen reader labels) • Responsive design verification (mobile, tablet, desktop) • Error boundaries and loading skeletons for all pages |
90%+ test coverage. All pages accessible and responsive. |
Agent Creation Wizard v1.12
Triggered from Team page → “New Agent” button (or by the COS agent requesting a new teammate). Multi-step wizard with progress persisted in browser localStorage. If the user leaves mid-wizard, they resume on return.
| Step | Name | UI Details |
|---|---|---|
| 1 | Identity | Name, display name, avatar upload/select. Choose a soul/personality from available templates or write a custom one. |
| 2 | Role | Job title, job description. Defines what this agent is responsible for within the instance. |
| 3 | Skills | Browse company skill catalog (published skill_definition records). Select skills, choose version per skill (default: latest). System validates tool requirements against instance availability; missing requirements flagged with warnings. |
| 4 | MCP Servers | Select from instance’s configured mcp_server records. Shows which skills require which MCP connections. |
| 5 | Tools | Assign additional “freestyle” tools beyond what skills bring. Browse instance’s available tools by category. |
| 6 | Model | Select LLM model (default from template). Show capability tier and cost per 1M tokens. |
| 7 | Review & Create | Summary of everything configured. Estimated cost per task based on model + tool usage. “Create Agent” button. |
Published Profile Fork Workflow v1.12
When a Super Operator clicks “Fork” on a published agent profile from the company catalog, this 7-step workflow creates a fully configured agent in a single transaction.
| Step | Name | Details |
|---|---|---|
| 1 | Select Version | UI shows available versions from published_agent_profile_version. User picks one (default: latest). |
| 2 | Customize | Pre-filled form with the version’s soul, role_title, job_description, personality_traits, model. User can modify any field. |
| 3 | Review Skills | List of skills from profile_skill_assignment for that version. Toggle skills on/off. For each skill, system checks tool requirements against instance’s available tools and MCP servers. Missing requirements flagged with warnings. |
| 4 | Configure MCP | For each profile_mcp_configuration, user must select or create a matching mcp_server in their instance (profiles specify server TYPE, not specific connections). Auto-selected if matching server exists. |
| 5 | Review Tools | Direct tool assignments from profile_tool_assignment. User confirms or adjusts. |
| 6 | Create | Single transaction creates: team_member (member_type=AGENT) + agent + agent_skill records (pinned to profile versions) + agent_mcp_assignment records + agent_tool_assignment records + profile_fork_record (provenance). |
| 7 | Verify & Activate | Summary card showing the new agent, its skills, MCP connections, and tools. “Activate” button sets agent status to ACTIVE. |
Dependencies
Production
Development / Testing
Dockerfile (Multi-Stage Build)
Build stage compiles TypeScript and bundles with Vite. Serve stage uses Nginx to serve static files with proper SPA routing (fallback to index.html).
# Stage 1: Build FROM node:22-slim AS build WORKDIR /app COPY package.json pnpm-lock.yaml ./ RUN corepack enable && pnpm install --frozen-lockfile COPY . . RUN pnpm build # Stage 2: Serve FROM nginx:alpine COPY --from=build /app/dist /usr/share/nginx/html COPY nginx.conf /etc/nginx/conf.d/default.conf # SPA routing: all routes fallback to index.html # nginx.conf must include: try_files $uri $uri/ /index.html; EXPOSE 80
Test Requirements
Vitest + Testing Library for all component and page tests. MSW (Mock Service Worker) for API mocking. No real API calls in tests.
Mandatory Test Coverage
- Pages: Every page renders without errors with mocked data (11 page tests minimum)
- Auth: Login redirect, token parsing, permission evaluation, logout cleanup
- Type tables: Store loads all tables, byId lookup works, handles loading state
- WebSocket: Connection, reconnection on close, event dispatch to query invalidation
- SSE streaming: Token events append to message, tool_call events render ToolCallCard, error events show error state
- Permissions: IAM policy evaluation (Allow, Deny, Deny-takes-precedence), UI elements conditionally render
- Components: StatusBadge renders correct color/label from type table, KanbanBoard renders columns, ProjectCard shows health indicator
- API client: qqq-client methods attach JWT, handle 401 (redirect to login), handle 403 (show forbidden)
- Coverage target: 90%+
Acceptance Criteria
- All 11 primary pages render correctly with real data from qqq API
- Schedule Management: list view, detail/edit with cron picker, run history, + New Schedule form
- Authentik OIDC login/logout flow works end-to-end (redirect, token, refresh)
- Permission-driven UI: sidebar items and action buttons respect IAM policies from JWT
- All 59 type tables load on startup and drive StatusBadge/PriorityBadge/HealthBadge rendering
- WebSocket connection established with auto-reconnect; cache invalidation working on push events
- SSE chat streaming works: tokens stream in real-time, tool calls visualized, thinking indicator shown
- Dashboard shows live project health, team status, escalation summary, meeting schedule, cost burn rate
- Project detail page: all 6 tabs render (overview, kanban, team, milestones, meetings, activity)
- Team page: agent detail shows skills radar, memory explorer, tool assignments, execution history
- Tool Forge: split-pane build mode works (chat left, code preview right, approve/reject actions)
- Command palette (Cmd+K) searches across projects, agents, tasks, conversations, tools
- Setup wizards: instance setup, project setup, and agent creation wizards complete with progress persistence
- Agent creation wizard: 7-step flow (identity, role, skills, MCP, tools, model, review) with skill tool-requirement validation
- Published profile fork workflow: fork from company catalog creates agent + all assignments in single transaction
- Docker image builds (multi-stage: Node build + Nginx serve) and serves SPA correctly
- All tests passing with 90%+ coverage (Vitest + Testing Library + MSW)
- Responsive design works on mobile, tablet, and desktop viewports
CLAUDE.md Rules for Frontend Agents
These rules MUST be placed in the CLAUDE.md file at the root of the praesidium2-web/ repository.
# praesidium2-web CLAUDE.md ## Project Context This is the React 19 SPA frontend for Praesidium2 -- the command center for Greater Goods' AI-augmented workforce. It provides 11 primary pages for managing projects, agents, conversations, tools, approvals, KPIs, schedules, and escalations. Meetings and forge pages are deferred to V1.5. ## Architecture Rules - Two API backends: qqq API (data CRUD) and Agent Gateway (AI interactions) - Use TanStack Router for ALL routing (file-based, type-safe) - Use TanStack Query for ALL server state (qqq API reads + mutations) - Use Zustand ONLY for client-side state (auth, type tables, WebSocket status) - NEVER store server state in Zustand -- that is TanStack Query's job - All type table lookups go through the Zustand type-table store (cached at load) - StatusBadge, PriorityBadge, HealthBadge MUST read from type tables -- never hardcode values - WebSocket events MUST invalidate TanStack Query caches -- never manually refetch - SSE streaming MUST use the useSSEStream hook -- never raw EventSource - Schedule pages (/schedules, /schedules/:id) use qqq API for scheduled_job CRUD -- cron picker component validates cron expressions client-side ## Auth Rules - JWT stored in memory ONLY (Zustand auth store) -- NEVER in localStorage or cookies - Every API request MUST include Authorization: Bearer header - On 401 response: clear auth store, redirect to Authentik login - Permission checks use IAM-style policy evaluation (Deny takes precedence) - UI elements MUST be permission-gated (sidebar items, action buttons) ## Coding Standards - TypeScript strict mode -- no any types, no type assertions unless unavoidable - All components are functional (no class components) - Use shadcn/ui primitives as the base -- customize via Tailwind, don't fork - pnpm for package management (not npm, not yarn) - Vitest + Testing Library for all tests - MSW for API mocking in tests -- no real API calls - Target 90%+ test coverage ## What NOT to Do - Do NOT install axios -- use fetch or the typed qqq-client/gateway-client - Do NOT use Redux, MobX, or any state library besides TanStack Query + Zustand - Do NOT create custom CSS files -- use Tailwind utility classes only - Do NOT hardcode status labels, colors, or icons -- always use type table lookups - Do NOT store JWTs in localStorage or sessionStorage - Do NOT make API calls outside of TanStack Query hooks or mutation wrappers - Do NOT skip writing tests -- every page and critical component needs tests
Local Dev Environment
Epic 8 — Days 1, 4Day 1 deliverable. All other services depend on a working local dev environment.
Local development runs on the same k8s stack as every other environment. A local k8s cluster (Docker Desktop or OrbStack) provides the runtime. Tilt watches source directories and hot-deploys to a praesidium-local namespace via the same Helm charts from praesidium-cd used in dev/staging/prod. Ollama provides local LLM inference (replacing Anthropic API). Each service repo also has a docker/compose.yml for standalone local dev. Full design: docs/superpowers/specs/2026-04-16-local-development-design.md
Local Development Environment (k8s-native via Tilt)
A single tilt up builds all images from Dockerfiles in each service repo, deploys to the local k8s cluster via Helm charts from praesidium-cd, and live-reloads on source changes. The same Helm charts run across local/dev/staging/prod.
brew install orbstack, lighter on RAM/CPU, Kubernetes included). The Tiltfile targets the docker-desktop context by default — update allow_k8s_contexts() if using OrbStack. A Java 21 JDK must be available on the host to build the qqq-api Maven image.
Prerequisites
| Tool | Purpose | Install |
|---|---|---|
Docker Desktop or OrbStack | Container runtime + local k8s cluster | Docker Desktop: enable Kubernetes in Settings; OrbStack: brew install orbstack |
kubectl | Cluster management CLI | Bundled with Docker Desktop / OrbStack, or brew install kubectl |
Helm 3 | Chart-based deployment (same charts as prod) | brew install helm |
Tilt | Live-reload orchestrator for local k8s | brew install tilt-dev/tap/tilt |
Java 21 JDK | Required to build the qqq-api Maven image | brew install openjdk@21 |
Node.js 20+ | Required for web Vite/React image build | brew install node or via nvm |
Python 3.11+ | Required for gateway / worker image builds | Bundled in images; host Python needed only for pre-commit hooks |
Dev Workflow
./scripts/local-dev-setup.sh— one-time namespace + dependency setuptilt up— builds images, deploys via Helm, watches for changes- Gateway + Worker: Tilt
live_updatesyncssrc/into running pods, uvicorn/celery auto-reloads - Web: Tilt
live_updatesyncssrc/, Vite HMR inside the pod - qqq: Full image rebuild on change (Java requires recompile)
- Authentik admin at
https://localhost:9443/if/admin/(port-forwarded) - Tilt UI at
http://localhost:10350— logs, build status, resource health
Port Forwards (automatic via Tilt)
| Service | Local Port | Pod Port |
|---|---|---|
qqq-api | 8080 | 8080 |
gateway | 8000 | 8000 |
web | 3000 | 3000 |
authentik | 9443 | 9443 |
postgresql | 5432 | 5432 |
redis | 6379 | 6379 |
minio | 9000 / 9001 | 9000 / 9001 |
ollama | 11434 | 11434 |
jaeger | 16686 | 16686 |
Tiltfile
The Tiltfile builds all service images from Dockerfiles in each service repo, deploys via Helm charts from praesidium-cd to the praesidium-local namespace, watches source directories for live-reload, and port-forwards key services.
# Tiltfile
# ── Configuration ──────────────────────────────────
load('ext://namespace', 'namespace_create')
namespace_create('praesidium-local')
allow_k8s_contexts('docker-desktop')
# ── Infrastructure Dependencies (Helm charts) ─────
# PostgreSQL (bitnami)
helm_resource(
'postgresql',
'oci://registry-1.docker.io/bitnamicharts/postgresql',
flags=['--namespace=praesidium-local',
'--set=auth.database=praesidium',
'--set=auth.username=praesidium',
'--set=auth.password=localdev',
'--set=image.tag=16',
'--set=primary.extendedConfiguration=shared_preload_libraries=vector'],
port_forwards=['5432:5432'],
labels=['infra']
)
# Redis (bitnami)
helm_resource(
'redis',
'oci://registry-1.docker.io/bitnamicharts/redis',
flags=['--namespace=praesidium-local',
'--set=auth.enabled=false',
'--set=architecture=standalone'],
port_forwards=['6379:6379'],
labels=['infra']
)
# Authentik
helm_resource(
'authentik',
'oci://ghcr.io/goauthentik/helm/authentik',
flags=['--namespace=praesidium-local',
'--set=authentik.secret_key=local-dev-secret-key-change-in-prod',
'--set=authentik.postgresql.host=postgresql',
'--set=authentik.postgresql.user=praesidium',
'--set=authentik.postgresql.password=localdev',
'--set=authentik.postgresql.name=authentik',
'--set=authentik.redis.host=redis-master'],
port_forwards=['9443:9443'],
resource_deps=['postgresql', 'redis'],
labels=['infra']
)
# MinIO (S3-compatible object storage)
helm_resource(
'minio',
'oci://registry-1.docker.io/bitnamicharts/minio',
flags=['--namespace=praesidium-local',
'--set=auth.rootUser=praesidium',
'--set=auth.rootPassword=localdevkey'],
port_forwards=['9000:9000', '9001:9001'],
labels=['infra']
)
# Ollama (local LLM inference)
helm_resource(
'ollama',
'./charts/ollama',
flags=['--namespace=praesidium-local'],
port_forwards=['11434:11434'],
labels=['llm']
)
# Jaeger (distributed tracing)
helm_resource(
'jaeger',
'jaegertracing/jaeger',
flags=['--namespace=praesidium-local',
'--set=allInOne.enabled=true'],
port_forwards=['16686:16686'],
labels=['observability']
)
# ── Application Services ──────────────────────────
# qqq-api (Java — full rebuild on change)
docker_build(
'praesidium2-qqq',
context='./praesidium2-qqq',
dockerfile='./praesidium2-qqq/Dockerfile'
)
# gateway (Python — live_update for hot reload)
docker_build(
'praesidium2-gateway',
context='./praesidium2-python',
dockerfile='./praesidium2-python/services/agent-gateway/Dockerfile',
live_update=[
sync('./praesidium2-python/services/agent-gateway/src', '/app/src'),
run('cd /app && pip install -e .', trigger=['./praesidium2-python/services/agent-gateway/pyproject.toml']),
]
)
# worker (Python — live_update for hot reload)
docker_build(
'praesidium2-worker',
context='./praesidium2-python',
dockerfile='./praesidium2-python/services/agent-worker/Dockerfile',
live_update=[
sync('./praesidium2-python/services/agent-worker/src', '/app/src'),
run('cd /app && pip install -e .', trigger=['./praesidium2-python/services/agent-worker/pyproject.toml']),
]
)
# web (React — live_update for Vite HMR)
docker_build(
'praesidium2-web',
context='./praesidium2-web',
dockerfile='./praesidium2-web/Dockerfile',
live_update=[
sync('./praesidium2-web/src', '/app/src'),
run('cd /app && pnpm install', trigger=['./praesidium2-web/package.json']),
]
)
# tool-runner (Python — no live_update, runs as Job)
docker_build(
'praesidium2-tool-runner',
context='./praesidium2-tool-runner',
dockerfile='./praesidium2-tool-runner/Dockerfile'
)
# ── Deploy via Helm (charts from praesidium-cd) ───
k8s_yaml(helm(
'./praesidium-cd',
name='praesidium',
namespace='praesidium-local',
values=['./praesidium-cd/values-local.yaml']
))
# ── Port Forwards ─────────────────────────────────
k8s_resource('qqq-api', port_forwards=['8080:8080'], resource_deps=['postgresql', 'authentik'], labels=['app'])
k8s_resource('gateway', port_forwards=['8000:8000'], resource_deps=['qqq-api', 'redis', 'ollama'], labels=['app'])
k8s_resource('worker', resource_deps=['qqq-api', 'redis', 'ollama'], labels=['app'])
k8s_resource('web', port_forwards=['3000:3000'], resource_deps=['gateway'], labels=['app'])
Helm values-local.yaml
Local overrides for the same Helm charts used in dev/staging/prod. Single-replica services, local infrastructure dependencies, observability disabled.
# praesidium-cd/values-local.yaml
# Local k8s development overrides (used by Tiltfile)
global:
imageRegistry: "" # local images, no registry prefix
imagePullPolicy: Never # use locally-built images
instance:
namespace: praesidium-local
id: "local-dev"
# ── Per-service: single replica, local images ─────
qqq-api:
replicas: 1
image: praesidium2-qqq
tag: latest
resources:
requests: { cpu: 250m, memory: 512Mi }
limits: { cpu: 1000m, memory: 2Gi }
env:
SPRING_PROFILES_ACTIVE: local
SPRING_DATASOURCE_URL: jdbc:postgresql://postgresql:5432/praesidium
SPRING_DATASOURCE_USERNAME: praesidium
SPRING_DATASOURCE_PASSWORD: localdev
AUTHENTIK_ISSUER_URL: https://authentik:9443/application/o/praesidium/
AUTHENTIK_JWKS_URL: https://authentik:9443/application/o/praesidium/jwks/
gateway:
replicas: 1
image: praesidium2-gateway
tag: latest
resources:
requests: { cpu: 100m, memory: 256Mi }
limits: { cpu: 500m, memory: 1Gi }
env:
QQQ_API_URL: http://qqq-api:8080
REDIS_URL: redis://redis-master:6379/0
MINIO_ENDPOINT: http://minio:9000
MINIO_ACCESS_KEY: praesidium
MINIO_SECRET_KEY: localdevkey
AUTHENTIK_ISSUER_URL: https://authentik:9443/application/o/praesidium/
LLM_PROVIDER: ollama
LLM_BASE_URL: http://ollama:11434
LLM_MODEL: llama3.1:70b
worker:
replicas: 1
image: praesidium2-worker
tag: latest
hpa:
enabled: false
resources:
requests: { cpu: 250m, memory: 512Mi }
limits: { cpu: 1000m, memory: 2Gi }
env:
QQQ_API_URL: http://qqq-api:8080
REDIS_URL: redis://redis-master:6379/0
MINIO_ENDPOINT: http://minio:9000
CELERY_BROKER_URL: redis://redis-master:6379/1
CELERY_RESULT_BACKEND: redis://redis-master:6379/2
LLM_PROVIDER: ollama
LLM_BASE_URL: http://ollama:11434
LLM_MODEL: llama3.1:70b
web:
replicas: 1
image: praesidium2-web
tag: latest
resources:
requests: { cpu: 50m, memory: 64Mi }
limits: { cpu: 100m, memory: 128Mi }
env:
VITE_API_URL: http://localhost:8080
VITE_GATEWAY_URL: http://localhost:8000
VITE_AUTHENTIK_URL: https://localhost:9443
# ── Infrastructure (deployed by Tilt helm_resource) ─
postgresql:
enabled: false # managed by Tilt helm_resource
redis:
enabled: false # managed by Tilt helm_resource
authentik:
enabled: false # managed by Tilt helm_resource
# ── Observability (disabled locally) ──────────────
datadog:
enabled: false
apm: false
rum: false
Local Setup Script
./scripts/local-dev-setup.sh — one-time setup for local k8s development. Idempotent (safe to re-run).
#!/usr/bin/env bash
set -euo pipefail
# scripts/local-dev-setup.sh
# One-time local k8s development environment setup
echo "==> Verifying prerequisites..."
command -v docker >/dev/null || { echo "ERROR: Docker not found"; exit 1; }
command -v kubectl >/dev/null || { echo "ERROR: kubectl not found"; exit 1; }
command -v helm >/dev/null || { echo "ERROR: Helm not found"; exit 1; }
command -v tilt >/dev/null || { echo "ERROR: Tilt not found"; exit 1; }
# Verify Docker Desktop k8s is running
kubectl cluster-info --context docker-desktop >/dev/null 2>&1 \
|| { echo "ERROR: Docker Desktop Kubernetes not running. Enable it in Docker Desktop settings."; exit 1; }
echo "==> Setting kubectl context to docker-desktop..."
kubectl config use-context docker-desktop
echo "==> Creating namespace praesidium-local..."
kubectl create namespace praesidium-local --dry-run=client -o yaml | kubectl apply -f -
echo "==> Adding Helm repos..."
helm repo add bitnami https://charts.bitnami.com/bitnami 2>/dev/null || true
helm repo add authentik https://charts.goauthentik.io 2>/dev/null || true
helm repo update
echo "==> Setup complete. Run 'tilt up' to start the development environment."
echo " Tilt UI will be available at http://localhost:10350"
Multi-Stage Dockerfiles (in each service repo)
Each service repo owns its own Dockerfile with a multi-stage build for minimal image size and layer caching. Faber uses these for CI builds; Tilt uses them for local k8s dev.
qqq-api (Java)
FROM eclipse-temurin:21-jdk AS build WORKDIR /app COPY build.gradle.kts settings.gradle.kts ./ COPY gradle/ gradle/ RUN ./gradlew dependencies --no-daemon COPY src/ src/ RUN ./gradlew bootJar --no-daemon FROM eclipse-temurin:21-jre COPY --from=build /app/build/libs/*.jar app.jar EXPOSE 8080 ENTRYPOINT ["java", "-jar", "app.jar"]
gateway / worker (Python)
FROM python:3.12-slim AS base
WORKDIR /app
RUN pip install uv
COPY pyproject.toml uv.lock ./
RUN uv sync --frozen --no-dev
FROM python:3.12-slim
WORKDIR /app
COPY --from=base /app/.venv /app/.venv
COPY src/ src/
ENV PATH="/app/.venv/bin:$PATH"
# gateway:
EXPOSE 8000
CMD ["uvicorn", "src.main:app", \
"--host", "0.0.0.0", "--port", "8000"]
# worker:
CMD ["celery", "-A", "src.worker", "worker", \
"--loglevel=info", "--concurrency=4"]
web (React SPA)
FROM node:20-alpine AS build WORKDIR /app COPY package.json pnpm-lock.yaml ./ RUN corepack enable && pnpm install --frozen-lockfile COPY . . RUN pnpm build FROM nginx:alpine COPY --from=build /app/dist /usr/share/nginx/html COPY nginx.conf /etc/nginx/conf.d/default.conf EXPOSE 80
tool-runner (Sandbox)
FROM python:3.12-slim WORKDIR /sandbox RUN pip install uv COPY tool_runner/ tool_runner/ COPY requirements.txt . RUN uv pip install -r requirements.txt # No EXPOSE — runs as k8s Job # Entry point set dynamically via Job spec ENTRYPOINT ["python", "-m", "tool_runner"]
Kubernetes Manifests
Each service gets a Deployment, Service, ConfigMap, and NetworkPolicy. Worker pods get HPA and PDB for production resilience.
Per-Service Resource Matrix
| Service | Replicas (dev / prod) | CPU Request | Memory Request | CPU Limit | Memory Limit |
|---|---|---|---|---|---|
qqq-api | 1 / 2 | 500m | 1Gi | 1000m | 2Gi |
gateway | 1 / 2 | 250m | 512Mi | 500m | 1Gi |
worker | 1 / 2-8 (HPA) | 500m | 1Gi | 2000m | 4Gi |
web | 1 / 2 | 50m | 64Mi | 100m | 128Mi |
tool-runner | Job (on-demand) | Varies by risk_level | 512Mi-4Gi | 500m-2000m | 512Mi-4Gi |
Example: Gateway Deployment + Service
apiVersion: apps/v1
kind: Deployment
metadata:
name: gateway
namespace: {{ .Values.instance.namespace }}
spec:
replicas: {{ .Values.gateway.replicas }}
selector:
matchLabels:
app: gateway
template:
metadata:
labels:
app: gateway
spec:
terminationGracePeriodSeconds: 60
containers:
- name: gateway
image: {{ .Values.gateway.image }}:{{ .Values.gateway.tag }}
ports:
- containerPort: 8000
envFrom:
- configMapRef:
name: gateway-config
- secretRef:
name: gateway-secrets
livenessProbe:
httpGet:
path: /healthz
port: 8000
initialDelaySeconds: 10
periodSeconds: 15
readinessProbe:
httpGet:
path: /readyz
port: 8000
initialDelaySeconds: 5
periodSeconds: 10
resources:
requests:
cpu: {{ .Values.gateway.resources.requests.cpu }}
memory: {{ .Values.gateway.resources.requests.memory }}
limits:
cpu: {{ .Values.gateway.resources.limits.cpu }}
memory: {{ .Values.gateway.resources.limits.memory }}
---
apiVersion: v1
kind: Service
metadata:
name: gateway
spec:
selector:
app: gateway
ports:
- port: 8000
targetPort: 8000
NetworkPolicy (restrict pod-to-pod)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: gateway-netpol
spec:
podSelector:
matchLabels:
app: gateway
policyTypes: [Ingress, Egress]
ingress:
- from:
- podSelector:
matchLabels:
app: web # frontend can reach gateway
ports:
- port: 8000
egress:
- to:
- podSelector:
matchLabels:
app: qqq-api # gateway calls qqq
ports:
- port: 8080
- to:
- podSelector:
matchLabels:
app: redis
ports:
- port: 6379
PodDisruptionBudget (production)
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: qqq-api-pdb
spec:
minAvailable: 1
selector:
matchLabels:
app: qqq-api
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: gateway-pdb
spec:
minAvailable: 1
selector:
matchLabels:
app: gateway
Helm Chart Structure (in praesidium-cd)
Umbrella chart with sub-charts per service. Lives in the praesidium-cd repo (not owned by Praesidium2 team). Environment-specific values files override defaults.
values.yaml (defaults)
# Global
global:
imageRegistry: "" # ghcr.io/dmdbrands in prod
imagePullPolicy: IfNotPresent
instance:
namespace: praesidium-dev
id: "" # set per-instance
# Per-service
qqq-api:
replicas: 1
image: praesidium2-qqq
tag: latest
resources:
requests: { cpu: 500m, memory: 1Gi }
limits: { cpu: 1000m, memory: 2Gi }
env:
SPRING_PROFILES_ACTIVE: dev
gateway:
replicas: 1
image: praesidium2-gateway
tag: latest
resources:
requests: { cpu: 250m, memory: 512Mi }
limits: { cpu: 500m, memory: 1Gi }
worker:
replicas: 1
image: praesidium2-worker
tag: latest
hpa:
enabled: false
minReplicas: 2
maxReplicas: 8
targetCPU: 70
resources:
requests: { cpu: 500m, memory: 1Gi }
limits: { cpu: 2000m, memory: 4Gi }
web:
replicas: 1
image: praesidium2-web
tag: latest
resources:
requests: { cpu: 50m, memory: 64Mi }
limits: { cpu: 100m, memory: 128Mi }
Per-Instance Provisioning
Each business unit gets its own k8s namespace. ArgoCD ApplicationSet generates one Application per instance YAML file in the instances/ directory.
Instance YAML Example (tech-team.yaml)
# instances/tech-team.yaml
instance:
id: "tech-team"
namespace: "praesidium-tech-team"
displayName: "CTO Engineering"
# Instance-specific overrides
worker:
replicas: 2
hpa:
enabled: true
minReplicas: 2
maxReplicas: 6
# Instance-specific env vars (merged with defaults)
env:
INSTANCE_ID: "uuid-for-tech-team"
AUTHENTIK_GROUP: "tech-team"
DEFAULT_LLM_MODEL: "claude-sonnet-4-6"
DAILY_BUDGET_USD: "50.00"
Adding a New Business Unit Instance
- Create a new YAML file in
instances/(e.g.,finance.yaml) - Create Authentik company group with matching
praesidiumInstanceIdattribute - Commit and push — ArgoCD ApplicationSet auto-creates the Application
- ArgoCD syncs: creates namespace, deploys all services, applies instance-specific values
Acceptance Criteria
tilt updeploys all services to local k8s with zero manual steps- All health checks pass within 90 seconds of startup
- Live-reload works for gateway, worker (Python via
live_update), and web (Vite HMR vialive_update) - Helm chart installs cleanly on local k8s (Docker Desktop with Kubernetes enabled)
- Same Helm charts from
praesidium-cdused in local/dev/staging/prod (onlyvalues-*.yamldiffers) - Each service has Deployment, Service, ConfigMap, NetworkPolicy, and PDB manifests
- Worker HPA scales pods based on CPU utilization
- Instance YAML in
instances/creates a fully isolated namespace deployment - All Docker images build in <3 minutes with layer caching
./scripts/local-dev-setup.shis idempotent and validates all prerequisites
CLAUDE.md Rules for Local Dev / k8s / CI
# Per-Service Repo CLAUDE.md Rules ## CI/CD Pattern (each service repo) Each service repo has: - .faber.yml at root (pipeline: java-webapp, python-webapp, or node-webapp) - Dockerfile (or docker/Dockerfile) — owned by this repo - docker/compose.yml for local dev - Faber handles: lint, test, Docker build, push to GHCR, update CD repo Helm values - Helm charts and ArgoCD config live in praesidium-cd (NOT owned by Praesidium2 team) - Docker registry: ghcr.io/dmdbrands ## Architecture Rules - Local dev uses Tilt + Docker Desktop k8s — Helm charts from praesidium-cd - Every service MUST have a health check in k8s manifests - Every service MUST have resource requests AND limits in k8s - NetworkPolicy MUST be defined for every pod (default deny, explicit allow) - PDB MUST be defined for qqq-api, gateway, and worker - Worker HPA MUST scale on CPU — never scale to zero - Instance YAML in instances/ MUST NOT contain secrets (use Secrets Manager refs) ## What NOT to Do - Do NOT hardcode image tags — always use Helm values - Do NOT use hostPath volumes in k8s manifests - Do NOT expose services via NodePort — always use ClusterIP + Ingress - Do NOT skip multi-stage Docker builds - Do NOT put credentials in ConfigMaps — use Secrets - Do NOT modify praesidium-cd or praesidium-infrastructure repos directly
CI/CD (Faber Pipeline)
Epic 8 — Days 3, 5No PR merges without passing CI. Faber pipeline blocks on any test failure or coverage drop.
Faber is Greater Goods' CI/CD platform. Each service repo has a .faber.yml at its root (pipeline type: java-webapp, python-webapp, or node-webapp). Faber handles: lint, test, Docker build, push to GHCR (ghcr.io/dmdbrands), and update CD repo Helm values. Pipelines run Lint, Test, Build, Push, and GitOps Update stages in order.
Pipeline Stages
Every push to any branch triggers the pipeline. Only develop and main branches push Docker images and update GitOps values.
Per-Language Pipeline Configuration
Python, Java, and TypeScript services each have language-specific lint and test tooling. All three run in parallel during the Lint + Test stages.
Python (Gateway + Worker)
| Lint | ruff check . + ruff format --check . |
| Type Check | mypy src/ |
| Unit Tests | pytest tests/unit/ -v --cov=src --cov-report=xml |
| Integration | pytest tests/integration/ -v --cov-append |
| Coverage Gate | 90% |
| Security | pip-audit + semgrep --config=auto |
Java (qqq API)
| Lint | ./gradlew checkstyleMain |
| Unit Tests | ./gradlew test |
| Integration | ./gradlew integrationTest (Testcontainers) |
| Coverage Gate | 90% |
| Security | ./gradlew dependencyCheckAnalyze |
TypeScript (Web SPA)
| Lint | eslint . && prettier --check . |
| Type Check | tsc --noEmit |
| Unit Tests | vitest run --coverage |
| E2E Tests | playwright test (headless Chromium) |
| Coverage Gate | 90% |
| Security | pnpm audit + semgrep |
Docker Build + GHCR Push
Multi-stage builds with Docker BuildKit layer caching. Each service repo owns its own Dockerfile. Images are pushed to ghcr.io/dmdbrands and tagged with both git SHA and environment label.
Build Configuration
# Faber build stage (per service, Dockerfile in each service repo) docker buildx build \ --cache-from type=registry,ref=$GHCR_REPO:cache \ --cache-to type=registry,ref=$GHCR_REPO:cache,mode=max \ --tag $GHCR_REPO:$GIT_SHA \ --tag $GHCR_REPO:$ENVIRONMENT \ --push \ -f Dockerfile .
| Service | GHCR Repository | Avg Build Time |
|---|---|---|
| qqq-api | ghcr.io/dmdbrands/praesidium/qqq-api | ~90s |
| gateway | ghcr.io/dmdbrands/praesidium/gateway | ~45s |
| worker | ghcr.io/dmdbrands/praesidium/worker | ~45s |
| web | ghcr.io/dmdbrands/praesidium/web | ~60s |
| tool-runner | ghcr.io/dmdbrands/praesidium/tool-runner | ~30s |
Image Tagging Strategy
| Tag Pattern | When Applied | Example |
|---|---|---|
{git-sha} | Every build | a1b2c3d |
develop | Push to develop | Latest develop build |
staging | Push to staging | Staging candidate |
latest | Push to main | Production release |
v{semver} | Git tag | v1.0.0 |
GHCR Image Retention
- Keep last 30 untagged images (cleanup old SHA tags)
- Keep all tagged images (
latest,develop,staging,v*) - Trivy scan on push for CVE vulnerabilities
GitOps Update (Helm Values in praesidium-cd)
After a successful image push to GHCR, Faber commits the new image tag to the praesidium-cd GitOps repository. ArgoCD watches this repo and auto-syncs. Helm charts also live in praesidium-cd.
# Faber GitOps update step (runs after GHCR push)
git clone $GITOPS_REPO /tmp/gitops # praesidium-cd
cd /tmp/gitops
# Determine target values file from branch
case "$BRANCH" in
develop) VALUES_FILE="values-dev.yaml" ;;
staging) VALUES_FILE="values-staging.yaml" ;;
main) VALUES_FILE="values-prod.yaml" ;;
esac
# Update image tags for the changed service
yq eval ".${SERVICE}.tag = \"${GIT_SHA}\"" -i "$VALUES_FILE"
git add "$VALUES_FILE"
git commit -m "ci(${SERVICE}): update to ${GIT_SHA} [skip ci]"
git push origin main
Branch Strategy + Promotion Flow
feature/*→ Lint + Test only (no image push, no deploy)develop→ Lint + Test + Build + Push + Deploy to dev (auto-sync)staging→ Lint + Test + Build + Push + Deploy to staging (auto-sync)main→ Lint + Test + Build + Push + Update prod values (manual ArgoCD sync)- PRs require: passing CI, 1 approval, no unresolved comments
- Hotfix: branch from
main, PR directly tomain, cherry-pick back todevelop
CI Time Budget
Target: full pipeline completes in under 5 minutes. Parallelism is key.
| Stage | Duration Target | Parallel? | Notes |
|---|---|---|---|
| Lint | <30s | Yes (all 3 languages) | Fastest to fail, runs first |
| Test | <3 min | Yes (Python, Java, TS in parallel) | Testcontainers adds ~15s startup |
| Build | <90s | Yes (all images in parallel) | BuildKit cache cuts rebuilds to ~20s |
| Push | <30s | Yes | Only changed service images |
| GitOps | <10s | No | Single commit to values file |
Faber Pipeline Configuration
Each service repository includes a .faber.yml configuration file that Faber reads to determine build, test, and deployment behavior.
# praesidium2-qqq/.faber.yml
pipeline: java-webapp
image_name: praesidium2-qqq
java_version: "21"
docker:
registry: ghcr.io/dmdbrands
cd:
repo: dmdbrands/praesidium-cd
format: helm
coverage:
min_instruction: "90"
health:
path: /qqq-api/health
port: 8000
db: true
services:
- image: pgvector/pgvector:pg17
port: 5432
healthcheck: "pg_isready -U test"
env:
POSTGRES_USER: test
POSTGRES_PASSWORD: testing
POSTGRES_DB: praesidium_test
Each repo has its own .faber.yml. The pipeline type varies: python-webapp for praesidium2-python, node-webapp for praesidium2-web. The pattern is the same — image name, registry, CD repo, coverage gates, and health checks — adjusted per language and service.
Acceptance Criteria
- Pipeline runs lint + test + build for all 3 languages in parallel
- Coverage gates enforced: Python 90%, Java 90%, TypeScript 90%
- Docker images pushed to GHCR (
ghcr.io/dmdbrands) tagged with git SHA + environment - GitOps repo auto-updated with new image tags on develop/staging/main
- Feature branches run lint + test only (no image push)
- Full pipeline completes in <5 minutes
- PRs blocked on failing CI
- Security scans (dependency audit, Semgrep, container scan) run on every build
.faber.ymlconfiguration template committed to repository root
ArgoCD GitOps
Epic 8 — Day 4ArgoCD is the ONLY way services get deployed to Kubernetes. No manual kubectl apply.
ArgoCD watches the GitOps repository and reconciles cluster state with the declared Helm values. One ApplicationSet generates an Application per business unit instance.
ApplicationSet Configuration
A single ApplicationSet resource scans the instances/ directory in the GitOps repo. Each YAML file generates a full Praesidium2 deployment in its own namespace.
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: praesidium-instances
namespace: argocd
spec:
generators:
- git:
repoURL: https://github.com/dmdbrand/praesidium-cd.git
revision: main
files:
- path: "instances/*.yaml"
template:
metadata:
name: "praesidium-{{instance.id}}"
labels:
app.kubernetes.io/part-of: praesidium
praesidium.io/instance: "{{instance.id}}"
spec:
project: praesidium
source:
repoURL: https://github.com/dmdbrand/praesidium-cd.git
targetRevision: main
path: charts/praesidium
helm:
valueFiles:
- values.yaml # base defaults
- "values-{{instance.environment}}.yaml" # env overrides
- "../../instances/{{instance.id}}.yaml" # instance overrides
destination:
server: https://kubernetes.default.svc
namespace: "{{instance.namespace}}"
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
- PrunePropagationPolicy=foreground
retry:
limit: 3
backoff:
duration: 5s
factor: 2
maxDuration: 1m
How Instance Discovery Works
- ArgoCD scans
instances/*.yamlin the GitOps repo - Each file is parsed as Helm values — the
instance.idandinstance.namespacefields drive Application naming - Adding a new file = adding a new business unit. ArgoCD auto-creates the Application within seconds.
- Removing a file = tearing down an instance (prune: true deletes the namespace resources)
Sync Policy by Environment
Dev and staging auto-sync on every commit. Production requires manual approval via the ArgoCD UI or CLI.
| Environment | Auto-Sync | Self-Heal | Prune | Approval |
|---|---|---|---|---|
| dev | Enabled | Enabled | Enabled | None — fully automatic |
| staging | Enabled | Enabled | Enabled | None — fully automatic |
| prod | Disabled | Enabled | Disabled | Manual sync via ArgoCD UI + Slack confirmation |
Production Deployment Process
- CI pushes image and updates
values-prod.yamlin the GitOps repo - ArgoCD detects the diff and marks the Application as OutOfSync
- Slack bot posts a deployment notification to
#praesidium-deployswith diff summary - On-call engineer reviews the diff in ArgoCD UI and clicks Sync
- ArgoCD performs a rolling update with health check verification
- Post-sync notification posted to Slack with success/failure status
Rollback Procedure
Rollbacks are git operations, not kubectl commands. Reverting the values file triggers ArgoCD to automatically restore the previous version.
Automated Rollback (GitOps)
# Revert the last values commit cd praesidium-cd/ git revert HEAD --no-edit git push origin main # ArgoCD detects the revert commit # → Auto-syncs dev/staging immediately # → Prod: marks OutOfSync, awaits manual sync
ArgoCD restores the previous image tag. Kubernetes performs a rolling update back to the prior version. Total rollback time: ~60s for dev, ~2 min for prod (after manual approval).
Emergency Rollback (ArgoCD UI)
- Open ArgoCD UI → select the Application
- Click History and Rollback
- Select the previous healthy revision
- Click Rollback — ArgoCD syncs to that revision
- Follow up with a git revert to keep GitOps repo in sync
Important
Always follow up an ArgoCD UI rollback with a git revert. Otherwise ArgoCD will mark the Application as OutOfSync on the next reconciliation loop.
Health Check Verification
ArgoCD verifies each service is healthy after every sync. A sync is only marked Healthy when all pods pass readiness probes.
| Service | Readiness Probe | Startup Timeout | ArgoCD Degraded After |
|---|---|---|---|
qqq-api | GET /readyz (DB migrations complete) | 90s | 120s |
gateway | GET /readyz (qqq API reachable, Redis connected) | 30s | 60s |
worker | Celery inspect active (at least 1 worker responding) | 60s | 90s |
web | GET / (Nginx serves static files) | 10s | 30s |
Sync Failure Behavior
- If a pod fails readiness within the timeout, ArgoCD marks the sync as Degraded
- The
syncPolicy.retryconfiguration retries up to 3 times with exponential backoff - After all retries fail, the Application stays in Degraded state — Slack alert fires
- Self-heal ensures any manual kubectl changes get reverted to match git state
Notifications (Slack Integration)
ArgoCD Notifications controller sends deploy status to Slack. Every sync event triggers a notification.
# argocd-notifications-cm ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-notifications-cm
data:
service.slack: |
token: $slack-token
trigger.on-sync-succeeded: |
- when: app.status.operationState.phase in ['Succeeded']
send: [slack-deploy-success]
trigger.on-sync-failed: |
- when: app.status.operationState.phase in ['Error', 'Failed']
send: [slack-deploy-failure]
trigger.on-health-degraded: |
- when: app.status.health.status == 'Degraded'
send: [slack-health-degraded]
template.slack-deploy-success: |
slack:
channel: praesidium-deploys
blocks: |
- type: section
text:
type: mrkdwn
text: |
:white_check_mark: *{{.app.metadata.name}}* deployed successfully
*Revision:* {{.app.status.sync.revision | truncate 7}}
*Environment:* {{.app.spec.destination.namespace}}
template.slack-deploy-failure: |
slack:
channel: praesidium-deploys
blocks: |
- type: section
text:
type: mrkdwn
text: |
:x: *{{.app.metadata.name}}* deploy FAILED
*Revision:* {{.app.status.sync.revision | truncate 7}}
*Error:* {{.app.status.operationState.message}}
| Event | Slack Channel | Mention |
|---|---|---|
| Sync Succeeded | #praesidium-deploys | None |
| Sync Failed | #praesidium-deploys | @oncall-eng |
| Health Degraded | #praesidium-alerts | @oncall-eng |
| Prod Pending Sync | #praesidium-deploys | @james |
Acceptance Criteria
- ApplicationSet generates one Application per instance YAML file
- Adding a new instance YAML auto-creates the full namespace deployment
- Dev/staging auto-sync within 30 seconds of a values commit
- Production requires manual sync approval
- Rollback via git revert restores previous version within 60 seconds
- Health checks verified for all pods before sync marked Healthy
- Slack notifications fire on sync success, failure, and health degradation
- Self-heal reverts manual kubectl changes within the reconciliation loop (default 3 min)
AWS Setup
Epic 8 — Phase 5Phase 5 deliverable. AWS deployment leverages existing EKS cluster + Faber CI/CD + ArgoCD. Production deploy in Phase 5 (Engineering team).
AWS services replace local k8s equivalents: RDS for PostgreSQL, ElastiCache for Redis, S3 for MinIO. EKS runs all application pods. All services are HIPAA-eligible.
EKS Cluster Configuration
Managed Kubernetes cluster with separate node groups for application workloads and tool execution sandboxes.
Cluster Settings
| Kubernetes Version | 1.29+ |
| Region | us-east-1 |
| VPC | Dedicated VPC with private subnets across 3 AZs |
| Control Plane Logging | API, Audit, Authenticator logs to CloudWatch |
| Encryption | Secrets encrypted at rest with KMS |
| Add-ons | CoreDNS, kube-proxy, VPC CNI, EBS CSI, Metrics Server |
Node Groups
| Node Group | Instance Type | Min / Max | Purpose |
|---|---|---|---|
app-nodes | t3.xlarge (4 vCPU, 16 GB) | 2 / 6 | qqq-api, gateway, worker, web |
tool-nodes | t3.medium (2 vCPU, 4 GB) | 0 / 10 | tool-runner Jobs (scale to zero) |
Autoscaling
app-nodes: Cluster Autoscaler scales based on pending pod requeststool-nodes: Scale from 0 — tool-runner Jobs trigger node provisioning (~60s cold start)- Spot instances for
tool-nodes(cost savings, Jobs are short-lived and retryable)
RDS PostgreSQL + pgvector
Managed PostgreSQL with pgvector extension for embedding storage. Multi-AZ for production resilience.
| Engine | PostgreSQL 17.x |
| Instance Class | db.r6g.large (2 vCPU, 16 GB) |
| Storage | 100 GB gp3 (auto-scaling to 500 GB) |
| Multi-AZ | Yes (synchronous replication) |
| Read Replica | 1 replica (for memory search, activity feeds) |
| Encryption | AES-256 at rest (KMS), TLS 1.3 in transit |
| Backups | Automated daily snapshots (35-day retention) + PITR |
| Extensions | vector, pg_stat_statements, pg_trgm |
RDS Proxy (Connection Pooling)
| Purpose | Pool DB connections for worker pods (many short connections) |
| Max Connections | 200 (RDS limit), Proxy multiplexes to ~50 backend conns |
| IAM Auth | Enabled — pods use IRSA to authenticate to Proxy |
| TLS | Required for all connections |
Connection Routing
- qqq-api: Connects to RDS Proxy endpoint (read/write)
- gateway: Connects to read replica endpoint for heavy read queries
- worker: Connects to RDS Proxy (high connection churn)
- LangGraph checkpoints: Same RDS instance, separate schema
ElastiCache Redis
Two separate Redis clusters for workload isolation: one for task queues (durable) and one for caching (ephemeral).
| Cluster | Instance Type | Nodes | Persistence | Eviction | Purpose |
|---|---|---|---|---|---|
| Queue | cache.r6g.large | Primary + 1 replica (Multi-AZ) | AOF (appendfsync everysec) |
noeviction |
Celery broker, task queue. Data must not be lost. |
| Cache | cache.r6g.medium | Primary + 1 replica (Multi-AZ) | None (RDB snapshots daily) | allkeys-lru |
WebSocket pub/sub, session cache, rate limiting, agent hydration cache |
Redis Connection Configuration
- Both clusters: encryption at rest (AES-256) + encryption in transit (TLS 1.3)
- Auth: Redis AUTH token stored in Secrets Manager, injected as env var
- Monitoring: CloudWatch alarms on memory >80%, replication lag >1s, eviction rate spike
- Automatic failover: <30s when primary node fails
S3 Buckets
Three purpose-specific buckets with versioning, encryption, and lifecycle policies.
| Bucket | Purpose | Versioning | Encryption | Lifecycle |
|---|---|---|---|---|
praesidium-artifacts-{env} |
Task artifacts, reports, generated files | Enabled | SSE-S3 (AES-256) | Move to Glacier after 1 year, delete after 3 years |
praesidium-tools-{env} |
CUSTOM tool source code, built packages | Enabled | SSE-KMS | Keep all versions (no expiration) |
praesidium-knowledge-{env} |
Knowledge docs (uploads, Confluence mirrors) | Enabled | SSE-KMS | Move to IA after 6 months, Glacier after 2 years |
S3 Access Controls
- Block all public access (account-level setting)
- Bucket policies restrict to EKS pod roles only
- Access logging enabled to a separate audit bucket
- Cross-region replication for
praesidium-tools(critical data)
Local Dev Equivalent
- MinIO container maps 1:1 to S3 API
- Three buckets auto-created via init script on first
tilt up - Same SDK code works for both (AWS SDK S3 client with endpoint override for MinIO)
Secrets Manager
All secrets stored in AWS Secrets Manager with hierarchical naming and automatic rotation policies.
Secret Organization Structure
praesidium/
global/ # Shared across all instances
authentik-admin-password
ecr-pull-credentials
argocd-slack-token
{instance-id}/ # Per-instance secrets
db-credentials # RDS username + password
redis-auth-token # ElastiCache AUTH
authentik-oidc-client-secret # OAuth2 client secret
service-account-jwt # Worker service identity
secrets/ # User-managed secrets (via secret_ref table)
ANTHROPIC_API_KEY
OPENAI_API_KEY
SLACK_BOT_TOKEN
JIRA_API_TOKEN
GITHUB_PAT
GOOGLE_SERVICE_ACCOUNT_JSON
ZOHO_OAUTH_REFRESH_TOKEN
HUBSPOT_API_KEY
Rotation Policies
| Secret | Rotation | Method |
|---|---|---|
| DB credentials | Every 30 days | RDS-native rotation Lambda |
| Redis AUTH | Every 90 days | Custom Lambda + ElastiCache modify |
| Service account JWT | Every 24 hours | Authentik token refresh (80% of expiry) |
| API keys (user-managed) | Alert at 30/7/1 day before expires_at | Manual (escalation created) |
Pod Access to Secrets
- IRSA (IAM Roles for Service Accounts) for pod-level access
- Each service's k8s ServiceAccount maps to an IAM role
- IAM role grants
secretsmanager:GetSecretValuefor specific secret paths - Secrets injected as env vars via
external-secrets-operatoror CSI driver - Never mounted as files (prevents accidental logging)
ALB / Route53
Application Load Balancer routes traffic to services. Route53 manages DNS per instance.
ALB Configuration
| Type | Application Load Balancer (Layer 7) |
| TLS | ACM certificate, TLS 1.3 termination |
| Ingress Controller | AWS Load Balancer Controller (k8s Ingress annotations) |
| WAF | AWS WAF attached (OWASP Core Rule Set) |
| Access Logs | S3 bucket for ALB access logs |
Route53 DNS (Per Instance)
| Record | Target |
|---|---|
{instance}.praesidium.dmdbrand.com | ALB (web SPA) |
api.{instance}.praesidium.dmdbrand.com | ALB → qqq-api |
ws.{instance}.praesidium.dmdbrand.com | ALB → gateway (WebSocket) |
auth.praesidium.dmdbrand.com | ALB → Authentik |
# Kubernetes Ingress (per instance, via Helm template)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: praesidium-ingress
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:ACCOUNT:certificate/CERT-ID
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
alb.ingress.kubernetes.io/ssl-redirect: "443"
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/wafv2-acl-arn: arn:aws:wafv2:us-east-1:ACCOUNT:regional/webacl/praesidium-waf
spec:
rules:
- host: "{{.Values.instance.id}}.praesidium.dmdbrand.com"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: web
port: { number: 80 }
- host: "api.{{.Values.instance.id}}.praesidium.dmdbrand.com"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: qqq-api
port: { number: 8080 }
- host: "ws.{{.Values.instance.id}}.praesidium.dmdbrand.com"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: gateway
port: { number: 8000 }
CloudWatch Monitoring
Centralized logging and metrics via CloudWatch. Fluent Bit DaemonSet ships container logs. Container Insights provides cluster-level metrics.
Log Groups
| Log Group | Source | Retention |
|---|---|---|
/praesidium/qqq-api | qqq-api pods | 90 days |
/praesidium/gateway | gateway pods | 90 days |
/praesidium/worker | worker pods | 90 days |
/praesidium/tool-runner | tool-runner Jobs | 30 days |
/praesidium/web | web pods (Nginx) | 30 days |
/praesidium/audit | audit_access_log records | 7 years (HIPAA) |
CloudWatch Alarms
| Alarm | Threshold | Action |
|---|---|---|
| RDS CPU | >80% for 5 min | SNS → Slack alert |
| RDS Free Storage | <10 GB | SNS → PagerDuty |
| Redis Memory | >80% | SNS → Slack alert |
| EKS Pod Restarts | >3 in 5 min | SNS → Slack alert |
| ALB 5xx Rate | >1% for 5 min | SNS → PagerDuty |
| ALB Latency p99 | >5s | SNS → Slack alert |
IAM Roles (IRSA)
Each Kubernetes ServiceAccount maps to an IAM role via IRSA (IAM Roles for Service Accounts). Least-privilege access to AWS services.
| Service Account | IAM Role | Permissions |
|---|---|---|
qqq-api-sa |
praesidium-qqq-api-role |
rds-db:connect (RDS Proxy IAM auth),
secretsmanager:GetSecretValue (db-credentials)
|
gateway-sa |
praesidium-gateway-role |
secretsmanager:GetSecretValue (redis-auth, oidc-secret, user-managed secrets),
s3:GetObject/PutObject (artifacts, knowledge)
|
worker-sa |
praesidium-worker-role |
secretsmanager:GetSecretValue (all instance secrets),
s3:* (artifacts, tools, knowledge),
batch:SubmitJob (tool-runner),
rds-db:connect (for LangGraph checkpoints)
|
tool-runner-sa |
praesidium-tool-runner-role |
secretsmanager:GetSecretValue (tool-specific secrets only),
s3:GetObject (tool source code)
|
fluent-bit-sa |
praesidium-fluent-bit-role |
logs:CreateLogStream, logs:PutLogEvents
|
Infrastructure-as-Code (OpenTofu + Terragrunt)
All AWS resources are managed via Infrastructure-as-Code. Praesidium2 runs in its own dedicated AWS accounts under AWS Control Tower (not in a shared legacy account like me-health). This provides billing isolation, blast-radius containment, independent IAM policies, and cleaner security boundaries.
IaC Stack
| IaC Engine | OpenTofu 1.11+ (Terraform-compatible, open-source) |
| Environment Mgmt | Terragrunt (DRY env configs, dependency ordering) |
| State Backend | S3 + DynamoDB locks per account |
| Secrets | SOPS + AWS KMS (encrypted in-repo) |
| CI/CD | CircleCI with Faber orb (plan on PR, apply on merge) |
AWS Account Structure (Control Tower)
Greater Goods AWS Control Tower └─ Management Account └─ praesidium-dev (Dev) └─ praesidium-staging (Staging) └─ praesidium-prod (Production)
Key Difference from me-health
Separate AWS accounts per environment (Control Tower) vs single shared account. Provides: billing isolation, blast-radius containment, independent IAM policies, cleaner security boundaries.
Terraform Module Structure
Modeled after me-health-portal-infrastructure layout with Praesidium-specific modules.
praesidium-infrastructure/ ├─ terraform/ │ ├─ modules/ │ │ ├─ vpc/ # VPC, subnets, endpoints, NAT │ │ ├─ eks/ # EKS cluster, managed node groups │ │ ├─ rds/ # Aurora PostgreSQL 17 + pgvector │ │ ├─ elasticache/ # Redis 7.0 (HA) │ │ ├─ secrets/ # AWS Secrets Manager │ │ ├─ iam/ # IRSA roles, service accounts │ │ ├─ acm/ # TLS certificates │ │ ├─ cloudtrail/ # HIPAA audit logging (6-year retention) │ │ ├─ cluster-addons/ # Helm releases (Datadog, ESO, External DNS, ALB Controller) │ │ ├─ s3/ # Application buckets │ │ └─ kms/ # Encryption keys │ ├─ live/ │ │ ├─ dev/terragrunt.hcl │ │ ├─ staging/terragrunt.hcl │ │ ├─ production/terragrunt.hcl │ │ └─ root.hcl │ └─ root/ │ ├─ main.tf # Module calls │ ├─ variables.tf # 100+ variables │ └─ outputs.tf ├─ .faber.yml # CI/CD config ├─ docs/ │ └─ adr/ # Architecture Decision Records (v1.10) └─ README.md
Per-Environment Resources
| Resource | Dev | Staging | Prod |
|---|---|---|---|
| EKS Nodes | 2x t3.medium (1-3 ASG) | 2x t3.medium (2-4 ASG) | 3x t3.large (2-6 ASG) |
| Aurora PostgreSQL | Serverless v2 (0.5-2 ACU) | Serverless v2 (1-4 ACU) | Provisioned (2x r6g.large) |
| Redis | 1x cache.t3.micro | 1x cache.t3.small | 2x cache.t3.medium (multi-AZ) |
| Datadog | Disabled | Enabled | Enabled |
IRSA Roles (6)
| Role | Purpose |
|---|---|
| AWS Load Balancer Controller | Manages ALB/NLB resources for Ingress |
| External Secrets Operator | Syncs Secrets Manager into K8s Secrets |
| External DNS | Manages Route53 records from Ingress annotations |
| Datadog Agent | Ships metrics, logs, traces to Datadog |
| Agent Worker | Blind Secret Pipeline — fetches from Secrets Manager |
| Tool Runner | Sandboxed, read-only secrets access for tool execution |
Day 1 HIPAA Hardening v1.10
These security controls ship in Phase 1 alongside infrastructure provisioning. No data-at-rest or data-in-transit may exist without encryption from Day 1.
| Service | Control | Configuration |
|---|---|---|
| RDS | Encryption at rest | storage_encrypted = true with KMS Customer Managed Key (CMK) |
| S3 | Default bucket encryption | SSE-KMS on all three buckets (artifacts, tools, knowledge). Bucket policy denies PutObject without encryption header. |
| ALB | HTTPS-only, TLS 1.2 minimum | Listener redirects HTTP → HTTPS. Security policy: ELBSecurityPolicy-TLS13-1-2-2021-06. No plaintext listeners. |
Acceptance Criteria
- EKS cluster running with app-nodes and tool-nodes node groups
- RDS PostgreSQL with pgvector extension, Multi-AZ, automated backups, RDS Proxy
- Two ElastiCache Redis clusters (queue + cache), both Multi-AZ with encryption
- Three S3 buckets with versioning, encryption, and lifecycle policies
- Secrets Manager hierarchy populated with all required secrets
- ALB with TLS termination, WAF, and per-instance routing
- Route53 records resolving to ALB for each instance
- CloudWatch log groups created, Fluent Bit shipping logs
- CloudWatch alarms configured and tested for critical thresholds
- IRSA configured — each pod has least-privilege access to AWS services
- All services are HIPAA-eligible AWS services
- Infrastructure-as-Code (OpenTofu + Terragrunt) defines all resources across Dev/Staging/Prod accounts
- All data-at-rest encrypted (RDS KMS CMK, S3 SSE-KMS), all data-in-transit TLS 1.2+ v1.10
Authentication (Authentik)
Epic 9 — Days 2-3Same pattern as me-health-portal. Authentik is the single identity provider for all Praesidium2 instances.
Authentik provides OAuth2/OIDC for human users and service account JWTs for agent workers. JWT custom claims carry IAM-style policies and instance scoping. qqq enforces all permissions — no authorization logic in application code.
Authentik Setup
Authentik runs as a k8s pod (local dev via Tilt Helm resource, production via EKS). Initial configuration creates the admin user, OAuth2 provider, and application.
Initial Admin Setup
| Admin URL | https://localhost:9443/if/admin/ (local dev) |
| Default Admin | akadmin (set password on first boot) |
| MFA | Required for all human users (TOTP or WebAuthn) |
| Session Expiry | Access token: 15 min, Refresh token: 7 days |
| JWKS Endpoint | /application/o/praesidium/jwks/ |
Provider Configuration
- Create OAuth2/OIDC Provider named
praesidium - Client type: Confidential
- Redirect URIs:
http://localhost:3000/auth/callback+ production URLs - Scopes:
openid profile email praesidium - Signing key: RS256 auto-generated
- Create Application linked to the Provider
- Configure custom scope mappings (see blueprint below)
Authentik Blueprint (YAML)
Blueprints automate Authentik configuration. This blueprint creates a single unified scope mapping (containing both praesidiumInstanceId and policies claims), groups, and the OAuth2 application. Applied via authentik blueprints apply or auto-loaded from the blueprints/ directory.
# blueprints/praesidium-setup.yaml
version: 1
metadata:
name: Praesidium2 Configuration
entries:
# ── Unified Custom Scope Mapping: praesidium ──
# Single mapping returns both praesidiumInstanceId and policies claims.
# Merged from two separate mappings to avoid scope_name conflict.
- model: authentik_providers_oauth2.scopemapping
id: praesidium-unified-scope
attrs:
name: "Praesidium Claims (Instance ID + IAM Policies)"
scope_name: praesidium
expression: |
claims = {}
merged_statements = []
for group in request.user.ak_groups.all():
attrs = group.attributes
# Extract instance ID from company group
if "praesidiumInstanceId" in attrs and "praesidiumInstanceId" not in claims:
claims["praesidiumInstanceId"] = attrs["praesidiumInstanceId"]
claims["praesidiumAllAccess"] = attrs.get("praesidiumAllAccess", False)
# Merge IAM policies from all role groups
policies = attrs.get("policies", {})
stmts = policies.get("Statement", [])
merged_statements.extend(stmts)
if merged_statements:
claims["policies"] = {
"PolicySpec": "praesidium-iam",
"PolicyVersion": "1.0.0",
"Statement": merged_statements
}
return claims
# ── Company Groups (one per instance) ──
- model: authentik_core.group
id: group-tech-team
attrs:
name: "Tech Team"
attributes:
praesidiumInstanceId: "uuid-for-tech-team"
praesidiumAllAccess: false
- model: authentik_core.group
id: group-hr
attrs:
name: "HR Department"
attributes:
praesidiumInstanceId: "uuid-for-hr"
praesidiumAllAccess: false
# ── Role Groups (define permissions) ──
- model: authentik_core.group
id: role-admin
attrs:
name: "Praesidium Admin"
attributes:
policies:
PolicySpec: praesidium-iam
PolicyVersion: "1.0.0"
Statement:
- Effect: Allow
Action: ["read", "insert", "edit", "delete"]
Resource: ["*"]
- model: authentik_core.group
id: role-manager
attrs:
name: "Praesidium Manager"
attributes:
policies:
PolicySpec: praesidium-iam
PolicyVersion: "1.0.0"
Statement:
- Effect: Allow
Action: ["read", "insert", "edit"]
Resource: ["*"]
- Effect: Deny
Action: ["delete"]
Resource: ["agent_template", "integration_config"]
- model: authentik_core.group
id: role-viewer
attrs:
name: "Praesidium Viewer"
attributes:
policies:
PolicySpec: praesidium-iam
PolicyVersion: "1.0.0"
Statement:
- Effect: Allow
Action: ["read"]
Resource: ["*"]
ProdAuthenticationCustomizer (qqq Java Class)
Copied from the me-health-portal pattern. This Java class validates Authentik JWTs and maps custom claims to qqq permissions and RecordSecurityLock values.
// ProdAuthenticationCustomizer.java (simplified)
public class ProdAuthenticationCustomizer implements QAuthenticationModuleCustomizer {
@Override
public QSession createSession(HttpServletRequest request) {
String token = extractBearerToken(request);
DecodedJWT jwt = verifyToken(token); // validates against JWKS endpoint
// Extract instance scoping
String instanceId = jwt.getClaim("praesidiumInstanceId").asString();
boolean allAccess = jwt.getClaim("praesidiumAllAccess").asBoolean();
// Extract IAM policies
Map<String, Object> policiesMap = jwt.getClaim("policies").asMap();
List<PolicyStatement> statements = parsePolicyStatements(policiesMap);
// Build qqq session
QSession session = new QSession();
session.setSecurityKeyValues(Map.of("instance_id", instanceId));
session.setAllAccess(allAccess);
// Map policy Statements to qqq table permissions
for (PolicyStatement stmt : statements) {
for (String resource : stmt.getResources()) {
for (String action : stmt.getActions()) {
TablePermission perm = mapActionToPermission(action);
if (stmt.getEffect() == Effect.ALLOW) {
session.addTablePermission(resource, perm);
} else {
session.removeTablePermission(resource, perm);
}
}
}
}
return session;
}
// Action mapping: read→READ, insert→INSERT, edit→EDIT, delete→DELETE
// Resource "*" applies to all tables
// Deny statements override Allow (evaluated in order, Deny wins)
}
Key Behaviors
- Validates JWT signature against Authentik JWKS endpoint (cached, refreshed every 5 min)
- Extracts
praesidiumInstanceId→ sets as RecordSecurityLock value - Parses IAM-style policy JSON → maps to qqq table permissions (read/insert/edit/delete)
- If
praesidiumAllAccess = true, bypasses all policy checks (admin only) - Deny statements override Allow (evaluated in order, most restrictive wins)
- Missing/expired/invalid tokens → HTTP 401
- Valid token but insufficient permissions → HTTP 403
IAM-Style Policy Format
Policies are JSON objects stored as Authentik group attributes. Users inherit policies from all role groups they belong to. Statements are merged at token issuance time.
Policy Structure
{
"PolicySpec": "praesidium-iam",
"PolicyVersion": "1.0.0",
"Statement": [
{
"Effect": "Allow", // Allow or Deny
"Action": [ // qqq operations
"read",
"insert",
"edit",
"delete"
],
"Resource": [ // table names or "*"
"project",
"task",
"conversation"
]
}
]
}
Default Policy Sets
| Role | Can Read | Can Write | Can Delete |
|---|---|---|---|
| Admin | All tables | All tables | All tables |
| Manager | All tables | All tables | Operational tables only |
| Member | All tables | task, message, conversation | Own records only |
| Viewer | All tables | None | None |
Evaluation Logic
- Default deny: no permissions unless explicitly granted
- Statements evaluated in order from all role groups
- Deny always overrides Allow (regardless of order)
- Resource
"*"matches all tables - Multiple role groups: union of all Allow statements, intersection with all Deny
Worker Service Identity
Agent workers authenticate to qqq API using a service account JWT issued by Authentik. This is NOT an all-access admin account — workers have scoped permissions.
Service Account Configuration
- One service account per instance, created during provisioning
- JWT includes
praesidiumInstanceId→ qqq RLS scopes all queries praesidiumAllAccessis false — workers do NOT bypass RLS- 24-hour token expiry with automatic refresh at 80% of window
- Refresh handled proactively by the worker process to avoid mid-task failures
Worker Policy (Scoped)
{
"Statement": [
{
"Effect": "Allow",
"Action": ["read"],
"Resource": ["*"]
},
{
"Effect": "Allow",
"Action": ["insert", "edit"],
"Resource": [
"task", "task_comment",
"message", "conversation",
"conversation_participant",
"activity_log", "memory",
"job_execution", "escalation",
"artifact", "human_interaction",
"agent_skill", "knowledge_chunk"
]
},
{
"Effect": "Allow",
"Action": ["edit"],
"Resource": ["agent", "scheduled_job"]
},
{
"Effect": "Deny",
"Action": ["delete"],
"Resource": [
"project", "agent_template",
"team_member", "integration_config"
]
}
]
}
Instance Scoping (RecordSecurityLock)
Every root entity table gets a RecordSecurityLock on instance_id. The lock value comes from the JWT praesidiumInstanceId claim. Child tables inherit scoping via FK joins. Same pattern as me-health-portal's mhCompanyId.
How It Works
- qqq adds
instance_id UUID NOT NULLto every root entity table - Value auto-populated from JWT claim on INSERT
- All queries automatically filtered by
instance_id - No application code touches this column directly
- Users in instance A cannot see or modify instance B data
- Admin users with
praesidiumAllAccessbypass the lock (platform admin only)
RecordSecurityLock Config (Java)
// In QqqMetaDataProvider.java
table.withRecordSecurityLock(
new RecordSecurityLock()
.withFieldName("instance_id")
.withLockScope(RecordSecurityLock
.LockScope.READ_AND_WRITE)
.withNullValueBehavior(
NullValueBehavior.DENY)
);
// Child tables use joinNameChain:
childTable.withRecordSecurityLock(
new RecordSecurityLock()
.withFieldName("instance_id")
.withJoinNameChain(List.of(
"project")) // FK to root table
.withLockScope(RecordSecurityLock
.LockScope.READ_AND_WRITE)
);
Acceptance Criteria
- Authentik running with OAuth2/OIDC provider for Praesidium
- Blueprint auto-configures scope mappings, groups, and application
- Human users log in via OIDC flow and receive JWT with custom claims
- Worker service accounts receive scoped JWTs with instance_id
- ProdAuthenticationCustomizer validates JWT and maps policies to qqq permissions
- RecordSecurityLock enforces instance scoping on all root entity tables
- Policy evaluation: Deny overrides Allow, default deny for unmatched actions
- Invalid/expired tokens return 401, insufficient permissions return 403
- MFA enforced for all human users
Testing Strategy
Cross-CuttingTesting is built into every layer from Day 1. No PR merges without passing tests.
Every service has unit, integration, and contract tests. Coverage gates are enforced by CI. Agent-specific behavior tests verify tool call accuracy and memory extraction correctness.
Testing Pyramid
Six layers of testing, from fast unit tests to slow security scans. Lower layers run more frequently and catch issues earlier.
| Layer | Tool | Coverage Target | What It Tests | Run Frequency |
|---|---|---|---|---|
| Unit | pytest / JUnit / Vitest | 90%+ statement coverage | Individual functions, classes, utilities | Every commit |
| Integration | pytest + Testcontainers | 80%+ | Service-to-database, service-to-Redis, service-to-qqq API | Every commit |
| API Contract | pytest + OpenAPI validation | 100% of endpoints | Request/response schema compliance, error codes | Every commit |
| E2E | Playwright (frontend), pytest (backend) | Critical paths | Full user flows: login → create project → chat → execute task | Every PR |
| Load | Locust or k6 | Defined SLAs | Concurrent agent execution, API response times, WebSocket scalability | Weekly + pre-release |
| Security | OWASP ZAP, Snyk, Semgrep, Trivy | Zero HIGH/CRITICAL | XSS, SQLI, SSRF, dependency vulns, container scans, secrets scanning | Every commit |
Per-Service Test Setup
Each service has its own test infrastructure, framework, and coverage requirements.
Python (Gateway + Worker)
# pyproject.toml [tool.pytest]
testpaths = ["tests"]
markers = [
"unit: fast, no deps",
"integration: needs containers",
"contract: API schema tests",
]
# conftest.py
@pytest.fixture(scope="session")
def pg_container():
with PostgresContainer(
"pgvector/pgvector:pg16"
) as pg:
yield pg
@pytest.fixture(scope="session")
def redis_container():
with RedisContainer() as r:
yield r
@pytest.fixture(scope="session")
def minio_container():
with MinioContainer() as m:
yield m
| Framework | pytest 8.x |
| Async | pytest-asyncio |
| Coverage | pytest-cov (90% gate) |
| Mocking | unittest.mock + respx (HTTP) |
| Factories | factory_boy |
Java (qqq API)
// build.gradle.kts
dependencies {
testImplementation(
"org.testcontainers:postgresql")
testImplementation(
"org.testcontainers:junit-jupiter")
testImplementation(
"io.rest-assured:rest-assured")
}
// IntegrationTest base class
@Testcontainers
@SpringBootTest(
webEnvironment = RANDOM_PORT)
abstract class BaseIntegrationTest {
@Container
static PostgreSQLContainer<?> pg =
new PostgreSQLContainer<>(
"pgvector/pgvector:pg16");
}
| Framework | JUnit 5 |
| HTTP Client | RestAssured |
| Coverage | JaCoCo (90% gate) |
| Containers | Testcontainers |
TypeScript (Web SPA)
// vitest.config.ts
export default defineConfig({
test: {
environment: "jsdom",
globals: true,
coverage: {
provider: "v8",
reporter: ["text", "lcov"],
thresholds: {
statements: 90,
branches: 85,
functions: 90,
lines: 90,
},
},
setupFiles: [
"./src/test/setup.ts"
],
},
});
// E2E: playwright.config.ts
export default defineConfig({
use: { headless: true },
projects: [
{ name: "chromium" },
],
});
| Unit/Integration | Vitest + Testing Library |
| E2E | Playwright |
| Coverage | v8 provider (90% gate) |
| Snapshots | Vitest snapshot testing |
Testcontainers Infrastructure
Integration tests run against real dependencies via Testcontainers. No mocks for database, Redis, or object storage in integration tests.
| Container | Image | Used By | Purpose |
|---|---|---|---|
PostgreSQL + pgvector | pgvector/pgvector:pg17 | qqq-api, worker | Real database with vector extension for embedding tests |
Redis | redis:7-alpine | gateway, worker | Real queue/cache for pub/sub and task dispatch tests |
MinIO | minio/minio:latest | gateway, worker | S3-compatible storage for artifact and knowledge doc tests |
Test Data Management
- Factory pattern: Every entity has a factory (agent_factory, project_factory, task_factory, etc.)
- No magic data: All test data is explicit and reproducible
- Fixtures match production: Type table seed data in tests matches production seeds exactly
- Database cleanup: Each test class uses a transaction that rolls back after the test
CI Parallelism
- Python, Java, TypeScript test suites run in parallel in Faber
- Testcontainers add ~15s startup overhead per suite
- Total CI test time target: <3 minutes
- Each test suite gets its own container instances (no shared state)
CI Gates (Enforced by Faber)
PRs are blocked from merging if any of these gates fail.
| Gate | Threshold | Frequency | Blocks PR? |
|---|---|---|---|
| Python coverage | 90% | Every commit | Yes |
| Java coverage | 90% | Every commit | Yes |
| TypeScript coverage | 90% | Every commit | Yes |
| Lint (all languages) | Zero warnings | Every commit | Yes |
| API contract tests | 100% pass | Every commit | Yes |
| Security scan (Semgrep) | Zero HIGH/CRITICAL | Every commit | Yes |
| Dependency audit | Zero known vulns | Every commit | Yes |
| Container image scan (Trivy) | Zero CRITICAL | Every build | Yes |
| Mutation testing | >70% mutation score | Weekly (scheduled) | Advisory only |
| E2E (Playwright) | Critical paths pass | Every PR | Yes |
Mutation Testing (Weekly)
- Python:
mutmut— mutates source code, verifies tests detect changes - TypeScript:
Stryker— similar mutation testing for frontend code - Mutation score <70% triggers a Slack alert to the team (not a CI blocker)
- Purpose: verify tests actually catch bugs, not just exercise code paths
Agent-Specific Testing
Testing AI agent behavior requires a different approach than traditional software tests. We test what the agent does (tool calls, state transitions) not what it says (LLM output text).
Tool Call Verification
# test_agent_tools.py
async def test_agent_creates_task_on_request():
"""Given a user request to create a task,
verify the agent calls the create_task tool
with correct parameters."""
agent = build_test_agent(
tools=[create_task, get_project],
system_prompt=PM_PROMPT
)
result = await agent.execute(
"Create a task 'Fix login bug' on "
"project Alpha, priority HIGH"
)
# Verify tool was called
assert_tool_called(result, "create_task")
call = get_tool_call(result, "create_task")
assert call.args["title"] == "Fix login bug"
assert call.args["priority"] == "HIGH"
Memory Extraction Verification
# test_memory_extractor.py
async def test_extracts_decision_memory():
"""Verify the memory extractor correctly
identifies a DECISION memory type from
a conversation."""
conversation = [
msg(role="user",
text="Let's use PostgreSQL"),
msg(role="assistant",
text="Agreed. Decision: we will "
"use PostgreSQL 16 with "
"pgvector for the database.")
]
memories = await extract_memories(
conversation
)
assert len(memories) >= 1
decision = find_by_type(
memories, "DECISION"
)
assert decision is not None
assert "PostgreSQL" in decision.content
assert decision.confidence >= 0.8
Meeting Orchestration Tests
- Verify standup produces expected conversation flow (each agent reports)
- Verify action items are extracted as
taskrecords - Verify meeting summary is generated and stored
- Test with 2-agent, 5-agent, and 10-agent meetings
LLM Provider Fallback Tests
- Mock primary provider returning 5 consecutive errors
- Verify circuit breaker opens and system falls back to secondary model
- Verify the agent continues execution with the fallback model
- Verify the original provider is retried after cooldown period
Consumer-Driven Contract Tests (Pact)
Pact contract tests verify that service interfaces remain compatible as each service evolves independently. Contract tests run in CI before deployment — a broken contract blocks the build.
Contract Pairs
- Gateway → qqq: Gateway publishes consumer expectations for all qqq REST API endpoints it calls (agent hydration, task CRUD, search, etc.). qqq verifies provider compatibility.
- Worker → qqq: Worker publishes consumer expectations for qqq endpoints used during task execution, memory storage, and status updates.
- Frontend → Gateway: Frontend publishes expectations for SSE streaming, chat, and task dispatch endpoints.
CI Integration
- Each consumer service publishes its Pact file as a CI artifact
- Provider services verify all published Pacts in their own CI pipeline
- Contract verification runs before deployment — broken contracts block the build
- Pact Broker (self-hosted or Pactflow) stores contract versions and verification results
can-i-deploycheck gates production deploys on contract verification status
LLM Output Test Strategy
LLM output is inherently non-deterministic. This strategy prevents flaky tests while still verifying agent behavior correctness.
| Strategy | Environment | Details |
|---|---|---|
| Deterministic Mocks | CI (every commit) | Recorded LLM responses with temperature=0. Tests run against frozen responses — zero flakiness. All tool call verification and state transition tests use mocks. |
| Behavioral Assertions | CI + Live | Assert on output structure (e.g., “response contains a tool call to create_task”), not exact text. Check required fields, schema compliance, and action correctness. |
| Live LLM Integration | Weekly (scheduled) | End-to-end tests against real LLM providers with temperature=0. Not blocking CI — runs on a weekly schedule. Results tracked in dashboard for trend analysis. |
| Flaky Test Quarantine | CI | After 3 consecutive flaky failures, the test is auto-quarantined (skipped) and a Slack alert notifies the team. Quarantined tests are reviewed weekly and either fixed or removed. |
Load Testing
Load tests verify the system meets SLA targets under concurrent usage. Run weekly and before every production release.
Target SLAs
| Metric | Target |
|---|---|
| API availability | 99.9% |
| Chat time to first token (p95) | <2s |
| Task execution start (p95) | <10s |
| qqq API response time (p95) | <200ms |
| Concurrent WebSocket connections | 500+ |
| Concurrent agent executions | 20+ |
Load Test Configuration (k6)
// load-test.js (k6)
export const options = {
scenarios: {
chat_load: {
executor: "ramping-vus",
startVUs: 0,
stages: [
{ duration: "2m", target: 50 },
{ duration: "5m", target: 100 },
{ duration: "2m", target: 0 },
],
},
api_load: {
executor: "constant-arrival-rate",
rate: 200, // 200 RPS
duration: "5m",
preAllocatedVUs: 50,
},
},
thresholds: {
http_req_duration: ["p(95)<200"],
http_req_failed: ["rate<0.01"],
},
};
Acceptance Criteria
- Every service has unit, integration, and contract tests
- Coverage gates enforced in CI: Python 90%, Java 90%, TypeScript 90%
- Testcontainers used for all integration tests (no mocks for DB/Redis/S3)
- Test data factories exist for every entity type
- Agent behavior tests verify tool call correctness (not LLM text output)
- Memory extraction tests verify all 6 memory types are correctly identified
- Load tests pass SLA targets for API response time and concurrent connections
- Security scans (Semgrep, Trivy, dependency audit) have zero HIGH/CRITICAL findings
- Mutation testing runs weekly with >70% mutation score
- E2E tests cover critical user flows (login, create project, chat with agent, execute task)
- Pact consumer-driven contract tests verify Gateway↔qqq and Worker↔qqq interfaces
- LLM tests use deterministic mocks in CI; live LLM integration tests run weekly (non-blocking)
- Flaky test quarantine auto-skips after 3 consecutive failures with team alert
HIPAA Controls
Cross-CuttingHIPAA compliance is built in from Day 1, not bolted on later.
Praesidium2 operates in a healthcare-adjacent environment (Greater Goods' me-health-portal). While Praesidium2 itself does not store PHI directly, agents MAY interact with systems that contain PHI. Every layer must ensure PHI is never stored, logged, or transmitted to external LLM providers.
Encryption Matrix
Every data layer is encrypted at rest and in transit. No exceptions.
| Layer | At Rest | In Transit | Key Management |
|---|---|---|---|
PostgreSQL (RDS) |
AES-256 (RDS encryption) | TLS 1.3 (pg_hba.conf: hostssl) |
AWS KMS (customer-managed key) |
Redis (ElastiCache) |
AES-256 (ElastiCache encryption) | TLS 1.3 | AWS KMS |
S3 / MinIO |
AES-256 (SSE-KMS for sensitive, SSE-S3 for artifacts) | HTTPS only (deny HTTP via bucket policy) | AWS KMS (per-bucket keys) |
Secrets Manager |
AES-256 (AWS managed) | HTTPS only | AWS KMS (automatic) |
Pod-to-pod |
N/A | mTLS via service mesh or k8s NetworkPolicy + TLS | Cert-manager auto-rotation |
Browser-to-API |
N/A | TLS 1.3 (ALB terminates, internal via mTLS) | ACM certificate (auto-renew) |
EKS Secrets |
AES-256 (etcd encryption via KMS) | TLS 1.3 (kube-apiserver) | AWS KMS envelope encryption |
PHI Detection & Exclusion
The memory extractor and conversation logger must detect and exclude Protected Health Information. Agents using external LLM providers MUST NOT send PHI.
PHI Auto-Scan Patterns
The memory extractor runs a PHI detection pass before storing any memory. Matches trigger exclusion from the memory table.
| Pattern | Regex / Method | Action |
|---|---|---|
| SSN | \d{3}-\d{2}-\d{4} | Exclude from memory |
| Medical Record Number | MRN[-:\s]?\d{6,10} | Exclude from memory |
| Date of Birth | DOB[-:\s]?\d{1,2}/\d{1,2}/\d{2,4} | Exclude from memory |
| Patient Name Context | NER model: PERSON entity near medical terms | Flag for review |
| ICD/CPT Codes with PII | [A-Z]\d{2}\.?\d{0,4} near names | Exclude from memory |
| Health Plan Identifiers | member[-:\s]?ID[-:\s]?\w{8,} | Exclude from memory |
Memory Extractor Safeguards
# Memory extraction system prompt includes: MEMORY_EXTRACTION_RULES = """ Do NOT store any of the following as memories: - Patient names - Medical record numbers (MRNs) - Social Security Numbers (SSNs) - Dates of birth - Individually identifiable health info - Health plan beneficiary numbers - Any combination of name + medical condition If a conversation involves PHI: 1. Extract only procedural/process memories (e.g., "Team prefers async updates") 2. NEVER include patient-specific details 3. Flag the conversation as phi_involved=true """
LLM Provider Routing for PHI
If a conversation is flagged as PHI-adjacent, the system routes to LOCAL or SELF_HOSTED LLM providers only (Ollama/vLLM). External providers (Anthropic, OpenAI) are never sent PHI.
audit_access_log Implementation
Separate from activity_log (operational audit trail), the audit_access_log provides HIPAA-mandated access tracking with 7-year retention.
Table Schema
CREATE TABLE audit_access_log (
id UUID DEFAULT gen_random_uuid()
PRIMARY KEY,
team_member_id UUID NOT NULL
REFERENCES team_member(id),
resource_type VARCHAR(50) NOT NULL,
resource_id UUID NOT NULL,
action VARCHAR(20) NOT NULL,
ip_address VARCHAR(45),
user_agent VARCHAR(500),
justification TEXT,
phi_involved BOOLEAN DEFAULT false,
created_at TIMESTAMPTZ DEFAULT now()
) PARTITION BY RANGE (created_at);
-- Monthly partitions
CREATE TABLE audit_access_log_2026_04
PARTITION OF audit_access_log
FOR VALUES FROM ('2026-04-01')
TO ('2026-05-01');
-- Indexes
CREATE INDEX idx_aal_member
ON audit_access_log (team_member_id);
CREATE INDEX idx_aal_resource
ON audit_access_log (resource_type,
resource_id);
CREATE INDEX idx_aal_phi
ON audit_access_log (phi_involved)
WHERE phi_involved = true;
CREATE INDEX idx_aal_created
ON audit_access_log (created_at);
Partition Strategy
| Partition | Retention | Storage |
|---|---|---|
| Current month | Active | RDS (SSD) |
| Last 12 months | Active | RDS (SSD) |
| 1-7 years old | Archived | S3 Glacier (Parquet export) |
| >7 years | Purged | Secure delete |
What Gets Logged
- READ: Every qqq API GET that returns data (recorded by qqq middleware)
- WRITE: Every POST/PATCH that modifies data
- DELETE: Every DELETE operation
- phi_involved: Set to true when the resource is known to be PHI-adjacent
- Immutable: no UPDATE or DELETE allowed on this table (enforced by qqq config)
Automated Partition Management
A scheduled Liquibase job runs monthly to create the next month's partition and archive partitions older than 12 months to S3 Glacier via pg_dump + Parquet conversion.
Business Associate Agreement (BAA) Tracking
Every integration that could touch PHI needs a BAA. The system tracks BAA status per integration and blocks PHI-adjacent operations without an active BAA.
| Integration | BAA Status | HIPAA Tier | Notes |
|---|---|---|---|
| AWS (RDS, S3, ElastiCache, EKS) | Active | HIPAA Eligible | Standard AWS BAA in place |
| Anthropic | Required | Cloud API | BAA required if PHI context possible; else route PHI to LOCAL models |
| OpenAI | Required | Cloud API | Same as Anthropic |
| Slack | Available | Enterprise Grid | HIPAA-compliant tier available |
| Jira / Confluence | Available | Cloud Enterprise | Atlassian HIPAA compliance package |
| GitHub | Available | Enterprise | GitHub Enterprise offers BAA |
| Google Workspace | Active | Business Plus+ | BAA available on Business Plus and above |
-- changeset praesidium:235-baa-tracking-columns -- Liquibase migration: 200-entity-tables/235-baa-tracking-columns.sql -- Domain 2 (Tools) -- adds BAA tracking to integration_config ALTER TABLE integration_config ADD COLUMN baa_required BOOLEAN DEFAULT false, ADD COLUMN baa_signed_at TIMESTAMPTZ, ADD COLUMN baa_document_url VARCHAR(255), ADD COLUMN baa_counterparty VARCHAR(100); -- baa_required: true if this integration may touch PHI -- baa_signed_at: when the BAA was countersigned -- baa_document_url: link to the signed BAA document -- baa_counterparty: legal entity name on the BAA -- Agent workers check baa_required + baa_signed_at before sending PHI-adjacent data through an integration
Data Retention & Disposal Automation
Retention policies are enforced by scheduled Liquibase-based archival jobs. No manual intervention required.
| Data Type | Retention | Archival Method | Disposal |
|---|---|---|---|
Audit logs (activity_log, audit_access_log) | 7 years | Monthly partition → Parquet → S3 Glacier after 1 year | Secure delete after 7 years |
| Conversation messages | 3 years | Soft delete → hard delete after retention period | Cascading delete of message + tool_calls |
| Agent memories | Until expired or deleted | N/A (active data) | Secure wipe (overwrite embedding + delete record) |
| Artifacts | Project lifetime + 1 year | S3 lifecycle policy | S3 object expiration |
| Knowledge documents | Until manually removed | N/A (active data) | Delete chunks + embeddings + S3 object |
| LangGraph checkpoints | 90 days after workflow completion | Auto-purge scheduled job | Batch DELETE from checkpoint tables |
# Retention automation (scheduled via Celery Beat)
# 1. Archive old audit partitions (monthly)
@celery.task(name="retention.archive_audit_partitions")
async def archive_audit_partitions():
"""Export partitions >12 months to S3 Glacier as Parquet."""
cutoff = datetime.now() - timedelta(days=365)
partitions = get_partitions_before(cutoff)
for partition in partitions:
export_to_parquet(partition, s3_bucket="praesidium-audit-archive")
drop_partition(partition) # only after confirmed S3 upload
# 2. Purge expired conversations (daily)
@celery.task(name="retention.purge_expired_conversations")
async def purge_expired_conversations():
"""Hard-delete soft-deleted messages older than 3 years."""
cutoff = datetime.now() - timedelta(days=3*365)
await qqq_client.delete("message",
filters={"deleted_at__lt": cutoff.isoformat()})
# 3. Purge LangGraph checkpoints (daily)
@celery.task(name="retention.purge_langgraph_checkpoints")
async def purge_langgraph_checkpoints():
"""Delete checkpoints for workflows completed >90 days ago."""
cutoff = datetime.now() - timedelta(days=90)
await qqq_client.delete("langgraph_checkpoint",
filters={"completed_at__lt": cutoff.isoformat()})
Automated HIPAA Control Verification
Compliance tests run as part of CI/CD. These are automated checks that verify HIPAA controls are correctly implemented — not just documented.
| Control | Test | Frequency | Pass Criteria |
|---|---|---|---|
| Encryption at rest (RDS) | AWS API: describe-db-instances → StorageEncrypted: true |
Daily | All RDS instances encrypted |
| Encryption at rest (S3) | AWS API: get-bucket-encryption → SSE-KMS or SSE-S3 |
Daily | All buckets have default encryption |
| Encryption in transit (Redis) | AWS API: describe-replication-groups → TransitEncryptionEnabled: true |
Daily | All Redis clusters enforce TLS |
| Access logging enabled | Verify audit_access_log records exist for test API calls |
Every commit | Every qqq API call generates an audit record |
| PHI exclusion from memory | Feed PHI-containing text to memory extractor, verify no PHI in output | Every commit | Zero PHI patterns in extracted memories |
| MFA enforcement | Authentik API: verify all human users have MFA configured | Daily | 100% MFA coverage |
| Instance isolation (RLS) | Create data in Instance A, attempt read from Instance B token | Every commit | Zero cross-instance data leaks |
| Audit log immutability | Attempt UPDATE/DELETE on audit_access_log → expect 403 |
Every commit | All mutations rejected |
| S3 public access blocked | AWS API: get-public-access-block → all flags true |
Daily | No public access on any bucket |
| Secrets not in logs | Scan last 24h of CloudWatch logs for known secret patterns | Daily | Zero secret values in log output |
# tests/compliance/test_hipaa_controls.py
class TestHIPAAControls:
"""Automated HIPAA control verification tests.
Run in CI and as daily scheduled jobs."""
def test_rds_encryption_at_rest(self, aws_client):
"""Verify all RDS instances have encryption enabled."""
instances = aws_client.describe_db_instances()
for db in instances["DBInstances"]:
assert db["StorageEncrypted"] is True, \
f"RDS instance {db['DBInstanceIdentifier']} is NOT encrypted"
def test_audit_log_immutable(self, qqq_client, admin_token):
"""Verify audit_access_log rejects UPDATE and DELETE."""
record = qqq_client.get("audit_access_log", limit=1)[0]
with pytest.raises(HTTPError, match="403"):
qqq_client.patch(f"audit_access_log/{record['id']}",
{"justification": "tampered"})
with pytest.raises(HTTPError, match="403"):
qqq_client.delete(f"audit_access_log/{record['id']}")
def test_phi_not_in_memory(self, memory_extractor):
"""Verify PHI is excluded from memory extraction."""
conversation = [
{"role": "user", "content":
"Patient John Smith, MRN-123456, DOB 01/15/1980, "
"diagnosed with diabetes. Use Metformin protocol."}
]
memories = memory_extractor.extract(conversation)
for mem in memories:
assert "John Smith" not in mem.content
assert "MRN-123456" not in mem.content
assert "01/15/1980" not in mem.content
def test_cross_instance_isolation(self, qqq_client):
"""Verify Instance A cannot read Instance B data."""
# Create project in Instance A
project = qqq_client.post("project",
{"name": "Secret Project"},
token=instance_a_token)
# Attempt read from Instance B
results = qqq_client.get("project",
filters={"id": project["id"]},
token=instance_b_token)
assert len(results) == 0, "Cross-instance data leak detected"
Access Control Summary
| Control | Implementation | Enforcement Point |
|---|---|---|
| Authentication | Authentik OIDC (JWT) | qqq ProdAuthenticationCustomizer |
| Authorization (RBAC) | IAM-style policies in JWT claims | qqq table permissions |
| Row-level security | RecordSecurityLock on instance_id | qqq query filter (automatic) |
| MFA | Authentik TOTP / WebAuthn | Authentik login flow |
| Session management | 15 min access token, 7 day refresh with rotation | Authentik token endpoint |
| Audit trail | activity_log + audit_access_log | qqq middleware (immutable, append-only) |
| Default deny | No permissions unless explicitly granted via policy | ProdAuthenticationCustomizer |
| Least privilege | Scoped worker service account, role-based user policies | Authentik group attributes |
Acceptance Criteria
- All data encrypted at rest (RDS, ElastiCache, S3, EKS Secrets) using AES-256
- All data encrypted in transit (TLS 1.3 everywhere, no plaintext connections)
- PHI detection patterns prevent PHI from being stored in agent memories
- PHI-adjacent conversations routed to LOCAL LLM providers only
audit_access_logrecords every data access, partitioned by month, 7-year retention- BAA status tracked per integration, PHI operations blocked without active BAA
- Data retention automation running: audit archival, conversation purge, checkpoint cleanup
- Automated compliance tests pass in CI: encryption, access logging, PHI exclusion, RLS isolation
- MFA enforced for 100% of human users
- All AWS services used are HIPAA-eligible
Review Summary
Findings by Severity
The implementation plan is comprehensive and ready for Claude Code agents to begin work. It provides file trees, acceptance criteria, CLAUDE.md rules, day-by-day tasks, and detailed code samples for every service. All 2 blockers resolved in v1.8. 7 gaps resolved via Gemini spec integration (AGENT_RUNTIME_SPEC, HITL_ORCHESTRATION_SPEC, DATADOG_INTEGRATION_SPEC, IMPLEMENTATION_FIX_PLAN). 1 additional gap resolved in v1.9 (S-Q1: Java coverage aligned to 90%). 3 additional gaps resolved in v1.10 (GAP-TR3: Python-only Tool Runner, GAP-AWS1: IaC templates, GAP-H1: BAA migration changeset). 10 additional gaps resolved in v1.11 (shared Python packages, health check standardization, frontend client paths, tool execution modes, Faber config, Authentik scope merge, contract tests, LLM test strategy). 3 remaining gaps are minor consistency issues.
Per-Section Scores
| Section | Score | Blockers | Gaps | Suggestions | Status |
|---|---|---|---|---|---|
| qqq Data Layer | 9/10 | 0 | 3 | 1 | Ready |
| Agent Gateway | 10/10 | 0 | 0 | 1 | Ready |
| Agent Workers | 9/10 | 0 | 1 | 1 | Ready |
| Tool Runner | 10/10 | 0 | 0 | 1 | Ready |
| Frontend SPA | 9/10 | 0 | 0 | 2 | Ready |
| Local Dev / k8s | 9/10 | 0 | 0 | 1 | Ready |
| CI/CD (Faber) | 10/10 | 0 | 0 | 1 | Ready |
| ArgoCD | 9/10 | 0 | 0 | 1 | Ready |
| AWS Setup | 9/10 | 0 | 1 | 1 | Ready |
| Authentication | 10/10 | 0 | 0 | 1 | Ready |
| Testing Strategy | 10/10 | 0 | 0 | 1 | Ready |
| HIPAA Controls | 10/10 | 0 | 0 | 1 | Ready |
Blocking Issues (All Resolved)
The secret_ref scope hierarchy was inconsistent between Worker and Tool Runner sections. RESOLVED: Authoritative hierarchy is now AGENT > TOOL > PROJECT > INSTANCE (narrowest scope wins) consistently across Worker day plan, Worker acceptance criteria, and Tool Runner credential injection flow.
The RESOLVED: audit_access_log table was defined in HIPAA Controls but missing from qqq data layer.audit_access_log added to Domain 6 Operations in the Table Implementation Order and 206-audit-access-log.sql added to the 200-entity-tables/ Liquibase changelog structure.
Gaps (3 open / 7 resolved in v1.8 / 1 resolved in v1.9 / 3 resolved in v1.10 / 10 resolved in v1.11)
| ID | Section | Finding |
|---|---|---|
| GAP-Q1 | qqq | llm_provider and llm_model table schemas.provider_type, api_base_url, context_window, cost_per_1m_tokens columns. |
| GAP-Q2 | qqq | secret_ref table schema. |
| GAP-Q3 | qqq | llm_provider and secret_ref listed in root tables but missing from entity table implementation order. |
| GAP-GW1 | Gateway | llm/ package but no plan for how to share it (monorepo, published package, symlink).shared/ directory contains llm/ and hydration/ packages. Both Gateway and Worker install via pip install -e ../shared in Docker builds. Post-v1: publish to private PyPI. |
| GAP-GW2 | Gateway | /healthz in file tree, /health in docker-compose, /health//ready in k8s manifests./healthz (liveness) and /readyz (readiness) across k8s manifests and ArgoCD health checks. Docker-compose removed in v1.12. |
| GAP-W1 | Workers | services/hydration.py says “same as gateway” but no sharing mechanism specified.shared/hydration/ package installed by both Gateway and Worker via pip install -e. See “Shared Python Packages” section in Gateway tab. |
| GAP-W2 | Workers | GET /api/internal/worker/hydrate/{agent_id} endpoint added to Gateway. Auth via Service Account JWT (issued by Authentik, scoped to Agent’s Instance ID). See AGENT_RUNTIME_SPEC. |
| GAP-W3 | Workers | Celery Beat leader election has minimal detail (just “Redis SET NX EX with 30s TTL”). Insufficient for autonomous implementation. |
| GAP-FE1 | Frontend | POST /chat) don’t match gateway router paths (POST /api/chat/{agent_id}). Path parameters missing from client spec.gateway-client.ts updated to use correct /api/ prefix and path parameters: POST /api/chat/{agent_id}, POST /api/task/execute, GET /api/stream/{id}. |
| GAP-FE2 | Frontend | GET /api/search endpoint added to Gateway. Uses pgvector semantic search across projects, agents, tasks, conversations, and tools. Command Palette (Cmd+K) wired to this backend. |
| GAP-FE3 | Frontend | Type table list in Zustand store callout lists 59 names which matches the 59 type tables in the implementation plan. RESOLVED. |
| GAP-TR1 | Tool Runner | --branch=${GIT_SHA} for git clone.git clone --no-checkout followed by git checkout ${GIT_SHA} (A-004 fix). |
| GAP-TR2 | Tool Runner | execution_mode field added to tool table. Default: ephemeral (sandboxed k8s Job). warm_pool only for read-only tools with no side effects. Decision made at tool registration time. See “Execution Mode Decision Logic” in Tool Runner tab. |
| GAP-TR3 | Tool Runner | dist/runner.js but no TypeScript runner implementation is described anywhere in the plan.Dockerfile.typescript removed from file tree and Dockerfile section. The Python sandbox (runner.py + Dockerfile.python) is the sole execution environment for K8s Jobs. |
| GAP-D1 | Local Dev | *authentik-env YAML anchor but the anchor is never defined.&authentik-env anchor fixed. SUPERSEDED (v1.12): docker-compose.yml removed entirely. Local dev now uses Tilt + Helm charts deploying to Docker Desktop k8s. Gap is moot. |
| GAP-D2 | Local Dev | /health but gateway defines /healthz (liveness) and /readyz (readiness). No /health endpoint exists./healthz (liveness) and /readyz (readiness). SUPERSEDED (v1.12): docker-compose removed; health checks defined in Helm chart k8s probes only. |
| GAP-D3 | Local Dev | qqq-api dependency.resource_deps in the Tiltfile. |
| GAP-CI1 | CI/CD | .faber.yml configuration template added to CI/CD tab. Defines terraform (OpenTofu) and docker service pipelines for all 4 services. |
| GAP-AWS1 | AWS | |
| GAP-AWS2 | AWS | AWS Setup Epic badge says “Day 19” but the timeline shows deployment on Days 13-14. Inconsistent. |
| GAP-AUTH1 | Auth | scope_name: praesidium. May conflict — should be merged into a single mapping.praesidium-unified-scope mapping that returns both praesidiumInstanceId and policies claims in one scope expression. |
| GAP-T1 | Testing | can-i-deploy gating. |
| GAP-T2 | Testing | temperature=0 recorded responses) for CI. Behavioral assertions on output structure, not exact text. Weekly live LLM integration tests (non-blocking). Flaky test quarantine after 3 consecutive failures. |
| GAP-H1 | HIPAA | ALTER TABLE integration_config) not included in the qqq Liquibase changelog structure. Needs a migration changeset.235-baa-tracking-columns.sql added to Domain 2 (Tools) in the entity table changelog. Adds baa_required, baa_signed_at, baa_document_url, baa_counterparty columns to integration_config. |
Suggestions (13 findings)
| ID | Section | Suggestion |
|---|---|---|
| S-Q1 | qqq | |
| S-GW1 | Gateway | Specify rate limiter algorithm (token bucket vs sliding window) for per-agent per-tool limits. |
| S-W1 | Workers | Add brief interface spec per integration tool (Jira, Slack, GitHub, etc.) to accelerate implementation. |
| S-TR1 | Tool Runner | Add FastAPI to warm pool dependencies or specify alternative HTTP framework for http_server.py. |
| S-FE1 | Frontend | Define React Error Boundary pattern, fallback UI components, and error reporting strategy. |
| S-FE2 | Frontend | Align Node.js version in Dockerfiles: node:22-slim in repo setup vs node:20-alpine in Local Dev tab. |
| S-D1 | Local Dev | Provide MinIO bucket auto-creation init script for local dev environment. |
| S-CI1 | CI/CD | Align Java coverage gate to 90% to match Python/TypeScript and quality target of 90%+ across all packages. |
| S-A1 | ArgoCD | Clarify whether praesidium-cd.git is the same repo as Helm charts or a separate GitOps repository. |
| S-AWS1 | AWS | Note RDS (db.r6g.large) and ElastiCache instance sizes as right-sizing candidates after initial load testing. |
| S-AUTH1 | Auth | Define automated bootstrap procedure for first admin user and first instance provisioning. |
| S-T1 | Testing | Separate internal latency (hydration + queue) from total latency (including LLM provider) in chat TTFT SLA. |
| S-H1 | HIPAA | Add NER dependency (spaCy or similar) to worker pyproject.toml for PHI detection “PERSON entity near medical terms.” |
Design Spec Cross-Reference
The implementation plan was cross-referenced against the full design spec (2026-04-02-praesidium-v2-design.md). Key alignment checks:
| Design Spec Element | Implementation Plan | Status |
|---|---|---|
| 5 deployable services (qqq, gateway, workers, tool-runner, web) | All 5 services have dedicated tabs with file trees, endpoints, acceptance criteria | Aligned |
| 58 entity tables + 59 type tables = 117 total | 117 tables including 10 business ops tables + 4 new tables (llm_provider, llm_model, secret_ref, audit_access_log) + scheduler_heartbeat (v1.9). 59 type tables with changesets 100-159. | Aligned |
| Anthropic SDK + LangGraph (scoped) architecture | Worker tab defines direct SDK agent loops for tasks/meetings, LangGraph for durable/HITL workflows (jobs/forge) with correct state graphs | Aligned |
| Authentik + qqq auth pattern (same as me-health-portal) | Auth tab provides full blueprint, ProdAuthenticationCustomizer, IAM policies, and RecordSecurityLock config | Aligned |
| 3 Celery queues (quick/standard/long) | Exact match: quick 300s, standard 3600s, long 7200s with acks_late configuration | Aligned |
| Memory system (extract, embed, deduplicate, store, recall) | Worker tab provides full memory extractor pseudocode with embedding + cosine similarity dedup | Aligned |
| Tool safety (LOW/MEDIUM/HIGH/CRITICAL risk levels) | Worker tab defines risk-level gates with exact matching behavior per level | Aligned |
| Frontend: 11 pages with React 19 + TanStack | Frontend tab defines all 11 pages (including Schedules v1.9) with 5-phase implementation order and component library | Aligned |
| Per-instance isolation via k8s namespaces | Local Dev tab + ArgoCD tab define ApplicationSet per instance with namespace isolation | Aligned |
| HIPAA controls (encryption, PHI detection, audit logging) | HIPAA tab provides encryption matrix, PHI patterns, audit_access_log schema, automated compliance tests | Aligned |
| llm_provider / llm_model tables | Schemas defined in AGENT_RUNTIME_SPEC. Added to Domain 1 in table implementation order (v1.8). | Aligned |
| secret_ref table | Schema defined in AGENT_RUNTIME_SPEC. Added to Domain 1 in table implementation order (v1.8). | Aligned |
| HITL Orchestration (interrupt/resume) | HITL_ORCHESTRATION_SPEC integrated: task_approval table, interrupt() logic, resume protocol, action hash verification | Aligned |
| Datadog Unified Tracing (5-service waterfall) | DATADOG_INTEGRATION_SPEC integrated: RUM, ddtrace, DD_TRACE_ID env vars, DogStatsD custom metrics, SDS PHI redaction | Aligned |
| Re-sequenced timeline (Hands before Brain) | IMPLEMENTATION_FIX_PLAN integrated: Tool Runner Phase 2, Agent Workers Phase 3, 7-phase build plan | Aligned |
Review Metadata
Reviewed: 2026-04-03 • Reviewer: Claude Opus 4 (automated review) • Document: implementation.html (~7059 lines) • Cross-reference: 2026-04-02-praesidium-v2-design.md
Business Operations
V1 — Phases 5-7Operational readiness features required for real business deployment.
These features bridge the gap between a working AI platform and a production system that business teams can trust. Agent Output Approval ensures humans stay in the loop. Success Metrics prove ROI. Human Feedback enables continuous improvement. Rollback/Recovery provides safety nets. Phased Rollout manages deployment risk.
Agent Output Approval Workflow
Ensures high-risk agent actions require human approval before execution. Integrates with the existing risk-level safety gates and extends them with a persistent approval queue.
Database
| Table | Key Columns | Purpose |
|---|---|---|
task_approval_queue |
id, agent_id, action_type, action_payload (JSONB), risk_level_id, status_id (PENDING/APPROVED/REJECTED/EXPIRED), reviewer_id, reviewed_at, expires_at, created_at |
Persistent queue of agent actions awaiting human review |
approval_policy |
id, action_pattern, risk_threshold, auto_approve (boolean), required_approvers, timeout_minutes, escalation_policy |
Configurable policies controlling which actions need approval |
Backend Components
- Policy Engine: Evaluates each agent action against
approval_policyrules. AUTO_APPROVE for LOW risk, REQUIRE_APPROVAL for HIGH/CRITICAL. Configurable per-team and per-action-type. - Approval API:
GET /api/approvals(pending list),POST /api/approvals/{id}/approve,POST /api/approvals/{id}/reject,POST /api/approvals/{id}/request-changes - Notification Pipeline: Slack DM to designated approver on new pending item. Escalation after timeout (configurable, default 30 min).
- Execution Resume: On approval, the queued action is dispatched to the appropriate Celery queue for execution. On rejection, the agent is notified and the task status updated.
Frontend Components
- Approval Dashboard: (
/approvals) Pending items sorted by urgency/age. Action preview with diff view for data mutations. Approve/Reject/Request Changes buttons. - Sidebar Badge: Real-time count of pending approvals via WebSocket. Notification dot on sidebar icon.
- Inline Approval: Approval prompts embedded in chat conversation view when agent hits a HIGH/CRITICAL action.
Success Metrics / KPIs
Baseline capture begins in the Pre-Build phase with a 2-week time-tracking survey, then continues with automated measurement to prove AI augmentation ROI.
Pre-Build Baseline
- Time-Tracking Survey: 2-week survey where each team lead logs daily hours on tasks agents will handle (status reporting, Jira grooming, Slack triage, meeting scheduling, document creation, code review coordination).
- Historical Data Pull: Automated extraction from Jira, GitHub, and Slack APIs for quantitative baselines: average task completion time, throughput per team member, PR cycle time, error/rework rates, meeting hours per week.
- Success Targets: Stakeholder-agreed measurable targets (e.g., 30% reduction in status reporting time, 50% faster Jira triage).
Baseline Import (Phase 5)
- Data Import: Pre-Build survey data loaded into
kpi_baselinetable for automated comparison. - Storage:
kpi_baselinetable withmetric_name,team_id,baseline_value,captured_at,data_source(SURVEY | JIRA_API | GITHUB_API | SLACK_API).
KPI Measurement Service
- Scheduled Job: Celery Beat task running daily, computing current values for all tracked KPIs from task/agent/activity data.
- Tracked Metrics: Tasks completed per agent per day, average task duration, human intervention rate, approval turnaround time, cost per task, agent quality score (from feedback).
- Storage:
kpi_measurementtable withmetric_name,team_id,value,period_start,period_end,comparison_to_baseline(percentage).
Dashboard Component
- KPI Dashboard: Recharts line/bar charts showing metric trends over time with baseline comparison line.
- Comparison View: Before/after cards showing percentage improvement vs baseline for each tracked metric.
- Export: CSV export for executive reporting.
Human Feedback
Structured feedback collection on agent outputs to drive continuous improvement and compute agent quality scores.
Frontend Widget
- Feedback Widget: Thumbs up/down + optional comment on every agent output (task result, chat response, approval action). Appears inline in chat and on task detail pages.
- Rating Categories: Accuracy, Helpfulness, Timeliness, Safety (optional drill-down).
Backend
- Storage:
output_feedbacktable withid,agent_id,task_id,conversation_id,rating(1-5),thumbs(UP/DOWN),comment,category,reviewer_id,created_at. - Aggregation Service: Celery Beat job computing rolling quality scores per agent (weighted average of last 30 days of feedback). Updates
agent.quality_score. - Quality Score:
quality_score = (0.4 * accuracy + 0.3 * helpfulness + 0.2 * timeliness + 0.1 * safety), normalized to 0-100. Agents below 70 trigger alerts.
Rollback / Recovery
Safety nets for when agent actions need to be undone. Every mutating action is logged with enough context to reverse it.
Action Logging Middleware
- Middleware: Wraps all agent tool executions. Logs action type, input, output, and “undo payload” (what is needed to reverse the action) to
agent_action_logtable. - Undo Payloads: For Jira: store previous status/fields. For Slack: store message timestamp for deletion. For GitHub: store PR number for close. For internal: store previous record state.
Undo API Endpoints
POST /api/actions/{id}/undo— Reverses a single logged action using its undo payload.POST /api/tasks/{id}/rollback— Rolls back all actions from a task execution in reverse order.- Limitations: Some actions are irreversible (sent Slack messages can be deleted but recipients may have already read them). UI shows reversibility status.
Git Rollback Pull Request Protocol v1.9
- Protected Branch Rule: For Git-based tools (
git_merge,github_push_files,github_create_pr),revert()MUST NOT perform a directgit reverton protected branches. - PR Flow: When
tool.requires_review = true,revert()creates a branchpraesidium-rollback/{execution_id}, commits revert changes, and opens a Revert Pull Request tagged[Praesidium-Rollback]. - PR Body: Includes original action log ID, rollback reason, diff summary, and link to Praesidium audit trail.
- Schema:
ALTER TABLE tool ADD COLUMN requires_review BOOLEAN DEFAULT false;— all Git-based tools ship withrequires_review = trueby default. - Fallback: Tools with
requires_review = falsecontinue to use the standard directrevert()flow.
Emergency Kill Switch
- Admin Panel Control: Red button in Settings page that immediately: (1) revokes all active Celery tasks for the instance, (2) sets all agents to PAUSED status, (3) sends Slack alert to all admins.
- API:
POST /api/admin/emergency-stopwith confirmation token. Requires ADMIN role. - Recovery:
POST /api/admin/resumere-enables agents one-by-one with manual approval.
Agent Performance Management
Manages agent lifecycle states based on performance metrics. Poor-performing agents are automatically flagged, placed on probation, and suspended if issues persist.
Agent Lifecycle States
| State | Description | Transition Trigger |
|---|---|---|
| Active | Agent is operating normally | Default state; restored after successful probation review |
| Probation | Agent flagged for review; limited autonomy | Any auto-suspension trigger fires |
| Suspended | Agent halted; no task execution | Probation conditions not improved within 48 hours |
| Terminated | Agent permanently deactivated | Manual admin action after sustained failure |
Auto-Suspension Triggers
- Low Feedback Score: Rolling 30-day average feedback score < 2.5/5 (from
output_feedbacktable) - High Error Rate: Error rate > 25% over last 50 tasks (computed from task execution outcomes)
- Excessive Escalations: 3+ escalations in a single day (from
escalationtable)
Performance Review Dashboard
- Agent Health Card: Per-agent card showing lifecycle state, quality score trend, error rate, escalation count, and feedback distribution.
- Performance History: Time-series chart of
agent_quality_scorewith threshold lines for probation/suspension triggers. - Admin Actions: Buttons to manually promote (Probation → Active), suspend, or terminate agents. Requires ADMIN role.
- Stored in:
agent_quality_scoretable withagent_id,score,accuracy,helpfulness,timeliness,safety,error_rate,escalation_count,period_start,period_end,computed_at.
Configurable Agent Personality
Agents can adopt different communication personalities to make interactions more engaging for internal teams while maintaining professionalism for external-facing outputs.
Database
- Schema:
personality_traitsJSONB field onagent_templatetable. Stores personality mode, tone modifiers, and custom prompt fragments. - Structure:
{"mode": "WWE", "tone": "enthusiastic", "custom_prompt": "...", "external_override": true}
Personality Modes
| Mode | Description | Example Tone |
|---|---|---|
Professional | Default corporate tone (always used for external outputs) | “Task completed. Status report attached.” |
WWE | Wrestling-inspired hype and energy | “OH YEAH! That PR just got BODY-SLAMMED into main!” |
LOTR | Lord of the Rings / fantasy epic | “The deployment quest is complete, the pipeline holds strong.” |
GreysAnatomy | Medical drama emotional intensity | “We’re losing this build—PUSH! Clear! ...it’s alive!” |
StarWars | Star Wars saga references | “The force is strong with this commit. Deploy we must.” |
Custom | User-defined custom prompt fragment | User-provided personality instructions |
Frontend Components
- Settings Page: Personality mode selector (radio group) per agent template in Settings → Agent Templates.
- Custom Mode Editor: Textarea for custom prompt fragment (shown when mode = Custom).
- External Override Toggle: Checkbox that forces Professional mode for all external-facing outputs (Slack channels, Jira comments, client emails). Enabled by default.
Backend Integration
- Hydration: Agent hydration service reads
personality_traitsfrom template and appends personality instructions to the system prompt. - External Override: When
external_override: trueand the output target is an external integration (Slack channel, Jira, etc.), the personality is forced to Professional regardless of template setting.
Business-Value Kill Criteria
Automated evaluation gates that determine whether agent deployment is delivering measurable business value. If gates fail, the system recommends scaling back or terminating the deployment.
Kill Criteria Gates
| Gate | Time Saved | Feedback Score | Utilization | Action on Fail |
|---|---|---|---|---|
| 30-Day Gate | ≥ 5 hrs/week saved | > 3.5 / 5 | > 60% utilization | Alert CTO + team leads; begin remediation plan |
| 60-Day Gate | ≥ 8 hrs/week saved | > 4.0 / 5 | > 75% utilization | Executive review; scale-back or terminate deployment |
Kill Criteria Evaluation Dashboard
- Data Source: Reads from
kpi_measurementandkpi_baselinetables to compute time-saved, feedback averages, and utilization rates. - Dashboard Component: Card per gate showing current values vs thresholds with pass/fail indicators. Trend chart showing trajectory toward or away from gates.
- Automated Gate Check: Celery Beat job runs weekly, evaluating all active teams against their applicable gate. On failure, sends email notification to CTO + team leads via integration notification pipeline.
- Gate Status API:
GET /api/admin/kill-criteriareturns current gate evaluations per team.POST /api/admin/kill-criteria/evaluatetriggers manual re-evaluation.
User Training & Onboarding
Structured onboarding program to ensure each team can effectively use the AI workforce platform from day one.
Per-Role Quickstart Guides
- Delivery Format: Auto-generated Confluence pages (one per role: Engineering Lead, Marketing Manager, Operations Manager, Finance Analyst, Team Member).
- Content: Role-specific workflows, common agent interactions, approval workflow guide, KPI interpretation, FAQ.
- Generation: Template-based generation using team-specific agent configuration + integration setup. Triggered during team provisioning.
30-Minute Team Walkthrough Template
- Structure: Slide deck template covering: Platform overview (5 min), Live demo: chat with agent (10 min), Approval workflow demo (5 min), KPI dashboard walkthrough (5 min), Q&A (5 min).
- Customization: Template auto-populates with team-specific agent names, integration status, and configured workflows.
In-App First-Time User Onboarding
- Tooltip Tour: Step-by-step guided tour using a tooltip overlay (e.g., react-joyride) that highlights: sidebar navigation, dashboard metrics, how to start a chat, how to review approvals, where to find KPIs.
- Progress Tracking: Onboarding progress stored per user in
user_preferences(JSONB field). Tour auto-launches on first login, can be replayed from Settings. - Completion Badge: “Onboarding Complete” badge shown in user profile after finishing the tour.
“Agent Did Something Wrong” FAQ
- Location: Accessible from Help menu in header and linked from every agent output’s feedback widget.
- Content: Step-by-step guides for: “How to undo an agent action”, “How to report bad output”, “How to adjust agent permissions”, “How to escalate to a human”, “Emergency kill switch location”.
- Format: In-app help page with collapsible FAQ sections. Also exported to Confluence for offline reference.
Phased Rollout
Structured deployment process to manage risk. Engineering first (Phase 5), Marketing second (Phase 6), then additional teams in V1.1+.
Deployment Checklist
| Phase | Go/No-Go Criteria | Rollback Trigger |
|---|---|---|
| Staging | All E2E tests pass. Security scan clean. Performance benchmarks met. | Any critical test failure. |
| Engineering | Staging stable for 48h. KPI baseline captured. Kill switch tested. Approval workflow validated. | Critical bug affecting data integrity. Agent taking unauthorized actions. |
| Marketing | Engineering stable for 1 week. No critical bugs. Positive feedback (quality score > 75). KPI trending positive. | Quality score drops below 60. Critical integration failure. |
| Additional Teams | Two teams stable. Approval workflow scaled. Cost per task within budget. V1.1 integrations available. | Per-team evaluation using same criteria. |
Team Provisioning Scripts
- Provisioning Script: Creates team instance with: Authentik group + policies, default agent templates for team type (engineering/marketing/ops), integration configs (Slack channel, Jira project), KPI baseline job trigger.
- Validation Script: Post-provisioning smoke test: verifies auth flow, API access, agent execution, integration connectivity, approval workflow.
- Teardown Script: Clean removal of team instance if rollback needed. Archives data, removes Authentik group, pauses agents.
Day-by-Day Tasks (Business Operations Track)
Business operations features are built in Phases 4-5 by Backend Track and Frontend Track, alongside integration and deployment work.
| Period | Backend Track | Frontend Track |
|---|---|---|
| P4 D16-17 | task_approval_queue + approval_policy tables via qqq. Policy engine (auto-approve LOW risk, require approval for HIGH/CRITICAL). Approval API endpoints: GET /api/approvals, POST /api/approvals/{id}/approve, POST /api/approvals/{id}/reject. |
Approval dashboard page (/approvals): pending items list, action preview with diff view, approve/reject/request-changes buttons. Approval history view. |
| P4 D18-19 | Slack DM notification pipeline for pending approvals. Approval timeout + escalation logic (configurable, default 30 min). Approval notification via WebSocket for real-time sidebar badge. | Approval notification indicators in sidebar (real-time count badge via WebSocket). Inline approval prompts in chat conversation view for HIGH/CRITICAL actions. |
| P5 D21-22 | Rollback/recovery: agent_action_log + rollback_record tables. Undo API endpoints (POST /api/actions/{id}/undo, POST /api/tasks/{id}/rollback). Git Rollback PR protocol: when tool.requires_review = true, revert() creates a [Praesidium-Rollback] tagged PR instead of direct git revert. Emergency kill switch API (POST /api/admin/emergency-stop, POST /api/admin/resume). output_feedback collection endpoint. |
Human Feedback widget (thumbs up/down + comment on every agent output). Rollback controls in task detail UI. Emergency kill switch button in admin panel (Settings page). |
| P5 D23-24 | KPI measurement service: Celery Beat daily job computing metrics from task/agent data. kpi_measurement + kpi_baseline tables. agent_quality_score computation (rolling 30-day weighted average). Kill criteria gate evaluation job. Agent performance lifecycle state machine. |
KPI dashboard page (/kpis): baseline comparison charts (Recharts), before/after improvement cards, CSV export. Performance management UI: agent health cards, quality score trends, admin action buttons. Kill criteria gate status cards. |
| P5 D25 | Personality traits integration in agent hydration. Kill criteria notification pipeline. Quickstart guide generation (Confluence template). End-to-end business ops testing. | Personality mode selector in agent template settings. In-app onboarding tooltip tour (react-joyride). “Agent did something wrong” FAQ help page. Onboarding progress tracking. |
Acceptance Criteria
- Approval workflow captures HIGH/CRITICAL actions and blocks until human review
- Approval dashboard shows pending items with real-time count updates
- KPI baseline captured for Engineering team before AI deployment
- KPI dashboard shows trend comparison vs baseline
- Human feedback widget appears on all agent outputs
- Agent quality scores computed and updated daily
- Action logging captures undo payloads for all mutating tool executions
- Undo API successfully reverses Jira, Slack, and GitHub actions
- Emergency kill switch stops all agent activity within 30 seconds
- Team provisioning script creates fully functional team instance
- Engineering team deployed and stable for 1 week before Marketing rollout
- Go/no-go criteria documented and enforced at each deployment phase
- Agent performance management: lifecycle state transitions working (Active → Probation → Suspended)
- Auto-suspension triggers fire correctly for low feedback, high error rate, and excessive escalations
- Performance review dashboard shows agent health cards with quality score trends
- Configurable personality modes selectable per agent template (6 modes + Custom)
- External-facing outputs forced to Professional mode when override enabled
- Business-value kill criteria gates evaluated automatically at 30-day and 60-day marks
- Kill criteria gate failure sends email notification to CTO + team leads
- Per-role quickstart guides generated during team provisioning
- In-app tooltip onboarding tour launches for first-time users
- “Agent did something wrong” FAQ page accessible from help menu and feedback widget
Project Setup
Pre-BuildEverything here must be done BEFORE Phase 1 starts.
This is the complete project infrastructure plan: GitHub repositories, Jira project, Confluence space, roadmap, and the day-by-day pre-build schedule. All items are blockers for the 7-phase build clock.
1. GitHub Repositories
Decision: Multi-repo with one shared-packages exception. Each of the 4 service repos (plus infra and CD) deploys on its own cadence. Separate repos give each service its own CI pipeline, release tags, CLAUDE.md, and .faber.yml. Each service repo owns its own Dockerfile and docker/compose.yml for local dev. Helm charts live in praesidium-cd. The exception: Gateway and Worker share praesidium-llm and praesidium-hydration packages via a lightweight Python monorepo.
| # | Repository | Language | Description | GHCR Image |
|---|---|---|---|---|
| 1 | praesidium2-qqq | Java 21 | qqq data layer — 117 tables, REST API, Liquibase migrations, RLS. Owns its Dockerfile + .faber.yml (java-webapp) | ghcr.io/dmdbrands/praesidium/qqq-api |
| 2 | praesidium2-python | Python 3.12 | Monorepo: agent-gateway (FastAPI), agent-worker (Celery), shared packages (llm, hydration). Owns its Dockerfiles + .faber.yml (python-webapp) | ghcr.io/dmdbrands/praesidium/gateway, ghcr.io/dmdbrands/praesidium/worker |
| 3 | praesidium2-tool-runner | Python 3.12 | K8s Job sandbox for custom tool execution. Minimal deps (pydantic, httpx only). Owns its Dockerfile + .faber.yml (python-webapp) | ghcr.io/dmdbrands/praesidium/tool-runner |
| 4 | praesidium2-web | TypeScript | React 19 SPA — TanStack Router/Query, Zustand, shadcn/ui, Tailwind. Owns its Dockerfile + .faber.yml (node-webapp) | ghcr.io/dmdbrands/praesidium/web |
| 5 | praesidium-infrastructure | HCL | OpenTofu/Terragrunt — AWS IaC (EKS, RDS, ElastiCache, Secrets Manager) | N/A |
| 6 | praesidium-cd | YAML | ArgoCD ApplicationSets, Helm charts, Helm values per environment, instance YAML | N/A |
Repos 5 & 6 already exist and are NOT owned by the Praesidium2 team.
Do not modify them directly. Infrastructure and CD changes go through the existing change request process. Helm charts and ArgoCD config live in praesidium-cd.
Python Monorepo Structure
praesidium2-python/
CLAUDE.md # Root-level rules (applies to both services)
shared/
llm/
__init__.py
protocol.py # LLMClient Protocol interface
anthropic_client.py # V1 implementation
factory.py # build_llm_client() by provider type
pyproject.toml # name = "praesidium-llm"
hydration/
__init__.py
hydrator.py # Agent hydration: qqq fetch + Redis cache
models.py # HydratedAgent, HydrationContext
pyproject.toml # name = "praesidium-hydration"
services/
agent-gateway/
CLAUDE.md # Gateway-specific rules
pyproject.toml
src/
tests/
agent-worker/
CLAUDE.md # Worker-specific rules
pyproject.toml
src/
tests/
Branch Strategy
| Branch | Purpose | Protection | Deploy Target |
|---|---|---|---|
main | Production releases | 1 approval, passing CI, no force push, signed commits | Production (via ArgoCD) |
develop | Integration branch | 1 approval, passing CI, no force push | Staging (via ArgoCD) |
feature/* | Feature work | None (CI runs on push) | Local dev only |
bugfix/* | Bug fixes | None (CI runs on push) | Local dev only |
hotfix/* | Production hotfixes | Branch from main, merge to main AND develop | Production (expedited) |
Flow: feature/* → develop (PR) → main (PR, release)
Naming: feature/PR2-123-short-description, bugfix/PR2-456-fix-something, hotfix/PR2-789-critical-fix
PR Review Requirements
| Rule | Setting |
|---|---|
| Minimum approvals | 1 (human engineer) |
| Dismiss stale reviews on new push | Yes |
| Require CI to pass | Yes (Faber: lint + test + build) |
| Require up-to-date branch | Yes (rebase before merge) |
| Allow squash merge | Yes (default) |
| Allow merge commits | No |
| Allow rebase merge | Yes |
| Require signed commits | Yes (on main only) |
| CODEOWNERS required review | Yes |
| Auto-delete head branches | Yes |
CLAUDE.md per Repo
| Repository | CLAUDE.md Key Rules |
|---|---|
praesidium2-qqq | qqq conventions, table definitions in QqqMetaDataProvider.java, Liquibase SQL migrations, RecordSecurityLock on instance_id, Testcontainers for integration tests, 90%+ coverage |
praesidium2-python (root) | Cross-service rules: all data via qqq REST API (never touch DB directly), shared LLMClient abstraction, structlog for logging, Pydantic models at all boundaries |
agent-gateway/ | SSE streaming, WebSocket pub/sub, JWT auth on all routes except /healthz, agent hydration with Redis cache (5min TTL), async/await everywhere, httpx.AsyncClient |
agent-worker/ | Direct Anthropic SDK for “WHO does WHAT”, LangGraph for “HOW complex work flows” (durable/HITL only), HITL interrupt/resume, acks_late=True, tool safety rules |
praesidium2-tool-runner | Minimal deps (pydantic + httpx), non-root container, read-only rootfs, NetworkPolicy egress restrictions, “Blind” Secret Pipeline |
praesidium2-web | React 19, TanStack Router + Query, Zustand for client state only, JWT in memory only, shadcn/ui + Tailwind, pnpm, Vitest + Testing Library + MSW, 90%+ coverage |
2. Jira Project
| Setting | Value |
|---|---|
| Project Name | Praesidium2 |
| Project Key | PR2 |
| Project Type | Software (Scrum) |
| Project Lead | James Maes |
| Board Type | Scrum |
| Sprint Duration | 1 week (Monday to Friday) |
| Estimation | Story Points (Fibonacci: 1, 2, 3, 5, 8, 13) |
10 Epics Mapped to Build Phases
| Epic | Epic Name | Timeline | Key Deliverables |
|---|---|---|---|
| E1 | Foundations | Phase 1 | qqq DB (117 tables), Authentik config, Frontend App Shell, Datadog infra, Faber pipeline skeleton, Helm charts, ArgoCD ApplicationSet |
| E2 | Tool Runner | Phase 2 | K8s Job sandbox, “Blind” Secret Pipeline (IRSA + Secrets Manager), tool execution framework, CI pipeline complete |
| E3 | Agent Workers | Phase 3 | LangGraph state machines, Postgres Checkpointing, HITL interrupt/resume, direct SDK agent execution, Celery task queues |
| E4 | Gateway | Phase 4 | SSE streaming, chat UI, task orchestration, /api/search (pgvector), WebSocket broadcasts, OAuth setup |
| E5 | Integrations | Phase 5 | Slack, Jira, GitHub toolsets, webhook listeners, integration tool dispatch, secret resolution |
| E6 | Business Ops | Phase 6 | KPI dashboards, HITL Approval UI, schedule management, rollback logic, staging + production deploy |
| E7 | Stabilization | Phase 7 | Security audit (OWASP), load testing (k6), bug fixes, Marketing rollout, KPI comparison, API docs |
| E8 | Pre-Build | Pre-Build | KPI baseline survey, historical metrics extraction, target definition, environment setup, design finalization |
| E9 | V1.1 Integrations | Post-Launch | Confluence, Zoho CRM, Google Workspace connectors |
| E10 | V1.5 Advanced Features | Post-Launch | Tool Forge, Memory/RAG pipeline, Meeting Orchestration |
9 Components
| Component | Lead | Description |
|---|---|---|
| qqq-api | James Maes | Java data layer, REST API, migrations |
| agent-gateway | US Eng Manager 2 | FastAPI, SSE, WebSocket, chat, routing |
| agent-worker | US Eng Manager 2 | Celery, Anthropic SDK, LangGraph, task execution |
| tool-runner | US Eng Manager 2 | K8s Job sandbox, tool execution |
| web | US Eng Manager 3 | React 19 SPA, all 11 pages |
| infrastructure | James Maes | Docker, Helm, k8s, AWS IaC |
| integrations | James Maes | Slack, Jira, GitHub, OAuth connectors |
| security | James Maes | Authentik, RBAC, secrets, audit |
| observability | James Maes | Datadog APM/RUM/Logs, OpenTelemetry |
Labels
| Category | Labels |
|---|---|
| Service | svc:qqq-api, svc:gateway, svc:worker, svc:tool-runner, svc:web, svc:helm, svc:infra |
| Agent Work | agent-coded (written by Claude Code), agent-reviewed (reviewed by F.R.I.D.A.Y.) |
| Scope | scope:v1, scope:v1.1, scope:v1.5 |
| Risk | risk:security, risk:data-loss, risk:breaking-change |
| Type | tech-debt, performance, accessibility, documentation |
| Track | track:backend, track:frontend, track:infra, track:data |
Automation Rules
| Rule | Trigger | Action |
|---|---|---|
| Auto-assign on transition | Issue moved to IN PROGRESS | Assign to current user if unassigned |
| PR merge → IN REVIEW | PR merged in GitHub (webhook) | Transition linked issue to IN REVIEW |
| Close on merge to main | PR merged to main | Transition linked issue to DONE |
| Sprint auto-close | Sprint ends | Move incomplete issues to next sprint |
| Stale issue alert | Issue in IN PROGRESS > 3 days, no activity | Post Slack notification to assigned engineer |
| Coverage gate | PR fails coverage check | Add comment to linked Jira issue with coverage delta |
| Label sync | Issue created with component | Auto-apply matching svc:* label |
Board Configuration
Columns: Backlog (hidden) → To Do → In Progress → In Review → Done
Swimlanes: By Epic
Quick Filters: My Issues, Backend Track, Frontend Track, Infrastructure Track, Blocked Items, Agent-Coded, Current Sprint Only
3. Confluence Space
| Setting | Value |
|---|---|
| Space Name | Praesidium2 |
| Space Key | PR2 |
| Space Type | Team Space |
| Space Admin | James Maes |
| Permissions | Engineering Leadership Team (edit), All Engineering (view) |
Page Hierarchy
Praesidium2 (Home) | +-- Architecture | +-- System Design (mirrors system-design.html, kept up to date) | +-- Service Map (5 services, dependencies, communication patterns) | +-- Data Model (117 tables, ERD, type table catalog) | +-- Security Architecture (Authentik, RLS, RBAC, secrets) | +-- Observability Architecture (Datadog, OpenTelemetry, correlation IDs) | +-- Integration Architecture (Slack, Jira, GitHub, OAuth flows) | +-- Implementation Plan | +-- Timeline Overview (7-phase Gantt view) | +-- Phase 1: Foundations | +-- Phase 2: Tool Runner | +-- Phase 3: Agent Workers | +-- Phase 4: Gateway | +-- Phase 5: Integrations | +-- Phase 6: Business Ops | +-- Phase 7: Stabilization | +-- Pre-Build Checklist | +-- V1.1 Plan (Post-Launch) | +-- V1.5 Plan (Post-Launch) | +-- Runbooks | +-- qqq Configuration & Table Setup | +-- Authentik Setup (Groups, Policies, OIDC, Scope Mappings) | +-- AWS Deployment (EKS, RDS, ElastiCache, Secrets Manager) | +-- ArgoCD Operations (Sync, Rollback, ApplicationSet) | +-- Faber CI Pipeline Configuration | +-- Local Development Environment (Tilt + k8s) | +-- Redis Operations (Celery broker, pub/sub, cache) | +-- Datadog Setup (APM, RUM, Logs, SDS, Dashboards) | +-- Incident Response Playbook | +-- Secrets Rotation Procedures | +-- ADRs (Architecture Decision Records) | +-- ADR-001: Multi-Repo with Python Monorepo Exception | +-- ADR-002: Anthropic SDK + LangGraph (Scoped) Architecture | +-- ADR-003: qqq as Data Layer (generated REST APIs) | +-- ADR-004: Redis as Celery Broker (over RabbitMQ/SQS) | +-- ADR-005: Faber CI over CircleCI/GitHub Actions | +-- ADR-006: Datadog over ELK/Grafana for Observability | +-- ADR-007: "Blind" Secret Pipeline (IRSA + Secrets Manager) | +-- ADR-008: Tool Runner Isolation (K8s Jobs) | +-- Sprint Reports (Sprint 1-7, one per build phase) +-- Retrospectives (30-Day, 60-Day, 90-Day) +-- Team Directory +-- Onboarding Guide
Templates
| Template | Used For |
|---|---|
| Sprint Report | Weekly sprint summaries (velocity, completed stories, blockers, next sprint goals) |
| Retrospective | 30/60/90 day reviews (what went well, what didn’t, action items) |
| ADR | Architecture decisions (context, decision, consequences, status) |
| Runbook | Operational procedures (prerequisites, steps, rollback, verification) |
| Phase Implementation Update | Per-phase build progress (exit criteria status, blockers, risks) |
Initial Pages to Create Before Phase 1
- Home page with roadmap link and quick links to all sections
- Architecture section populated from system-design.html
- Implementation Plan section populated from implementation.html
- Onboarding Guide with environment setup, repository tour, and dev workflow
- Team Directory with build team + leadership roster, roles, and time zones
- All 5 templates created and published to the space
- Confluence-Jira macros configured for live issue embedding
4. Roadmap
7-phase build plan + post-launch phases with all 10 Epics plotted. Use Jira’s built-in Roadmap view (formerly Plans), grouped by Epic.
Phase & Epic Map
10 Milestones
| Milestone | Target | Criteria |
|---|---|---|
| Pre-Build Complete | Pre-Build (end) | KPI baseline captured, dev environments ready, design finalized, all repos created |
| Phase 1: Foundation | Phase 1 (end) | qqq DB provisioned (117 tables), Authentik configured, Frontend App Shell running, Datadog instrumented, all Faber pipelines green |
| Phase 2: Hands Ready | Phase 2 (end) | Tool Runner executing in K8s Jobs, “Blind” Secret Pipeline working, CI pipeline complete |
| Phase 3: Brain Online | Phase 3 (end) | LangGraph state machines running, checkpoints surviving restarts, HITL interrupt working |
| Phase 4: Connected | Phase 4 (end) | SSE streaming live, chat functional, task orchestration working, OAuth flows registered |
| Phase 5: Integrated | Phase 5 (end) | Slack + Jira + GitHub end-to-end, webhook listeners active, secret resolution by scope |
| Phase 6: Deployed | Phase 6 (end) | Production deployed, Engineering team active on platform, KPI dashboards live |
| V1 Launch | Phase 7 (end) | Security audit clean, load testing passing, Marketing team onboarded, 90%+ coverage, zero critical bugs |
| V1.1 | Post-Launch | Confluence, Zoho CRM, Google Workspace connectors live |
| V1.5 | Post-Launch | Tool Forge, Memory/RAG, Meeting Orchestration live |
Release Versioning
| Version | Content | Type |
|---|---|---|
v0.1.0 | Phase 1 Foundation (qqq DB, Auth, App Shell) | Internal milestone |
v0.2.0 | Phase 2 + Tool Runner | Internal milestone |
v0.3.0 | Phase 3 + Agent Workers | Internal milestone |
v0.4.0 | Phase 4 + Gateway | Internal milestone |
v0.5.0 | Phase 5 + Integrations | Internal milestone |
v0.9.0 | Phase 6 + Business Ops (staging) | Release candidate |
v1.0.0 | Phase 7 + Stabilization (V1 Launch) | Production release |
v1.1.0 | Confluence, Zoho, Google Workspace | Feature release |
v1.5.0 | Tool Forge, Memory/RAG, Meetings | Feature release |
5. Pre-Build Checklist
Everything to set up before the Phase 1 build clock starts. All items are blockers.
- Create all 7 GitHub repos with branch protection
- Add CODEOWNERS, CLAUDE.md, .gitignore, LICENSE to each repo
- Set up GHCR repositories (
ghcr.io/dmdbrands) for all service images - Set up Jira project with 10 Epics, board, components, labels, automation rules
- Create Confluence space with full page hierarchy
- Create Confluence templates (Sprint Report, Retro, ADR, Runbook)
- Configure
.faber.ymlin each service repo (java-webapp, python-webapp, node-webapp) - Set up AWS accounts (Dev, Staging, Prod) in Control Tower
- Run terraform init + first plan on dev account
- Create service account keys for CI/CD
- Set up Datadog organization with API keys
- Configure VPN + bastion host access
- Set up ArgoCD + CD repo configuration
- Begin KPI baseline survey (2-week time tracking)
- Verify all build team + leadership members have access to GitHub, Jira, Confluence, AWS (dev)
6. Day-by-Day Pre-Build Schedule
Parallel workstreams across the three US engineering leads. Target: 10 business days before Phase 1.
| Day | James (CTO) | Mark (SysAdmin) | Barath (Tech Lead) |
|---|---|---|---|
| Day −10 | Create GitHub repos + branch rules | Set up AWS Control Tower accounts | Set up Jira project + Epics |
| Day −9 | Write CLAUDE.md for each repo | Run terraform init on dev | Create Confluence space + pages |
| Day −8 | Configure .faber.yml per service repo | Set up Datadog org + API keys | Distribute KPI baseline survey |
| Day −7 | Set up ArgoCD + CD repo | Configure VPN + bastion | Create sprint board + automation |
| Day −5 to −1 | Review infra plan, finalize architecture | Verify all AWS resources provisioned | Onboard India team to repos/Jira/Confluence |
Acceptance Criteria
Pre-Build is complete when ALL of the following are verified:
- All 7 GitHub repos created with CLAUDE.md, branch protection, CODEOWNERS
- Jira PR2 project with 10 Epics, board, components, automation rules
- Confluence PR2 space with architecture, runbooks, ADR, onboarding pages
- CI/CD pipeline running lint/test on at least one repo
- AWS dev account provisioned with VPC, EKS, RDS (basic terraform apply)
- KPI baseline survey distributed to all 5 team leads
- All build team + leadership members have access to GitHub, Jira, Confluence, AWS (dev)