System evidence · architecture · operations

A system agents can understand—and cannot casually break.

This is the system the AI engineering method must survive. Explicit ownership, repeatable patterns, constrained trust, executable quality and observable runtime behavior make change inspectable and keep local mistakes from silently becoming system truth.

1. Engineering Purpose

The architecture is not a catalogue of fashionable technologies. It creates explicit places for business decisions, reliable paths between them and evidence when those paths fail.

Agent-friendly architecture makes the right change discoverable. Agent-resilient architecture prevents one wrong change from silently becoming truth.

BlueShop uses naming, ownership, contracts, database constraints, security boundaries, tests and runtime telemetry as mutually reinforcing controls.

01Business behavior firstDomain language, aggregates and state transitions define what the system means before adapters define how it moves.
02One authority per decisionStock owns availability, Cart owns quotes, Order owns order history and downstream projections do not rewrite those facts.
03Explicit propagationHTTP is used for immediate decisions; Kafka carries committed facts and rebuildable projections.
04Failure is observableTraces, logs, metrics, correlation IDs and retained events connect visible symptoms to the responsible boundary.

2. System Context

Customer and employee Angular applications enter through Traefik and the API Gateway. Reactive domain services own operational decisions. Kafka connects asynchronous facts to projections, notifications and analytics.

Customer request and derived insight

Customers and employeesStorefront, backoffice and identity experience

→

Traefik + API GatewayRouting, token validation and edge policy

→

Domain servicesReactive business decisions and service-owned state

→

Kafka projectionsSearch, notifications, recommendations and analytical facts

→

ClickHouse + MetabaseDerived business views, never transactional authority

Operational decisions stay in the domain services. Analytics is downstream and rebuildable; it never participates in transactional writes.

Layer	Current technology	Architectural role
Applications	Angular 21, Nx, signals, Signal Store, Playwright	Customer and role-scoped employee experiences
Edge	Traefik, Spring Cloud Gateway, Keycloak	Ingress, routing, authentication and edge policy
Services	Java 25, Spring Boot 4, WebFlux, Reactor	Domain behavior and non-blocking orchestration
Operational data	PostgreSQL/R2DBC, Redis, Elasticsearch	Service-owned state, cache/session and search projections
Events	Kafka KRaft, OAuth bearer identity, CloudEvents	Committed facts, projections and workflow decoupling
Analytics	ClickHouse, Metabase, bounded JDBC scheduler	Event-sourced facts and dashboard read models
Platform	K3s, ArgoCD, Kustomize, Terraform, Vault, SigNoz	Self-hosted runtime, desired state, secrets and observability

3. Domain Ownership

A deployment unit is not automatically a bounded context. BlueShop treats business services as candidate boundaries only where they own a domain model, language and integration contract.

A commerce decision chain without shared ownership

Discovery

Catalog owns product→Stock projects availability→Recommendation ranks candidates→Storefront presents

Purchase

Cart owns quote→Stock owns reservation→Order snapshots acceptance→Logistics owns shipment

Aftercare

Review owns moderation→Support owns recovery case→Analytics projects outcomes

Each arrow passes facts or requests. It does not transfer ownership of the source decision.

Boundary	Authoritative responsibility	Important boundary rule
Catalog	Products, variants, categories, brands, attributes and curated relations	Search and inventory visibility are projections, not new product authorities
Stock	Availability, movements and reservations	Only Stock may grant the final unit
Cart	Shopping session, checkout preparation and authoritative quote	Downstream services snapshot accepted value
Order	Order identity, history and commercial snapshots	Does not recompute historical price or own shipment work
Payment	Provider transactions and payment lifecycle	Order derives payment state from outcomes
Customer	Profiles, addresses and loyalty state	Marketing owns campaigns, not customer identity
Logistics	Shipment and delivery lifecycle	Reports accepted warehouse facts to Order
Recommendation	Candidate generation, ranking, serving and measurement	Projects source facts; does not own products, stock or purchases
Support and Review	Case recovery and moderated feedback	Validate against Order evidence without absorbing Order ownership
Analytics	Derived facts and dashboards	Never becomes a transactional write dependency

4. From Business Language to Executable Behavior

Specifications are useful only when identities survive the path into executable scenarios and durable reports.

Traceability chain

Business ruleUbiquitous language and invariant

→

Requirement + ACStable IDs and expected outcome

→

Gherkin scenarioGiven, When, Then behavior

→

ImplementationDomain model and adapters

→

Evidence reportTest result tied back to the criterion

The specification checker validates schema, identities, cross-references, rule tags and report-backed verification. A declared status without durable evidence is not accepted as verified.

5. Immediate Decisions and Asynchronous Facts

HTTP answers questions that must be decided now. Events report committed facts and let independent projections move at their own pace.

Two communication paths

Answer now

Customer asks→Owner checks identity and rules→Non-blocking database and API work→Return one answer

Share a fact

Save state and event together⇢Publish the committed fact⇢Ignore duplicate delivery⇢Update the local projection

If a consumer creates another business fact, that follow-up event is saved with its own local update. It is optional, not a step every projection performs.

Immediate path

Answer while the customer waits.

The owning service validates the request, applies its domain rules and composes PostgreSQL or downstream API work without holding a thread open. Timeouts and resilience policies bound every dependency before one response returns.

WebFlux, Reactor, R2DBC and WebClient implement this path. Production request code does not call .block().

Fact propagation

Tell other boundaries what happened.

The producer saves business state and its outbox event in one transaction. After publication, a consumer records the event identity before applying the local effect, so redelivery cannot repeat that effect.

CloudEvent identity, aggregate-key ordering and transactional inbox/outbox records make retries safe. ClickHouse JDBC is the explicit blocking exception and runs on a bounded scheduler.

6. Data Architecture

Storage follows ownership and workload. Operational services retain transaction authority; projections remain rebuildable and analytics remains downstream.

Store	Used for	Boundary
PostgreSQL	Operational service schemas, aggregates, outbox/inbox and RLS	Service-owned logical state; dynamic Vault database leases
Redis	Bounded cache and session use cases	Never replaces authoritative business state
Elasticsearch	Catalog-owned storefront search projection	Rebuildable through aliases and durable index work
Kafka	Retained integration facts and projection feeds	Transport and ordering boundary, not a domain database
ClickHouse	Event-sourced analytical facts and dimensions	Read-time deduplication; no operational write dependency
Longhorn volumes	Persistent state for self-hosted workloads	Shutdown and recovery ordering matter for stateful safety

Migration discipline: applied Flyway migrations are immutable. New changes receive new versions. Analytics uses versioned ClickHouse SQL applied through its migration runner rather than application-startup DDL.

TRUST AND OPERATION

The system assumes every boundary can fail—and every identity needs a reason to cross it.

Security, platform reconciliation, quality and observability make the architecture defensive rather than merely understandable.

7. Security and Trust Boundaries

Identity is propagated across protected boundaries. Internal routing and agent tooling do not become authorization bypasses.

Layered trust

Identity

Keycloak OIDCUser and service rolesBearer propagationKafka workload identity

Application

Gateway validationService validationMethod authorizationMCP capability scopes

Data

PostgreSQL RLSService rolesColumn-safe API viewsClickHouse read boundaries

Secrets

VaultDynamic DB leasesExternal SecretsNo committed values

Network

Ingress policyNamespace isolationService allowlistsDefault deny

A valid route is not sufficient. The caller needs a valid identity, the correct application authority and a permitted data context.

Defense in depth: PostgreSQL RLS filters rows, not sensitive columns. APIs and dedicated views still control field-level exposure.

8. Self-Hosted Platform

Agents first built and evolved the infrastructure, then used it as the execution environment for the commerce system and its engineering feedback loops.

How the platform operates

Build and deliver

Agents change Git→GitLab tests and builds→Harbor stores the image→ArgoCD reconciles K3s

Serve traffic

Traefik receives the request→Applications run on K3s→PostgreSQL, Kafka and projections hold state

Control every layer

Keycloak proves identity·Vault supplies secrets·OpenTelemetry sends evidence to SigNoz

Kustomize and Terraform describe desired state; ArgoCD applies the Git-managed workload state. Longhorn and network policies support the runtime beneath these paths.

The platform is intentionally real enough to expose scheduling, credential rotation, persistent storage and recovery problems that a local mock environment would hide.

9. Executable Quality Architecture

Different checks answer different questions. A successful compile cannot prove a state machine, a browser journey or a safe distributed failure path.

Evidence widens with the change

StructureTypes, null safety and architecture boundaries

→

BehaviorDomain rules, transitions and failure branches

→

IntegrationDatabase, event, HTTP and security boundaries

→

AcceptanceBusiness scenarios and browser journeys

→

System evidenceCross-service behavior, load and runtime traces

A small domain rule needs focused behavioral proof. A shared, event or security change needs evidence across every boundary it can affect.

Quality question	What must be proven
Does the behavior match the rule?	Domain invariants, legal transitions, reactive success and failure paths behave as specified.
Does the customer experience still work?	Components render correctly, remain accessible, complete the browser journey and build in every impacted application.
Did the architecture drift?	Domain ownership, non-blocking execution and dependency boundaries remain intact.
Are the tests strong enough?	Critical branches are exercised and mutation analysis exposes assertions that would miss behavioral changes.
Are integrations compatible?	API, event and specification changes preserve required consumers and traceability.
Is unauthorized behavior denied?	Roles, row-level security, secret handling and network boundaries reject invalid access.
Is the whole system stable?	Cross-service journeys repeat reliably and performance remains within explicit limits.

10. Resilience and Recovery

Resilience is distributed across domain idempotency, communication policy, data recovery and platform lifecycle. It is not one circuit-breaker annotation.

01Transactional boundariesDomain state, outbox and inbox effects commit together where message loss or duplication would corrupt behavior.
02Controlled degradationTimeouts, retries, circuit breakers, caches and fallback behavior are explicit at synchronous dependencies.
03Rebuildable projectionsSearch, recommendation and analytical state can be reconstructed from owners or retained facts.
04Credential continuityVault database leases rotate without static application credentials; lease errors retain recoverable state.
05Stateful shutdownVault, PostgreSQL, Kafka, ClickHouse, Elasticsearch and persistent volumes drain before K3s stops.
06Declarative recoveryDesired-state fixes remain in Git so emergency understanding does not become permanent hidden drift.

Example: an unsafe host shutdown once produced an all-NUL Vault lease file. The durable correction was an ordered stateful drain plus repeated power-cycle proof—not a startup retry hiding storage corruption.

11. Observability and Root-Cause Debugging

OpenTelemetry connects customer-visible behavior to edge, service, event and persistence evidence in SigNoz. Diagnosis follows the path of the request or fact instead of guessing from deployment status.

From symptom to durable correction

Visible symptomBrowser, API, business outcome or failed assertion

→

CorrelationTrace ID, entity ID, route and time window

→

Distributed evidenceSpans, logs, Kafka offsets, SQL and resource pressure

→

Owning root causeCode, policy, data, contract or infrastructure boundary

→

RegressionSmallest check that fails if the defect returns

Runtime evidence can disprove an attractive code-level theory. The regression then captures the corrected understanding.

Signals connected

HTTP route and status;
reactive service spans;
database calls and pool pressure;
Kafka produce, consume and lag;
trace and correlation identifiers;
CPU throttling, memory and pod scheduling.

Questions answered

Did the request reach the owner?
Was the state committed?
Was the event emitted and consumed?
Did the projection apply the correct identity?
Was latency compute, pool, network or dependency?
Did the user-visible result agree?

12. The Engineering Feedback Loop

Architecture, delivery and observation form one learning system. A defect is not complete when patched; the corrected understanding must become executable and discoverable.

Specification to stable evolution

Design

Business language→Ownership and contract→Acceptance scenario

Build

Domain behavior→Adapters and infrastructure→Executable quality

Learn

Running system→Evidence and root cause→Regression + durable context ↺

The final learning returns to architecture, specifications, tests and agent context. That is how the system becomes easier to change after each real failure.

See how agents operate inside this loop.

13. Current Limits and Transitional Boundaries

01Not every service is a proven bounded contextThe context map remains incomplete and deployment topology must not substitute for domain evidence.
02Payment is not externally integratedCurrent order creation records a selected method but does not prove authorization, capture, refund or chargeback.
03Fulfilment ownership is transitionalCart currently commits the Stock reservation after Order creation while Kafka provides idempotent recovery; a future Fulfilment boundary is only a candidate.
04Some projections fail open on unknown stateCatalog and Recommendation can retain eligibility when Stock state is missing or unknown.
05Development topology is not universal HASingle-node components and the self-hosted laboratory should not be presented as a production-scale availability claim.
06Traceability remains partialThe specification corpus is structurally checked, but most active scenarios are not yet tied to durable verified reports.

Result

Explicit architecture turns AI work into engineering work.

BlueShop is navigable because responsibilities, contracts and evidence have names. It is resilient because authority, data, security and recovery do not depend on an agent understanding everything perfectly.

Continue to AI engineering · See the applied commerce result · · Return to the site map