From Agents to Architecture: Determinism, Observability & Provider Boundaries

This week’s focus

Stabilising the system architecture before expanding it.

This week I focused on evolving the product foundation as deterministic, observable, and vendor-agnostic. I also strengthened the spine — analytics determinism, provider abstraction, and deployment reliability.

Related experiment: Runner Agentic Intelligence

What actually happened

The week unfolded as a structural consolidation across three layers:

Deterministic Analytics Core
- Refactored weekly snapshots and trends into deterministic engines.
- Enforced that everything must be recomputable from raw runs.
- Implemented recompute-on-startup to restore historical consistency.
Centralised Multi-Provider Integration
- Refactored LLM usage behind a unified LLMClient abstraction.
- Centralised Gemini, OpenAI, and HuggingFace models under LiteLLM.
- Enforced structured outputs consistently across providers.
- Shifted provider selection from configuration sprawl to capability boundary.
Observability Maturation
- Strengthened structured logging.
- Introduced consistent trace_id propagation.
- Added span lifecycle discipline (start/end boundaries).
- Used trace context to debug cross-agent behaviour.
- Reinforced observability as architectural necessity, not tooling add-on.

Additionally:

Fixed HuggingFace Spaces restart loop caused by improper async lifecycle ownership.

The system moved from experimentation-heavy to architecture-driven.

Key trade-offs

Chose structural refactor over new feature development.
Accepted short-term instability for long-term clarity.
Prioritised deterministic engines over agent flexibility.
Delayed UI improvements to stabilise domain boundaries.
Centralised provider abstraction before optimising model routing logic.

What changed in my thinking

Deterministic computation must form the spine; LLMs belong at the interpretation edge.
Provider abstraction is strategic leverage, not implementation detail.
Invariants reduce cognitive overload during deep refactors.
Platform lifecycle constraints (async ownership in HF) shape system design.
Empty states must be designed intentionally — not handled reactively.

Architecture signals (optional)

Deterministic engine for analytics; agents for interpretation.
Recompute-from-raw as non-negotiable invariant.
Null-object pattern to stabilise empty domain states.
Provider abstraction boundary via LiteLLM.
Observability as first-class system concern.
Platform runtime constraints influence architecture.
Stepwise execution reduces execution risk under complexity.

Key takeaways

Make raw data the single source of truth.
Separate deterministic computation from generative reasoning.
Centralise provider logic early to reduce blast radius.
Observability must be embedded before scale, without traceability, agent systems degrade into guesswork.

Assumptions invalidated (optional)

Recompute would implicitly restore deleted weeks.
Storage-disabled mode would degrade gracefully without strict contracts.
Async bootstrap patterns behave identically across environments.
Ingestion pipelines were “complete enough.”
Basic logging would suffice for multi-agent orchestration.

Looking ahead (optional)

Automate invariant testing (snapshot and trend recompute).
Strengthen ingestion robustness (GPX HR parsing tests).
Expand trace context into metrics and structured spans.
Design historical backfill using deterministic engine foundation.
Introduce more advanced longitudinal insights safely.

Architecture signals & invalidated assumptions

Note: This Weekly Learning was produced using the Ideas to Life Weekly Learning system.
See: Weekly Learning system map