Explanation: Live Sync & Data Freshness

This document explains the technical architecture of the Danvas external synchronization pipeline, including Dagster orchestration, Next.js 16 caching layers, and the circuit-breaker status engine.

The Synchronization PipelineCopy Anchor Link

Danvas synchronizes data from Toast POS (sales, clock-ins, orders, discounts) and 7shifts (scheduling, employees, roles). Integration runs are orchestrated by Dagster assets and background data-sync packages (@repo/data-sync, @repo/toast):

┌────────────────┐      Partition-native extraction       ┌───────────────────┐
│ Toast / 7shifts├───────────────────────────────────────►│ Dagster Pipelines │
└────────────────┘                                        └─────────┬─────────┘
                                                                    │ Writes to DB
                                                                    ▼
┌────────────────┐          Reads (cacheLife)             ┌───────────────────┐
│ apps/app (RSC) ◄────────────────────────────────────────┤ Neon Database     │
└────────────────┘                                        └───────────────────┘

Ingestion & Active WritersCopy Anchor Link

Orchestration is owned by Dagster assets (dagster/canvas_forge/) and database-backed sync packages:

1. Adapter Layer: @repo/toast/client and @repo/sevenshifts handle connection pools, retries, and token management.
2. Staging & Marts: Assets extract raw data into PostgreSQL staging tables (stg_toast_*) and transform them into analytics.f_* marts.
3. State Tracking: Every execution updates high-watermark timestamps and circuit-breaker states in the sync_state table.

Next.js 16 cacheLife OptimizationCopy Anchor Link

The authenticated app uses Next.js 16 Cache Components for bounded live reads:

• Dashboard and monitoring caches: getCachedDashboardSummary() and getCachedMonitoringSummary() use the custom live profile. The profile allows a 30-second client stale window, revalidates server results every 30 seconds, and expires entries after 60 seconds.
• Tag-based invalidation: Dashboard mutations invalidate dashboardSummaryTag(...) with updateTag() for read-your-own-writes. Shift-task and location mutations invalidate their team-scoped tags with revalidateTag(..., "max").
• Dynamic holes: These live reads remain request-time streamed content; their short expiration intentionally keeps them out of the static App Shell.

The client still performs non-blocking refreshes for status indicators. Cache invalidation and freshness are separate concerns: invalidation removes known stale entries, while the live profile bounds freshness when no mutation signal is available.

Circuit-Breaker Status EngineCopy Anchor Link

To prevent cascading failures and continuous hammering when third-party APIs experience outages:

• The sync manager tracks sequential execution failures in sync_state.
• If a sync source fails 3 times consecutively, the circuit opens, marking the source as disabled and triggering an alert.
• While the circuit is open, subsequent reads serve cached local marts directly, preventing UI requests from blocking.
• The circuit breaker transitions to half_open after a 5-minute cooldown period to attempt a recovery sync.