Assignment 06. Delivery-State Replication and Final Reliability Demo

Week 12 · Assignment brief from the course outline; submission and evaluation results have not been recorded.

Dependency: Extends A5 delivery coordination and complete system

Scenario and assignment objective

A delivery-state replica disappears temporarily. The platform must catch it up and report when replicas have converged.

Required system features

1. Build scoped primary-backup replication for delivery metadata across three replica processes.
2. Every state change has increasing sequence number, event identity, durable primary log and target replicas.
3. Replicas expose last_applied_seq and reject duplicate/out-of-order application safely.
4. Implement gap detection and recovery through replay after disconnect/restart.
5. Explicitly declare acknowledgment semantics (e.g. primary-durable ACK); do not promise zero data loss.
6. Integrate order->payment->delivery flow; add correlation IDs and recovery/latency measurements.

Implementation guide - suggested steps

1. Model delivery change log with seq, delivery_id, action and payload; persist on primary.
2. Create append/apply endpoint for replicas; handle repeated sequence safely and reject missing predecessor.
3. Track last_applied_seq for each replica and implement GET missing-log entries to catch up.
4. Disconnect one replica, generate delivery changes, reconnect and replay until up to date.
5. Compare final record counts/state hash across replicas and measure replication lag/recovery time.
6. Run full Compose stack from clean start and demonstrate selected failure matrix with an evidence report.

Hints (not complete solutions)

• Avoid trying to implement full Raft; the goal is replication semantics, not automatic safe leader failover.
• Use per-replica last_applied_seq and replay from last_applied_seq + 1.
• Do not let a lagging replica become writable leader automatically; report this as a limitation.
• Report p50/p95 only from a repeatable measured workload; note number of requests and machine config.

API / data contracts

• GET /replication/status -> {node_id,role,last_applied_seq,replication_lag}.
• Documentation declares write ACK policy and stale-read restrictions.

Mandatory verification and failure experiments

1. Disconnect replica, send >=10 state changes, reconnect and verify convergence.
2. Restart replica and ensure last-applied marker persists.
3. Submit >=50 test orders; report p50/p95 latency and time required for replica catch-up.
4. Demonstrate no duplicate simulated payment and no duplicate courier assignment after recovery.

Required individual submission

• Entire individual repository and complete Compose startup instructions.
• Architecture and sequence diagrams, final tests, benchmark table, failure matrix, limitations.

Acceptance checklist

• Replicas converge after recovery under stated model.
• Complete checkout and delivery flow operates through gateway.
• Quantitative results reproducible; safety limitations clearly articulated.

Marks: 7 / 40 lab marks. Use the common rubric: implementation 40%, tests 25%, fault experiment 20%, documentation/viva 15%.