# Queues, Horizon & Redis

🟢 **Decided 2026-06-13.** How background jobs and the analytics firehose run on
Redis without the analytics volume ever slowing important jobs — sized to LAFF's
actual measured traffic.

---

## Stack
- **Laravel Horizon + Redis** for all background work (queues, workers, monitoring).
- Horizon supervises **separate worker pools** for ops vs analytics.
- Analytics events buffer through a **Redis Stream**, drained in batches — NOT
  one queued job per event.

This is the concrete implementation of the inviolable rule from
[data-stores-and-logging.md](data-stores-and-logging.md): *analytics writes are
never synchronous in a request path.*

---

## Two isolation problems (they are different layers)

| Isolation | Solves | Mechanism |
|---|---|---|
| **Worker / CPU** | analytics jobs starving payment/mail jobs of *workers* | separate Horizon pools (separate processes) |
| **Redis memory / throughput** | the firehose evicting *cache* or saturating Redis ops | separate Redis connection → eventually separate instance |

> Separate **pools** ≠ separate **Redis**. Pools isolate CPU on the worker side;
> a separate Redis isolates memory/throughput on the data side. They're orthogonal
> — you climb to each only when traffic warrants (see ladder).

---

## Architecture (current decision — Stage 0)

```
                    ┌─────────────────────────────────────┐
  HORIZON           │  Redis (single instance, Stage 0)    │
   ├─ "ops" pool ───┼─→ queues: webhooks, mail, default    │
   │   high prio    │   + cache + sessions                 │
   │                │   db1 (separate logical): analytics  │
   └─ "analytics" ──┼─→ Redis Stream: event buffer         │
       pool, low    └─────────────────────────────────────┘
       prio                       │
                                  ▼  worker: XREADGROUP ~1000 at a time
                          one bulk INSERT per batch → analytics store
```

- **ops pool** — webhooks (payments!), mail, certificates, drip, AC sync. Few
  jobs, each matters, retried, monitored. Reserved processes so a log spike can
  never delay a payment confirmation.
- **analytics pool** — drains the Redis Stream in batches. High volume, low
  priority, disposable. Scales independently.
- **Redis Stream** chosen over per-event queued jobs: built for append +
  consumer-group batch reads + auto-trim; minimal per-event overhead; keeps
  Horizon focused on real jobs.

---

## Sizing: grounded in LAFF's real traffic (the key decision input)

**Measured:** ~100,000 website clicks/month (Berner, 2026-06-13).
**Current analytics writes** (`wp_thrive_reporting_logs`): ~1,562/day avg,
~1,800/day peak — i.e. ~1 event every 50s. Trivial.

**Projected full clickstream** (navigation + A/B exposure ≈ ~15 events/visit):
100k visits × 15 ≈ **~1.5M events/month ≈ ~50k/day ≈ ~0.6/sec avg, ~5–10/sec peak.**

A single modest Redis does **tens of thousands of ops/sec**. At ~10/sec peak we
are using a tiny fraction of one Redis. **→ Stage 0 is correct; a separate Redis
now would protect against load 3–4 orders of magnitude below capacity.**

---

## The split ladder (climb only on measured triggers)

| Stage | Events/day (sustained) | Peak/sec | Setup |
|---|---|---|---|
| **0 — now (LAFF is here)** | < ~50k | < ~10 | **1 Redis** + separate Horizon pools + Stream buffer |
| 1 — growing | ~50k–1M | ~10–50 | analytics on a **separate logical Redis DB + connection** (config only) |
| 2 — heavy | ~1M–10M | ~50–500 | **separate Redis instance** for the analytics stream |
| 3 — firehose | > ~10M | > ~500 | separate Redis **+ ClickHouse** storage; consider Kafka if Streams strain |

### Triggers to climb (watch these — not vibes)
- 🔴 **`evicted_keys > 0`** on Redis — analytics is pushing out cache. **#1 signal.**
- `used_memory` approaching `maxmemory`.
- `instantaneous_ops_per_sec` trending toward instance ceiling.
- Horizon **wait time** rising on the `ops` pool (webhooks/mail).
- Analytics **stream backlog** growing faster than the worker drains it.

### Why each climb is cheap
The analytics path uses a **named Redis connection** and a **Stream** behind the
`analytics.track()` interface. So Stage 0→1→2 are **config changes** (repoint the
connection), not code changes. Instrument once; topology evolves underneath.

**At current trajectory (~50k/day projected), the first split is ~20× away —
a long runway.** Build Stage 0, instrument, let the numbers say when to climb.

---

## Open follow-ups
- Pick Redis maxmemory + eviction policy (cache keys evictable; analytics stream
  should NOT silently evict — trim explicitly or split early).
- Horizon pool process counts + `balance` strategy per environment.
- Stream trimming / retention policy (how long raw events live in Redis before
  they're safely in the store).
See [../00-index/open-questions.md](../00-index/open-questions.md).