Cross-Server Chat

The single-server chat hits a wall

In Lesson 22 you built a multi-room chat on one php app.php instance. Every connected browser talks to the same OpenSwoole process; a message sent by user A lands in worker N and is pushed to user B’s $fd right there. No coordination needed.

Now you put two php app.php instances behind a load balancer — one on :8080, one on :9090. User A connects to the first, user B to the second. A sends “hi”. B never sees it. Why?

       Browser A ↔ LB → [server :8080] worker 0  fd=12   user A
       Browser B ↔ LB → [server :9090] worker 1  fd=12   user B   ← different process!

  A sends “hi”. Server :8080 worker 0 receives it and tries
  $server->push($fd_of_B, "hi")  — but $fd_of_B doesn’t exist on :8080.
  The message dies in worker 0’s memory. B never sees it.

The mental model: $fd is process-local

$fd is a per-process integer handle into OpenSwoole’s worker-local connection table. Worker A’s $fd=12 is unrelated to worker B’s $fd=12 on the same process, and on a different OpenSwoole process the integer is meaningless — $server->push($fd_from_other_process, "hi") silently drops the message.

Only the worker that accepted the connection on the process that owns the listening socket can push to that $fd. Everyone else has to route through it.

That routing is what Lessons 22 build: a shared map saying “client X is owned by server Y”, plus a messaging fabric so any server can ask Y to push.

The three primitives

Three pieces, all already in ZealPHP — no extra packages, no custom infra.

1. A shared Store backend

The default Store uses OpenSwoole\Table — process-local shared memory. That’s fast (nanoseconds) but stops at the process boundary. Flip to Redis with one line:

use ZealPHP\Store;

// In app.php, BEFORE $app->run() or App::run():
Store::defaultBackend(Store::BACKEND_REDIS);
// Or via env: ZEALPHP_STORE_BACKEND=redis ZEALPHP_REDIS_URL=redis://cache:6379

Every existing Store::make(), Store::set(), Store::get() call keeps working unchanged — now backed by Redis (or Valkey) and visible to every worker on every node. See /store#redis for the full backend doc.

2. An ownership map

Each connected client needs to be discoverable from any server. A Store table keyed by client_id with two columns — the server identity and the worker-local fd:

Store::make('ws_owner', 4096, [
    'server' => [Store::TYPE_STRING, 64],   // e.g. "host-7:31415"
    'fd'     => [Store::TYPE_INT],
]);

“Who owns Alice?” becomes one Redis HGET. “Alice disconnected” becomes one Redis DEL. Cross-node consistency for free.

3. A per-server inbox

Each server identifies itself (hostname + pid is enough) and subscribes to a channel keyed on that identity. Messages published to ws:server:$myId only get delivered to this server — no broadcast storms, no filtering.

$myId = gethostname() . ':' . getmypid();

App::subscribe("ws:server:{$myId}", function (string $payload): void {
    $msg = json_decode($payload, true);
    // ... deliver locally to $msg['fd'] ...
});

App::subscribe spawns a dedicated subscriber coroutine per worker at onWorkerStart. Cleanup on shutdown is automatic. See /pubsub#quickstart.

Build it — four small deltas

Start from the Lesson 22 chat. The four changes below are isolated — you can apply them one at a time and rerun your chat between each step.

Step 1: Switch the Store backend

// app.php, near the top
use ZealPHP\Store;

Store::defaultBackend(Store::BACKEND_REDIS);

// Existing Store tables (chat history, presence, …) keep working unchanged.

Step 2: Claim ownership on connect, release on disconnect

$myId = gethostname() . ':' . getmypid();

// One-time: declare the ownership table BEFORE App::run()
Store::make('ws_owner', 4096, [
    'server' => [Store::TYPE_STRING, 64],
    'fd'     => [Store::TYPE_INT],
]);

App::ws('/chat',
    onMessage: function ($server, $frame) { /* handle inbound */ },

    onOpen: function ($server, $request) use ($myId) {
        $clientId = $request->get['user'] ?? bin2hex(random_bytes(8));
        Store::set('ws_owner', $clientId, [
            'server' => $myId,
            'fd'     => $request->fd,
        ]);
    },

    onClose: function ($server, $fd) use ($myId) {
        // Find which client owned this fd and drop the row.
        foreach (Store::iterate('ws_owner') as $clientId => $row) {
            if ($row['server'] === $myId && (int)$row['fd'] === $fd) {
                Store::del('ws_owner', $clientId);
                break;
            }
        }
    }
);

Step 3: Subscribe to your own inbox

App::subscribe("ws:server:{$myId}", function (string $payload) use (&$server): void {
    $msg = json_decode($payload, true);
    $fd  = (int)($msg['fd'] ?? 0);
    if ($fd > 0 && $server->isEstablished($fd)) {
        $server->push($fd, $msg['data']);
    }
});

isEstablished guards against the race where the client dropped between the publish and the local delivery — push on a dead fd is a no-op, but the guard avoids the stderr log noise.

Step 4: Send via lookup + publish, not direct push

function sendToClient(string $clientId, string $data): bool {
    $owner = Store::get('ws_owner', $clientId);
    if (!$owner) {
        return false;   // client not connected anywhere we know about
    }
    Store::publish("ws:server:{$owner['server']}", json_encode([
        'fd'   => $owner['fd'],
        'data' => $data,
    ]));
    return true;
}

// Use it everywhere you previously called $server->push():
sendToClient('alice', json_encode(['from' => 'bob', 'text' => 'hi']));

That’s the entire change. Local message? The owning server is you — one Redis publish, one local subscriber dispatch, one local push. Remote? Same code path, the publish crosses the network instead.

Try it live: two ports, one Redis

Set the Store backend via env, start two instances on different ports, open two browser tabs — one per port — and watch messages cross.

# Terminal 1
ZEALPHP_STORE_BACKEND=redis \
ZEALPHP_REDIS_URL=redis://127.0.0.1:6379 \
php app.php start -p 8080

# Terminal 2
ZEALPHP_STORE_BACKEND=redis \
ZEALPHP_REDIS_URL=redis://127.0.0.1:6379 \
php app.php start -p 9090

# Browser
open http://localhost:8080/chat?user=alice
open http://localhost:9090/chat?user=bob

# Alice types "hi bob" — Bob sees it (delivered via Redis).
# Stop the :8080 instance — Bob stays connected to :9090 (his fd belongs there).
# Alice reconnects to :9090, the ws_owner row updates, sending resumes.

Don’t have Redis running locally? Valkey (Redis-compatible) is a drop-in: docker run -p 6379:6379 valkey/valkey:8. The ZealPHP test suite uses Valkey on :16379 — either is fine.

The WSRouter shortcut

Steps 2–4 above are the same five lines for every app that wants this pattern. The framework bundles them into ZealPHP\WSRouter:

use ZealPHP\WSRouter;

// app.php (before App::run())
// Defaults are demo-grade: 4,096 owner rows + 16,384 room-member rows. Bump
// inline for production (HARD CAPS on the Table backend; informational on Redis):
WSRouter::init(
    ownerCapacity:       200_000,    // max concurrent WS connections cluster-wide
    roomMembersCapacity: 1_000_000,  // max (room × member) pairs cluster-wide
    slowConsumerBytes:   4 * 1024 * 1024,  // per-fd send-queue drop threshold
);
// Or call WSRouter::initOptions(...) BEFORE init() — same setters, different shape.

App::ws('/chat',
    onMessage: function ($server, $frame) { /* … */ },
    onOpen:    function ($server, $request) {
                   WSRouter::own($request->get['user'], $request->fd);
               },
    onClose:   function ($server, $fd) {
                   // optional — release() looks up by client id if you tracked one
               },
);

// Anywhere:
WSRouter::sendToClient('alice', json_encode(['from' => 'bob', 'text' => 'hi']));
WSRouter::broadcast('chat:room:42', json_encode(['hello' => 'everyone']));

Same machinery, one boot call, two per-connection calls, two send helpers. Use the helper for new code; the four-step manual build above is for understanding what the helper is doing.

Beyond two servers — first-class rooms

Direct send (sendToClient) routes to one client. For “every member of room 42” (chat rooms, presence, live leaderboards), the framework ships a first-class Room object — cluster-wide membership, presence events, fan-out broadcast, and a handler registry, all behind 4 verbs:

First: how is the user identified?

The framework doesn’t impose a user-identity scheme — you supply a stable string $clientId (session ID, user ID, email, JWT subject) and it’s used consistently across the routing fabric. The typical wiring inside onOpen:

use ZealPHP\G;

App::ws('/chat',
    onOpen: function ($server, $request) {
        $g = G::instance();   // canonical per-coroutine context; works in BOTH
                              // superglobals modes. Raw $_SESSION races across
                              // coroutines in coroutine mode — always use $g.
        // 1. Identify the user — pick ONE pattern that suits your app:
        //    a) Cookie session (Apache/PHP-mod parity, default in ZealPHP):
        $sessionId = $request->cookie['PHPSESSID'] ?? null;
        $username  = $g->session['username'] ?? 'guest';
        //    b) Query param (demos / quick tests): $request->get['user']
        //    c) JWT in `Sec-WebSocket-Protocol` header / Authorization: parse + verify
        if ($username === 'guest') { $server->disconnect($request->fd, WSRouter::CLOSE_AUTH_REQUIRED); return; }

        // 2. Register cluster-wide ownership using THAT identifier.
        //    $clientId is the value you'll thread through every send below.
        WSRouter::own($username, $request->fd);

        // 3. Join whatever rooms the user belongs to.
        WSRouter::room('chat:room:42')->join($username);
    },
    onClose: function ($server, $fd) {
        // Find which user owned this fd + release on disconnect:
        // (a `WSRouter::releaseByFd($fd)` helper is on the roadmap; for now
        //  apps store the reverse-map in $g->openswoole_request data.)
    },
);

Use WSRouter::CLOSE_AUTH_REQUIRED (4001) / CLOSE_AUTH_INVALID (4002) / CLOSE_FORBIDDEN (4003) when refusing the upgrade — clients can react to those codes specifically. The full set lives at WSRouter::CLOSE_*.

The 4 Room verbs (post-identity)

use ZealPHP\WSRouter;

// In your handlers, $username is the SAME identifier you passed to WSRouter::own():
$room = WSRouter::room('chat:room:42');
$room->join($username);                 // SADD-equivalent + presence event broadcast cluster-wide

// From anywhere on any server. Payload schema is APP-DEFINED — the
// framework doesn't enforce a 'from' key. Common convention:
$room->push(
    ['from' => $username, 'text' => 'lunch!', 'ts' => time()],
    fromClientId: $username,            // optional — wires WS-4 per-client rate limit
);
$room->size();                                         // cluster-wide member count (SCARD)
$room->members();                                      // cluster-wide roster (SSCAN-drained)
$room->membersPaged($cursor, 100);                     // paginated roster for very large rooms

// Optional: handlers on EACH server that receive the broadcast + fan out
// to each server's locally-owned fds. Registered ONCE at boot.
$room->onMessage(function (array $msg, string $room) {
    // $msg is the decoded payload; push to your local fds here, e.g.
    // foreach (yourLocalFdsFor($room) as $fd) { $server->push($fd, json_encode($msg)); }
});
$room->onPresence(function (array $event, string $room) {
    // $event = ['type' => 'join'|'leave', 'client_id' => '...', 'ts' => ...]
    // — the 'client_id' here IS the value you passed to join().
});

// Cleanup:
$room->leave($username);

How it federates. A single PSUBSCRIBE ws:room:* pattern subscriber per worker covers every room you ever create — no per-room subscriber explosion. Membership lives in the cluster-wide ws_room_members Store table; size + roster lookups go via a per-room Redis SET (O(1) SCARD; paginated SSCAN for very large rooms). Server-side enforcement: filling a capped membership table throws WS\CapacityException with an actionable bump hint — the framework refuses to silently drop joins.

The lower-level primitives (WSRouter::broadcast($channel, $payload) + App::subscribe('your:channel:*', fn ...)) are still available for cases that don’t fit the Room shape — presence-only feeds, cross-app event buses, telemetry. They’re what WSRouter::room() uses under the hood.

No fan-out service to deploy — Redis pub/sub does it. For at-least-once delivery (audit logs, payments, work queues), swap Store::publish for Store::publishReliable and App::subscribe for App::subscribeReliable — same shape, backed by Redis Streams with consumer groups. See /pubsub#reliable.

Production notes

• Capacity defaults are demo-grade. WSRouter::init() creates ws_owner sized for 4,096 rows and ws_room_members sized for 16,384 rows — with two different meanings depending on the backend:
  • Table backend (single OpenSwoole process): these are HARD CAPS on shared memory allocated at master fork. Scope = workers within THIS process, NOT cross-host. So ws_owner at 4,096 means “max concurrent WS connections this one server can hold.”
  • Redis backend: the numbers are informational only. Redis is a global KV with no per-table cap, and the rows are visible across every server in the cluster. Bound disk/memory growth via Redis-server maxmemory + maxmemory-policy allkeys-lru. The “cluster-wide capacity” promise is the Redis story.
  Bump inline for production:

  WSRouter::init(
      ownerCapacity:        200_000,
      roomMembersCapacity:  1_000_000,
      slowConsumerBytes:    4 * 1024 * 1024,   // per-fd backpressure threshold (default)
  );

  Filling a capped Table throws WS\CapacityException with an actionable hint — the framework refuses to silently drop owners.
• Driver choice. Both phpredis (preferred when ext-redis is loaded) and predis SUBSCRIBE loops yield correctly under HOOK_ALL. phpredis is ~2× faster on hot CRUD; pick it when you can. The only nuance: phpredis SUBSCRIBE blocks the worker without HOOK_ALL — HOOK_ALL is on by default in coroutine mode, so this only matters if you’ve disabled it explicitly. See /store#phpredis-pubsub-caveat.
• Sticky load balancer? Doesn’t matter — the routing fabric is keyed on client_id, not source IP. Sticky LBs reduce ws_owner churn (the same client always lands on the same server); non-sticky LBs just write more rows on reconnect. Pick whichever fits your infra.
• Owner-row staleness. If a server dies hard, its ws_owner rows linger until the clients reconnect (which they will — the LB sends them elsewhere). Add a TTL on the table (Store::make('ws_owner', …, ['mode' => 'ttl', 'ttl' => 3600])) for automatic cleanup, or wire App::onWorkerStop to release this worker’s rows on graceful shutdown.
• Cross-region? Redis pub/sub is in-cluster; for multi-region you want a Redis replica in each region with publishing pinned to a primary, or a dedicated message bus (NATS, Kafka). Same routing pattern, different transport.

Key takeaways

• $fd is process-local. Cross-server WebSocket delivery needs a routing fabric — you can’t just push from a worker that didn’t accept the connection.
• Store::publish is fire-and-forget pub/sub (cache invalidation, WS routing, presence). Store::publishReliable is at-least-once via Redis Streams (orders, audit, work queues).
• WSRouter bundles the “owner of fd pushes; everyone else publishes” pattern in five calls. Use the helper for new code; the manual four-step build is the mental model.
• One Store::defaultBackend(Store::BACKEND_REDIS) line takes the WHOLE app from single-box to multi-node — sessions, cache, counters, ws routing, the lot.

Want the portfolio-page deep dive? See /pubsub for the API reference + live demo, /ws#scaling for the cross-server scaling story, and /store#pubsub for the receiver-count semantics.