RoutedRaftTransport

Deliver Raft messages to nodes that sit behind another server, without ever losing track of who really sent each one — the routed Raft transport.

A federated game runs one Raft cluster whose members are spread across several servers: the servers form a small fully-meshed core, and each player connects to whichever server is nearest. An ordinary transport can only talk to nodes it is directly wired to, so a leader on one server could never reach a player behind a different one — that player would never receive the committed log. This decorator fixes that by relaying: when a message is addressed to a node it cannot reach directly, it wraps the message in a RaftRelay and hands it one hop closer, along the bounded path player → server → core → server → player. Every hop is a single-addressee send — the relay is never a broadcast, so a message meant for one node never leaks to a second.

It disappears when there is nothing to relay

Wrapping inner is inert off a federation. When every addressee is a direct peer of inner (a single server, a LAN, an in-memory test), sendTo delegates straight through and no relay frame is ever produced. The decorator only does work once a message is addressed to a node that is not directly reachable.

Preserving the true origin (why this exists at all)

On a direct link the fabric stamps the sender for you, and it can never be forged. Relaying breaks that: the frame's fabric-sender becomes the relaying server, not the node the Raft engine must credit. So the real sender rides inside the RaftRelay as its origin, and this transport preserves it verbatim — a relayed frame surfaces as RaftEnvelope(from = origin), never the relay's own sender. (The engine keys vote tallies, matchIndex, CheckQuorum, ReadIndex acks and leadership-transfer auth on from; re-stamping would break every one.)

First-hop origin validation (commit-safety)

Because origin now travels inside a forgeable frame, it is validated before a relayed message is ever handed to the engine or forwarded on. The two roles validate differently:

A server applies the sender-based first-hop rule (validFirstHop):

  • A frame from a spoke (the fabric-sender is not a core member) is accepted only if its origin equals that sender — a player may speak only for itself, never forge a vote or a matchIndex-advancing response on another node's behalf.

  • A frame from the core (the fabric-sender is a core member) is trusted to carry an already-validated origin; core servers preserve identity.

  • A frame that arrived from the core and is not for a locally reachable node is dropped, never re-forwarded onto the core — the loop guard that keeps the hop bound at one core crossing.

A player cannot trust the sender: its one peer (the relay server) need not be a voter, and a co-player behind the same server could wrap a forged AppendEntries as RaftRelay(origin = self, dest = victim, …) that the server forwards down. So a player instead trusts a down-frame only when its true origin is a known voter — read live via voters on every frame so a committed membership growth is honoured. This is strictly tighter than the sender rule and closes that log-corruption vector: the victim's engine does no from validation and would otherwise truncate-and-append the forged log.

The spoke→voter reach, and how it is contained (#1383)

This relay is deliberately reachability-complete: the bounded spoke → core → core → spoke path lets any admitted learner address every cluster member with honest-origin frames. That reach is the point — it is how a far player's AppendEntries response reaches a leader on another server. Both directions are now hardened:

  • Origin spoofing is blocked in every direction. A spoke cannot claim to be another node: the first-hop rule (validFirstHop) rejects a spoke frame whose origin isn't the sender, and the player-side origin ∈ voters() gate rejects a fellow-spoke's forged down-frame. So no node is ever impersonated.

  • A voter accepting an honest-origin RPC it should never process is blocked too. A malicious-but-admitted learner can address a voter with a RaftRelay carrying its own honest origin but an AppendEntries / InstallSnapshot body — RPC types only a leader should originate. validFirstHop passes (origin == sender) and the frame reaches the engine, so the containment cannot live here at the send-side decorator. It lives at the engine: RaftEngine.onMessage applies a §5.2/§8 leader-authority gate — an AppendEntries/InstallSnapshot whose sender is not a current voter (membershipState.isVoter) is dropped before dispatch. Only the engine can do this: the RPC type is an internal RaftMessage the relay layer cannot decode, and the engine's membershipState is the live committed voter set. Without that gate an accepted forged AppendEntries truncates-and-appends the voter's log and an InstallSnapshot overwrites its state — log corruption, not merely the term-inflation a spoof-only view would suggest (votes and matchIndex are keyed on a validated from, but the log itself was not). The star topology this replaced happened to confine such a frame to one server; the engine gate restores that containment for the identity-preserving cluster-wide reach.

One class, two roles

A isServer server may take one core hop when it holds a frame for a player behind another server (its next hop comes from attachment, the live "which server is this player behind" lookup). A player always forwards to its single server (the sole non-self peer of relayChannel) and never routes for anyone else. Build a server via serverRelayTransport and a player via playerRelayTransport.

Ownership & threading

Takes sole ownership of relayChannel's incoming stream (a single relay coroutine, launched in scope, pulls envelopes off it), per the single-collection contract — do not run another collector over the same seam. incoming merges inner's own frames with the relayed frames destined for this node. There is no shared mutable state: routing reads only the live inner/relayChannel peer sets and the injected attachment function, so the transport is correct under a multi-threaded dispatcher.

Parameters

inner

the direct transport this wraps (typically a SeamRaftTransport). Sends to a direct peer, and this node's own directly-received frames, pass through it unchanged.

relayChannel

the seam relay envelopes are sent and received on. For a server it reaches the local players and the other core servers; for a player it reaches its one server. The transport owns its incoming.

core

the node ids of the fully-meshed server core — the sender-based trust boundary for a server's first-hop validation. Unused by a player (which validates by origin against voters); pass the empty set.

isServer

whether this endpoint is a core server (may take one core hop) or a player (always forwards to its one server).

scope

the CoroutineScope whose Job parents the relay coroutine. Required — no real-dispatcher default; inject a test scope's backgroundScope under virtual time.

attachment

for a server, the live (player) -> the server it is behind lookup used to pick the core hop for a remote player; ignored by a player (which always forwards to its one server). Pass AttachmentDirectory.lookup-style function on a server.

voters

for a player, the live provider of the current voter node ids — a down-frame is accepted only when its origin is one of these, read per frame so a committed membership growth is picked up (never captured at construction). Unused by a server; pass { emptySet() }.

headerBudget

the RaftRelay envelope allowance subtracted from inner's frame limit; defaults to RELAY_HEADER_BUDGET.

Properties

Link copied to clipboard
open override val incoming: Flow<RaftEnvelope>
Link copied to clipboard
open override val maxPayloadBytes: Int?

The largest payload a single sendTo may carry: inner's limit less the headerBudget the RaftRelay envelope consumes, so a ceiling-sized chunk plus its envelope still fits a framed fabric. null (unbounded) when inner is unbounded — an unbounded fabric never chunks, so the envelope's few bytes are immaterial.

Link copied to clipboard
open override val peers: StateFlow<Set<NodeId>>

Reachability as inner sees it — the directly wired nodes. Nodes reachable only across the core are absent here; the Raft engine still addresses them (it sends to configured voters regardless of reachability) and sendTo relays those.

Link copied to clipboard
open override val selfId: NodeId

Functions

Link copied to clipboard
fun close()

Stop relaying and release the relay coroutine. Idempotent.

Link copied to clipboard
open suspend override fun sendTo(peer: NodeId, message: ByteArray)