Federation

Your BBS talks to other BBSes.

In the 1980s and 90s, FidoNet proved that independent BBSes could form a network — dial-up calls, store-and-forward message packets, zone coordinators. A message typed in the Bronxville BBS on Monday morning could reach a sysop in Ohio by Tuesday night, hopping node to node. It was slow, fragile, and revolutionary.

The Exchange brings federation back, but with modern infrastructure underneath:

The design goal: make adding a new federated resource as cheap as writing a handler. The sysop declares the type, calls sync_create, and handles on sync_receive. Everything else — serialization, ID assignment, dedup, ordering, backfill, compression, tombstones, network retries — is the runtime’s job.


Declaring Federated Resources

Every BBS node has a node_name and a list of peers it federates with. Resources are federated by setting federated = true on the resource declaration. The default is false — anything not explicitly federated stays local and never leaves the node.

bbs "The Exchange" {
    node_name = "bronxville"
    federate_with = [
        "ohio.boremaj.net:2323",
        "florida.boremaj.net:2323"
    ]

    // Federated — posts sync to all connected peers
    board "General" {
        read = ["users"]
        write = ["users"]
        federated = true
    }

    // Local only — never replicated, never leaves this node
    board "Family Office" {
        read = ["vip"]
        write = ["vip"]
        federated = false
    }

    // Federated — messages relay across all nodes in real-time
    chat "#lobby" {
        topic = "The main lobby"
        federated = true
    }

    // Local only — sysop coordination stays on this node
    chat "#sysop" {
        topic = "SysOp coordination"
        federated = false
    }
}

How it works:

The node_name is your identity in the federation. Every envelope you originate carries this name as its origin field. Peers use it to track your sequence numbers, dedup your messages, and backfill from you on reconnect.

federate_with is a list of peer addresses (host:port). The transport layer (jnet) handles the actual TCP connections, reconnection with backoff, and peer authentication during the handshake.


Events vs Entities

Federated resources come in two flavors — events and entities. The distinction is fundamental: it determines how identity is assigned, whether updates are allowed, and how deletion works.

PropertyEvent (mode = "event")Entity (mode = "entity")
Identityorigin:sequence (from envelope)Natural key (declared fields)
UpdatesNot allowed — always new entriesUpsert by key
DeletionTombstone (marked deleted, not removed)Delete by key
DeduplicationAutomatic via sequence numberBy key — same key = same entity
Use caseTrade signals, chat messages, achievementsUser profiles, game scores, settings

Events — append-only, immutable

Events are the simple case. The runtime assigns identity as origin:sequence — for example, bronxville:4824. This pair is globally unique. You never specify an ID. You can’t update an event, can’t overwrite one, and can’t delete it (only tombstone it). Deduplication is automatic: if the same envelope arrives twice due to a network retry, the runtime sees it already has bronxville:4824 and skips it.

syncable "trade_signal" {
    mode = "event"
    // Identity = origin:sequence from the envelope.
    // No separate ID field. Can't update, can't be overwritten.
    // Deduplication is automatic via sequence number.

    fields = {
        symbol: String,
        action: String,      // "buy" or "sell"
        shares: Int,
        price: Float,
        user: String,
        timestamp: Time
    }

    conflict_resolution = "last_write_wins"
    retention = 7d          // purge after 7 days
    write = ["vip"]         // only vip role can create trade signals
}

Entities — updatable by natural key

Entities are updatable. You declare one or more fields as the natural key — the fields that uniquely identify an entity across the federation. sync_create is an upsert: if no local entry exists with that key, it creates one; if an entry exists, it updates it (running conflict resolution first).

syncable "user_profile" {
    mode = "entity"
    key = ["user_id"]       // which field(s) form the unique identity.
                            // sync_create upserts: creates if key doesn't exist,
                            // updates if it does (with conflict resolution).

    fields = {
        user_id: Int,
        display_name: String,
        bio: String,
        avatar: String,
        location: String
    }

    conflict_resolution = "merge"
    retention = forever     // entities are current state — keep until deleted
    write = ["users"]       // any logged-in user can update their profile
}

Composite keys

Keys can span multiple fields. This is useful for resources like game scores, where the identity is “this game + this user” — a single user can have scores in multiple games, but only one score per game.

syncable "door_game_score" {
    mode = "entity"
    key = ["game", "user"]  // unique per game+user. Updates overwrite the
                            // existing entry for the same game+user pair.

    fields = {
        game: String,
        user: String,
        score: Int,
        node: String,
        played_at: Time
    }

    conflict_resolution = "highest_value"  // highest score wins
    retention = forever
    write = ["users"]
}

Sync Functions

Three functions drive federated data: sync_create, sync_update, and sync_delete. The runtime wraps every call in a standard envelope, assigns identity, and pushes to peers. On the receiving side, on sync_receive fires for creates and updates; on sync_delete fires for tombstones.

Creating an event

// Event — always creates a new entry
on trade_executed {
    sync_create("trade_signal", {
        symbol: event.symbol,
        action: event.action,
        shares: event.shares,
        price: event.price,
        user: user.username,
        timestamp: now()
    })
    // Runtime wraps in envelope: {origin: "bronxville", sequence: 4824, ...}
    // Pushes to peers. Peers check: have I seen bronxville:4824? No → apply.
}

Creating/updating an entity (upsert)

// Entity — upsert by key
on profile_update {
    sync_create("user_profile", {
        user_id: user.id,
        display_name: "Joe",
        bio: "Updated bio",
        avatar: "joe.ans",
        location: "Bronxville"
    })
    // Runtime checks: does user_profile with key user_id=1 exist locally?
    // No → create. Yes → update (with conflict resolution).
    // Either way, pushes to peers. Peers upsert by the same key.
}

Updating specific fields

// Entity — update specific fields only
sync_update("user_profile", {
    user_id: user.id,
    bio: "New bio only"
})
// Finds existing entry by key user_id=1, merges the update.
// Fields not present in the update are left unchanged.

Deleting by key

// Entity — delete by key
sync_delete("user_profile", key: {user_id: 1})
// Removes entry with key user_id=1, sends tombstone to peers.
// Peers receive the tombstone and remove their local copy.

Receiving synced data — events

// Event — fires on append
on sync_receive type "trade_signal" {
    // envelope.origin and envelope.sequence identify this event
    // payload is the data fields
    say "#trading: Federated trade: ${payload.action} ${payload.shares} ${payload.symbol} @ $${payload.price} (from ${envelope.origin})"

    // Store locally — no dedup check needed, runtime already checked sequence
    sql_exec("INSERT INTO federated_trades (origin, sequence, symbol, action, shares, price, timestamp)
              VALUES (?, ?, ?, ?, ?, ?, ?)",
              envelope.origin, envelope.sequence, payload.symbol, payload.action,
              payload.shares, payload.price, payload.timestamp)

    fire("federated_trade", payload)
}

Receiving synced data — entities

// Entity — fires on create or update
on sync_receive type "user_profile" {
    // entry exists locally if this was an update, null if new
    existing = sql_one("SELECT * FROM federated_profiles WHERE user_id = ?", payload.user_id)

    if existing == null {
        sql_exec("INSERT INTO federated_profiles (user_id, display_name, bio, avatar, location, origin)
                  VALUES (?, ?, ?, ?, ?, ?)",
                  payload.user_id, payload.display_name, payload.bio, payload.avatar,
                  payload.location, envelope.origin)
    } else {
        // Conflict resolution already applied by runtime, just store the result
        sql_exec("UPDATE federated_profiles SET display_name = ?, bio = ?, avatar = ?, location = ?
                  WHERE user_id = ?",
                  payload.display_name, payload.bio, payload.avatar, payload.location, payload.user_id)
    }
}

// Entity — fires on delete (tombstone received from peer)
on sync_delete type "user_profile" {
    sql_exec("DELETE FROM federated_profiles WHERE user_id = ?", payload.user_id)
}

Built-in types

Built-in resources like board posts and chat messages are already syncable — you don’t declare them. Just handle the receive event:

on sync_receive type "board_post" {
    // Runtime already inserted into bbs.posts
    say "#lobby: New federated post from ${envelope.origin}: ${payload.subject}"
}

Conflict Resolution

When two nodes update the same entity, the runtime resolves the conflict before the handler sees the data. The handler receives the resolved result and simply stores it. Six strategies are built in:

StrategyDescriptionUse case
last_write_winsMost recent timestamp winsChat topics, board posts
first_write_winsEarliest timestamp winsAchievements, immutable events
highest_valueHighest numeric field winsHigh scores, rankings
lowest_valueLowest numeric field winsResponse time records
mergeMerge non-conflicting fields, last-write-wins on conflictsUser profiles
custom: function_nameCustom Phosphor functionComplex merge logic

For events, there’s no conflict — they’re append-only. The strategy only matters for entities, where two nodes may update the same key concurrently.

Using a built-in strategy

syncable "user_profile" {
    fields = {
        user_id: Int,
        display_name: String,
        bio: String,
        avatar: String,
        location: String
    }
    conflict_resolution = "merge"
    retention = forever
    write = ["users"]
}

Custom conflict resolver

When the built-in strategies aren’t enough, you can write a Phosphor function that receives both the local and remote versions and returns the merged result. The runtime calls your function instead of applying a built-in strategy:

// Custom conflict resolver — called by the runtime when a conflict is detected
function resolve_profile_conflict(local, remote) {
    // Merge: take remote display_name if newer, keep local bio, merge location
    return {
        user_id: local.user_id,
        display_name: if remote.timestamp > local.timestamp then remote.display_name else local.display_name,
        bio: local.bio,                              // local wins for bio
        avatar: remote.avatar,                        // remote wins for avatar
        location: remote.location ?? local.location   // remote if set, else local
    }
}

syncable "user_profile" {
    mode = "entity"
    key = ["user_id"]
    fields = {
        user_id: Int,
        display_name: String,
        bio: String,
        avatar: String,
        location: String
    }
    conflict_resolution = "custom: resolve_profile_conflict"
    retention = forever
    write = ["users"]
}

The function receives the local and remote entity records and returns the merged entity. The runtime then hands the result to on sync_receive, which stores it. The handler doesn’t decide who wins — it just persists the runtime’s decision.


Envelope Format

Every federated message is wrapped in a standard envelope. This is the single wire format that carries any Phosphor data type — board posts, chat messages, custom syncables, everything. The envelope is what makes adding a new federated resource cheap: no custom serialization, no custom jnet message types.

Structure

FieldTypeDescription
protocolStringAlways "phosphor-sync/1.0" — identifies the wire format
typeStringThe syncable type: "board_post", "chat_message", "trade_signal", "user_profile", or any custom type
originStringThe node_name that created this entry (e.g. "bronxville")
timestampStringISO-8601 UTC timestamp of creation (e.g. "2026-07-30T14:32:01Z")
sequenceIntegerMonotonic per-origin counter for ordering and dedup
resourceStringBoard name, channel name, or resource key
actionString"create", "update", or "delete"
payloadObjectThe actual data — type-specific fields
signatureStringHMAC for integrity (optional)

Example envelope

{
    "protocol": "phosphor-sync/1.0",
    "type": "trade_signal",
    "origin": "bronxville",
    "timestamp": "2026-07-30T14:32:01Z",
    "sequence": 4823,
    "resource": "trade_signal",
    "action": "create",
    "payload": {
        "symbol": "AAPL",
        "action": "buy",
        "shares": 100,
        "price": 212.50,
        "user": "joe",
        "timestamp": "2026-07-30T14:32:01Z"
    },
    "signature": "a1b2c3d4e5f6..."
}

The origin:sequence pair (bronxville:4823) is the global identity of this event. No other event will ever have that pair. The runtime uses it for deduplication and backfill tracking.

The signature field is optional. If the node has federation security configured, the runtime computes an HMAC over the envelope contents and includes it. Peers verify the signature on receipt; if it doesn’t match, the envelope is rejected.


Layered Runtime

The sync system is layered. Each layer has a clear responsibility, and handlers only touch the top layer. This separation is what lets sysops write federated features in 10 lines of Phosphor instead of 200 lines of Java.

The four layers

Layer 1 — Handler (Phosphor .phos code)

This is what the sysop writes:

What the handler does NOT deal with:

Layer 2 — Sync Runtime (Java, inside PhosphorRuntime)

The runtime does automatically:

Layer 3 — Transport (jnet mesh — existing Java)

The transport layer knows nothing about Phosphor types, envelopes, syncables, or conflict resolution. It just moves bytes between nodes.

Layer 4 — Storage (Postgres — existing)

The runtime manages its own tables. The handler manages its own tables. They don’t share tables.

Data flow diagram

SENDING
─────────────────────────────────────────────────────────────
Handler calls sync_create("trade_signal", {symbol: "AAPL", ...})
    │
    ▼
Runtime: assign sequence number (bronxville:4824)
    │
    ▼
Runtime: wrap in envelope {type, origin, sequence, payload, action, timestamp}
    │
    ▼
Runtime: compress + batch with other pending envelopes
    │
    ▼
Runtime: hand batch to jnet transport
    │
    ▼
jnet: deliver to all connected peers (ohio, florida)


RECEIVING
─────────────────────────────────────────────────────────────
jnet: batch arrives from peer (ohio)
    │
    ▼
Runtime: decompress + unbatch
    │
    ▼
Runtime: for each envelope:
    │
    ├─ Event? check origin:sequence in seen table
    │   ├─ Already seen → skip (dedup)
    │   └─ New → record in seen table, apply
    │
    ├─ Entity? resolve by natural key
    │   ├─ No local entry → create (fire on sync_receive)
    │   ├─ Local entry exists → run conflict resolution
    │   │   ├─ Result differs → update (fire on sync_receive)
    │   │   └─ Result same → skip (no change)
    │   └─ Tombstone (action=delete) → remove (fire on sync_delete)
    │
    ▼
Handler: on sync_receive type "trade_signal" { ... }
    │
    ▼
Handler: sql_exec("INSERT INTO federated_trades ...")
    (handler stores in its own table)

Backfill on Reconnect

Nodes go offline. Network drops happen. When a peer reconnects, it needs to catch up on everything it missed — in order, with conflict resolution, exactly as if the entries had arrived live.

The runtime tracks last_sequence_seen for every (origin, syncable_type) pair. On reconnect, it sends a backfill request to the peer: “give me all entries after sequence N.” The peer responds with the gap from its envelope log. The runtime applies each entry in sequence order, running the same receive flow as live messages.

// Automatic on peer connect — the runtime handles the backfill request itself.
// The sysop can observe the event:
on peer_connected {
    say "#sysop: Peer ${peer.node_name} connected (${peer.address})"
    // Runtime sends: "Give me all syncable entries since sequence 4823"
    // Peer responds with all entries from sequence 4824 onward
    // The runtime applies them in order, running conflict resolution on each
    // The handler sees on sync_receive as if they arrived live

    say "#sysop: Syncing ${sync.pending_count} entries from ${peer.node_name}"
}

on peer_disconnected {
    say "#sysop: Peer ${peer.node_name} disconnected"
}

on sync_complete {
    say "#sysop: Sync complete — ${sync.received} entries received, ${sync.conflicts_resolved} conflicts resolved"
}

Backfill data flow

jnet: peer (ohio) reconnects after downtime
    │
    ▼
Runtime: check last_sequence_seen for ohio → 4800
    │
    ▼
Runtime: send backfill request to ohio: "send entries 4801+"
    │
    ▼
jnet: ohio responds with entries 4801-4830
    │
    ▼
Runtime: apply each entry in order (same receive flow as live)
    │
    ▼
Runtime: update last_sequence_seen for ohio → 4830
    │
    ▼
Handler: fires on sync_receive for each entry, as if they arrived live

The handler code is identical whether entries arrive live or via backfill. The handler doesn’t know the difference. It just sees on sync_receive fire for each entry in order.

Runtime-managed tables

The runtime maintains two tables that the handler never touches:

-- Tracks which sequence numbers we've seen from each origin
-- (for dedup + backfill)
CREATE TABLE bbs.phosphor_sync_state (
    origin           VARCHAR(64) NOT NULL,
    syncable_type    VARCHAR(128) NOT NULL,
    last_sequence_seen BIGINT NOT NULL,
    last_updated     TIMESTAMPTZ DEFAULT now(),
    PRIMARY KEY (origin, syncable_type)
);

-- Envelope log — stores all received envelopes for retention period
-- (used for serving backfill requests to other nodes)
CREATE TABLE bbs.phosphor_sync_envelopes (
    id           BIGSERIAL PRIMARY KEY,
    origin       VARCHAR(64) NOT NULL,
    sequence     BIGINT NOT NULL,
    syncable_type VARCHAR(128) NOT NULL,
    action       VARCHAR(16) NOT NULL,    -- create, update, delete
    payload      JSONB NOT NULL,
    received_at  TIMESTAMPTZ DEFAULT now(),
    expires_at   TIMESTAMPTZ,             -- null for entities, set for events
    UNIQUE (origin, sequence, syncable_type)
);

-- Index for backfill queries
CREATE INDEX idx_sync_envelopes_backfill
    ON bbs.phosphor_sync_envelopes (origin, syncable_type, sequence);

When a peer asks for backfill, the runtime queries phosphor_sync_envelopes for entries from that origin after the requested sequence number. This is why every received envelope is logged — the log is the source of truth for backfill.


Tombstones and Retention

How deletes propagate

When you call sync_delete, the runtime doesn’t just remove the local entry — it sends a tombstone to all peers. A tombstone is an envelope with action = "delete". Peers that receive the tombstone remove their local copy and store the tombstone itself (for retention tracking).

// Local delete — removes entry and sends tombstone
sync_delete("user_profile", key: {user_id: 1})

On the receiving side:

// Fires when a tombstone arrives from a peer
on sync_delete type "user_profile" {
    sql_exec("DELETE FROM federated_profiles WHERE user_id = ?", payload.user_id)
}

Tombstones are stored in the envelope log with action = "delete". They expire after the retention period, at which point the runtime purges them. This prevents tombstones from accumulating forever while still ensuring deletes propagate to peers that reconnect later.

Event expiry

Events have a retention period. The runtime purges event entries older than the retention window. For example, retention = 7d means trade signals older than 7 days are automatically removed from the envelope log. The expires_at column in phosphor_sync_envelopes is set for events and null for entities.

Entity persistence

Entities are current state, not history. They’re kept until explicitly deleted — retention = forever is the norm. There’s no expiry. The runtime doesn’t purge entities based on age; it only removes them when a tombstone arrives or when the handler calls sync_delete.

Resource typeRetention behavior
Event (mode = "event")Purged after retention period. expires_at is set.
Entity (mode = "entity")Kept until deleted. expires_at is null.
TombstonePurged after retention period.

Security

Federation is powerful, but you need to control who can write what. The runtime provides three layers of security:

HMAC signatures (optional)

If the node has federation security configured, the runtime computes an HMAC over the envelope contents and includes it in the signature field. Peers verify the signature on receipt. If it doesn’t match — tampered in transit, or sent by an impostor — the envelope is rejected and logged.

This is opt-in. If you’re running a trusted mesh on a private network, you may skip it. If you’re federating across the public internet, you should enable it.

Peer authentication

The jnet transport handles peer authentication during the connection handshake. Before any sync data flows, jnet verifies the peer’s identity. Unauthenticated peers don’t receive federated data. This is existing jnet functionality — Phosphor builds on top of it.

Write permissions

Every syncable declaration includes a write field that restricts who can create, update, or delete entries:

syncable "trade_signal" {
    mode = "event"
    fields = { symbol: String, action: String, shares: Int, price: Float, user: String, timestamp: Time }
    conflict_resolution = "last_write_wins"
    retention = 7d
    write = ["vip"]      // only vip role can create trade signals
}

syncable "user_profile" {
    mode = "entity"
    key = ["user_id"]
    fields = { user_id: Int, display_name: String, bio: String, avatar: String, location: String }
    conflict_resolution = "merge"
    retention = forever
    write = ["users"]    // any logged-in user can update profiles
}

syncable "user_achievement" {
    mode = "event"
    fields = { user_id: Int, badge_name: String, awarded_at: Time }
    conflict_resolution = "first_write_wins"
    retention = forever
    write = ["sysop"]    // only sysops award achievements
}

The write field is a list of roles. Only users with one of those roles can call sync_create, sync_update, or sync_delete for that type. The runtime enforces this before any envelope is created.


A Real Example: Board Post from Node A to Node B

Let’s trace a federated board post as it flows from the Bronxville node to the Ohio node, step by step. This shows every layer in action.

Node A — Bronxville (sending)

A user posts to the “General” board, which is federated = true.

Step 1 — Handler creates the post:

on board_post board "General" {
    // User submitted a post. The BBS already stored it locally in bbs.posts.
    // Now we sync it to peers.
    sync_create("board_post", {
        board: "General",
        author: user.username,
        subject: event.subject,
        body: event.body,
        posted_at: now()
    })
}

Step 2 — Runtime assigns identity:

The runtime assigns origin = "bronxville" and sequence = 4824 (the next monotonic counter for this origin). No ID field is needed — the identity is bronxville:4824.

Step 3 — Runtime wraps in envelope:

{
    "protocol": "phosphor-sync/1.0",
    "type": "board_post",
    "origin": "bronxville",
    "timestamp": "2026-07-30T14:32:01Z",
    "sequence": 4824,
    "resource": "General",
    "action": "create",
    "payload": {
        "board": "General",
        "author": "joe",
        "subject": "Re: Market open",
        "body": "Futures are green. Let's see if it holds.",
        "posted_at": "2026-07-30T14:32:01Z"
    },
    "signature": "f3a1b2c3d4e5..."
}

Step 4 — Runtime compresses and batches:

If there are other pending envelopes, the runtime groups them (up to 100 per batch) and gzip-compresses if the batch exceeds 1KB. In this case, it’s just one envelope, so it goes through uncompressed.

Step 5 — Runtime hands batch to jnet transport:

The runtime passes the envelope to jnet, which handles the actual TCP delivery.

Step 6 — jnet delivers to all peers:

jnet sends the envelope to every connected peer in federate_with — in this case, ohio.boremaj.net:2323 and florida.boremaj.net:2323.

Node B — Ohio (receiving)

Step 7 — jnet receives the envelope:

The jnet transport layer on the Ohio node receives the message frame and hands it to the sync runtime.

Step 8 — Runtime decompresses and unbatches:

The runtime unpacks the batch. One envelope: bronxville:4824, type board_post, action create.

Step 9 — Runtime checks dedup:

The runtime looks up phosphor_sync_state for (origin = "bronxville", syncable_type = "board_post"). The last sequence seen was 4823. This is 4824 — new. Record it.

If the same envelope had arrived twice (network retry), the runtime would see 4824 is already in the seen table and skip it. Dedup is automatic.

Step 10 — Runtime applies conflict resolution:

board_post is an event (append-only), so there’s no conflict — the runtime always appends. For an entity, it would resolve by key and run the declared conflict strategy.

Step 11 — Runtime stores in envelope log:

The envelope is written to phosphor_sync_envelopes so Ohio can serve it as backfill to other nodes later. expires_at is set based on the retention period.

Step 12 — Runtime fires the handler:

on sync_receive type "board_post" {
    // Runtime already inserted into bbs.posts
    say "#lobby: New federated post from ${envelope.origin}: ${payload.subject}"
}

The handler fires. envelope.origin is "bronxville". payload.subject is "Re: Market open". The runtime has already inserted the post into bbs.posts (built-in type), so the handler just announces it to the lobby.

Step 13 — Ohio’s users see the post:

A user browsing the “General” board on the Ohio node now sees Joe’s post from Bronxville. It’s stored locally, so it persists even if the connection drops. If Ohio later backfills to a third node (Florida), it can forward this same envelope — bronxville:4824 — and Florida will dedup it the same way.

What the handler never saw

Throughout this entire flow, the Ohio handler:

The handler declared the type, called sync_create (on the sending side), and handled on sync_receive (on the receiving side). That’s it. The runtime, transport, and storage layers handled everything else.


Adding a new federated type takes 3 steps

  1. Declare it with syncable "name" { ... }
  2. Create it with sync_create("name", data) from any handler
  3. Handle it with on sync_receive type "name" { ... } on the receiving side

No custom Java serialization. No custom jnet message types. No custom deserializer. The generic envelope handles everything. That’s the point — you go from “write 200 lines of Java protocol code” to “write 10 lines of Phosphor.”