Preplay Shreds is now a Yellowstone stream you can filter

Preplay Shreds now serves Yellowstone gRPC's SubscribeDeshred. You send account filters, and the stream carries only the transactions that match, decoded, with their lookup-table addresses resolved. What you get is unchanged: every transaction a leader produces, in leader order, before the block is finished or replayed.

Same endpoint, same API key.

What changed

Until now Preplay Shreds used SubscribeEntries. It took an empty request and sent every batch of ledger entries as raw bytes. You decoded them yourself, then threw away most of what you received.

Vote transactions are not sent.

Filters

A subscription carries up to 16 named filters, each with three account lists:

• account_include: the transaction uses at least one of these
• account_exclude: it uses none of these
• account_required: it uses all of these

Accounts are matched against the transaction's static keys and the addresses it loads from lookup tables, so a swap that reaches a pool through a lookup table still matches on the pool. Each message names the filters it matched, so one stream can feed several consumers. To change filters, send a new request on the same stream; there is no need to reconnect.

A filter with no accounts matches every transaction. Nothing is delivered until you send at least one filter.

Lookup tables

A v0 transaction names some of its accounts by lookup table and index. With raw entries, resolving them meant reading each table over RPC while the stream kept moving. Each message now carries loaded_writable_addresses and loaded_readonly_addresses. Instruction account indexes point into the static keys, then the loaded writable addresses, then the loaded read-only ones, which is the order the runtime uses:

let keys: Vec<&[u8]> = message
    .account_keys
    .iter()
    .chain(&info.loaded_writable_addresses)
    .chain(&info.loaded_readonly_addresses)
    .map(Vec::as_slice)
    .collect();

Every instruction account resolves from that list with no RPC call. In the rare case a table cannot be resolved, both loaded lists are empty while address_table_lookups is not. Treat that transaction as incomplete.

Yellowstone clients

SubscribeDeshred is part of the upstream Yellowstone protocol, so there is no Sodae-specific proto to vendor. yellowstone-grpc-client 14 for Rust and @triton-one/yellowstone-grpc 7 for TypeScript both support it, and Go clients generate from the upstream geyser.proto. Send your key as x-token, as with Yellowstone gRPC.

let request = SubscribeDeshredRequest {
    deshred_transactions: HashMap::from([(
        "pumpswap".to_string(),
        SubscribeRequestFilterDeshredTransactions {
            account_include: vec!["pAMMBay6oceH9fJKBRHGP5D4bD4sWpmSwMn52FMfXEA".to_string()],
            ..Default::default()
        },
    )]),
    ..Default::default()
};
let (mut sink, mut updates) = client.subscribe_deshred_with_request(Some(request)).await?;

Staying connected

The server sends a ping every 15 seconds. Reply with a request carrying ping so proxies between you and us keep the connection open; the reply leaves your filters alone.

Read continuously. A client that falls behind, or reads nothing for 10 seconds, is disconnected with DATA_LOSS. If you cannot keep up with the full stream, narrow your filters.

Data use

Per-GB plans and trials count the bytes streamed to you, so a filter that narrows the stream also cuts what you use. For the full stream around the clock, the fixed-price Binary Decoded Shreds plan has no data cap.

Migrating

A SubscribeEntries client needs four changes:

1. Use a Yellowstone client, or generate from geyser.proto and solana-storage.proto.
2. Open SubscribeDeshred and send a request with at least one filter. An empty filter, {}, gets you everything.
3. Drop the entry decoder. Read transaction.message and resolve accounts with the loaded address lists.
4. Answer pings.

Keep the maximum decoded message size at 64 MB or more.

Examples

The Rust, TypeScript and Go examples in sodae-io/sodae-docs now use SubscribeDeshred. Without an argument they print each slot's transaction count. Given a program id, they ask the server for only that program's transactions and print each one's signer, version and instructions.

git clone https://github.com/sodae-io/sodae-docs
cd sodae-docs/examples/rust
SODAE_TOKEN=... cargo run --example preplay -- pAMMBay6oceH9fJKBRHGP5D4bD4sWpmSwMn52FMfXEA

The docs cover the rest: the API and filter rules, the transaction format and the examples.

References

• yellowstone-grpc #684, which added SubscribeDeshred to the proto and Rust client
• yellowstone-grpc changelog
• geyser.proto and solana-storage.proto, the definitions Preplay Shreds serves

Discord / X