NNTP pipelining1.x
Pipelining keeps a Usenet connection busy by asking for the next articles before the previous ones finish arriving. Depth is how many asks stay outstanding on that connection (default 8 for queue imports).
OffOne ask at a time
On, depth 3Three asks in flight
- Ask sent
- Waiting on the provider
- Article arriving
Three controls
Section titled “Three controls”| Setting | What it does |
|---|---|
usenet.queue-pipelining.enabled |
Queue first-segment fetch and provider benchmark batches. Location: Settings → Usenet. Default off. |
usenet.pipelined-body-requests |
Enable WebDAV streaming BODY batches. Location: Settings → Streaming. Default on. |
usenet.streaming-body-batch-width |
Maximum articles per streaming BODY batch (1–8). Location: Settings → Streaming. Default 4. |
Legacy keys usenet.pipelining.enabled and usenet.pipelining.depth remain honored as fallbacks (including via NZBDAV_CONFIG__USENET__PIPELINING__* env vars). Upgrades copy their SQLite values into the new queue keys; rename env vars to NZBDAV_CONFIG__USENET__QUEUE_PIPELINING__* when possible.
What queue pipelining speeds up
Section titled “What queue pipelining speeds up”| Path | Without | With |
|---|---|---|
| Queue first-segment fetch (0→50%) | one BODY per file across connections | depth-sized batches on a connection |
| Provider benchmark | one BODY per article | depth-sized batches |
Health/import existence checks use concurrent STAT and are unaffected. Queue pipelining does not change WebDAV playback — that path uses the streaming toggles above.
How connections, batch width, and memory interact
Section titled “How connections, batch width, and memory interact”During WebDAV playback with batched BODY requests enabled:
- Connections engaged per stream ≈ outstanding segment window ÷ batch width.
- At stream construction, infinidysk sizes the segment task window and prefetch byte ceiling from the configured batch width and article buffer. Those ceilings stay fixed for the life of the stream; adaptive narrowing only shrinks future batch sizes, not the retained window.
- Decoded bytes retained across all streams are capped host-wide by
usenet.in-flight-article-budget-mb(25% of the detected managed-heap ceiling by default, clamped to about 67 MB–8.6 GB). One stream with a wide batch width can consume most of that budget and starve other concurrent viewers.
Provider behavior varies: some throttle per connection (wider batches can help), others per account (more connections / narrower batches). If Auto-tune reports queue pipelining is unsafe for a provider, treat wide streaming batch widths cautiously too — both use the same NNTP pipelining mechanism.
Interleaved streaming batches
Section titled “Interleaved streaming batches”A connection returns a batch’s articles one after another, so a batch of consecutive articles makes playback wait on that single connection for each next article while other connections sit on finished data. WebDAV playback therefore interleaves batches across the connections expected for the stream: with 4 connections and width 4, the first batch asks for articles 1, 5, 9, 13, the second for 2, 6, 10, 14, and so on, and the next articles playback needs arrive in parallel.
The batch width setting keeps its meaning (articles per batch), and the bytes served are unchanged. When the decoded-article budget cannot hold a whole interleaved group at once, the stream falls back to consecutive batches rather than waiting, so interleaving adds no new partial-reservation wait between concurrent viewers. Consecutive batches reserve the budget the same way they did before interleaving.
The number of interleaved batches is the stream’s connection target when playback opens, counting only providers with byte quota left. It is not re-measured while connections are still opening: on real providers, interleaving across connections that are still connecting reached the next articles sooner than waiting for them to open. If a batch then has to wait for a connection because the stream lost one it had been using (another viewer or a health check took it), the stream interleaves across only the connections still downloading for it. It widens again when more of its batches run at once. Articles already in the segment cache or repair store are handed to playback straight away; they do not wait for the connection that fetches the rest of their batch.
Enabling
Section titled “Enabling”- Prefer Auto-tune on a provider before enabling queue pipelining.
- Settings → Usenet → Queue pipelining + queue depth (1–64, default 8). Per-provider depth overrides optional.
- Streaming: Batched article downloads and optional Streaming batch width on the Streaming tab.
Limitations
Section titled “Limitations”- Pipelined batches use the same per-segment failover as
DecodedBodiesAsync. - Per-queue-item article cache can bypass pipelined queue paths when caching is enabled.