Python Spool Rollover: Zero Thresholds and fileno()

If a SpooledTemporaryFile(max_size=0) keeps growing in memory, do not interpret zero as “spill after zero bytes.” In the CPython versions tested here, ordinary writes with this default do not trigger size-based rollover. Set a positive threshold for a small-file spool, or choose TemporaryFile() when the workflow needs a temporary filesystem file from the start.

The other surprising boundary is fileno(): asking for a descriptor can trigger rollover even when the content is tiny. A size check that calls os.fstat(spool.fileno()) therefore changes the storage state it is trying to inspect.

This guide tests the synchronous standard-library class in binary mode. It does not establish the behavior of an asynchronous wrapper or every framework's upload configuration. A historical reader report identifies the zero-threshold ambiguity; its age is not evidence of current search volume.

Observe rollover without inspecting private state

Save this as spool-boundaries.py and run python3 spool-boundaries.py. The subclass records calls to the public rollover method. All files contain disposable test bytes, and context managers close them. The probe does not use directory listings, a guessed filename, or the private _rolled attribute as a production contract.

import os
import tempfile

class ObservedSpool(tempfile.SpooledTemporaryFile):
    def __init__(self, *args, **kwargs):
        self.rollover_calls = 0
        super().__init__(*args, **kwargs)

    def rollover(self):
        self.rollover_calls += 1
        return super().rollover()

with ObservedSpool(max_size=0, mode="w+b") as spool:
    spool.write(b"abcdef")
    spool.flush()
    assert spool.rollover_calls == 0
    print("zero, after write and flush:", spool.rollover_calls)
    spool.seek(2)
    descriptor = spool.fileno()
    assert spool.rollover_calls == 1
    assert spool.tell() == 2
    assert os.fstat(descriptor).st_size == 6
    assert spool.read() == b"cdef"
    print("zero, after fileno:", spool.rollover_calls)
assert spool.closed

with ObservedSpool(max_size=8, mode="w+b") as spool:
    spool.write(b"12345678")
    assert spool.rollover_calls == 0
    print("eight-byte limit, eight written:", spool.rollover_calls)
    spool.write(b"9")
    assert spool.rollover_calls == 1
    spool.seek(0)
    assert spool.read() == b"123456789"
    print("eight-byte limit, nine written:", spool.rollover_calls)
assert spool.closed

with ObservedSpool(max_size=1024, mode="w+b") as spool:
    spool.write(b"abcdef")
    spool.seek(2)
    spool.rollover()
    assert spool.rollover_calls == 1
    assert spool.tell() == 2
    assert spool.read() == b"cdef"
    print("explicit rollover preserved position and bytes")
assert spool.closed

Our runs on CPython 3.11.6, 3.13.2 and 3.14.3 produced the same five lines: zero calls after the default-threshold write and flush, one after fileno(), zero at exactly eight bytes, one at nine bytes, and the final explicit-rollover confirmation. These observations cover these small binary examples; they are not a memory benchmark or a guarantee about all operations.

Notice that the exact threshold is tested separately from exceeding it. A threshold of eight is deliberately small to make the boundary visible. It is not a recommended deployment setting. We also check the file position after rollover, because a consumer reading from the wrong offset can receive a suffix or empty input even when every byte was preserved.

Choose storage policy before passing the object downstream

For ordinary small-file buffering, specify a positive value such as max_size=1024 * 1024. Choose it from the application's measured concurrency and typical payloads. The threshold concerns spool storage; it does not reject oversized input. Once rollover has happened, subsequent data can continue growing in the temporary filesystem.

If a downstream API requires an OS descriptor, expect disk-backed temporary storage when it requests fileno(). Raising the threshold will not prevent that descriptor-triggered change. If the consumer accepts a file-like object, pass that object using its documented interface; do not request a descriptor merely to prove it is “a real file.”

For a workflow that deliberately needs filesystem storage immediately, the following standalone example avoids the ambiguous zero setting entirely:

import tempfile

with tempfile.TemporaryFile(mode="w+b") as spool:
    spool.write(b"disposable payload")
    spool.seek(0)
    assert spool.read() == b"disposable payload"
assert spool.closed
print("temporary file closed")

The current tempfile documentation describes threshold, descriptor and explicit rollover. It also explains why a TemporaryFile need not have a visible directory entry on Unix. Do not use the absence of a new filename as evidence that rollover failed. Filesystem backing also does not prove a physical disk write or durable persistence; operating-system caching is a separate layer.

Measure length without casually requesting a descriptor

For a seekable binary stream owned by the current operation, save the current position, seek to the end, record that offset, then restore the position in a finally block. Here is a complete example using a standard spool:

import tempfile

def binary_length_preserving_position(stream):
    position = stream.tell()
    try:
        return stream.seek(0, 2)
    finally:
        stream.seek(position)

with tempfile.SpooledTemporaryFile(max_size=1024, mode="w+b") as spool:
    spool.write(b"abcdef")
    spool.seek(2)
    assert binary_length_preserving_position(spool) == 6
    assert spool.tell() == 2
    assert spool.read() == b"cdef"
print("length checked and reader position restored")

This is our tested helper for a seekable binary object. It is unsuitable for a nonseekable source, for text-mode position cookies treated as byte lengths, or for a stream concurrently shared with another reader. A producer that already records the byte count may be able to pass that count instead. Neither approach authorizes trusting an unverified upload size for security limits.

Keep storage thresholds separate from resource limits

A positive spool threshold is not a hard process-memory ceiling. The producer still holds its input chunk, buffering and concurrent requests consume resources, and crossing the threshold requires migration of existing content. Avoid allocating an entire huge input merely to write it into a spool in one call. Use bounded reads and enforce a separate accepted-input limit at the application boundary.

When the service must avoid temporary filesystem use, choose an in-memory interface and an explicit rejection policy for inputs that exceed its budget. Merely setting a very large threshold leaves a descriptor call able to trigger rollover. Conversely, moving content to temporary storage does not protect against filling the filesystem. Test the permitted temporary directory, failure handling and cleanup in the actual deployment environment.

The CPython 3.14.3 implementation confirms the write-threshold condition checked by these examples. We do not generalize it into “zero can never roll over”: descriptor calls, explicit rollover and other operations must be considered separately. Historical truncate reports are not instructions to manipulate private members.

Check the caller that changed the state

If a tiny spool unexpectedly switches storage, inspect the next consumer's documented requirements and trace calls to fileno() or rollover() in a disposable reproduction. If default-threshold writes unexpectedly retain memory, rerun the first example under the exact interpreter used by the service. Record the Python implementation and version alongside the result.

If the bytes appear lost, check the saved position and seek to the intended start before transfer. If the process still runs out of resources, investigate accepted input size, chunk size, concurrency and temporary-filesystem capacity separately. A storage-mode fix is not evidence of lower latency, lower hosting cost or successful upload delivery.

Use a named temporary file for a named-path contract

SpooledTemporaryFile is not a promise that a second process can reopen a stable pathname. Its rolled backing follows TemporaryFile behavior, which differs by platform and may have no usable directory entry. If a subprocess API requires a pathname, select a named-file workflow and review close, reopen, and cleanup rules for the target operating system.

Keep ownership clear: the context manager owns and closes the temporary resource. A receiver that retains it beyond the with block needs a different lifetime arrangement. Conversely, a receiver that closes caller-owned input can invalidate later operations. Document that boundary instead of relying on garbage collection to resolve it.