Python File Checksums: Compute Multiple Digests in One Read Pass

A release manifest needs both SHA-256 and SHA-512 for the same artifact. Calling hashlib.file_digest() once for each algorithm is convenient, but it is not a request to compute both values during one traversal. For a large file or an input that cannot be replayed, make the read loop own the bytes and feed each block to every hash object.

The useful distinction is between one file handle, one read pass, and one hash calculation. Reusing a handle does not rewind it automatically. Reopening a path lets you calculate another digest, but reads the file again and may observe a changed file. A single loop shares the exact byte sequence among the selected algorithms while bounding the size of each input block.

First decide whether one algorithm is enough

For one digest on Python 3.11 or later, open a fresh binary file and use hashlib.file_digest(f, "sha256"). Read the result with .hexdigest() and let the surrounding with close the file. Do not continue reading from that object on the assumption that its position or buffering is unchanged: the documented helper can bypass normal Python I/O and leaves the input state unspecified.

The helper takes one algorithm name, constructor, or callable returning a hash object. Passing a list of algorithm names does not select a built-in multiple-digest interface. A callable is useful for a supported hash or HMAC object; it does not make a list into that object.

This article addresses generating two required checksums for one artifact. It does not choose a password storage scheme or turn an untrusted checksum into proof of who published a file.

Feed every digest from the same block

The following example is self-contained and uses an in-memory binary stream. Replace that stream with a fresh open(path, "rb") context when processing a file. The function consumes from the caller's current position to EOF and leaves closing the stream to the caller.

import hashlib
import io


def digest_stream(stream, names=("sha256", "sha512"), block_size=65536):
    if not isinstance(block_size, int) or isinstance(block_size, bool):
        raise ValueError("block_size must be a positive integer")
    if block_size <= 0:
        raise ValueError("block_size must be a positive integer")
    names = tuple(names)
    if not names or len(set(names)) != len(names):
        raise ValueError("choose distinct algorithm names")
    if any(name not in {"sha256", "sha512"} for name in names):
        raise ValueError("this helper supports sha256 and sha512 only")
    hashes = {name: hashlib.new(name) for name in names}
    while True:
        block = stream.read(block_size)
        if block is None:
            raise BlockingIOError("use a blocking binary input")
        if not isinstance(block, bytes):
            raise TypeError("stream.read() must return bytes")
        if block == b"":
            break
        for digest in hashes.values():
            digest.update(block)
    return {name: digest.hexdigest() for name, digest in hashes.items()}


payload = b"release\x00data\r\n" * 100
actual = digest_stream(io.BytesIO(payload), block_size=7)
assert actual == {
    "sha256": hashlib.sha256(payload).hexdigest(),
    "sha512": hashlib.sha512(payload).hexdigest(),
}
assert digest_stream(io.BytesIO(b""))["sha256"] == hashlib.sha256(b"").hexdigest()
print("two algorithms and empty input verified")

Every successful read supplies the same block to both objects. The last block may be smaller than the requested size; that is normal and must still be hashed. Only an empty bytes result ends this blocking-stream contract. A nonblocking read returning None must not be mistaken for EOF, because that would produce a digest of an incomplete prefix.

The narrow algorithm allowlist is deliberate. This helper returns fixed-length hexadecimal strings without additional parameters. It is not a generic wrapper for SHAKE algorithms, which require an output length, or arbitrary user-selected algorithms. Add an algorithm only after defining its result contract.

Verify the read behavior, not just the final strings

Run this block after the preceding definition. A counting stream records each request without measuring the filesystem or a storage device. It proves that adding an algorithm does not add another read traversal in this implementation.

class CountingInput(io.BytesIO):
    def __init__(self, data):
        super().__init__(data)
        self.requests = []
        self.bytes_returned = 0

    def read(self, size=-1):
        self.requests.append(size)
        block = super().read(size)
        self.bytes_returned += len(block)
        return block


source = CountingInput(b"abcdefghij")
result = digest_stream(source, block_size=4)
assert source.requests == [4, 4, 4, 4]
assert source.bytes_returned == 10
assert result["sha256"] == hashlib.sha256(b"abcdefghij").hexdigest()
assert result["sha512"] == hashlib.sha512(b"abcdefghij").hexdigest()
assert not source.closed
print("three data reads, one EOF read, ten input bytes")

This observation is not a speed benchmark. Two hash algorithms still perform two hash calculations, and OS caches can change physical I/O. The loop limits each requested block to 65,536 bytes by default, plus the memory for the selected hash objects and transient read results. It does not prove that the stream implementation itself has no additional buffering.

Make the current-position contract explicit

Run this block after the function definition. It verifies that a consumed prefix is excluded. Starting from the beginning is the responsibility of the code opening the input or explicitly positioning a seekable stream.

source = io.BytesIO(b"header:payload")
assert source.read(7) == b"header:"
actual = digest_stream(source)
assert actual["sha256"] == hashlib.sha256(b"payload").hexdigest()
assert actual["sha256"] != hashlib.sha256(b"header:payload").hexdigest()
for bad_size in (0, -1, True):
    try:
        digest_stream(io.BytesIO(b"x"), block_size=bad_size)
    except ValueError:
        pass
    else:
        raise AssertionError("invalid block size was accepted")
try:
    digest_stream(io.StringIO("text"))
except TypeError:
    pass
else:
    raise AssertionError("text input was accepted")
print("position, validation, and binary-input contract verified")

Use binary mode so newline decoding and text encoding do not change the input bytes. If checksums disagree with another tool, first compare the artifact bytes, selected algorithm, starting offset, and whether a trailing newline was included. The related command substitution newline guide explains one way shell transport can alter those bytes before hashing.

Keep failure and file ownership visible

The function returns only after reaching EOF successfully. A read error propagates to the caller; do not catch it and publish a partial dictionary as a completed manifest. Close a real file with a with context even when reading fails. A nonblocking source needs a separate readiness-aware protocol; this helper intentionally does not wait or retry it.

One pass means both digests receive the same sequence of returned blocks. It does not create an atomic filesystem snapshot. If another process modifies a file during the read, the sequence may not represent a stable artifact. Generate the checksums from a finalized, immutable build output or an appropriate snapshot. Reopening a path twice can make that consistency problem worse because the two traversals can see different content.

A checksum comparison also depends on the expected value coming from a trusted channel. An attacker who replaces both an artifact and its untrusted manifest can make the pair agree. Keep integrity checking, publisher authentication, and transport trust as separate decisions.

Check that input failures cannot return a completed result

Run this block after the function definition. Both test inputs are local and finite. The first models a would-block result, while the second models a read error after one successful block.

class NotReady:
    def read(self, size):
        return None


class FailsAfterBlock:
    def __init__(self):
        self.calls = 0

    def read(self, size):
        self.calls += 1
        if self.calls == 1:
            return b"prefix"
        raise OSError("fixture read failed")


for stream, expected in ((NotReady(), BlockingIOError),
                         (FailsAfterBlock(), OSError)):
    try:
        digest_stream(stream)
    except expected:
        pass
    else:
        raise AssertionError("incomplete input returned a result")
print("would-block and midstream failures propagate")

The practical completion check is concrete: both returned strings match independent known-byte calculations, the reader consumes the bytes once, an empty file succeeds, a preconsumed prefix is handled as documented, and an input error cannot silently become a successful manifest.