A file indexer can appear to freeze while printing longer versions of the same directory path. If it uses os.walk(..., followlinks=True), check directory symlinks before adding threads or increasing a timeout. A link back to an ancestor lets traversal re-enter a directory it already visited. Even without a cycle, two links to the same directory can produce duplicate work.
The first decision is whether directory links belong in your scan. Leave link following disabled when they do not. If they do, choose whether to process each underlying directory once or preserve every logical path. Those policies produce different results. The code below implements the first policy for a stable, trusted local tree; it is not a filesystem sandbox.
Start with the default when directory links are unnecessary
os.walk defaults to followlinks=False. A directory symlink can still appear in the yielded dirs list, but that does not mean the walker descends into it. Applications that separately enqueue every entry in dirs can accidentally reintroduce link following outside the walker.
Use topdown=True when pruning. In that mode, change dirs in place to control which children the walker subsequently visits. Assigning a new list to the local variable does not change the list held by the generator. In bottom-up mode, modifying the list cannot prevent descent that has already happened.
Here is a complete temporary fixture. It creates one real child directory, an alias to that child, and a link from the child back to the root. The intentionally unguarded example takes only eight results; do not turn it into an unbounded list(os.walk(...)) call.
import os
from itertools import islice
from pathlib import Path
from tempfile import TemporaryDirectory
with TemporaryDirectory() as tmp:
root = Path(tmp)
child = root / "real"
child.mkdir()
(child / "note.txt").write_text("example", encoding="utf-8")
(root / "alias").symlink_to(child, target_is_directory=True)
(child / "back").symlink_to(root, target_is_directory=True)
default = list(os.walk(root, followlinks=False))
assert len(default) == 2
repeated = list(islice(os.walk(root, followlinks=True), 8))
assert len(repeated) == 8
identities = [
(os.stat(path).st_dev, os.stat(path).st_ino)
for path, _, _ in repeated
]
assert len(set(identities)) == 2
print("default visits:", len(default))
print("bounded link-following visits:", len(repeated))
print("underlying directories:", len(set(identities)))The expected counts are 2, 8, and 2. The text of individual paths depends on directory enumeration order and the temporary directory name. The fixture needs symlink support and permission to create links. A Windows machine without the necessary configuration may reject fixture creation; that failure does not show that cycles are impossible there.
Process each underlying directory once
A path string is a name, not a unique directory identity. Resolving paths can simplify names, but an identity policy should explicitly state what it considers the same object. This example uses the pair st_dev and st_ino obtained from os.stat, which follows the directory link. The pair distinguishes inode values on different devices on filesystems that provide meaningful values.
When a directory has already been seen, clear its descendants and skip processing the current result. Clearing descendants alone prevents more recursion but still allows duplicate indexing of the current directory. Skipping without clearing descendants still permits further traversal. Both operations matter.
import os
from pathlib import Path
from tempfile import TemporaryDirectory
def raise_walk_error(error):
raise error
def walk_unique_directories(root):
seen = set()
for path, dirs, files in os.walk(
root, topdown=True, followlinks=True, onerror=raise_walk_error
):
info = os.stat(path) # Follow the link to identify its target.
identity = (info.st_dev, info.st_ino)
if identity in seen:
dirs[:] = []
continue
seen.add(identity)
yield path, dirs, files
with TemporaryDirectory() as tmp:
root = Path(tmp)
(root / "real").mkdir()
(root / "real" / "note.txt").write_bytes(b"example")
(root / "alias").symlink_to(root / "real", target_is_directory=True)
(root / "real" / "back").symlink_to(root, target_is_directory=True)
visits = list(walk_unique_directories(root))
assert len(visits) == 2
assert sum(files.count("note.txt") for _, _, files in visits) == 1
print("unique visits:", len(visits))
print("note.txt entries:", 1)This produces two unique directory results and one entry for note.txt. Which name represents the child, real or alias, is not guaranteed. If output paths are part of a public index, define a preferred-name rule instead of relying on directory enumeration order. Sorting dirs makes ordering more predictable for a stable fixture, but does not establish a universal canonical name.
The seen set belongs to a single scan. Reusing it indefinitely can suppress later work after files or directories change. It also uses memory proportional to the number of directory identities encountered. This is a correctness example, not a measured performance improvement.
Global deduplication and cycle prevention are different policies
A global identity set intentionally discards alternate directory paths after their first visit. That is useful for a content scanner that wants one pass over each directory target. It is unsuitable when the output must include both an alias path and the original path.
For path-preserving traversal, design a recursive traversal with identities tracked along the current ancestry chain. Reject a target already in that chain, then remove identities as recursion unwinds. That stops ancestor cycles while allowing an independent branch to visit the same target under another name. Such a scanner may legitimately produce many paths. Do not silently replace that contract with global deduplication just to reduce output.
The sample also does not deduplicate files by inode. Two hard-linked filenames inside the visited directories remain two entries. Decide separately whether the reader needs filenames, unique file objects, or unique content. These are three different inventories.
Surface errors and define filesystem boundaries
By default, os.walk ignores errors from its directory scanning call. The example supplies onerror to raise those errors and lets a failed os.stat propagate. An audit scan should not silently present an incomplete tree as complete. A best-effort search tool can record failures and continue, provided its result explicitly says which paths were skipped.
A link may lead outside the starting directory or onto another filesystem. The identity set prevents repeat processing; it does not enforce root containment, mount restrictions, or authorization. A directory can also be replaced between the scanning and stat operations. On a changing or adversarial tree, path-based checks have race conditions. Use an operating-system-specific design and review if containment is a security requirement.
The examples were exercised on local macOS filesystems with Python 3.11, 3.13, and 3.14. Network filesystem identity behavior, Windows junctions, concurrent mutation, permission-denied trees, and production-scale memory use require separate validation. Do not turn the successful small fixture into a claim about those environments.
Verify the scan before re-enabling downstream work
Keep the fixture small and assert directory counts, identity counts, and output counts separately. Test a plain tree, an alias, a link to an ancestor, and an inaccessible directory using an environment where that failure can actually be produced. Avoid testing failure behavior only as an administrator, where permissions may differ from the production process.
If links are unnecessary, retain followlinks=False and check that your own queue does not follow them later. If links are necessary, document the alias policy next to the traversal function and keep error reporting visible. A scanner that terminates but silently loses required logical paths has exchanged one bug for another.