highCVE-2026-55520Aug 28, 2026

CVE-2026-55520: Protego ReDoS via robots.txt Wildcard Directive

Rohit Hatagale
AI Security Researcher, SecureLayer7

Protego's robots.txt parser can be frozen indefinitely by a crafted Disallow directive containing many asterisks, because each wildcard becomes a lazy regex piece that causes catastrophic exponential…

PackageProtego
Ecosystempip
Affected<= 0.6.1
Fixed in0.6.2
CVE-2026-55520: Protego ReDoS via robots.txt Wildcard Directive

The problem

Protego translates every * in a robots.txt Allow/Disallow directive into a lazy .*? regex fragment inside _URLPattern._prepare_pattern_for_regex(). A directive with many interleaved asterisks and literal characters produces a regex that backtracks exponentially when matched against a near-miss URL.

An attacker who can serve a crafted robots.txt to a Scrapy spider (or any consumer of protego.Protego.parse()) can stall the parser indefinitely. The call to can_fetch() never returns, effectively freezing the crawling process and causing a denial of service.

Proof of concept

A working proof-of-concept for CVE-2026-55520 in Protego, with the exact payload below.

python
from protego import Protego

# Directive produces: /.*?1.*?1.*?1....*?1.*?1.*?Z as a regex
# The near-miss URL has no trailing 'Z', triggering catastrophic backtracking
robots = """
User-agent: *
Disallow: /" + "*1" * 12 + "*Z
"""

# Cleaner equivalent:
robotstxt = "User-agent: *\nDisallow: /" + "*1" * 12 + "*Z\n"
rp = Protego.parse(robotstxt)
url = "/" + "1" * 60
rp.can_fetch(url, "mybot")  # freezes -- never returns

The vulnerable pattern is in protego._urlpattern._URLPattern._prepare_pattern_for_regex(), which replaces each * with .*?. With 13 wildcards in the directive and a 60-character near-miss URL, Python's re engine explores an exponential number of ways to split the input across the lazy quantifiers before concluding no match exists.

The fix in 0.6.2 (commit 785940181659bf440ba82f1da148fade5087e858) eliminates the backtracking. On Python 3.11 and newer the wildcards can be replaced with possessive quantifiers (.*+) or wrapped in atomic groups ((?>.*)), both of which prevent the engine from re-examining already-consumed characters.

For older Python targets an equivalent approach is to split the directive on * and match each literal segment sequentially, never constructing a regex with multiple adjacent open-ended quantifiers.

Root cause: CWE-1333 (Inefficient Regular Expression Complexity) / CWE-400 (Uncontrolled Resource Consumption).

The fix

Upgrade Protego to version 0.6.2 or later (pip install 'Protego>=0.6.2'). The patch commit is 785940181659bf440ba82f1da148fade5087e858 on the scrapy/protego repository. If an immediate upgrade is not possible, pre-validate or reject any robots.txt directive containing more than a small, bounded number of * characters before passing it to Protego.parse().

Reported by Scrapy Security (reported via scrapy/protego security advisory).

References: [1][2][3][4]

Related research