Skip to content

⚡ Bolt: [Improve yEnc decoding performance using C-backed bytes methods] #29

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
xbmc4lyfe wants to merge 1 commit into
base: main
Choose a base branch
from
Open

⚡ Bolt: [Improve yEnc decoding performance using C-backed bytes methods] #29

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 25 additions & 13 deletions verify_nzb.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,20 +115,32 @@ def _parse_yenc_attrs(line: bytes) -> dict[str, str]:
return attrs


_YENC_DECODE_TABLE = bytes((i - 42) % 256 for i in range(256))


def _decode_yenc_lines(lines: Iterable[bytes]) -> bytes:
decoded = bytearray()
for line in lines:
index = 0
while index < len(line):
byte = line[index]
if byte == 61:
index += 1
if index >= len(line):
raise ValueError("dangling yEnc escape")
byte = (line[index] - 64) % 256
decoded.append((byte - 42) % 256)
index += 1
return bytes(decoded)
"""
Decodes yEnc-encoded lines using C-backed bytes methods for significant performance gain.
Expects ~8-10x speedup by avoiding Python byte-by-byte iteration.
"""
data = b"".join(lines)
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot May 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve line-local dangling escape validation

Because validate_yenc_body passes the physical NNTP body lines after stripping CRLF, joining all data lines before scanning lets an escape marker at the end of one yEnc line consume the first byte of the next line. yEnc line wrapping must keep the escape marker and escaped byte on the same line, and the previous implementation rejected this as dangling yEnc escape; with inputs like [b"abc=", b"def"] this now decodes across the boundary and can report a malformed article as valid if the advertised size/CRC match the mis-decoded bytes.

Useful? React with 👍 / 👎.

if b"=" not in data:
return data.translate(_YENC_DECODE_TABLE)

chunks = []
start = 0
while True:
idx = data.find(b"=", start)
if idx == -1:
chunks.append(data[start:].translate(_YENC_DECODE_TABLE))
break
chunks.append(data[start:idx].translate(_YENC_DECODE_TABLE))
if idx + 1 >= len(data):
raise ValueError("dangling yEnc escape")
escaped_byte = data[idx + 1]
chunks.append(bytes([(escaped_byte - 106) % 256]))
start = idx + 2
return b"".join(chunks)
Comment on lines +126 to +143
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot May 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Preserve dangling-escape validation at original line boundaries.

data = b"".join(lines) removes line boundaries before checking escapes, so = at the end of an individual yEnc line is no longer treated as dangling. That changes validation behavior and can let malformed yEnc line structure pass further checks.

Proposed fix 
 def _decode_yenc_lines(lines: Iterable[bytes]) -> bytes:
@@
-    data = b"".join(lines)
+    buffered_lines: list[bytes] = []
+    for line in lines:
+        if line.endswith(b"="):
+            raise ValueError("dangling yEnc escape")
+        buffered_lines.append(line)
+
+    data = b"".join(buffered_lines)
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@verify_nzb.py` around lines 126 - 143, The current logic joins all input
lines into one blob (data = b"".join(lines)) which hides line endings so
trailing '=' at the end of an original yEnc line is no longer detected as a
dangling escape; fix by processing each element of lines individually (or
reintroducing their original line terminators) and apply the escape-parsing
logic per-line: for each line in lines, if it contains '=' check if any '='
occurs as the very last byte of that line and raise ValueError("dangling yEnc
escape") when found, otherwise decode that line using the same
translate(_YENC_DECODE_TABLE) + escaped byte handling and append to chunks;
finally return the concatenation of per-line decoded chunks.



def validate_yenc_body(lines: Iterable[bytes | str]) -> YencValidationResult:
Expand Down