⚡️ Speed up function replace_mime_encodings by 22%

[codeflash-ai]

📄 22% (0.22x) speedup for replace_mime_encodings in unstructured/cleaners/core.py

⏱️ Runtime : 330 microseconds → 270 microseconds (best of 64 runs)

📝 Explanation and details

The optimized code achieves a 22% speedup by adding a single decorator: @lru_cache(maxsize=128) to the format_encoding_str function. This is a pure memoization optimization that caches the results of encoding string formatting.

Why this optimization works:

1. Repeated encoding values: In real-world usage, applications typically use a small set of encoding strings repeatedly (e.g., "utf-8", "UTF-8", "utf_8", "iso-8859-1"). The format_encoding_str function performs string operations (.lower() and .replace()) on every call, even when processing the same encoding value multiple times.

2. Cache eliminates redundant work: With lru_cache, after the first call with a given encoding string, subsequent calls return the cached result immediately without executing the function body. This eliminates:

   • String lowercasing operation (~44.6% of original function time)
   • List creation for annotated_encodings (~24% of original function time)
   • String replacement and membership checks (~18.3% of original function time)

3. Evidence from profiling: The line profiler shows format_encoding_str calls dropped from 813,499 ns to 385,714 ns in replace_mime_encodings (52% faster), accounting for the overall 22% speedup.

Test results confirm the optimization pattern:

• First calls show ~20-30% speedup
• Repeated calls show dramatically better performance (e.g., test_performance_repeated_operations: 33.7% → 33.6% → 26.3% faster across successive calls, with absolute times dropping from 4.49μs → 1.42μs → 1.08μs)
• Tests with repeated encoding variations (underscores, case) benefit most (up to 58% faster on second call with utf_8)
• Large-scale tests with 500+ repetitions show consistent 15-25% improvements

Impact on workloads:
This optimization is particularly effective when:

• Processing multiple documents/strings with the same encoding
• Batch processing operations where replace_mime_encodings is called repeatedly
• Long-running applications that handle MIME-encoded content continuously

The 128-entry cache size is appropriate since encoding names are a finite, small set in practice, keeping memory overhead minimal while maximizing hit rates.

✅ Correctness verification report:

Test	Status
⚙️ Existing Unit Tests	✅ 20 Passed
🌀 Generated Regression Tests	✅ 57 Passed
⏪ Replay Tests	🔘 None Found
🔎 Concolic Coverage Tests	✅ 1 Passed
📊 Tests Coverage	100.0%

⚙️ Click to see Existing Unit Tests

Test File::Test Function	Original ⏱️	Optimized ⏱️	Speedup
cleaners/test_core.py::test_replace_mime_encodings	5.79μs	4.59μs	26.2%✅
cleaners/test_core.py::test_replace_mime_encodings_works_with_different_encodings	5.12μs	4.23μs	20.9%✅
cleaners/test_core.py::test_replace_mime_encodings_works_with_right_to_left_encodings	11.0μs	9.66μs	14.2%✅

🌀 Click to see Generated Regression Tests

from typing import List

# imports
import pytest  # used for our unit tests

from unstructured.cleaners.core import replace_mime_encodings

# function to test


def format_encoding_str(encoding: str) -> str:
    """Format input encoding string (e.g., `utf-8`, `iso-8859-1`, etc).
    Parameters
    ----------
    encoding
        The encoding string to be formatted (e.g., `UTF-8`, `utf_8`, `ISO-8859-1`, `iso_8859_1`,
        etc).
    """
    formatted_encoding = encoding.lower().replace("_", "-")

    # Special case for Arabic and Hebrew charsets with directional annotations
    annotated_encodings = ["iso-8859-6-i", "iso-8859-6-e", "iso-8859-8-i", "iso-8859-8-e"]
    if formatted_encoding in annotated_encodings:
        formatted_encoding = formatted_encoding[:-2]  # remove the annotation

    return formatted_encoding


def test_no_encodings_returns_same_string():
    """
    Basic: If the input contains no MIME/quoted-printable sequences the function
    should return the original string unchanged.
    """
    input_text = "Hello, World!"  # plain ASCII
    # Default encoding is utf-8; expect exact identity
    codeflash_output = replace_mime_encodings(input_text)  # 4.55μs -> 3.40μs (33.8% faster)


def test_decode_utf8_multibyte_sequence_right_single_quote():
    """
    Basic: Decode a UTF-8 multibyte quoted-printable sequence.
    The bytes E2 80 99 are the UTF-8 encoding for the right single quote (’).
    Example input follows the common quoted-printable hex form: =E2=80=99
    """
    input_text = "5 w=E2=80=99s"  # quoted-printable representation of "5 w’s"
    expected = "5 w’s"
    codeflash_output = replace_mime_encodings(
        input_text, encoding="utf-8"
    )  # 5.57μs -> 4.62μs (20.5% faster)


def test_decode_cafe_with_various_encoding_name_forms():
    """
    Basic: Verify that different representations of the encoding name (underscore,
    uppercase) are normalized by format_encoding_str and that the decoding works.
    The UTF-8 quoted-printable for 'é' is =C3=A9, so 'caf=C3=A9' should become 'café'.
    """
    inputs_and_encodings = [
        ("caf=C3=A9", "utf-8"),
        ("caf=C3=A9", "UTF_8"),
        ("caf=C3=A9", "utf_8"),
    ]
    for text, enc in inputs_and_encodings:
        # Each variant of the encoding name should decode to the same output.
        codeflash_output = replace_mime_encodings(
            text, encoding=enc
        )  # 8.99μs -> 7.13μs (26.0% faster)


def test_space_decoding_A20_to_space():
    """
    Basic: Quoted-printable represents space as =20. Ensure it becomes an actual space.
    """
    codeflash_output = replace_mime_encodings("A=20B")  # 4.46μs -> 3.39μs (31.4% faster)


def test_iso_8859_1_decoding_single_byte_charset():
    """
    Basic: Test decoding using a single-byte charset (iso-8859-1). The character 'á'
    in ISO-8859-1 is 0xE1, represented as =E1 in quoted-printable.
    """
    # The input "ol=E1" encoded with iso-8859-1 should decode to "olá".
    codeflash_output = replace_mime_encodings(
        "ol=E1", encoding="iso-8859-1"
    )  # 5.27μs -> 4.24μs (24.4% faster)


def test_empty_string_returns_empty():
    """
    Edge: An empty input string should simply return an empty string.
    """
    codeflash_output = replace_mime_encodings(
        "", encoding="utf-8"
    )  # 4.92μs -> 3.85μs (27.6% faster)


def test_annotated_encoding_names_are_normalized_and_no_error_on_ascii_text():
    """
    Edge: Encodings with directional annotations (e.g., iso-8859-6-i) should be
    normalized by format_encoding_str. Using such an annotated encoding on simple ASCII
    input should not raise and should return the original ASCII.
    """
    annotated = "ISO_8859_6_I"  # upper + underscore + annotation
    # Now ensure replace_mime_encodings uses that normalization internally and works
    codeflash_output = replace_mime_encodings(
        "ASCII only text", encoding=annotated
    )  # 11.6μs -> 9.86μs (17.6% faster)


def test_invalid_encoding_raises_LookupError():
    """
    Edge: If an unknown encoding name is provided, Python's codec lookup should raise
    a LookupError when calling .encode(...) or .decode(...). The function should not
    swallow that exception.
    """
    with pytest.raises(LookupError):
        # We expect a LookupError because 'no_such_encoding' is not a valid codec name.
        replace_mime_encodings(
            "abc=20def", encoding="no_such_encoding"
        )  # 7.33μs -> 6.13μs (19.6% faster)


def test_soft_line_breaks_are_removed():
    """
    Edge: Quoted-printable soft line breaks are indicated by '=' at end-of-line
    followed by a newline. They should be removed, joining the lines.
    For example "soft=\nline" -> "softline".
    """
    input_text = "soft=\nline"
    codeflash_output = replace_mime_encodings(input_text)  # 4.55μs -> 3.54μs (28.4% faster)


def test_trailing_equals_sign_preserved_if_not_soft_break():
    """
    Edge: A single trailing '=' without a following newline is not a soft line break.
    quopri.decodestring typically leaves such trailing '=' characters untouched.
    Ensure function behaviour is deterministic: if the underlying library leaves it,
    our function returns it unchanged.
    """
    # This behavior documents expected behavior: trailing '=' remains.
    # If the underlying quopri behavior changes, this test will fail and flag that mutation.
    codeflash_output = replace_mime_encodings("abc=")  # 4.45μs -> 3.48μs (28.1% faster)


def test_large_scale_repeated_sequences_performance_and_correctness():
    """
    Large scale: Create a moderately large payload by repeating a quoted-printable
    sequence many times (but keep repetitions < 1000 per instructions).
    Ensure the function decodes every occurrence correctly and runs deterministically.
    """
    repetitions = 500  # within allowed limit (< 1000)
    # Use UTF-8 quoted-printable for 'café' -> 'caf=C3=A9'
    token = "caf=C3=A9"
    # Join with a single space to form a large string; final output should contain 'café' repeated.
    large_input = " ".join([token] * repetitions)
    codeflash_output = replace_mime_encodings(large_input, encoding="utf-8")
    output = codeflash_output  # 20.5μs -> 19.6μs (4.68% faster)
    # All tokens should decode to 'café' and be separated by single spaces -> count occurrences.
    # Use split to avoid potential overlapping substring matches.
    parts: List[str] = output.split(" ")
    # Verify every decoded token equals 'café'
    for part in parts:
        pass


# codeflash_output is used to check that the output of the original code is the same as that of the optimized code.

from unstructured.cleaners.core import replace_mime_encodings


def test_basic_mime_decoding():
    """Test basic MIME encoded string decoding with default UTF-8 encoding."""
    # Input: Simple MIME-encoded text with soft hyphen
    text = "5 w=E2=80=99s"
    codeflash_output = replace_mime_encodings(text)
    result = codeflash_output  # 5.36μs -> 4.32μs (24.1% faster)


def test_basic_mime_decoding_no_encoding():
    """Test MIME decoding with default UTF-8 when no encoding is specified."""
    # Input: MIME-encoded text without explicit encoding parameter
    text = "Hello=20World"
    codeflash_output = replace_mime_encodings(text)
    result = codeflash_output  # 4.44μs -> 3.37μs (31.5% faster)


def test_empty_string():
    """Test that an empty string returns an empty string."""
    # Input: Empty string
    text = ""
    codeflash_output = replace_mime_encodings(text)
    result = codeflash_output  # 4.56μs -> 3.56μs (27.9% faster)


def test_no_mime_encoding():
    """Test that plain text without MIME encoding is returned unchanged."""
    # Input: Regular ASCII text without any MIME encoding
    text = "Hello World"
    codeflash_output = replace_mime_encodings(text)
    result = codeflash_output  # 4.41μs -> 3.34μs (31.8% faster)


def test_multiple_mime_encodings():
    """Test multiple consecutive MIME-encoded characters."""
    # Input: Multiple MIME-encoded characters
    text = "test=3D=3D"
    codeflash_output = replace_mime_encodings(text)
    result = codeflash_output  # 4.53μs -> 3.47μs (30.5% faster)


def test_mixed_content():
    """Test text with both plain and MIME-encoded content."""
    # Input: Mix of plain text and MIME-encoded sequences
    text = "Name=3A John"
    codeflash_output = replace_mime_encodings(text)
    result = codeflash_output  # 4.47μs -> 3.35μs (33.4% faster)


def test_special_characters_encoded():
    """Test MIME encoding of common special characters."""
    # Input: MIME-encoded equals sign and comma
    text = "value=3Dtest,another=3Dvalue"
    codeflash_output = replace_mime_encodings(text)
    result = codeflash_output  # 4.40μs -> 3.46μs (27.3% faster)


def test_encoding_parameter_utf8():
    """Test explicitly specifying UTF-8 encoding."""
    # Input: MIME text with explicit UTF-8 encoding
    text = "Hello=20World"
    codeflash_output = replace_mime_encodings(text, encoding="utf-8")
    result = codeflash_output  # 4.88μs -> 3.80μs (28.4% faster)


def test_encoding_parameter_uppercase():
    """Test that encoding parameter accepts uppercase values."""
    # Input: MIME text with uppercase encoding name
    text = "Hello=20World"
    codeflash_output = replace_mime_encodings(text, encoding="UTF-8")
    result = codeflash_output  # 4.88μs -> 3.74μs (30.3% faster)


def test_encoding_parameter_underscore():
    """Test that encoding parameter handles underscores in encoding names."""
    # Input: MIME text with underscore in encoding name
    text = "Hello=20World"
    codeflash_output = replace_mime_encodings(text, encoding="utf_8")
    result = codeflash_output  # 5.19μs -> 3.76μs (38.0% faster)


def test_incomplete_mime_sequence():
    """Test handling of incomplete MIME encoding sequences."""
    # Input: Text with incomplete MIME sequence (only one hex digit)
    text = "test=2"
    codeflash_output = replace_mime_encodings(text)
    result = codeflash_output  # 4.51μs -> 3.35μs (34.6% faster)


def test_invalid_hex_characters():
    """Test MIME encoding with invalid hex characters."""
    # Input: MIME sequence with non-hex characters
    text = "test=ZZ"
    codeflash_output = replace_mime_encodings(text)
    result = codeflash_output  # 4.46μs -> 3.52μs (26.7% faster)


def test_trailing_equals():
    """Test text ending with an incomplete MIME sequence."""
    # Input: Text ending with equals sign
    text = "test="
    codeflash_output = replace_mime_encodings(text)
    result = codeflash_output  # 4.31μs -> 3.39μs (27.0% faster)


def test_single_equals():
    """Test text that is just an equals sign."""
    # Input: Single equals character
    text = "="
    codeflash_output = replace_mime_encodings(text)
    result = codeflash_output  # 4.05μs -> 3.11μs (30.1% faster)


def test_consecutive_mime_encodings():
    """Test multiple consecutive MIME-encoded sequences without plain text."""
    # Input: Only MIME-encoded content
    text = "=20=20=20"
    codeflash_output = replace_mime_encodings(text)
    result = codeflash_output  # 4.42μs -> 3.41μs (29.5% faster)


def test_mime_encoding_at_start():
    """Test MIME encoding at the beginning of string."""
    # Input: String starting with MIME-encoded character
    text = "=48ello"
    codeflash_output = replace_mime_encodings(text)
    result = codeflash_output  # 4.23μs -> 3.39μs (24.7% faster)


def test_mime_encoding_at_end():
    """Test MIME encoding at the end of string."""
    # Input: String ending with MIME-encoded character
    text = "Hell=6F"
    codeflash_output = replace_mime_encodings(text)
    result = codeflash_output  # 4.51μs -> 3.33μs (35.5% faster)


def test_mime_null_character():
    """Test MIME encoding of null character."""
    # Input: MIME-encoded null character
    text = "test=00end"
    codeflash_output = replace_mime_encodings(text)
    result = codeflash_output  # 4.41μs -> 3.43μs (28.6% faster)


def test_lowercase_hex_values():
    """Test MIME encoding with lowercase hex values."""
    # Input: MIME encoding using lowercase hex
    text = "test=2b=3d"
    codeflash_output = replace_mime_encodings(text)
    result = codeflash_output  # 4.43μs -> 3.58μs (23.6% faster)


def test_uppercase_hex_values():
    """Test MIME encoding with uppercase hex values."""
    # Input: MIME encoding using uppercase hex
    text = "test=2B=3D"
    codeflash_output = replace_mime_encodings(text)
    result = codeflash_output  # 4.49μs -> 3.56μs (26.3% faster)


def test_mixed_case_hex_values():
    """Test MIME encoding with mixed case hex values."""
    # Input: MIME encoding with mixed case hex
    text = "test=2B=3d=2A"
    codeflash_output = replace_mime_encodings(text)
    result = codeflash_output  # 4.30μs -> 3.45μs (24.8% faster)


def test_iso_8859_1_encoding():
    """Test with ISO-8859-1 encoding."""
    # Input: MIME text with ISO-8859-1 encoding
    text = "caf=E9"
    codeflash_output = replace_mime_encodings(text, encoding="iso-8859-1")
    result = codeflash_output  # 5.23μs -> 4.24μs (23.4% faster)


def test_iso_8859_6_special_case():
    """Test that ISO-8859-6 with directional annotation is handled."""
    # Input: MIME text with ISO-8859-6-i encoding
    text = "test=20string"
    codeflash_output = replace_mime_encodings(text, encoding="iso-8859-6-i")
    result = codeflash_output  # 11.0μs -> 9.36μs (18.0% faster)


def test_iso_8859_8_special_case():
    """Test that ISO-8859-8 with directional annotation is handled."""
    # Input: MIME text with ISO-8859-8-e encoding
    text = "test=20string"
    codeflash_output = replace_mime_encodings(text, encoding="iso-8859-8-e")
    result = codeflash_output  # 10.8μs -> 8.99μs (20.2% faster)


def test_encoding_case_variations():
    """Test various case formats of encoding names."""
    # Input: Same MIME text
    text = "Hello=20World"
    # Test with different case variations
    codeflash_output = replace_mime_encodings(text, encoding="UTF-8")
    result1 = codeflash_output  # 4.74μs -> 3.75μs (26.4% faster)
    codeflash_output = replace_mime_encodings(text, encoding="utf-8")
    result2 = codeflash_output  # 1.68μs -> 1.34μs (25.5% faster)
    codeflash_output = replace_mime_encodings(text, encoding="Utf-8")
    result3 = codeflash_output  # 1.20μs -> 1.11μs (8.04% faster)


def test_encoding_underscore_variations():
    """Test various underscore/dash formats of encoding names."""
    # Input: Same MIME text
    text = "Hello=20World"
    # Test with different separator formats
    codeflash_output = replace_mime_encodings(text, encoding="utf-8")
    result1 = codeflash_output  # 4.87μs -> 3.75μs (29.8% faster)
    codeflash_output = replace_mime_encodings(text, encoding="utf_8")
    result2 = codeflash_output  # 2.00μs -> 1.27μs (58.0% faster)


def test_large_plain_text_no_encoding():
    """Test performance with large plain text without MIME encoding."""
    # Input: Large string of plain text (no MIME encoding)
    text = "Hello World " * 100
    codeflash_output = replace_mime_encodings(text)
    result = codeflash_output  # 7.72μs -> 6.59μs (17.0% faster)


def test_large_mime_encoded_text():
    """Test performance with large MIME-encoded content."""
    # Input: Repeatedly MIME-encoded space character
    text = "Word=20" * 100 + "Test"
    codeflash_output = replace_mime_encodings(text)
    result = codeflash_output  # 6.40μs -> 5.36μs (19.3% faster)


def test_large_mixed_content():
    """Test performance with large mixed plain and MIME content."""
    # Input: Large alternating pattern of plain and MIME content
    text = "".join(["word" + "=20" for _ in range(100)]) + "end"
    codeflash_output = replace_mime_encodings(text)
    result = codeflash_output  # 6.24μs -> 5.15μs (21.2% faster)


def test_large_special_characters():
    """Test performance with many MIME-encoded special characters."""
    # Input: Many MIME-encoded equals signs
    text = "=3D" * 200
    codeflash_output = replace_mime_encodings(text)
    result = codeflash_output  # 6.24μs -> 5.17μs (20.7% faster)


def test_long_string_with_various_encodings():
    """Test performance with long string containing various MIME sequences."""
    # Input: Long string with different MIME-encoded characters
    base = "test=20text=3D"
    text = base * 50
    codeflash_output = replace_mime_encodings(text)
    result = codeflash_output  # 6.16μs -> 5.39μs (14.2% faster)


def test_performance_repeated_operations():
    """Test that repeated operations on same text produce consistent results."""
    # Input: Original text
    text = "data=20value=3Dtest"
    # Perform multiple operations
    codeflash_output = replace_mime_encodings(text)
    result1 = codeflash_output  # 4.49μs -> 3.36μs (33.7% faster)
    codeflash_output = replace_mime_encodings(text)
    result2 = codeflash_output  # 1.42μs -> 1.06μs (33.6% faster)
    codeflash_output = replace_mime_encodings(text)
    result3 = codeflash_output  # 1.08μs -> 859ns (26.3% faster)


def test_large_text_idempotent():
    """Test that applying function to already-decoded text doesn't change it."""
    # Input: Already decoded text
    text = "This is plain text with no encoding"
    # Apply function
    codeflash_output = replace_mime_encodings(text)
    result = codeflash_output  # 4.52μs -> 3.40μs (33.0% faster)


def test_many_consecutive_sequences():
    """Test performance with many consecutive MIME sequences."""
    # Input: 500 consecutive MIME-encoded spaces
    text = "=20" * 500
    codeflash_output = replace_mime_encodings(text)
    result = codeflash_output  # 7.99μs -> 6.90μs (15.8% faster)


def test_large_alternating_valid_invalid():
    """Test with large alternating pattern of valid and invalid sequences."""
    # Input: Pattern of valid MIME codes and plain text
    text = "".join(["=20" + chr(65 + i % 26) for i in range(100)])
    codeflash_output = replace_mime_encodings(text)
    result = codeflash_output  # 5.33μs -> 4.27μs (24.8% faster)


def test_memory_efficiency_repeated_small_text():
    """Test memory efficiency with many repetitions of small text."""
    # Input: Small MIME text repeated many times
    base_text = "a=20b=20c"
    text = base_text * 100
    codeflash_output = replace_mime_encodings(text)
    result = codeflash_output  # 6.74μs -> 5.66μs (19.0% faster)


def test_large_single_long_line():
    """Test with large text as single line without breaks."""
    # Input: Very long single line with MIME encoding
    text = "word=20" * 150 + "end"
    codeflash_output = replace_mime_encodings(text)
    result = codeflash_output  # 7.47μs -> 6.08μs (22.9% faster)


def test_boundary_encoding_parameters():
    """Test with various encoding parameters on large text."""
    # Input: Large MIME text
    text = "test=20value=20data" * 50
    # Test with different encodings
    codeflash_output = replace_mime_encodings(text, encoding="utf-8")
    result_utf8 = codeflash_output  # 7.32μs -> 6.35μs (15.3% faster)
    codeflash_output = replace_mime_encodings(text, encoding="iso-8859-1")
    result_iso = codeflash_output  # 4.12μs -> 3.70μs (11.6% faster)


# codeflash_output is used to check that the output of the original code is the same as that of the optimized code.

import pytest

from unstructured.cleaners.core import replace_mime_encodings


def test_replace_mime_encodings():
    with pytest.raises(LookupError, match="unknown\\ encoding:\\ "):
        replace_mime_encodings("", encoding="")

🔎 Click to see Concolic Coverage Tests

Test File::Test Function	Original ⏱️	Optimized ⏱️	Speedup
codeflash_concolic_xdo_puqm/tmpvb4dgkld/test_concolic_coverage.py::test_replace_mime_encodings	5.83μs	5.40μs	8.00%✅

To edit these changes git checkout codeflash/optimize-replace_mime_encodings-mkrvo33a and push.