⚡️ Speed up function autodetect_ner_format by 41%

[codeflash-ai]

📄 41% (0.41x) speedup for autodetect_ner_format in spacy/cli/convert.py

⏱️ Runtime : 1.26 milliseconds → 896 microseconds (best of 204 runs)

📝 Explanation and details

The optimized code achieves a 41% speedup through two key optimizations that eliminate redundant work:

1. Pre-compiled regex patterns at module level
The original code recompiles the same regex patterns (\S+\|(O|[IB]-\S+) and `\S+\s+(O|[IB]-\S+)#### 📝 Explanation and details

) on every function call. The optimization moves these to module-level constants _IOB_RE and _NER_RE, eliminating 31.5% of the original runtime (lines showing 17.7% + 13.8% in profiler). Regex compilation is expensive in Python, involving pattern parsing and finite state machine construction.

2. Optimized string splitting with early termination
Instead of input_data.split("\n")[:20] which splits the entire string then slices, the optimization uses input_data.split('\n', 20) with a maxsplit parameter. This stops splitting after finding 20 newlines, avoiding unnecessary work on large files. The profiler shows this reduces splitting time from 12.9% to 6.6% of total runtime.

Performance characteristics by test case:

• Large files see the biggest gains (88.9% speedup for 1000-line inputs) due to the split optimization
• Empty/small inputs benefit most from regex pre-compilation (68.7% speedup for empty strings)
• All test cases improve with consistent 13-40% gains across different input patterns

These optimizations are particularly valuable since this function appears to be a format detection utility that would likely be called repeatedly on multiple files or data chunks in document processing pipelines. The module-level regex compilation provides cumulative benefits that scale with usage frequency.

✅ Correctness verification report:

Test	Status
⚙️ Existing Unit Tests	🔘 None Found
🌀 Generated Regression Tests	✅ 85 Passed
⏪ Replay Tests	🔘 None Found
🔎 Concolic Coverage Tests	🔘 None Found
📊 Tests Coverage	100.0%

🌀 Generated Regression Tests and Runtime

import re
from typing import Optional

imports

import pytest # used for our unit tests
from spacy.cli.convert import autodetect_ner_format

unit tests

BASIC TEST CASES

def test_ner_format_basic():
# Simple NER format: whitespace-separated token and tag
data = "John B-PER\nlives O\nin O\nNew B-LOC\nYork I-LOC"
codeflash_output = autodetect_ner_format(data) # 7.85μs -> 6.35μs (23.6% faster)

def test_iob_format_basic():
# Simple IOB format: pipe-separated token|tag
data = "John|B-PER\nlives|O\nin|O\nNew|B-LOC\nYork|I-LOC"
codeflash_output = autodetect_ner_format(data) # 9.80μs -> 8.41μs (16.6% faster)

def test_mixed_format_none():
# Mixed lines, should return None (ambiguous)
data = "John|B-PER\nlives O\nin|O\nNew O\nYork|I-LOC"
codeflash_output = autodetect_ner_format(data) # 8.62μs -> 7.60μs (13.4% faster)

def test_empty_string():
# Empty input should return None
codeflash_output = autodetect_ner_format("") # 2.91μs -> 1.72μs (68.7% faster)

def test_only_whitespace_lines():
# Only whitespace lines should return None
data = "\n \n\t\n"
codeflash_output = autodetect_ner_format(data) # 4.08μs -> 2.70μs (51.0% faster)

def test_only_non_matching_lines():
# Lines that do not match either format
data = "This is a sentence.\nAnother one here."
codeflash_output = autodetect_ner_format(data) # 8.91μs -> 7.33μs (21.7% faster)

def test_ner_format_with_extra_spaces():
# NER format with extra spaces between token and tag
data = "John B-PER\nlives O\nNew B-LOC"
codeflash_output = autodetect_ner_format(data) # 7.01μs -> 5.52μs (27.0% faster)

def test_iob_format_with_extra_spaces():
# IOB format with spaces around pipe (should not match)
data = "John | B-PER\nlives | O"
codeflash_output = autodetect_ner_format(data) # 6.64μs -> 5.36μs (23.7% faster)

def test_ner_format_with_leading_trailing_spaces():
# NER format with leading/trailing spaces
data = " John B-PER \n lives O "
codeflash_output = autodetect_ner_format(data) # 5.76μs -> 4.24μs (35.9% faster)

def test_iob_format_with_leading_trailing_spaces():
# IOB format with leading/trailing spaces
data = " John|B-PER \n lives|O "
codeflash_output = autodetect_ner_format(data) # 6.88μs -> 5.54μs (24.1% faster)

EDGE TEST CASES

def test_ner_format_with_nonstandard_tags():
# NER format with nonstandard tags (should still match)
data = "John B-PERSON\nlives O\nin O\nNew B-LOCATION"
codeflash_output = autodetect_ner_format(data) # 7.63μs -> 5.88μs (29.9% faster)

def test_iob_format_with_nonstandard_tags():
# IOB format with nonstandard tags (should still match)
data = "John|B-PERSON\nlives|O\nNew|B-LOCATION"
codeflash_output = autodetect_ner_format(data) # 9.42μs -> 7.88μs (19.5% faster)

def test_ner_format_with_numbers_and_symbols():
# NER format with tokens containing numbers and symbols
data = "123 B-NUM\n$ O\n@user B-MISC"
codeflash_output = autodetect_ner_format(data) # 6.39μs -> 4.58μs (39.4% faster)

def test_iob_format_with_numbers_and_symbols():
# IOB format with tokens containing numbers and symbols
data = "123|B-NUM\n$|O\n@user|B-MISC"
codeflash_output = autodetect_ner_format(data) # 7.87μs -> 6.31μs (24.7% faster)

def test_ner_format_with_short_lines():
# NER format with very short lines
data = "a O\nb B-X"
codeflash_output = autodetect_ner_format(data) # 5.08μs -> 3.58μs (42.0% faster)

def test_iob_format_with_short_lines():
# IOB format with very short lines
data = "a|O\nb|B-X"
codeflash_output = autodetect_ner_format(data) # 5.45μs -> 3.97μs (37.3% faster)

def test_ner_format_with_long_lines():
# NER format with long tokens
data = "Supercalifragilisticexpialidocious B-WORD\nPneumonoultramicroscopicsilicovolcanoconiosis O"
codeflash_output = autodetect_ner_format(data) # 14.2μs -> 12.9μs (10.1% faster)

def test_iob_format_with_long_lines():
# IOB format with long tokens
data = "Supercalifragilisticexpialidocious|B-WORD\nPneumonoultramicroscopicsilicovolcanoconiosis|O"
codeflash_output = autodetect_ner_format(data) # 31.0μs -> 29.2μs (6.31% faster)

def test_ner_format_with_empty_lines():
# NER format with empty lines in between
data = "John B-PER\n\nlives O\n\nNew B-LOC"
codeflash_output = autodetect_ner_format(data) # 7.17μs -> 5.68μs (26.2% faster)

def test_iob_format_with_empty_lines():
# IOB format with empty lines in between
data = "John|B-PER\n\nlives|O\n\nNew|B-LOC"
codeflash_output = autodetect_ner_format(data) # 8.26μs -> 6.65μs (24.2% faster)

def test_ner_format_with_tab_separator():
# NER format with tab separator (should not match)
data = "John\tB-PER\nlives\tO"
codeflash_output = autodetect_ner_format(data) # 5.33μs -> 4.00μs (33.2% faster)

def test_iob_format_with_tab_separator():
# IOB format with tab separator (should not match)
data = "John\t|B-PER\nlives\t|O"
codeflash_output = autodetect_ner_format(data) # 6.67μs -> 5.13μs (30.0% faster)

def test_ner_format_with_multiple_spaces_and_nonmatching():
# NER format with multiple spaces and a non-matching line
data = "John B-PER\nlives O\nNotMatchingLine"
codeflash_output = autodetect_ner_format(data) # 9.21μs -> 7.46μs (23.5% faster)

def test_iob_format_with_multiple_pipes():
# IOB format with multiple pipes (should not match)
data = "John|B-PER|X\nlives|O|Y"
codeflash_output = autodetect_ner_format(data) # 7.28μs -> 5.95μs (22.3% faster)

def test_ner_format_with_lowercase_tags():
# NER format with lowercase tags (should still match)
data = "John b-per\nlives o"
codeflash_output = autodetect_ner_format(data) # 6.12μs -> 4.78μs (27.9% faster)

def test_iob_format_with_lowercase_tags():
# IOB format with lowercase tags (should still match)
data = "John|b-per\nlives|o"
codeflash_output = autodetect_ner_format(data) # 6.69μs -> 5.36μs (24.8% faster)

def test_ner_format_with_dash_in_token():
# NER format with dash in token
data = "Jean-Luc B-PER\nPicard O"
codeflash_output = autodetect_ner_format(data) # 5.96μs -> 4.58μs (30.2% faster)

def test_iob_format_with_dash_in_token():
# IOB format with dash in token
data = "Jean-Luc|B-PER\nPicard|O"
codeflash_output = autodetect_ner_format(data) # 7.49μs -> 5.77μs (29.7% faster)

def test_ner_format_with_non_ascii():
# NER format with non-ASCII characters
data = "José B-PER\nMünchen B-LOC"
codeflash_output = autodetect_ner_format(data) # 6.38μs -> 4.93μs (29.4% faster)

def test_iob_format_with_non_ascii():
# IOB format with non-ASCII characters
data = "José|B-PER\nMünchen|B-LOC"
codeflash_output = autodetect_ner_format(data) # 8.05μs -> 6.67μs (20.6% faster)

def test_ner_format_with_more_than_20_lines():
# Only first 20 lines should be considered
data = "\n".join([f"Token{i} B-TYPE" for i in range(25)])
codeflash_output = autodetect_ner_format(data) # 18.1μs -> 16.9μs (7.36% faster)

def test_iob_format_with_more_than_20_lines():
# Only first 20 lines should be considered
data = "\n".join([f"Token{i}|B-TYPE" for i in range(25)])
codeflash_output = autodetect_ner_format(data) # 38.6μs -> 36.7μs (5.43% faster)

def test_mixed_format_with_first_20_lines_ner():
# First 20 lines NER, rest IOB, should return NER
data = "\n".join([f"Token{i} B-TYPE" for i in range(20)] + [f"Token{i}|B-TYPE" for i in range(20,25)])
codeflash_output = autodetect_ner_format(data) # 18.0μs -> 16.5μs (9.02% faster)

def test_mixed_format_with_first_20_lines_iob():
# First 20 lines IOB, rest NER, should return IOB
data = "\n".join([f"Token{i}|B-TYPE" for i in range(20)] + [f"Token{i} B-TYPE" for i in range(20,25)])
codeflash_output = autodetect_ner_format(data) # 38.5μs -> 37.2μs (3.38% faster)

def test_mixed_format_with_first_20_lines_mixed():
# First 10 lines NER, next 10 IOB, should return None
data = "\n".join([f"Token{i} B-TYPE" for i in range(10)] + [f"Token{i}|B-TYPE" for i in range(10,20)])
codeflash_output = autodetect_ner_format(data) # 29.4μs -> 27.9μs (5.16% faster)

LARGE SCALE TEST CASES

def test_large_scale_ner_format():
# Large number of NER lines (up to 1000)
data = "\n".join([f"Token{i} B-TYPE" for i in range(1000)])
# Only first 20 lines are checked, so should return "ner"
codeflash_output = autodetect_ner_format(data) # 34.7μs -> 18.4μs (88.9% faster)

def test_large_scale_iob_format():
# Large number of IOB lines (up to 1000)
data = "\n".join([f"Token{i}|B-TYPE" for i in range(1000)])
# Only first 20 lines are checked, so should return "iob"
codeflash_output = autodetect_ner_format(data) # 55.2μs -> 37.8μs (46.2% faster)

def test_large_scale_mixed_format():
# First 500 NER, next 500 IOB
data = "\n".join([f"Token{i} B-TYPE" for i in range(500)] + [f"Token{i}|B-TYPE" for i in range(500,1000)])
# Only first 20 lines are NER, so should return "ner"
codeflash_output = autodetect_ner_format(data) # 34.1μs -> 18.3μs (86.4% faster)

def test_large_scale_mixed_format_none():
# Alternating NER and IOB in first 20 lines
lines = []
for i in range(20):
if i % 2 == 0:
lines.append(f"Token{i} B-TYPE")
else:
lines.append(f"Token{i}|B-TYPE")
data = "\n".join(lines + [f"Token{i} B-TYPE" for i in range(20, 1000)])
# Should return None due to ambiguity in first 20 lines
codeflash_output = autodetect_ner_format(data) # 44.9μs -> 29.1μs (54.1% faster)

def test_large_scale_non_matching_lines():
# All lines non-matching
data = "\n".join([f"This is line {i}" for i in range(1000)])
codeflash_output = autodetect_ner_format(data) # 43.7μs -> 27.5μs (59.1% faster)

def test_large_scale_with_empty_lines():
# Large input with many empty lines
data = "\n".join([""] * 1000)
codeflash_output = autodetect_ner_format(data) # 12.7μs -> 4.75μs (168% faster)

def test_large_scale_with_leading_trailing_spaces():
# Large NER input with leading/trailing spaces
data = "\n".join([f" Token{i} B-TYPE " for i in range(1000)])
codeflash_output = autodetect_ner_format(data) # 36.6μs -> 19.8μs (85.2% faster)

codeflash_output is used to check that the output of the original code is the same as that of the optimized code.

#------------------------------------------------
import re
from typing import Optional

imports

import pytest # used for our unit tests
from spacy.cli.convert import autodetect_ner_format

unit tests

------------------- Basic Test Cases -------------------

def test_ner_format_basic():
# Typical NER format: "word LABEL"
data = "London B-LOC\nis O\na O\ncity O"
codeflash_output = autodetect_ner_format(data) # 8.20μs -> 6.53μs (25.6% faster)

def test_iob_format_basic():
# Typical IOB format: "word|LABEL"
data = "London|B-LOC\nis|O\na|O\ncity|O"
codeflash_output = autodetect_ner_format(data) # 8.62μs -> 6.95μs (24.0% faster)

def test_ner_format_with_trailing_spaces():
# NER with trailing spaces and tabs
data = "London B-LOC \n is\tO\ncity O"
codeflash_output = autodetect_ner_format(data) # 6.81μs -> 5.29μs (28.6% faster)

def test_iob_format_with_extra_spaces():
# IOB with extra spaces before/after pipe
data = "London | B-LOC\nis | O"
# This should not match IOB regex, so it returns None
codeflash_output = autodetect_ner_format(data) # 6.96μs -> 5.32μs (30.7% faster)

def test_mixed_format_returns_none():
# Data with both formats present
data = "London|B-LOC\nis O\ncity|O\n"
codeflash_output = autodetect_ner_format(data) # 8.00μs -> 6.58μs (21.6% faster)

def test_empty_string_returns_none():
# Empty input should return None
data = ""
codeflash_output = autodetect_ner_format(data) # 3.01μs -> 1.73μs (74.4% faster)

def test_only_labels_returns_none():
# Only labels, no tokens
data = "B-LOC\nO\nO"
codeflash_output = autodetect_ner_format(data) # 4.66μs -> 3.28μs (42.2% faster)

def test_only_tokens_returns_none():
# Only tokens, no labels
data = "London\nis\na\ncity"
codeflash_output = autodetect_ner_format(data) # 5.50μs -> 4.00μs (37.4% faster)

def test_ner_format_lowercase_labels():
# NER with lowercase labels
data = "London b-loc\nis o"
# Should not match the regex (expects [IB]- or O)
codeflash_output = autodetect_ner_format(data) # 6.45μs -> 4.74μs (36.1% faster)

def test_iob_format_lowercase_labels():
# IOB with lowercase labels
data = "London|b-loc\nis|o"
# Should not match the regex (expects [IB]- or O)
codeflash_output = autodetect_ner_format(data) # 6.95μs -> 5.46μs (27.2% faster)

------------------- Edge Test Cases -------------------

def test_ner_format_with_non_ascii():
# NER with non-ASCII tokens
data = "München B-LOC\nist O\nschön O"
codeflash_output = autodetect_ner_format(data) # 7.46μs -> 6.08μs (22.6% faster)

def test_iob_format_with_non_ascii():
# IOB with non-ASCII tokens
data = "München|B-LOC\nist|O\nschön|O"
codeflash_output = autodetect_ner_format(data) # 8.68μs -> 7.05μs (23.1% faster)

def test_ner_format_with_numbers():
# NER with numbers as tokens
data = "123 B-NUM\n456 O"
codeflash_output = autodetect_ner_format(data) # 5.53μs -> 3.96μs (39.5% faster)

def test_iob_format_with_numbers():
# IOB with numbers as tokens
data = "123|B-NUM\n456|O"
codeflash_output = autodetect_ner_format(data) # 6.26μs -> 4.75μs (32.0% faster)

def test_ner_format_with_punctuation():
# NER with punctuation as tokens
data = "London B-LOC\n. O"
codeflash_output = autodetect_ner_format(data) # 5.44μs -> 4.25μs (28.2% faster)

def test_iob_format_with_punctuation():
# IOB with punctuation as tokens
data = "London|B-LOC\n.|O"
codeflash_output = autodetect_ner_format(data) # 6.48μs -> 5.07μs (27.8% faster)

def test_ner_format_with_empty_lines():
# NER with empty lines in between
data = "London B-LOC\n\nis O\n\ncity O"
codeflash_output = autodetect_ner_format(data) # 7.06μs -> 5.64μs (25.1% faster)

def test_iob_format_with_empty_lines():
# IOB with empty lines in between
data = "London|B-LOC\n\nis|O\n\ncity|O"
codeflash_output = autodetect_ner_format(data) # 8.08μs -> 6.51μs (24.1% faster)

def test_ner_format_with_leading_trailing_newlines():
# NER with leading/trailing newlines
data = "\nLondon B-LOC\nis O\ncity O\n"
codeflash_output = autodetect_ner_format(data) # 6.95μs -> 5.28μs (31.6% faster)

def test_iob_format_with_leading_trailing_newlines():
# IOB with leading/trailing newlines
data = "\nLondon|B-LOC\nis|O\ncity|O\n"
codeflash_output = autodetect_ner_format(data) # 8.03μs -> 6.37μs (26.0% faster)

def test_ner_format_with_unusual_labels():
# NER with labels containing hyphens and numbers
data = "London B-LOC-1\nis O\ncity O"
codeflash_output = autodetect_ner_format(data) # 6.21μs -> 5.05μs (23.0% faster)

def test_iob_format_with_unusual_labels():
# IOB with labels containing hyphens and numbers
data = "London|B-LOC-1\nis|O\ncity|O"
codeflash_output = autodetect_ner_format(data) # 7.79μs -> 6.25μs (24.6% faster)

def test_ner_format_with_tabs():
# NER with tabs instead of spaces
data = "London\tB-LOC\nis\tO"
codeflash_output = autodetect_ner_format(data) # 5.48μs -> 4.16μs (31.5% faster)

def test_iob_format_with_tabs():
# IOB with tabs, should not match regex (expects '|')
data = "London\t|\tB-LOC\nis\t|\tO"
codeflash_output = autodetect_ner_format(data) # 6.64μs -> 5.25μs (26.5% faster)

def test_ner_format_with_multiple_spaces():
# NER with multiple spaces between token and label
data = "London B-LOC\nis O"
codeflash_output = autodetect_ner_format(data) # 6.01μs -> 4.38μs (37.3% faster)

def test_iob_format_with_multiple_pipes():
# IOB with multiple pipes, should not match regex
data = "London|B-LOC|X\nis|O|Y"
codeflash_output = autodetect_ner_format(data) # 7.48μs -> 6.04μs (23.7% faster)

def test_ner_format_with_long_labels():
# NER with very long label names
data = "London B-LOCATION-ENTITY-EXTREMELY-LONG-LABEL\nis O"
codeflash_output = autodetect_ner_format(data) # 9.16μs -> 7.81μs (17.3% faster)

def test_iob_format_with_long_labels():
# IOB with very long label names
data = "London|B-LOCATION-ENTITY-EXTREMELY-LONG-LABEL\nis|O"
codeflash_output = autodetect_ner_format(data) # 19.0μs -> 17.6μs (8.01% faster)

def test_ner_format_with_label_at_start():
# NER with label at start (should not match)
data = "B-LOC London\nO is"
codeflash_output = autodetect_ner_format(data) # 6.11μs -> 4.79μs (27.6% faster)

def test_iob_format_with_label_at_start():
# IOB with label at start (should not match)
data = "B-LOC|London\nO|is"
codeflash_output = autodetect_ner_format(data) # 6.62μs -> 5.42μs (22.2% faster)

def test_ner_format_with_only_spaces():
# Input with only spaces
data = " \n "
codeflash_output = autodetect_ner_format(data) # 3.52μs -> 2.26μs (55.8% faster)

def test_iob_format_with_only_pipes():
# Input with only pipes
data = "|||\n||"
codeflash_output = autodetect_ner_format(data) # 4.33μs -> 3.03μs (43.1% faster)

def test_ner_format_with_comment_lines():
# NER with comment lines (should ignore comments)
data = "# This is a comment\nLondon B-LOC\n# Another comment\nis O"
codeflash_output = autodetect_ner_format(data) # 10.7μs -> 9.29μs (15.2% faster)

def test_iob_format_with_comment_lines():
# IOB with comment lines (should ignore comments)
data = "# This is a comment\nLondon|B-LOC\n# Another comment\nis|O"
codeflash_output = autodetect_ner_format(data) # 11.1μs -> 9.72μs (13.7% faster)

------------------- Large Scale Test Cases -------------------

def test_ner_format_large_scale():
# Large NER dataset (1000 lines)
data = "\n".join([f"word{i} B-ENTITY" if i % 10 == 0 else f"word{i} O" for i in range(1000)])
codeflash_output = autodetect_ner_format(data) # 32.2μs -> 15.6μs (106% faster)

def test_iob_format_large_scale():
# Large IOB dataset (1000 lines)
data = "\n".join([f"word{i}|B-ENTITY" if i % 10 == 0 else f"word{i}|O" for i in range(1000)])
codeflash_output = autodetect_ner_format(data) # 38.9μs -> 22.6μs (72.4% faster)

def test_large_mixed_format_returns_none():
# Large dataset with both formats present in first 20 lines
lines = []
for i in range(10):
lines.append(f"word{i}|B-ENTITY")
lines.append(f"word{i} B-ENTITY")
lines += [f"word{i} O" for i in range(960)]
data = "\n".join(lines)
codeflash_output = autodetect_ner_format(data) # 65.4μs -> 30.6μs (114% faster)

def test_large_ner_format_with_noise():
# Large NER dataset with some noisy lines
lines = [f"word{i} B-ENTITY" if i % 10 == 0 else f"word{i} O" for i in range(980)]
lines += ["", " ", "# comment", "nonsense"]
data = "\n".join(lines)
codeflash_output = autodetect_ner_format(data) # 32.7μs -> 14.6μs (125% faster)

def test_large_iob_format_with_noise():
# Large IOB dataset with some noisy lines
lines = [f"word{i}|B-ENTITY" if i % 10 == 0 else f"word{i}|O" for i in range(980)]
lines += ["", " ", "# comment", "nonsense"]
data = "\n".join(lines)
codeflash_output = autodetect_ner_format(data) # 38.7μs -> 21.9μs (76.7% faster)

def test_large_empty_input():
# Large input with only empty lines
data = "\n" * 999
codeflash_output = autodetect_ner_format(data) # 12.6μs -> 4.87μs (159% faster)

def test_large_all_non_matching_lines():
# Large input with lines that don't match either format
data = "\n".join([f"word{i}-label{i}" for i in range(1000)])
codeflash_output = autodetect_ner_format(data) # 58.3μs -> 41.1μs (41.8% faster)

def test_large_first_20_lines_are_ner_rest_iob():
# First 20 lines are NER, rest are IOB, should detect as NER
data = "\n".join([f"word{i} B-ENTITY" for i in range(20)] + [f"word{i}|B-ENTITY" for i in range(20, 1000)])
codeflash_output = autodetect_ner_format(data) # 35.4μs -> 19.3μs (83.8% faster)

def test_large_first_20_lines_are_iob_rest_ner():
# First 20 lines are IOB, rest are NER, should detect as IOB
data = "\n".join([f"word{i}|B-ENTITY" for i in range(20)] + [f"word{i} B-ENTITY" for i in range(20, 1000)])
codeflash_output = autodetect_ner_format(data) # 59.3μs -> 42.7μs (38.9% faster)

codeflash_output is used to check that the output of the original code is the same as that of the optimized code.

To edit these changes git checkout codeflash/optimize-autodetect_ner_format-mhwsmgws and push.