Skip to content

perf: inline is_ident_char() in hot tokenizer loops via CHAR_IDENT bi…#149

Merged
bartveneman merged 1 commit intomainfrom
perf/tokenizer-ident-char
Feb 26, 2026
Merged

perf: inline is_ident_char() in hot tokenizer loops via CHAR_IDENT bi…#149
bartveneman merged 1 commit intomainfrom
perf/tokenizer-ident-char

Conversation

@bartveneman
Copy link
Member

…tmask

Add CHAR_IDENT (1 << 5) to the lookup table and replace all 4 is_ident_char() callsites in tight loops with a direct bitmask check, eliminating the 3-function call chain (is_ident_char → is_ident_start → is_alpha) on every character scanned.

Benchmark results vs main:

  • Tokenizer - Large CSS: 29,920 → 32,064 ops/sec (+7.2%)
  • Tokenizer - Bootstrap CSS: 356 → 401 ops/sec (+12.6%)
  • Tokenizer - Tailwind CSS: 28 → 31 ops/sec (+10.7%)
  • Parser - Large CSS: 18,801 → 19,908 ops/sec (+5.9%)
  • Parser - Bootstrap CSS: 231 → 244 ops/sec (+5.6%)
  • Parser - Tailwind CSS: 17 → 18 ops/sec (+5.9%)
  • Parse/walk - Bootstrap CSS: 204 → 215 ops/sec (+5.4%)
  • Parse/walk - Tailwind CSS: 15 → 16 ops/sec (+6.7%)

…tmask

Add CHAR_IDENT (1 << 5) to the lookup table and replace all 4 is_ident_char()
callsites in tight loops with a direct bitmask check, eliminating the
3-function call chain (is_ident_char → is_ident_start → is_alpha) on every
character scanned.

Benchmark results vs main:
- Tokenizer - Large CSS:      29,920 → 32,064 ops/sec  (+7.2%)
- Tokenizer - Bootstrap CSS:    356  →    401 ops/sec (+12.6%)
- Tokenizer - Tailwind CSS:      28  →     31 ops/sec (+10.7%)
- Parser - Large CSS:         18,801 → 19,908 ops/sec  (+5.9%)
- Parser - Bootstrap CSS:       231  →    244 ops/sec  (+5.6%)
- Parser - Tailwind CSS:         17  →     18 ops/sec  (+5.9%)
- Parse/walk - Bootstrap CSS:   204  →    215 ops/sec  (+5.4%)
- Parse/walk - Tailwind CSS:     15  →     16 ops/sec  (+6.7%)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@codecov-commenter
Copy link

Bundle Report

Changes will increase total bundle size by 235 bytes (0.16%) ⬆️. This is within the configured threshold ✅

Detailed changes
Bundle name Size Change
@projectwallace/css-parser-esm 146.94kB 235 bytes (0.16%) ⬆️

Affected Assets, Files, and Routes:

view changes for bundle: @projectwallace/css-parser-esm

Assets Changed:

Asset Name Size Change Total Size Change (%)
tokenize.js 231 bytes 19.13kB 1.22%
char-*.js 4 bytes 1.46kB 0.27%

Files in tokenize.js:

  • ./src/tokenize.ts → Total Size: 18.53kB

Files in char-*.js:

  • ./src/char-types.ts → Total Size: 1.3kB

@bartveneman bartveneman merged commit e6065a1 into main Feb 26, 2026
5 checks passed
@bartveneman bartveneman deleted the perf/tokenizer-ident-char branch February 26, 2026 19:33
@codecov-commenter
Copy link

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.71%. Comparing base (557e0d7) to head (c315eeb).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #149      +/-   ##
==========================================
- Coverage   94.87%   94.71%   -0.17%     
==========================================
  Files          16       16              
  Lines        2714     2726      +12     
  Branches      712      713       +1     
==========================================
+ Hits         2575     2582       +7     
- Misses        139      144       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants