add lag + index head support to ctlog

[tarunKoyalwar]

CT Logs: Add Index Head and Lag Calculation

Summary

Added real-time progress tracking for CT log streaming, showing current index, total entries (head), and processing
lag.

Changes

• Enhanced metadata: Added TreeSize, Lag, LogURL to EntryMeta
• Progress tracking: Shows Index X/Y (Z.Z%), Lag: W in verbose mode
• JSON output: Added ctl_index, ctl_tree_size, ctl_lag fields
• Backward compatibility: All existing functionality preserved

Usage

Verbose progress display

tlsx -ctl -v
[CT] Google 'Argon2023': Index 12345/50000 (24.7%), Lag: 37655

Enhanced JSON metadata

tlsx -ctl -j | jq '{progress: (.ctl_index/.ctl_tree_size*100), lag: .ctl_lag}'

Files Modified

• pkg/ctlogs/options.go - Enhanced EntryMeta/CTLogSource structures
• pkg/ctlogs/ctlogs.go - Added tree size tracking and lag calculation
• pkg/tlsx/clients/clients.go - Added CT log metadata to Response
• internal/runner/runner.go - Enhanced callback with progress display

Benefits

• Monitoring: Track CT log processing health and lag
• Visibility: Understand indexing progress and coverage gaps
• Zero breaking changes: Purely additive enhancement

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing Touches

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch show-index-head

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

• Add @coderabbitai ignore or @coderabbit ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

• Visit our Status Page to check the current availability of CodeRabbit.
• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.