Skip to content

Feature/stream sdk#81

Open
boy-hack wants to merge 14 commits intomainfrom
feature/stream-sdk
Open

Feature/stream sdk#81
boy-hack wants to merge 14 commits intomainfrom
feature/stream-sdk

Conversation

@boy-hack
Copy link
Owner

No description provided.

boy-hack added 14 commits March 17, 2026 22:02
- Add EnumStream(ctx, domain, callback func(Result)) error
- Add VerifyStream(ctx, domains, callback func(Result)) error
- Introduce streamCollector (implements outputter.Output) that forwards
  each result to the callback in real-time, no buffering
- Refactor: extract buildOptions(), buildDictChan(), parseResult() to
  eliminate duplication between blocking and stream paths
- Remove obsolete Timeout/DynamicTimeout fields from Config (now always
  dynamic with 10s upper bound)
- DefaultConfig and NewScanner() updated accordingly
Callers can now inject custom output sinks:

    type myWriter struct{}
    func (w *myWriter) WriteDomainResult(r result.Result) error { ... }
    func (w *myWriter) Close() error { return nil }

    scanner := sdk.NewScanner(&sdk.Config{
        ExtraWriters: []outputter.Output{&myWriter{}},
    })

ExtraWriters are appended after the internal primaryWriter so the SDK's
own collect/stream logic is unaffected. Close() is called on each writer
by the runner after the scan finishes.
- New pkg/core/errors package with:
  ErrPermissionDenied, ErrDeviceNotFound, ErrDeviceNotActive,
  ErrPcapInit, ErrDomainChanNil
- device.go: wrap three fmt.Errorf strings with %w + sentinel
- result.go: replace fmt.Errorf string with ErrDomainChanNil
- sdk.go: re-export sentinels as package-level vars so callers need
  only import the sdk package and use errors.Is(err, sdk.ErrPermissionDenied)
- Architecture diagram showing goroutine data flow
- Goroutine responsibility table
- SDK quick-start with blocking + stream API examples
- Config field table (notes that Timeout is not configurable)
- Low-level Options field table
- Custom output sink guide (outputter.Output interface)
- Typed error handling with errors.Is examples
- Backpressure description
- Build / test commands
- Practical notes: one-instance-only, sudo, macOS BPF, WSL2
screen.go:
- Remove \r prefix from all output paths (was used to overwrite
  progress bar in-place, but corrupts piped output)
- --od --silent path now emits clean 'domain\n' lines, directly
  consumable by httpx / grep / awk
- Non-silent path pads to terminal width without leading \r
- Guard against negative padding width (long domain + narrow terminal)

jsonl.go:
- Replace per-record file.Sync() with bufio.Writer (64 KiB buffer)
  to reduce syscall overhead under high throughput
- Flush + close in Close(); data is always complete after scanner exits
- Extract parseAnswers() helper: handles CNAME/NS/PTR/TXT/AAAA/A
  prefixes, shared with beautified.go to eliminate duplication
- Add AAAA record type detection (was falling through to plain-IP path)
- Stable JSON field names: domain / type / records / timestamp

beautified.go:
- Use shared parseAnswers() instead of inline type-parsing loop
- Add Runner.SuccessCount() uint64 method (reads successCount atomically)
- enum.go + verify.go: call os.Exit(1) after Close() when SuccessCount==0
- Enables correct shell pipeline semantics:
    ksubdomain enum -d example.com && httpx ...
  (httpx only runs if at least one subdomain was found)
quickstart.md:
- Prerequisites, installation, first scan examples
- Common flags table, output formats, bandwidth tuning
- Quick troubleshooting table

api.md:
- Full SDK API reference: Config, Result, Scanner methods
- Error sentinels with errors.Is examples
- Custom output sink interface + example
- Complete working example

best-practices.md:
- Bandwidth selection table by connection type
- DNS resolver selection and anti-overload guidance
- Wildcard DNS detection and filter-mode guide
- Pipe-friendly --silent --od recipe
- JSONL + jq analysis examples
- Large-scale enumeration tips
- SDK cancellation and thread-safety guidance

faq.md:
- Permissions: sudo vs CAP_NET_RAW
- Interface errors: not found, not active, auto-detect
- macOS ENOBUFS / BPF buffer
- WSL2 interface and libpcap setup
- DNS: wildcard, SERVFAIL, incomplete results
- httpx garbled input fix
- jq JSONL parse errors
- Build: libpcap cross-compile
- Exit code table
…endCycleWithContext

RunEnumeration: goroutine topology diagram showing full data flow from
  domainChan through statusDB, retry loop, recv pipeline to outputters.

sendCycleWithContext: document batching rationale (amortised serialisation
  cost), 10ms ticker flush, and backpressure protocol with recv.go.
simple/main.go:
- Fix Records[0] index-out-of-range (use strings.Join instead)
- Add errors.Is checks for ErrPermissionDenied / ErrDeviceNotFound

advanced/main.go:
- Remove Timeout field (removed from Config)
- Use errors.Is for context.DeadlineExceeded and sdk.ErrPermissionDenied
- Guard speed calc against zero elapsed

.gitignore: add 'simple' and 'advanced' binary names
- Trigger on push to main/dev/feature/** and PRs to main/dev
- Build matrix: Linux amd64 (static libpcap), Linux arm64 (CGO=0),
  macOS amd64, macOS arm64, Windows amd64 (cross-compiled CGO=0)
- fail-fast: false so all platform results are visible simultaneously
- Separate lint job: go vet + staticcheck (no root/pcap required)
- Uses actions/checkout@v4, actions/setup-go@v5 (latest stable)
- pkg/core/conf/config.go: change Version from const to var so ldflags
  can override it at link time; default value '2.4-dev' distinguishes
  untagged dev builds from official releases
- build.yml: add -ldflags to all three go build invocations, injecting
  the git tag captured in get_version step
- build.sh: rewrite as proper bash script; auto-detect version from
  'git describe --tags --always'; support override via $1 argument;
  add Windows cross-compile step
Revert .github/workflows/build.yml to main version.
Remove .github/workflows/ci.yml (added in P3-7).

CI matrix and ldflags changes can be applied directly on GitHub
or after re-generating a token with 'workflow' scope.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant