Skip to content

feat: enhance telemetry, transport architecture, and memory management#4

Merged
gsmlg merged 11 commits intomainfrom
develop
Oct 29, 2025
Merged

feat: enhance telemetry, transport architecture, and memory management#4
gsmlg merged 11 commits intomainfrom
develop

Conversation

@gsmlg
Copy link
Copy Markdown
Contributor

@gsmlg gsmlg commented Oct 29, 2025

Summary

This PR introduces several major improvements and bug fixes to enhance telemetry, modularize the transport layer, and add configurable memory management.

New Features

  • Modular UDP transport architecture

    • Abyss.Transport.UDP.Core - Core UDP socket operations
    • Abyss.Transport.UDP.Unicast - Unicast-specific functionality with proper resource cleanup
    • Abyss.Transport.UDP.Broadcast - Broadcast and multicast support
  • Configurable telemetry sampling

    • connection_telemetry_sample_rate (default: 0.05) - Control connection event sampling
  • Configurable handler memory monitoring

    • handler_memory_check_interval (default: 10s)
    • handler_memory_warning_threshold (default: 100MB)
    • handler_memory_hard_limit (default: 150MB)
  • Comprehensive CHANGELOG.md documentation

Improvements

  • Better listener scaling: Changed algorithm from 1:1000 to 1:100 ratio (1 listener per 100 connections) for better granularity under varying loads
  • Enhanced telemetry: Rolling window rate calculations with atomic ETS operations
  • Improved configuration validation: Clear error messages for all new options
  • Better documentation: Updated README with new features and architecture details

Bug Fixes

  • Critical: Fixed ETS race condition in telemetry table creation
  • Critical: Fixed time unit inconsistency in adaptive timeout calculation
  • Critical: Fixed socket leak in UDP unicast send_recv function
  • Medium: Fixed telemetry sampling to use configurable rate from ServerConfig
  • Medium: Fixed rolling window rate calculation in integration tests

Test Improvements

  • Increased test coverage: 40% → 62%+
  • Removed 5 skipped/placeholder tests for cleaner test suite
  • Updated all tests to match improved algorithms
  • Result: 237 tests passing, 0 failures, 0 skipped

Breaking Changes

None - all changes are backward compatible with sensible defaults.

Checklist

  • All tests passing (237 tests, 0 failures)
  • Code formatted (mix format)
  • Test coverage increased (62%+)
  • Documentation updated (README.md, CHANGELOG.md)
  • No breaking changes

This commit introduces several major improvements and bug fixes:

## New Features

- Add modular UDP transport architecture with Core, Unicast, and Broadcast modules
- Add configurable telemetry sampling rate (connection_telemetry_sample_rate)
- Add configurable handler memory monitoring:
  - handler_memory_check_interval (default: 10s)
  - handler_memory_warning_threshold (default: 100MB)
  - handler_memory_hard_limit (default: 150MB)
- Add comprehensive CHANGELOG.md documentation

## Improvements

- Improve listener scaling algorithm: 1:100 ratio (was 1:1000)
  Provides better granularity for low to medium connection loads
- Enhanced telemetry with rolling window rate calculations
- Add atomic ETS operations for concurrent metric updates
- Improve configuration validation with clear error messages

## Bug Fixes

- Fix critical ETS race condition in telemetry table creation
- Fix time unit inconsistency in adaptive timeout calculation
- Fix socket leak in UDP unicast send_recv function
- Fix telemetry sampling to use configurable rate from ServerConfig
- Fix rolling window rate calculation in telemetry integration tests

## Test Improvements

- Increase test coverage from 40% to 62%+
- Remove 5 skipped/placeholder tests for cleaner test suite
- Update all tests to match improved algorithms
- All 237 tests passing with 0 failures and 0 skipped

## Documentation

- Update README.md with new configuration options
- Document modular transport architecture
- Add comprehensive CHANGELOG.md
- Update inline code documentation
- Fix code formatting in server_config_test.exs
- Fix Dialyzer type spec for get_metrics/0 with detailed return type
- Fix unmatched return warnings for :ets.update_counter calls
- Fix unreachable pattern match in listener_pool_scaler.ex
- All 237 tests passing, 0 failures, 62%+ coverage

CI checks fixed:
- Format check: PASS
- Dialyzer: PASS (all type issues resolved)
- Tests: PASS (237 tests, 0 failures)

Note: Credo warnings remain (58 total) but are style-related,
not functional issues. These can be addressed in a future PR.
- Remove priv/plts/ directory from repository
- Add priv/plts/ to .gitignore
- PLT files should be generated locally, not committed
- This fixes Dialyzer CI failure due to hardcoded local paths
Fixed 24 Credo readability warnings for numbers larger than 9999.
All numbers now use underscores for better readability following
Elixir conventions.

Changes:
- test/abyss/handler_test.exs: 10 number formatting fixes
- test/abyss/connection_test.exs: 4 number formatting fixes
- test/abyss/server_config_test.exs: 4 number formatting fixes
- test/support/test_transport.ex: 2 number formatting fixes
- test/abyss/transport_udp_comprehensive_test.exs: 2 fixes
- test/abyss/transport/udp_test.exs: 1 fix
- test/abyss/transport/udp/unicast_test.exs: 1 fix
- test/abyss_test.exs: 1 fix

Examples:
- 12345 → 12_345 (port numbers)
- 10000 → 10_000 (timeouts)
- 16384 → 16_384 (buffer sizes)
- 99999 → 99_999 (test values)

All tests passing (237 tests, 0 failures). No functionality changes.
Fixed 18 Credo software design warnings by aliasing nested modules
at the top of modules. This improves code readability by using
shorter module references throughout the codebase.

Changes:
- lib/abyss/handler.ex: Added UDP alias in genserver_impl quote block (5 usages)
- lib/abyss/connection.ex: Added UDP alias at module level (1 usage)
- test/support/test_helper.ex: Added UDP alias (3 usages)
- test/support/test_handler.ex: Added UDP alias in TestEchoHandler (1 usage)
- test/integration/echo_test.exs: Added UDP alias (8 usages)

Before: Abyss.Transport.UDP.controlling_process(socket, pid)
After:  UDP.controlling_process(socket, pid)

All tests passing (237 tests, 0 failures). No functionality changes.

Credo warnings reduced from 34 to 16 (18 fixed).
Cumulative: 72% reduction (58 → 16 warnings).
- Add :rate_limit_exceeded and :packet_too_large to telemetry event_name type
- Fix retry_start and retry_start_active return type specs to include all possible returns
- Add explicit discard assignments (_ =) for unmatched returns in:
  - Connection.start calls in listener.ex
  - UDP.controlling_process in connection.ex
  - Task.start calls for non-blocking retries
  - Process.send_after calls
- Fix listener_pool_scaler check_and_scale typespec to accept GenServer.server()
- Simplify DynamicSupervisor.which_children pattern matching (always returns list)

Reduced Dialyzer errors from 22 to 3 (86% reduction)
- Change listener.ex init/1 return type from state to map()
- Refine retry_start and retry_start_active list parameter types from list() to [term, ...]

All Dialyzer errors resolved (100% - from 22 errors to 0)
…ity)

- Extract try block bodies into private do_* functions in server.ex:
  - resume -> do_resume
  - suspend -> do_suspend
  - listener_pool_pid -> do_listener_pool_pid
  - connection_sup_pid -> do_connection_sup_pid
- Extract try block bodies into private do_* functions in listener_pool.ex:
  - listener_pids -> do_listener_pids
  - suspend -> do_suspend
  - resume -> do_resume

All 7 explicit try block warnings resolved. Credo warnings reduced from 16 to 9.
- Reverse negated condition in rate_limiter.ex (if not enabled -> if enabled)
- Extract rate calculation logic into calculate_rate/3 helper function
- Reduce nesting in get_accept_rate and get_response_rate by using helper

Credo warnings reduced from 9 to 6.
Reduced all Credo refactoring warnings from 6 to 0:

- Converted nested if-else to cond in listener.ex (reduced nesting depth)
- Extracted handle_received_packet helper to reduce cyclomatic complexity
- Extracted validation helpers in server_config.ex (reduced complexity from 10 to 3)
- Extracted check_memory_after_gc helper in handler macro (reduced nesting)
- Added Credo disable comments for legitimate macro complexity

All changes maintain test coverage at 64.99% with 237 tests passing.
@gsmlg gsmlg merged commit ef5ee6b into main Oct 29, 2025
6 checks passed
@gsmlg gsmlg deleted the develop branch October 29, 2025 09:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants