Enhance DecentDB integration, import features, and logging#1
Merged
Conversation
- Updated GitHub Actions workflow to streamline DecentDB native library downloads for Linux, macOS, and Windows. - Removed Nim setup step and integrated direct asset downloading based on the resolved DecentDB version. - Implemented a new `DecentDbNativeReleaseAsset` class to manage the downloading and caching of native library assets. - Modified the native library resolver to include cached asset candidates and environment variable support. - Updated README and documentation to reflect changes in native library handling and installation. - Added tests for the new asset management functionality and ensured existing tests are compatible with the new structure. - Adjusted CMake configuration to stage the DecentDB native library during the Linux build process.
- Introduced a headless CLI import mode for Decent Bench, allowing users to run imports without the desktop UI. This includes options for inferred defaults, explicit plans, and silent mode. - Removed the proposed external TOML theme system ADR as it has been replaced with a new version. - Updated references in the custom SQL editor surface rendering ADR to point to the new theme system ADR. - Added tests for SQL dump import models, Excel source preparer, layout persistence service, shortcut config service, and SQL dump import support. - Expanded import support to include CSV, JSON, XML, HTML tables, and ZIP/GZip archives, while maintaining existing dedicated wizards for Excel, SQLite, and SQL dumps. - Established a rationale for using a Git dependency for the DecentDB package to ensure reproducible builds and direct access to upstream Dart FFI bindings.
…ogging, and native library support
…pport, and adjust mock file description
There was a problem hiding this comment.
Pull request overview
This PR updates Decent Bench’s native DecentDB integration and import UX, aiming to make CI/release packaging more reliable by downloading prebuilt native assets, expanding archive detection, and improving operational visibility via in-app log viewing.
Changes:
- Modernize CI/release workflows and staging tooling to use pinned, prebuilt DecentDB native release assets instead of building DecentDB from source.
- Expand import detection/extraction (notably tar+gzip / tar+bzip2 paths) and improve import failure UX with blocking dialogs and richer summaries.
- Bump versioning/docs (1.1.0), add ADRs documenting import scope and native dependency rationale, and add/adjust tests accordingly.
Reviewed changes
Copilot reviewed 60 out of 62 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| test-data/sql_related/mysql_mock_export.bak | Updates .bak fixture text used for detection/UI tests. |
| design/adr/0027-container-assisted-proprietary-imports.md | Adds ADR proposal for container-assisted .bak imports. |
| design/adr/0026-tar-bzip2-gzip-archive-import.md | Adds ADR documenting tar+bzip2/tar+gzip import approach. |
| design/adr/0025-decentdb-git-dependency-rationale.md | Adds ADR explaining pinned git dependency strategy for decentdb. |
| design/adr/0024-import-scope-expansion-beyond-prd-mvp.md | Adds ADR documenting import scope expansion beyond PRD MVP. |
| design/adr/0023-external-toml-theme-system.md | Fixes ADR header numbering/reference consistency. |
| design/adr/0015-custom-sql-editor-surface-rendering.md | Updates ADR reference link to renamed/renumbered ADR. |
| design/SPEC.md | Documents shipped headless CLI import mode in the spec. |
| apps/decent-bench/windows/runner/Runner.rc | Bumps Windows resource version strings to 1.1.0. |
| apps/decent-bench/tool/stage_decentdb_native.dart | Staging helper now auto-ensures pinned native asset when --source omitted. |
| apps/decent-bench/test/support/import_fixture_manifest.dart | Updates expected detection results for .bak fixtures. |
| apps/decent-bench/test/support/fakes.dart | Extends fake gateway summaries to include new SQLite import summary fields. |
| apps/decent-bench/test/shared/widgets/import_failure_dialog_test.dart | Adds unit tests for import failure summary extraction. |
| apps/decent-bench/test/features/workspace/presentation/shell/app_menu_bar_test.dart | Adds widget test ensuring “View Log” appears in Tools menu. |
| apps/decent-bench/test/features/workspace/infrastructure/sqlite_import_support_test.dart | Adds tests for high-precision timestamp parsing and sparse temporal columns. |
| apps/decent-bench/test/features/workspace/infrastructure/sql_dump_import_support_test.dart | Adds tests for SQL dump inspection/materialization/type mapping and encoding handling. |
| apps/decent-bench/test/features/workspace/infrastructure/shortcut_config_service_test.dart | Adds tests for shortcut parsing/display formatting and default binding load behavior. |
| apps/decent-bench/test/features/workspace/infrastructure/native_library_resolver_test.dart | Updates resolver tests for cached pinned asset resolution behavior. |
| apps/decent-bench/test/features/workspace/infrastructure/layout_persistence_service_test.dart | Adds tests for layout persistence normalization and round-trips. |
| apps/decent-bench/test/features/workspace/infrastructure/excel_source_preparer_test.dart | Adds tests for Excel normalization and temp file cleanup behavior. |
| apps/decent-bench/test/features/workspace/infrastructure/decentdb_native_release_asset_test.dart | Adds unit tests for lockfile parsing and asset selection logic. |
| apps/decent-bench/test/features/workspace/infrastructure/decentdb_bridge_smoke_test.dart | Updates smoke tests for resolver candidates + adds timestamp import coverage + summary assertions. |
| apps/decent-bench/test/features/workspace/domain/sql_dump_import_models_test.dart | Adds tests for SQL dump import domain model behaviors. |
| apps/decent-bench/test/features/workspace/application/workspace_controller_test.dart | Adds tests for View Log open + import failure state propagation. |
| apps/decent-bench/test/features/import/infrastructure/import_fixture_round_trip_test.dart | Updates system library resolution to use resolver candidates. |
| apps/decent-bench/test/features/import/infrastructure/import_detection_tar_test.dart | Adds tests validating tar+gzip/tar+bzip2 candidate listing and extraction. |
| apps/decent-bench/test/app/theme_system/theme_manager_test.dart | Updates theme compatibility messages to 1.1.0. |
| apps/decent-bench/test/app/headless_import_runner_test.dart | Ensures native asset availability during headless import tests and refactors formatting. |
| apps/decent-bench/pubspec.yaml | Bumps app version and updates decentdb git ref to v2.3.0. |
| apps/decent-bench/pubspec.lock | Updates locked decentdb version/ref to v2.3.0. |
| apps/decent-bench/linux/runner/main.cc | Updates CLI help/version output to 1.1.0. |
| apps/decent-bench/linux/CMakeLists.txt | Adds install-time native staging into Linux bundle. |
| apps/decent-bench/lib/shared/widgets/import_failure_dialog.dart | Adds reusable blocking import failure dialog + summarizer. |
| apps/decent-bench/lib/features/workspace/presentation/workspace_screen.dart | Routes .bak to new dialog; adds “Tools -> View Log”; updates unknown-type supported list. |
| apps/decent-bench/lib/features/workspace/presentation/sqlite_import_dialog.dart | Shows blocking failure dialog on failed SQLite import; enriches summary display. |
| apps/decent-bench/lib/features/workspace/presentation/sql_dump_import_dialog.dart | Shows blocking failure dialog on failed SQL dump import; improves empty summary messaging. |
| apps/decent-bench/lib/features/workspace/presentation/shell/app_menu_bar.dart | Adds “View Log” menu item to platform + in-app menu structures. |
| apps/decent-bench/lib/features/workspace/presentation/ms_sql_bak_import_dialog.dart | Adds initial .bak import dialog that checks Docker availability (scaffold). |
| apps/decent-bench/lib/features/workspace/presentation/excel_import_dialog.dart | Shows blocking failure dialog on failed Excel import; improves empty summary messaging. |
| apps/decent-bench/lib/features/workspace/infrastructure/sqlite_import_support.dart | Adds checkpoint/WAL cleanup + richer summary metrics + improved timestamp parsing/inference. |
| apps/decent-bench/lib/features/workspace/infrastructure/sql_dump_import_support.dart | Adds checkpointing after SQL dump import commit. |
| apps/decent-bench/lib/features/workspace/infrastructure/native_library_resolver.dart | Adds cached pinned-asset candidates into native library search path order. |
| apps/decent-bench/lib/features/workspace/infrastructure/excel_import_support.dart | Adds checkpointing after Excel import commit. |
| apps/decent-bench/lib/features/workspace/infrastructure/decentdb_native_release_asset.dart | New helper to locate/download/cache pinned DecentDB native release assets. |
| apps/decent-bench/lib/features/workspace/domain/sqlite_import_models.dart | Extends SQLite import summary model with object counts and file/WAL sizes. |
| apps/decent-bench/lib/features/workspace/application/workspace_controller.dart | Adds openLogDatabase() and improves failure state propagation to workspace errors. |
| apps/decent-bench/lib/features/import/utils/docker_cli.dart | New helper to check Docker availability. |
| apps/decent-bench/lib/features/import/presentation/generic_import_dialog.dart | Adds blocking failure dialog handling and improved summary-step behavior. |
| apps/decent-bench/lib/features/import/infrastructure/import_format_registry.dart | Adds .bak format + marks bzip2 wrapper complete; updates wrapper metadata. |
| apps/decent-bench/lib/features/import/infrastructure/import_execution_service.dart | Adds checkpointing after generic import commit. |
| apps/decent-bench/lib/features/import/infrastructure/import_detection_service.dart | Adds tar+gzip and tar+bzip2 candidate listing/extraction paths. |
| apps/decent-bench/lib/features/import/domain/import_models.dart | Adds msSqlBak to ImportFormatKey. |
| apps/decent-bench/lib/app/logging/import_log_details.dart | Logs new SQLite import summary metrics. |
| apps/decent-bench/lib/app/app_metadata.dart | Bumps app metadata version string to 1.1.0. |
| apps/decent-bench/integration_test/workspace_shell_test.dart | Adds additional UI interaction coverage and test infrastructure changes. |
| apps/decent-bench/README.md | Updates app README to describe pinned native assets and staging behavior. |
| THIRD_PARTY_NOTICES.md | Updates attribution/source details for decentdb and adds dependency notes. |
| README.md | Updates root README features/support matrix, onboarding, and roadmap for 1.1.0. |
| CHANGELOG.md | Adds 1.1.0 release notes and updates compare links. |
| .github/workflows/release.yml | Switches releases to download pinned DecentDB assets; adds arch suffixes and stricter upload checks. |
| .github/workflows/flutter-build.yml | Switches CI to download/install pinned DecentDB assets instead of building from source. |
| .github/GEMINI.md | Adds documentation for maintaining .github/skills/* content. |
Comment on lines
+197
to
+200
| supportState: ImportSupportState.complete, | ||
| extensions: <String>['.bak'], | ||
| implementationKind: ImportImplementationKind.legacyWizard, | ||
| description: 'Container-assisted MS SQL backup import.', |
Comment on lines
258
to
+275
| ImportFormatDefinition( | ||
| key: ImportFormatKey.gzipArchive, | ||
| label: 'GZip Wrapper', | ||
| family: ImportFamily.compressedArchive, | ||
| supportState: ImportSupportState.complete, | ||
| extensions: <String>['.gz'], | ||
| implementationKind: ImportImplementationKind.wrapper, | ||
| description: | ||
| 'Single-file wrapper that unwraps supported CSV/JSON/NDJSON/XML/HTML/SQL/Excel/SQLite files.', | ||
| ), | ||
| ImportFormatDefinition( | ||
| key: ImportFormatKey.bzip2Archive, | ||
| label: 'BZip2 Wrapper', | ||
| label: 'BZip2 / Tar+BZip2 Wrapper', | ||
| family: ImportFamily.compressedArchive, | ||
| supportState: ImportSupportState.investigate, | ||
| supportState: ImportSupportState.complete, | ||
| extensions: <String>['.bz2'], | ||
| implementationKind: ImportImplementationKind.recognizedUnsupported, | ||
| description: 'BZip2 compressed wrapper support.', | ||
| implementationKind: ImportImplementationKind.wrapper, | ||
| description: |
Comment on lines
+1861
to
+1896
| if (inferredType == 'TIMESTAMP' && | ||
| !_columnContainsOnlyTimestampLikeValues(database, tableName, column)) { | ||
| continue; | ||
| } | ||
| if (inferredType != null && inferredType != column.inferredTargetType) { | ||
| inferredColumns[column.sourceName] = inferredType; | ||
| } | ||
| } | ||
| return inferredColumns; | ||
| } | ||
|
|
||
| bool _columnContainsOnlyTimestampLikeValues( | ||
| sqlite.Database database, | ||
| String tableName, | ||
| SqliteImportColumnDraft column, | ||
| ) { | ||
| final quotedTable = _quoteSqliteIdent(tableName); | ||
| final quotedColumn = _quoteSqliteIdent(column.sourceName); | ||
| final statement = database.prepare( | ||
| 'SELECT $quotedColumn AS value FROM $quotedTable WHERE $quotedColumn IS NOT NULL', | ||
| ); | ||
| try { | ||
| final cursor = statement.selectCursor(); | ||
| while (cursor.moveNext()) { | ||
| if (tryParseSqliteTimestampValue( | ||
| cursor.current['value'], | ||
| columnName: column.sourceName, | ||
| ) == | ||
| null) { | ||
| return false; | ||
| } | ||
| } | ||
| return true; | ||
| } finally { | ||
| statement.close(); | ||
| } |
Comment on lines
+221
to
+329
| Future<DecentDbNativeReleaseDownload> _resolveDownload() async { | ||
| final metadata = await _fetchReleaseMetadata(tag); | ||
| return selectDownload( | ||
| metadata: metadata, | ||
| tag: tag, | ||
| releaseSuffix: releaseSuffix, | ||
| archiveExtension: archiveExtension, | ||
| ); | ||
| } | ||
|
|
||
| static DecentDbNativeReleaseDownload selectDownload({ | ||
| required Map<String, Object?> metadata, | ||
| required String tag, | ||
| required String releaseSuffix, | ||
| required String archiveExtension, | ||
| }) { | ||
| final rawAssets = metadata['assets']; | ||
| if (rawAssets is! List) { | ||
| throw StateError( | ||
| 'Release metadata for $tag did not include an asset list.', | ||
| ); | ||
| } | ||
| final suffix = '-$releaseSuffix.$archiveExtension'; | ||
| final matches = | ||
| rawAssets | ||
| .whereType<Map<String, Object?>>() | ||
| .map(DecentDbNativeReleaseDownload.fromJson) | ||
| .where( | ||
| (asset) => | ||
| asset.name.endsWith(suffix) && | ||
| asset.name.startsWith('decentdb-') && | ||
| !asset.name.startsWith('decentdb-jdbc-') && | ||
| !asset.name.startsWith('decentdb-dbeaver-'), | ||
| ) | ||
| .toList() | ||
| ..sort((left, right) { | ||
| final leftPriority = left.name.contains('dart-native') ? 0 : 1; | ||
| final rightPriority = right.name.contains('dart-native') ? 0 : 1; | ||
| final priorityCompare = leftPriority.compareTo(rightPriority); | ||
| if (priorityCompare != 0) { | ||
| return priorityCompare; | ||
| } | ||
| return left.name.compareTo(right.name); | ||
| }); | ||
|
|
||
| if (matches.isEmpty) { | ||
| final assetNames = | ||
| rawAssets | ||
| .whereType<Map<String, Object?>>() | ||
| .map((asset) => asset['name']) | ||
| .whereType<String>() | ||
| .toList() | ||
| ..sort(); | ||
| throw StateError( | ||
| 'No DecentDB release asset matched $tag / $releaseSuffix.$archiveExtension. ' | ||
| 'Available assets: ${assetNames.join(', ')}', | ||
| ); | ||
| } | ||
|
|
||
| return matches.first; | ||
| } | ||
|
|
||
| static Future<Map<String, Object?>> _fetchReleaseMetadata(String tag) async { | ||
| final responseBytes = await _downloadArchive( | ||
| Uri.parse( | ||
| 'https://api.github.com/repos/sphildreth/decentdb/releases/tags/$tag', | ||
| ), | ||
| requestJson: true, | ||
| ); | ||
| final decoded = jsonDecode(utf8.decode(responseBytes)); | ||
| if (decoded is! Map<String, Object?>) { | ||
| throw StateError('Unexpected GitHub release metadata payload for $tag.'); | ||
| } | ||
| return decoded; | ||
| } | ||
|
|
||
| static Future<Uint8List> _downloadArchive( | ||
| Uri uri, { | ||
| bool requestJson = false, | ||
| }) async { | ||
| final client = HttpClient(); | ||
| try { | ||
| final request = await client.getUrl(uri); | ||
| if (requestJson) { | ||
| request.headers.set( | ||
| HttpHeaders.acceptHeader, | ||
| 'application/vnd.github+json', | ||
| ); | ||
| } | ||
| request.headers.set( | ||
| HttpHeaders.userAgentHeader, | ||
| 'decent-bench-native-asset', | ||
| ); | ||
| final response = await request.close(); | ||
| if (response.statusCode != HttpStatus.ok) { | ||
| throw HttpException( | ||
| 'Failed to download $uri (HTTP ${response.statusCode})', | ||
| uri: uri, | ||
| ); | ||
| } | ||
| final bytes = BytesBuilder(copy: false); | ||
| await for (final chunk in response) { | ||
| bytes.add(chunk); | ||
| } | ||
| return bytes.takeBytes(); | ||
| } finally { | ||
| client.close(force: true); | ||
| } | ||
| } |
Comment on lines
+13
to
+17
| setUpAll(() async { | ||
| await DecentDbNativeReleaseAsset.ensureAvailableForCurrentProject( | ||
| startPath: Directory.current.path, | ||
| ); | ||
| }); |
Comment on lines
+74
to
+75
| | `.gz`, `.tar.gz`, `.tgz` | **Unwrap & Import** | Supports single-file gzip unwrap and tar+gzip archive inspection/extraction. | | ||
| | `.bz2`, `.tar.bz2`, `.tbz2` | **Unwrap & Import** | Supports single-file bzip2 unwrap and tar+bzip2 archive inspection/extraction. | |
Comment on lines
96
to
+101
| - os: macos-latest | ||
| flutter_target: macos | ||
| bundle_path: build/macos/Build/Products/Release/decent_bench.app | ||
| native_lib: ../../../decentdb/build/libdecentdb.dylib | ||
| decentdb_release_suffix: macOS-arm64 | ||
| native_archive_extension: tar.gz | ||
| native_lib_name: libdecentdb.dylib |
Comment on lines
+121
to
+127
| try { | ||
| final result = Process.runSync( | ||
| 'tar', | ||
| listArgs, | ||
| stdoutEncoding: utf8, | ||
| stderrEncoding: utf8, | ||
| ); |
Comment on lines
+341
to
+357
| final baseName = p.basename(entryPath); | ||
| final inferredFormat = _inferFormatForExtensionlessTarEntry(entryPath); | ||
| final extension = inferredFormat.extensions.isNotEmpty | ||
| ? inferredFormat.extensions.first | ||
| : ''; | ||
| final outputFileName = | ||
| extension.isNotEmpty && !baseName.toLowerCase().endsWith(extension) | ||
| ? '$baseName$extension' | ||
| : baseName; | ||
| final outputPath = p.join(tempDir.path, outputFileName); | ||
|
|
||
| final result = Process.runSync( | ||
| 'tar', | ||
| [extractFlag, archivePath, '-C', tempDir.path, '--no-anchored', baseName], | ||
| stdoutEncoding: utf8, | ||
| stderrEncoding: utf8, | ||
| ); |
Comment on lines
+620
to
+630
| DetectionFixtureEntry( | ||
| relativePath: 'test-data/sql_related/mysql_mock_export.bak', | ||
| expectedFormatKey: ImportFormatKey.unknown, | ||
| expectedSupportState: ImportSupportState.notStarted, | ||
| expectedImplementationKind: ImportImplementationKind.unknown, | ||
| expectedFormatKey: ImportFormatKey.msSqlBak, | ||
| expectedSupportState: ImportSupportState.complete, | ||
| expectedImplementationKind: ImportImplementationKind.legacyWizard, | ||
| ), | ||
| DetectionFixtureEntry( | ||
| relativePath: 'test-data/sql_related/mariadb_mock_export.bak', | ||
| expectedFormatKey: ImportFormatKey.unknown, | ||
| expectedSupportState: ImportSupportState.notStarted, | ||
| expectedImplementationKind: ImportImplementationKind.unknown, | ||
| expectedFormatKey: ImportFormatKey.msSqlBak, | ||
| expectedSupportState: ImportSupportState.complete, | ||
| expectedImplementationKind: ImportImplementationKind.legacyWizard, |
Owner
Author
|
@copilot apply changes based on the comments in this thread |
- Fix catch (e) → catch (_) in docker_cli.dart - Downgrade msSqlBak to investigate/recognizedUnsupported (not implemented) - Add .tgz/.tbz2 extensions and route via tar detection/extraction - Update .bak fixture expectations to match new support state - Bound _columnContainsOnlyTimestampLikeValues() to 1000 rows - Support GITHUB_TOKEN env var for GitHub API rate limit mitigation - Gate native download in headless test behind DECENT_BENCH_DOWNLOAD_NATIVE=1 - Compute macOS arch suffix dynamically from RUNNER_ARCH - Replace Process.runSync with async Process.run in _detectTarCandidates - Fix _extractTarEntry to use exact path instead of --no-anchored basename Agent-Logs-Url: https://github.com/sphildreth/decent-bench/sessions/c7be6d7e-0219-4294-8d4f-8129405fc85d Co-authored-by: sphildreth <193334+sphildreth@users.noreply.github.com>
Contributor
Applied all review feedback in commit b019d1d:
|
Owner
Author
|
@copilot resolve failing PR check |
…nux desktop shutdown crash Agent-Logs-Url: https://github.com/sphildreth/decent-bench/sessions/0f034488-3b7b-48d0-8e9a-46e4e4dec500 Co-authored-by: sphildreth <193334+sphildreth@users.noreply.github.com>
Contributor
The failing check was the "Flutter integration test" step in Fixed in commit Warning Firewall rules blocked me from connecting to one or more addresses (expand for details)I tried to connect to the following addresses, but was blocked by firewall rules:
If you need me to access, download, or install something from one of these locations, you can either:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request modernizes and simplifies the native library integration for Decent Bench desktop builds and releases, switching from building DecentDB from source to downloading prebuilt native assets. It also improves cross-platform support, workflow reliability, and release artifact naming. Additionally, the changelog is updated for version 1.1.0, and minor documentation fixes are included.
CI/CD Workflow Modernization and Native Library Handling:
flutter-build.ymlandrelease.yml. This removes the need for Nim and manual dependency setup, making builds faster and more reliable. [1] [2] [3] [4] [5] [6]x64,arm64), and updates release upload patterns to match new artifact names and formats. [1] [2] [3] [4] [5] [6]Documentation and Changelog Updates:
CHANGELOG.mdfor the 1.1.0 release, summarizing new features, changes, and fixes, including improved native library handling, import workflow enhancements, and bug fixes. [1] [2]README.mdnavigation links.