Feat/data profiling#7
Open
Tomic-Riedel wants to merge 27 commits intoHPI-Information-Systems:mainfrom
Open
Conversation
pd.Series(dtype=int, index=...) initializes with NaN which forces float64 conversion. Building a dict first preserves int64 dtype.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a comprehensive data profiling system to Metis that provides caching of profiling computations and support for importing pre-computed profiles from external tools. The system includes 30+ profiling functions covering cardinalities, value distributions, patterns, data types, domain classification, and similarity metrics. A DataProfileManager singleton manages database-backed caching, and a flexible importer system supports various input formats including HyFD, AIDFD, and CFDFinder outputs.
Changes:
- Renamed
mesTimetotimestampin DQResult class and database models for consistency - Added DataProfile database model for storing cached profiling results
- Implemented DataProfileManager singleton with database-backed caching
- Added 30+ profiling functions organized into cardinalities, value distribution, patterns/types, domain classification, and sketches
- Created importer system supporting inline JSON and external file formats for pre-computed profiles
- Integrated profiling system into DQOrchestrator with automatic context management
- Added comprehensive documentation for import formats
Reviewed changes
Copilot reviewed 57 out of 64 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
| requirements.txt | Added numpy and datasketch dependencies |
| metis/writer/database_writer.py | Updated to use timestamp instead of mesTime |
| metis/utils/result.py | Renamed mesTime field to timestamp in DQResult class |
| metis/utils/data_config.py | Added data_profiles configuration field, removed duplicate lines |
| metis/database_models.py | Added DataProfile model for caching profiling results |
| metis/dq_orchestrator.py | Integrated DataProfileManager initialization and profile importing |
| metis/profiling/data_profile_manager.py | Singleton manager for profile caching with serialization/deserialization |
| metis/profiling/cache.py | Decorator for transparent profiling function caching |
| metis/profiling/init.py | Exports cached versions of all profiling functions |
| metis/profiling/importers/* | Importer classes for various profile formats (FD, UCC, IND, histograms, etc.) |
| metis/profiling/single_column/* | Cached wrappers for profiling functions |
| metis/utils/data_profiling/* | Core implementations of 30+ profiling functions |
| metis/metric/* | Updated metrics to use timestamp instead of mesTime |
| docs/DATA_PROFILE_IMPORT_FORMATS.md | Comprehensive documentation of import formats |
| README.md | Added data profiling documentation and examples |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
metis/profiling/single_column/summaries_and_sketches/minhash.py
Outdated
Show resolved
Hide resolved
metis/utils/data_profiling/single_column/summaries_and_sketches/jaccard_similarity.py
Show resolved
Hide resolved
metis/utils/data_profiling/single_column/value_distribution/constancy.py
Show resolved
Hide resolved
metis/utils/data_profiling/single_column/domain_classification/domain.py
Show resolved
Hide resolved
metis/profiling/single_column/summaries_and_sketches/jaccard_similarity.py
Show resolved
Hide resolved
metis/utils/data_profiling/single_column/domain_classification/domain.py
Outdated
Show resolved
Hide resolved
Tomic-Riedel
force-pushed
the
feat/data-profiling
branch
from
68c0112 to
7846ef5
Compare
lisehr
approved these changes
Feb 25, 2026
Collaborator
lisehr
left a comment
lisehr
left a comment
There was a problem hiding this comment.
Thanks for all the great extension, I reviewed it and looks really good to me.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a data profiling module with functions for cardinality metrics, value distributions, patterns, and domain classification. All profiling results are cached in the database via a DataProfileManager singleton, which eliminates redundant computations across metric executions. Pre-computed profiles from external tools like HyFD or CFDFinder can be imported through the data_profiles field in data loader configs.