diff --git a/.github/workflows/repo-checks.yml b/.github/workflows/repo-checks.yml new file mode 100644 index 0000000..6f645f2 --- /dev/null +++ b/.github/workflows/repo-checks.yml @@ -0,0 +1,26 @@ +name: Repo checks + +on: + push: + branches: + - "**" + pull_request: + +permissions: + contents: read + +jobs: + repo-checks: + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Setup Python + uses: actions/setup-python@v5 + with: + python-version: "3.12" + + - name: Run repo checks + run: python3 scripts/repo_checks.py + diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..eda5d3c --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,18 @@ +## Contributing + +Contributions are welcome: new questions, fixes, better links, and formatting improvements. + +### Quick checks + +Run the local repository checks before opening a PR: + +```bash +python3 scripts/repo_checks.py +``` + +### Content guidelines + +- Keep files **UTF-8** encoded. +- Avoid placeholder links like `TODO` in markdown or HTML. +- If you add links in `content/full.md`, make sure the referenced heading exists in the target file. + diff --git a/README.md b/README.md index 6188ccb..ea5b7ed 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,9 @@

More than 2000+ questions for preparing a Data Engineer interview.

Full list of questions

+

Pick a topic below or use the full list to practice end-to-end.

Interview questions for Data Engineer

- @@ -91,7 +91,7 @@ - + @@ -100,6 +100,20 @@ + + + + + + + + + + + + + + @@ -117,7 +131,7 @@ - + @@ -132,7 +146,7 @@ - + @@ -194,6 +208,17 @@ + + + + + + + + + + + @@ -203,12 +228,60 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + @@ -224,19 +297,19 @@ - + - + - + @@ -245,4 +318,5 @@

Contribution

Please contribute to this repository to help it make better. Any change like new question, code improvement, doc improvement etc is very welcome.

-
\ No newline at end of file +

See CONTRIBUTING.md for quick checks and guidelines.

+ diff --git a/content/cdc.md b/content/cdc.md new file mode 100644 index 0000000..afc041e --- /dev/null +++ b/content/cdc.md @@ -0,0 +1,114 @@ +## [Main title](../README.md) +### [Interview questions](full.md) +# +# Change Data Capture (CDC) ++ [What is CDC (Change Data Capture)?](#What-is-CDC-(Change-Data-Capture)) ++ [When would you choose CDC over batch ingestion?](#When-would-you-choose-CDC-over-batch-ingestion) ++ [What are common CDC sources (WAL/binlog/triggers) and trade-offs?](#What-are-common-CDC-sources-(WAL/binlog/triggers)-and-trade-offs) ++ [What is the difference between snapshot and incremental CDC?](#What-is-the-difference-between-snapshot-and-incremental-CDC) ++ [What delivery semantics exist (at-most-once/at-least-once/exactly-once)?](#What-delivery-semantics-exist-(at-most-once/at-least-once/exactly-once)) ++ [How do you make CDC ingestion idempotent?](#How-do-you-make-CDC-ingestion-idempotent) ++ [How do you handle deletes in CDC pipelines?](#How-do-you-handle-deletes-in-CDC-pipelines) ++ [How do you handle updates when the source does not provide full row images?](#How-do-you-handle-updates-when-the-source-does-not-provide-full-row-images) ++ [What is ordering and why is it hard in CDC?](#What-is-ordering-and-why-is-it-hard-in-CDC) ++ [What is a watermark/offset and where should it be stored?](#What-is-a-watermark/offset-and-where-should-it-be-stored) ++ [How do you handle schema evolution with CDC?](#How-do-you-handle-schema-evolution-with-CDC) ++ [What is the outbox pattern and why is it useful?](#What-is-the-outbox-pattern-and-why-is-it-useful) ++ [How do you monitor CDC lag and where can lag come from?](#How-do-you-monitor-CDC-lag-and-where-can-lag-come-from) ++ [How do you design a safe backfill/reprocessing strategy for CDC?](#How-do-you-design-a-safe-backfill/reprocessing-strategy-for-CDC) ++ [What are the most common failure modes in CDC pipelines?](#What-are-the-most-common-failure-modes-in-CDC-pipelines) + +## What is CDC (Change Data Capture)? +CDC is a technique to capture row-level changes (inserts, updates, deletes) from a source system and stream them downstream. In databases, CDC is commonly log-based (reading WAL/binlog) so it can capture changes without repeatedly scanning full tables. CDC pipelines often feed event logs (append-only) and/or materialized current-state tables. + +[Table of Contents](#Change-Data-Capture-(CDC)) + +## When would you choose CDC over batch ingestion? +Choose CDC when you need low-latency updates, continuous incremental processing, or a reliable change history. CDC is also useful when full-table scans are too expensive because the source is large or changes frequently. Batch ingestion is often simpler when latency requirements are relaxed and operational complexity should be minimized. + +[Table of Contents](#Change-Data-Capture-(CDC)) + +## What are common CDC sources (WAL/binlog/triggers) and trade-offs? +Common sources are: ++ database logs (WAL/binlog): low overhead and high fidelity, but requires privileges and careful offset handling ++ triggers: can work without log access, but adds write overhead and is easy to break ++ timestamp-based queries: simple but can miss updates or create duplicates if clocks/transactions behave unexpectedly +Log-based CDC is usually preferred for production at scale. + +[Table of Contents](#Change-Data-Capture-(CDC)) + +## What is the difference between snapshot and incremental CDC? +A snapshot reads an initial consistent baseline of existing rows. Incremental CDC then streams changes after that baseline using a log position/offset. Many systems require both: snapshot for bootstrap, incremental for ongoing updates. + +[Table of Contents](#Change-Data-Capture-(CDC)) + +## What delivery semantics exist (at-most-once/at-least-once/exactly-once)? +At-most-once can lose messages but avoids duplicates. At-least-once avoids loss but can produce duplicates on retries. Exactly-once is hard end-to-end; many systems implement “effectively once” by combining at-least-once delivery with idempotent writes and deterministic merge logic. + +[Table of Contents](#Change-Data-Capture-(CDC)) + +## How do you make CDC ingestion idempotent? +You make downstream writes idempotent by using stable keys (primary key + source LSN/transaction id) and applying deterministic upsert/merge logic. A common pattern is to load changes into an append-only raw log, then build current-state tables using “latest per key” logic. Idempotency must hold across retries and replays. + +[Table of Contents](#Change-Data-Capture-(CDC)) + +## How do you handle deletes in CDC pipelines? +Deletes may appear as explicit delete events, tombstones, or a “before image” without an “after image”. Downstream you can: ++ apply hard deletes (remove row) ++ apply soft deletes (is_deleted flag with deleted_at) ++ keep full history (append-only) and compute current state with delete logic +The choice depends on compliance, analytics needs, and downstream consumers. + +[Table of Contents](#Change-Data-Capture-(CDC)) + +## How do you handle updates when the source does not provide full row images? +If updates are partial (only changed columns), you often need to reconstruct the full row by joining against previous state. Some CDC tools can be configured to emit before/after images; if not, current-state reconstruction becomes part of your pipeline. You must also handle out-of-order updates carefully. + +[Table of Contents](#Change-Data-Capture-(CDC)) + +## What is ordering and why is it hard in CDC? +Ordering means applying changes in the same order they happened in the source. It is hard because events can be partitioned (multiple tables/partitions), delivered out of order, or retried. Correctness often requires per-key ordering (or per-partition ordering) using source transaction metadata. + +[Table of Contents](#Change-Data-Capture-(CDC)) + +## What is a watermark/offset and where should it be stored? +An offset (watermark) is the position in the source log (or stream) up to which changes have been processed. It should be stored durably in the ingestion system (connector state store) and, for reliability, often also in your platform (for example a control table) so you can audit and recover. Offsets must be updated atomically with downstream writes to avoid gaps/duplicates. + +[Table of Contents](#Change-Data-Capture-(CDC)) + +## How do you handle schema evolution with CDC? +You need a strategy for compatible changes (additive columns) and breaking changes (renames, type changes). Common tools include schema registries and compatibility policies, plus strict validation in CI. Downstream tables should tolerate additive fields, and pipelines should alert on breaking changes before production data is corrupted. + +[Table of Contents](#Change-Data-Capture-(CDC)) + +## What is the outbox pattern and why is it useful? +The outbox pattern writes business events into an “outbox” table in the same transaction as the application write. CDC then captures the outbox changes and publishes them reliably. This avoids dual-write problems where the database update succeeds but the event publish fails (or vice versa). + +[Table of Contents](#Change-Data-Capture-(CDC)) + +## How do you monitor CDC lag and where can lag come from? +Lag can come from: ++ source: heavy write load, log retention limits ++ connector: slow polling, backpressure, failures ++ broker/stream: throughput limits, partition hotspots ++ downstream: slow merges/compaction, expensive transformations +You monitor lag using source LSN vs processed LSN, consumer offsets, and end-to-end freshness in target tables. + +[Table of Contents](#Change-Data-Capture-(CDC)) + +## How do you design a safe backfill/reprocessing strategy for CDC? +A safe strategy usually separates raw change storage from derived tables. You keep the raw CDC log (immutable) and rebuild derived current-state/marts from it when needed. For backfills, you often run in a separate environment or commit to new snapshots/versions, then switch consumers after validation. + +[Table of Contents](#Change-Data-Capture-(CDC)) + +## What are the most common failure modes in CDC pipelines? +Common issues include: ++ duplicates due to retries/rebalances ++ missed events due to offset mismanagement or log retention ++ schema drift/breaking changes ++ incorrect delete handling ++ out-of-order application of updates +Good pipelines treat CDC as “production software” with strong observability and tests. + +[Table of Contents](#Change-Data-Capture-(CDC)) + diff --git a/content/cost-optimization.md b/content/cost-optimization.md new file mode 100644 index 0000000..2d312ac --- /dev/null +++ b/content/cost-optimization.md @@ -0,0 +1,82 @@ +## [Main title](../README.md) +### [Interview questions](full.md) +# +# Cost Optimization ++ [Why is cost optimization a core data engineering skill?](#Why-is-cost-optimization-a-core-data-engineering-skill) ++ [What are the main cost drivers in data platforms?](#What-are-the-main-cost-drivers-in-data-platforms) ++ [How does partitioning affect cost and performance?](#How-does-partitioning-affect-cost-and-performance) ++ [What is the small files problem and why does it increase cost?](#What-is-the-small-files-problem-and-why-does-it-increase-cost) ++ [How do you choose a target file size for Parquet tables?](#How-do-you-choose-a-target-file-size-for-Parquet-tables) ++ [How do you detect and fix data skew in distributed processing?](#How-do-you-detect-and-fix-data-skew-in-distributed-processing) ++ [What is shuffle and how do you reduce it in Spark?](#What-is-shuffle-and-how-do-you-reduce-it-in-Spark) ++ [When should you pre-aggregate or materialize tables?](#When-should-you-pre-aggregate-or-materialize-tables) ++ [How do you prevent runaway queries and protect shared clusters?](#How-do-you-prevent-runaway-queries-and-protect-shared-clusters) ++ [How do you optimize joins in large-scale analytics?](#How-do-you-optimize-joins-in-large-scale-analytics) ++ [How do you plan and estimate the cost of a backfill?](#How-do-you-plan-and-estimate-the-cost-of-a-backfill) ++ [What metrics would you track for FinOps in data engineering?](#What-metrics-would-you-track-for-FinOps-in-data-engineering) + +## Why is cost optimization a core data engineering skill? +Data platforms can scale costs linearly or worse with data volume and usage. Cost optimization ensures the platform remains sustainable while meeting SLAs. It requires understanding storage layout, compute behavior, query patterns, and operational practices like backfills and compaction. + +[Table of Contents](#Cost-Optimization) + +## What are the main cost drivers in data platforms? +Typical drivers include: ++ compute time (clusters, warehouses, serverless slots) ++ bytes scanned and shuffles ++ storage growth (raw + derived + duplicates) ++ data movement (egress, cross-region transfers) ++ operational overhead (frequent backfills, retries) + +[Table of Contents](#Cost-Optimization) + +## How does partitioning affect cost and performance? +Good partitioning reduces scanned data by enabling partition pruning. Bad partitioning creates too many small partitions, increasing metadata overhead and small files. Partitioning should match common query filters and data distribution, and be reevaluated as workloads evolve. + +[Table of Contents](#Cost-Optimization) + +## What is the small files problem and why does it increase cost? +Small files increase scheduling and metadata overhead and reduce scan efficiency. Engines spend more time opening and planning files than processing data. Small files often appear from streaming writes or overly granular partitions and typically require compaction or clustering to fix. + +[Table of Contents](#Cost-Optimization) + +## How do you choose a target file size for Parquet tables? +You choose a size that balances parallelism and overhead. Too small increases file count and planning cost; too large reduces parallelism and can slow selective queries. Many teams target hundreds of MB per file, but the correct value depends on the engine, storage, and query patterns. + +[Table of Contents](#Cost-Optimization) + +## How do you detect and fix data skew in distributed processing? +Skew happens when some partitions have much more data than others, causing straggler tasks. Detect it via task duration distributions and partition size metrics. Fixes include salting keys, using skew-aware joins, repartitioning, or changing the join strategy (broadcast when possible). + +[Table of Contents](#Cost-Optimization) + +## What is shuffle and how do you reduce it in Spark? +Shuffle is data redistribution across executors (for joins, group by, distinct). It is expensive due to network IO and disk spills. You reduce shuffle by using proper partitioning, avoiding wide transformations, filtering early, broadcasting small tables, and tuning shuffle partitions. + +[Table of Contents](#Cost-Optimization) + +## When should you pre-aggregate or materialize tables? +Materialize when many downstream queries reuse expensive computations or when interactive BI requires low latency. Avoid over-materialization because it increases storage and refresh complexity. A good approach is to materialize stable, high-value marts and keep exploratory logic as views. + +[Table of Contents](#Cost-Optimization) + +## How do you prevent runaway queries and protect shared clusters? +Use workload management: query timeouts, concurrency limits, resource quotas, and separate compute for heavy workloads. Enforce best practices with guardrails (linting, cost alerts) and educate users with profiling tools. Multi-tenant platforms typically need isolation to prevent one team from impacting others. + +[Table of Contents](#Cost-Optimization) + +## How do you optimize joins in large-scale analytics? +You optimize joins by ensuring join keys are clean and well-distributed, filtering before joins, and choosing appropriate join strategies (broadcast vs shuffle). You also reduce the size of join inputs (select only needed columns) and consider pre-joining into curated marts when joins are repeated. + +[Table of Contents](#Cost-Optimization) + +## How do you plan and estimate the cost of a backfill? +Estimate based on data volume, compute requirements, and expected scan/shuffle behavior. Run on a small sample window to measure throughput and extrapolate. Backfills should be staged, monitored, and preferably run in off-peak windows, with a clear rollback plan. + +[Table of Contents](#Cost-Optimization) + +## What metrics would you track for FinOps in data engineering? +Common metrics include cost by team/project, cost per pipeline, bytes scanned per query, cluster utilization, storage growth by layer, and top expensive datasets/queries. You also track trend changes (week-over-week) and tie cost to value (usage, criticality). + +[Table of Contents](#Cost-Optimization) + diff --git a/content/data-governance.md b/content/data-governance.md new file mode 100644 index 0000000..b984d74 --- /dev/null +++ b/content/data-governance.md @@ -0,0 +1,81 @@ +## [Main title](../README.md) +### [Interview questions](full.md) +# +# Data Governance ++ [What is data governance and why does it matter?](#What-is-data-governance-and-why-does-it-matter) ++ [What is the difference between governance and security?](#What-is-the-difference-between-governance-and-security) ++ [What are common roles in data governance (owner/steward/custodian)?](#What-are-common-roles-in-data-governance-(owner/steward/custodian)) ++ [What is a data catalog and what should it contain?](#What-is-a-data-catalog-and-what-should-it-contain) ++ [What is business glossary vs technical metadata?](#What-is-business-glossary-vs-technical-metadata) ++ [RBAC vs ABAC: what is the difference?](#RBAC-vs-ABAC:-what-is-the-difference) ++ [What is row-level and column-level security?](#What-is-row-level-and-column-level-security) ++ [How do you handle PII/PHI data in analytics platforms?](#How-do-you-handle-PII/PHI-data-in-analytics-platforms) ++ [What is data masking and tokenization?](#What-is-data-masking-and-tokenization) ++ [How do you implement auditability for data access?](#How-do-you-implement-auditability-for-data-access) ++ [How do you handle GDPR “right to be forgotten” in a lakehouse/warehouse?](#How-do-you-handle-GDPR-“right-to-be-forgotten”-in-a-lakehouse/warehouse) ++ [What are data retention policies and how do you enforce them?](#What-are-data-retention-policies-and-how-do-you-enforce-them) + +## What is data governance and why does it matter? +Data governance is the set of processes, policies, and responsibilities that ensure data is discoverable, trusted, secure, and used correctly. It matters because data platforms scale to many teams; without governance, definitions diverge, access becomes risky, and compliance issues appear. Good governance enables faster, safer self-service analytics. + +[Table of Contents](#Data-Governance) + +## What is the difference between governance and security? +Security focuses on preventing unauthorized access and ensuring confidentiality/integrity (access controls, encryption). Governance includes security but also covers ownership, definitions, quality, lifecycle, and change management. Governance answers “who owns this data and what does it mean”, not just “who can access it”. + +[Table of Contents](#Data-Governance) + +## What are common roles in data governance (owner/steward/custodian)? +Common roles are: ++ data owner: accountable for the dataset and its business definition ++ data steward: maintains quality, metadata, and processes ++ data custodian: operates the technical platform (storage, access, backups) +Clear roles prevent “nobody owns it” incidents. + +[Table of Contents](#Data-Governance) + +## What is a data catalog and what should it contain? +A data catalog is an inventory of datasets with searchable metadata. It typically contains dataset descriptions, owners, lineage, classifications, freshness/SLA, schema, and links to dashboards. A good catalog reduces time to discovery and improves trust and reuse. + +[Table of Contents](#Data-Governance) + +## What is business glossary vs technical metadata? +A business glossary defines business terms (for example “active user”, “net revenue”) and their approved definitions. Technical metadata describes datasets and fields (schemas, types, partitions, lineage). Both are needed: glossary aligns meaning, technical metadata enables implementation and discovery. + +[Table of Contents](#Data-Governance) + +## RBAC vs ABAC: what is the difference? +RBAC (Role-Based Access Control) grants permissions to roles (analyst, engineer) and assigns users to roles. ABAC (Attribute-Based Access Control) uses attributes (department, region, sensitivity) to evaluate policies dynamically. ABAC is more flexible but can be harder to manage without good tooling. + +[Table of Contents](#Data-Governance) + +## What is row-level and column-level security? +Row-level security restricts which rows a user can see (for example only their region). Column-level security restricts which columns are visible (for example hide salary or PII). These controls are important for multi-tenant analytics and compliance. + +[Table of Contents](#Data-Governance) + +## How do you handle PII/PHI data in analytics platforms? +You classify sensitive fields, minimize exposure, and enforce least privilege. Common controls include encryption, masking/tokenization, column-level access, and auditing. You also ensure downstream marts only include necessary fields and implement secure sharing patterns. + +[Table of Contents](#Data-Governance) + +## What is data masking and tokenization? +Masking hides sensitive values (for example showing only last 4 digits) while keeping the dataset usable. Tokenization replaces sensitive values with tokens that can be detokenized only with access to a secure mapping. Tokenization is useful when analytics needs stable identifiers without exposing raw PII. + +[Table of Contents](#Data-Governance) + +## How do you implement auditability for data access? +You log access events (who queried what, when, from where) and retain logs per policy. You also track permission changes and administrative actions. Audit logs should be queryable for investigations and compliance reporting. + +[Table of Contents](#Data-Governance) + +## How do you handle GDPR “right to be forgotten” in a lakehouse/warehouse? +You need a process to delete or anonymize personal data across raw, derived, and serving layers. This can involve targeted deletes (where supported), reprocessing from raw logs without the user, or storing personal data separately to allow deletion without rewriting large datasets. You also document what is feasible given retention and legal constraints. + +[Table of Contents](#Data-Governance) + +## What are data retention policies and how do you enforce them? +Retention policies define how long data is kept and when it must be deleted or archived. Enforcement can be implemented with partition expiry jobs, lifecycle rules on object storage, snapshot expiration, and access reviews. Retention must align with legal requirements and business needs. + +[Table of Contents](#Data-Governance) + diff --git a/content/data-modeling.md b/content/data-modeling.md new file mode 100644 index 0000000..e5f9f1c --- /dev/null +++ b/content/data-modeling.md @@ -0,0 +1,109 @@ +## [Main title](../README.md) +### [Interview questions](full.md) +# +# Data Modeling ++ [What is data modeling in analytics and why does it matter?](#What-is-data-modeling-in-analytics-and-why-does-it-matter) ++ [What is grain and why is it the first question to answer?](#What-is-grain-and-why-is-it-the-first-question-to-answer) ++ [What is the difference between fact and dimension tables?](#What-is-the-difference-between-fact-and-dimension-tables) ++ [What is a star schema and what are its benefits?](#What-is-a-star-schema-and-what-are-its-benefits) ++ [Star schema vs snowflake schema: what are the trade-offs?](#Star-schema-vs-snowflake-schema:-what-are-the-trade-offs) ++ [What are the main types of fact tables?](#What-are-the-main-types-of-fact-tables) ++ [What is a surrogate key and when should you use it?](#What-is-a-surrogate-key-and-when-should-you-use-it) ++ [What is a conformed dimension?](#What-is-a-conformed-dimension) ++ [What is SCD (Slowly Changing Dimension)?](#What-is-SCD-(Slowly-Changing-Dimension)) ++ [How do you implement SCD Type 2?](#How-do-you-implement-SCD-Type-2) ++ [What is a factless fact table?](#What-is-a-factless-fact-table) ++ [What is a bridge table and when do you need it?](#What-is-a-bridge-table-and-when-do-you-need-it) ++ [What are degenerate dimensions?](#What-are-degenerate-dimensions) ++ [How do you prevent double counting in analytical models?](#How-do-you-prevent-double-counting-in-analytical-models) ++ [How do you validate a data model after changes?](#How-do-you-validate-a-data-model-after-changes) + +## What is data modeling in analytics and why does it matter? +Data modeling in analytics is the process of structuring data so it can be queried reliably and efficiently for business questions. A good model makes metrics consistent, reduces complexity for consumers, and improves performance by aligning storage with query patterns. A bad model leads to inconsistent KPIs, confusing joins, and expensive queries. + +[Table of Contents](#Data-Modeling) + +## What is grain and why is it the first question to answer? +Grain is the level of detail of a table, for example “one row per order line” or “one row per user per day”. It determines which metrics are additive, how joins behave, and what can be uniquely identified. If grain is unclear, you will get duplicates, incorrect aggregations, and ambiguous keys. + +[Table of Contents](#Data-Modeling) + +## What is the difference between fact and dimension tables? +Facts store measurable events (orders, payments, clicks) and typically contain foreign keys to dimensions. Dimensions store descriptive attributes (user, product, date) that provide context for slicing and filtering. Facts answer “what happened”, dimensions answer “who/what/where/when”. + +[Table of Contents](#Data-Modeling) + +## What is a star schema and what are its benefits? +A star schema has a central fact table connected to multiple dimension tables. Benefits include simpler queries, clear join paths, and good performance because joins are typically fact-to-dimension (many-to-one). It also encourages consistent definitions of metrics and attributes. + +[Table of Contents](#Data-Modeling) + +## Star schema vs snowflake schema: what are the trade-offs? +Snowflake schema normalizes dimensions into multiple related tables. It can reduce redundancy and improve dimension maintainability, but it increases join complexity and can hurt performance. Star schemas are usually preferred for BI and ad-hoc analytics because they are easier for users and tools. + +[Table of Contents](#Data-Modeling) + +## What are the main types of fact tables? +Common types are: ++ transactional: one row per business event (order, payment) ++ periodic snapshot: one row per entity per period (daily account balance) ++ accumulating snapshot: one row that is updated as a process progresses (order lifecycle) +Choosing the right type depends on the questions you need to answer. + +[Table of Contents](#Data-Modeling) + +## What is a surrogate key and when should you use it? +A surrogate key is a synthetic identifier used instead of a natural key. It is useful when natural keys are not stable, when you need SCD history, or when integrating multiple sources with different key systems. Surrogate keys also allow consistent joins even if source keys change. + +[Table of Contents](#Data-Modeling) + +## What is a conformed dimension? +A conformed dimension is a dimension shared consistently across multiple fact tables or data marts. For example, the same “customer” dimension used for orders and support tickets. Conformed dimensions enable consistent reporting and cross-domain analysis. + +[Table of Contents](#Data-Modeling) + +## What is SCD (Slowly Changing Dimension)? +SCD describes how you handle changes in dimension attributes over time (for example a user changing address). Type 1 overwrites history, Type 2 tracks history with multiple rows and effective dates, and Type 3 keeps limited history in additional columns. The choice depends on whether “as-of” reporting is required. + +[Table of Contents](#Data-Modeling) + +## How do you implement SCD Type 2? +SCD2 typically stores: ++ a surrogate key ++ the natural/business key ++ effective_start and effective_end timestamps (or end_date) ++ a current_flag +On change, you “close” the previous record and insert a new version. You must define how to detect changes and handle late arriving updates. + +[Table of Contents](#Data-Modeling) + +## What is a factless fact table? +A factless fact table records events that do not have numeric measures, for example “student attended class” or “user is eligible for campaign”. The fact itself can be counted, and the table is useful for coverage, eligibility, and relationship tracking. + +[Table of Contents](#Data-Modeling) + +## What is a bridge table and when do you need it? +A bridge table resolves many-to-many relationships, for example products belonging to multiple categories or users having multiple roles. It can also store weights for allocation (for example revenue split). Without a bridge table, joins can explode row counts and break aggregations. + +[Table of Contents](#Data-Modeling) + +## What are degenerate dimensions? +Degenerate dimensions are identifiers stored in the fact table without a separate dimension table, such as order_id or invoice_number. They are useful for drill-through and filtering while avoiding unnecessary dimension tables. + +[Table of Contents](#Data-Modeling) + +## How do you prevent double counting in analytical models? +You prevent double counting by enforcing correct grain, using proper join keys (many-to-one), and avoiding many-to-many joins without a bridge table. You also define metric rules (sum vs distinct count vs last value) and test them with reconciliation queries. + +[Table of Contents](#Data-Modeling) + +## How do you validate a data model after changes? +Typical validation includes: ++ row count and key uniqueness checks at each layer ++ reconciliation of totals against source systems ++ sampling and drill-through checks on known entities ++ regression tests for key business metrics +This catches subtle issues like join duplication or missing late data. + +[Table of Contents](#Data-Modeling) + diff --git a/content/data-quality.md b/content/data-quality.md new file mode 100644 index 0000000..16b3b30 --- /dev/null +++ b/content/data-quality.md @@ -0,0 +1,117 @@ +## [Main title](../README.md) +### [Interview questions](full.md) +# +# Data Quality ++ [What does data quality mean in data engineering?](#What-does-data-quality-mean-in-data-engineering) ++ [What are common dimensions of data quality?](#What-are-common-dimensions-of-data-quality) ++ [What is data freshness and how do you measure it?](#What-is-data-freshness-and-how-do-you-measure-it) ++ [What is the difference between data validation and data reconciliation?](#What-is-the-difference-between-data-validation-and-data-reconciliation) ++ [What are row-level checks and aggregate checks?](#What-are-row-level-checks-and-aggregate-checks) ++ [How do you design quality checks for incremental pipelines?](#How-do-you-design-quality-checks-for-incremental-pipelines) ++ [How do you handle late arriving data from a quality perspective?](#How-do-you-handle-late-arriving-data-from-a-quality-perspective) ++ [How do you detect schema drift and breaking changes?](#How-do-you-detect-schema-drift-and-breaking-changes) ++ [What are data contracts?](#What-are-data-contracts) ++ [How do you avoid noisy alerts (false positives) in data quality monitoring?](#How-do-you-avoid-noisy-alerts-(false-positives)-in-data-quality-monitoring) ++ [What is anomaly detection for metrics and when is it useful?](#What-is-anomaly-detection-for-metrics-and-when-is-it-useful) ++ [How do you quarantine bad data without blocking the entire pipeline?](#How-do-you-quarantine-bad-data-without-blocking-the-entire-pipeline) ++ [How do you test data transformations?](#How-do-you-test-data-transformations) ++ [What is the minimum set of checks you would add to every table?](#What-is-the-minimum-set-of-checks-you-would-add-to-every-table) ++ [What are common data quality failure modes?](#What-are-common-data-quality-failure-modes) + +## What does data quality mean in data engineering? +Data quality means the data is fit for its intended use. In practice, it means data is correct, complete, consistent, timely, and well-defined so downstream consumers can trust metrics. Data quality is both a technical problem (pipelines) and a product problem (definitions and contracts). + +[Table of Contents](#Data-Quality) + +## What are common dimensions of data quality? +Common dimensions include: ++ completeness (missing rows/fields) ++ validity (values in allowed ranges/domains) ++ accuracy (matches real-world or source truth) ++ consistency (no contradictions across tables) ++ timeliness/freshness (arrives within SLA) ++ uniqueness (no duplicates where keys should be unique) + +[Table of Contents](#Data-Quality) + +## What is data freshness and how do you measure it? +Freshness is how recently data was updated relative to expected cadence. You can measure it by comparing the maximum event/ingestion timestamp in a table to current time, and alert when it exceeds an SLA threshold. Freshness should be defined per dataset because “daily” and “real-time” pipelines differ. + +[Table of Contents](#Data-Quality) + +## What is the difference between data validation and data reconciliation? +Validation checks whether a dataset satisfies rules (types, constraints, ranges). Reconciliation compares datasets to ensure they match expected totals or invariants (for example source row count vs target row count, or sum(amount) in source vs warehouse). Reconciliation is important when multiple systems can drift. + +[Table of Contents](#Data-Quality) + +## What are row-level checks and aggregate checks? +Row-level checks validate each record (for example `not_null`, regex, range constraints). Aggregate checks validate the dataset as a whole (row counts, distinct counts, distribution checks, totals). Aggregate checks often catch issues that row-level rules miss, like missing partitions. + +[Table of Contents](#Data-Quality) + +## How do you design quality checks for incremental pipelines? +Checks should be scoped to the incremental window and also protect the full table invariants. For example, validate uniqueness on the keys in the new batch, validate that the batch size is within expected bounds, and validate that total counts change consistently. You also need idempotency checks to detect duplicates on retries. + +[Table of Contents](#Data-Quality) + +## How do you handle late arriving data from a quality perspective? +You define allowed lateness (for example 7 days), track completeness by event time, and implement backfills to “close” historical partitions. Quality checks should consider both ingestion time and event time so you can detect gaps in past periods. + +[Table of Contents](#Data-Quality) + +## How do you detect schema drift and breaking changes? +You compare incoming schemas to an expected contract (column names, types, nullability) and alert on changes. Additive changes can be allowed with policy, but breaking changes (rename/type change) should fail fast. Many teams use CI checks and schema registries to enforce compatibility. + +[Table of Contents](#Data-Quality) + +## What are data contracts? +Data contracts are explicit agreements between producers and consumers about schema, semantics, SLAs, and ownership. Contracts reduce surprises by defining what can change and what cannot, and they enable automated validation. A contract can live as code (YAML/JSON) and be enforced in CI/CD. + +[Table of Contents](#Data-Quality) + +## How do you avoid noisy alerts (false positives) in data quality monitoring? +Use dynamic thresholds, seasonality-aware baselines, and multi-signal alerts (freshness + volume + error rate). Also route alerts by severity and dataset criticality, and suppress alerts during known maintenance windows. Good alerting includes a runbook and clear ownership. + +[Table of Contents](#Data-Quality) + +## What is anomaly detection for metrics and when is it useful? +Anomaly detection flags unusual changes in metrics (spikes/drops) that may indicate pipeline issues or business events. It is useful for detecting silent failures where data still arrives but is wrong. It should be combined with context (releases, campaigns) to avoid alert fatigue. + +[Table of Contents](#Data-Quality) + +## How do you quarantine bad data without blocking the entire pipeline? +A common pattern is to split data into “good” and “quarantine” outputs based on validation rules. You can store bad rows with error reasons for debugging while allowing the pipeline to continue for valid data. For critical datasets, you may still block publishing to marts while keeping raw ingestion running. + +[Table of Contents](#Data-Quality) + +## How do you test data transformations? +You can test transformations by: ++ unit-like tests on small fixtures (golden datasets) ++ property tests (invariants like non-negative totals) ++ reconciliation tests against known sources ++ regression tests on key metrics +In dbt-style workflows, tests often run in the warehouse as SQL assertions. + +[Table of Contents](#Data-Quality) + +## What is the minimum set of checks you would add to every table? +Minimum checks often include: ++ freshness (max timestamp within SLA) ++ row count / volume sanity check ++ primary key uniqueness (if applicable) ++ not_null checks on critical columns ++ referential integrity for key joins (where applicable) + +[Table of Contents](#Data-Quality) + +## What are common data quality failure modes? +Common failures include: ++ missing partitions or partial loads ++ duplicates due to retries/reprocessing ++ schema drift and type coercion issues ++ timezone and date boundary bugs ++ silent truncation or rounding changes ++ incorrect joins causing duplication + +[Table of Contents](#Data-Quality) + diff --git a/content/dbt.md b/content/dbt.md new file mode 100644 index 0000000..14a8e54 --- /dev/null +++ b/content/dbt.md @@ -0,0 +1,105 @@ +## [Main title](../README.md) +### [Interview questions](full.md) +# +# dbt ++ [What is dbt?](#What-is-dbt) ++ [How is dbt different from Airflow?](#How-is-dbt-different-from-Airflow) ++ [What is a dbt model?](#What-is-a-dbt-model) ++ [What does ref() do and why is it important?](#What-does-ref()-do-and-why-is-it-important) ++ [What are typical layers in a dbt project (staging/intermediate/marts)?](#What-are-typical-layers-in-a-dbt-project-(staging/intermediate/marts)) ++ [What is a materialization in dbt?](#What-is-a-materialization-in-dbt) ++ [When would you use view vs table materializations?](#When-would-you-use-view-vs-table-materializations) ++ [What is an incremental model and what problems does it solve?](#What-is-an-incremental-model-and-what-problems-does-it-solve) ++ [How do you design a reliable unique_key for incremental models?](#How-do-you-design-a-reliable-unique_key-for-incremental-models) ++ [What are dbt tests and what types exist?](#What-are-dbt-tests-and-what-types-exist) ++ [What are sources in dbt and how do you test them?](#What-are-sources-in-dbt-and-how-do-you-test-them) ++ [What are snapshots in dbt and when would you use them?](#What-are-snapshots-in-dbt-and-when-would-you-use-them) ++ [What are macros in dbt and when should you use them?](#What-are-macros-in-dbt-and-when-should-you-use-them) ++ [What is dbt state selection (slim CI) and why is it useful?](#What-is-dbt-state-selection-(slim-CI)-and-why-is-it-useful) ++ [How do you approach CI/CD for dbt projects?](#How-do-you-approach-CI/CD-for-dbt-projects) + +## What is dbt? +dbt (data build tool) is a transformation framework where you write transformations as SQL (and Jinja-templated SQL), and dbt builds a DAG of models and runs them in the right order. It is commonly used to transform raw data into cleaned, documented, tested analytics tables in a warehouse/lakehouse. + +[Table of Contents](#dbt) + +## How is dbt different from Airflow? +Airflow is an orchestration system that schedules and coordinates tasks of any kind. dbt focuses specifically on SQL-based transformations and their dependency graph, plus testing and documentation. In practice, Airflow often runs dbt as one step of a larger pipeline (ingestion → transform → publish). + +[Table of Contents](#dbt) + +## What is a dbt model? +A dbt model is a select query stored as a `.sql` file in the `models/` directory. dbt materializes the model into a relation in your warehouse (view/table/incremental) and manages dependencies between models. + +[Table of Contents](#dbt) + +## What does ref() do and why is it important? +`ref('model_name')` declares a dependency on another dbt model. dbt uses it to build the DAG, order execution, and resolve the correct schema/database per environment. It also enables features like automated lineage in dbt docs. + +[Table of Contents](#dbt) + +## What are typical layers in a dbt project (staging/intermediate/marts)? +A common pattern is: ++ staging: light cleaning and renaming, close to sources ++ intermediate: reusable transformations and joins ++ marts: business-facing fact/dimension tables and metrics-ready datasets +This makes models easier to test, reuse, and maintain. + +[Table of Contents](#dbt) + +## What is a materialization in dbt? +A materialization defines how dbt builds a model in the warehouse (for example as a view, table, or incremental table). It controls the build strategy and how changes are applied over time. + +[Table of Contents](#dbt) + +## When would you use view vs table materializations? +Views are useful for fast iteration and when compute cost per query is acceptable. Tables are useful when queries are expensive or many downstream queries reuse the same result. The trade-off is storage cost and the need to keep tables updated. + +[Table of Contents](#dbt) + +## What is an incremental model and what problems does it solve? +An incremental model only processes new or changed data instead of rebuilding the entire table each run. It reduces runtime and cost for large datasets. It requires correct keys and logic to avoid duplicates and missed updates. + +[Table of Contents](#dbt) + +## How do you design a reliable unique_key for incremental models? +The `unique_key` should uniquely identify a record in the target table, usually a stable business key or a surrogate key derived from stable columns. If the source can update records, you also need a deterministic “latest record” rule (for example using an `updated_at` column) so merges are correct. + +[Table of Contents](#dbt) + +## What are dbt tests and what types exist? +dbt supports: ++ generic tests (built-in or custom): `unique`, `not_null`, `accepted_values`, `relationships` ++ singular tests: custom SQL queries that should return zero failing rows +Tests run as queries in the warehouse and fail the run if assertions are violated. + +[Table of Contents](#dbt) + +## What are sources in dbt and how do you test them? +Sources declare upstream tables and their metadata in `sources.yml`. You can test sources with freshness checks and column-level tests (like `not_null`) to detect broken ingestion or schema drift early. + +[Table of Contents](#dbt) + +## What are snapshots in dbt and when would you use them? +Snapshots capture slowly changing dimension history by tracking changes in source records over time. They are useful when you need point-in-time analysis or to implement SCD2-style history from mutable source tables. + +[Table of Contents](#dbt) + +## What are macros in dbt and when should you use them? +Macros are Jinja functions that generate SQL. Use them to avoid repetition (standardized column selection, reusable filters, dynamic SQL) and to keep business logic consistent across models. Overusing macros for complex logic can hurt readability and debugging. + +[Table of Contents](#dbt) + +## What is dbt state selection (slim CI) and why is it useful? +State selection runs only models that changed (and their dependents) compared to a previous artifact state. In CI, this reduces runtime by avoiding full rebuilds, while still validating that changes compile, run, and pass tests for affected parts of the DAG. + +[Table of Contents](#dbt) + +## How do you approach CI/CD for dbt projects? +A common approach is: ++ in CI: `dbt deps`, `dbt compile`, run selected models/tests (state-based or tags) ++ in CD: deploy artifacts and run scheduled jobs in production +You usually also enforce code review, documentation, and data tests for critical marts. + +[Table of Contents](#dbt) + diff --git a/content/full.md b/content/full.md index 75af3e8..23365ed 100644 --- a/content/full.md +++ b/content/full.md @@ -26,6 +26,17 @@ + [Greenplum](#Greenplum) + [Redshift](#Redshift) + [Data Structures](#Data-Structures) ++ [dbt](#dbt) ++ [Apache Iceberg](#Apache-Iceberg) ++ [Change Data Capture (CDC)](#Change-Data-Capture-(CDC)) ++ [Data Modeling](#Data-Modeling) ++ [Data Quality](#Data-Quality) ++ [Data Observability](#Data-Observability) ++ [Data Governance](#Data-Governance) ++ [Apache Hudi](#Apache-Hudi) ++ [Cost Optimization](#Cost-Optimization) ++ [Python for Data Engineering](#Python-for-Data-Engineering) ++ [Data System Design](#Data-System-Design) ## Apache Hadoop + [What are the main components of a Hadoop Application?](hadoop.md#What-are-the-main-components-of-a-Hadoop-Application) @@ -235,6 +246,196 @@ [Table of Contents](#Interview-questions-for-Data-Engineer) +## dbt ++ [What is dbt?](dbt.md#What-is-dbt) ++ [How is dbt different from Airflow?](dbt.md#How-is-dbt-different-from-Airflow) ++ [What is a dbt model?](dbt.md#What-is-a-dbt-model) ++ [What does ref() do and why is it important?](dbt.md#What-does-ref()-do-and-why-is-it-important) ++ [What are typical layers in a dbt project (staging/intermediate/marts)?](dbt.md#What-are-typical-layers-in-a-dbt-project-(staging/intermediate/marts)) ++ [What is a materialization in dbt?](dbt.md#What-is-a-materialization-in-dbt) ++ [When would you use view vs table materializations?](dbt.md#When-would-you-use-view-vs-table-materializations) ++ [What is an incremental model and what problems does it solve?](dbt.md#What-is-an-incremental-model-and-what-problems-does-it-solve) ++ [How do you design a reliable unique_key for incremental models?](dbt.md#How-do-you-design-a-reliable-unique_key-for-incremental-models) ++ [What are dbt tests and what types exist?](dbt.md#What-are-dbt-tests-and-what-types-exist) ++ [What are sources in dbt and how do you test them?](dbt.md#What-are-sources-in-dbt-and-how-do-you-test-them) ++ [What are snapshots in dbt and when would you use them?](dbt.md#What-are-snapshots-in-dbt-and-when-would-you-use-them) ++ [What are macros in dbt and when should you use them?](dbt.md#What-are-macros-in-dbt-and-when-should-you-use-them) ++ [What is dbt state selection (slim CI) and why is it useful?](dbt.md#What-is-dbt-state-selection-(slim-CI)-and-why-is-it-useful) ++ [How do you approach CI/CD for dbt projects?](dbt.md#How-do-you-approach-CI/CD-for-dbt-projects) + +[Table of Contents](#Interview-questions-for-Data-Engineer) + +## Apache Iceberg ++ [What is Apache Iceberg?](iceberg.md#What-is-Apache-Iceberg) ++ [What problems does Iceberg solve compared to plain Parquet datasets?](iceberg.md#What-problems-does-Iceberg-solve-compared-to-plain-Parquet-datasets) ++ [What is a snapshot in Iceberg?](iceberg.md#What-is-a-snapshot-in-Iceberg) ++ [What is time travel and why is it useful?](iceberg.md#What-is-time-travel-and-why-is-it-useful) ++ [How does Iceberg handle concurrent writes?](iceberg.md#How-does-Iceberg-handle-concurrent-writes) ++ [What is hidden partitioning in Iceberg?](iceberg.md#What-is-hidden-partitioning-in-Iceberg) ++ [What is partition evolution and why is it important?](iceberg.md#What-is-partition-evolution-and-why-is-it-important) ++ [How does schema evolution work in Iceberg?](iceberg.md#How-does-schema-evolution-work-in-Iceberg) ++ [What are equality deletes and positional deletes?](iceberg.md#What-are-equality-deletes-and-positional-deletes) ++ [How do upserts/merges work with Iceberg?](iceberg.md#How-do-upserts/merges-work-with-Iceberg) ++ [Why does the small files problem happen and how do you mitigate it?](iceberg.md#Why-does-the-small-files-problem-happen-and-how-do-you-mitigate-it) ++ [What maintenance operations are common for Iceberg tables?](iceberg.md#What-maintenance-operations-are-common-for-Iceberg-tables) ++ [What is an Iceberg catalog and what options exist?](iceberg.md#What-is-an-Iceberg-catalog-and-what-options-exist) ++ [How would you migrate an existing dataset to Iceberg?](iceberg.md#How-would-you-migrate-an-existing-dataset-to-Iceberg) ++ [Iceberg vs Delta vs Hudi: when would you choose Iceberg?](iceberg.md#Iceberg-vs-Delta-vs-Hudi:-when-would-you-choose-Iceberg) + +[Table of Contents](#Interview-questions-for-Data-Engineer) + +## Change Data Capture (CDC) ++ [What is CDC (Change Data Capture)?](cdc.md#What-is-CDC-(Change-Data-Capture)) ++ [When would you choose CDC over batch ingestion?](cdc.md#When-would-you-choose-CDC-over-batch-ingestion) ++ [What are common CDC sources (WAL/binlog/triggers) and trade-offs?](cdc.md#What-are-common-CDC-sources-(WAL/binlog/triggers)-and-trade-offs) ++ [What is the difference between snapshot and incremental CDC?](cdc.md#What-is-the-difference-between-snapshot-and-incremental-CDC) ++ [What delivery semantics exist (at-most-once/at-least-once/exactly-once)?](cdc.md#What-delivery-semantics-exist-(at-most-once/at-least-once/exactly-once)) ++ [How do you make CDC ingestion idempotent?](cdc.md#How-do-you-make-CDC-ingestion-idempotent) ++ [How do you handle deletes in CDC pipelines?](cdc.md#How-do-you-handle-deletes-in-CDC-pipelines) ++ [How do you handle updates when the source does not provide full row images?](cdc.md#How-do-you-handle-updates-when-the-source-does-not-provide-full-row-images) ++ [What is ordering and why is it hard in CDC?](cdc.md#What-is-ordering-and-why-is-it-hard-in-CDC) ++ [What is a watermark/offset and where should it be stored?](cdc.md#What-is-a-watermark/offset-and-where-should-it-be-stored) ++ [How do you handle schema evolution with CDC?](cdc.md#How-do-you-handle-schema-evolution-with-CDC) ++ [What is the outbox pattern and why is it useful?](cdc.md#What-is-the-outbox-pattern-and-why-is-it-useful) ++ [How do you monitor CDC lag and where can lag come from?](cdc.md#How-do-you-monitor-CDC-lag-and-where-can-lag-come-from) ++ [How do you design a safe backfill/reprocessing strategy for CDC?](cdc.md#How-do-you-design-a-safe-backfill/reprocessing-strategy-for-CDC) ++ [What are the most common failure modes in CDC pipelines?](cdc.md#What-are-the-most-common-failure-modes-in-CDC-pipelines) + +[Table of Contents](#Interview-questions-for-Data-Engineer) + +## Data Modeling ++ [What is data modeling in analytics and why does it matter?](data-modeling.md#What-is-data-modeling-in-analytics-and-why-does-it-matter) ++ [What is grain and why is it the first question to answer?](data-modeling.md#What-is-grain-and-why-is-it-the-first-question-to-answer) ++ [What is the difference between fact and dimension tables?](data-modeling.md#What-is-the-difference-between-fact-and-dimension-tables) ++ [What is a star schema and what are its benefits?](data-modeling.md#What-is-a-star-schema-and-what-are-its-benefits) ++ [Star schema vs snowflake schema: what are the trade-offs?](data-modeling.md#Star-schema-vs-snowflake-schema:-what-are-the-trade-offs) ++ [What are the main types of fact tables?](data-modeling.md#What-are-the-main-types-of-fact-tables) ++ [What is a surrogate key and when should you use it?](data-modeling.md#What-is-a-surrogate-key-and-when-should-you-use-it) ++ [What is a conformed dimension?](data-modeling.md#What-is-a-conformed-dimension) ++ [What is SCD (Slowly Changing Dimension)?](data-modeling.md#What-is-SCD-(Slowly-Changing-Dimension)) ++ [How do you implement SCD Type 2?](data-modeling.md#How-do-you-implement-SCD-Type-2) ++ [What is a factless fact table?](data-modeling.md#What-is-a-factless-fact-table) ++ [What is a bridge table and when do you need it?](data-modeling.md#What-is-a-bridge-table-and-when-do-you-need-it) ++ [What are degenerate dimensions?](data-modeling.md#What-are-degenerate-dimensions) ++ [How do you prevent double counting in analytical models?](data-modeling.md#How-do-you-prevent-double-counting-in-analytical-models) ++ [How do you validate a data model after changes?](data-modeling.md#How-do-you-validate-a-data-model-after-changes) + +[Table of Contents](#Interview-questions-for-Data-Engineer) + +## Data Quality ++ [What does data quality mean in data engineering?](data-quality.md#What-does-data-quality-mean-in-data-engineering) ++ [What are common dimensions of data quality?](data-quality.md#What-are-common-dimensions-of-data-quality) ++ [What is data freshness and how do you measure it?](data-quality.md#What-is-data-freshness-and-how-do-you-measure-it) ++ [What is the difference between data validation and data reconciliation?](data-quality.md#What-is-the-difference-between-data-validation-and-data-reconciliation) ++ [What are row-level checks and aggregate checks?](data-quality.md#What-are-row-level-checks-and-aggregate-checks) ++ [How do you design quality checks for incremental pipelines?](data-quality.md#How-do-you-design-quality-checks-for-incremental-pipelines) ++ [How do you handle late arriving data from a quality perspective?](data-quality.md#How-do-you-handle-late-arriving-data-from-a-quality-perspective) ++ [How do you detect schema drift and breaking changes?](data-quality.md#How-do-you-detect-schema-drift-and-breaking-changes) ++ [What are data contracts?](data-quality.md#What-are-data-contracts) ++ [How do you avoid noisy alerts (false positives) in data quality monitoring?](data-quality.md#How-do-you-avoid-noisy-alerts-(false-positives)-in-data-quality-monitoring) ++ [What is anomaly detection for metrics and when is it useful?](data-quality.md#What-is-anomaly-detection-for-metrics-and-when-is-it-useful) ++ [How do you quarantine bad data without blocking the entire pipeline?](data-quality.md#How-do-you-quarantine-bad-data-without-blocking-the-entire-pipeline) ++ [How do you test data transformations?](data-quality.md#How-do-you-test-data-transformations) ++ [What is the minimum set of checks you would add to every table?](data-quality.md#What-is-the-minimum-set-of-checks-you-would-add-to-every-table) ++ [What are common data quality failure modes?](data-quality.md#What-are-common-data-quality-failure-modes) + +[Table of Contents](#Interview-questions-for-Data-Engineer) + +## Data Observability ++ [What is data observability?](observability.md#What-is-data-observability) ++ [How is data observability different from application observability?](observability.md#How-is-data-observability-different-from-application-observability) ++ [What are the key signals you monitor for data pipelines?](observability.md#What-are-the-key-signals-you-monitor-for-data-pipelines) ++ [How do you define and measure an end-to-end SLA for data?](observability.md#How-do-you-define-and-measure-an-end-to-end-SLA-for-data) ++ [What should you log for each pipeline run?](observability.md#What-should-you-log-for-each-pipeline-run) ++ [How do you detect silent failures?](observability.md#How-do-you-detect-silent-failures) ++ [What is lineage and how does it help during incidents?](observability.md#What-is-lineage-and-how-does-it-help-during-incidents) ++ [How do you monitor and debug a broken metric in BI?](observability.md#How-do-you-monitor-and-debug-a-broken-metric-in-BI) ++ [How do you design alerts to avoid alert fatigue?](observability.md#How-do-you-design-alerts-to-avoid-alert-fatigue) ++ [How do you approach backfills safely?](observability.md#How-do-you-approach-backfills-safely) ++ [What is a runbook and what should it contain?](observability.md#What-is-a-runbook-and-what-should-it-contain) ++ [What are common incident patterns in data platforms?](observability.md#What-are-common-incident-patterns-in-data-platforms) + +[Table of Contents](#Interview-questions-for-Data-Engineer) + +## Data Governance ++ [What is data governance and why does it matter?](data-governance.md#What-is-data-governance-and-why-does-it-matter) ++ [What is the difference between governance and security?](data-governance.md#What-is-the-difference-between-governance-and-security) ++ [What are common roles in data governance (owner/steward/custodian)?](data-governance.md#What-are-common-roles-in-data-governance-(owner/steward/custodian)) ++ [What is a data catalog and what should it contain?](data-governance.md#What-is-a-data-catalog-and-what-should-it-contain) ++ [What is business glossary vs technical metadata?](data-governance.md#What-is-business-glossary-vs-technical-metadata) ++ [RBAC vs ABAC: what is the difference?](data-governance.md#RBAC-vs-ABAC:-what-is-the-difference) ++ [What is row-level and column-level security?](data-governance.md#What-is-row-level-and-column-level-security) ++ [How do you handle PII/PHI data in analytics platforms?](data-governance.md#How-do-you-handle-PII/PHI-data-in-analytics-platforms) ++ [What is data masking and tokenization?](data-governance.md#What-is-data-masking-and-tokenization) ++ [How do you implement auditability for data access?](data-governance.md#How-do-you-implement-auditability-for-data-access) ++ [How do you handle GDPR “right to be forgotten” in a lakehouse/warehouse?](data-governance.md#How-do-you-handle-GDPR-“right-to-be-forgotten”-in-a-lakehouse/warehouse) ++ [What are data retention policies and how do you enforce them?](data-governance.md#What-are-data-retention-policies-and-how-do-you-enforce-them) + +[Table of Contents](#Interview-questions-for-Data-Engineer) + +## Apache Hudi ++ [What is Apache Hudi?](hudi.md#What-is-Apache-Hudi) ++ [What problems does Hudi solve in a data lake?](hudi.md#What-problems-does-Hudi-solve-in-a-data-lake) ++ [What is the difference between Copy-on-Write (COW) and Merge-on-Read (MOR)?](hudi.md#What-is-the-difference-between-Copy-on-Write-(COW)-and-Merge-on-Read-(MOR)) ++ [How do you choose between COW and MOR?](hudi.md#How-do-you-choose-between-COW-and-MOR) ++ [What is a record key, partition path, and precombine field?](hudi.md#What-is-a-record-key-partition-path-and-precombine-field) ++ [How does Hudi support upserts?](hudi.md#How-does-Hudi-support-upserts) ++ [What is compaction in Hudi?](hudi.md#What-is-compaction-in-Hudi) ++ [What is clustering in Hudi and when do you need it?](hudi.md#What-is-clustering-in-Hudi-and-when-do-you-need-it) ++ [Why does the small files problem happen and how do you mitigate it in Hudi?](hudi.md#Why-does-the-small-files-problem-happen-and-how-do-you-mitigate-it-in-Hudi) ++ [How do you handle CDC with Hudi?](hudi.md#How-do-you-handle-CDC-with-Hudi) ++ [What are common operational metrics for Hudi tables?](hudi.md#What-are-common-operational-metrics-for-Hudi-tables) ++ [When would you choose Hudi vs Iceberg vs Delta?](hudi.md#When-would-you-choose-Hudi-vs-Iceberg-vs-Delta) + +[Table of Contents](#Interview-questions-for-Data-Engineer) + +## Cost Optimization ++ [Why is cost optimization a core data engineering skill?](cost-optimization.md#Why-is-cost-optimization-a-core-data-engineering-skill) ++ [What are the main cost drivers in data platforms?](cost-optimization.md#What-are-the-main-cost-drivers-in-data-platforms) ++ [How does partitioning affect cost and performance?](cost-optimization.md#How-does-partitioning-affect-cost-and-performance) ++ [What is the small files problem and why does it increase cost?](cost-optimization.md#What-is-the-small-files-problem-and-why-does-it-increase-cost) ++ [How do you choose a target file size for Parquet tables?](cost-optimization.md#How-do-you-choose-a-target-file-size-for-Parquet-tables) ++ [How do you detect and fix data skew in distributed processing?](cost-optimization.md#How-do-you-detect-and-fix-data-skew-in-distributed-processing) ++ [What is shuffle and how do you reduce it in Spark?](cost-optimization.md#What-is-shuffle-and-how-do-you-reduce-it-in-Spark) ++ [When should you pre-aggregate or materialize tables?](cost-optimization.md#When-should-you-pre-aggregate-or-materialize-tables) ++ [How do you prevent runaway queries and protect shared clusters?](cost-optimization.md#How-do-you-prevent-runaway-queries-and-protect-shared-clusters) ++ [How do you optimize joins in large-scale analytics?](cost-optimization.md#How-do-you-optimize-joins-in-large-scale-analytics) ++ [How do you plan and estimate the cost of a backfill?](cost-optimization.md#How-do-you-plan-and-estimate-the-cost-of-a-backfill) ++ [What metrics would you track for FinOps in data engineering?](cost-optimization.md#What-metrics-would-you-track-for-FinOps-in-data-engineering) + +[Table of Contents](#Interview-questions-for-Data-Engineer) + +## Python for Data Engineering ++ [Why is Python widely used in data engineering?](python.md#Why-is-Python-widely-used-in-data-engineering) ++ [How do iterators and generators help with large data processing?](python.md#How-do-iterators-and-generators-help-with-large-data-processing) ++ [What is the difference between threads, multiprocessing, and async IO in Python?](python.md#What-is-the-difference-between-threads-multiprocessing-and-async-IO-in-Python) ++ [What is the GIL and why does it matter?](python.md#What-is-the-GIL-and-why-does-it-matter) ++ [How do you read and write Parquet efficiently in Python?](python.md#How-do-you-read-and-write-Parquet-efficiently-in-Python) ++ [How do you process large CSV files without running out of memory?](python.md#How-do-you-process-large-CSV-files-without-running-out-of-memory) ++ [How do you implement retries with exponential backoff?](python.md#How-do-you-implement-retries-with-exponential-backoff) ++ [What logging practices are important for data pipelines?](python.md#What-logging-practices-are-important-for-data-pipelines) ++ [How do you structure a Python project for data pipelines?](python.md#How-do-you-structure-a-Python-project-for-data-pipelines) ++ [How do you manage dependencies and reproducible environments?](python.md#How-do-you-manage-dependencies-and-reproducible-environments) ++ [How do you test data transformations in Python?](python.md#How-do-you-test-data-transformations-in-Python) ++ [How do you profile and optimize slow Python code?](python.md#How-do-you-profile-and-optimize-slow-Python-code) + +[Table of Contents](#Interview-questions-for-Data-Engineer) + +## Data System Design ++ [How would you design an end-to-end batch analytics pipeline?](system-design.md#How-would-you-design-an-end-to-end-batch-analytics-pipeline) ++ [How would you design a near-real-time ingestion pipeline?](system-design.md#How-would-you-design-a-near-real-time-ingestion-pipeline) ++ [How do you ensure idempotency in data pipelines?](system-design.md#How-do-you-ensure-idempotency-in-data-pipelines) ++ [How do you handle late arriving events and backfills?](system-design.md#How-do-you-handle-late-arriving-events-and-backfills) ++ [How do you choose between batch and streaming?](system-design.md#How-do-you-choose-between-batch-and-streaming) ++ [How do you model raw/silver/gold layers (bronze/silver/gold)?](system-design.md#How-do-you-model-raw/silver/gold-layers-(bronze/silver/gold)) ++ [How do you design a data platform for multiple teams (multi-tenancy)?](system-design.md#How-do-you-design-a-data-platform-for-multiple-teams-(multi-tenancy)) ++ [How do you design for schema evolution?](system-design.md#How-do-you-design-for-schema-evolution) ++ [What are the main reliability patterns for pipelines?](system-design.md#What-are-the-main-reliability-patterns-for-pipelines) ++ [How do you design observability for a data platform?](system-design.md#How-do-you-design-observability-for-a-data-platform) ++ [How do you manage cost while meeting SLAs?](system-design.md#How-do-you-manage-cost-while-meeting-SLAs) + +[Table of Contents](#Interview-questions-for-Data-Engineer) + ## Apache Flume + [What is Flume?](flume.md#What-is-Flume) + [What is Apache Flume?](flume.md#What-is-Apache-Flume) @@ -1746,7 +1947,7 @@ + [Write query to find employees with duplicate email.](sql.md#Write-query-to-find-employees-with-duplicate-email) + [Write a query to find all employee whose name contains the word "rich", regardless of case.](sql.md#Write-a-query-to-find-all-employee-whose-name-contains-the-word-"rich",-regardless-of-case) + [Is it safe to use rowid to locate a record in oracle sql queries?](sql.md#Is-it-safe-to-use-rowid-to-locate-a-record-in-oracle-sql-queries) -+ [What is a pseudoпїЅolumn?](sql.md#What-is-a-pseudoпїЅolumn) ++ [What is a pseudo-column?](sql.md#What-is-a-pseudo-column) + [What are the reasons for denormalizing the data?](sql.md#What-are-the-reasons-for-denormalizing-the-data) + [What is the feature in sql for writing if and else statements?](sql.md#What-is-the-feature-in-sql-for-writing-if-and-else-statements) + [What is the difference between delete and truncate in sql?](sql.md#What-is-the-difference-between-delete-and-truncate-in-sql) diff --git a/content/hudi.md b/content/hudi.md new file mode 100644 index 0000000..657bbf2 --- /dev/null +++ b/content/hudi.md @@ -0,0 +1,77 @@ +## [Main title](../README.md) +### [Interview questions](full.md) +# +# Apache Hudi ++ [What is Apache Hudi?](#What-is-Apache-Hudi) ++ [What problems does Hudi solve in a data lake?](#What-problems-does-Hudi-solve-in-a-data-lake) ++ [What is the difference between Copy-on-Write (COW) and Merge-on-Read (MOR)?](#What-is-the-difference-between-Copy-on-Write-(COW)-and-Merge-on-Read-(MOR)) ++ [How do you choose between COW and MOR?](#How-do-you-choose-between-COW-and-MOR) ++ [What is a record key, partition path, and precombine field?](#What-is-a-record-key-partition-path-and-precombine-field) ++ [How does Hudi support upserts?](#How-does-Hudi-support-upserts) ++ [What is compaction in Hudi?](#What-is-compaction-in-Hudi) ++ [What is clustering in Hudi and when do you need it?](#What-is-clustering-in-Hudi-and-when-do-you-need-it) ++ [Why does the small files problem happen and how do you mitigate it in Hudi?](#Why-does-the-small-files-problem-happen-and-how-do-you-mitigate-it-in-Hudi) ++ [How do you handle CDC with Hudi?](#How-do-you-handle-CDC-with-Hudi) ++ [What are common operational metrics for Hudi tables?](#What-are-common-operational-metrics-for-Hudi-tables) ++ [When would you choose Hudi vs Iceberg vs Delta?](#When-would-you-choose-Hudi-vs-Iceberg-vs-Delta) + +## What is Apache Hudi? +Apache Hudi is an open-source data lakehouse framework that brings database-like capabilities (upserts, deletes, incremental pulls) to data lakes on object storage. It manages table metadata and timelines so batch and streaming engines can write and read consistently. + +[Table of Contents](#Apache-Hudi) + +## What problems does Hudi solve in a data lake? +Hudi enables efficient updates and deletes without rewriting entire datasets, supports incremental processing (read only changes), and helps manage small files and write amplification. It is commonly used for near-real-time ingestion and CDC-style workloads where tables must be kept up-to-date. + +[Table of Contents](#Apache-Hudi) + +## What is the difference between Copy-on-Write (COW) and Merge-on-Read (MOR)? +COW stores data in columnar files and rewrites them on updates, optimizing read performance at the cost of write amplification. MOR stores base files plus delta log files and merges them at read time (or during compaction), optimizing write performance at the cost of more complex reads. + +[Table of Contents](#Apache-Hudi) + +## How do you choose between COW and MOR? +Choose COW when read latency and query simplicity are most important and updates are moderate. Choose MOR when write throughput and near-real-time updates are critical, and you can manage compaction and read-time merges. The choice also depends on engine support and operational constraints. + +[Table of Contents](#Apache-Hudi) + +## What is a record key, partition path, and precombine field? +Record key identifies a row uniquely (like a primary key). Partition path controls how data is grouped for storage and pruning. Precombine field is used to resolve multiple records for the same key in a batch (for example choose the latest by `updated_at`) to ensure deterministic upserts. + +[Table of Contents](#Apache-Hudi) + +## How does Hudi support upserts? +Hudi indexes records by key and can locate existing records to update them. On upsert, it writes new versions and updates the timeline/metadata so readers see a consistent view. Correctness depends on stable keys and proper handling of late and duplicate events. + +[Table of Contents](#Apache-Hudi) + +## What is compaction in Hudi? +Compaction merges delta log files into base columnar files (mostly relevant for MOR tables). It improves read performance and reduces the overhead of reading many log files. Compaction must be scheduled and monitored to avoid backlog. + +[Table of Contents](#Apache-Hudi) + +## What is clustering in Hudi and when do you need it? +Clustering reorganizes data files to improve layout (file sizes, sorting, locality) and query performance. You use clustering to combat small files, improve pruning, or optimize for common access patterns (for example sorting by event time). + +[Table of Contents](#Apache-Hudi) + +## Why does the small files problem happen and how do you mitigate it in Hudi? +It happens when writers commit frequently with small batches or when partitions are highly granular. Mitigations include tuning write parallelism and file sizing, using clustering/compaction, and adjusting partitioning strategy to reduce tiny partitions. + +[Table of Contents](#Apache-Hudi) + +## How do you handle CDC with Hudi? +Hudi can ingest CDC events and apply them as upserts/deletes to maintain a current-state table. You typically standardize event ordering and deduplication (record key + precombine field) and monitor lag/compaction. Some setups also keep an append-only log for auditing alongside the current-state table. + +[Table of Contents](#Apache-Hudi) + +## What are common operational metrics for Hudi tables? +Common metrics include commit/compaction backlog, file counts per partition, average file size, write latency, failed commits, and query scan metrics. Operational alerts often focus on increasing small files and compaction lag. + +[Table of Contents](#Apache-Hudi) + +## When would you choose Hudi vs Iceberg vs Delta? +Hudi is often chosen for CDC-heavy and upsert/delete workloads with near-real-time ingestion requirements. Iceberg is often chosen for engine-agnostic open tables with strong metadata and evolution features. Delta is often chosen when you are aligned with the Delta ecosystem and its tooling. The best choice depends on engines, governance, and operational capabilities. + +[Table of Contents](#Apache-Hudi) + diff --git a/content/iceberg.md b/content/iceberg.md new file mode 100644 index 0000000..20a868e --- /dev/null +++ b/content/iceberg.md @@ -0,0 +1,106 @@ +## [Main title](../README.md) +### [Interview questions](full.md) +# +# Apache Iceberg ++ [What is Apache Iceberg?](#What-is-Apache-Iceberg) ++ [What problems does Iceberg solve compared to plain Parquet datasets?](#What-problems-does-Iceberg-solve-compared-to-plain-Parquet-datasets) ++ [What is a snapshot in Iceberg?](#What-is-a-snapshot-in-Iceberg) ++ [What is time travel and why is it useful?](#What-is-time-travel-and-why-is-it-useful) ++ [How does Iceberg handle concurrent writes?](#How-does-Iceberg-handle-concurrent-writes) ++ [What is hidden partitioning in Iceberg?](#What-is-hidden-partitioning-in-Iceberg) ++ [What is partition evolution and why is it important?](#What-is-partition-evolution-and-why-is-it-important) ++ [How does schema evolution work in Iceberg?](#How-does-schema-evolution-work-in-Iceberg) ++ [What are equality deletes and positional deletes?](#What-are-equality-deletes-and-positional-deletes) ++ [How do upserts/merges work with Iceberg?](#How-do-upserts/merges-work-with-Iceberg) ++ [Why does the small files problem happen and how do you mitigate it?](#Why-does-the-small-files-problem-happen-and-how-do-you-mitigate-it) ++ [What maintenance operations are common for Iceberg tables?](#What-maintenance-operations-are-common-for-Iceberg-tables) ++ [What is an Iceberg catalog and what options exist?](#What-is-an-Iceberg-catalog-and-what-options-exist) ++ [How would you migrate an existing dataset to Iceberg?](#How-would-you-migrate-an-existing-dataset-to-Iceberg) ++ [Iceberg vs Delta vs Hudi: when would you choose Iceberg?](#Iceberg-vs-Delta-vs-Hudi:-when-would-you-choose-Iceberg) + +## What is Apache Iceberg? +Apache Iceberg is an open table format for large analytics datasets. It defines how table metadata, snapshots, and data files are managed so multiple engines can read and write the same tables safely. Iceberg tables are usually stored on object storage (S3/GCS/ADLS) with files like Parquet, but the table semantics (transactions, schema/partition evolution) come from Iceberg metadata. + +[Table of Contents](#Apache-Iceberg) + +## What problems does Iceberg solve compared to plain Parquet datasets? +A plain Parquet “dataset in folders” lacks transactional guarantees and consistent metadata. Iceberg adds ACID-like commits, consistent snapshots, reliable schema/partition evolution, and query planning features like partition pruning based on metadata. It also supports deletes and updates in a table-like way rather than “rewrite everything”. + +[Table of Contents](#Apache-Iceberg) + +## What is a snapshot in Iceberg? +A snapshot is a point-in-time view of the table. Each commit creates a new snapshot that references a set of data and delete files through metadata (manifest lists/manifests). Readers can query a specific snapshot to get consistent results even while writers continue to append or modify the table. + +[Table of Contents](#Apache-Iceberg) + +## What is time travel and why is it useful? +Time travel means querying the table “as of” a previous snapshot (or timestamp). It is useful for debugging regressions, auditing changes, reproducing past reports, and safe backfills (compute against a stable snapshot and then commit a new one). + +[Table of Contents](#Apache-Iceberg) + +## How does Iceberg handle concurrent writes? +Iceberg typically uses optimistic concurrency control. Writers create new metadata based on a known current snapshot and then attempt to commit; if the base snapshot changed, the commit may conflict and must be retried. This prevents silent lost updates and keeps commits atomic. + +[Table of Contents](#Apache-Iceberg) + +## What is hidden partitioning in Iceberg? +Hidden partitioning means partitioning is defined at the table metadata level (for example `day(ts)`), not as user-managed folder paths. Engines can still prune partitions, but you are not tied to a physical directory layout. This makes partition evolution safer and avoids leaking storage details into query logic. + +[Table of Contents](#Apache-Iceberg) + +## What is partition evolution and why is it important? +Partition evolution allows changing the partition spec over time (for example switching from `day(ts)` to `month(ts)` or adding a new partition field) without rewriting all historical data. It matters because real datasets change: query patterns evolve, and the “right” partitioning today may not be right in a year. + +[Table of Contents](#Apache-Iceberg) + +## How does schema evolution work in Iceberg? +Iceberg tracks columns by unique field IDs, enabling safe add/drop/rename operations without breaking readers. Schema evolution can be applied while keeping historical snapshots queryable. Some changes (like certain type changes) depend on engine support and can still be risky if consumers assume fixed schemas. + +[Table of Contents](#Apache-Iceberg) + +## What are equality deletes and positional deletes? +Deletes in Iceberg can be represented as separate delete files: ++ equality deletes: delete rows matching a key predicate (for example `id = 123`) ++ positional deletes: delete specific row positions in data files +They allow deletes without rewriting full data files, but too many delete files can hurt performance until compacted. + +[Table of Contents](#Apache-Iceberg) + +## How do upserts/merges work with Iceberg? +Upserts are usually implemented by engines (Spark/Flink/Trino) using `MERGE INTO` semantics, which may create new data files and delete files. The exact behavior depends on engine and write mode. Operationally, you often follow merges with maintenance to compact data and remove old snapshots. + +[Table of Contents](#Apache-Iceberg) + +## Why does the small files problem happen and how do you mitigate it? +Small files appear when many writers write tiny batches or when streaming jobs checkpoint frequently. Too many files increase planning overhead and reduce scan efficiency. Mitigations include setting target file sizes, batching writes, and running compaction/rewrite jobs to merge files. + +[Table of Contents](#Apache-Iceberg) + +## What maintenance operations are common for Iceberg tables? +Common maintenance includes: ++ rewriting data files (compaction) to target file sizes ++ rewriting manifests to reduce metadata overhead ++ expiring old snapshots to control metadata growth ++ removing orphan files +These jobs keep query planning and storage cost under control. + +[Table of Contents](#Apache-Iceberg) + +## What is an Iceberg catalog and what options exist? +A catalog stores table metadata pointers and namespace information. Common options include Hive metastore, REST catalog, and cloud-native catalogs (for example AWS Glue). Catalog choice affects authorization, interoperability, and operational complexity. + +[Table of Contents](#Apache-Iceberg) + +## How would you migrate an existing dataset to Iceberg? +You typically create an Iceberg table schema and then either: ++ register existing files (where supported) if they conform to the expected layout ++ rewrite data into Iceberg using an engine (Spark/Flink) for safety +After migration, you validate row counts, key constraints, and query performance, then switch readers to the new table. + +[Table of Contents](#Apache-Iceberg) + +## Iceberg vs Delta vs Hudi: when would you choose Iceberg? +Iceberg is a strong choice when you need an open, engine-agnostic table format with robust metadata, time travel, and evolution features. It fits well in environments with multiple query engines (Spark + Trino + Flink) and a preference for open standards. The best choice still depends on your stack, operational tooling, and update/CDC requirements. + +[Table of Contents](#Apache-Iceberg) + diff --git a/content/observability.md b/content/observability.md new file mode 100644 index 0000000..6f0cb07 --- /dev/null +++ b/content/observability.md @@ -0,0 +1,95 @@ +## [Main title](../README.md) +### [Interview questions](full.md) +# +# Data Observability ++ [What is data observability?](#What-is-data-observability) ++ [How is data observability different from application observability?](#How-is-data-observability-different-from-application-observability) ++ [What are the key signals you monitor for data pipelines?](#What-are-the-key-signals-you-monitor-for-data-pipelines) ++ [How do you define and measure an end-to-end SLA for data?](#How-do-you-define-and-measure-an-end-to-end-SLA-for-data) ++ [What should you log for each pipeline run?](#What-should-you-log-for-each-pipeline-run) ++ [How do you detect silent failures?](#How-do-you-detect-silent-failures) ++ [What is lineage and how does it help during incidents?](#What-is-lineage-and-how-does-it-help-during-incidents) ++ [How do you monitor and debug a broken metric in BI?](#How-do-you-monitor-and-debug-a-broken-metric-in-BI) ++ [How do you design alerts to avoid alert fatigue?](#How-do-you-design-alerts-to-avoid-alert-fatigue) ++ [How do you approach backfills safely?](#How-do-you-approach-backfills-safely) ++ [What is a runbook and what should it contain?](#What-is-a-runbook-and-what-should-it-contain) ++ [What are common incident patterns in data platforms?](#What-are-common-incident-patterns-in-data-platforms) + +## What is data observability? +Data observability is the practice of monitoring, alerting, and diagnosing the health of data and data pipelines. It focuses on whether datasets are arriving on time, with correct volume and values, and whether downstream consumers can rely on them. The goal is to reduce “data downtime” and speed up root-cause analysis. + +[Table of Contents](#Data-Observability) + +## How is data observability different from application observability? +Application observability focuses on service health (latency, error rates, traces) for request/response systems. Data observability focuses on dataset health and pipeline behavior over time (freshness, completeness, distribution shifts). Data issues can be silent, delayed, and cumulative, so you need different signals and baselines. + +[Table of Contents](#Data-Observability) + +## What are the key signals you monitor for data pipelines? +Common signals include: ++ freshness (max event/ingestion time) ++ volume (row counts, bytes, partitions) ++ schema changes (added/removed/typed columns) ++ quality checks (null rates, uniqueness, referential integrity) ++ pipeline run status and duration ++ cost signals (query time, bytes scanned, cluster utilization) + +[Table of Contents](#Data-Observability) + +## How do you define and measure an end-to-end SLA for data? +You define the SLA from source event time to availability in the final dataset or BI metric. Measure it with timestamps at each stage (ingest, transform, publish) and compute percentiles (p50/p95) as well as breach counts. SLAs should reflect consumer needs and include ownership and escalation paths. + +[Table of Contents](#Data-Observability) + +## What should you log for each pipeline run? +At minimum: ++ inputs (source tables/partitions, offsets, watermarks) ++ outputs (target tables/partitions, row counts written) ++ runtime and resource usage ++ warnings/errors and retry attempts ++ data quality results summary +This makes runs reproducible and speeds up debugging. + +[Table of Contents](#Data-Observability) + +## How do you detect silent failures? +Silent failures happen when the job succeeds but data is wrong (missing subset, wrong join, truncation). Detect them with dataset-level checks: freshness, volume bounds, distribution checks, and reconciliations against sources. Monitoring key business KPIs for unexpected changes is also effective. + +[Table of Contents](#Data-Observability) + +## What is lineage and how does it help during incidents? +Lineage describes upstream/downstream dependencies between datasets, jobs, and dashboards. During incidents, lineage helps you assess blast radius (which reports are impacted) and prioritize fixes. It also helps prevent regressions by making dependencies explicit. + +[Table of Contents](#Data-Observability) + +## How do you monitor and debug a broken metric in BI? +You start from the metric definition and trace dependencies to the underlying tables. Then check data freshness and recent changes (deploys, schema changes, upstream outages). Use reconciliation queries and sampling to locate where the metric diverged, then confirm whether it is a data issue or a definition change. + +[Table of Contents](#Data-Observability) + +## How do you design alerts to avoid alert fatigue? +Alerts should be actionable, scoped, and owned. Use severity levels, suppression windows, and baselines that account for seasonality. Prefer multi-signal alerts (freshness + volume) over single noisy checks, and include runbooks with clear next steps. + +[Table of Contents](#Data-Observability) + +## How do you approach backfills safely? +Backfills can be expensive and risky because they rewrite history and can change metrics. A safe approach includes running in smaller batches, validating each batch, and separating “raw” from “published” layers so you can rebuild derived tables without re-ingesting sources. You also communicate expected metric changes and schedule around peak usage. + +[Table of Contents](#Data-Observability) + +## What is a runbook and what should it contain? +A runbook is a step-by-step guide for responding to an alert or incident. It should include impact assessment, common root causes, validation queries, mitigation steps, rollback/backfill steps, and escalation contacts. Good runbooks reduce time-to-recovery for on-call engineers. + +[Table of Contents](#Data-Observability) + +## What are common incident patterns in data platforms? +Common patterns include: ++ upstream outages and late data ++ schema changes breaking transforms ++ duplicates from retries/reprocessing ++ incorrect joins causing metric inflation ++ partition boundary/timezone bugs ++ cost spikes due to inefficient queries + +[Table of Contents](#Data-Observability) + diff --git a/content/python.md b/content/python.md new file mode 100644 index 0000000..7fe8c1d --- /dev/null +++ b/content/python.md @@ -0,0 +1,77 @@ +## [Main title](../README.md) +### [Interview questions](full.md) +# +# Python for Data Engineering ++ [Why is Python widely used in data engineering?](#Why-is-Python-widely-used-in-data-engineering) ++ [How do iterators and generators help with large data processing?](#How-do-iterators-and-generators-help-with-large-data-processing) ++ [What is the difference between threads, multiprocessing, and async IO in Python?](#What-is-the-difference-between-threads-multiprocessing-and-async-IO-in-Python) ++ [What is the GIL and why does it matter?](#What-is-the-GIL-and-why-does-it-matter) ++ [How do you read and write Parquet efficiently in Python?](#How-do-you-read-and-write-Parquet-efficiently-in-Python) ++ [How do you process large CSV files without running out of memory?](#How-do-you-process-large-CSV-files-without-running-out-of-memory) ++ [How do you implement retries with exponential backoff?](#How-do-you-implement-retries-with-exponential-backoff) ++ [What logging practices are important for data pipelines?](#What-logging-practices-are-important-for-data-pipelines) ++ [How do you structure a Python project for data pipelines?](#How-do-you-structure-a-Python-project-for-data-pipelines) ++ [How do you manage dependencies and reproducible environments?](#How-do-you-manage-dependencies-and-reproducible-environments) ++ [How do you test data transformations in Python?](#How-do-you-test-data-transformations-in-Python) ++ [How do you profile and optimize slow Python code?](#How-do-you-profile-and-optimize-slow-Python-code) + +## Why is Python widely used in data engineering? +Python is popular because it is productive, readable, and has a strong ecosystem for data processing, orchestration, and integration. It is commonly used for glue code, transformations, APIs, and automation, even when the heavy compute runs on distributed engines. + +[Table of Contents](#Python-for-Data-Engineering) + +## How do iterators and generators help with large data processing? +Iterators and generators allow streaming data instead of loading everything into memory. They enable processing large datasets row-by-row or batch-by-batch. This reduces memory usage and often improves stability of ETL jobs. + +[Table of Contents](#Python-for-Data-Engineering) + +## What is the difference between threads, multiprocessing, and async IO in Python? +Threads are useful for IO-bound workloads (network, disk) but limited by the GIL for CPU-bound tasks. Multiprocessing runs multiple processes and can use multiple CPU cores for CPU-bound work. Async IO uses an event loop to handle many concurrent IO operations efficiently when tasks spend time waiting. + +[Table of Contents](#Python-for-Data-Engineering) + +## What is the GIL and why does it matter? +The Global Interpreter Lock (GIL) allows only one thread to execute Python bytecode at a time in CPython. This limits CPU-bound parallelism with threads. Many data engineering tasks are IO-bound, so threads can still help, but CPU-bound tasks often require multiprocessing or native libraries. + +[Table of Contents](#Python-for-Data-Engineering) + +## How do you read and write Parquet efficiently in Python? +Using `pyarrow` is common because it supports efficient columnar reads, predicate pushdown (engine-dependent), and schema handling. You should select only needed columns, write reasonably sized row groups/files, and avoid creating many tiny files. For large datasets, integrate with a distributed engine rather than pure local processing. + +[Table of Contents](#Python-for-Data-Engineering) + +## How do you process large CSV files without running out of memory? +You can stream the file in chunks (for example using chunked readers), process line-by-line, or use a library that supports streaming and type inference control. You also avoid loading wide columns you do not need, and you write results incrementally. For very large data, consider Parquet and/or distributed processing. + +[Table of Contents](#Python-for-Data-Engineering) + +## How do you implement retries with exponential backoff? +Retries should include exponential backoff and jitter to avoid thundering herds. You should only retry on transient errors (timeouts, rate limits) and limit total retry time. Always make operations idempotent or use deduplication to prevent duplicates on retries. + +[Table of Contents](#Python-for-Data-Engineering) + +## What logging practices are important for data pipelines? +Logs should include pipeline identifiers, run ids, input/output counts, and key timestamps. Structured logging (JSON) helps search and correlation. You should log errors with context and avoid logging sensitive data. + +[Table of Contents](#Python-for-Data-Engineering) + +## How do you structure a Python project for data pipelines? +A common structure separates reusable library code from job entrypoints. You keep configuration separate, use dependency injection for external services, and standardize logging and error handling. Small CLI entrypoints improve local runs and CI tests. + +[Table of Contents](#Python-for-Data-Engineering) + +## How do you manage dependencies and reproducible environments? +Use pinned dependencies and a lockfile (or equivalent) and isolate environments (virtualenv/containers). Reproducibility also requires pinning Python versions and building deterministic images. CI should use the same environment definition as production. + +[Table of Contents](#Python-for-Data-Engineering) + +## How do you test data transformations in Python? +Use small fixture datasets and assert expected outputs (golden tests). Add property tests for invariants (non-negative amounts, uniqueness). Integration tests validate real connectors and schemas, but they should be isolated and stable. + +[Table of Contents](#Python-for-Data-Engineering) + +## How do you profile and optimize slow Python code? +Profile first to find hotspots (CPU, IO, memory). Common fixes include vectorizing operations, reducing data copies, using efficient data structures, and offloading heavy compute to native libraries or distributed engines. Optimization should be guided by measurable impact. + +[Table of Contents](#Python-for-Data-Engineering) + diff --git a/content/sql.md b/content/sql.md index c7f86cf..77f42dd 100644 --- a/content/sql.md +++ b/content/sql.md @@ -17,7 +17,7 @@ + [Write query to find employees with duplicate email.](#Write-query-to-find-employees-with-duplicate-email) + [Write a query to find all employee whose name contains the word "rich", regardless of case.](#Write-a-query-to-find-all-employee-whose-name-contains-the-word-"rich",-regardless-of-case) + [Is it safe to use rowid to locate a record in oracle sql queries?](#Is-it-safe-to-use-rowid-to-locate-a-record-in-oracle-sql-queries) -+ [What is a pseudoпїЅolumn?](#What-is-a-pseudoпїЅolumn) ++ [What is a pseudo-column?](#What-is-a-pseudo-column) + [What are the reasons for denormalizing the data?](#What-are-the-reasons-for-denormalizing-the-data) + [What is the feature in sql for writing if and else statements?](#What-is-the-feature-in-sql-for-writing-if-and-else-statements) + [What is the difference between delete and truncate in sql?](#What-is-the-difference-between-delete-and-truncate-in-sql) @@ -626,7 +626,7 @@ ROWID is the physical location of a row. We can do very fast lookup based on ROW [Table of Contents](#SQL) -## What is a pseudo�olumn? +## What is a pseudo-column? A Pseudocolumn is like a table column, but it is not stored in the same table. We can select from a Pseudocolumn, but we can not insert, update or delete on a Pseudocolumn. A Pseudocolumn is like a function with no arguments. Two most popular Pseudocolumns in Oracle are ROWID and ROWNUM. NEXTVAL and CURRVAL are also pseudo columns. [Table of Contents](#SQL) diff --git a/content/system-design.md b/content/system-design.md new file mode 100644 index 0000000..4d81b35 --- /dev/null +++ b/content/system-design.md @@ -0,0 +1,71 @@ +## [Main title](../README.md) +### [Interview questions](full.md) +# +# Data System Design ++ [How would you design an end-to-end batch analytics pipeline?](#How-would-you-design-an-end-to-end-batch-analytics-pipeline) ++ [How would you design a near-real-time ingestion pipeline?](#How-would-you-design-a-near-real-time-ingestion-pipeline) ++ [How do you ensure idempotency in data pipelines?](#How-do-you-ensure-idempotency-in-data-pipelines) ++ [How do you handle late arriving events and backfills?](#How-do-you-handle-late-arriving-events-and-backfills) ++ [How do you choose between batch and streaming?](#How-do-you-choose-between-batch-and-streaming) ++ [How do you model raw/silver/gold layers (bronze/silver/gold)?](#How-do-you-model-raw/silver/gold-layers-(bronze/silver/gold)) ++ [How do you design a data platform for multiple teams (multi-tenancy)?](#How-do-you-design-a-data-platform-for-multiple-teams-(multi-tenancy)) ++ [How do you design for schema evolution?](#How-do-you-design-for-schema-evolution) ++ [What are the main reliability patterns for pipelines?](#What-are-the-main-reliability-patterns-for-pipelines) ++ [How do you design observability for a data platform?](#How-do-you-design-observability-for-a-data-platform) ++ [How do you manage cost while meeting SLAs?](#How-do-you-manage-cost-while-meeting-SLAs) + +## How would you design an end-to-end batch analytics pipeline? +A typical design is ingestion into a raw layer (immutable), transformation into curated datasets (cleaned and modeled), and publishing into marts/BI-serving datasets. You define SLAs, data quality checks, and ownership for each dataset. Batch pipelines are often scheduled daily/hourly and optimized for correctness and cost. + +[Table of Contents](#Data-System-Design) + +## How would you design a near-real-time ingestion pipeline? +You typically use an event log or CDC stream (Kafka/PubSub/Kinesis) and write to a raw append-only store with durable offsets. Downstream, you materialize current-state tables and metrics with windowing and deduplication. You must design for replays, ordering, and late events. + +[Table of Contents](#Data-System-Design) + +## How do you ensure idempotency in data pipelines? +Idempotency means retries do not change the final result. Common strategies include writing with deterministic keys (upserts/merges), using exactly-once features where available, and keeping raw data immutable while rebuilding derived layers. You also store checkpoints/watermarks consistently with writes. + +[Table of Contents](#Data-System-Design) + +## How do you handle late arriving events and backfills? +You define allowed lateness and design tables by event time, not just ingestion time. For late events, you reprocess affected windows/partitions and update derived outputs deterministically. Backfills should be incremental, validated, and isolated so they do not corrupt current reporting. + +[Table of Contents](#Data-System-Design) + +## How do you choose between batch and streaming? +Streaming is chosen for low latency and continuous updates, but it adds complexity (state, ordering, exactly-once). Batch is simpler and often cheaper for many analytics use cases. Many real systems are hybrid: streaming ingestion into raw, batch modeling into marts. + +[Table of Contents](#Data-System-Design) + +## How do you model raw/silver/gold layers (bronze/silver/gold)? +Bronze/raw is immutable ingestion with minimal transformation. Silver is cleaned, standardized, and deduplicated data with consistent schemas. Gold is business-ready marts and aggregates designed for BI and applications. This layering helps with reprocessing, governance, and clear contracts. + +[Table of Contents](#Data-System-Design) + +## How do you design a data platform for multiple teams (multi-tenancy)? +You need isolation (separate compute or quotas), clear ownership, and strong governance. Provide shared standards for naming, testing, and documentation, and a catalog for discovery. Multi-tenancy also requires controlling cost and preventing one team’s workloads from impacting others. + +[Table of Contents](#Data-System-Design) + +## How do you design for schema evolution? +You define compatibility rules, version schemas/contracts, and enforce them in CI and ingestion. Downstream transforms should tolerate additive fields but fail fast on breaking changes. You also maintain backward-compatible outputs where multiple consumers depend on stable schemas. + +[Table of Contents](#Data-System-Design) + +## What are the main reliability patterns for pipelines? +Common patterns include idempotent writes, retries with backoff, dead-letter queues, exactly-once or effectively-once processing, and separation of raw from derived layers. You also use checkpointing and consistent watermark management to avoid gaps and duplicates. + +[Table of Contents](#Data-System-Design) + +## How do you design observability for a data platform? +You monitor freshness, volume, quality, and lineage, plus pipeline runtime and failure rates. Alerts should be actionable with runbooks. Observability should be end-to-end: from source to final BI datasets and critical metrics. + +[Table of Contents](#Data-System-Design) + +## How do you manage cost while meeting SLAs? +You optimize storage layout (partitioning, file sizes), avoid unnecessary recomputation, and use selective materialization. You also schedule heavy jobs off-peak, enforce resource quotas, and track cost per dataset/pipeline. Cost is part of design, not just an afterthought. + +[Table of Contents](#Data-System-Design) + diff --git a/scripts/generate_full.py b/scripts/generate_full.py new file mode 100644 index 0000000..2515223 --- /dev/null +++ b/scripts/generate_full.py @@ -0,0 +1,100 @@ +#!/usr/bin/env python3 +""" +Generate content/full.md from headings in content/*.md. + +This is optional (not run in CI) but helps keep the "full list" consistent. + +Usage: + python3 scripts/generate_full.py > content/full.md + +Or (safer): + python3 scripts/generate_full.py > /tmp/full.md && diff -u content/full.md /tmp/full.md +""" + +from __future__ import annotations + +import re +import sys +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] +CONTENT_DIR = ROOT / "content" + + +def github_slugify(text: str) -> str: + s = text.strip().lower() + s = re.sub(r"[^a-z0-9 \-]", "", s) + s = re.sub(r"\s+", "-", s) + s = re.sub(r"-{2,}", "-", s) + return s.strip("-") + + +def iter_content_files() -> list[Path]: + files = sorted(CONTENT_DIR.glob("*.md")) + return [p for p in files if p.name != "full.md"] + + +def parse_topic_and_questions(path: Path) -> tuple[str, list[str]]: + topic = path.stem + questions: list[str] = [] + + h1_re = re.compile(r"^#\s+(.+?)\s*$") + h2_re = re.compile(r"^##\s+(.+?)\s*$") + + for line in path.read_text(encoding="utf-8").splitlines(): + if line.startswith("## [Main title]"): + continue + if line.startswith("### [Interview questions]"): + continue + + m1 = h1_re.match(line) + if m1 and topic == path.stem: + topic = m1.group(1).strip() + continue + + m2 = h2_re.match(line) + if m2: + q = m2.group(1).strip() + if q: + questions.append(q) + + return topic, questions + + +def main() -> int: + content_files = iter_content_files() + if not content_files: + print("No content files found.", file=sys.stderr) + return 2 + + topics: list[tuple[str, str, list[str]]] = [] + for p in content_files: + topic, questions = parse_topic_and_questions(p) + topics.append((p.name, topic, questions)) + + out: list[str] = [] + out.append("## [Main title](../README.md)") + out.append("") + for _filename, topic, _questions in topics: + out.append(f"+ [{topic}](#{github_slugify(topic)})") + out.append("") + + for filename, topic, questions in topics: + out.append(f"## {topic}") + base_counts: dict[str, int] = {} + for q in questions: + base = github_slugify(q) + n = base_counts.get(base, 0) + base_counts[base] = n + 1 + slug = base if n == 0 else f"{base}-{n}" + out.append(f"+ [{q}]({filename}#{slug})") + out.append("") + + sys.stdout.write("\n".join(out).rstrip() + "\n") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) + diff --git a/scripts/repo_checks.py b/scripts/repo_checks.py new file mode 100644 index 0000000..07df4b5 --- /dev/null +++ b/scripts/repo_checks.py @@ -0,0 +1,225 @@ +#!/usr/bin/env python3 +""" +Lightweight repo checks for this markdown-based project. + +What it checks: +- No placeholder TODO links in README (e.g. href="TODO") +- No obvious encoding artifacts (U+FFFD replacement char, "пїЅ") +- Internal links in content/full.md that point to other content/*.md headings exist +""" + +from __future__ import annotations + +import re +import sys +from dataclasses import dataclass +import os +from pathlib import Path +from typing import Iterable +from urllib.parse import unquote + + +ROOT = Path(__file__).resolve().parents[1] +CONTENT_DIR = ROOT / "content" +README = ROOT / "README.md" +FULL = CONTENT_DIR / "full.md" + + +FORBIDDEN_TEXT_PATTERNS: list[tuple[str, re.Pattern[str]]] = [ + ("replacement character (U+FFFD)", re.compile("\uFFFD")), + ("mojibake marker (пїЅ)", re.compile(r"пїЅ")), +] + + +@dataclass(frozen=True) +class Problem: + file: Path + line_no: int + message: str + line: str + + +def _read_text(path: Path) -> str: + # Explicit UTF-8 helps surface bad encodings early. + return path.read_text(encoding="utf-8") + + +def _iter_lines(path: Path) -> Iterable[tuple[int, str]]: + for idx, line in enumerate(_read_text(path).splitlines(), start=1): + yield idx, line + + +def _github_slugify(text: str) -> str: + """ + Approximate GitHub heading id generation: + - lower-case + - remove punctuation + - spaces -> hyphens + - collapse multiple hyphens + + Note: GitHub's exact algorithm has edge cases; this is intentionally simple + and good enough for catching broken anchors in this repo. + """ + s = text.strip().lower() + # Keep alphanumerics, spaces, and hyphens; drop the rest. + s = re.sub(r"[^a-z0-9 \-]", "", s) + s = re.sub(r"\s+", "-", s) + s = re.sub(r"-{2,}", "-", s) + return s.strip("-") + + +def _collect_heading_slugs(md_path: Path) -> set[str]: + """ + Collect all GitHub-like slugs for headings within a markdown file. + Includes duplicate handling: 'title', 'title-1', 'title-2', ... + """ + slugs: set[str] = set() + counts: dict[str, int] = {} + heading_re = re.compile(r"^(#{1,6})\s+(.+?)\s*$") + + for _line_no, line in _iter_lines(md_path): + m = heading_re.match(line) + if not m: + continue + heading_text = m.group(2) + base = _github_slugify(heading_text) + if not base: + continue + n = counts.get(base, 0) + counts[base] = n + 1 + slug = base if n == 0 else f"{base}-{n}" + slugs.add(slug) + return slugs + + +def check_readme_placeholders(problems: list[Problem]) -> None: + readme_text = _read_text(README) + if 'href="TODO"' in readme_text: + for line_no, line in _iter_lines(README): + if 'href="TODO"' in line: + problems.append( + Problem( + file=README, + line_no=line_no, + message='README contains placeholder link href="TODO"', + line=line, + ) + ) + + # Verify all linked content files exist (simple sanity check). + for line_no, line in _iter_lines(README): + for m in re.finditer(r'href="(\./content/[^"]+\.md)"', line): + rel = m.group(1) + target = (ROOT / rel).resolve() + if not target.exists(): + problems.append( + Problem( + file=README, + line_no=line_no, + message=f"README links to missing file: {rel}", + line=line, + ) + ) + + +def check_forbidden_text(problems: list[Problem]) -> None: + paths = [README, FULL, *sorted(CONTENT_DIR.glob("*.md"))] + for path in paths: + try: + text = _read_text(path) + except UnicodeDecodeError as e: + problems.append( + Problem( + file=path, + line_no=1, + message=f"File is not valid UTF-8: {e}", + line="", + ) + ) + continue + + for label, pattern in FORBIDDEN_TEXT_PATTERNS: + for line_no, line in enumerate(text.splitlines(), start=1): + if pattern.search(line): + problems.append( + Problem( + file=path, + line_no=line_no, + message=f"Found {label}", + line=line, + ) + ) + + +def check_full_internal_links(problems: list[Problem], *, strict_anchors: bool) -> None: + if not FULL.exists(): + return + + # Cache heading slugs for all content files. + slug_cache: dict[Path, set[str]] = {} + + link_re = re.compile(r"\(([^)]+\.md#[^)]+)\)") + for line_no, line in _iter_lines(FULL): + for raw_target in link_re.findall(line): + if "://" in raw_target: + continue + file_part, frag = raw_target.split("#", 1) + target_path = (CONTENT_DIR / file_part).resolve() + if not target_path.exists(): + problems.append( + Problem( + file=FULL, + line_no=line_no, + message=f"Broken link target file not found: {file_part}", + line=line, + ) + ) + continue + + if strict_anchors: + if target_path not in slug_cache: + slug_cache[target_path] = _collect_heading_slugs(target_path) + + frag = unquote(frag) + frag_slug = _github_slugify(frag.replace("-", " ")) + if frag_slug and frag_slug not in slug_cache[target_path]: + problems.append( + Problem( + file=FULL, + line_no=line_no, + message=f"Broken anchor: {file_part}#{frag} (normalized: #{frag_slug})", + line=line, + ) + ) + + +def main() -> int: + problems: list[Problem] = [] + + if not CONTENT_DIR.exists(): + print(f"Expected directory not found: {CONTENT_DIR}", file=sys.stderr) + return 2 + + check_readme_placeholders(problems) + check_forbidden_text(problems) + strict_anchors = os.environ.get("STRICT_ANCHORS", "").strip() == "1" + check_full_internal_links(problems, strict_anchors=strict_anchors) + + if problems: + print("Repo checks failed:\n", file=sys.stderr) + for p in problems[:200]: + rel = p.file.relative_to(ROOT) + print(f"- {rel}:{p.line_no}: {p.message}", file=sys.stderr) + if p.line: + print(f" {p.line}", file=sys.stderr) + if len(problems) > 200: + print(f"\n... and {len(problems) - 200} more", file=sys.stderr) + return 1 + + print("Repo checks passed.") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) +
Databases and Data Warehouses
Parquet Apache Parquet Apache Parquet is a column-oriented data file format designed for efficient data storage and retrieval.TODOParquet format · Docs
DeltaDelta Lake is a storage framework that enables building a Lakehouse architecture with compute engines Delta examples
IcebergIcebergApache IcebergApache Iceberg is an open table format for huge analytic datasets.Iceberg docs
HudiHudiApache HudiApache Hudi brings upserts, deletes, and incremental processing to data lakes.Hudi docs
Big Data FrameworksFlume Apache Flume Apache Flume is a distributed, reliable, and available software for efficiently collecting, aggregating, and moving large amounts of log data.TODOFlume User Guide
HadoopImpala Apache Impala Apache Impala is a parallel processing SQL query engine for data stored in a computer cluster running Apache Hadoop.TODOImpala docs
KafkaGoogle Cloud Platform is a suite of cloud computing services. Awesome GCP
Modern Data Stack
dbtdbtdbtdbt is a transformation framework for building tested and documented SQL models.dbt tests
TheoryDWH Architectures A data warehouse architecture is a method of defining the overall architecture of data communication processing and presentation that exist for end-clients computing within the enterprise. Awesome databases
CDCChange Data Capture (CDC)CDC captures inserts/updates/deletes from source systems for low-latency ingestion.Debezium docs
Data ModelingData ModelingDimensional modeling concepts used to build reliable analytics datasets.Kimball Group
Data QualityData QualityTests, monitoring, and practices to ensure datasets are trusted and correct.Great Expectations docs
Data ObservabilityData ObservabilityMonitoring and incident response practices for pipeline and dataset health.OpenLineage
Data GovernanceData GovernanceOwnership, policies, privacy, and access controls for data platforms.DataHub
Cost OptimizationCost OptimizationPractical techniques to reduce compute and storage costs while meeting SLAs.Spark tuning
PythonPython for Data EngineeringPython fundamentals for reliable, scalable data pipelines and tooling.PyArrow docs
System DesignData System DesignSystem design interview questions for batch/streaming data platforms.Data mesh overview
Airflow Data Structures A data structure is a specialized format for organizing, processing, retrieving and storing data. TODOAwesome Algorithms
SQLTableau Tableau Tableau is a powerful data visualization tool used in the Business Intelligence.TODOTableau Desktop docs
Looker Looker Looker is an enterprise platform for BI, data applications, and embedded analytics that helps you explore and share insights in real time.TODOLooker docs
Kafka Apache Superset Apache Superset Superset is a modern data exploration and data visualization platformTODOSuperset docs