Предложения по репозиторию

.github/workflows/repo-checks.yml

@@ -0,0 +1,26 @@
+name: Repo checks
+
+on:
+  push:
+    branches:
+      - "**"
+  pull_request:
+
+permissions:
+  contents: read
+
+jobs:
+  repo-checks:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Run repo checks
+        run: python3 scripts/repo_checks.py
+

CONTRIBUTING.md

@@ -0,0 +1,18 @@
+## Contributing
+
+Contributions are welcome: new questions, fixes, better links, and formatting improvements.
+
+### Quick checks
+
+Run the local repository checks before opening a PR:
+
+```bash
+python3 scripts/repo_checks.py
+```
+
+### Content guidelines
+
+- Keep files **UTF-8** encoded.
+- Avoid placeholder links like `TODO` in markdown or HTML.
+- If you add links in `content/full.md`, make sure the referenced heading exists in the target file.
+

README.md

@@ -1,9 +1,9 @@
 <h1 align="center">More than 2000+ questions for preparing a Data Engineer interview.</h1>
 <h2 align="center"><a href="./content/full.md">Full list of questions</a></h2>
+<p align="center">Pick a topic below or use the full list to practice end-to-end.</p>
 <h1 align="center">Interview questions for Data Engineer</h1>
 <div>
 <table>
-  <tr>
   <tr>
     <th colspan="5">Databases and Data Warehouses</th>
   </tr>
@@ -91,7 +91,7 @@
     <th><a href="https://parquet.apache.org/"><img style="vertical-align:middle" src="img/icon/parquet.ico" alt="Parquet"></a></th>
     <th><a href="./content/parquet.md">Apache Parquet</a></th>
     <th>Apache Parquet is a column-oriented data file format designed for efficient data storage and retrieval.</th>
-    <th><a href="TODO">TODO</a></th>
+    <th><a href="https://github.com/apache/parquet-format">Parquet format</a> · <a href="https://parquet.apache.org/docs/">Docs</a></th>
   </tr>
   <tr>
     <th><a href="https://github.com/delta-io"><img  style="vertical-align:middle"  src="img/icon/github.ico" alt="Delta"></a></th>
@@ -100,6 +100,20 @@
     <th>Delta Lake is a storage framework that enables building a Lakehouse architecture with compute engines</th>
     <th><a href="https://github.com/MrPowers/delta-examples">Delta examples</a></th>
   </tr>
+  <tr>
+    <th><a href="https://github.com/apache/iceberg"><img  style="vertical-align:middle"  src="img/icon/github.ico" alt="Iceberg"></a></th>
+    <th><a href="https://iceberg.apache.org/"><img style="vertical-align:middle" src="img/icon/fire.ico" alt="Iceberg"></a></th>
+    <th><a href="./content/iceberg.md">Apache Iceberg</a></th>
+    <th>Apache Iceberg is an open table format for huge analytic datasets.</th>
+    <th><a href="https://iceberg.apache.org/docs/latest/">Iceberg docs</a></th>
+  </tr>
+  <tr>
+    <th><a href="https://github.com/apache/hudi"><img  style="vertical-align:middle"  src="img/icon/github.ico" alt="Hudi"></a></th>
+    <th><a href="https://hudi.apache.org/"><img style="vertical-align:middle" src="img/icon/fire.ico" alt="Hudi"></a></th>
+    <th><a href="./content/hudi.md">Apache Hudi</a></th>
+    <th>Apache Hudi brings upserts, deletes, and incremental processing to data lakes.</th>
+    <th><a href="https://hudi.apache.org/docs/overview/">Hudi docs</a></th>
+  </tr>
  <th colspan="5"><a></a></th>
   <tr>
     <th colspan="5">Big Data Frameworks</th>
@@ -117,7 +131,7 @@
     <th><a href="https://flume.apache.org/"><img style="vertical-align:middle" src="img/icon/flume.ico" alt="Flume"></a></th>
     <th><a href="./content/flume.md">Apache Flume</a></th>
     <th>Apache Flume is a distributed, reliable, and available software for efficiently collecting, aggregating, and moving large amounts of log data.</th>
-    <th><a href="TODO">TODO</a></th>
+    <th><a href="https://flume.apache.org/FlumeUserGuide.html">Flume User Guide</a></th>
   </tr>
   <tr>
     <th><a href="https://github.com/apache/hadoop"><img  style="vertical-align:middle"  src="img/icon/github.ico" alt="Hadoop"></a></th>
@@ -132,7 +146,7 @@
     <th><a href="https://impala.apache.org/"><img style="vertical-align:middle" src="img/icon/impala.ico" alt="Impala"></a></th>
     <th><a href="./content/impala.md">Apache Impala</a></th>
     <th>Apache Impala is a parallel processing SQL query engine for data stored in a computer cluster running Apache Hadoop.</th>
-    <th><a href="TODO">TODO</a></th>
+    <th><a href="https://impala.apache.org/docs/build/html/">Impala docs</a></th>
   </tr>
   <tr>
     <th><a href="https://github.com/apache/kafka"><img  style="vertical-align:middle"  src="img/icon/github.ico" alt="Kafka"></a></th>
@@ -194,6 +208,17 @@
     <th>Google Cloud Platform is a suite of cloud computing services.</th>
     <th><a href="https://github.com/GoogleCloudPlatform/awesome-google-cloud">Awesome GCP</a></th>
   </tr>
+<th colspan="5"><a></a></th>
+  <tr>
+    <th colspan="5">Modern Data Stack</th>
+  </tr>
+  <tr>
+    <th><a href="https://github.com/dbt-labs/dbt-core"><img  style="vertical-align:middle"  src="img/icon/github.ico" alt="dbt"></a></th>
+    <th><a href="https://docs.getdbt.com/"><img style="vertical-align:middle" src="img/icon/fire.ico" alt="dbt"></a></th>
+    <th><a href="./content/dbt.md">dbt</a></th>
+    <th>dbt is a transformation framework for building tested and documented SQL models.</th>
+    <th><a href="https://docs.getdbt.com/docs/build/tests">dbt tests</a></th>
+  </tr>
 <th colspan="5"><a></a></th>
   <tr>
     <th colspan="5"><b>Theory</b></th>
@@ -203,12 +228,60 @@
     <th><a href="./content/dwha.md">DWH Architectures</a></th>
     <th>A data warehouse architecture is a method of defining the overall architecture of data communication processing and presentation that exist for end-clients computing within the enterprise.</th>
     <th><a href="https://github.com/numetriclabz/awesome-db">Awesome databases</a></th>
+  </tr>
+    <tr>
+    <th colspan="2"><a href="./content/cdc.md"><img style="vertical-align:middle" src="img/icon/fire.ico" alt="CDC"></a></th>
+    <th><a href="./content/cdc.md">Change Data Capture (CDC)</a></th>
+    <th>CDC captures inserts/updates/deletes from source systems for low-latency ingestion.</th>
+    <th><a href="https://debezium.io/documentation/">Debezium docs</a></th>
+  </tr>
+    <tr>
+    <th colspan="2"><a href="./content/data-modeling.md"><img style="vertical-align:middle" src="img/icon/fire.ico" alt="Data Modeling"></a></th>
+    <th><a href="./content/data-modeling.md">Data Modeling</a></th>
+    <th>Dimensional modeling concepts used to build reliable analytics datasets.</th>
+    <th><a href="https://www.kimballgroup.com/">Kimball Group</a></th>
+  </tr>
+    <tr>
+    <th colspan="2"><a href="./content/data-quality.md"><img style="vertical-align:middle" src="img/icon/fire.ico" alt="Data Quality"></a></th>
+    <th><a href="./content/data-quality.md">Data Quality</a></th>
+    <th>Tests, monitoring, and practices to ensure datasets are trusted and correct.</th>
+    <th><a href="https://greatexpectations.io/docs/">Great Expectations docs</a></th>
+  </tr>
+    <tr>
+    <th colspan="2"><a href="./content/observability.md"><img style="vertical-align:middle" src="img/icon/fire.ico" alt="Data Observability"></a></th>
+    <th><a href="./content/observability.md">Data Observability</a></th>
+    <th>Monitoring and incident response practices for pipeline and dataset health.</th>
+    <th><a href="https://openlineage.io/">OpenLineage</a></th>
+  </tr>
+    <tr>
+    <th colspan="2"><a href="./content/data-governance.md"><img style="vertical-align:middle" src="img/icon/fire.ico" alt="Data Governance"></a></th>
+    <th><a href="./content/data-governance.md">Data Governance</a></th>
+    <th>Ownership, policies, privacy, and access controls for data platforms.</th>
+    <th><a href="https://github.com/datahub-project/datahub">DataHub</a></th>
+  </tr>
+    <tr>
+    <th colspan="2"><a href="./content/cost-optimization.md"><img style="vertical-align:middle" src="img/icon/fire.ico" alt="Cost Optimization"></a></th>
+    <th><a href="./content/cost-optimization.md">Cost Optimization</a></th>
+    <th>Practical techniques to reduce compute and storage costs while meeting SLAs.</th>
+    <th><a href="https://spark.apache.org/docs/latest/tuning.html">Spark tuning</a></th>
+  </tr>
+    <tr>
+    <th colspan="2"><a href="./content/python.md"><img style="vertical-align:middle" src="img/icon/fire.ico" alt="Python"></a></th>
+    <th><a href="./content/python.md">Python for Data Engineering</a></th>
+    <th>Python fundamentals for reliable, scalable data pipelines and tooling.</th>
+    <th><a href="https://arrow.apache.org/docs/python/">PyArrow docs</a></th>
+  </tr>
+    <tr>
+    <th colspan="2"><a href="./content/system-design.md"><img style="vertical-align:middle" src="img/icon/fire.ico" alt="System Design"></a></th>
+    <th><a href="./content/system-design.md">Data System Design</a></th>
+    <th>System design interview questions for batch/streaming data platforms.</th>
+    <th><a href="https://martinfowler.com/articles/data-monolith-to-mesh.html">Data mesh overview</a></th>
   </tr>
     <tr>
     <th colspan="2"><a href="./content/data-structure.md"><img style="vertical-align:middle" src="img/icon/datastruct.ico" alt="Airflow"></a></th>
     <th><a href="./content/data-structure.md">Data Structures</a></th>
     <th>A data structure is a specialized format for organizing, processing, retrieving and storing data. </th>
-    <th><a href="TODO">TODO</a></th>
+    <th><a href="https://github.com/tayllan/awesome-algorithms">Awesome Algorithms</a></th>
   </tr>
     <tr>
     <th colspan="2"><a href="./content/sql.md"><img style="vertical-align:middle" src="img/icon/sql.ico" alt="SQL"></a></th>
@@ -224,19 +297,19 @@
     <th colspan="2"><a href="./content/tableau.md"><img style="vertical-align:middle" src="img/icon/tableau.ico" alt="Tableau"></a></th>
     <th><a href="./content/tableau.md">Tableau</a></th>
     <th>Tableau is a powerful data visualization tool used in the Business Intelligence.</th>
-    <th><a href="TODO">TODO</a></th>
+    <th><a href="https://help.tableau.com/current/pro/desktop/en-us/default.htm">Tableau Desktop docs</a></th>
   </tr>
     <th colspan="2"><a href="./content/looker.md"><img style="vertical-align:middle" src="img/icon/looker.ico" alt="Looker"></a></th>
     <th><a href="./content/looker.md">Looker</a></th>
     <th>Looker is an enterprise platform for BI, data applications, and embedded analytics that helps you explore and share insights in real time.</th>
-    <th><a href="TODO">TODO</a></th>
+    <th><a href="https://cloud.google.com/looker/docs">Looker docs</a></th>
   </tr>
  <tr>
     <th><a href="https://github.com/apache/superset"><img  style="vertical-align:middle"  src="img/icon/github.ico" alt="Kafka"></a></th>
     <th colspan="2"><a href="https://superset.apache.org/"><img style="vertical-align:middle" src="img/icon/superset.ico" alt="Apache Superset"></a></th>
     <th><a href="./content/superset.md">Apache Superset</a></th>
     <th>Superset is a modern data exploration and data visualization platform</th>
-    <th><a href="TODO">TODO</a></th>
+    <th><a href="https://superset.apache.org/docs/intro">Superset docs</a></th>
   </tr>
 
 
@@ -245,4 +318,5 @@
 <div>
 <h2 align="center"> Contribution </h2>
 <h3>Please contribute to this repository to help it make better. Any change like new question, code improvement, doc improvement etc is very welcome.</h3>
-</div>
+<p align="center">See <a href="./CONTRIBUTING.md">CONTRIBUTING.md</a> for quick checks and guidelines.</p>
+</div>

content/cdc.md

@@ -0,0 +1,114 @@
+## [Main title](../README.md)
+### [Interview questions](full.md)
+#
+# Change Data Capture (CDC)
++ [What is CDC (Change Data Capture)?](#What-is-CDC-(Change-Data-Capture))
++ [When would you choose CDC over batch ingestion?](#When-would-you-choose-CDC-over-batch-ingestion)
++ [What are common CDC sources (WAL/binlog/triggers) and trade-offs?](#What-are-common-CDC-sources-(WAL/binlog/triggers)-and-trade-offs)
++ [What is the difference between snapshot and incremental CDC?](#What-is-the-difference-between-snapshot-and-incremental-CDC)
++ [What delivery semantics exist (at-most-once/at-least-once/exactly-once)?](#What-delivery-semantics-exist-(at-most-once/at-least-once/exactly-once))
++ [How do you make CDC ingestion idempotent?](#How-do-you-make-CDC-ingestion-idempotent)
++ [How do you handle deletes in CDC pipelines?](#How-do-you-handle-deletes-in-CDC-pipelines)
++ [How do you handle updates when the source does not provide full row images?](#How-do-you-handle-updates-when-the-source-does-not-provide-full-row-images)
++ [What is ordering and why is it hard in CDC?](#What-is-ordering-and-why-is-it-hard-in-CDC)
++ [What is a watermark/offset and where should it be stored?](#What-is-a-watermark/offset-and-where-should-it-be-stored)
++ [How do you handle schema evolution with CDC?](#How-do-you-handle-schema-evolution-with-CDC)
++ [What is the outbox pattern and why is it useful?](#What-is-the-outbox-pattern-and-why-is-it-useful)
++ [How do you monitor CDC lag and where can lag come from?](#How-do-you-monitor-CDC-lag-and-where-can-lag-come-from)
++ [How do you design a safe backfill/reprocessing strategy for CDC?](#How-do-you-design-a-safe-backfill/reprocessing-strategy-for-CDC)
++ [What are the most common failure modes in CDC pipelines?](#What-are-the-most-common-failure-modes-in-CDC-pipelines)
+
+## What is CDC (Change Data Capture)?
+CDC is a technique to capture row-level changes (inserts, updates, deletes) from a source system and stream them downstream. In databases, CDC is commonly log-based (reading WAL/binlog) so it can capture changes without repeatedly scanning full tables. CDC pipelines often feed event logs (append-only) and/or materialized current-state tables.
+
+[Table of Contents](#Change-Data-Capture-(CDC))
+
+## When would you choose CDC over batch ingestion?
+Choose CDC when you need low-latency updates, continuous incremental processing, or a reliable change history. CDC is also useful when full-table scans are too expensive because the source is large or changes frequently. Batch ingestion is often simpler when latency requirements are relaxed and operational complexity should be minimized.
+
+[Table of Contents](#Change-Data-Capture-(CDC))
+
+## What are common CDC sources (WAL/binlog/triggers) and trade-offs?
+Common sources are:
++ database logs (WAL/binlog): low overhead and high fidelity, but requires privileges and careful offset handling
++ triggers: can work without log access, but adds write overhead and is easy to break
++ timestamp-based queries: simple but can miss updates or create duplicates if clocks/transactions behave unexpectedly
+Log-based CDC is usually preferred for production at scale.
+
+[Table of Contents](#Change-Data-Capture-(CDC))
+
+## What is the difference between snapshot and incremental CDC?
+A snapshot reads an initial consistent baseline of existing rows. Incremental CDC then streams changes after that baseline using a log position/offset. Many systems require both: snapshot for bootstrap, incremental for ongoing updates.
+
+[Table of Contents](#Change-Data-Capture-(CDC))
+
+## What delivery semantics exist (at-most-once/at-least-once/exactly-once)?
+At-most-once can lose messages but avoids duplicates. At-least-once avoids loss but can produce duplicates on retries. Exactly-once is hard end-to-end; many systems implement “effectively once” by combining at-least-once delivery with idempotent writes and deterministic merge logic.
+
+[Table of Contents](#Change-Data-Capture-(CDC))
+
+## How do you make CDC ingestion idempotent?
+You make downstream writes idempotent by using stable keys (primary key + source LSN/transaction id) and applying deterministic upsert/merge logic. A common pattern is to load changes into an append-only raw log, then build current-state tables using “latest per key” logic. Idempotency must hold across retries and replays.
+
+[Table of Contents](#Change-Data-Capture-(CDC))
+
+## How do you handle deletes in CDC pipelines?
+Deletes may appear as explicit delete events, tombstones, or a “before image” without an “after image”. Downstream you can:
++ apply hard deletes (remove row)
++ apply soft deletes (is_deleted flag with deleted_at)
++ keep full history (append-only) and compute current state with delete logic
+The choice depends on compliance, analytics needs, and downstream consumers.
+
+[Table of Contents](#Change-Data-Capture-(CDC))
+
+## How do you handle updates when the source does not provide full row images?
+If updates are partial (only changed columns), you often need to reconstruct the full row by joining against previous state. Some CDC tools can be configured to emit before/after images; if not, current-state reconstruction becomes part of your pipeline. You must also handle out-of-order updates carefully.
+
+[Table of Contents](#Change-Data-Capture-(CDC))
+
+## What is ordering and why is it hard in CDC?
+Ordering means applying changes in the same order they happened in the source. It is hard because events can be partitioned (multiple tables/partitions), delivered out of order, or retried. Correctness often requires per-key ordering (or per-partition ordering) using source transaction metadata.
+
+[Table of Contents](#Change-Data-Capture-(CDC))
+
+## What is a watermark/offset and where should it be stored?
+An offset (watermark) is the position in the source log (or stream) up to which changes have been processed. It should be stored durably in the ingestion system (connector state store) and, for reliability, often also in your platform (for example a control table) so you can audit and recover. Offsets must be updated atomically with downstream writes to avoid gaps/duplicates.
+
+[Table of Contents](#Change-Data-Capture-(CDC))
+
+## How do you handle schema evolution with CDC?
+You need a strategy for compatible changes (additive columns) and breaking changes (renames, type changes). Common tools include schema registries and compatibility policies, plus strict validation in CI. Downstream tables should tolerate additive fields, and pipelines should alert on breaking changes before production data is corrupted.
+
+[Table of Contents](#Change-Data-Capture-(CDC))
+
+## What is the outbox pattern and why is it useful?
+The outbox pattern writes business events into an “outbox” table in the same transaction as the application write. CDC then captures the outbox changes and publishes them reliably. This avoids dual-write problems where the database update succeeds but the event publish fails (or vice versa).
+
+[Table of Contents](#Change-Data-Capture-(CDC))
+
+## How do you monitor CDC lag and where can lag come from?
+Lag can come from:
++ source: heavy write load, log retention limits
++ connector: slow polling, backpressure, failures
++ broker/stream: throughput limits, partition hotspots
++ downstream: slow merges/compaction, expensive transformations
+You monitor lag using source LSN vs processed LSN, consumer offsets, and end-to-end freshness in target tables.
+
+[Table of Contents](#Change-Data-Capture-(CDC))
+
+## How do you design a safe backfill/reprocessing strategy for CDC?
+A safe strategy usually separates raw change storage from derived tables. You keep the raw CDC log (immutable) and rebuild derived current-state/marts from it when needed. For backfills, you often run in a separate environment or commit to new snapshots/versions, then switch consumers after validation.
+
+[Table of Contents](#Change-Data-Capture-(CDC))
+
+## What are the most common failure modes in CDC pipelines?
+Common issues include:
++ duplicates due to retries/rebalances
++ missed events due to offset mismanagement or log retention
++ schema drift/breaking changes
++ incorrect delete handling
++ out-of-order application of updates
+Good pipelines treat CDC as “production software” with strong observability and tests.
+
+[Table of Contents](#Change-Data-Capture-(CDC))
+