From 806c5964418e39a0ecb17f42abf49fd62545c78f Mon Sep 17 00:00:00 2001 From: Rafael Riki Ogawa Osiro Date: Sat, 20 Jun 2026 00:51:16 -0300 Subject: [PATCH 1/3] docs(after-date-time): restructure check docs into hub pattern --- docs/data-quality-checks/after-date-check.md | 320 ------------------ .../after-date-time/api.md | 101 ++++++ .../after-date-time/examples.md | 265 +++++++++++++++ .../after-date-time/faq.md | 75 ++++ .../after-date-time/how-it-works.md | 115 +++++++ .../after-date-time/introduction.md | 88 +++++ .../overview-of-a-check.md | 2 +- .../rule-types-overview.md | 2 +- docs/operations/profile/profile.md | 2 +- docs/stylesheets/extra.css | 13 + mkdocs.yml | 10 +- 11 files changed, 668 insertions(+), 325 deletions(-) delete mode 100644 docs/data-quality-checks/after-date-check.md create mode 100644 docs/data-quality-checks/after-date-time/api.md create mode 100644 docs/data-quality-checks/after-date-time/examples.md create mode 100644 docs/data-quality-checks/after-date-time/faq.md create mode 100644 docs/data-quality-checks/after-date-time/how-it-works.md create mode 100644 docs/data-quality-checks/after-date-time/introduction.md diff --git a/docs/data-quality-checks/after-date-check.md b/docs/data-quality-checks/after-date-check.md deleted file mode 100644 index abf3a00911..0000000000 --- a/docs/data-quality-checks/after-date-check.md +++ /dev/null @@ -1,320 +0,0 @@ -# After Date Time - -Use the `afterDateTime` rule when you need to ensure that a timestamp value occurs **strictly after a defined cutoff date and time**. - -This rule is commonly used for **data freshness validation**, **cutover enforcement**, and **post-migration correctness**. - -## What is After Date Time? - -Think of **After Date Time** as a **gatekeeper for time-based data**. - -Just like a security guard checks that people enter a building **after opening time**, this rule ensures that every value in a timestamp field occurs **after a specific date and time**. - -If any record shows up **before or exactly at the cutoff**, it gets flagged as an anomaly. - -## Add After Date Time Check - -Use the After Date Time Check to validate timestamp fields against a fixed cutoff and detect records that violate expected time boundaries. - -
- -## What Does After Date Time Do? - -After Date Time helps you answer questions like: - -- “Are all records created **after the system go-live time**?” -- “Did any old data sneak in after migration?” -- “Are late or backdated records affecting reports?” -- “Is this dataset truly fresh and recent?” - -**In simple terms:** It guarantees that your data is **newer than a specific moment in time**. - -## How Does After Date Time Work? - -### Step 1: Select the Timestamp Field - -You choose a **single date or timestamp field**, such as: - -- `created_at` -- `order_date` -- `event_time` -- `processed_at` - -### Step 2: Define the Cutoff Date & Time - -You specify the **exact date and time** that acts as the lower boundary. - -**Example:** 2025-12-01 06:15:00 - -Only records **after this moment** are valid. - -### Step 3: Rule Evaluation - -For each record, Qualytics checks: - -- Is the field value **greater than** the cutoff timestamp? - -### Step 4: Get Your Results - -- **Pass** – All timestamps are after the cutoff -- **Anomalies Found** – One or more records violate the rule and are flagged - -## Why Should You Use After Date Time? - -### 1. Enforce System Cutovers - -During migrations or system upgrades, old timestamps can corrupt reports. -This rule ensures **only post-cutover data exists**. - -### 2. Validate Data Freshness - -Helps confirm that pipelines are producing **new data**, not reprocessing old records. - -### 3. Prevent Reporting Errors - -Backdated entries can: - -- Inflate historical metrics -- Break daily dashboards -- Trigger incorrect alerts - -After Date Time stops this at the source. - -### 4. Strengthen Compliance & Audits - -Regulated environments often require strict timestamp validation. -This rule provides a **clear, auditable boundary**. - -## Real-Life Example: Enforcing a Login Cutover After System Migration - -### The Situation - -**BrightCart**, an e-commerce company, migrated its **user authentication system** to a new identity provider on **December 1, 2025 at 06:15 UTC**. - -From this point onward: - -- All user login activity must be recorded by the new system -- Any login timestamp before this cutoff is considered invalid -- Old timestamps indicate legacy data, cache issues, or failed sync jobs - -This login data is critical because it feeds: - -- Account security audits -- User activity tracking -- Fraud detection -- Session-based personalization - -### The Problem They Faced - -A few days after the migration, the security team noticed something unusual: - -- Some users appeared inactive for years -- Fraud detection rules started misfiring -- Recently active customers were flagged as **“dormant”** - -On investigation, they found that: - -- The `LAST_LOGIN_TS` field still contained old timestamps -- Many records had values like `2022-03-01` -- These values should not exist after a **2025 cutover** - -Manual checking wasn’t scalable — the table contained **millions of rows**. - -### The Solution: After Date Time - -The data team configured an **After Date Time** check to enforce a hard time boundary. - -
- -#### What they configured: - -- **Table:** `ECOMMERCE_ORDERS` -- **Field:** `LAST_LOGIN_TS` -- **Cutoff:** `2025-12-01 06:15 UTC` -- **Coverage:** 100% (every record must comply) - -**Rule intent:** -Every user record must have a `LAST_LOGIN_TS` that occurred **after** the authentication system migration. - -### Sample Records (What the Data Looked Like) - -| LAST_LOGIN_TS | CUSTOMER_EMAIL | CUSTOMER_ID | -|--------------|----------------|-------------| -| 2025-12-03 09:42 | customer043@example.com | C013 | -| 2022-03-01 10:00 | customer041@example.com | C011 | -| 2022-03-01 10:00 | customer040@example.com | C010 | -| 2025-12-05 18:21 | customer031@example.com | C001 | - -### What After Date Time Discovered - -When the check ran, Qualytics flagged **110 anomalous records**. - -All failed records shared the same issue: - -- Their `LAST_LOGIN_TS` occurred **before the cutover** -- Some timestamps were copied from legacy systems -- Others were default or cached values - -!!! warning "ANOMALIES DETECTED" - - Rule Applied: After Date Time - - Field Checked: LAST_LOGIN_TS - - Cutoff: 2025-12-01 06:15 UTC - - Anomalous Records: 110 - -### Anomaly Output (Source Records View) - -| LAST_LOGIN_TS | CUSTOMER_EMAIL | CUSTOMER_ID | -|---------------------------------------|-----------------------------|-------------| -| 2022-03-01T10:00:00.000Z | customer043@example.com | C013 | -| 2022-03-01T10:00:00.000Z | customer041@example.com | C011 | -| 2022-03-01T10:00:00.000Z | customer040@example.com | C010 | -| 2022-03-01T10:00:00.000Z | customer031@example.com | C001 | -| 2022-03-01T10:00:00.000Z | customer028@example.com | C028 | -| 2022-03-01T10:00:00.000Z | customer026@example.com | C026 | -| 2022-03-01T10:00:00.000Z | customer021@example.com | | -| 2022-03-01T10:00:00.000Z | customer014@example.com | C014 | -| 2022-03-01T10:00:00.000Z | customer012@example.com | C012 | -| 2022-03-01T10:00:00.000Z | customer009@example.com | C009 | - -![anomaly-result](../assets/data-quality-checks/after-date-time/anomaly-result.png) - -**Why they failed:** - -- The rule requires timestamps to be **strictly later than** the cutoff -- Any value before or equal to the cutoff violates post-migration expectations - -### Why This Was a Serious Issue - -Without this check: - -- Active users looked inactive -- Security reports were inaccurate -- Fraud systems operated on false signals -- Business decisions were based on stale behavior data - -### The Outcome - -#### Immediate Fix - -- Identified the service populating `LAST_LOGIN_TS` -- Corrected the data pipeline to read from the new auth system -- Backfilled affected records with correct timestamps - -#### Long-Term Protection - -- Any future regression is detected automatically -- Cutover boundaries are continuously enforced -- Teams trust that login activity reflects real user behavior - -## Key Takeaways - -- After Date Time enforces strict time boundaries -- It guarantees post-cutover data correctness -- It prevents legacy or backdated records from polluting datasets -- It runs automatically once configured - -## When Should You Use After Date Time? - -Use After Date Time whenever you need to: - -- Enforce system go-live or migration cutoffs -- Validate ingestion or processing timestamps -- Detect stale or replayed data -- Protect dashboards from historical pollution - -### Field Scope - -**Single:** The rule evaluates a single specified field. - -**Accepted Types** - -| Type | Supported | -|---------|:------------------------:| -| `Date` |
:material-check-circle:{ style="color: #4caf50" }
| -| `Timestamp` |
:material-check-circle:{ style="color: #4caf50" }
| - -### General Properties - -{% - include-markdown "components/general-props/index.md" - start='' - end='' -%} - -### Specific Properties - -Specify a particular date and time to act as the threshold for the rule. - -| Name | Description | -|----------------|---------------------------------------------------------------| -|
Date
| The timestamp used as the lower boundary. Values in the selected field should be after this timestamp. | - -### Anomaly Types - -{% - include-markdown "components/anomaly-support/index.md" - start='' - end='' -%} - -### Example - -**Objective**: *Ensure that all O_ORDERDATE entries in the ORDERS table are later than 10:30 AM on December 31st, 1991.* - -**Sample Data** - -| O_ORDERKEY | O_ORDERDATE | -|------------|-------------| -| 1 | 1991-12-31 10:30:00 | -| 2 | 1992-01-02 09:15:00 | -| 3 | 1991-12-14 10:25:00 | - -=== "Payload example" - ``` json - { - "description": "Ensure that all O_ORDERDATE entries in the ORDERS table are later than 10:30 AM on December 31st, 1991.", - "coverage": 1, - "properties": { - "datetime": "1991-12-31 10:30:00" - }, - "tags": [], - "fields": ["O_ORDERDATE"], - "additional_metadata": {"key 1": "value 1", "key 2": "value 2"}, - "rule": "afterDateTime", - "container_id": {container_id}, - "template_id": {template_id}, - "filter": "1=1" - } - ``` - -**Anomaly Explanation** - -In the sample data above, the entries with `O_ORDERKEY` **1** and **3** do not satisfy the rule because their `O_ORDERDATE` values are not after **1991-12-31 10:30:00**. - -=== "Flowchart" - ``` mermaid - graph TD - A[Start] --> B[Retrieve O_ORDERDATE] - B --> C{Is O_ORDERDATE > '1991-12-31 10:30:00'?} - C -->|Yes| D[Move to Next Record/End] - C -->|No| E[Mark as Anomalous] - E --> D - ``` -=== "SQL" - ```sql - -- An illustrative SQL query demonstrating the rule applied to example dataset(s) - select - o_orderkey - , o_orderdate - from orders - where - o_orderdate <= '1991-12-31 10:30:00' - ``` - -**Potential Violation Messages** - -!!! example "Record Anomaly" - The `O_ORDERDATE` value of `1991-12-14 10:30:00` is not later than **1991-12-31 10:30:00** - -!!! example "Shape Anomaly" - In `O_ORDERDATE`, 66.667% of 3 filtered records (2) are not later than **1991-12-31 10:30:00** diff --git a/docs/data-quality-checks/after-date-time/api.md b/docs/data-quality-checks/after-date-time/api.md new file mode 100644 index 0000000000..8846f127a4 --- /dev/null +++ b/docs/data-quality-checks/after-date-time/api.md @@ -0,0 +1,101 @@ +# :material-api:{ .middle style="color: var(--q-brick)" } After Date Time Check API + +The After Date Time check is created and managed through the standard Quality Checks API by setting `rule` to `afterDateTime` and providing a single `datetime` value under `properties`. The `fields` array must contain exactly one `Date` or `Timestamp` field. + +!!! tip + For complete API documentation, including request and response schemas, visit the [API docs](https://demo.qualytics.io/api/docs){:target="_blank"}. + +## Endpoints + +| Method | Path | Purpose | +|:---|:---|:---| +| `POST` | `/api/quality-checks` | Create a new After Date Time check. | +| `GET` | `/api/quality-checks/{id}` | Retrieve an After Date Time check by ID. | +| `PUT` | `/api/quality-checks/{id}` | Update an existing After Date Time check. | +| `DELETE` | `/api/quality-checks/{id}` | Delete (or archive) an After Date Time check. | + +**Permissions** (on the target container's team): + +- **`POST`** and **`PUT`** for an Active check: **Author** team permission (or above). +- **`POST`** and **`PUT`** for a Draft check: **Drafter** team permission (or above). +- **`DELETE`**: **Author** team permission (or above). +- **`GET`**: **Reporter** team permission (or above). + +The team-permission hierarchy is Editor > Author > Drafter > Viewer > Reporter. "Or above" includes every higher level. + +## Payload Example + +Create an After Date Time check on `o_orderdate` with a 1991-12-31 10:30 UTC cutoff. The payload below shows every field accepted by the create endpoint: + +```json +{ + "description": "Ensure that all o_orderdate values in the ORDERS table are later than 1991-12-31 10:30 UTC.", + "rule": "afterDateTime", + "fields": ["o_orderdate"], + "container_id": 145, + "coverage": 1, + "filter": null, + "properties": { + "datetime": "1991-12-31T10:30:00Z" + }, + "tags": ["migration"], + "additional_metadata": {"jira": "DATA-1234"}, + "anomaly_message_field": null, + "template_id": null, + "status": "Active", + "owner_id": 7, + "default_anomaly_assignee_id": 12 +} +``` + +## Field Notes + +| Field | Required | Notes | +|:---|:---:|:---| +| `description` | Yes | Free-text description shown in the UI. | +| `rule` | Yes | Must be `"afterDateTime"`. | +| `fields` | Yes | Array with exactly one entry. Must reference a `Date` or `Timestamp` field. | +| `container_id` | Yes | ID of the container (table or file) the check runs against. | +| `coverage` | No | Fractional value between `0` and `1`. Defaults to `1` (strict; any violating row fires a Shape Anomaly). | +| `filter` | No | SQL `WHERE` expression. Applied **before** the comparison, so only filtered rows are evaluated. Send `null` for no filter. | +| `properties` | Yes | Object with a single key, `datetime`, holding an ISO-8601 timestamp string (for example, `"2025-12-01T06:15:00Z"`). Stored as a UTC instant. | +| `tags` | No | List of tag names applied to the check for filtering and organization. | +| `additional_metadata` | No | Free-form key-value pairs (typically links to catalog entries, tickets, governance records). | +| `anomaly_message_field` | No | Name of a source-record field whose value should be used as the Record Anomaly message instead of the default template-generated one. After Date Time emits Record Anomalies, so this field is honored at evaluation time. Shape Anomalies always use the fixed template. Send `null` to use the default Record template. | +| `template_id` | No | ID of a Check Template to associate the check with. `null` if not using a template. | +| `status` | No | `"Active"` (default) or `"Draft"`. Draft checks are not evaluated by Scans. | +| `owner_id` | No | ID of the user who owns the check. Defaults to the user creating the check when omitted. | +| `default_anomaly_assignee_id` | No | ID of the user automatically assigned to anomalies produced by the check. | + +## What `PUT` Can Change + +A `PUT` to `/api/quality-checks/{id}` can update most fields on an existing After Date Time check. Three values stay immutable: the rule type, the container the check is attached to, and the associated Check Template. If any of these needs to change, delete the check and create a new one. + +!!! info "Editable vs Immutable on PUT" + + **Editable** + + - `description` + - `coverage` + - `filter` + - `properties.datetime` (move the cutoff) + - `tags` + - `additional_metadata` + - `anomaly_message_field` + - `fields` (must remain a single `Date` or `Timestamp` field) + - `status` + - `owner_id` + - `default_anomaly_assignee_id` + + **Immutable** + + - `rule` + - `container_id` + - `template_id` + +## Related + +- [Introduction](introduction.md){:target="_blank"}: formal definition, field scope, and general/anomaly properties. +- [How It Works](how-it-works.md){:target="_blank"}: full semantics, NULL handling, filter behavior, and anomaly templates. +- [Examples](examples.md){:target="_blank"}: three production scenarios with sample data and resulting anomalies. +- [FAQ](faq.md){:target="_blank"}: short answers to the most frequent questions. diff --git a/docs/data-quality-checks/after-date-time/examples.md b/docs/data-quality-checks/after-date-time/examples.md new file mode 100644 index 0000000000..a6b2eafd4e --- /dev/null +++ b/docs/data-quality-checks/after-date-time/examples.md @@ -0,0 +1,265 @@ +# After Date Time Check Examples + +Three real-world scenarios that show how the After Date Time check is typically used in production: enforcing a system cutover, validating a daily ingestion window with a filter, and detecting pre-migration timestamps that should no longer exist. + +=== "System Cutover Enforcement" + + **The situation:** BrightCart migrated its identity provider on `2025-12-01 06:15 UTC`. Every row in the `users` table must carry a `last_login_ts` that is strictly later than the cutover; older timestamps mean the row was copied from the legacy system and is no longer authoritative. + + **Check configuration** + + | Field | Value | + |:---|:---| + | Rule | After Date Time | + | Field | `last_login_ts` | + | Date | `2025-12-01 06:15 UTC` | + | Filter | *(none)* | + | Coverage | 100% | + | Custom Anomaly Description | Off | + | Status | Active | + | Owner | *(check creator)* | + | Anomaly Assignee | *(Identity Platform on-call)* | + | Tags | `migration`, `auth` | + | Additional Metadata | `jira: DATA-1024` | + | Description | Login timestamps must be later than the identity-provider migration cutover. | + + **Payload** + + ```json + { + "description": "Login timestamps must be later than the identity-provider migration cutover.", + "rule": "afterDateTime", + "fields": ["last_login_ts"], + "container_id": 145, + "coverage": 1, + "filter": null, + "properties": { + "datetime": "2025-12-01T06:15:00Z" + }, + "tags": ["migration", "auth"], + "additional_metadata": {"jira": "DATA-1024"}, + "anomaly_message_field": null, + "template_id": null, + "status": "Active", + "owner_id": 7, + "default_anomaly_assignee_id": 12 + } + ``` + + **Sample Data** + + | user_id | last_login_ts | email | + |---------|---------------------------|------------------------| + | C001 | 2025-12-05 18:21:00 | customer031@example.com | + | C010 | 2022-03-01 10:00:00 | customer040@example.com | + | C011 | 2022-03-01 10:00:00 | customer041@example.com | + | C013 | 2025-12-03 09:42:00 | customer043@example.com | + + **What gets flagged** + + Rows `C010` and `C011` both carry `last_login_ts = 2022-03-01 10:00:00`, which is earlier than the cutover. Both rows fire a Record Anomaly and contribute to a Shape Anomaly for the dataset. + + !!! example "Record Anomaly" + The field `last_login_ts` has value `2022-03-01 10:00:00`, which is not later than **2025-12-01T06:15:00Z** + + !!! example "Shape Anomaly" + For the field `last_login_ts`, 50.000% of 4 records (2) are not later than **2025-12-01T06:15:00Z** + + **Flowchart** + + ```mermaid + graph TD + A["No filter, evaluate all rows"] --> B["Interpret last_login_ts as timestamp"] + B --> C{"Is last_login_ts >
2025-12-01T06:15:00Z?"} + C -->|Yes| D["Row passes"] + C -->|No| E["Flag row.
Record Anomaly per row;
Shape Anomaly if rate > coverage."] + ``` + + **Equivalent SQL** + + ```sql + -- Rows the After Date Time check would flag. + SELECT u.* + FROM users u + WHERE u.last_login_ts IS NOT NULL + AND u.last_login_ts <= TIMESTAMP '2025-12-01 06:15:00 UTC'; + ``` + +=== "Daily Ingestion Window with Filter" + + **The situation:** A pipeline writes a `processed_at` value to every row of the `events` table when it lands. Today's batch must contain only rows whose `processed_at` is strictly later than the start of the batch window (`2026-06-19 00:00 UTC`). Earlier `processed_at` values indicate a replay or a backfill that bypassed the daily window. + + **Check configuration** + + | Field | Value | + |:---|:---| + | Rule | After Date Time | + | Field | `processed_at` | + | Date | `2026-06-19 00:00 UTC` | + | Filter | `event_date = current_date()` | + | Coverage | 100% | + | Custom Anomaly Description | Off | + | Status | Active | + | Owner | *(check creator)* | + | Anomaly Assignee | *(Ingestion on-call)* | + | Tags | `ingestion`, `freshness` | + | Additional Metadata | `jira: DATA-2045` | + | Description | Today's events must have processed_at after the start of today's window. | + + **Payload** + + ```json + { + "description": "Today's events must have processed_at after the start of today's window.", + "rule": "afterDateTime", + "fields": ["processed_at"], + "container_id": 212, + "coverage": 1, + "filter": "event_date = current_date()", + "properties": { + "datetime": "2026-06-19T00:00:00Z" + }, + "tags": ["ingestion", "freshness"], + "additional_metadata": {"jira": "DATA-2045"}, + "anomaly_message_field": null, + "template_id": null, + "status": "Active", + "owner_id": 7, + "default_anomaly_assignee_id": 18 + } + ``` + + **Sample Data** (filtered to `event_date = current_date()`) + + | event_id | event_date | processed_at | + |----------|------------|-----------------------------| + | e-001 | today | 2026-06-19 02:11:00 | + | e-002 | today | 2026-06-18 23:55:00 | + | e-003 | today | 2026-06-19 04:47:00 | + + **Why the filter matters** + + The filter runs **before** the comparison, so yesterday's events do not influence today's window. Only rows in the current day are tested against the `2026-06-19 00:00 UTC` boundary. + + **What gets flagged** + + `e-002` has `processed_at = 2026-06-18 23:55:00`, earlier than the start-of-day cutoff for today's window, so it is the only row reported. + + !!! example "Record Anomaly" + The field `processed_at` has value `2026-06-18 23:55:00`, which is not later than **2026-06-19T00:00:00Z** [filter: event_date = current_date()] + + !!! example "Shape Anomaly" + For the field `processed_at`, 33.333% of 3 records (1) are not later than **2026-06-19T00:00:00Z** [filter: event_date = current_date()] + + **Flowchart** + + ```mermaid + graph TD + A["Apply filter: event_date = current_date()"] --> B["Interpret processed_at as timestamp"] + B --> C{"Is processed_at >
2026-06-19T00:00:00Z?"} + C -->|Yes| D["Row passes"] + C -->|No| E["Flag row.
Anomaly message ends with
[filter: event_date = current_date()]"] + ``` + + **Equivalent SQL** + + ```sql + -- Rows the check would flag in today's window. + SELECT e.* + FROM events e + WHERE e.event_date = current_date() + AND e.processed_at IS NOT NULL + AND e.processed_at <= TIMESTAMP '2026-06-19 00:00:00 UTC'; + ``` + +=== "Detecting Pre-Migration Survivors" + + **The situation:** A warehouse migration moved historical `orders` data into a new schema on `1992-01-01 00:00 UTC`. Every order row in the new schema is expected to carry an `o_orderdate` later than that boundary. NULL order dates are allowed on the `unconfirmed` rows produced by a separate manual process and should not be flagged. + + **Check configuration** + + | Field | Value | + |:---|:---| + | Rule | After Date Time | + | Field | `o_orderdate` | + | Date | `1992-01-01 00:00 UTC` | + | Filter | *(none)* | + | Coverage | 99.5% | + | Custom Anomaly Description | Off | + | Status | Active | + | Owner | *(check creator)* | + | Anomaly Assignee | *(Warehouse Migration team)* | + | Tags | `migration`, `historical` | + | Additional Metadata | `jira: DATA-3071` | + | Description | Migrated orders must have an order date later than the migration cutoff. | + + **Payload** + + ```json + { + "description": "Migrated orders must have an order date later than the migration cutoff.", + "rule": "afterDateTime", + "fields": ["o_orderdate"], + "container_id": 318, + "coverage": 0.995, + "filter": null, + "properties": { + "datetime": "1992-01-01T00:00:00Z" + }, + "tags": ["migration", "historical"], + "additional_metadata": {"jira": "DATA-3071"}, + "anomaly_message_field": null, + "template_id": null, + "status": "Active", + "owner_id": 7, + "default_anomaly_assignee_id": 24 + } + ``` + + **Sample Data** + + | o_orderkey | o_orderdate | o_orderstatus | + |------------|------------------------------|---------------| + | 1 | 1991-12-31 10:30:00 | F | + | 2 | 1992-03-15 09:15:00 | O | + | 3 | 1991-12-14 10:25:00 | F | + | 4 | *(null)* | unconfirmed | + + **What gets flagged** + + Rows 1 and 3 fail because their `o_orderdate` is earlier than the cutoff. Row 4 is NULL, so it passes without firing an anomaly but still counts in the scanned total. The 50% violation rate exceeds the 0.5% tolerance, so a Shape Anomaly is emitted in addition to the two Record Anomalies. + + !!! example "Record Anomaly" + The field `o_orderdate` has value `1991-12-31 10:30:00`, which is not later than **1992-01-01T00:00:00Z** + + !!! example "Shape Anomaly" + For the field `o_orderdate`, 50.000% of 4 records (2) are not later than **1992-01-01T00:00:00Z** + + **Flowchart** + + ```mermaid + graph TD + A["No filter, evaluate all rows"] --> B["Interpret o_orderdate as timestamp"] + B --> C{"Is value NULL?"} + C -->|Yes| D["Row passes"] + C -->|No| E{"Is o_orderdate >
1992-01-01T00:00:00Z?"} + E -->|Yes| D + E -->|No| F["Flag row"] + ``` + + **Equivalent SQL** + + ```sql + -- Rows the check would flag. + SELECT o.* + FROM orders o + WHERE o.o_orderdate IS NOT NULL + AND o.o_orderdate <= TIMESTAMP '1992-01-01 00:00:00 UTC'; + ``` + +## Related + +- [Introduction](introduction.md){:target="_blank"}: formal definition, field scope, and general/anomaly properties. +- [How It Works](how-it-works.md){:target="_blank"}: full semantics, NULL handling, filter behavior, and edge cases. +- [API](api.md){:target="_blank"}: payload shape and field notes for creating an After Date Time check programmatically. +- [FAQ](faq.md){:target="_blank"}: short answers to the most frequent questions. diff --git a/docs/data-quality-checks/after-date-time/faq.md b/docs/data-quality-checks/after-date-time/faq.md new file mode 100644 index 0000000000..02d81a387b --- /dev/null +++ b/docs/data-quality-checks/after-date-time/faq.md @@ -0,0 +1,75 @@ +# :material-help-circle-outline:{ .middle style="color: var(--q-brick)" } After Date Time Check FAQ + +Common questions about how the After Date Time check evaluates timestamps, how it handles NULLs and time zones, and how anomalies are reported. + +## Behavior + +### Is the cutoff inclusive or exclusive? + +Exclusive. The check uses strict greater-than (`>`), so a row whose value equals the cutoff is flagged. If you need inclusive semantics (`>=`), set the cutoff one second (for timestamps) or one day (for dates) earlier than the value you want to allow. + +### How are NULL values treated? + +NULLs **pass**. A row with `NULL` in the evaluated field is not counted as a violation. After Date Time only asserts that present values fall after the cutoff. If the field must also be populated, pair the check with a [Not Null](../not-null-check.md){:target="_blank"} check on the same field. + +### What happens when the cutoff date and the field have different time zones? + +The cutoff is stored as a UTC instant. `Timestamp` fields that carry an explicit time zone are compared directly. Naive `Timestamp` and `Date` fields are interpreted using the platform's configured session time zone before the comparison. When the source data carries mixed-zone values, normalize the field upstream with a Computed Field so the boundary is unambiguous. + +### Does the filter run before or after the comparison? + +Before. The platform applies the filter first and then evaluates the comparison only on the rows that pass the filter. Filtered-out rows cannot trigger an anomaly and are not counted in the totals. + +### Can the field be a `String` that looks like a date? + +The UI restricts the field picker to `Date` and `Timestamp` columns, so the answer is effectively no. If the source column is stored as a string, expose it as a native `Timestamp` upstream (or with a Computed Field) before running After Date Time. + +## Anomaly Reporting + +### What do the anomaly messages look like? + +**Record Anomaly:** `The field '' has value '', which is not later than ` + +**Shape Anomaly:** `For the field '', X.XXX% of N records (K) are not later than ` + +When a filter is set, both Record and Shape Anomaly messages end with `[filter: ]`. + +### Does After Date Time produce Record Anomalies, Shape Anomalies, or both? + +Both. Every violating row produces a Record Anomaly. When the violation rate crosses the coverage threshold, a Shape Anomaly is emitted in addition. Lowering the coverage threshold (for example to `0.995`) keeps Record Anomalies per row but suppresses the Shape Anomaly when the rate stays below the threshold. + +### Why is the cutoff in the message shown as an ISO-8601 string with a `Z` suffix? + +The cutoff is stored as a UTC instant and the anomaly messages echo that ISO-8601 string. Even when you entered the cutoff in a local time zone in the UI, the persisted value is UTC; the anomaly message reflects that. + +### Does Custom Anomaly Description work for After Date Time? + +Yes. After Date Time emits Record Anomalies, so the `anomaly_message_field` payload field (and the **Custom Anomaly Description** toggle in the UI) replaces the Record Anomaly message with the value of the named column on the violating row. When that column is null, missing, or empty for a violating row, the standard Record template is used instead. The Shape Anomaly always uses the fixed template. + +## Configuration + +### Can I lower the coverage on an After Date Time check? + +Yes. Coverage at `1.0` (100%) means every row must pass; lowering it to `0.995` allows up to 0.5% of rows to fail without firing a Shape Anomaly. Per-row Record Anomalies are still produced for every violating row, regardless of coverage. Use lower coverage with care: a real regression that happens to fall just under the threshold will not fire a Shape alert. + +### Can I change the cutoff date on an existing check? + +Yes. A `PUT` to `/api/quality-checks/{id}` can update `properties.datetime` (along with most other fields). The rule type, the target container, and the associated Check Template stay immutable. See the [API page](api.md){:target="_blank"} for the editable/immutable matrix. + +### How do I enforce both a lower and an upper time bound? + +Two options: + +- Pair After Date Time with [Before Date Time](../before-date-time-check.md){:target="_blank"} on the same field. Each rule handles one side of the window. +- Use [Between](../between-check.md){:target="_blank"} when the window is fixed; prefer the After/Before pair when the two bounds are managed independently. + +### How does After Date Time differ from Freshness? + +After Date Time enforces a **fixed cutoff** across every row in the dataset. [Freshness](../freshness-check.md){:target="_blank"} asserts that the most recent value is within an interval of the current time, so the boundary moves with the clock. Use Freshness when the question is "is this data recent enough"; use After Date Time when the question is "is this value after a specific moment in time". + +## Related + +- [Introduction](introduction.md){:target="_blank"}: formal definition, field scope, and general/anomaly properties. +- [How It Works](how-it-works.md){:target="_blank"}: full semantics, NULL handling, filter behavior, and edge cases. +- [Examples](examples.md){:target="_blank"}: three production scenarios with sample data and resulting anomalies. +- [API](api.md){:target="_blank"}: payload example and field notes for creating an After Date Time check programmatically. diff --git a/docs/data-quality-checks/after-date-time/how-it-works.md b/docs/data-quality-checks/after-date-time/how-it-works.md new file mode 100644 index 0000000000..852a5be0e0 --- /dev/null +++ b/docs/data-quality-checks/after-date-time/how-it-works.md @@ -0,0 +1,115 @@ +# How After Date Time Checks Work + +This page covers everything the After Date Time check does, in detail: how it decides which rows are violations, how NULLs are handled, how the cutoff is compared against the field value, the resulting anomaly, and how the rule relates to other time-boundary rule types. + +If you only need a quick reference, the [Introduction](introduction.md){:target="_blank"} page covers the formal definition, field scope, and general/anomaly properties. This page is the detailed reference. + +## Evaluation Flow + +Every After Date Time check follows the same four-step evaluation flow: + +1. **Apply the filter clause.** If the check has a `filter` set, only rows matching the filter expression continue to the next step. Rows outside the filter are ignored and cannot contribute to the violation count. +2. **Read the field value as a timestamp.** The platform interprets each row's value as a timestamp for comparison; values that cannot be interpreted are treated as failing the comparison (see below). +3. **Compare against the cutoff.** The cutoff (stored as a UTC instant) is compared to the field value using strict greater-than (`>`). A row passes only when the field value is strictly later than the cutoff. +4. **Report violations.** Each row that fails the comparison contributes to the anomaly count and produces a Record Anomaly. If the violating fraction crosses the coverage threshold, a Shape Anomaly is emitted in addition. + +The comparison is strict, so a row whose field value equals the cutoff is flagged. To get inclusive semantics, set the cutoff one second (for timestamps) or one day (for dates) earlier than the boundary you want to allow. + +## NULL Handling + +The check **passes NULL values**: a row with `NULL` in the evaluated field is not counted as a violation. This is intentional. After Date Time only asserts that *present* values fall after the cutoff and does not enforce mandatory presence on the field. + +If the field must also be populated, pair After Date Time with a [Not Null](../not-null-check.md){:target="_blank"} check on the same field. Together they enforce *"the field must be present AND must be later than the cutoff"*. + +Values that cannot be interpreted as a timestamp are **flagged as violations**. Because the field picker accepts only `Date` and `Timestamp` columns, this rarely happens in practice. + +## Cutoff and Time Zone Behavior + +The cutoff is stored as a UTC instant. When you set the cutoff to `2025-12-01 06:15:00` in the UI, the platform records it as `2025-12-01T06:15:00Z` and uses that exact instant for every evaluation. + +The platform interprets the field side using its configured session time zone: + +- `Timestamp` fields stored with explicit time zone offsets are compared directly against the UTC cutoff. +- Naive `Timestamp` fields (no explicit zone) are treated according to the session time zone and converted to instants before comparison. +- `Date` fields are interpreted as the start of that calendar day in the session time zone, then compared to the cutoff instant. + +If your source data carries values in mixed or local time zones, normalize the field upstream with a Computed Field before running After Date Time so the comparison is unambiguous. + +## The Filter Clause + +The filter clause is a SQL `WHERE` expression applied **before** the evaluation. Filtered-out rows are ignored entirely (they cannot trigger a violation and are not counted in the totals). + +Common uses: + +- Restricting the boundary to a subset of the dataset (`tenant_id = 42`, `status = 'active'`). +- Excluding known-bad legacy rows that are tracked by a separate clean-up task. +- Scoping the check to a partition (`event_date >= '2025-12-01'`). + +When a filter is set, both the Record Anomaly and the Shape Anomaly messages end with `[filter: ]` so the evaluated scope is visible in the alert. + +## Coverage and Tolerance + +Coverage is a fractional value between `0` and `1` that sets the threshold for emitting a Shape Anomaly: + +- **`1.0` (100%):** every row in the filtered set must pass. Any violating row fires a Shape Anomaly for the dataset, together with one Record Anomaly per failing row. This is the default and the strictest setting. +- **`< 1.0`:** the check tolerates a fraction of violating rows before a Shape Anomaly fires. Record Anomalies still describe each individual failing row. + +Lower coverage values are useful when a small, known fraction of pre-cutover rows is expected (for example, during a slow migration). Use coverage carefully: a 0.5% tolerance can mask a real regression that happens to fall just under the threshold. + +## The Resulting Anomalies + +After Date Time produces both **Record Anomalies** (one per violating row) and **Shape Anomalies** (one per dataset, when the violation rate crosses the coverage threshold). The two templates are: + +**Record Anomaly** + +``` +The field '' has value '', which is not later than +``` + +**Shape Anomaly** + +``` +For the field '', X.XXX% of N records (K) are not later than +``` + +When a filter is set, both Record and Shape Anomaly messages end with `[filter: ]`. + +### What the numbers mean + +- **X.XXX%:** the fraction of filtered rows that fail the comparison. +- **N:** the total number of rows the check evaluated (after the filter, if any). +- **K:** the number of rows that fail the comparison. + +### Source Records Behavior + +The offending cell is highlighted with an orange outline and a warning-tinted background in the Source Records view, mirroring the platform's standard violation rendering. Only the cell carrying the violating value is highlighted; the rest of the row renders normally. + +## Custom Anomaly Description + +After Date Time supports Custom Anomaly Description because it emits Record Anomalies. When `anomaly_message_field` (or the **Custom Anomaly Description** toggle in the UI) is set to another column on the same row, the Record Anomaly message is replaced by the value of that column for the violating row. The Shape Anomaly always uses the fixed template. + +## Relationship with Other Rule Types + +After Date Time is part of a small family of time-boundary rule types. Use them together when a single boundary is not enough. + +| Rule Type | Why pair it with After Date Time | +|:---|:---| +| [Before Date Time](../before-date-time-check.md){:target="_blank"} | After Date Time defines a lower bound; Before Date Time defines an upper bound. Pair them to enforce an open time window (`cutoff_low < field < cutoff_high`). Both comparisons are strict. | +| [Between](../between-check.md){:target="_blank"} | Same effect as pairing After Date Time + Before Date Time, but expressed as a single rule. Prefer Between when the window is fixed; use the After/Before pair when the bounds are managed independently. | +| [Not Future](../not-future-check.md){:target="_blank"} | Enforces an *upper* bound at the current moment. Pair with After Date Time to require that timestamps fall after a historical cutoff AND not in the future. | +| [Not Null](../not-null-check.md){:target="_blank"} | After Date Time passes NULL values. Add a Not Null check to require presence in addition to the boundary. | +| [Freshness](../freshness-check.md){:target="_blank"} | Asserts that the *most recent* value is within an interval of the current time. After Date Time enforces a fixed cutoff across all rows. Choose Freshness when the bound is "recent enough" and After Date Time when it is "after a specific moment". | + +## Performance Considerations + +After Date Time evaluates one column with a `>` comparison, so it is one of the cheaper rule types to run. The cost scales linearly with the number of rows that pass the filter. Two practical implications: + +- A filter that narrows the dataset to the rows you actually care about (today's partition, a single tenant, only active rows) reduces the work proportionally. +- Comparisons on a native `Timestamp` column are cheaper than re-interpreting a non-typed value at scan time. If the upstream column is a string, project it through a Computed Field with `Timestamp` type so each scan reuses the already-typed value. + +## Related + +- [Introduction](introduction.md){:target="_blank"}: formal definition, field scope, and general/anomaly properties. +- [Examples](examples.md){:target="_blank"}: three production scenarios with sample data and resulting anomalies. +- [API](api.md){:target="_blank"}: payload shape and field notes for creating an After Date Time check programmatically. +- [FAQ](faq.md){:target="_blank"}: short answers to the most frequent questions. diff --git a/docs/data-quality-checks/after-date-time/introduction.md b/docs/data-quality-checks/after-date-time/introduction.md new file mode 100644 index 0000000000..9016655f15 --- /dev/null +++ b/docs/data-quality-checks/after-date-time/introduction.md @@ -0,0 +1,88 @@ +# :material-clock-check-outline:{ .middle style="color: var(--q-brick)" } After Date Time Check + +## Definition + +*Asserts that every value in a Date or Timestamp field is strictly later than a chosen cutoff date and time.* + +## Overview + +The After Date Time rule defines a lower time boundary on a single date or timestamp field. Each row in the target container must hold a value that is greater than the cutoff; any value equal to or earlier than the cutoff fires an anomaly. The comparison is strict (`>`), not inclusive (`>=`), and the cutoff is stored as a UTC instant. + +Typical use cases: + +- Enforce a system go-live or migration cutoff. +- Validate ingestion or processing timestamps against a freshness floor. +- Detect stale, replayed, or backfilled rows that slipped through a date-scoped pipeline. + +## Field Scope + +**Single:** The rule evaluates exactly one field per check. + +**Accepted Types** + +| Type | Supported | +|-------------|:-------------------------:| +| `Date` |
:material-check-circle:{ .lg style="color: #4caf50" }
| +| `Timestamp` |
:material-check-circle:{ .lg style="color: #4caf50" }
| + +## General Properties + +{% + include-markdown "components/general-props/index.md" + start='' + end='' +%} + +## Specific Properties + +After Date Time has one rule-specific property: + +| Name | Description | +|------|-------------| +|
Date
| The cutoff date and time. The check flags every row whose field value is **not strictly later than** this value. | + +## Anomaly Types + +{% + include-markdown "components/anomaly-support/index.md" + start='' + end='' +%} + +## Next Steps + +
+ +- :material-information-outline:{ .lg .middle } **How It Works** + + --- + + Full semantics: evaluation flow, NULL handling, filter behavior, coverage, anomaly templates, and how After Date Time relates to other rule types. + + [:octicons-arrow-right-24: How It Works](how-it-works.md) + +- :material-clipboard-text-outline:{ .lg .middle } **Examples** + + --- + + Three production scenarios with sample data, anomaly messages, and the SQL equivalent of what the check evaluates. + + [:octicons-arrow-right-24: Examples](examples.md) + +- :material-api:{ .lg .middle } **API** + + --- + + Payload shape and field notes for creating an After Date Time check programmatically. + + [:octicons-arrow-right-24: API](api.md) + +- :material-help-circle-outline:{ .lg .middle } **FAQ** + + --- + + Short answers to questions about NULLs, inclusivity, time zones, and anomaly reporting. + + [:octicons-arrow-right-24: FAQ](faq.md) + +
diff --git a/docs/data-quality-checks/overview-of-a-check.md b/docs/data-quality-checks/overview-of-a-check.md index faf8627146..433011ab5f 100644 --- a/docs/data-quality-checks/overview-of-a-check.md +++ b/docs/data-quality-checks/overview-of-a-check.md @@ -156,7 +156,7 @@ For more details about check rule types, please refer to the [**Rule Types Overv | Rule Type | Description | |---------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------| -| [After Date Time](../data-quality-checks/after-date-check.md) | Asserts that the field is a timestamp later than a specific date and time. | +| [After Date Time](../data-quality-checks/after-date-time/introduction.md) | Asserts that the field is a timestamp later than a specific date and time. | | [Aggregation Comparison](../data-quality-checks/aggregation-comparison-check.md) | Verifies that the specified comparison operator evaluates true when applied to two aggregation expressions. | | [Any Not Null](../data-quality-checks/any-not-null-check.md) | Asserts that one of the fields must not be null. | | [Before Date Time](../data-quality-checks/before-date-time-check.md) | Asserts that the field is a timestamp earlier than a specific date and time. | diff --git a/docs/data-quality-checks/rule-types-overview.md b/docs/data-quality-checks/rule-types-overview.md index 7f08905007..2822f8d8ba 100644 --- a/docs/data-quality-checks/rule-types-overview.md +++ b/docs/data-quality-checks/rule-types-overview.md @@ -8,7 +8,7 @@ Here’s an overview of the rule types and their purposes: | Rule Type | Description | |---------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------| -| [After Date Time](../data-quality-checks/after-date-check.md) | Asserts that the field is a timestamp later than a specific date and time. | +| [After Date Time](../data-quality-checks/after-date-time/introduction.md) | Asserts that the field is a timestamp later than a specific date and time. | | [Aggregation Comparison](../data-quality-checks/aggregation-comparison-check.md) | Verifies that the specified comparison operator evaluates true when applied to two aggregation expressions. | | [Any Not Null](../data-quality-checks/any-not-null-check.md) | Asserts that one of the fields must not be null. | | [Before DateTime](../data-quality-checks/before-date-time-check.md) | Asserts that the field is a timestamp earlier than a specific date and time. | diff --git a/docs/operations/profile/profile.md b/docs/operations/profile/profile.md index e7cb91d550..ee92ca73f3 100644 --- a/docs/operations/profile/profile.md +++ b/docs/operations/profile/profile.md @@ -203,7 +203,7 @@ The following table shows the additional AI Managed checks generated at this lev | AI Managed Checks | Reference | |--------|-------| | Time Distribution Size | [See more.](https://userguide.qualytics.io/checks/time-distribution-size-check/) | -| After Date Time | [See more.](https://userguide.qualytics.io/checks/after-date-check/) | +| After Date Time | [See more.](https://userguide.qualytics.io/data-quality-checks/after-date-time/introduction/) | | Before Date Time | [See more.](https://userguide.qualytics.io/checks/before-date-time-check/) | | Greater Than | [See more.](https://userguide.qualytics.io/checks/greater-than-check/) | | Greater Than Field | [See more.](https://userguide.qualytics.io/checks/greater-than-field-check/) | diff --git a/docs/stylesheets/extra.css b/docs/stylesheets/extra.css index acec27b991..83052c2994 100644 --- a/docs/stylesheets/extra.css +++ b/docs/stylesheets/extra.css @@ -955,6 +955,19 @@ color: var(--q-brick); } +/* Cell highlight that mirrors the orange outline + warning-tinted background + used to flag anomalous values in the Source Records view. + Use inside markdown table cells to mark a single value as anomalous. */ +.anomalous-cell { + display: inline-block; + padding: 0.05rem 0.4rem; + border: 1px solid var(--q-orange); + border-radius: 4px; + background-color: rgba(249, 103, 25, 0.12); + color: var(--q-brick); + font-weight: 500; +} + .text-sm { font-size: 0.7rem; } diff --git a/mkdocs.yml b/mkdocs.yml index e54bcd4a70..cd6ae59653 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -302,7 +302,12 @@ nav: - FAQ: data-quality-checks/ai-managed/faq.md - Rule Types: - Rule Types Overview: data-quality-checks/rule-types-overview.md - - After Date Time: data-quality-checks/after-date-check.md + - After Date Time: + - Introduction: data-quality-checks/after-date-time/introduction.md + - How It Works: data-quality-checks/after-date-time/how-it-works.md + - Examples: data-quality-checks/after-date-time/examples.md + - API: data-quality-checks/after-date-time/api.md + - FAQ: data-quality-checks/after-date-time/faq.md - Aggregation Comparison: data-quality-checks/aggregation-comparison-check.md - Any Not Null: data-quality-checks/any-not-null-check.md - Before Date Time: data-quality-checks/before-date-time-check.md @@ -1150,7 +1155,8 @@ plugins: 'data-quality-checks/inferred/api.md': 'data-quality-checks/ai-managed/api.md' 'data-quality-checks/inferred/faq.md': 'data-quality-checks/ai-managed/faq.md' 'checks/rule-types-overview.md': 'data-quality-checks/rule-types-overview.md' - 'checks/after-date-check.md': 'data-quality-checks/after-date-check.md' + 'checks/after-date-check.md': 'data-quality-checks/after-date-time/introduction.md' + 'data-quality-checks/after-date-check.md': 'data-quality-checks/after-date-time/introduction.md' 'checks/aggregation-comparison-check.md': 'data-quality-checks/aggregation-comparison-check.md' 'checks/any-not-null-check.md': 'data-quality-checks/any-not-null-check.md' 'checks/before-date-time-check.md': 'data-quality-checks/before-date-time-check.md' From 24a8fd7def9cdfd8294c0772878003ca12c6288e Mon Sep 17 00:00:00 2001 From: Rafael Riki Ogawa Osiro Date: Sat, 20 Jun 2026 02:36:58 -0300 Subject: [PATCH 2/3] docs(after-date-time): align H1 + card icons with Standard Icons convention --- docs/data-quality-checks/after-date-time/how-it-works.md | 2 +- docs/data-quality-checks/after-date-time/introduction.md | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/data-quality-checks/after-date-time/how-it-works.md b/docs/data-quality-checks/after-date-time/how-it-works.md index 852a5be0e0..ec1e98247e 100644 --- a/docs/data-quality-checks/after-date-time/how-it-works.md +++ b/docs/data-quality-checks/after-date-time/how-it-works.md @@ -1,4 +1,4 @@ -# How After Date Time Checks Work +# :material-file-cog:{ .middle style="color: var(--q-brick)" } How After Date Time Checks Work This page covers everything the After Date Time check does, in detail: how it decides which rows are violations, how NULLs are handled, how the cutoff is compared against the field value, the resulting anomaly, and how the rule relates to other time-boundary rule types. diff --git a/docs/data-quality-checks/after-date-time/introduction.md b/docs/data-quality-checks/after-date-time/introduction.md index 9016655f15..7ab8b718ed 100644 --- a/docs/data-quality-checks/after-date-time/introduction.md +++ b/docs/data-quality-checks/after-date-time/introduction.md @@ -1,4 +1,4 @@ -# :material-clock-check-outline:{ .middle style="color: var(--q-brick)" } After Date Time Check +# :material-book-open-variant:{ .middle style="color: var(--q-brick)" } After Date Time Check ## Definition @@ -53,7 +53,7 @@ After Date Time has one rule-specific property:
-- :material-information-outline:{ .lg .middle } **How It Works** +- :material-file-cog:{ .lg .middle } **How It Works** --- @@ -61,7 +61,7 @@ After Date Time has one rule-specific property: [:octicons-arrow-right-24: How It Works](how-it-works.md) -- :material-clipboard-text-outline:{ .lg .middle } **Examples** +- **Examples** --- From c39734a320bf92283a891cac74d1fb350ad2a802 Mon Sep 17 00:00:00 2001 From: Rafael Riki Ogawa Osiro Date: Sat, 20 Jun 2026 02:39:05 -0300 Subject: [PATCH 3/3] docs(after-date-time): document anomaly_message_field fallback on api page (greptile) --- docs/data-quality-checks/after-date-time/api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/data-quality-checks/after-date-time/api.md b/docs/data-quality-checks/after-date-time/api.md index 8846f127a4..c44c121ceb 100644 --- a/docs/data-quality-checks/after-date-time/api.md +++ b/docs/data-quality-checks/after-date-time/api.md @@ -61,7 +61,7 @@ Create an After Date Time check on `o_orderdate` with a 1991-12-31 10:30 UTC cut | `properties` | Yes | Object with a single key, `datetime`, holding an ISO-8601 timestamp string (for example, `"2025-12-01T06:15:00Z"`). Stored as a UTC instant. | | `tags` | No | List of tag names applied to the check for filtering and organization. | | `additional_metadata` | No | Free-form key-value pairs (typically links to catalog entries, tickets, governance records). | -| `anomaly_message_field` | No | Name of a source-record field whose value should be used as the Record Anomaly message instead of the default template-generated one. After Date Time emits Record Anomalies, so this field is honored at evaluation time. Shape Anomalies always use the fixed template. Send `null` to use the default Record template. | +| `anomaly_message_field` | No | Name of a source-record field whose value should be used as the Record Anomaly message instead of the default template-generated one. After Date Time emits Record Anomalies, so this field is honored at evaluation time. When the named column is null, missing, or empty for a violating row, the standard Record template is used instead. Shape Anomalies always use the fixed template. Send `null` to use the default Record template. | | `template_id` | No | ID of a Check Template to associate the check with. `null` if not using a template. | | `status` | No | `"Active"` (default) or `"Draft"`. Draft checks are not evaluated by Scans. | | `owner_id` | No | ID of the user who owns the check. Defaults to the user creating the check when omitted. |