diff --git a/docs/data-operate/export/export-overview.md b/docs/data-operate/export/export-overview.md index cadfecb956d5b..670da7fbf0c99 100644 --- a/docs/data-operate/export/export-overview.md +++ b/docs/data-operate/export/export-overview.md @@ -143,7 +143,7 @@ When Apache Doris exports to the Parquet file format, it first converts the in-m | hll | binary | BYTE_ARRAY | | :::note -In versions 2.1.11 and 3.0.7, the `parquet.enable_int96_timestamps` property is supported to specify whether the Doris `datetimev2` type is stored as `INT96` or `INT64` in Parquet. The default is `INT96`. `INT96` has been deprecated in the Parquet standard and is used only for compatibility with legacy systems (such as Hive versions before 4.0). +For Parquet Export and Outfile, use the `enable_int96_timestamps` property to choose the physical encoding of Doris `DATETIMEV2` values. The default is `false`, which writes `INT64` with a Parquet timestamp logical type. Set it to `true` to write the deprecated `INT96` encoding only when a legacy downstream reader, such as Hive 2 or Hive 3, requires it. Asynchronous Export stores this property with the export job and applies it when the job executes. If Doris later reads an explicitly exported `INT96` file, set `hive.parquet.time-zone` to the export session time zone in the Hive Catalog or file table-valued function so FileScannerV2 restores the original wall-clock value. ::: ## Related Documents diff --git a/docs/lakehouse/catalogs/hive-catalog.mdx b/docs/lakehouse/catalogs/hive-catalog.mdx index 6a99635565995..5970c8d0d2905 100644 --- a/docs/lakehouse/catalogs/hive-catalog.mdx +++ b/docs/lakehouse/catalogs/hive-catalog.mdx @@ -72,6 +72,8 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES ( * `hive.staging_dir`: Used to set the temporary staging root directory for Hive write operations. During the write process, data is first written to this directory and then moved to the final Hive table directory after completion. This path can be either a relative path relative to the Hive table root directory or an absolute path. The default value is `/tmp/.doris_staging`. In some cases, when writing to Hive tables, you may encounter issues where the `/tmp` directory and the table directory belong to different namespaces, causing write failures. In such cases, you can modify this parameter to use a relative path to resolve the issue. This parameter is supported since version 4.0.3. + * `hive.parquet.time-zone`: The compatibility time zone used to restore legacy Parquet `INT96` timestamps when FileScannerV2 reads Hive or Hudi tables. The default is empty, which preserves the wall-clock fields stored in `INT96` without applying a time zone conversion. Set this property only when the `INT96` files in the Catalog were normalized by the writer with a known time zone. For details, see [Parquet INT96 Timestamp Compatibility](#timestamp-compatibility). + * `{CommonProperties}` The CommonProperties section is for entering common attributes. Please see the "Common Properties" section in the [Catalog Overview](../catalog-overview.md). @@ -208,6 +210,50 @@ Hive transactional tables are supported from version 3.x onwards. For details, r * [ORC](../file-formats/orc.md) +## Parquet INT96 Timestamp Compatibility {#timestamp-compatibility} + +Hive tables can contain timestamp files produced by different engines. The physical encoding and writer behavior determine whether a time zone conversion is required. Doris does not infer a writer time zone from a Parquet `INT96` value because `INT96` has no time zone annotation. + +### Parquet INT96 timestamps + +FileScannerV2 handles Parquet timestamps as follows: + +| Physical encoding | Default behavior | Effect of `hive.parquet.time-zone` | +|---|---|---| +| `INT96` mapped to `DATETIMEV2` | Preserve the date and time fields stored in the file. The SQL session time zone does not shift the value. | Treat the raw `INT96` value as a UTC instant and convert it to the configured compatibility time zone, restoring timestamps that a legacy writer normalized. | +| `INT64` with a timestamp logical type | Follow the Parquet logical-type semantics. | Not affected. | + +For example, suppose the intended local timestamp is `2021-01-01 10:11:00` in `Asia/Shanghai`: + +* If the file stores the wall-clock fields directly as `10:11:00`, leave `hive.parquet.time-zone` unset. Doris returns `10:11:00` even when the SQL session uses `Asia/Shanghai`. +* If a legacy writer normalizes the value to UTC before storing `INT96`, the file may contain `02:11:00`. Set `hive.parquet.time-zone` to `Asia/Shanghai` so that Doris restores `10:11:00`. + +Configure the compatibility time zone when creating the Catalog: + +```sql +CREATE CATALOG hive_legacy_int96 PROPERTIES ( + 'type' = 'hms', + 'hive.metastore.uris' = 'thrift://127.0.0.1:9083', + 'hive.parquet.time-zone' = 'Asia/Shanghai' +); +``` + +The value can be an IANA time zone name, such as `Asia/Shanghai`, or a UTC offset from `-12:00` through `+14:00`. An invalid value causes Catalog creation to fail. + +:::caution +`hive.parquet.time-zone` applies to every `INT96` timestamp column mapped to `DATETIMEV2` in Hive and Hudi tables in the Catalog. It does not affect Iceberg tables in the same Hive Metastore because Iceberg defines its own timestamp semantics. Parquet metadata cannot reliably tell Doris which convention an `INT96` writer used. Do not set the property merely to match the SQL session time zone. If one Hive Metastore contains Hive or Hudi tables with `INT96` files written using different conventions, expose them through separate Catalogs when possible. +::: + +The same property is accepted by file table-valued functions that read Parquet files. The compatibility conversion applies only to Parquet `INT96` decoding in FileScannerV2. FileScannerV1 is intentionally unchanged and can return different results; switching to FileScannerV1 is not a compatibility mechanism for this change. Parquet `INT64` logical timestamps are not affected. When an `INT96` column maps to `TIMESTAMPTZ`, Doris ignores this property and preserves the UTC instant; displaying that value still follows `TIMESTAMPTZ` and session-time-zone semantics. + +### Migration recommendations + +* Doris writes Parquet `INT64` logical timestamps by default. Set `enable_int96_timestamps = true` only when a legacy reader such as Hive 2 or Hive 3 requires `INT96`. +* When upgrading to a version with this FileScannerV2 behavior, check existing Hive and Hudi Catalogs before rollout. A Catalog containing `INT96` files that were normalized by a legacy writer can return a different wall-clock value after the upgrade. Compare representative values and set `hive.parquet.time-zone` to the writer time zone when conversion is required. +* Before setting `hive.parquet.time-zone`, inspect representative files and compare known timestamp values. An unnecessary setting introduces the same kind of offset that it is intended to correct. +* Keep files with different `INT96` writer conventions in separate Catalogs, or rewrite them to a consistent `INT64` encoding. +* When Doris explicitly exports `INT96` with `enable_int96_timestamps = true`, it normalizes timestamps using the export session time zone. To read the file back through FileScannerV2, set `hive.parquet.time-zone` to that export session time zone. See [Export Overview](../../data-operate/export/export-overview.md). + ## Column Type Mapping | Hive Type | Doris Type | Comment | diff --git a/docs/lakehouse/file-formats/orc.md b/docs/lakehouse/file-formats/orc.md index 63a8b3e8e0b3f..b64e01508dc8e 100644 --- a/docs/lakehouse/file-formats/orc.md +++ b/docs/lakehouse/file-formats/orc.md @@ -17,6 +17,18 @@ This document introduces the support for reading and writing ORC file formats in * Writing data during Export. * Writing data with Outfile. +## Timestamp Fractional-Second Precision + +ORC timestamps can store nanoseconds, while Doris `DATETIMEV2` and `TIMESTAMPTZ` support up to microseconds. FileScannerV2 rounds ORC fractional seconds to the nearest microsecond using round half up in Catalog scans, table-valued functions, and Broker Load: + +| ORC value | Doris value | +|---|---| +| `2024-01-01 00:00:00.123456499` | `2024-01-01 00:00:00.123456` | +| `2024-01-01 00:00:00.123456500` | `2024-01-01 00:00:00.123457` | +| `2024-01-01 00:00:00.999999500` | `2024-01-01 00:00:01.000000` | + +Both `DATETIMEV2` and `TIMESTAMPTZ` mappings use this rule, including a carry into the next second. Row decoding, statistics conversion, and predicate pushdown use matching boundaries. When ORC statistics do not have enough precision to represent a rounded boundary exactly, Doris widens the pruning boundary conservatively. If a timestamp predicate such as `!=` cannot be represented safely, Doris skips that search argument and evaluates the predicate on decoded rows. This can reduce pruning for that predicate, but prevents valid rows from being incorrectly skipped. + ## Supported Compression Formats * uncompressed diff --git a/docs/lakehouse/file-formats/parquet.md b/docs/lakehouse/file-formats/parquet.md index 48dc54ba429a7..d3ff574f790ff 100644 --- a/docs/lakehouse/file-formats/parquet.md +++ b/docs/lakehouse/file-formats/parquet.md @@ -17,6 +17,14 @@ This document introduces the support for reading and writing Parquet file format * Writing data during Export. * Writing data with Outfile. +## INT96 Timestamp Decoding + +Parquet `INT96` stores date and time fields without a time zone annotation. FileScannerV2 therefore preserves the raw wall-clock value by default instead of shifting it with the SQL session time zone. For example, a raw `2021-01-01 10:11:00` remains `10:11:00` when mapped to `DATETIMEV2`, including in Catalog scans, table-valued functions, and Broker Load. + +This behavior applies only to `INT96`. Parquet `INT64` values with a timestamp logical type continue to follow the logical-type semantics. If legacy Hive writers normalized `INT96` values with a known time zone, configure `hive.parquet.time-zone` in the [Hive Catalog](../catalogs/hive-catalog.mdx#timestamp-compatibility). External file table-valued functions also accept that property. Other FileScannerV2 entry points preserve the raw `INT96` wall-clock value. + +When an `INT96` column maps to `TIMESTAMPTZ`, Doris preserves the UTC instant rather than applying the compatibility time zone. + ## Supported Compression Formats * uncompressed diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/export/export-overview.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/export/export-overview.md index cc75b874bfb43..31f5056f5aa95 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/export/export-overview.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/export/export-overview.md @@ -143,7 +143,7 @@ Apache Doris 导出到 Parquet 文件格式时,会先将 Doris 内存数据转 | hll | binary | BYTE_ARRAY | | :::note -在 2.1.11 和 3.0.7 版本中,支持通过 `parquet.enable_int96_timestamps` 属性指定 Doris 的 `datetimev2` 类型在 Parquet 中是使用 `INT96` 还是 `INT64` 存储,默认为 `INT96`。`INT96` 在 Parquet 标准中已被废弃,仅用于兼容旧系统(如 Hive 4.0 之前的版本)。 +Parquet Export 和 Outfile 使用 `enable_int96_timestamps` 属性选择 Doris `DATETIMEV2` 值的物理编码。默认值为 `false`,写入带 Parquet 时间戳逻辑类型的 `INT64`。仅当 Hive 2、Hive 3 等下游旧版读取端要求时,才设置为 `true` 写入已废弃的 `INT96` 编码。异步 Export 会将该属性保存在导出作业中,并在作业执行时应用。如果 Doris 后续读取显式导出的 `INT96` 文件,请在 Hive Catalog 或文件表值函数中将 `hive.parquet.time-zone` 设置为导出会话时区,使 FileScannerV2 还原原始墙上时间。 ::: ## 相关文档 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/hive-catalog.mdx b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/hive-catalog.mdx index 3094ec39172e9..90cbedf174673 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/hive-catalog.mdx +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/hive-catalog.mdx @@ -74,6 +74,8 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES ( * `hive.staging_dir`:用于设置 Hive 写操作的临时 staging 根目录。写入过程中数据会先写入该目录,写入完成后再移动到最终的 Hive 表目录。该路径可以是相对于 Hive 表根目录的相对路径,也可以是绝对路径。默认值为 `/tmp/.doris_staging`。某些情况下,写入 Hive 表时会遇到 `/tmp` 目录和表目录分属不同 namespace,导致无法写入的问题,此时可以通过修改此参数,使用相对路径解决该问题。该参数自 4.0.3 版本开始支持。 + * `hive.parquet.time-zone`:FileScannerV2 读取 Hive 或 Hudi 表时,用于还原旧版 Parquet `INT96` 时间戳的兼容时区。默认值为空,此时 Doris 保留 `INT96` 中存储的日期和时间字段,不做时区转换。仅当 Catalog 中的 `INT96` 文件由写入端按已知时区进行过归一化时,才需要设置此属性。详情请参阅 [Parquet INT96 时间戳兼容性](#timestamp-compatibility)。 + * `{CommonProperties}` CommonProperties 部分用于填写通用属性。请参阅[ 数据目录概述 ](../catalog-overview.md)中【通用属性】部分。 @@ -224,6 +226,50 @@ ORDER BY entry_name; * [ORC](../file-formats/orc.md) +## Parquet INT96 时间戳兼容性 {#timestamp-compatibility} + +Hive 表中的时间戳文件可能由不同引擎生成。是否需要时区转换取决于物理编码和写入端行为。Parquet `INT96` 本身不包含时区标注,因此 Doris 不会根据 `INT96` 值推断写入端时区。 + +### Parquet INT96 时间戳 + +FileScannerV2 按以下规则处理 Parquet 时间戳: + +| 物理编码 | 默认行为 | `hive.parquet.time-zone` 的作用 | +|---|---|---| +| 映射到 `DATETIMEV2` 的 `INT96` | 保留文件中存储的日期和时间字段,SQL 会话时区不会使该值发生偏移。 | 将原始 `INT96` 值视为 UTC 时刻,再转换到配置的兼容时区,从而还原被旧版写入端归一化过的时间戳。 | +| 带时间戳逻辑类型的 `INT64` | 遵循 Parquet 逻辑类型语义。 | 不受影响。 | + +例如,假设预期的本地时间戳是 `Asia/Shanghai` 时区的 `2021-01-01 10:11:00`: + +* 如果文件直接存储墙上时间字段 `10:11:00`,请不要设置 `hive.parquet.time-zone`。即使 SQL 会话使用 `Asia/Shanghai`,Doris 仍返回 `10:11:00`。 +* 如果旧版写入端先将该值归一化为 UTC 再写入 `INT96`,文件中可能存储 `02:11:00`。将 `hive.parquet.time-zone` 设置为 `Asia/Shanghai` 后,Doris 会还原出 `10:11:00`。 + +创建 Catalog 时配置兼容时区: + +```sql +CREATE CATALOG hive_legacy_int96 PROPERTIES ( + 'type' = 'hms', + 'hive.metastore.uris' = 'thrift://127.0.0.1:9083', + 'hive.parquet.time-zone' = 'Asia/Shanghai' +); +``` + +该值可以是 `Asia/Shanghai` 等 IANA 时区名称,也可以是 `-12:00` 到 `+14:00` 范围内的 UTC 偏移。非法值会导致 Catalog 创建失败。 + +:::caution +`hive.parquet.time-zone` 作用于该 Catalog 中所有 Hive 和 Hudi 表内映射到 `DATETIMEV2` 的 `INT96` 时间戳列。该属性不影响同一 Hive Metastore 中的 Iceberg 表,因为 Iceberg 定义了自己的时间戳语义。Parquet 元数据无法可靠地告诉 Doris 某个 `INT96` 写入端使用了哪种约定。不要仅为了匹配 SQL 会话时区而设置此属性。如果同一个 Hive Metastore 中的 Hive 或 Hudi 表包含使用不同约定写入的 `INT96` 文件,应尽量通过不同 Catalog 分别访问。 +::: + +读取 Parquet 文件的文件表值函数也接受同名属性。该兼容转换仅影响 FileScannerV2 对 Parquet `INT96` 的解码。FileScannerV1 有意保持不变,可能返回不同结果;切换到 FileScannerV1 不是本次变更的兼容方案。Parquet `INT64` 逻辑时间戳不受影响。`INT96` 列映射到 `TIMESTAMPTZ` 时,Doris 会忽略该属性并保留 UTC 时刻;该值的显示仍遵循 `TIMESTAMPTZ` 和会话时区语义。 + +### 迁移建议 + +* Doris 默认写入带逻辑类型的 Parquet `INT64` 时间戳。仅当 Hive 2、Hive 3 等旧版读取端要求 `INT96` 时,才设置 `enable_int96_timestamps = true`。 +* 升级到采用此 FileScannerV2 行为的版本前,请先检查现有 Hive 和 Hudi Catalog。包含由旧版写入端归一化过的 `INT96` 文件时,升级后返回的墙上时间可能发生变化。请比对有代表性的已知值,并在需要转换时将 `hive.parquet.time-zone` 设置为写入端时区。 +* 设置 `hive.parquet.time-zone` 前,请检查有代表性的文件并与已知时间戳值进行比对。不必要的配置反而会产生原本希望消除的时间偏移。 +* 对使用不同 `INT96` 写入约定的文件,使用不同 Catalog 分别访问,或将文件重写为统一的 `INT64` 编码。 +* Doris 通过 `enable_int96_timestamps = true` 显式导出 `INT96` 时,会使用导出会话时区归一化时间戳。通过 FileScannerV2 读回该文件时,请将 `hive.parquet.time-zone` 设置为相同的导出会话时区。详情请参阅[数据导出概述](../../data-operate/export/export-overview.md)。 + ## 列类型映射 | Hive Type | Doris Type | Comment | diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/file-formats/orc.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/file-formats/orc.md index 7e2bfa5b36c0d..91230cc017a40 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/file-formats/orc.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/file-formats/orc.md @@ -21,6 +21,18 @@ * Outfile 中对数据的写入操作。 +## 时间戳小数秒精度 + +ORC 时间戳可以保存纳秒,而 Doris `DATETIMEV2` 和 `TIMESTAMPTZ` 最高支持微秒。FileScannerV2 在 Catalog 扫描、表值函数和 Broker Load 中使用四舍五入规则,将 ORC 小数秒舍入到最接近的微秒: + +| ORC 值 | Doris 值 | +|---|---| +| `2024-01-01 00:00:00.123456499` | `2024-01-01 00:00:00.123456` | +| `2024-01-01 00:00:00.123456500` | `2024-01-01 00:00:00.123457` | +| `2024-01-01 00:00:00.999999500` | `2024-01-01 00:00:01.000000` | + +`DATETIMEV2` 和 `TIMESTAMPTZ` 映射均使用该规则,包括向下一秒进位。行解码、统计信息转换和谓词下推使用一致的边界。当 ORC 统计信息的精度不足以准确表示舍入边界时,Doris 会保守地扩大裁剪边界。如果 `!=` 等时间戳谓词无法安全表示,Doris 会跳过对应的搜索条件,并在解码后的数据行上计算谓词。这可能降低该谓词的裁剪效果,但可以避免错误跳过有效数据。 + ## 支持的压缩格式 * umcomressed diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/file-formats/parquet.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/file-formats/parquet.md index 2c029d1c5cfb3..5a5830e690de8 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/file-formats/parquet.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/file-formats/parquet.md @@ -21,6 +21,14 @@ * Outfile 中对数据的写入操作。 +## INT96 时间戳解码 + +Parquet `INT96` 存储日期和时间字段,但不包含时区标注。因此 FileScannerV2 默认保留原始墙上时间值,而不会根据 SQL 会话时区产生偏移。例如,映射到 `DATETIMEV2` 时,原始值 `2021-01-01 10:11:00` 在 Catalog 扫描、表值函数和 Broker Load 中均保持为 `10:11:00`。 + +该行为仅适用于 `INT96`。带时间戳逻辑类型的 Parquet `INT64` 值仍遵循逻辑类型语义。如果旧版 Hive 写入端按已知时区归一化过 `INT96` 值,请在 [Hive Catalog](../catalogs/hive-catalog.mdx#timestamp-compatibility) 中配置 `hive.parquet.time-zone`。外部文件表值函数也接受该属性。其他 FileScannerV2 入口会保留原始 `INT96` 墙上时间值。 + +`INT96` 列映射到 `TIMESTAMPTZ` 时,Doris 会保留 UTC 时刻,而不应用兼容时区。 + ## 支持的压缩格式 * umcomressed