The benchmark has a time drift problem: several gold SQL pipelines depend on runtime values such as current_date, current_timestamp, now(), randomness, and execution order. Unless date, timestamp, seed, and execution determinism are fixed, rerunning the gold SQL may not reproduce the same CFS outputs.
This means a gold run.py rerun can only be expected to score 100% CFS against a reference that was built with the same fixed date/time/seed settings. It should not necessarily score 100% against the shipped gold .duckdb tables, because those shipped tables appear to encode variable task-specific bake dates.
Why This Affects Benchmark Validity
Some expected outputs are not stable business facts. They are artifacts of when the SQL happened to run.
For example, in dacomp-de-evol-022:
sql/intermediate/int_hubspot__contact_lifecycle_metrics.sql
datediff('day', c.created_date, current_date) as contact_age_days,
case
when datediff('day', c.created_date, current_date) < 30 then 'New Contact'
when datediff('day', c.created_date, current_date) < 90 then 'Recent Contact'
when datediff('day', c.created_date, current_date) < 365 then 'Established Contact'
else 'Mature Contact'
end as contact_maturity_segmentLets suppose the raw contacts are from 2021 (fixed) and the benchmark is run with an as-of date in 2025, most or all contacts become Mature Contact. An agent could arrive at the same output through an overly broad or even incorrect SQL transformation, and CFS would still mark it correct because it matches the time-specific snapshot rather than the intended transformation logic.
Another example is dacomp-de-evol-004:
sql/intermediate/int_pendo__real_time_integration_health.sql
where cast(substr(occurred_at, 1, 19) as timestamp) >= current_timestamp - interval '24 hours'
and later:
datediff('minute', max(re.event_timestamp), current_timestamp) as minutes_since_latest_eventThis model changes depending on the exact timestamp used during execution. A different current_timestamp can turn the same raw data into a non-empty real-time health table, an empty table, or different latency/status classifications. That makes the expected answer dependent on benchmark runtime rather than only on the task data.
A third example is date spine generation, such as dacomp-de-evol-007:
sql/intermediate/int_salesforce__date_spine.sql
min(created_date::date) as min_date,
current_date as max_date
The number of rows in the date spine changes every day unless current_date is fixed. In practice, shipped gold tables can have a different max date than a fresh deterministic rebuild, so shipped gold cannot reliably serve as a reproducible target unless the original bake date is known and reused.
Observed Behavior
After fixing date/time/seed settings, fresh gold SQL reruns become reproducible against a newly rebuilt reference snapshot. However, they do not necessarily match the shipped gold .duckdb tables, because the shipped tables were likely generated with different as-of dates/timestamps across tasks.
This creates two problems:
- Gold reproducibility depends on hidden runtime state.
- CFS can reward SQL that reproduces snapshot artifacts, even when the SQL does not reflect the intended business logic.
Expected Invariant
For a valid deterministic benchmark:
gold SQL + fixed raw data + fixed date/time/seed
=
same gold tables on every rerun
Only under that invariant is CFS a meaningful measure of whether an agent reproduced the intended transformation.
Could you please clarify whether this understanding of the evaluation setup is correct? I would also appreciate any guidance if there is a gap in this interpretation or any important context I may be missing.
The benchmark has a time drift problem: several gold SQL pipelines depend on runtime values such as
current_date,current_timestamp,now(), randomness, and execution order. Unless date, timestamp, seed, and execution determinism are fixed, rerunning the gold SQL may not reproduce the same CFS outputs.This means a gold
run.pyrerun can only be expected to score 100% CFS against a reference that was built with the same fixed date/time/seed settings. It should not necessarily score 100% against the shipped gold.duckdbtables, because those shipped tables appear to encode variable task-specific bake dates.Why This Affects Benchmark Validity
Some expected outputs are not stable business facts. They are artifacts of when the SQL happened to run.
For example, in
dacomp-de-evol-022:sql/intermediate/int_hubspot__contact_lifecycle_metrics.sqlLets suppose the raw contacts are from 2021 (fixed) and the benchmark is run with an as-of date in 2025, most or all contacts become
Mature Contact. An agent could arrive at the same output through an overly broad or even incorrect SQL transformation, and CFS would still mark it correct because it matches the time-specific snapshot rather than the intended transformation logic.Another example is
dacomp-de-evol-004:sql/intermediate/int_pendo__real_time_integration_health.sqland later:
This model changes depending on the exact timestamp used during execution. A different
current_timestampcan turn the same raw data into a non-empty real-time health table, an empty table, or different latency/status classifications. That makes the expected answer dependent on benchmark runtime rather than only on the task data.A third example is date spine generation, such as
dacomp-de-evol-007:sql/intermediate/int_salesforce__date_spine.sqlThe number of rows in the date spine changes every day unless
current_dateis fixed. In practice, shipped gold tables can have a different max date than a fresh deterministic rebuild, so shipped gold cannot reliably serve as a reproducible target unless the original bake date is known and reused.Observed Behavior
After fixing date/time/seed settings, fresh gold SQL reruns become reproducible against a newly rebuilt reference snapshot. However, they do not necessarily match the shipped gold
.duckdbtables, because the shipped tables were likely generated with different as-of dates/timestamps across tasks.This creates two problems:
Expected Invariant
For a valid deterministic benchmark:
Only under that invariant is CFS a meaningful measure of whether an agent reproduced the intended transformation.
Could you please clarify whether this understanding of the evaluation setup is correct? I would also appreciate any guidance if there is a gap in this interpretation or any important context I may be missing.