Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/help/describegpt.md
Original file line number Diff line number Diff line change
Expand Up @@ -226,7 +226,7 @@ qsv describegpt --help
|--------|------|-------------|--------|
|  `‑‑num‑examples`  | integer | The number of Example values to include in the dictionary. | `5` |
|  `‑‑truncate‑str`  | integer | The maximum length of an Example value in the dictionary. An ellipsis is appended to the truncated value. If zero, no truncation is performed. | `25` |
| &nbsp;`‑‑infer‑content‑type`&nbsp; | flag | Also have the LLM classify each field's semantic "Content Type", mapped to a curated, documented vocabulary (e.g. email, city, category, name, credit card, etc.) see <https://github.com/dathere/qsv/blob/master/src/cmd/synthesize/faker_map.rs>. Adds a "Content Type" column/field to the Data Dictionary output. Fields where cardinality equals the row count (i.e. every row has a distinct non-null value - primary keys, surrogate keys, sequence numbers) are deterministically classified as "unique_id", overriding any token the LLM returned for that field. For Date/DateTime fields, the LLM also infers the column's strftime date format (e.g. "date:%m/%d/%Y"); the Markdown, JSON & JSON Schema dictionaries then render Min/Max AND Examples in that inferred format so they match how the dates actually appear in the data, instead of qsv's normalized form. (TSV output keeps Min/Max & Examples in qsv's raw normalized form.) | |
| &nbsp;`‑‑infer‑content‑type`&nbsp; | flag | Also have the LLM classify each field's semantic "Content Type", mapped to a curated, documented vocabulary (e.g. email, city, category, name, credit card, etc.) see <https://github.com/dathere/qsv/blob/master/src/cmd/synthesize/faker_map.rs>. Adds a "Content Type" column/field to the Data Dictionary output. Fields where cardinality equals the row count (i.e. every row has a distinct non-null value - primary keys, surrogate keys, sequence numbers) are deterministically classified as "unique_id", overriding any token the LLM returned for that field. For Date/DateTime fields, the LLM also infers the column's strftime date format (e.g. "date:%m/%d/%Y"); the Markdown, JSON & JSON Schema dictionaries then render Min/Max AND Examples in that inferred format so they match how the dates actually appear in the data, instead of qsv's normalized form. (TSV output keeps Min/Max & Examples in qsv's raw normalized form.) For a CONTINUOUS numeric measure on a canonical scale (a percentage, a 0-1 ratio/probability, a bounded index), the LLM also proposes an "x-qsv.gauge_range" [min, max]; qsv keeps it only when the field is a numeric measure AND the observed data lies within it, so that a "viz smart" dictionary-driven dashboard draws that KPI tile as a GAUGE. (A KPI "vs target" delta uses "x-qsv.target", which is a GOAL you hand-author - never inferred.) | |
| &nbsp;`‑‑infer‑null‑values`&nbsp; | flag | Also have the LLM propose each field's null sentinels - literal values that stand in for "missing" (e.g. NULL, N/A, -999, 9999-12-31). Emitted into the JSON Schema dictionary's per-property "x-qsv" object, split into two lists that carry different warranties.<ul><li>"null_values" - proposed by the LLM AND independently confirmed by qsv to occur as a literal value in that String column. Every observed casing is listed.</li><li>"null_candidates" - proposed by the LLM but NOT confirmable by any scan; numeric/date placeholders that parse as valid values of the column's own type (-999 is a legal integer), or tokens never seen in the data. Each carries "confirm_required": true.</li></ul> A proposal that is IMPOSSIBLE is dropped rather than reported: a non-numeric token in a numeric column (no cell could hold it, or the column would not have typed as numeric), and an echo of the null placeholder the Frequency Distribution displays for cells qsv already counts as empty. A confirmed "null_values" entry means only that the literal is PRESENT in the column; the judgment that it MEANS "missing" remains the LLM's. This complements "qsv denull", which reports a narrower set - only the columns that would promote to a numeric type once their sentinels are blanked. A purely categorical column (status = ok/pending/NULL) is outside denull's remit. Sentinels are REPORTED, never applied - qsv does not modify your data on an LLM's say-so. Only "qsv denull --apply" masks values, and only in the columns it independently confirmed. Only rendered by "--format JSONSchema"; the other formats ignore them. | |
| &nbsp;`‑‑two‑pass`&nbsp; | flag | Run a second LLM call that takes the full first-pass Data Dictionary as JSON context and refines each field's Label, Description and (when --infer-content-type is set) Content Type using cross-field awareness. The LLM can then relate fields that belong together (e.g. street_no + street_name + city + state + zip describing a single mailing address; first_name + last_name naming a single person; lat + lng forming a coordinate pair). The refined dictionary becomes the emitted output and is also what downstream Description, Tags and Prompt inference phases see as dictionary context. Roughly doubles dictionary LLM cost and latency, so opt-in. Most useful when combined with --infer-content-type. Allowed with the --dictionary, --all and --prompt inference flags. Mutually exclusive with --prepare-context and --process-response (MCP sampling is single-turn per inference phase). | |
| &nbsp;`‑‑addl‑cols`&nbsp; | flag | Add additional columns to the dictionary from the Summary Statistics. | |
Expand Down
11 changes: 10 additions & 1 deletion docs/help/viz.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,15 @@ points overlaid by a size heuristic, see --box-points)
- ID-like (near-unique) and all-empty columns are skipped

Overview panels (each leads the dashboard on its own full-width row):
- KPI overview row (leads the dashboard when the dataset has headline numeric
measures): a strip of "big number" tiles, one per headline measure (summed
for extensive quantities, averaged for intensive ones). A measure tile
becomes a GAUGE when the dictionary supplies a validated `x-qsv.gauge_range`
that contains the value, and gains a "vs target" DELTA when it supplies an
`x-qsv.target` (see --dictionary). Omitted for image exports and for datasets
with no headline measure. (Overall dataset completeness - the share of
non-empty cells - is a quiet "Completeness:" line in the header metadata
table, not a KPI tile.)
- correlation heatmap, when 2+ continuous numeric columns exist (one extra
data pass for Pearson correlations). If the strongest pair is at least
moderately correlated, a drill-down is added beside it: a scatter (or a 2D
Expand Down Expand Up @@ -397,7 +406,7 @@ qsv viz --help
| &nbsp;`‑‑no‑other`&nbsp; | flag | Omit the "Other (N)" aggregate bar from frequency bar charts. It collects the categories beyond --limit (N = how many distinct categories were rolled up) and is shown by default. | |
| &nbsp;`‑‑smarter`&nbsp; | flag | Before building the dashboard, run `qsv moarstats --advanced` to enrich the stats cache with distribution-shape statistics (bimodality, entropy, skewness, outlier share). This unlocks histograms for bimodal columns, frequency bars for concentrated high-cardinality columns, and skew/outlier hints on box panels. Costs one extra pass over the data and writes <stem>.stats.csv, its sidecars, and an .idx index (like running `qsv moarstats` manually). On geocode-enabled builds it also enriches map point hovers with the US FIPS code and annotates the spatial-extent summary with the country's continent; the county is always shown in map hovers, with or without --smarter. Only affects `smart`. Applied only with default parsing; inputs using --no-headers or a custom --delimiter fall back to the standard dashboard. | |
| &nbsp;`‑‑hierarchy‑style`&nbsp; | string | For `smart`, the chart used for the categorical part-to-whole hierarchy panel (built when 2+ low-cardinality dimensions exist). One of: auto (default), treemap, sunburst, icicle. auto follows best practice — a treemap for a shallow 2-level hierarchy (accurate size comparison) and a sunburst for a deep 3-level one (parent child structure); icicle is an opt-in level-aligned alternative. Only affects `smart`. | |
| &nbsp;`‑‑dictionary`&nbsp; | string | Use a describegpt Data Dictionary to guide panel selection from each field's semantic role/concept (falling back to its content type) instead of relying on column statistics alone: dimensions and numeric codes (ward, census_tract, zone) become bars, measures get box/correlation/trend panels, date/datetime columns feed the time-series panel (not noisy frequency bars), identifiers / PII / free-text are skipped, and lat/lon feed the map. Field labels are shown as panel subtitles beneath the field-name titles. Columns the dictionary cannot classify still use the statistical heuristic. <src> is one of: "infer" to run describegpt on the input now (with description, infer-content-type, two-pass and jsonschema output; requires an LLM configured) and use its output; or a path to an existing describegpt dictionary file (jsonschema or json). With "infer", the generated dictionary is saved beside the input as <stem>.schema.json so you can fine-tune it; if that file already exists, it is reused as-is (skipping the LLM) - edit it to fine-tune, or delete it to force a fresh re-infer. Generation/read failures soft-fall back to the stats-only dashboard. Only affects `smart`. | |
| &nbsp;`‑‑dictionary`&nbsp; | string | Use a describegpt Data Dictionary to guide panel selection from each field's semantic role/concept (falling back to its content type) instead of relying on column statistics alone: dimensions and numeric codes (ward, census_tract, zone) become bars, measures get box/correlation/trend panels, date/datetime columns feed the time-series panel (not noisy frequency bars), identifiers / PII / free-text are skipped, and lat/lon feed the map. Field labels are shown as panel subtitles beneath the field-name titles. Columns the dictionary cannot classify still use the statistical heuristic. <src> is one of: "infer" to run describegpt on the input now (with description, infer-content-type, two-pass and jsonschema output; requires an LLM configured) and use its output; or a path to an existing describegpt dictionary file (jsonschema or json). With "infer", the generated dictionary is saved beside the input as <stem>.schema.json so you can fine-tune it; if that file already exists, it is reused as-is (skipping the LLM) - edit it to fine-tune, or delete it to force a fresh re-infer. Generation/read failures soft-fall back to the stats-only dashboard. The dictionary also drives the KPI overview row via two optional per-field hints in a property's "x-qsv" object (edit them in the saved schema to fine-tune). A "gauge_range" of [min, max] on a continuous numeric measure renders its KPI tile as a GAUGE on that canonical scale (e.g. [0,1] for a ratio, [0,100] for a percent); qsv keeps it only when the observed data lies within the range, so a mis-scaled range can't draw a misleading dial, and "infer" emits it for canonical-scale measures. A "target" number on a measure renders a "vs target" DELTA against that goal (value minus target) - a GOAL you supply, never a fabricated prior-period baseline, so "infer" never emits it; hand-author it. Only affects `smart`. | |
| &nbsp;`‑‑dictionary‑context`&nbsp; | string | Path to a file with extra context about the dataset (a glossary, README, data dictionary, PDF, etc.) forwarded to describegpt as --context-file when `--dictionary infer` generates the dictionary. Better context yields better role/concept/label/grain tags, hence a better dashboard. Ignored unless `--dictionary infer` is used (it does not apply when reading an existing dictionary file). Only affects `smart`. | |
| &nbsp;`‑‑dict‑info`&nbsp; | flag | When a usable Data Dictionary is available (per --dictionary), add a "Data Dictionary" link beneath the dashboard title and an info icon on each panel title: hovering shows that column's dictionary description; clicking opens a human-friendly rendering of the dictionary in a side drawer NEXT TO the plots (embedded in the dashboard file - no extra file is written), scrolled to and highlighting that column's entry. The drawer is open by default on load and can be dismissed with its close button or Esc. The dictionary page carries a role-tinted table of contents and per-column "View chart" links back to the panels. The drawer's popout button opens the same document in its own browser tab instead (needs a browser that allows user-initiated pop-ups). HTML output only; ignored with a note when no dictionary is available or when exporting an image. Only affects `smart`. | |
| &nbsp;`‑‑dataset‑pid`&nbsp; | string | A persistent identifier (PID) for the dataset - typically a full URL such as a DOI (<https://doi.org/10.1234/abc>) or other citable link. When set, a clickable "PID" row is added to the metadata table at the top of the dashboard, using the value verbatim as the link target. HTML output only. Only affects `smart`. | |
Expand Down
Loading
Loading