Skip to content

Legacy trader count #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mspinola merged 2 commits into from
Jul 14, 2026
Merged

Legacy trader count #7

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
51270a1
feat: add Trader counts to Legacy dataset
mspinola Jul 14, 2026
7e461b4
docs: update Legacy schema in README to include trader count columns
mspinola Jul 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@ The primary source for Legacy positioning data (CFTC Legacy Futures Report). **H
> **Legacy Reports**: The Legacy reports are broken down by exchange. These reports have a futures only report and a combined futures and options report. Legacy reports break down the reportable open interest positions into two classifications: non-commercial and commercial traders. The `cotdata` pipeline strictly downloads the **Futures-only** reports (located at `https://www.cftc.gov/files/dea/history/dea_fut_xls_{YEAR}.zip`).

> [!NOTE]
> **Column Subset**: While the raw CFTC `.xls` files contain [well over 100 columns](https://www.cftc.gov/MarketReports/CommitmentsofTraders/HistoricalViewable/cotvariableslegacy.html) (including spreading, concentration ratios, etc.), the producer pipeline explicitly discards them. The parquet files only maintain the exact 10-column subset listed below to keep the file sizes extremely small and strictly focused on what the downstream models require. To include additional data points from the raw reports, simply add the exact CFTC column name to the `TARGET_COLS` list inside `src/cotdata/providers/cftc.py`.
> **Column Subset**: While the raw CFTC `.xls` files contain [well over 100 columns](https://www.cftc.gov/MarketReports/CommitmentsofTraders/HistoricalViewable/cotvariableslegacy.html) (including spreading, concentration ratios, etc.), the producer pipeline explicitly discards them. The parquet files only maintain the exact 15-column subset listed below to keep the file sizes extremely small and strictly focused on what the downstream models require. To include additional data points from the raw reports, simply add the exact CFTC column name to the `TARGET_COLS` list inside `src/cotdata/providers/cftc.py`.

| Column | Type | Description |
|--------|------|-------------|
Expand All @@ -200,6 +200,11 @@ The primary source for Legacy positioning data (CFTC Legacy Futures Report). **H
| `NonComm_Positions_Short_All` | float | Non-Commercial (Large Speculator) Short positions. |
| `NonRept_Positions_Long_All` | float | Non-Reportable (Small Speculator) Long positions. |
| `NonRept_Positions_Short_All` | float | Non-Reportable (Small Speculator) Short positions. |
| `Traders_Tot_All` | float | Total number of reportable traders. |
| `Traders_Comm_Long_All` | float | Number of Commercial Long traders. |
| `Traders_Comm_Short_All` | float | Number of Commercial Short traders. |
| `Traders_NonComm_Long_All` | float | Number of Non-Commercial Long traders. |
| `Traders_NonComm_Short_All` | float | Number of Non-Commercial Short traders. |

### COT Disaggregated Data (`cot_disagg/{code}.parquet`)
The primary source for entity-specific positioning and trader counts (CFTC Disaggregated Futures-Only Report). **History starts in 2006.** Indexed by tz-naive `Report_Date_as_MM_DD_YYYY`.
Expand Down
10 changes: 9 additions & 1 deletion src/cotdata/providers/cftc.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,17 @@
NONREPT_LONG = "NonRept_Positions_Long_All"
NONREPT_SHORT = "NonRept_Positions_Short_All"

TRADERS_TOT = "Traders_Tot_All"
TRADERS_COMM_LONG = "Traders_Comm_Long_All"
TRADERS_COMM_SHORT = "Traders_Comm_Short_All"
TRADERS_NONCOMM_LONG = "Traders_NonComm_Long_All"
TRADERS_NONCOMM_SHORT = "Traders_NonComm_Short_All"

TARGET_COLS = [MARKET_NAME, REPORT_DATE, CONTRACT_CODE, OPEN_INTEREST,
COMM_LONG, COMM_SHORT, NONCOMM_LONG, NONCOMM_SHORT,
NONREPT_LONG, NONREPT_SHORT]
NONREPT_LONG, NONREPT_SHORT,
TRADERS_TOT, TRADERS_COMM_LONG, TRADERS_COMM_SHORT,
TRADERS_NONCOMM_LONG, TRADERS_NONCOMM_SHORT]


def _cache_dir() -> Path:
Expand Down
Loading