This section outlines the workflow for processing data, building the ground truth, and running evaluations.
Use download_hf_dataset.py to pull the newest librarian-bots/model_cards_with_metadata or librarian-bots/dataset_cards_with_metadata parquet shards into a date-tagged folder (for example, data/raw_251117). The script automatically enumerates the parquet shards available on Hugging Face Hub and stores them locally so downstream steps can point to a specific snapshot when re-running the pipeline.
Important:
- Model cards and dataset cards are stored in the same
data/raw_<date>directory but with different filename prefixes to avoid conflicts:- Modelcard files:
train-*.parquet(no prefix, original format) - Datasetcard files:
datasetcard-train-*.parquet(with prefix to avoid conflict)
- Modelcard files:
- Both can use the same tag (e.g.,
251117) to keep them synchronized - The old
data/rawdirectory (without date tag) maintains the original format for backward compatibility
mkdir logs
# Download modelcards/datasetcards dataset
python -m src.data_preprocess.download_hf_dataset --date 251117 --type modelcard/datasetcard > logs/download_hf_dataset_251117.log 2>&1This step extracts key metadata from model cards and associated links.
# Split readme and tags, parse URLs, parse BibTeX entries.
# Output: ['modelId', 'author', 'last_modified', 'downloads', 'likes', 'library_name', 'tags', 'pipeline_tag', 'createdAt', 'card', 'card_tags', 'card_readme', 'pdf_link', 'github_link', 'all_links', 'extracted_bibtex', 'extracted_bibtex_tuple', 'parsed_bibtex_tuple_list', 'successful_parse_count']
#python -m src.data_preprocess.step1_parse --raw-date 251117 --versioning --baseline-step1 data/processed/modelcard_step1.parquet > logs/step1_parse_251117.log 2>&1 # incremental mode, based on the previous step1 result
# or
python -m src.data_preprocess.step1_parse --raw-date 251117 > logs/step1_parse_251117.log 2>&1 # output: modelcard_step1_251117.parquet
python -m src.data_preprocess.step1_down_giturl --tag 251117 --versioning --baseline-cache data/processed/github_readme_cache.parquet > logs/step1_down_giturl_251117.log 2>&1 # Download GitHub READMEs; Input: modelcard_step1_251117.parquet. Download: only new files to data/downloaded_github_readmes_251117/. Output: (1) dir data/downloaded_github_readmes_251117/, (2) github_readmes_info_251117.parquet, (3) github_readme_cache_251117.parquet. All saved paths in (2)(3) are unified as data/downloaded_github_readmes_251117/ (reused from baseline are not re-downloaded; run ln_giturl to symlink baseline into this dir).
python -m src.data_preprocess.ln_giturl --source-dir data/downloaded_github_readmes --target-dir data/downloaded_github_readmes_251117 > logs/ln_giturl_251117.log 2>&1 # Symlink all .md from source into target; skip if name already in target. In case that we try to analyze github readmes folder in the future
# (Optional) find data/downloaded_github_readmes -type f -exec stat -f "%z %N" {} + | sort -nr | head -n 50 | awk '{printf "%.2f MB %s\n", $1/1024/1024, $2}' > logs/find_large_readmes.log 2>&1 # some readme files are too large, they are actually model filesThis step extracts tabular data from various sources and processes it.
# Extract tables from Hugging Face Model Cards and GitHub READMEs. Saves CSVs to local folder.
# Versioning mode (with tag):
# Process downloaded GitHub HTML files to Markdown. Skips when output file already exists (e.g. ln -s into _processed).
# Input: data/downloaded_github_readmes_<tag>/
# Output: data/downloaded_github_readmes_<tag>_processed/, data/processed/md_parsing_results_v2_<tag>.parquet
python -m src.data_preprocess.step2_git_md2text --tag 251117 > logs/step2_git_md2text_251117.log 2>&1
# Extract tables from Hugging model cards + GitHub READMEs. Input: modelcard_step1, github_readmes_info, downloaded_github_readmes_<tag>/ (not _processed).
# Input: data/processed/modelcard_step1_<tag>.parquet, github_readmes_info_<tag>.parquet, downloaded_github_readmes_<tag>/
# Output: data/processed/modelcard_step2_v2_<tag>.parquet, data/processed/deduped_hugging_csvs_v2_<tag>/, data/processed/hugging_deduped_mapping_v2_<tag>.json, data/processed/deduped_github_csvs_v2_<tag>/, md_to_csv_mapping.json
############################################### Here we only keep v2 version for extracting table as this is more accurate; see v1 extracting, check the previous packaged version on github
python -m src.data_preprocess.step2_hugging_github_extract --tag 251117 > logs/step2_hugging_github_extract_251117.log 2>&1
# Extract titles from arXiv and GitHub URLs (not S2ORC). For BibTeX entries and PDF URLs.
# Input: modelcard_step1_<tag>.parquet, github_readme_cache_<tag>.parquet, downloaded_github_readmes_<tag>_processed/, PDF/GitHub URLs
# Output: modelcard_all_title_list_<tag>.parquet, all_title_list_intra_row_dedup_groups_<tag>.json
# (Output but not used anymore) github_readme_cache_update_<tag>.parquet, github_extraction_cache_<tag>.json, all_links_with_category_<tag>.csv
python -m src.data_preprocess.step2_arxiv_github_title --tag 251117 > logs/step2_arxiv_github_title_251117.log 2>&1 # This one is slow..
# (One-time fix without run all: PYTHONPATH=. python bak/dedup_all_title_list_intra_row_251117.py)
# Save deduplicated titles for querying Semantic Scholar (S2ORC). Cross-row dedup: same normalize. Output: modelcard_dedup_titles_<tag>.json, s2orc_cross_row_dedup_groups_<tag>.json
python -m src.data_preprocess.step2_s2orc_save --tag 251117 > logs/step2_s2orc_save_251117.log 2>&1
# non-main pipeline scripts are documented in `docs/depre_scripts.md`.
#### Option1:
# save some searched results, only search the missing titles
#cp -r data/processed/s2orc_titles2ids.parquet data/processed/s2orc_titles2ids_251117.parquet
#cp -r data/processed/s2orc_references_cache.parquet data/processed/s2orc_references_cache_251117.parquet
# Query Semantic Scholar API for citation information (alternative to local database if no key, but may hit rate limits).
# Why API over local (build_mini_citation_es): (1) API provides fresher citations/references; (2) API's title fuzzy matching is more accurate (commercialized) than our local ES fuzzy match.
python -m src.data_preprocess.s2orc_title2ids_API --tag 251117 > logs/s2orc_title2ids_API_251117.log 2>&1 # Input: modelcard_dedup_titles_<tag>.json Output: s2orc_titles2ids_<tag>.parquet
python -m src.data_preprocess.s2orc_refcit_API --tag 251117 > logs/s2orc_refcit_API_251117.log 2>&1 # Input: s2orc_titles2ids_<tag>.parquet Output: s2orc_references_cache_<tag>.parquet
# Download arXiv HTML, extract tables from arXiv HTML files.
#python -m bak.analyze_bibtex_arxiv_ids --tag 251117 > logs/analyze_bibtex_arxiv_ids_251117.log 2>&1 # Input: s2orc_titles2ids_<tag>.parquet, modelcard_all_title_list_<tag>.parquet, Output: bibte_title_arxiv_s2orc_<tag>.parquet # try saving some title:arxiv from bibtex
# Resolve title→arxiv_id. Init: s2orc + arxiv_titles_cache concat; then bibtex + OAI rescue; sync html_path from folder.
# Input: s2orc_titles2ids_<tag>.parquet (query_title, retrieved_title), arxiv_titles_cache_<tag>.json (url→title, init only)
# bibtex_title_arxiv_<tag>.parquet, title2arxiv_oai_index_<tag>.parquet
# Output: title2arxiv_cache_<tag>.parquet, final_missing_titles_from_cache_<tag>.txt
python -m src.data_preprocess.arxiv_title2ids_oai --tag 251117 > logs/arxiv_title2ids_oai_251117_5.log 2>&1
bash scripts/ln_arxiv_html.sh 251117 # ln data/arxiv_fulltext_html/*.html → data/arxiv_fulltext_html_<tag>/, save downloading
# Download HTML for arxiv_ids in cache. Input: title2arxiv_cache_<tag>.parquet. Output: data/arxiv_fulltext_html_<tag>/*.html (no parquet write)
python -m src.data_preprocess.arxiv_fulltext_api --tag 251117 > logs/arxiv_fulltext_api_251117.log 2>&1
# run python -m src.data_preprocess.arxiv_title2ids_oai --tag 251117 > logs/arxiv_title2ids_oai_251117_4.log 2>&1 again to sync html_path from folder to cache.
# Extract tables from arXiv HTML files (v2: rowspan/colspan, ltx_table).
# Input: arxiv_fulltext_html_<tag>/*.html. Output: tables_output_v2_<tag>/*.csv, html_parsing_results_v2_<tag>.parquet
# Incremental by default (skips paper_ids already in parquet). Use --overwrite for full reprocess.
###############################################
# we don't ln v1 to v2, because we change parsing logic
python -m src.data_preprocess.step2_arxiv_parse_v2 --n_jobs 16 --tag 251117 --save_mode csv > logs/step2_arxiv_parse_v2_251117.log 2>&1 # --overwrite for full run; save_mode: csv|duckdb Finally, we merge table list from different sources back to modelID level.
To substitute step2_integration_s2orc_llm, step2_llm_save (we skip llm tables as it is unstable), step2_merge_tables, we can use step2_merge_tables_simplify to directly generate the final merged table list at modelID level (without running the LLM table extraction pipeline).
python -m src.data_preprocess.step2_merge_tables_simplify --tag 251117 --v2_mode > logs/step2_merge_tables_simplify_251117.log 2>&1
# Input: s2orc_titles2ids_<tag>.parquet, title2arxiv_cache_<tag>.parquet, modelcard_all_title_list_<tag>.parquet,
# hugging_deduped_mapping_v2_<tag>.json, deduped_github_csvs_v2_<tag>/md_to_csv_mapping.json. html_parsing_results_v2_<tag>.parquet, modelcard_step2_v2_<tag>.parquet,
# Output: modelcard_step3_merged_v2_<tag>.parquetEnsure data quality and consistency before generating final ground truth.
# Umm, dedup better happen before merge, maybe in the future TODO
python -m src.data_preprocess.step2_dedup_tables --tag 251117 --v2_mode > logs/step2_dedup_tables_v2_251117.log 2>&1 # Deduplicate raw tables, prioritizing Hugging Face > GitHub > HTML > LLM. Input: modelcard_step3_merged_v2_<tag>.parquet. Output: modelcard_step3_dedup_v2_<tag>.parquet, and others
python -m src.data_analysis.qc_dedup_fig --tag 251117 --v2_mode > logs/qc_dedup_fig_v2_251117.log 2>&1 # Generate heatmaps from dedup results. Input: deduped_v2_<tag>/dup_matrix_v2_<tag>.pkl, deduped_v2_<tag>/stats_v2_<tag>.json. Output: heatmaps heatmap_overlap_v2_<tag>.pdf / heatmap_percentage_v2_<tag>.pdf in data/analysis/
PYTHONUNBUFFERED=1 python -m src.data_analysis.qc_stats --tag 251117 --v2_mode > logs/qc_stats_v2_251117.log 2>&1 # Print table #rows #cols. Input: modelcard_step3_dedup_v2_<tag>.parquet. s2orc_titles2ids_<tag>.parquet. Output: benchmark_results_v2_<tag>.parquet, all_title_list_valid_v2_<tag>.parquet, all_valid_title_valid_v2_<tag>.txt. Here we filter out over large tables (max_cols=100, max_rows=200)
python -m src.data_analysis.qc_stats_fig --tag 251117 --v2_mode --exclude_resources llm > logs/qc_stats_fig_v2_251117.log 2>&1 # Plot benchmark results. Input: benchmark_results_v2_<tag>.parquet. Output: benchmark_metrics_vertical_v2_<tag>.pdf/pngWe could go for starmie searching and baselines searching. We need groundtruth for evaluation based on searched results and groundtruth results.
Generate the definitive ground truth files for evaluation.
This section details the process of generating ground truth labels for table unionability.
python -m src.data_gt.paper_citation_overlap --tag 251117 > logs/paper_citation_overlap_251117.log 2>&1 # Compute paper-pair citation overlap scores for ground truth. Input: s2orc_references_cache_<tag>.parquet (use columns), s2orc_titles2ids_<tag>.parquet (use minimum corpusIds as main Key). Output: modelcard_citation_all_matrices_<tag>.pkl.gz (REQUIRED for step3_gt)
PYTHONUNBUFFERED=1 python -m src.data_gt.step3_gt --tag 251117 --v2_mode > logs/step3_gt_v2_251117.log 2>&1 # Build ground truth (paper-level). Input: modelcard_citation_all_matrices_<tag>.pkl.gz, modelcard_step3_dedup_v2_<tag>.parquet, s2orc_titles2ids_<tag>.parquet, modelcard_all_title_list_<tag>.parquet. Output: data/gt/* (no versioning)
# Process SQLite ground truth into pickle files (if applicable from other benchmarks).
python -m src.data_gt.turn_tus_into_pickle > logs/turn_tus_into_pickle.log 2>&1
# (deprecate) python -m src.data_gt.gt_combine > logs/gt_combine.log 2>&1
python -m src.data_gt.modelcard_matrix --tag 251117 --v2_mode > logs/modelcard_matrix_v2_251117.log 2>&1 # Add other two levels of citation graphs (modelcard and dataset). Input: modelcard_step1_<tag>.parquet, modelcard_step3_dedup_v2_<tag>.parquet, modelcard_step3_merged_v2_<tag>.parquet. Output: modelcard_gt_related_model_v2_<tag>.parquet, data/gt/scilake_gt_modellink_*_v2_<tag>.npz
python -m src.data_gt.merge_union --level direct --tag 251117 --v2_mode > logs/merge_union_v2_251117.log 2>&1 # Merge union ground truth. Input: data/gt/*_v2_<tag>.npz, *_v2_<tag>.pkl. Output: data/gt/csv_pair_union_*_v2_<tag>_processed.npz
python -m src.data_gt.gt_distri --tag 251117 --v2_mode > logs/gt_distri_251117.log 2>&1 # Plot GT length distribution (boxplot/violin). Input: data/gt/*_v2_<tag>.npz and *_v2_<tag>_processed.npz (requires merge_union first). Use same --tag as merge_union.
python -m src.data_gt.nonzeroedge --tag 251117 --v2_mode > logs/nonzeroedge_v2_251117.log 2>&1 # Compute non-zero edge statistics for citation graphs. Input: data/gt/*_v2_<tag>.npz
# input: table, query how many tables related to this table in the ground truth
python -m src.data_gt.query_table2related_from_gt --query 1910.09700_table0.csv --levels direct --out tmp/direct_table_1910.txt
python -m src.data_gt.query_table2related_from_gt --query 1910.09700_table0.csv --targets table_b.csv table_c.csv --level directPrepare data and augmentations for integration with the Starmie benchmark framework.
Two main steps:
- zip and transfer the data to the server
- Create augmented table folders (tr/str): Generate transpose and string-augmented versions of tables
- Create symlinks: Link ModelTables tables to starmie_internal/data/scilake_final_/datalake
bash src/postprocess/zip_with_mask.sh 251117 # Step 0: zip with mask
# Step 1: Create augmented table folders (tr/str) deduped_hugging_csvs_v2_251117_tr, deduped_hugging_csvs_v2_251117_str
python -m src.data_symlink.trick_aug --repo_root /u1/z6dong/Repo/ModelTables/data/processed --mode tr --tag 251117 --v2_mode > logs/trick_aug_tr_v2_251117.log 2>&1
python -m src.data_symlink.trick_aug --repo_root /u1/z6dong/Repo/ModelTables/data/processed --mode str --tag 251117 --v2_mode > logs/trick_aug_str_v2_251117.log 2>&1
# Step 2: Create symlinks from ModelTables to starmie_internal/data/scilake_final_<tag>/datalake
python -m src.data_symlink.ln_scilake_new --repo_root /u1/z6dong/Repo --tag 251117 --v2_mode --n_jobs 32 > logs/ln_scilake_new_251117.log 2>&1Execute Starmie's pipeline for contrastive learning, embedding extraction, and search
python -m src.data_symlink.prepare_sample --tag 251117 --v2_mode --root_dir /u1/z6dong/Repo --output_file data/analysis/scilake_final_filelist_v2_251117.txt --limit 1000 --seed 42 > logs/prepare_sample_v2_251117.log 2>&1 # sample files for pretraining
# hands to starmie
cd ../starmie_internal
# Using date-based tag (e.g., 251117)
bash scripts/step1_pretrain.sh > logs/step1_pretrain_251117.log 2>&1 # Fine-tune contrastive learning model
bash scripts/step2_extractvectors.sh > logs/step2_extractvectors_251117.log 2>&1 # Encode embeddings for query and datalake items
bash scripts/step3_hnsw_search.sh > logs/step3_hnsw_search_251117.log 2>&1 # Perform data lake search (retrieval)
bash scripts/step3_processmetrics.sh > logs/step3_processmetrics_251117.log 2>&1 # Extract metrics based on ground truth and retrieval results; plot figures
bash scripts/step3_processmetrics_all.sh <EXPERIMENT_INDEX> > logs/step3_processmetrics_all_251117.log 2>&1 # run baselines
bash eval_per_resource.sh > logs/eval_per_resource_251117.log 2>&1 # Run ablation study on different resources (after getting results)Run baseline table embedding and retrieval methods for comparison, for faiss cpu/gpu installation, see FAISS GitHub repository.
7.1 Baseline1: Dense Search Unified script - supports base/str/tr modes Note: All three modes use the same Python script (table_retrieval_pipeline.py) with different --mode arguments The unified script replaces the separate pipeline_str.sh and pipeline_tr.sh scripts
This works on table
# encode
python src/baseline1/table_retrieval_pipeline.py encode --base_path /u501/z6dong/Repo/ModelTables --mask_file /u501/z6dong/Repo/ModelTables/data/analysis/all_valid_title_valid_v2_251117.txt --model_name all-MiniLM-L6-v2 --batch_size 512 --output_npz /u501/z6dong/Repo/ModelTables/data/baseline1_251117/valid_tables_v2_251117_embeddings.npz
# search
python src/baseline1/table_retrieval_pipeline.py search --emb_npz /u501/z6dong/Repo/ModelTables/data/baseline1_251117/valid_tables_v2_251117_embeddings.npz --top_k 5 --output_json /u501/z6dong/Repo/ModelTables/data/baseline1_251117/table_neighbors_v2_251117.json
# postprocess: add .csv back to id
python src/baseline1/table_retrieval_pipeline.py postprocess --input_json /u501/z6dong/Repo/ModelTables/data/baseline1_251117/table_neighbors_v2_251117.json
# Augmentation experiment
# Step1: encode: run above src/baseline1/table_retrieval_pipeline.py encode only for str and tr modes
# TODO:
# Step1.2: merge: run merge scripts for ori+tr, ori+str, ori+tr+str
python -m src.baseline1.aug_merge_embeddings --v2_mode --tag 251117 > logs/baseline1_aug_merge_embeddings_251117.log 2>&1 # for augmented ablation studies: step2 combine embedding and jsonl for ori+tr, ori+str, ori+tr+str
# Step2: search: run search scripts for ori+tr, ori+str, ori+tr+str
python -m src.baseline1.table_retrieval_pipeline \
search --emb_npz data/baseline1_251117/valid_tables_ori_tr_251117_embeddings.npz \
--top_k 11 \
--output_json data/baseline1_251117/table_neighbors_ori_tr_251117.json
# TODO for ori+str and ori+tr+str
# step3: postprocess: split into ori / tr / str json
V2_MODE=true TAG=251117 bash src/baseline1/aug_postprocess.sh > logs/baseline1_aug_postprocess_251117.log 2>&1
# step4: postprocess: all files back to ori csv name
V2_MODE=true TAG=251117 bash src/baseline1/aug_suffix_remove.sh > logs/baseline1_aug_suffix_remove_251117.log 2>&1
# Hands to starmie for computing metrics
bash scripts/step3_processmetrics_all.sh <index> > logs/baseline1_processmetrics_251117.log 2>&1 7.2 Baseline2: Sparse search Note: Requires pyserini and Java/JDK (for pyserini to work) Recommended conda environment: faiss_gpu_env (or any environment with pyserini installed) Output: data/tmp/baseline2_sparse_results_251117.json
This works on modelcard content retrieval
# 1. generate mapping from csv_path:readme_path
python src/baseline2/create_raw_csv_to_text_mapping.py --tag 251117 --v2_mode
# 2. get incontext embedding for each csv_path, save to data/tmp/corpus/collection.jsonl
python src/baseline2/create_dedup_table_to_text_mapping.py --tag 251117 --v2_mode
# 3. build index: sparse retrieval by pyserini
python -m pyserini.index.lucene --collection JsonCollection --input data/tmp/corpus_v2_251117 --index data/tmp/index_sparse_v2_251117 --generator DefaultLuceneDocumentGenerator --threads 1 --storePositions --storeDocvectors --storeRaw
# 4. build up tsv
python src/baseline2/create_queries_from_table.py --tag 251117 --v2_mode
# or python src/baseline2/create_queries_from_corpus.py
# 4. search pyserini
#python -m pyserini.search.lucene --index data/tmp/index_sparse_v2_251117 --topics data/tmp/queries_table_v2_251117.tsv --output data/tmp/search_result.txt --bm25 --hits 11 --threads 8 --batch-size 64 # as this can not solve truncating clause automatically
# or python batch_search.py
python src/baseline2/search_with_pyserini.py --top_k 11 --output data/tmp/baseline2_sparse_results_v2_251117.json
# 5. postprocess
python -m src.baseline1.table_retrieval_pipeline postprocess --input_json data/tmp/baseline2_sparse_results_v2_251117.json7.3 Baseline3: Hybrid (Sparse + Dense search) Note: Hybrid search uses Python scripts with command-line arguments
# Requires:
# - Sparse index from Baseline2: data/tmp/index_251117
# - Dense index directory: data/tmp/index_dense_251117/ (must contain index.faiss or index file)
# - To create dense index, first encode corpus and build faiss index:
# mkdir -p data/tmp/index_dense_251117
# python src/baseline1/table_retrieval_pipeline.py encode \
# --base_path data/processed --mask_file data/analysis/all_valid_title_valid.txt \
# --model_name sentence-transformers/all-MiniLM-L6-v2 \
# --batch_size 256 --output_index data/tmp/index_dense_251117/index.faiss --device cuda
# Output: data/tmp/search_result_hybrid_251117.json (then postprocess to baseline3_hybrid_results_251117.json)
python -m src.baseline2.search_with_pyserini_hybrid --top_k 11 --alpha 0.45 --tag 251117 --v2_mode > logs/baseline2_hybrid_search_v2_251117.log 2>&1
python -m src.baseline1.table_retrieval_pipeline postprocess --input_json data/tmp/baseline3_hybrid_results_v2_251117.jsonFor Blend baseline, refer to blend repository for instructions.
# figure1 + 2: qc_stats_fig + gt_distri's output figures
# figure3: count v.s. time. Based on step3_dedup_<tag>.parquet, all_title_list_valid_<tag>.parquet, output table_model_counts_over_time_<tag>.pdf/png
python -m src.data_analysis.table_model_counts_over_time --tag 251117 --v2_mode > logs/table_model_counts_over_time_v2_251117.log 2>&1 # step3_dedup_<tag>.parquet, all_title_list_valid_<tag>.parquet, output table_model_counts_over_time_<tag>.pdf/png
# step by step filtering img
python -m src.data_analysis.card_statistics --tag 251117 > logs/card_statistics_251117.log 2>&1 # get statistics of model cards
python -m src.data_analysis.hf_models_analysis --tag 251117 --v2_mode > logs/hf_models_analysis_v2_251117.log 2>&1 # get statistics of models in Hugging Face: hf_models_analysis.png and hf_cross_analysis.png
python -m src.data_analysis.model_snapshot_overlap > logs/model_snapshot_overlap_251117.log 2>&1 # compare modelId overlap between two fixed snapshots: V1 (no tag, no v2) vs V2 (tag 251117 + v2): data/analysis/model_snapshot_overlap.png
python -m src.data_analysis.align_tables_output_versions --dir-a data/processed/tables_output --dir-b data/processed/tables_output_v2_251117 > logs/align_tables_output_arxiv.log 2>&1
python -m src.data_analysis.compare_tables_by_content 2503.03556v1 > logs/compare_tables_by_content.log 2>&1 # compare tables by content for base id 2409.19581
# after carefully examining
python -m src.data_analysis.filtered_gt_visualization > logs/filtered_gt_visualization.log 2>&1
python -m src.data_analysis.quick_visualization_final > logs/quick_visualization_final.log 2>&1
# get relational keys from other key automatically (require logs/parquet_schema.log)
python -m src.data_analysis.get_from --target html_table_list_mapped_dedup --source modelId --value google-bert/bert-base-uncased > logs/get_from.log 2>&1
python -m src.data_analysis.get_from --target readme_path --source csv_paths --value "64dc62e53f_table2.csv" >> logs/get_from.log 2>&1
python -m src.data_analysis.get_from --target modelId --source hugging_table_list --value data/processed/deduped_hugging_csvs/021f09961f_table1.csv >> logs/get_from.log 2>&1
python -m src.data_analysis.get_from --target modelId --source pdf_link --value https://arxiv.org/pdf/0803.1019 >> logs/get_from.log 2>&1
# or
python -m src.postprocess.relational_parquet_strategies --tag 251117 --v2_mode > logs/relational_parquet_strategies_251117.log 2>&1 # python -m src.data_analysis.valid_table_shapes --tag 251117 --v2_mode > logs/valid_table_shapes_v2_251117.log 2>&1 # table shapes from valid table list, Input: all_valid_title_valid_v2_<tag>.txt, Output: valid_table_shapes_v2_<tag>.parquet; We double check the qc_stats filtering
#could execute sql on valid_table_shapes.parquet, get anomaly tables with extremely large rows or columns
python -m src.data_analysis.table_usage_stats --tag 251117 --v2_mode > logs/table_usage_stats_v2_251117.log 2>&1 # table usage value counts, Input: valid_table_shapes_v2_<tag>.parquet, Output: table_usage_stats_v2_<tag>.parquet