Supported File Types

Detailed Information on File Types

This page provides a comprehensive overview of the file types supported by the Hugging Face Backup Tool. Understanding these file types can help you select the correct files for backup and ensure that your backups are complete.

Table of Contents

1. Supported File Types
2. Selecting File Types for Backup
3. Custom File Types (Future Enhancement)

1. Supported File Types

The Hugging Face Backup Tool supports a variety of file types commonly used in machine learning and data science. The tool determines the file type based on the file extension.

File Type	File Extension(s)	Description	Typical Use Cases
SafeTensors	.safetensors	A secure and performant file format for storing tensors. It's designed to be safe to load, preventing arbitrary code execution.	Storing model weights and other tensor data, especially for models distributed by Hugging Face.
PyTorch Models	.pt	Files containing serialized PyTorch models, including model architecture and weights.	Saving and loading PyTorch models.
PyTorch Legacy	.pth	Older format for PyTorch models.	Saving and loading legacy PyTorch models.
ONNX Models	.onnx	Open Neural Network Exchange (ONNX) models. A standard format for representing machine learning models, enabling interoperability between different frameworks.	Sharing models between different frameworks, optimized for inference.
TensorFlow Models	.pb	TensorFlow protocol buffer files. Often used to store the model definition and trained weights.	Saving and loading TensorFlow models (e.g., saved models).
Keras Models	.h5	Hierarchical Data Format (HDF5) files containing Keras models.	Saving and loading Keras models.
Checkpoints	.ckpt	Files used to store the state of a model at a specific point in training (e.g., weights, optimizer state).	Saving and resuming model training.
Binary Files	.bin	Generic binary files.	Storing various binary data, like custom data or other model components.
JSON Files	.json	JavaScript Object Notation (JSON) files. Used to store structured data in a human-readable format.	Storing model configuration, metadata, or other structured data.
YAML Files	.yaml, .yml	YAML Ain't Markup Language (YAML) files. Another human-readable data serialization format, often used for configuration files.	Storing model configuration, dataset descriptions, or other configuration data.
Text Files	.txt	Plain text files.	Storing text-based data, like training logs, data descriptions, or model card information.
CSV Files	.csv	Comma-Separated Values (CSV) files. Used to store tabular data.	Storing datasets.
Pickle Files	.pkl	Python pickle files. Used to serialize and store Python objects.	Storing Python objects, but be cautious about security when loading pickle files from untrusted sources.
PNG Images	.png	Portable Network Graphics (PNG) image files. A lossless image format.	Storing image data (e.g., visualizations, dataset images).
JPEG Images	.jpg, .jpeg	Joint Photographic Experts Group (JPEG) image files. A lossy image format, good for photographs.	Storing image data (e.g., visualizations, dataset images).
WebP Images	.webp	WebP image files. A modern image format that provides better compression than JPEG or PNG.	Storing image data, often used for web-based images.
GIF Images	.gif	Graphics Interchange Format (GIF) image files. A format that supports animation.	Storing animated images.
ZIP Archives	.zip	ZIP archive files. Used for compressing and packaging files and folders.	Creating compressed backups, storing multiple related files in a single archive.
TAR Files	.tar	Tape Archive (TAR) files. Used for creating archives of files, often used in conjunction with compression.	Creating archives, often used in combination with compression algorithms like gzip or bzip2 (e.g., .tar.gz).
GZ Archives	.gz	Gzip compressed files. A common compression format.	Compressing individual files or archives (e.g., .tar.gz).

2. Selecting File Types for Backup

When using the Hugging Face Backup Tool, you select which file types to include in your backups. Consider the following when making your selection:

• Completeness: Ensure that you select all the file types that are relevant to your models, datasets, or spaces.
• Size: Be aware of the file sizes of the types you select, especially for images or large datasets, as this will affect the backup time and storage requirements.
• Relevance: Only back up the files you need. Avoid backing up temporary or irrelevant files.

3. Custom File Types (Future Enhancement)

In the future, the tool might support custom file type definitions. This would allow you to specify patterns for files not covered by the default extensions.