diff --git a/skills/tos-file-access/SKILL.md b/skills/tos-file-access/SKILL.md index 47864eb3..d342553c 100644 --- a/skills/tos-file-access/SKILL.md +++ b/skills/tos-file-access/SKILL.md @@ -1,16 +1,16 @@ --- name: tos-file-access -description: Upload files or directories to Volcano Engine TOS (Torch Object Storage) and download files from URLs. Use this skill when (1) Upload Agent-generated files or directories (like videos, images, reports, output folders) to TOS for sharing, (2) Download files from URLs before Agent processing. +description: Upload files or directories to TOS-compatible object storage for Volcano Engine or BytePlus and download files from URLs. Use this skill when (1) Upload Agent-generated files or directories for sharing, (2) Download files from URLs before Agent processing. license: Complete terms in LICENSE.txt --- # TOS File Access -This skill provides utilities for uploading files and directories to Volcano Engine TOS (Torch Object Storage) and downloading files from URLs. +This skill provides utilities for uploading files and directories to TOS-compatible object storage for Volcano Engine or BytePlus, and downloading files from URLs. ## Overview -**TOS (Torch Object Storage)** is Volcano Engine's object storage service, similar to AWS S3. This skill enables: +**TOS-compatible object storage** is used by Volcano Engine and BytePlus, similar to AWS S3. This skill enables: - **Upload**: Upload Agent-generated files or entire directories to TOS and get shareable signed URLs (for files) or TOS paths (for directories) - **Download**: Download files from URLs to local storage for Agent processing @@ -54,19 +54,16 @@ python scripts/tos_upload.py /path/to/report.pdf --bucket my-bucket --region cn- Download files from URLs to local storage. **Usage:** - ```bash python scripts/file_download.py [url2 ...] [--save-dir DIR] [--filenames NAME1 NAME2 ...] ``` **Arguments:** - - `urls`: One or more URLs to download (positional, required) - `--save-dir`: Save directory (optional, defaults to `/tmp`) - `--filenames`: Custom filenames for downloaded files (optional, must match number of URLs) **Examples:** - ```bash # Download single file to /tmp python scripts/file_download.py https://tos-cn-beijing.volces.com/bucket/file.pdf @@ -89,45 +86,39 @@ python scripts/file_download.py \ Upload files or directories to TOS and generate signed access URLs (for files) or TOS paths (for directories). **Key Features:** - - **Auto-detection**: Automatically detects whether the path is a file or directory - **Session-based paths**: Uses `TOOL_USER_SESSION_ID` environment variable to organize uploads - **Preserves structure**: For directories, maintains the full directory structure in TOS - **Automatic bucket creation**: Creates bucket if it doesn't exist (with private ACL) **Usage:** - ```bash python scripts/tos_upload.py --bucket BUCKET [--region REGION] [--expires SECONDS] ``` **Arguments:** - - `path`: Local file or directory path to upload (positional, required) - `--bucket`: TOS bucket name (required) -- `--region`: TOS region (optional, defaults to `cn-beijing`) +- `--region`: TOS region (optional, defaults to `cn-beijing` for `volcengine`, `ap-southeast-1` for `byteplus`) - `--expires`: Signed URL expiration in seconds (optional, defaults to 604800 = 7 days, only applies to file uploads) **Upload Structure:** - - **File**: `upload/{session_prefix}/{filename}` - Example: `upload/skill_agent_veadk_default_user_tmp-session-20251210150057/video.mp4` - **Directory**: `upload/{session_prefix}/{directory_name}/{relative_path}` - Example: `upload/skill_agent_veadk_default_user_tmp-session-20251210150057/output_folder/file1.txt` **Session Prefix:** - - If `TOOL_USER_SESSION_ID` is set, uses that value as prefix - Otherwise, falls back to timestamp format `YYYYMMDD_HHMMSS` -**Authentication:** +**Cloud provider and authentication:** Requires one of: - +- `CLOUD_PROVIDER`: `volcengine` or `byteplus` (optional, defaults to `volcengine`) - Environment variables: `VOLCENGINE_ACCESS_KEY` and `VOLCENGINE_SECRET_KEY` - VeFaaS IAM Role (automatic credential retrieval) **Examples:** - ```bash # Upload single file (auto-detected) python scripts/tos_upload.py /workspace/output.mp4 --bucket my-bucket @@ -148,12 +139,10 @@ python scripts/tos_upload.py /workspace/output_dir \ ``` **Returns:** - - **For files**: Prints a signed URL that can be shared with users (valid for specified duration) - **For directories**: Prints a TOS path in format `tos://bucket-name/path/to/directory` **Output Examples:** - ```bash # File upload output ============================================================ @@ -174,14 +163,15 @@ tos://my-bucket/upload/skill_agent_xxx/output_folder ## Environment Variables -- `VOLCENGINE_ACCESS_KEY`: Volcano Engine access key for TOS authentication -- `VOLCENGINE_SECRET_KEY`: Volcano Engine secret key for TOS authentication +- `CLOUD_PROVIDER`: Cloud provider, `volcengine` or `byteplus` (optional, defaults to `volcengine`) +- `VOLCENGINE_ACCESS_KEY`: Access key for object storage authentication +- `VOLCENGINE_SECRET_KEY`: Secret key for object storage authentication +- `REGION` or `DATABASE_TOS_REGION`: Optional region fallback when `--region` is not provided - `TOOL_USER_SESSION_ID`: Session ID used to generate organized upload paths (optional, falls back to timestamp) ## Common Use Cases 1. **Video Processing**: Download source video → process → upload result - ```bash # Download python scripts/file_download.py https://example.com/input.mp4 --save-dir /workspace @@ -191,7 +181,6 @@ tos://my-bucket/upload/skill_agent_xxx/output_folder ``` 2. **Data Analysis**: Download datasets → analyze → upload charts/reports folder - ```bash # Download data python scripts/file_download.py https://example.com/data.csv --save-dir /workspace @@ -201,7 +190,6 @@ tos://my-bucket/upload/skill_agent_xxx/output_folder ``` 3. **Batch Processing**: Process multiple files and upload output directory - ```bash # Process files (your code here, generates /workspace/batch_output/) # Upload entire output directory @@ -209,18 +197,12 @@ tos://my-bucket/upload/skill_agent_xxx/output_folder ``` 4. **Project Export**: Upload a complete project directory structure - ```bash # Upload project directory with subdirectories python scripts/tos_upload.py /workspace/my_project --bucket project-storage # Result: tos://project-storage/upload/{session_id}/my_project/... ``` - -## Error Handling - -- IF the script raises the error `VOLCENGINE_ACCESS_KEY and VOLCENGINE_SECRET_KEY are not provided or IAM Role is not configured.`, inform the user that they need to provide the `VOLCENGINE_ACCESS_KEY` and `VOLCENGINE_SECRET_KEY` environment variable. Write it to the environment variable file in the workspace. If the file already exists, append it to the end. Ensure the environment variable format is correct, make the environment variable effective, and retry the task that just failed. - ## Notes - **Auto-detection**: No need to specify whether uploading a file or directory - the script detects automatically @@ -234,3 +216,4 @@ tos://my-bucket/upload/skill_agent_xxx/output_folder - **IAM Role support**: Scripts automatically retrieve credentials from VeFaaS IAM when available - **Error handling**: Scripts print clear error messages for network, permission, or file issues - **Bucket requirement**: Bucket name must be specified via `--bucket` parameter (required) +- **Provider selection**: Set `CLOUD_PROVIDER=byteplus` to use BytePlus endpoints; otherwise Volcano Engine endpoints are used diff --git a/skills/tos-file-access/scripts/tos_upload.py b/skills/tos-file-access/scripts/tos_upload.py index 5f6ca6d2..2e401f20 100644 --- a/skills/tos-file-access/scripts/tos_upload.py +++ b/skills/tos-file-access/scripts/tos_upload.py @@ -11,23 +11,19 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. - """ TOS file/directory upload utility -Provides functionality to upload files or directories to Volcano Engine TOS object storage and returns signed access URLs +Provides functionality to upload files or directories to TOS object storage and returns signed access URLs """ import argparse import logging import os -import requests import sys from datetime import datetime from pathlib import Path -from pydantic import BaseModel from typing import Optional, Union -from dotenv import load_dotenv import tos from tos import HttpMethodType @@ -35,6 +31,7 @@ sys.path.append(str(Path(__file__).resolve().parent)) # Parent directory sys.path.append(str(Path(__file__).resolve().parent.parent)) +from veadk.auth.veauth.utils import get_credential_from_vefaas_iam logger = logging.getLogger(__name__) logger.setLevel(logging.INFO) @@ -47,85 +44,6 @@ logger.addHandler(console_handler) -def success_loaded_openclaw_dotenv() -> bool: - openclaw_env = Path("/root/.openclaw/.env") - # openclaw_env = Path( - # "/Users/bytedance/workspace/agentkit/agentkit-samples/skills/.venv/.openclaw/.env" - # ) - if openclaw_env.exists(): - success = load_dotenv(openclaw_env) - logger.info(f"Successfully loaded environment variables from {openclaw_env}") - return success - - -LOAD_OPENCLAW_DOTENV_SUCCESS = success_loaded_openclaw_dotenv() - - -class VeIAMCredential(BaseModel): - access_key_id: str - secret_access_key: str - session_token: str - - -def get_credential_from_service() -> VeIAMCredential: - """Get credential from credential service""" - - endpoint = os.getenv("CREDENTIAL_SERVICE_ENDPOINT") - api_key = os.getenv("CREDENTIAL_SERVICE_API_KEY") - - if not endpoint or not api_key: - logger.error( - "CREDENTIAL_SERVICE_ENDPOINT and CREDENTIAL_SERVICE_API_KEY environment variables must be set to fetch credentials from service." - ) - return None - - try: - response = requests.get( - url=f"{endpoint}/credential", - headers={"Authorization": f"Bearer {api_key}"}, - timeout=5, - ) - response.raise_for_status() - credential_data = response.json()["data"] - logger.info(f"Successfully fetched credentials from service {endpoint}.") - print(f"Credential data from service: {credential_data}") # Debug output - return VeIAMCredential( - access_key_id=credential_data["access_key_id"], - secret_access_key=credential_data["secret_access_key"], - session_token=credential_data["session_token"], - ) - except Exception as e: - logger.error(f"Failed to fetch credentials from service: {e}") - return None - - -def identify_volc_env() -> str: - """ - Identify the Volcano Engine environment (vefaas or ecs). - """ - - VEFAAS_IAM_CRIDENTIAL_PATH = "/var/run/secrets/iam/credential" - ECS_CLOUD_LINUX_ENV_PATH = "/etc/cloud/cloud.cfg" - ECS_CLOUD_WINDOWS_ENV_PATH = r"C:\Program Files\Cloudbase Solutions\Cloudbase-Init" - - if os.path.exists(VEFAAS_IAM_CRIDENTIAL_PATH): - return "vefaas" - elif os.path.exists(ECS_CLOUD_LINUX_ENV_PATH): - return "ecs" - elif os.path.exists(ECS_CLOUD_WINDOWS_ENV_PATH): - return "ecs" - else: - return "unknown" - - -VOLC_ENV = identify_volc_env() -if VOLC_ENV == "vefaas": - try: - from veadk.auth.veauth.utils import get_credential_from_vefaas_iam - except ImportError: - logger.error("vefaas environment detected but veadk import failed.") - - def _get_session_prefix() -> str: """Extract session prefix from TOOL_USER_SESSION_ID environment variable @@ -141,10 +59,31 @@ def _get_session_prefix() -> str: return datetime.now().strftime("%Y%m%d_%H%M%S") +def _resolve_tos_config(region: Optional[str] = None) -> tuple[str, str, str]: + """Resolve cloud provider, region, and TOS endpoint from environment.""" + provider = (os.getenv("CLOUD_PROVIDER") or "volcengine").lower() + if provider == "byteplus": + default_region = "ap-southeast-1" + sld = "bytepluses" + else: + provider = "volcengine" + default_region = "cn-beijing" + sld = "volces" + + resolved_region = ( + region + or os.getenv("REGION") + or os.getenv("DATABASE_TOS_REGION") + or default_region + ) + endpoint = f"tos-{resolved_region}.{sld}.com" + return provider, resolved_region, endpoint + + def upload_file_to_tos( file_path: str, bucket_name: str, - region: str = "cn-beijing", + region: Optional[str] = None, ak: Optional[str] = None, sk: Optional[str] = None, session_token: Optional[str] = None, @@ -156,7 +95,7 @@ def upload_file_to_tos( Args: file_path: Local file path bucket_name: TOS bucket name - region: TOS region, defaults to cn-beijing + region: TOS region; defaults to cn-beijing for volcengine and ap-southeast-1 for byteplus ak: Access Key; if empty, reads from environment variables sk: Secret Key; if empty, reads from environment variables session_token: Session token @@ -167,8 +106,9 @@ def upload_file_to_tos( None: Returns None if upload fails Environment variables: - VOLCENGINE_ACCESS_KEY: Volcano Engine access key - VOLCENGINE_SECRET_KEY: Volcano Engine secret key + CLOUD_PROVIDER: Cloud provider, volcengine or byteplus + VOLCENGINE_ACCESS_KEY: Access key + VOLCENGINE_SECRET_KEY: Secret key TOOL_USER_SESSION_ID: Session ID for generating object key prefix """ if bucket_name is None: @@ -190,38 +130,19 @@ def upload_file_to_tos( session_token = session_token or "" if not (access_key and secret_key): - # First try to get credentials from Credential Service if environment variables are set - if os.getenv("CREDENTIAL_SERVICE_ENDPOINT") and os.getenv( - "CREDENTIAL_SERVICE_API_KEY" - ): - logger.info("Trying to fetch credentials from Credential Service...") - try: - cred = get_credential_from_service() - access_key = cred.access_key_id - secret_key = cred.secret_access_key - session_token = cred.session_token - except Exception as e: - logger.warning(f"Failed to get credential from Credential Service: {e}") - - if VOLC_ENV == "vefaas": - if get_credential_from_vefaas_iam: - logger.info("Trying to fetch credentials from VeFaaS IAM...") - try: - cred = get_credential_from_vefaas_iam() - access_key = cred.access_key_id - secret_key = cred.secret_access_key - session_token = cred.session_token - except Exception as e: - logger.warning(f"Failed to get credential from vefaas iam: {e}") - else: - logger.warning( - "vefaas environment detected but get_credential_from_vefaas_iam is None." - ) + try: + cred = get_credential_from_vefaas_iam() + access_key = cred.access_key_id + secret_key = cred.secret_access_key + session_token = cred.session_token + except Exception as e: + logger.error(f"Failed to get credentials from IAM: {e}") if not access_key or not secret_key: - raise PermissionError( - "VOLCENGINE_ACCESS_KEY and VOLCENGINE_SECRET_KEY are not provided or IAM Role or Credential Service is not configured." + logger.error( + "Error: VOLCENGINE_ACCESS_KEY and VOLCENGINE_SECRET_KEY are not provided or IAM Role is not configured." ) + return None # Auto-generate object_key: upload/{session_prefix}/{filename} session_prefix = _get_session_prefix() @@ -232,15 +153,18 @@ def upload_file_to_tos( client = None try: # Initialize TOS client - endpoint = f"tos-{region}.volces.com" + provider, resolved_region, endpoint = _resolve_tos_config(region) client = tos.TosClientV2( ak=access_key, sk=secret_key, security_token=session_token, endpoint=endpoint, - region=region, + region=resolved_region, ) + logger.info(f"Cloud Provider: {provider}") + logger.info(f"TOS Region: {resolved_region}") + logger.info(f"TOS Endpoint: {endpoint}") logger.info(f"Starting file upload: {file_path}") logger.info(f"Target Bucket: {bucket_name}") logger.info(f"Object Key: {object_key}") @@ -308,7 +232,7 @@ def upload_file_to_tos( def upload_directory_to_tos( directory_path: str, bucket_name: str, - region: str = "cn-beijing", + region: Optional[str] = None, ak: Optional[str] = None, sk: Optional[str] = None, session_token: Optional[str] = None, @@ -320,7 +244,7 @@ def upload_directory_to_tos( Args: directory_path: Local directory path bucket_name: TOS bucket name - region: TOS region, defaults to cn-beijing + region: TOS region; defaults to cn-beijing for volcengine and ap-southeast-1 for byteplus ak: Access Key; if empty, reads from environment variables sk: Secret Key; if empty, reads from environment variables session_token: Session token @@ -331,8 +255,9 @@ def upload_directory_to_tos( None: Returns None if upload fails Environment variables: - VOLCENGINE_ACCESS_KEY: Volcano Engine access key - VOLCENGINE_SECRET_KEY: Volcano Engine secret key + CLOUD_PROVIDER: Cloud provider, volcengine or byteplus + VOLCENGINE_ACCESS_KEY: Access key + VOLCENGINE_SECRET_KEY: Secret key """ if bucket_name is None: logger.error("Error: bucket name The bucket has not been specified.") @@ -353,36 +278,17 @@ def upload_directory_to_tos( session_token = session_token or "" if not (access_key and secret_key): - # First try to get credentials from Credential Service if environment variables are set - if os.getenv("CREDENTIAL_SERVICE_ENDPOINT") and os.getenv( - "CREDENTIAL_SERVICE_API_KEY" - ): - logger.info("Trying to fetch credentials from Credential Service...") - try: - cred = get_credential_from_service() - access_key = cred.access_key_id - secret_key = cred.secret_access_key - session_token = cred.session_token - except Exception as e: - logger.warning(f"Failed to get credential from Credential Service: {e}") - - if VOLC_ENV == "vefaas": - if get_credential_from_vefaas_iam: - try: - cred = get_credential_from_vefaas_iam() - access_key = cred.access_key_id - secret_key = cred.secret_access_key - session_token = cred.session_token - except Exception as e: - logger.error(f"Failed to get credential from vefaas iam: {e}") - else: - logger.warning( - "vefaas environment detected but get_credential_from_vefaas_iam is None." - ) + try: + cred = get_credential_from_vefaas_iam() + access_key = cred.access_key_id + secret_key = cred.secret_access_key + session_token = cred.session_token + except Exception as e: + logger.error(f"Failed to get credentials from IAM: {e}") if not access_key or not secret_key: logger.error( - "Error: VOLCENGINE_ACCESS_KEY and VOLCENGINE_SECRET_KEY are not provided or IAM Role or Credential Service is not configured." + "Error: VOLCENGINE_ACCESS_KEY and VOLCENGINE_SECRET_KEY are not provided or IAM Role is not configured." ) return None @@ -396,15 +302,18 @@ def upload_directory_to_tos( try: # Initialize TOS client - endpoint = f"tos-{region}.volces.com" + provider, resolved_region, endpoint = _resolve_tos_config(region) client = tos.TosClientV2( ak=access_key, sk=secret_key, security_token=session_token, endpoint=endpoint, - region=region, + region=resolved_region, ) + logger.info(f"Cloud Provider: {provider}") + logger.info(f"TOS Region: {resolved_region}") + logger.info(f"TOS Endpoint: {endpoint}") logger.info(f"Starting directory upload: {directory_path}") logger.info(f"Target Bucket: {bucket_name}") logger.info(f"Object Key Prefix: {object_key_prefix}") @@ -438,12 +347,10 @@ def upload_directory_to_tos( # Upload file try: - result = client.put_object_from_file( + client.put_object_from_file( bucket=bucket_name, key=object_key, file_path=file_path ) - logger.info( - f"Uploaded: {file_path} -> {object_key}, result: {result}" - ) + logger.info(f"Uploaded: {file_path} -> {object_key}") except Exception as e: logger.error(f"Failed to upload {file_path}: {e}") @@ -476,7 +383,7 @@ def upload_directory_to_tos( def upload_to_tos( path: str, bucket_name: str, - region: str = "cn-beijing", + region: Optional[str] = None, ak: Optional[str] = None, sk: Optional[str] = None, session_token: Optional[str] = None, @@ -536,26 +443,27 @@ def upload_to_tos( def main(): """Command-line interface for tos_upload""" parser = argparse.ArgumentParser( - description="Upload files or directories to Volcano Engine TOS and generate signed URLs", + description="Upload files or directories to TOS and generate signed URLs", formatter_class=argparse.RawDescriptionHelpFormatter, epilog=""" Examples: # Upload a file (auto-detect) python tos_upload.py /path/to/file.mp4 --bucket my-bucket - + # Upload a directory (auto-detect) python tos_upload.py /path/to/directory --bucket my-bucket - + # Upload to different region with custom expiration python tos_upload.py /path/to/file.json --bucket my-bucket --region cn-beijing --expires 86400 File Upload Structure: File: upload/{session_prefix}/{filename} Directory: upload/{session_prefix}/{directory_name}/{relative_path} - + Environment Variables: - VOLCENGINE_ACCESS_KEY Volcano Engine access key - VOLCENGINE_SECRET_KEY Volcano Engine secret key + CLOUD_PROVIDER volcengine or byteplus + VOLCENGINE_ACCESS_KEY Access key + VOLCENGINE_SECRET_KEY Secret key TOOL_USER_SESSION_ID Session ID for generating object key prefix """, ) @@ -567,8 +475,8 @@ def main(): parser.add_argument( "--region", type=str, - default="cn-beijing", - help="TOS region (default: cn-beijing)", + default=None, + help="TOS region (default: cn-beijing for volcengine, ap-southeast-1 for byteplus)", ) parser.add_argument(