Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -493,6 +493,7 @@ result.enable_job_notifications(

Diagnose gradient spikes by identifying which training data lines were used in a specific training step. Enable with `enable_batch_sample_tracing=True` on `ForgeTrainer`, then call `trainer.trace_batch(result, step=N)` after the job completes. See [`docs/spec/service-classes.md`](docs/spec/service-classes.md) for full API details.


---
## Telemetry

Expand Down
3 changes: 2 additions & 1 deletion docs/spec/dataset.md
Original file line number Diff line number Diff line change
Expand Up @@ -414,6 +414,7 @@ def transform(
- `training_method` (TrainingMethod): Required. The training method (e.g., `TrainingMethod.SFT_LORA`).
- `model` (Model): Required. The target model.
- `eval_task` (EvaluationTask): Optional. Required when `training_method` is `EVALUATION`.
- `platform` (Platform): Required for RFT Multiturn methods (`RFT_MULTITURN_LORA`, `RFT_MULTITURN_FULL`). Determines the output format: `Platform.SMTJServerless` produces flat `{"prompt": "..."}` format, `Platform.SMHP` produces nested `{"id": "...", "metadata": {"prompt": "..."}}` format.
- `column_mappings` (dict): Optional. Maps standard column names to your dataset's column names.
- `multimodal_data_s3_path` (Optional[str]): S3 prefix where images will be uploaded during conversion (e.g., `s3://my-bucket/images/`). Required when the dataset contains `image_url` content blocks in OpenAI format. When saving the output to S3, the save path must use the same bucket.
- `multimodal_data_bucket_owner` (Optional[str]): AWS account ID that owns the S3 bucket. If not provided, auto-resolved via STS.
Expand Down Expand Up @@ -460,7 +461,7 @@ def validate(
- `method` (ValidateMethod): The validation method (default: `ValidateMethod.INVALID_RECORDS`). `ValidateMethod.SCHEMA` is deprecated but still supported for backward compatibility.
- `model` (Model): The Nova model version (e.g., `Model.NOVA_LITE`)
- `eval_task` (EvaluationTask): Optional. The evaluation task (e.g., `EvaluationTask.GEN_QA`)
- `platform` (Platform): Optional. The target platform (`Platform.SMTJ`, `Platform.SMHP`, or `Platform.BEDROCK`). Accepted for forward-compatibility; row-count checks are currently enforced only during `train()`, not `validate()`.
- `platform` (Platform): Optional. The target platform (`Platform.SMTJServerless`, `Platform.SMHP`, `Platform.SMTJ`, or `Platform.BEDROCK`). Required for RFT Multiturn methods — routes to `RFTMultiturnServerlessValidator` when `Platform.SMTJServerless`, validating the flat `{"prompt": "..."}` format.

**Returns:**
- None
Expand Down
40 changes: 33 additions & 7 deletions docs/spec/model-customizer.md
Original file line number Diff line number Diff line change
Expand Up @@ -212,6 +212,7 @@ def train(
- `loraplus_lr_ratio` (float): LoRA+ learning rate ratio
- `global_batch_size` (int): Global batch size
- `max_length` (int): Maximum sequence length
- `model_name_or_path` (str): S3 checkpoint path or a SageMaker Model Package ARN for iterative training
- A full list of available overrides can be found via the [Nova Customization public documentation](https://docs.aws.amazon.com/nova/latest/userguide/customize-fine-tune-sagemaker.html) or by referencing the training recipes [here](https://docs.aws.amazon.com/sagemaker/latest/dg/nova-model-recipes.html).
- `rft_lambda_arn` (Optional[str]): Rewards Lambda ARN (only used for RFT training methods). If passed, takes priority over `rft_lambda_arn` set on the `RuntimeManager`.
- `validation_data_s3_path` (Optional[str]): Validation S3 path, applicable for CPT and SFT on SMTJ/SMTJServerless/SMHP, or any method on Bedrock (optional)
Expand All @@ -225,8 +226,8 @@ def train(
- `started_time` (datetime): Job start timestamp
- `model_artifacts` (ModelArtifacts): Paths to model checkpoints and outputs
- `checkpoint_s3_path` (str, Optional): Path to the model checkpoint/trained model. For `SMTJServerless`, populated after job completion via `get_model_artifacts()`.
- `output_s3_path` (str): Path to the metrics and output tar file.
- `output_model_arn` (str, Optional): Model package ARN for `SMTJServerless` jobs. Use as `model_path` for iterative training.
- `output_s3_path` (str, Optional): Path to the metrics and output tar file. For MTRL jobs, fetched from the AgentRFT API.
- `output_model_arn` (str, Optional): Model package ARN for `SMTJServerless` jobs. Use as `model_arn` in `ForgeTrainer` for iterative training.
- `model_type` (Model): Model type of the model being trained

**Raises:**
Expand Down Expand Up @@ -382,7 +383,7 @@ def deploy(
* **Note:** If `model_artifact_path` is provided, we will NOT attempt to resolve `model_artifact_path` from `job_result` or the enclosing `NovaModelCustomizer` object.

**Parameters:**
- `model_artifact_path` (Optional[str]): S3 path to the trained model checkpoint. If not provided, will attempt to extract from job_result or the `job_id` field of the Customizer.
- `model_artifact_path` (Optional[str]): S3 path to the trained model checkpoint, or a SageMaker Model Package name/ARN. Model package ARNs are auto-detected for SageMaker deployments and passed via `ModelPackageName`. If not provided, will attempt to extract from job_result or the `job_id` field of the Customizer.
- `deploy_platform` (DeployPlatform): Platform to deploy the model to
- `DeployPlatform.BEDROCK_OD`: Bedrock On-Demand
- `DeployPlatform.BEDROCK_PT`: Bedrock Provisioned Throughput
Expand Down Expand Up @@ -432,6 +433,15 @@ sagemaker_deployment = customizer.deploy(
print(f"Model deployed: {sagemaker_deployment.endpoint.uri}")
print(f"Endpoint: {sagemaker_deployment.endpoint.endpoint_name}")
print(f"Status: {sagemaker_deployment.status}")

# Deploy to SageMaker using a model package ARN (auto-detected)
sagemaker_mp_deployment = customizer.deploy(
model_artifact_path="arn:aws:sagemaker:us-east-1:123456789012:model-package/my-group/1",
deploy_platform=DeployPlatform.SAGEMAKER,
unit_count=1,
endpoint_name="my-model-package-endpoint",
)
print(f"Model deployed: {sagemaker_mp_deployment.endpoint.uri}")
```

Optionally, you can provide a Bedrock execution role name to be used in deployment.
Expand All @@ -454,12 +464,16 @@ create_bedrock_execution_role(
```
---
#### `create_custom_model()`
Creates a Bedrock custom model from S3 artifacts, decoupled from endpoint deployment.
Creates a Bedrock custom model from S3 artifacts or a model package ARN, decoupled from endpoint deployment.

This method extracts the model-creation step from the deploy flow so users can
create a model independently of endpoint deployment, enabling retry of deployment
if it fails after model creation.

Either `model_artifact_path` (maps to `modelSourceConfig`) or `custom_model_data_source`
(maps to `customModelDataSource`) must be provided, but not both. If neither is provided,
the method attempts to resolve from `job_result`.

**Signature:**
```python
def create_custom_model(
Expand All @@ -470,26 +484,28 @@ def create_custom_model(
execution_role_name: Optional[str] = None,
tags: Optional[List[Dict[str, str]]] = None,
skip_model_reuse: bool = False,
custom_model_data_source: Optional[Dict[str, Any]] = None,
) -> ModelDeployResult
```

**Parameters:**
- `model_artifact_path` (Optional[str]): S3 path to trained model checkpoint. Takes precedence over `job_result` if both are provided.
- `model_artifact_path` (Optional[str]): S3 path to trained model checkpoint. Used to populate `modelSourceConfig.s3DataSource.s3Uri`. Takes precedence over `job_result` if both are provided.
- `job_result` (Optional[TrainingResult]): Training job result to extract checkpoint path from.
- `endpoint_name` (Optional[str]): Optional name prefix for the model name (auto-generated if not provided).
- `execution_role_name` (Optional[str]): IAM role name for Bedrock. Defaults to `BedrockDeployModelExecutionRole`.
- `tags` (Optional[List[Dict[str, str]]]): Optional list of `{"key": str, "value": str}` dicts for source tracking.
- `skip_model_reuse` (bool): If True, always create a new model even if one with the same escrow URI already exists. Default: False.
- `custom_model_data_source` (Optional[Dict[str, Any]]): Alternative data source configuration for the custom model. When provided, `modelSourceConfig` is omitted from the API call.

**Returns:**
- `ModelDeployResult`: Contains:
- `model_arn` (str): The Bedrock custom model ARN
- `model_name` (str): The model name passed to CreateCustomModel
- `escrow_uri` (str): S3 artifacts path used to create the model
- `escrow_uri` (str): S3 artifacts path or model package ARN used to create the model
- `created_at` (datetime): UTC timestamp when the model was created

**Raises:**
- `ValueError`: When neither `model_artifact_path` nor `job_result` is provided, or when checkpoint path cannot be resolved from `job_result`.
- `ValueError`: When neither `model_artifact_path`, `job_result`, nor `custom_model_data_source` is provided, or when both `model_artifact_path` and `custom_model_data_source` are provided.
- `RuntimeError`: When IAM role creation or custom model creation fails.

**Example:**
Expand All @@ -505,6 +521,16 @@ publish_result.dump(file_path="./results/")

# Or create from a training job result
publish_result = customizer.create_custom_model(job_result=training_result)

# Or create from a model package ARN
publish_result = customizer.create_custom_model(
custom_model_data_source={
"modelPackageArnDataSource": {
"modelPackageArn": "arn:aws:sagemaker:us-east-1:123456789012:model-package/my-group/1"
}
}
)
print(f"Model ARN: {publish_result.model_arn}")
```
---
#### `deploy_to_bedrock()`
Expand Down
12 changes: 11 additions & 1 deletion docs/spec/rft-multiturn.md
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,16 @@ print(f"Uploaded to: {custom_env.s3_uri}")
- `get_configuration()`: Get infrastructure config
- `get_recipe_overrides()`: Get recipe overrides for training

**Note:** RFT multiturn only supports SageMaker HyperPod (SMHP) platform and Nova 2.0 models (NOVA_LITE_2).
**Note:** RFT multiturn supports SageMaker HyperPod (SMHP) and SMTJServerless platforms with Nova 2.0 models (NOVA_LITE_2).

---

### Platform Requirements

#### MLFlow Required for MTRL on SMTJServerless

MTRL training and evaluation on `SMTJServerless` require `ForgeConfig.mlflow_monitor` with a valid `tracking_uri` — raises `ValueError` if missing.

MLFlow is optional on SMHP.

---
13 changes: 12 additions & 1 deletion docs/spec/runtime-managers.md
Original file line number Diff line number Diff line change
Expand Up @@ -426,7 +426,9 @@ def __init__(
security_group_ids: Optional[list[str]] = None,
max_job_runtime: Optional[int] = 86400,
rft_lambda: Optional[str] = None,
agent_core_arn: Optional[str] = None,
evaluator_name: Optional[str] = None,
intermediate_model_package_group_name: Optional[str] = None,
)
```

Expand All @@ -438,16 +440,25 @@ def __init__(
- `subnets` (Optional[list[str]]): Optional list of strings representing subnets. Default value is None.
- `security_group_ids` (Optional[list[str]]): Optional list of strings representing security group IDs. Default value is None.
- `max_job_runtime` (Optional[int]): Max Job Runtime in seconds (default: 1 day)
- `rft_lambda` (Optional[str]): Lambda ARN, SageMaker hub-content ARN, or local `.py` file path for the RFT reward function.
- `rft_lambda` (Optional[str]): Lambda ARN, SageMaker hub-content ARN, or local `.py` file path for the RFT reward function. Used for single-turn RFT or as an alternative agent environment for MTRL on serverless.
- **Lambda ARN**: Automatically registered as a hub-content `JsonDoc` evaluator during `train()`. The hub-content ARN is passed as `EvaluatorArn` in `ServerlessJobConfig`.
- **Hub-content ARN**: Passed directly as `EvaluatorArn` — no registration needed.
- **Local `.py` file**: Call `deploy_lambda()` first to deploy and get a Lambda ARN.
- `agent_core_arn` (Optional[str]): Bedrock AgentCore runtime ARN for MTRL training. If both `agent_core_arn` and `rft_lambda` are set, `agent_core_arn` takes priority as the agent environment.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this work only for the new MTRL serverless or does SMHP also support this?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MTRL training is only available for SMTJ Serverless

@cmahima cmahima Jun 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this works for SMTJ serverless only

- `evaluator_name` (Optional[str]): Optional human-readable name for the hub-content evaluator entry when auto-registering a Lambda ARN. Defaults to the Lambda function name.
- `intermediate_model_package_group_name` (Optional[str]): Model package group name for intermediate checkpoints during MTRL training. If not set, defaults to `{model_package_group_name}-checkpoints`.

**Example:**
```python
from amzn_nova_forge.manager import *

# With Bedrock AgentCore (for MTRL training)
infra = SMTJServerlessRuntimeManager(
model_package_group_name="nova-rft-serverless",
execution_role="arn:aws:iam::123456789012:role/my-role",
agent_core_arn="arn:aws:bedrock-agentcore:us-east-1:123456789012:runtime/my-agent",
)

# With a Lambda ARN (auto-registered as hub-content during train())
infra = SMTJServerlessRuntimeManager(
model_package_group_name="nova-rft-serverless",
Expand Down
Loading
Loading