Skip to content

sagemaker_config related Networking bugs in ModelTrainer #5766

@admivsn

Description

@admivsn

PySDK Version

  • PySDK V2 (2.x)
  • [x ] PySDK V3 (3.x)

Describe the bug
2 closely related Networking bugs in ModelTrainer that prevent the use of sagemaker_config.

  1. Incorrect parameter used here in Networking definition causing Pydantic ValidationError:
                self.networking = Networking(
                    default_enable_network_isolation=default_enable_network_isolation,
  1. Networking security_group_ids not successfully added here in a silent bug.
            if self.networking.subnets is None:
                self.networking.subnets = self.config_mgr.resolve_value_from_config(
                    config_path=TRAINING_JOB_SUBNETS_PATH
                )
            if self.networking.security_group_ids is None:
                self.networking.subnets = self.config_mgr.resolve_value_from_config( 
                    config_path=TRAINING_JOB_SUBNETS_PATH
                )

To reproduce

from sagemaker.core.helper.session_helper import Session
from sagemaker.train.model_trainer import ModelTrainer

sagemaker_config = {
    "SageMaker": {
        "TrainingJob": {
            "VpcConfig": {
                "SecurityGroupIds": ["sg-blah"],
                "Subnets": ["subnet-blah"]
            }
        }
    },
    "SchemaVersion": "1.0",
}

sagemaker_session = Session(sagemaker_config=sagemaker_config)

model_trainer = ModelTrainer(
    training_image="685385470294.dkr.ecr.eu-west-1.amazonaws.com/sagemaker-xgboost:1.7-1",
    sagemaker_session=sagemaker_session,
)

# Error occurs here when SDK tries to instantiate Networking 
model_trainer.train(input_data_config={})
ValidationError: 1 validation error for Networking
default_enable_network_isolation
  Extra inputs are not permitted [type=extra_forbidden, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden

Expected behavior
sagemaker_config variables to be successfully added to ModelTrainer.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions