diff --git a/docs/admin/deployment/gcp/02-prerequisites.md b/docs/admin/deployment/gcp/02-prerequisites.md index 184066eb..25502074 100644 --- a/docs/admin/deployment/gcp/02-prerequisites.md +++ b/docs/admin/deployment/gcp/02-prerequisites.md @@ -102,7 +102,6 @@ For GCP deployments, gcloud CLI serves dual purposes: GCP resource management an You will need access to the following repositories to complete the deployment: -- **Terraform Remote Backend:** [codemie-terraform-gcp-remote-backend](https://gitbud.epam.com/epm-cdme/codemie-terraform-gcp-remote-backend) - **Terraform Platform Modules:** [codemie-terraform-gcp-platform](https://gitbud.epam.com/epm-cdme/codemie-terraform-gcp-platform) - **Helm Charts:** [codemie-helm-charts](https://gitbud.epam.com/epm-cdme/codemie-helm-charts) diff --git a/docs/admin/deployment/gcp/04-infrastructure-deployment/index.md b/docs/admin/deployment/gcp/04-infrastructure-deployment/index.md index 4b66b138..2e76bb54 100644 --- a/docs/admin/deployment/gcp/04-infrastructure-deployment/index.md +++ b/docs/admin/deployment/gcp/04-infrastructure-deployment/index.md @@ -4,7 +4,7 @@ title: GCP Infrastructure Deployment sidebar_label: Infrastructure Deployment sidebar_position: 4 pagination_prev: admin/deployment/gcp/architecture -pagination_next: admin/deployment/gcp/infrastructure-deployment/infrastructure-manual-deployment +pagination_next: admin/deployment/gcp/infrastructure-deployment/infrastructure-scripted-deployment --- # GCP Infrastructure Deployment @@ -59,7 +59,7 @@ The core platform infrastructure provisions all GCP resources needed to run AI/R | **Subnets** | Network segmentation for GKE nodes and pods | | **Cloud NAT** | Provides consistent outbound public IP for internet connectivity | | **Cloud Router** | Enables dynamic routing for VPC | -| **DNS Zones** | Name resolution for CodeMie components (public or private) | +| **DNS Zones** | Name resolution for CodeMie components | | **Firewall Rules** | Network access control and traffic filtering | ### Data & Storage @@ -100,4 +100,13 @@ The core platform infrastructure leverages proven Terraform modules from the com With the infrastructure resources defined, you are now ready to proceed with the deployment of the GCP infrastructure. -Continue to the [Infrastructure Deployment Guide](./infrastructure-manual-deployment) to execute the deployment process and provision all required GCP resources for AI/Run CodeMie. +Proceed to the next step to deploy the infrastructure: + +- [**Scripted Deployment** →](./infrastructure-scripted-deployment) - Recommended automated deployment using the `gcp-terraform.sh` script +- [**Manual Deployment** →](./infrastructure-manual-deployment) - Advanced option for custom scenarios with manual Terraform control + +:::note Deployment Method Selection + +- **Scripted Deployment**: Handles prerequisites, validation, and orchestration automatically (recommended for most users) +- **Manual Deployment**: Provides full control over Terraform operations for advanced customization + ::: diff --git a/docs/admin/deployment/gcp/04-infrastructure-deployment/manual-deployment.md b/docs/admin/deployment/gcp/04-infrastructure-deployment/manual-deployment.md index a43cd806..e36709da 100644 --- a/docs/admin/deployment/gcp/04-infrastructure-deployment/manual-deployment.md +++ b/docs/admin/deployment/gcp/04-infrastructure-deployment/manual-deployment.md @@ -2,11 +2,14 @@ id: infrastructure-manual-deployment title: Manual Infrastructure Deployment sidebar_label: Manual Deployment -sidebar_position: 1 +sidebar_position: 2 pagination_prev: admin/deployment/gcp/infrastructure-deployment/infrastructure-deployment-overview pagination_next: admin/deployment/gcp/components-deployment/components-deployment-overview --- +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + # Manual Infrastructure Deployment This guide walks you through deploying GCP infrastructure for AI/Run CodeMie using Terraform with manual step-by-step instructions. This approach provides full control over each deployment phase and allows for customization at every step. @@ -28,24 +31,24 @@ Before starting the deployment, ensure you have completed all requirements from - [ ] **GCP Access**: Project Owner or Editor role with IAM permissions - [ ] **Required APIs Enabled**: Cloud IAP, Service Networking, Secret Manager, Vertex AI APIs -- [ ] **Tools Installed**: Terraform 1.5.7, gcloud CLI, kubectl, Helm, Docker -- [ ] **GCP Authentication**: Logged in with gcloud CLI and project set +- [ ] **Tools Installed**: Terraform 1.13.5, gcloud CLI, kubectl, Helm, Docker +- [ ] **GCP Authentication**: Logged in with gcloud CLI and application default credentials configured - [ ] **Repository Access**: Have access to Terraform and Helm repositories - [ ] **Network Planning**: Prepared list of authorized networks (if accessing GKE API from workstation) - [ ] **Domain & Certificate**: DNS zone and TLS certificate ready (for public access) or will use private DNS :::warning Authentication Required -You must be authenticated to GCP CLI before running Terraform. Run `gcloud auth login` and verify with `gcloud config get-value project`. +You must be authenticated to GCP CLI before running Terraform. Run `gcloud auth login` and `gcloud auth application-default login`. Verify the active project with `gcloud config get-value project`. ::: ## Deployment Phases -Manual deployment involves two sequential phases: +Manual deployment involves two sequential phases, both within the same repository: -| Phase | Description | Repository | -| ------------------------------------ | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | -| **Phase 1: State Backend** | Creates GCS bucket for Terraform state files | [codemie-terraform-gcp-remote-backend](https://gitbud.epam.com/epm-cdme/codemie-terraform-gcp-remote-backend) | -| **Phase 2: Platform Infrastructure** | Deploys GKE, networking, storage, databases, security components, and Bastion Host | [codemie-terraform-gcp-platform](https://gitbud.epam.com/epm-cdme/codemie-terraform-gcp-platform) | +| Phase | Description | Directory | +| ------------------------------------ | ---------------------------------------------------------------------------------- | ----------------- | +| **Phase 1: State Backend** | Creates GCS bucket for Terraform state files | `remote-backend/` | +| **Phase 2: Platform Infrastructure** | Deploys GKE, networking, storage, databases, security components, and Bastion Host | `platform/` | :::info Bastion Host Bastion Host is optional and only required for completely private GKE clusters with private DNS. For public clusters or clusters with authorized networks, you can access GKE API directly. @@ -59,63 +62,72 @@ The first step is to create a Google Cloud Storage bucket for storing Terraform The state backend ensures that your infrastructure state is stored securely and can be shared across your team. Without this, Terraform state would only exist locally on your machine. ::: -### Step 1: Clone the Repository - -Clone the Terraform state backend repository to your local machine: +1. Clone the platform repository to your local machine: ```bash -git clone git@gitbud.epam.com:epm-cdme/codemie-terraform-gcp-remote-backend.git -cd codemie-terraform-gcp-remote-backend +git clone https://gitbud.epam.com/epm-cdme/codemie-terraform-gcp-platform.git +cd codemie-terraform-gcp-platform ``` -### Step 2: Review Configuration Variables - -Open and review the `variables.tf` file to understand available configuration options: +2. Navigate to the `remote-backend/` directory and configure variables. There are two ways to provide Terraform variables: ```bash -# View all available variables -cat variables.tf +cd remote-backend ``` -Key variables to consider: - -- **project_id**: Your GCP project ID -- **storage_bucket_name**: Prefix for storage bucket name -- **region**: GCP region for the bucket (e.g., `europe-west3`) + + -If you need to customize any values, create a `terraform.tfvars` file with your overrides. +Load variables from `deployment.conf` (`set -a` enables auto-export of all variables): -### Step 3: Deploy the State Backend +```bash +set -a && source ../deployment.conf && set +a +``` Initialize Terraform and deploy the storage bucket: ```bash -# Initialize Terraform providers terraform init +terraform plan -out=tfplan +terraform apply tfplan +``` -# Preview the resources that will be created -terraform plan + + -# Create the GCS bucket -terraform apply +Create a `terraform.tfvars` file in the `remote-backend/` directory: + +```hcl +project_id = "your-gcp-project-id" +region = "europe-west3" +storage_bucket_name = "codemie-terraform-states" + +# Optional: Custom labels +labels = { + "sys_name" = "ai_run" + "environment" = "development" + "project" = "ai_run" +} ``` -When prompted, type `yes` to confirm the deployment. +Initialize Terraform and deploy the storage bucket: -### Step 4: Verify Deployment +```bash +terraform init +terraform plan -out=tfplan +terraform apply tfplan +``` -After successful deployment, verify the bucket was created: + + -```bash -# Check Terraform outputs -terraform output +3. After successful deployment, note the bucket name from Terraform outputs: -# Or verify via gcloud CLI -gcloud storage buckets list | grep terraform +```bash +export BACKEND_BUCKET=$(terraform output -raw terraform_states_storage_bucket_name) +echo "Backend bucket: $BACKEND_BUCKET" ``` -**Save the bucket name** - you'll need it for Phase 2 configuration. - :::tip Next Phase The storage bucket is now ready. Proceed to Phase 2 to deploy the main platform infrastructure. ::: @@ -124,123 +136,110 @@ The storage bucket is now ready. Proceed to Phase 2 to deploy the main platform This phase deploys all core GCP resources required to run AI/Run CodeMie. This includes the GKE cluster, networking components, databases, and security infrastructure. -### Step 1: Clone the Repository - -Clone the platform infrastructure Terraform repository: +1. Navigate to the `platform/` directory: ```bash -git clone https://gitbud.epam.com/epm-cdme/codemie-terraform-gcp-platform.git -cd codemie-terraform-gcp-platform +cd ../platform ``` -### Step 2: Configure Remote State Backend +2. Configure and deploy. There are two ways to provide Terraform variables: -Configure Terraform to use the GCS bucket created in Phase 1 for storing infrastructure state. + + -Edit the `versions.tf` file and update the backend configuration: +Load variables from `deployment.conf`: -```hcl -backend "gcs" { - bucket = "your-bucket-name-from-phase1" # Replace with actual bucket name - prefix = "terraform/platform/state" # Path prefix for state files -} +```bash +set -a && source ../deployment.conf && set +a ``` -### Step 3: Review and Configure Variables - -The `variables.tf` file contains all configurable parameters for your infrastructure deployment. Review these variables and create a `terraform.tfvars` file to customize values for your environment. - -#### Create Configuration File - -Create a `terraform.tfvars` file in the repository root: +Initialize Terraform with backend configuration and deploy: ```bash -# Create and edit configuration file -nano terraform.tfvars +terraform init \ + -backend-config="bucket=${BACKEND_BUCKET}" \ + -backend-config="prefix=${TF_VAR_region}/codemie/platform_terraform.tfstate" + +terraform plan -out=tfplan +terraform apply tfplan ``` -#### Essential Variables + + -Configure these required variables in `terraform.tfvars`: +Create a `terraform.tfvars` file with your configuration: ```hcl # GCP Project Configuration -project_id = "your-gcp-project-id" # Your GCP project ID -platform_name = "codemie" # Platform identifier +project_id = "your-gcp-project-id" +platform_name = "codemie" -# Network Access Control +# Network Access Control (only used when private_cluster = true) bastion_members = [ - "group:devops@airun.example.com", # Grant Bastion access to specific users/groups + "group:devops@airun.example.com", "user:admin@airun.example.com" ] -# DNS Configuration -dns_name = "codemie-example-com" # DNS zone name (hyphens, no dots) -dns_domain = "codemie.airun.example.com." # Full domain with trailing dot +# DNS Configuration (only used when create_private_dns_zone = true) +dns_name = "codemie-example-com" +dns_domain = "codemie.airun.example.com." # GKE API Access (optional) extra_authorized_networks = [ { - cidr_block = "x.x.x.x/24" # Your office/VPN CIDR + cidr_block = "x.x.x.x/24" display_name = "Office Network" - }, - { - cidr_block = "x.x.x.x/24" # Additional network if needed - display_name = "VPN Network" } ] # Cluster Configuration -private_cluster = false # Set to true for completely private GKE cluster -create_private_dns_zone = false # Set to true if using private DNS +private_cluster = false +create_private_dns_zone = false ``` :::info Configuration References -For all available variables and their descriptions, see: +For all available variables and their descriptions, see `variables.tf` in the `platform/` directory. +::: -- **Example configuration**: [terraform.tfvars.example](https://gitbud.epam.com/epm-cdme/codemie-terraform-gcp-platform/-/blob/main/terraform.tfvars.example?ref_type=heads) -- **Variable definitions**: [variables.tf](https://gitbud.epam.com/epm-cdme/codemie-terraform-gcp-platform/-/blob/main/variables.tf?ref_type=heads) - ::: +Set the backend bucket and region: -### Step 4: Deploy Platform Infrastructure +```bash +export BACKEND_BUCKET="your-bucket-name-from-phase1" +export REGION="europe-west3" +``` -Initialize Terraform and deploy the complete platform infrastructure: +Initialize Terraform with backend configuration and deploy: ```bash -# Initialize Terraform and download providers/modules -terraform init +terraform init \ + -backend-config="bucket=${BACKEND_BUCKET}" \ + -backend-config="prefix=${REGION}/codemie/platform_terraform.tfstate" -# Review the execution plan -terraform plan - -# Deploy all infrastructure resources -terraform apply +terraform plan -out=tfplan +terraform apply tfplan ``` -### Step 5: Verify Deployment + + -After successful deployment, verify all resources were created correctly: +3. After successful deployment, verify all resources were created correctly: ```bash -# View Terraform outputs (includes important connection information) +# View Terraform outputs terraform output # Verify GKE cluster exists -gcloud container clusters list --project=your-project-id +gcloud container clusters list --project= # Check Cloud SQL instance -gcloud sql instances list --project=your-project-id - -# List created VPC networks -gcloud compute networks list --project=your-project-id +gcloud sql instances list --project= ``` -**Save the Terraform outputs** - they contain critical information needed for subsequent steps, including: +**Save the Terraform outputs** — they contain critical information needed for subsequent steps, including: - GKE cluster connection commands - Bastion Host SSH/RDP commands - Cloud SQL connection details -- Service account information :::tip Infrastructure Ready The GCP infrastructure deployment is now complete. You can proceed to configure cluster access or continue with components deployment. @@ -265,9 +264,7 @@ The Bastion Host is a secure jump server that provides access to your private GK Use SSH to connect to the Bastion Host for deploying and managing Kubernetes resources. -#### Step 1: Connect to Bastion Host - -Retrieve the SSH command from Terraform outputs and connect: +1. Retrieve the SSH command from Terraform outputs and connect: ```bash # Get the SSH connection command @@ -284,9 +281,7 @@ gcloud compute ssh bastion-vm --project=your-project --zone=europe-west3-a Ensure your user account is listed in `bastion_members` variable from Phase 2 configuration. Only authorized users can SSH into the Bastion Host. ::: -#### Step 2: Set Root Password (Required for RDP) - -After connecting via SSH, set a root password for later RDP access: +2. After connecting via SSH, set a root password for later RDP access: ```bash # Switch to root user @@ -300,9 +295,7 @@ passwd The root password you set here will be used to login via RDP. Make sure to remember it or store it securely. ::: -#### Step 3: Configure Kubectl Access - -Fetch GKE cluster credentials to enable kubectl commands: +3. Fetch GKE cluster credentials to enable kubectl commands: ```bash # Get the kubectl configuration command @@ -315,12 +308,9 @@ terraform output get_kubectl_credentials_for_private_cluster gcloud container clusters get-credentials your-cluster-name --region=europe-west3 --project=your-project ``` -#### Step 4: Clone Deployment Repository - -Clone the Helm charts repository needed for component deployment: +4. Clone the Helm charts repository needed for component deployment: ```bash -# Clone the repository git clone https://gitbud.epam.com/epm-cdme/codemie-helm-charts.git cd codemie-helm-charts ``` @@ -335,9 +325,7 @@ Use RDP to access application web interfaces that are only available via private RDP is useful when you need to access web-based administrative interfaces that aren't exposed publicly. For kubectl/Helm operations, SSH access is sufficient. ::: -#### Step 1: Set Up RDP Port Forwarding - -Retrieve the RDP forwarding command from Terraform outputs: +1. Retrieve the RDP forwarding command from Terraform outputs and start the IAP tunnel: ```bash # Get the RDP forwarding command @@ -347,25 +335,22 @@ terraform output bastion_rdp_command # gcloud compute start-iap-tunnel bastion-vm 3389 --local-host-port=localhost:3389 --zone=europe-west3-a --project=your-project ``` -Run the command to create an IAP tunnel that forwards RDP traffic: +Run the command to create an IAP tunnel that forwards RDP traffic (keep this terminal open): ```bash -# Start the IAP tunnel (keep this terminal open) gcloud compute start-iap-tunnel bastion-vm 3389 \ --local-host-port=localhost:3389 \ --zone=europe-west3-a \ --project=your-project ``` -#### Step 2: Connect with Remote Desktop Client - -Open your Remote Desktop client and connect: +2. Open your Remote Desktop client and connect: | Setting | Value | | ------------ | -------------------------- | | **Computer** | `localhost:3389` | | **Username** | `root` | -| **Password** | Password set in SSH Step 2 | +| **Password** | Password set in SSH step 2 | ### Tips for Using the Bastion Host diff --git a/docs/admin/deployment/gcp/04-infrastructure-deployment/scripted-deployment.md b/docs/admin/deployment/gcp/04-infrastructure-deployment/scripted-deployment.md new file mode 100644 index 00000000..df66cb56 --- /dev/null +++ b/docs/admin/deployment/gcp/04-infrastructure-deployment/scripted-deployment.md @@ -0,0 +1,169 @@ +--- +id: infrastructure-scripted-deployment +title: Infrastructure Scripted Deployment +sidebar_label: Scripted Deployment +sidebar_position: 1 +pagination_prev: admin/deployment/gcp/infrastructure-deployment/infrastructure-deployment-overview +pagination_next: admin/deployment/gcp/components-deployment/components-deployment-overview +--- + +# Scripted Infrastructure Deployment + +This guide walks you through deploying GCP infrastructure for AI/Run CodeMie using the automated `gcp-terraform.sh` deployment script. The script handles all deployment phases automatically: Terraform state backend and core platform infrastructure. + +:::tip Recommended Approach +Scripted deployment is the recommended method as it handles prerequisite checks, configuration validation, and proper sequencing of Terraform operations automatically. +::: + +## Prerequisites + +Before starting the deployment, ensure you have completed all requirements from the [Prerequisites](../prerequisites) page: + +### Verification Checklist + +- [ ] **GCP Access**: Project Owner or Editor role with IAM permissions +- [ ] **Required APIs Enabled**: Cloud IAP, Service Networking, Secret Manager, Vertex AI APIs +- [ ] **Tools Installed**: tfenv, Terraform 1.13.5, gcloud CLI, kubectl, Helm, Docker +- [ ] **GCP Authentication**: Logged in with gcloud CLI and application default credentials configured +- [ ] **Repository Access**: Have access to Terraform and Helm repositories +- [ ] **Network Planning**: Prepared list of authorized networks (if accessing GKE API from workstation) + +:::warning Authentication Required +You must be authenticated to GCP CLI before running Terraform. Run `gcloud auth login` and `gcloud auth application-default login`. Verify the active project with `gcloud config get-value project`. +::: + +## Deployment Phases + +The script automatically deploys infrastructure in sequential phases: + +| Phase | Description | Required | +| ------------------------------------ | ---------------------------------------------------------------------------------- | -------- | +| **Phase 1: State Backend** | Creates GCS bucket for Terraform state files | Yes | +| **Phase 2: Platform Infrastructure** | Deploys GKE, networking, storage, databases, security components, and Bastion Host | Yes | + +## Step 1: Clone Platform Repository + +Clone the platform Terraform repository: + +```bash +git clone https://gitbud.epam.com/epm-cdme/codemie-terraform-gcp-platform.git +cd codemie-terraform-gcp-platform +``` + +## Step 2: Configure Platform Deployment + +Edit `deployment.conf` file to provide your GCP-specific configuration: + +```bash +# GCP project ID where the platform will be deployed +TF_VAR_project_id="my-gcp-project-id" + +# GCP region where the platform will be deployed +TF_VAR_region="europe-west3" + +# GCS bucket name prefix for Terraform remote state storage +TF_VAR_storage_bucket_name="codemie-terraform-states" + +TF_VAR_labels='{"sys_name":"ai_run","environment":"development","project":"ai_run"}' + +# Unique platform identifier used in resource naming +TF_VAR_platform_name="codemie" + +# Whether to create a private GKE cluster +# Setting to true will also deploy a bastion host for cluster access +TF_VAR_private_cluster=false + +# Users/groups who can access the bastion host via IAP (only used when private_cluster = true) +# Format: ["user:email@domain.com", "group:group@domain.com"] +TF_VAR_bastion_members='["user:user@example.com"]' + +# Machine type for worker nodes +TF_VAR_node_pool_machine_type="e2-standard-8" +TF_VAR_node_pool_min_count=2 +TF_VAR_node_pool_max_count=3 + +# Additional networks authorized to access the GKE cluster API server +TF_VAR_extra_authorized_networks='[]' +``` + +:::info Complete Variable List +For all available configuration options, refer to the `platform/terraform.tfvars.example` file in the repository. +::: + +## Step 3: Run Deployment Script + +Execute the automated deployment script: + +```bash +bash ./gcp-terraform.sh +``` + +The script will automatically execute the following operations: + +1. **Validate Configuration**: Check `deployment.conf` for required variables +2. **Verify Prerequisites**: Check for required tools (tfenv, terraform, gcloud CLI) +3. **Check Terraform Version**: Ensure correct Terraform version via tfenv +4. **Verify GCP Authentication**: Validate active gcloud session and application default credentials +5. **Deploy State Backend**: Create GCS bucket for Terraform state storage (`remote-backend/`) +6. **Deploy Platform Infrastructure**: Provision core GCP infrastructure (`platform/`) +7. **Generate Outputs**: Create `deployment_outputs.env` with infrastructure details required for the next phases + +## Deployment Outputs + +Upon successful deployment, the script generates a `deployment_outputs.env` file containing essential infrastructure details needed for the next deployment phase: + +```bash +# GKE Cluster +GKE_CLUSTER_NAME=codemie-gke +GKE_LOCATION=europe-west3 +GKE_NETWORK=codemie-cluster-network +GKE_SUBNET=codemie-cluster-subnet +KUBECTL_COMMAND=gcloud container clusters get-credentials --project my-gcp-project --zone europe-west3 codemie-gke + +# PostgreSQL +CODEMIE_POSTGRES_DATABASE_HOST=10.60.0.11 +CODEMIE_POSTGRES_DATABASE_PORT=5432 +CODEMIE_POSTGRES_DATABASE_NAME=codemie +CODEMIE_POSTGRES_DATABASE_USER=admin +CODEMIE_POSTGRES_DATABASE_INSTANCE=codemie-postgresql +CODEMIE_POSTGRES_DATABASE_SECRET=codeemiePGDB +CODEMIE_POSTGRES_DATABASE_PASSWORD=generated-password + +# GCP +VERTEX_PROJECT=my-gcp-project +``` + +:::tip Save These Outputs +The `deployment_outputs.env` file contains sensitive information. Store it securely, do not commit to version control and reference it during the Components Deployment phase. +::: + +## Post-Deployment Validation + +After deployment completes, verify that all infrastructure was created successfully: + +### Step 1: Verify GCP Resources + +```bash +# Verify GKE cluster status +gcloud container clusters list --project= + +# Check Cloud SQL instance +gcloud sql instances list --project= + +# Verify GCS state bucket +gcloud storage buckets list | grep terraform +``` + +### Step 2: Check Deployment Logs + +Review the deployment logs in the `logs/` directory for any warnings or errors: + +```bash +less logs/codemie_gcp_deployment_YYYY-MM-DD-HHMMSS.log +``` + +## Next Steps + +After successful infrastructure deployment and validation, proceed to: + +**[Components Deployment](../components-deployment/)** - Deploy AI/Run CodeMie application components to your GKE cluster diff --git a/sidebars.ts b/sidebars.ts index fd1e8715..9ef9a006 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -452,6 +452,7 @@ const sidebars: SidebarsConfig = { }, collapsed: true, items: [ + 'admin/deployment/gcp/infrastructure-deployment/infrastructure-scripted-deployment', 'admin/deployment/gcp/infrastructure-deployment/infrastructure-manual-deployment', ], },