Skip to content

develop.md

Latest commit

 

History

History
82 lines (54 loc) · 4.71 KB

develop.md

File metadata and controls

82 lines (54 loc) · 4.71 KB

Development Guide

This guide provides an overview of the Cortex development process, including setting up the development environment, developing plugins, and testing the service.

Repository structure

Cortex' repository is structured into multiple distinct directories to separate concerns. The following snippet provides an overview:

.
├── api/
│   ├── <version>/     # CRD definitions and webhooks
│   └── delegation/    # Delegation API definitions (external scheduler API)
├── config/            # Helm and Kustomize bases, such as CRDs
├── dist/              # Helm chart generated by kubebuilder helm plugin from bases in config/
├── bundles/           # Umbrella Helm charts for easily deploying Cortex
├── internal/          # Project-internal code
│   ├── scheduling     # Scheduling & workload placement related code
│   ├── reservations   # Resource reservations related code
│   └── knowledge      # Knowledge Database related code
├── pkg/               # Reusable/public library code
├── tools/             # Miscellaneous tools and scripts
└── cmd/               # Binary entrypoints

Golang Tooling, Testing, Linting

Cortex is developed using the Go programming language. To get started with the development, follow these steps:

Install Golang: Ensure you have Go installed on your machine by following the instructions here.

Install golangci-lint: Cortex uses golangci-lint for linting the code. You can install it by following the instructions here.

Run make in your terminal from the cortex root directory to perform linting and testing tasks.

Working on Tests

# Watch mode for continuous testing; print logs for failed tests only
make testsum WATCH=1

The testsum target provides cleaner output by showing only full verbose output for failing tests.

Available options:

  • WATCH=1 - Automatically re-run tests when files change
  • RUN=<pattern> - Run specific tests matching the pattern
  • PACKAGE=<pkg> - Test specific package(s)
  • FORMAT=<fmt> - Change output format (e.g., standard-verbose for verbose output on all tests)

Helm Charts

Helm charts bundle the application into a package, containing all the Kubernetes resources needed to run the application. The configuration for the application is specified in the Helm values.yaml.

Read the helm chart structure documentation for more information about the structure of the Helm charts used in this repository.

For local development, use the cortex.secrets.example.yaml file to override the default Helm values. You can write your OpenStack credentials here to include credentials, such as SSO certificates to access Prometheus metrics or OpenStack credentials to authenticate with Keystone.

Tilt Setup

Tilt acts like a makefile/dockerfile for Kubernetes services, allowing you to spin up something in a local Kubernetes cluster with all the services needed to run Cortex. Which kubernetes cluster you use is up to you; common choices are Minikube or Kind.

You need to tell Tilt where to find your custom configuration file. First, export the path of your secrets file with export TILT_VALUES_PATH="<path-to-cortex.secrets.yaml>". Then, start the cluster and the tilt setup:

export TILT_VALUES_PATH="${HOME}/cortex.secrets.yaml" && tilt up

The service will be accessible at http://localhost:10350/.

Prometheus Metrics and Alerts

Prometheus is used for monitoring and alerting.

The Tilt setup comes with a Prometheus operator that runs a Prometheus instance for local testing. You can access the Prometheus dashboard by clicking on Prometheus in the Tilt dashboard. From there you can explore the metrics collected from the various Cortex services.

Dashboards

The Tilt setup includes a Plutono instance, a fork of Grafana used to display dashboards for the services. Documentation for Plutono can be found here.

By using the Tilt dashboard, you can click on Plutono and then access the dashboards link. You can modify the dashboards as needed to display the metrics you want, and from the Plutono dashboard, you can save your progress by exporting the updated dashboard to JSON file.