grafana admin login with default password not working with COS stack deployed with Terraform

[codyshepherd]

CPC has encountered an issue with the grafana application in the COS HA stack deployed in PS6 where grafana returns 401 errors when attempting to log in with the admin username and the default secret created during the course of deployment of the stack.

Steps to reproduce: deploy the COS stack using terraform with the following as a main.tf:

terraform {
        required_version = ">= 1.5"
        required_providers {
                juju = {
                        source  = "juju/juju"
                        version = "~> 1.0"
                }
        }
}

data "juju_model" "my-model" {
        name  = "<model name>"
        owner = "admin"
}

variable "s3_secret_key" {
        default = "NO_VALUE_SET"
}

variable "s3_access_key" {
        default = "NO_VALUE_SET"
}

module "cos" {
        source                          = "git::https://github.com/canonical/observability-stack//terraform/cos"
        model_uuid                      = data.juju_model.my-model.uuid
        channel                         = "2/stable"
        anti_affinity                   = false
        internal_tls                    = false
        external_certificates_offer_url = null
        s3_endpoint                     = "https://radosgw.ps6.canonical.com"
        s3_secret_key                   = var.s3_secret_key
        s3_access_key                   = var.s3_access_key
        ssc                             = { channel = "1/stable" }
        traefik                         = { channel = "latest/stable" }
        loki_bucket                     = "<loki bucket name>"
        mimir_bucket                    = "<mimir bucket name>"
        tempo_bucket                    = "<tempo bucket name>"
}

I've confirmed that the juju secret, grafana secret (from inside the container), and the output of the get-admin-password all match. I've also tried resetting the password by sshing to the container and running the grafana tool to reset the password.