OIDC The Hard way - Mirecloud Home lab Part 3

#devops #kubernetes #security #tutorial

Eliminating password databases: OpenID Connect, front-channel vs. back-channel, role mapping, and the end of local authentication.
Overview

Parts 1 and 2 built the foundation: Vault manages all credentials, External Secrets Operator bridges them into Kubernetes, cert-manager automates TLS, and Keycloak runs as a production-grade identity provider with clustered session state.
Part 3 is where that infrastructure proves its value: integrating Grafana with Keycloak via OpenID Connect to eliminate Grafana's native login form entirely. By the end, there is no Grafana password database. No local admin account. Every login redirects to Keycloak, authenticates against the central identity layer, and maps realm roles to Grafana permissions automatically.
The deliverables:
Understanding the OIDC Authorization Code Flow
Configuring Keycloak as an Identity Provider (IdP)
Configuring Grafana as a Relying Party (RP)
Managing the client secret through Vault and ESO
Front-channel vs. back-channel URL configuration (the detail most guides get wrong)
Role mapping via JMESPath expressions

A Primer on OpenID Connect

Before diving into YAML, it is worth understanding what OpenID Connect actually does - because every configuration decision that follows is a direct consequence of how the protocol works.
The Problem It Solves

Without SSO, every service in your cluster has its own user database, its own password policy, its own session management. Add a user, you add them five times. Rotate a password, you rotate it five times. An employee leaves, you hope you remembered to revoke access in all five places.
OpenID Connect (OIDC) is an identity layer built on top of OAuth 2.0. It defines a standard protocol by which an application (the Relying Party, e.g., Grafana) can delegate authentication to a trusted external service (the Identity Provider, e.g., Keycloak). The application never handles passwords. It only receives a verified identity token.
The Authorization Code Flow

This is the flow used by Grafana when a user attempts to log in:
Step-by-step breakdown:
User navigates to Grafana → GET /
Grafana redirects to Keycloak → 302 with auth_url
Browser follows redirect to Keycloak → GET /auth/realms/mirecloud/protocol/openid-connect/auth
Keycloak renders login form → User sees username/password fields
User submits credentials → POST to Keycloak (Grafana never sees this)
Keycloak redirects back to Grafana → 302 with code=AUTH_CODE
Browser follows redirect to Grafana callback → GET /login/generic_oauth?code=...

From here, the flow switches to back-channel (server-to-server, no browser involved):
Grafana exchanges code for tokens (back-channel) → POST /token
Keycloak returns tokens → { access_token, id_token }
Grafana requests user info (back-channel) → GET /userinfo
Keycloak returns user claims → { sub, email, realm_access.roles }
Grafana creates session → Sets grafana_session cookie

Front-Channel vs. Back-Channel

The diagram reveals a critical distinction that most tutorials ignore:
Front-channel calls travel through the user's browser as HTTP redirects. The auth_url is a front-channel URL - the browser navigates to it directly. It must be publicly reachable: https://keycloak.mirecloud.com/...
Back-channel calls are made directly between Grafana's pod and Keycloak's pod, inside the Kubernetes cluster. The browser is not involved. These are the token exchange ( token_url) and user info ( api_url) calls.
This is why token_url in the Grafana configuration uses the internal Kubernetes service DNS name ( keycloak-keycloakx-http.keycloak.svc.cluster.local) rather than the public hostname.
Key Concepts

Step 1 - Configure Keycloak (One-Time Setup)

Create a Realm

Navigate to the Keycloak admin console → Create Realm.
Create a Client for Grafana

Inside the mirecloud realm, navigate to Clients → Create Client.
General Settings: Capability config:
Client authentication: ON
Authentication flow: Enable "Standard flow"

Navigate to Clients → grafana → Credentials tab. Copy the Client Secret and store it in Vault:
kubectl -n vault exec -ti vault-0 -- vault kv put secret/grafana/sso \ client_secret=''

Step 2 - ExternalSecret for Grafana

apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: name: grafana-keycloak-es namespace: monitoring spec: refreshInterval: 1m secretStoreRef: name: vault-backend kind: ClusterSecretStore target: name: grafana-keycloak-secret data: - secretKey: client_secret remoteRef: key: secret/grafana/sso property: client_secret

Step 3 - Grafana OIDC Configuration

kube-prometheus-stack: grafana: enabled: true envFromSecret: grafana-keycloak-secret grafana.ini: auth.generic_oauth: enabled: true name: "Keycloak" client_id: "grafana" client_secret: $__env{client_secret} # Front-channel URL (browser navigates here) auth_url: "https://keycloak.mirecloud.com/auth/realms/mirecloud/protocol/openid-connect/auth" # Back-channel URLs (pod-to-pod) token_url: "http://keycloak-keycloakx-http.keycloak.svc.cluster.local:80/auth/realms/mirecloud/protocol/openid-connect/token" api_url: "http://keycloak-keycloakx-http.keycloak.svc.cluster.local:80/auth/realms/mirecloud/protocol/openid-connect/userinfo" scopes: "openid profile email" allow_sign_up: true # Role mapping role_attribute_path: "contains(realm_access.roles[*], 'admin') && 'Admin' || 'Viewer'"

Configuration Breakdown

uses the public DNS name: https://keycloak.mirecloud.com/...
token_url and api_url use internal cluster DNS to avoid DNS hairpin issues in homelab environments.
Test the OIDC Flow

Navigate to https://grafana.mirecloud.com.
Click Sign in with Keycloak.
The browser redirects to Keycloak. Enter credentials for a user in the mirecloud realm.
Grafana exchanges the authorization code for tokens (back-channel, invisible to you) and creates a session.
You land on the Grafana dashboard. Your role (Admin or Viewer) is determined by the admin realm role assignment.
Security Posture

No Grafana password database - all authentication delegated to Keycloak
Client secret managed through Vault and ESO - never visible in Git
OIDC tokens transmitted securely (TLS on front-channel, internal service mesh for back-channel)
Role assignment driven by Keycloak realm roles - access control changes do not require Grafana restarts

What's Next: Part 4

Part 4 will cover GitLab OIDC configuration with discovery: false, explicit OAuth endpoint definition, and CA injection.
The complete repository is available at github.com/mirecloud/home_lab.
Emmanuel Catin - Senior Platform Engineer | Kubernetes, GitOps, Zero Trust
CKA (90%) | CKS in preparation | Montréal, QC