
Keyvault Csi Driver
- 91 installs
- 6 repo stars
- Updated July 22, 2026
- julianobarbosa/claude-code-skills
Helps with ai & agent building tasks.
About
keyvault-csi-driver is a Claude Code skill for ai & agent building. It helps solo builders move faster with AI-assisted development.
- keyvault-csi-driver
- AI & Agent Building
- AI-coding skill
Keyvault Csi Driver by the numbers
- 91 all-time installs (skills.sh)
- +1 installs in the week ending Aug 2, 2026 (Skillselion tracking)
- Ranked #4,798 of 16,546 AI & Agent Building skills by installs in the Skillselion catalog
- Data as of Aug 3, 2026 (Skillselion catalog sync)
npx skills add https://github.com/julianobarbosa/claude-code-skills --skill keyvault-csi-driverAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 91 |
|---|---|
| repo stars | ★ 6 |
| Last updated | July 22, 2026 |
| Repository | julianobarbosa/claude-code-skills ↗ |
What it does
Helps with ai & agent building tasks.
Files
Azure Key Vault CSI Driver Skill
Overview
This skill provides guidance for integrating Azure Key Vault with Kubernetes using the Secrets Store CSI Driver. All sensitive data in the Hypera clusters is stored in Azure Key Vault and accessed via the CSI driver.
Quick Reference
Environment Configuration
| Cluster | Key Vault | Managed Identity (Client ID) | Tenant ID |
|---|---|---|---|
| cafehyna-dev | kv-cafehyna-dev-hlg | f1a14a8f-6d38-40a0-a935-3cdd91a25f47 | 3f7a3df4-f85b-4ca8-98d0-08b1034e6567 |
| cafehyna-hub | kv-cafehyna-default | f1a14a8f-6d38-40a0-a935-3cdd91a25f47 | 3f7a3df4-f85b-4ca8-98d0-08b1034e6567 |
| cafehyna-prd | kv-cafehyna-prd | f1a14a8f-6d38-40a0-a935-3cdd91a25f47 | 3f7a3df4-f85b-4ca8-98d0-08b1034e6567 |
| painelclientes-dev | painel-clientes-hml | Check cluster identity | 3f7a3df4-f85b-4ca8-98d0-08b1034e6567 |
| painelclientes-prd | painel-clientes-prd | Check cluster identity | 3f7a3df4-f85b-4ca8-98d0-08b1034e6567 |
SecretProviderClass Template
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: <app>-secrets
namespace: <namespace>
labels:
app.kubernetes.io/name: <app>
app.kubernetes.io/component: secrets
spec:
provider: azure
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "true"
userAssignedIdentityID: "<managed-identity-client-id>"
keyvaultName: "<keyvault-name>"
cloudName: "AzurePublicCloud"
tenantId: "<tenant-id>"
objects: |
array:
- |
objectName: "<secret-name-in-keyvault>"
objectType: "secret"
objectAlias: "<ALIAS_FOR_MOUNT>"
# Optional: Sync to Kubernetes Secret
secretObjects:
- secretName: <k8s-secret-name>
type: Opaque
data:
- objectName: "<ALIAS_FOR_MOUNT>"
key: "<key-in-k8s-secret>"Pod Volume Mount
spec:
containers:
- name: app
volumeMounts:
- name: secrets-store
mountPath: "/mnt/secrets-store"
readOnly: true
volumes:
- name: secrets-store
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "<secretproviderclass-name>"Common Patterns
Pattern 1: Simple API Token (e.g., Cloudflare)
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: cloudflare-api-token-kv
namespace: external-dns
spec:
provider: azure
secretObjects:
- data:
- key: cloudflare_api_token
objectName: cloudflare-api-token
secretName: cloudflare-api-token
type: Opaque
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "true"
userAssignedIdentityID: "f1a14a8f-6d38-40a0-a935-3cdd91a25f47"
keyvaultName: "kv-cafehyna-dev-hlg"
objects: |
array:
- |
objectName: cloudflare-api-token
objectType: secret
tenantId: "3f7a3df4-f85b-4ca8-98d0-08b1034e6567"Pattern 2: Multiple Secrets to Multiple K8s Secrets
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: app-secrets
spec:
provider: azure
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "true"
userAssignedIdentityID: "<identity>"
keyvaultName: "<keyvault>"
tenantId: "<tenant>"
objects: |
array:
- |
objectName: "app-db-password"
objectType: "secret"
objectAlias: "DB_PASSWORD"
- |
objectName: "app-redis-password"
objectType: "secret"
objectAlias: "REDIS_PASSWORD"
secretObjects:
- secretName: app-db-secret
type: Opaque
data:
- objectName: "DB_PASSWORD"
key: "password"
- secretName: app-redis-secret
type: Opaque
data:
- objectName: "REDIS_PASSWORD"
key: "password"Pattern 3: TLS Certificate
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: tls-cert-provider
spec:
provider: azure
secretObjects:
- secretName: tls-secret
type: kubernetes.io/tls
data:
- objectName: tls-cert
key: tls.crt
- objectName: tls-key
key: tls.key
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "true"
userAssignedIdentityID: "<identity>"
keyvaultName: "<keyvault>"
tenantId: "<tenant>"
objects: |
array:
- |
objectName: my-certificate
objectType: cert
objectAlias: tls-cert
- |
objectName: my-certificate
objectType: secret
objectAlias: tls-keyFile Locations
SecretProviderClass files are stored in:
argo-cd-helm-values/kube-addons/<application>/<cluster>/secretproviderclass.yamlExamples:
argo-cd-helm-values/kube-addons/defectdojo/cafehyna-dev/secretproviderclass.yamlargo-cd-helm-values/kube-addons/external-dns/cafehyna-dev/secretproviderclass.yamlargo-cd-helm-values/kube-addons/cert-manager/cafehyna-dev/csi-cloudflare-api-key.yaml
Troubleshooting
Error: 403 Forbidden
Cause: Managed identity lacks Key Vault permissions.
Solution:
# Get identity info from error message, then:
az keyvault set-policy \
--name "<keyvault-name>" \
--object-id "<object-id-from-error>" \
--secret-permissions get list
# Or for RBAC-enabled Key Vaults:
az role assignment create \
--role "Key Vault Secrets User" \
--assignee-object-id "<object-id>" \
--assignee-principal-type ServicePrincipal \
--scope "/subscriptions/.../Microsoft.KeyVault/vaults/<kv-name>"Error: Secret Not Found
Cause: Secret name doesn't exist or case mismatch.
Solution:
# List secrets (names are case-sensitive)
az keyvault secret list --vault-name "<kv-name>" --query "[].name" -o tsvError: K8s Secret Not Created
Cause: No pod has mounted the CSI volume yet.
Solution: Deploy a pod that mounts the volume. K8s secrets are only created when at least one pod uses the SecretProviderClass.
Error: Pod Stuck in ContainerCreating
Diagnostic:
kubectl describe pod <pod-name> -n <namespace>
kubectl get pods -n kube-system | grep secrets-store
kubectl logs -n kube-system -l app=secrets-store-provider-azureScripts
Grant Key Vault Permissions
# Use the helper script
./scripts/grant-keyvault-permissions.sh
# Or quick manual command
az keyvault set-policy \
--name "kv-cafehyna-dev-hlg" \
--object-id "<object-id>" \
--secret-permissions get listCreate Secret in Key Vault
az keyvault secret set \
--vault-name "kv-cafehyna-dev-hlg" \
--name "my-app-secret" \
--value "secret-value"List All SecretProviderClasses
kubectl get secretproviderclass -ACheck CSI Driver Status
kubectl get pods -n kube-system | grep secrets-storeImportant Notes
1. CSI Volume Required: Even if using secretObjects to sync to K8s secrets, the pod MUST mount the CSI volume.
2. Secret Names: Key Vault secret names are case-sensitive. Use exact match.
3. Object Alias: Use objectAlias for filesystem-safe names when mounting.
4. Namespace Scope: SecretProviderClass is namespace-scoped. Create one per namespace that needs it.
5. RBAC vs Access Policies: Check Key Vault authorization model:
az keyvault show --name "<kv>" --query "properties.enableRbacAuthorization"Detailed Reference
For complete implementation examples and architecture:
- [references/architecture.md](references/architecture.md) - CSI driver architecture
- [references/examples.md](references/examples.md) - Real-world examples
- [references/troubleshooting.md](references/troubleshooting.md) - Extended troubleshooting
---
Gotchas
- K8s Secret doesn't exist until a pod mounts the CSI volume:
secretObjects:syncing is lazy — no pod, no Secret. Apps that consume the Secret directly (without mounting the CSI volume themselves) will fail on first deploy until a sibling pod mounts first. - Workload Identity vs VM Managed Identity are different code paths:
useVMManagedIdentity: "true"reads from IMDS; Workload Identity needsusePodIdentity: "false"+useVMManagedIdentity: "false"+ serviceAccount annotations. Mixing flags silently falls back to wrong identity. - Key Vault names are case-sensitive in `objects:`, K8s keys are not: Secret named
MyTokenin KV won't be found if you writemytokenin the SPC. The CSI logs say "secret not found" with no hint about casing. - RBAC vs Access Policy is per-vault, not per-tenant: Some vaults use access policies (
az keyvault set-policy), others use Azure RBAC (Key Vault Secrets Userrole). Checkproperties.enableRbacAuthorizationfirst — granting the wrong type returns 403 with identical error text. - TLS certs need two `objectType` entries: Mounting a cert as
tls.crt+tls.keyrequiresobjectType: cert(public) ANDobjectType: secret(full PEM with private key) — both pointing to the same KV cert name. Missing one breaks Ingress TLS silently. - CSI rotation is opt-in and deploy-only by default: Secrets mounted into pods don't refresh when the KV value changes unless
--enable-secret-rotation=trueis set on the driver AND the pod is restarted. Long-lived pods serve stale secrets.
CSI Driver Architecture Reference
Component Overview
┌─────────────────────────────────────────────────────────────────────────────┐
│ Secrets Store CSI Driver Stack │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────────────────┐│
│ │ Control Plane ││
│ │ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ ││
│ │ │ SecretProvider │ │ K8s API Server │ │ ArgoCD │ ││
│ │ │ Class (CRD) │ │ (Secret Sync) │ │ (GitOps Deploy) │ ││
│ │ └────────┬─────────┘ └────────▲─────────┘ └──────────────────┘ ││
│ │ │ │ ││
│ └───────────│──────────────────────│───────────────────────────────────────┘│
│ │ │ │
│ ┌───────────▼──────────────────────│───────────────────────────────────────┐│
│ │ Node Level ││
│ │ ┌──────────────────────────────────────────────────────────────────┐ ││
│ │ │ secrets-store-csi-driver (DaemonSet) │ ││
│ │ │ - Runs on every node │ ││
│ │ │ - Handles CSI volume mount requests │ ││
│ │ │ - Syncs mounted secrets to K8s Secrets (if configured) │ ││
│ │ │ - Manages secret rotation │ ││
│ │ └────────────────────────────┬─────────────────────────────────────┘ ││
│ │ │ ││
│ │ ┌────────────────────────────▼─────────────────────────────────────┐ ││
│ │ │ secrets-store-csi-driver-provider-azure (DaemonSet) │ ││
│ │ │ - Runs on every node alongside CSI driver │ ││
│ │ │ - Communicates with Azure Key Vault │ ││
│ │ │ - Handles authentication via Managed Identity │ ││
│ │ │ - Retrieves secrets/keys/certificates │ ││
│ │ └────────────────────────────┬─────────────────────────────────────┘ ││
│ │ │ ││
│ └───────────────────────────────│──────────────────────────────────────────┘│
│ │ │
│ ┌───────────────────────────────▼──────────────────────────────────────────┐│
│ │ Azure Services ││
│ │ ┌──────────────────┐ ┌──────────────────────────┐ ││
│ │ │ Azure Key Vault │◄────────────────────│ Azure Managed Identity │ ││
│ │ │ - Secrets │ Authentication │ - User-Assigned │ ││
│ │ │ - Keys │ │ - Or Workload Identity │ ││
│ │ │ - Certificates │ │ │ ││
│ │ └──────────────────┘ └──────────────────────────┘ ││
│ └──────────────────────────────────────────────────────────────────────────┘│
│ │
└─────────────────────────────────────────────────────────────────────────────┘Request Flow
1. Pod Creation with CSI Volume
┌──────────┐ ┌──────────┐ ┌──────────────┐ ┌──────────────┐ ┌─────────┐
│ User │ │ Kubelet │ │ CSI Driver │ │ Azure │ │ Azure │
│ (Deploy) │ │ │ │ │ │ Provider │ │ KeyVlt │
└────┬─────┘ └────┬─────┘ └──────┬───────┘ └──────┬───────┘ └────┬────┘
│ │ │ │ │
│ Create Pod │ │ │ │
│ with CSI Vol │ │ │ │
│───────────────>│ │ │ │
│ │ │ │ │
│ │ NodeStageVolume │ │ │
│ │──────────────────>│ │ │
│ │ │ │ │
│ │ │ Get Secrets │ │
│ │ │───────────────────>│ │
│ │ │ │ │
│ │ │ │ Auth + Fetch │
│ │ │ │─────────────────>│
│ │ │ │ │
│ │ │ │<─────────────────│
│ │ │ │ Secrets │
│ │ │<───────────────────│ │
│ │ │ Secret Data │ │
│ │ │ │ │
│ │ │ Write to tmpfs │ │
│ │ │ at mount path │ │
│ │<──────────────────│ │ │
│ │ Volume Ready │ │ │
│ │ │ │ │
│ │ (Optional) Sync │ │ │
│ │ to K8s Secret │ │ │
│ │ │ │ │
│<───────────────│ │ │ │
│ Pod Running │ │ │ │
│ │ │ │ │2. Secret Rotation Flow
┌──────────────┐ ┌──────────────┐ ┌─────────┐
│ CSI Driver │ │ Azure │ │ Azure │
│ (Rotation │ │ Provider │ │ KeyVlt │
│ Reconciler)│ │ │ │ │
└──────┬───────┘ └──────┬───────┘ └────┬────┘
│ │ │
│ Poll Interval │ │
│ (default: 2m) │ │
│ │ │
│ Check for Updates │ │
│───────────────────>│ │
│ │ │
│ │ Fetch Latest │
│ │──────────────────>│
│ │ │
│ │<──────────────────│
│ │ Secret (v2) │
│<───────────────────│ │
│ New Version │ │
│ │ │
│ Update mounted │ │
│ files in-place │ │
│ │ │
│ (If secretObjects) │ │
│ Update K8s Secret │ │
│ │ │CRD Structure: SecretProviderClass
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: example # Referenced by pod volume
namespace: default # Must match pod namespace
spec:
provider: azure # Provider type
parameters: # Provider-specific configuration
# Authentication
usePodIdentity: "false"
useVMManagedIdentity: "true"
userAssignedIdentityID: "client-id-of-managed-identity"
# Key Vault Configuration
keyvaultName: "vault-name"
cloudName: "AzurePublicCloud" # Or AzureUSGovernment, AzureChinaCloud
tenantId: "tenant-id"
# Objects to retrieve
objects: |
array:
- |
objectName: "secret-name" # Name in Key Vault
objectType: "secret" # secret, key, or cert
objectAlias: "MOUNT_NAME" # Optional: filename when mounted
objectVersion: "" # Optional: specific version
objectEncoding: "utf-8" # Optional: base64, hex, utf-8
filePermission: "0644" # Optional: file permissions
# Optional: Create K8s Secrets from mounted content
secretObjects:
- secretName: k8s-secret-name # Name of K8s Secret to create
type: Opaque # Secret type
data:
- objectName: "MOUNT_NAME" # Must match objectAlias above
key: "key-in-secret" # Key name in K8s SecretAuthentication Methods
User-Assigned Managed Identity (Recommended)
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "true"
userAssignedIdentityID: "<client-id>" # NOT object-idHow it works:
1. AKS cluster has a user-assigned managed identity attached to node VMSS 2. Provider uses this identity to authenticate to Key Vault 3. Identity must have Key Vault access policies or RBAC role
Workload Identity (Modern Alternative)
parameters:
usePodIdentity: "false"
clientID: "<managed-identity-client-id>"Requirements:
- AKS cluster with OIDC issuer enabled
- Workload identity enabled
- Federated credential configured
- Service account annotated
Pod Identity (Deprecated)
parameters:
usePodIdentity: "true"Note: Pod Identity is deprecated as of October 2022. Use Workload Identity instead.
Resource Requirements
CSI Driver DaemonSet
resources:
requests:
cpu: 50m
memory: 100Mi
limits:
cpu: 200m
memory: 200MiAzure Provider DaemonSet
resources:
requests:
cpu: 50m
memory: 100Mi
limits:
cpu: 200m
memory: 200MiPorts and Endpoints
| Component | Port | Purpose |
|---|---|---|
| CSI Driver | 9808 | Liveness probe |
| CSI Driver | 8095 | Metrics |
| Azure Provider | 8989 | Health probe |
| Azure Provider | 8898 | Metrics |
Metrics Available
CSI Driver Metrics (port 8095)
csi_operations_seconds- CSI operation latencytotal_node_publish- Volume publish operationstotal_node_unpublish- Volume unpublish operationstotal_sync_k8s_secret- K8s secret sync operations
Azure Provider Metrics (port 8898)
keyvault_request- Key Vault request countkeyvault_request_duration_seconds- Request latencytotal_rotation_reconcile- Rotation reconciliation counttotal_rotation_reconcile_error- Rotation errors
Real-World Examples
Example 1: External-DNS Cloudflare API Token
Location: argo-cd-helm-values/kube-addons/external-dns/cafehyna-dev/secretproviderclass.yaml
Key Vault Secret: cloudflare-api-token
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: cloudflare-api-token-kv
namespace: external-dns
spec:
provider: azure
secretObjects:
- data:
- key: cloudflare_api_token
objectName: cloudflare-api-token
secretName: cloudflare-api-token
type: Opaque
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "true"
userAssignedIdentityID: "f1a14a8f-6d38-40a0-a935-3cdd91a25f47"
keyvaultName: "kv-cafehyna-dev-hlg"
objects: |
array:
- |
objectName: cloudflare-api-token
objectType: secret
tenantId: "3f7a3df4-f85b-4ca8-98d0-08b1034e6567"Helm Values Usage:
# values.yaml
env:
- name: CF_API_TOKEN
valueFrom:
secretKeyRef:
name: cloudflare-api-token
key: cloudflare_api_token
extraVolumes:
- name: secrets-store
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: cloudflare-api-token-kv
extraVolumeMounts:
- name: secrets-store
mountPath: /mnt/secrets-store
readOnly: true---
Example 2: DefectDojo Multi-Secret Configuration
Location: argo-cd-helm-values/kube-addons/defectdojo/cafehyna-dev/secretproviderclass.yaml
Key Vault Secrets:
defectdojo-admin-passworddefectdojo-secret-keydefectdojo-credential-aes-keydefectdojo-metrics-passworddefectdojo-postgresql-passworddefectdojo-postgresql-postgres-passworddefectdojo-redis-passworddefectdojo-azuread-client-secret
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: defectdojo-secrets
namespace: monitoring
labels:
app.kubernetes.io/name: defectdojo
app.kubernetes.io/component: secrets
spec:
provider: azure
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "true"
userAssignedIdentityID: "f1a14a8f-6d38-40a0-a935-3cdd91a25f47"
keyvaultName: "kv-cafehyna-dev-hlg"
cloudName: "AzurePublicCloud"
tenantId: "3f7a3df4-f85b-4ca8-98d0-08b1034e6567"
objects: |
array:
- |
objectName: "defectdojo-admin-password"
objectType: "secret"
objectAlias: "DD_ADMIN_PASSWORD"
- |
objectName: "defectdojo-secret-key"
objectType: "secret"
objectAlias: "DD_SECRET_KEY"
- |
objectName: "defectdojo-credential-aes-key"
objectType: "secret"
objectAlias: "DD_CREDENTIAL_AES_256_KEY"
- |
objectName: "defectdojo-metrics-password"
objectType: "secret"
objectAlias: "METRICS_HTTP_AUTH_PASSWORD"
- |
objectName: "defectdojo-postgresql-password"
objectType: "secret"
objectAlias: "postgresql-password"
- |
objectName: "defectdojo-postgresql-postgres-password"
objectType: "secret"
objectAlias: "postgresql-postgres-password"
- |
objectName: "defectdojo-redis-password"
objectType: "secret"
objectAlias: "redis-password"
- |
objectName: "defectdojo-azuread-client-secret"
objectType: "secret"
objectAlias: "DD_SOCIAL_AUTH_AZUREAD_TENANT_OAUTH2_SECRET"
secretObjects:
# DefectDojo application secrets
- secretName: defectdojo
type: Opaque
data:
- objectName: "DD_ADMIN_PASSWORD"
key: "DD_ADMIN_PASSWORD"
- objectName: "DD_SECRET_KEY"
key: "DD_SECRET_KEY"
- objectName: "DD_CREDENTIAL_AES_256_KEY"
key: "DD_CREDENTIAL_AES_256_KEY"
- objectName: "METRICS_HTTP_AUTH_PASSWORD"
key: "METRICS_HTTP_AUTH_PASSWORD"
- objectName: "DD_SOCIAL_AUTH_AZUREAD_TENANT_OAUTH2_SECRET"
key: "DD_SOCIAL_AUTH_AZUREAD_TENANT_OAUTH2_SECRET"
# PostgreSQL secrets
- secretName: defectdojo-postgresql-specific
type: Opaque
data:
- objectName: "postgresql-password"
key: "postgresql-password"
- objectName: "postgresql-postgres-password"
key: "postgresql-postgres-password"
# Redis secrets
- secretName: defectdojo-redis-specific
type: Opaque
data:
- objectName: "redis-password"
key: "redis-password"Key Points:
- Uses
objectAliasto provide clear names - Creates 3 separate K8s secrets for different purposes
- Helm chart expects specific secret names (
defectdojo,defectdojo-postgresql-specific,defectdojo-redis-specific)
---
Example 3: Cert-Manager Cloudflare DNS01 Challenge
Location: argo-cd-helm-values/kube-addons/cert-manager/loyalty-dev/csi-cloudflare-api-key.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: cloudflare-api-key-csi
namespace: cert-manager
spec:
provider: azure
secretObjects:
- secretName: cloudflare-api-key
type: Opaque
data:
- objectName: api-token
key: api-token
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "true"
userAssignedIdentityID: "<cluster-managed-identity>"
keyvaultName: "kv-loyalty-qas"
tenantId: "3f7a3df4-f85b-4ca8-98d0-08b1034e6567"
objects: |
array:
- |
objectName: cloudflare-api-token
objectType: secret
objectAlias: api-tokenClusterIssuer Reference:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging-cloudflare
spec:
acme:
solvers:
- dns01:
cloudflare:
apiTokenSecretRef:
name: cloudflare-api-key
key: api-token---
Example 4: RabbitMQ Cluster Secrets
Location: infra-team/argocd/addons/rabbitmq-cluster-operator/overlays/painelclientes-prd/secretproviderclass-rabbitmq.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: rabbitmq-secrets
namespace: painelclientes-prd
spec:
provider: azure
secretObjects:
- secretName: rabbitmq-default-user
type: Opaque
data:
- objectName: username
key: username
- objectName: password
key: password
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "true"
userAssignedIdentityID: "<cluster-identity>"
keyvaultName: "painel-clientes-prd"
tenantId: "3f7a3df4-f85b-4ca8-98d0-08b1034e6567"
objects: |
array:
- |
objectName: rabbitmq-username
objectType: secret
objectAlias: username
- |
objectName: rabbitmq-password
objectType: secret
objectAlias: password---
Example 5: Robusta Monitoring Secrets
Location: infra-team/argocd/addons/robusta/base/secret-provider-class.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: robusta-secrets
namespace: robusta
spec:
provider: azure
secretObjects:
- secretName: robusta-credentials
type: Opaque
data:
- objectName: signing-key
key: ROBUSTA_SIGNING_KEY
- objectName: account-id
key: ROBUSTA_ACCOUNT_ID
- objectName: sink-token
key: ROBUSTA_SINK_TOKEN
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "true"
userAssignedIdentityID: "$(MANAGED_IDENTITY_CLIENT_ID)"
keyvaultName: "$(KEY_VAULT_NAME)"
tenantId: "$(TENANT_ID)"
objects: |
array:
- |
objectName: robusta-signing-key
objectType: secret
objectAlias: signing-key
- |
objectName: robusta-account-id
objectType: secret
objectAlias: account-id
- |
objectName: robusta-sink-token
objectType: secret
objectAlias: sink-tokenKustomize Overlay (dev):
# overlays/dev/secret-provider-patch.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: robusta-secrets
spec:
parameters:
userAssignedIdentityID: "f1a14a8f-6d38-40a0-a935-3cdd91a25f47"
keyvaultName: "kv-cafehyna-dev-hlg"
tenantId: "3f7a3df4-f85b-4ca8-98d0-08b1034e6567"---
Common Patterns Summary
Pattern: Helm Chart Without extraObjects Support
When a Helm chart doesn't support extraObjects, deploy SecretProviderClass as a separate manifest in the same directory:
argo-cd-helm-values/kube-addons/<app>/<cluster>/
├── values.yaml # Helm values
└── secretproviderclass.yaml # CSI configurationConfigure ApplicationSet to include both:
sources:
- chart: <chart-name>
helm:
valueFiles:
- $values/kube-addons/<app>/{{cluster}}/values.yaml
- repoURL: <values-repo>
path: kube-addons/<app>/{{cluster}}
directory:
include: "secretproviderclass.yaml"Pattern: Base + Overlay with Kustomize
For applications using Kustomize:
infra-team/argocd/addons/<app>/
├── base/
│ ├── kustomization.yaml
│ └── secret-provider-class.yaml
└── overlays/
├── dev/
│ ├── kustomization.yaml
│ └── secret-provider-patch.yaml
└── prd/
├── kustomization.yaml
└── secret-provider-patch.yamlPattern: Secret Sync Pod
When secrets need to be created before the main application starts:
apiVersion: v1
kind: Pod
metadata:
name: secret-sync
annotations:
argocd.argoproj.io/hook: PreSync
spec:
containers:
- name: sync
image: busybox
command: ["sleep", "5"]
volumeMounts:
- name: secrets-store
mountPath: /mnt/secrets-store
volumes:
- name: secrets-store
csi:
driver: secrets-store.csi.k8s.io
volumeAttributes:
secretProviderClass: <name>
restartPolicy: NeverTroubleshooting Reference
Quick Diagnostic Commands
# Check CSI driver pods
kubectl get pods -n kube-system -l app=secrets-store-csi-driver
# Check Azure provider pods
kubectl get pods -n kube-system -l app=secrets-store-provider-azure
# List all SecretProviderClasses
kubectl get secretproviderclass -A
# Check CSI driver logs
kubectl logs -n kube-system -l app=secrets-store-csi-driver --tail=100
# Check provider logs
kubectl logs -n kube-system -l app=secrets-store-provider-azure --tail=100
# Describe problematic pod
kubectl describe pod <pod-name> -n <namespace>
# Check synced secrets
kubectl get secret -n <namespace> | grep <expected-secret-name>Error Reference
Error: 403 Forbidden / Access Denied
Full Error:
MountVolume.SetUp failed for volume "secrets-store"
rpc error: code = Unknown desc = failed to mount secrets store objects
for pod <namespace>/<pod-name>, err: rpc error: code = Unknown desc =
failed to process mount request, error: keyvault.BaseClient#GetSecret:
Failure responding to request: StatusCode=403 -- Original Error:
autorest/azure: Service returned an error. Status=403 Code="Forbidden"
Message="The user, group or application... does not have secrets get
permission on key vault..."Cause: The managed identity doesn't have permission to access Key Vault secrets.
Solution Steps:
1. Extract identity info from error:
# Look for object_id and client_id in the error message2. Check Key Vault authorization model:
az keyvault show --name "<kv-name>" \
--query "properties.enableRbacAuthorization" -o tsv3. Grant permissions:
For Access Policies (enableRbacAuthorization=false):
az keyvault set-policy \
--name "<kv-name>" \
--object-id "<object-id-from-error>" \
--secret-permissions get listFor RBAC Authorization (enableRbacAuthorization=true):
az role assignment create \
--role "Key Vault Secrets User" \
--assignee-object-id "<object-id-from-error>" \
--assignee-principal-type ServicePrincipal \
--scope "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.KeyVault/vaults/<kv-name>"4. Restart the pod:
kubectl delete pod <pod-name> -n <namespace>---
Error: Secret Not Found
Full Error:
failed to get objectType:secret, objectName:<secret-name>, objectVersion::
keyvault.BaseClient#GetSecret: Failure responding to request:
StatusCode=404 -- Original Error: autorest/azure: Service returned an error.
Status=404 Code="SecretNotFound" Message="A secret with (name/id) <name> was not found"Cause: Secret doesn't exist in Key Vault or name is incorrect (case-sensitive).
Solution Steps:
1. List secrets in Key Vault:
az keyvault secret list --vault-name "<kv-name>" \
--query "[].name" -o tsv2. Check exact name (case-sensitive):
az keyvault secret show --vault-name "<kv-name>" \
--name "<exact-secret-name>"3. Create secret if missing:
az keyvault secret set \
--vault-name "<kv-name>" \
--name "<secret-name>" \
--value "<secret-value>"4. Verify SecretProviderClass objectName matches exactly.
---
Error: Kubernetes Secret Not Created
Symptom: Pod starts successfully but expected K8s Secret doesn't exist.
Causes:
1. No pod has mounted the CSI volume yet 2. secretObjects configuration is wrong 3. objectName in secretObjects.data doesn't match objectAlias (or objectName if no alias)
Diagnostic:
# Check if secret exists
kubectl get secret <expected-name> -n <namespace>
# Check SecretProviderClass configuration
kubectl get secretproviderclass <name> -n <namespace> -o yaml
# Verify pod has the CSI volume mounted
kubectl get pod <pod-name> -n <namespace> -o yaml | grep -A10 "volumes:"Solution:
1. Ensure at least one pod mounts the CSI volume 2. Verify objectName in secretObjects.data matches the alias:
parameters:
objects: |
array:
- |
objectName: "kv-secret-name"
objectAlias: "MY_ALIAS" # <-- This name
secretObjects:
- secretName: k8s-secret
data:
- objectName: "MY_ALIAS" # <-- Must match alias above
key: "secret-key"---
Error: Pod Stuck in ContainerCreating
Symptom: Pod stays in ContainerCreating state indefinitely.
Diagnostic:
# Check pod events
kubectl describe pod <pod-name> -n <namespace>
# Check CSI driver logs
kubectl logs -n kube-system -l app=secrets-store-csi-driver --tail=50
# Check provider logs
kubectl logs -n kube-system -l app=secrets-store-provider-azure --tail=50Common Causes:
1. CSI driver not running:
kubectl get pods -n kube-system | grep secrets-store
# Should see both driver and provider pods on each node2. SecretProviderClass not found:
kubectl get secretproviderclass <name> -n <namespace>3. Network connectivity to Key Vault:
- Check if VPN is connected
- Verify private endpoint configuration (if used)
4. Wrong namespace:
- SecretProviderClass must be in the same namespace as the pod
---
Error: Wrong Managed Identity Used
Symptom: Getting 403 errors even after granting permissions.
Cause: Multiple managed identities exist, and the wrong one is being used.
Diagnostic:
# Get cluster's kubelet identity (used by CSI driver)
az aks show -g <rg> -n <cluster> \
--query "identityProfile.kubeletidentity.clientId" -o tsv
# Compare with userAssignedIdentityID in SecretProviderClass
kubectl get secretproviderclass <name> -n <namespace> -o yaml | grep userAssignedIdentityIDSolution: Update userAssignedIdentityID to match the cluster's kubelet identity, or grant permissions to the identity that's actually being used.
---
Error: Secret Rotation Not Working
Symptom: Secrets in Key Vault were updated but pods still see old values.
Diagnostic:
# Check if rotation is enabled on cluster
az aks show -g <rg> -n <cluster> \
--query "addonProfiles.azureKeyvaultSecretsProvider.config.enableSecretRotation"
# Check rotation poll interval
az aks show -g <rg> -n <cluster> \
--query "addonProfiles.azureKeyvaultSecretsProvider.config.rotationPollInterval"
# Check provider logs for rotation activity
kubectl logs -n kube-system -l app=secrets-store-provider-azure | grep rotationSolution:
1. Enable rotation on cluster:
az aks addon update \
--resource-group <rg> \
--name <cluster> \
--addon azure-keyvault-secrets-provider \
--enable-secret-rotation \
--rotation-poll-interval 2m2. For environment variables from K8s Secrets, the pod must restart. Use Stakater Reloader:
metadata:
annotations:
reloader.stakater.com/auto: "true"3. For volume mounts, the application must re-read the file.
---
Error: Authentication Timeout
Full Error:
context deadline exceededCause: Network connectivity issue to Azure.
Solution:
1. Check VPN connection (for private clusters) 2. Verify DNS resolution:
kubectl run test --rm -it --image=busybox -- nslookup <keyvault-name>.vault.azure.net3. Check if private endpoint is configured correctly
---
Verification Checklist
Before Deployment
- [ ] Key Vault secret exists with correct name
- [ ] Managed identity has Key Vault permissions
- [ ] CSI driver pods are running on all nodes
- [ ] SecretProviderClass uses correct parameters
After Deployment
# 1. Check pod is running
kubectl get pod <name> -n <namespace>
# 2. Check secrets are mounted
kubectl exec <pod> -n <namespace> -- ls -la /mnt/secrets-store/
# 3. Check K8s secrets were created (if using secretObjects)
kubectl get secret -n <namespace>
# 4. Verify secret content (careful with sensitive data)
kubectl exec <pod> -n <namespace> -- cat /mnt/secrets-store/<alias>Scripts
Quick Permission Grant Script
Save as grant-kv-access.sh:
#!/bin/bash
KV_NAME="${1:?Key Vault name required}"
OBJECT_ID="${2:?Object ID required}"
RBAC_ENABLED=$(az keyvault show -n "$KV_NAME" --query "properties.enableRbacAuthorization" -o tsv)
if [ "$RBAC_ENABLED" == "true" ]; then
SCOPE=$(az keyvault show -n "$KV_NAME" --query "id" -o tsv)
az role assignment create \
--role "Key Vault Secrets User" \
--assignee-object-id "$OBJECT_ID" \
--assignee-principal-type ServicePrincipal \
--scope "$SCOPE"
else
az keyvault set-policy \
--name "$KV_NAME" \
--object-id "$OBJECT_ID" \
--secret-permissions get list
fi
echo "Permissions granted for $OBJECT_ID on $KV_NAME"Usage:
./grant-kv-access.sh "kv-cafehyna-dev-hlg" "b31833cc-acab-4afd-adb1-9004c3331359"Diagnose CSI Issues Script
#!/bin/bash
NS="${1:-default}"
SPC="${2:-}"
echo "=== CSI Driver Pods ==="
kubectl get pods -n kube-system | grep secrets-store
echo -e "\n=== SecretProviderClasses in $NS ==="
kubectl get secretproviderclass -n "$NS"
if [ -n "$SPC" ]; then
echo -e "\n=== SecretProviderClass Details: $SPC ==="
kubectl describe secretproviderclass "$SPC" -n "$NS"
fi
echo -e "\n=== Recent CSI Driver Logs ==="
kubectl logs -n kube-system -l app=secrets-store-csi-driver --tail=20
echo -e "\n=== Recent Provider Logs ==="
kubectl logs -n kube-system -l app=secrets-store-provider-azure --tail=20#!/bin/bash
#
# Create or update a secret in Azure Key Vault
#
# Usage: ./create-keyvault-secret.sh <key-vault-name> <secret-name> [secret-value]
#
# If secret-value is not provided, it will be read from stdin or prompted.
#
# Examples:
# ./create-keyvault-secret.sh kv-cafehyna-dev-hlg my-app-password "secret123"
# echo "secret123" | ./create-keyvault-secret.sh kv-cafehyna-dev-hlg my-app-password
# ./create-keyvault-secret.sh kv-cafehyna-dev-hlg my-app-password # prompts for value
set -e
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'
KV_NAME="${1:?Usage: $0 <key-vault-name> <secret-name> [secret-value]}"
SECRET_NAME="${2:?Usage: $0 <key-vault-name> <secret-name> [secret-value]}"
SECRET_VALUE="${3:-}"
echo -e "${GREEN}=== Azure Key Vault Secret Creator ===${NC}"
echo "Key Vault: $KV_NAME"
echo "Secret Name: $SECRET_NAME"
echo ""
# Check Azure CLI authentication
if ! az account show &>/dev/null; then
echo -e "${RED}Error: Not logged in to Azure CLI. Run: az login${NC}"
exit 1
fi
# Verify Key Vault exists
echo -e "${YELLOW}Verifying Key Vault...${NC}"
if ! az keyvault show -n "$KV_NAME" &>/dev/null; then
echo -e "${RED}Error: Key Vault '$KV_NAME' not found${NC}"
exit 1
fi
echo -e "${GREEN}Key Vault found${NC}"
echo ""
# Get secret value if not provided
if [ -z "$SECRET_VALUE" ]; then
if [ -t 0 ]; then
# Interactive mode - prompt for password
echo -e "${YELLOW}Enter secret value (input hidden):${NC}"
read -s SECRET_VALUE
echo ""
else
# Read from stdin
SECRET_VALUE=$(cat)
fi
fi
if [ -z "$SECRET_VALUE" ]; then
echo -e "${RED}Error: Secret value cannot be empty${NC}"
exit 1
fi
# Check if secret already exists
echo -e "${YELLOW}Checking if secret exists...${NC}"
if az keyvault secret show --vault-name "$KV_NAME" --name "$SECRET_NAME" &>/dev/null; then
echo -e "${YELLOW}Secret '$SECRET_NAME' already exists. It will be updated.${NC}"
ACTION="updated"
else
echo "Secret '$SECRET_NAME' does not exist. It will be created."
ACTION="created"
fi
echo ""
# Create or update secret
echo -e "${YELLOW}Setting secret value...${NC}"
RESULT=$(az keyvault secret set \
--vault-name "$KV_NAME" \
--name "$SECRET_NAME" \
--value "$SECRET_VALUE" \
--query "{name:name, created:attributes.created, updated:attributes.updated, version:id}" \
-o json)
echo -e "${GREEN}Secret $ACTION successfully!${NC}"
echo ""
echo "Details:"
echo "$RESULT" | jq '.'
echo ""
echo -e "${GREEN}=== Complete ===${NC}"
echo ""
echo "Next steps:"
echo "1. Create/update SecretProviderClass to reference this secret"
echo "2. If pods are already running, delete them to pick up the new secret"
echo ""
echo "SecretProviderClass object example:"
echo " objects: |"
echo " array:"
echo " - |"
echo " objectName: \"$SECRET_NAME\""
echo " objectType: \"secret\""
#!/bin/bash
#
# Diagnose Secrets Store CSI Driver issues
#
# Usage: ./diagnose-csi.sh [namespace] [secretproviderclass-name]
#
# Examples:
# ./diagnose-csi.sh # Check kube-system only
# ./diagnose-csi.sh external-dns # Check specific namespace
# ./diagnose-csi.sh external-dns cloudflare-api-token-kv # Check specific SPC
set -e
NS="${1:-}"
SPC="${2:-}"
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
RED='\033[0;31m'
CYAN='\033[0;36m'
NC='\033[0m'
echo -e "${GREEN}=== Secrets Store CSI Driver Diagnostics ===${NC}"
echo ""
# Check CSI Driver pods
echo -e "${CYAN}[1/5] CSI Driver Pods${NC}"
kubectl get pods -n kube-system -l 'app in (secrets-store-csi-driver, secrets-store-provider-azure)' -o wide
echo ""
# Check CSI Driver status
DRIVER_PODS=$(kubectl get pods -n kube-system -l app=secrets-store-csi-driver -o jsonpath='{.items[*].status.phase}' 2>/dev/null || echo "")
PROVIDER_PODS=$(kubectl get pods -n kube-system -l app=secrets-store-provider-azure -o jsonpath='{.items[*].status.phase}' 2>/dev/null || echo "")
if [[ "$DRIVER_PODS" == *"Running"* ]] && [[ "$PROVIDER_PODS" == *"Running"* ]]; then
echo -e "${GREEN}CSI Driver and Provider are running${NC}"
else
echo -e "${RED}CSI Driver or Provider not healthy!${NC}"
echo "Driver status: $DRIVER_PODS"
echo "Provider status: $PROVIDER_PODS"
fi
echo ""
# Check CRD
echo -e "${CYAN}[2/5] SecretProviderClass CRD${NC}"
if kubectl get crd secretproviderclasses.secrets-store.csi.x-k8s.io &>/dev/null; then
echo -e "${GREEN}SecretProviderClass CRD is installed${NC}"
else
echo -e "${RED}SecretProviderClass CRD is NOT installed!${NC}"
fi
echo ""
# Check CSI Driver registration
echo -e "${CYAN}[3/5] CSI Driver Registration${NC}"
kubectl get csidriver secrets-store.csi.k8s.io -o jsonpath='{.metadata.name}' 2>/dev/null && echo -e " ${GREEN}(registered)${NC}" || echo -e "${RED}CSI Driver not registered!${NC}"
echo ""
# List SecretProviderClasses
echo -e "${CYAN}[4/5] SecretProviderClasses${NC}"
if [ -n "$NS" ]; then
echo "Namespace: $NS"
kubectl get secretproviderclass -n "$NS" 2>/dev/null || echo "No SecretProviderClasses found in $NS"
else
echo "All namespaces:"
kubectl get secretproviderclass -A 2>/dev/null || echo "No SecretProviderClasses found"
fi
echo ""
# Detailed SPC info if specified
if [ -n "$SPC" ] && [ -n "$NS" ]; then
echo -e "${CYAN}[4b] SecretProviderClass Details: $SPC${NC}"
echo ""
if kubectl get secretproviderclass "$SPC" -n "$NS" &>/dev/null; then
echo -e "${YELLOW}Configuration:${NC}"
kubectl get secretproviderclass "$SPC" -n "$NS" -o yaml | grep -A50 "spec:"
echo ""
echo -e "${YELLOW}Events:${NC}"
kubectl describe secretproviderclass "$SPC" -n "$NS" | grep -A20 "Events:" || echo "No events"
else
echo -e "${RED}SecretProviderClass $SPC not found in $NS${NC}"
fi
echo ""
fi
# Recent logs
echo -e "${CYAN}[5/5] Recent Logs${NC}"
echo ""
echo -e "${YELLOW}CSI Driver logs (last 10 lines):${NC}"
kubectl logs -n kube-system -l app=secrets-store-csi-driver --tail=10 2>/dev/null || echo "No logs available"
echo ""
echo -e "${YELLOW}Azure Provider logs (last 10 lines):${NC}"
kubectl logs -n kube-system -l app=secrets-store-provider-azure --tail=10 2>/dev/null || echo "No logs available"
echo ""
# Check for common errors in logs
echo -e "${CYAN}[Bonus] Checking for common errors...${NC}"
ERRORS=$(kubectl logs -n kube-system -l app=secrets-store-provider-azure --tail=100 2>/dev/null | grep -i "error\|failed\|forbidden" || echo "")
if [ -n "$ERRORS" ]; then
echo -e "${RED}Errors found in provider logs:${NC}"
echo "$ERRORS" | head -20
else
echo -e "${GREEN}No obvious errors in recent provider logs${NC}"
fi
echo ""
echo -e "${GREEN}=== Diagnostics Complete ===${NC}"
#!/bin/bash
#
# Grant Azure Key Vault access to a managed identity
#
# Usage: ./grant-keyvault-access.sh <key-vault-name> <object-id>
#
# This script automatically detects whether the Key Vault uses RBAC or
# Access Policies and grants the appropriate permissions.
set -e
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'
KV_NAME="${1:?Usage: $0 <key-vault-name> <object-id>}"
OBJECT_ID="${2:?Usage: $0 <key-vault-name> <object-id>}"
echo -e "${GREEN}=== Key Vault Access Grant Script ===${NC}"
echo "Key Vault: $KV_NAME"
echo "Object ID: $OBJECT_ID"
echo ""
# Check Azure CLI authentication
if ! az account show &>/dev/null; then
echo -e "${RED}Error: Not logged in to Azure CLI. Run: az login${NC}"
exit 1
fi
# Get Key Vault details
echo -e "${YELLOW}Fetching Key Vault details...${NC}"
KV_INFO=$(az keyvault show -n "$KV_NAME" --query "{id:id, rbac:properties.enableRbacAuthorization, rg:resourceGroup}" -o json 2>/dev/null)
if [ -z "$KV_INFO" ]; then
echo -e "${RED}Error: Key Vault '$KV_NAME' not found${NC}"
exit 1
fi
RBAC_ENABLED=$(echo "$KV_INFO" | jq -r '.rbac')
KV_ID=$(echo "$KV_INFO" | jq -r '.id')
RG_NAME=$(echo "$KV_INFO" | jq -r '.rg')
echo "Resource Group: $RG_NAME"
echo "RBAC Enabled: $RBAC_ENABLED"
echo ""
if [ "$RBAC_ENABLED" == "true" ]; then
echo -e "${YELLOW}Granting RBAC role 'Key Vault Secrets User'...${NC}"
if az role assignment create \
--role "Key Vault Secrets User" \
--assignee-object-id "$OBJECT_ID" \
--assignee-principal-type ServicePrincipal \
--scope "$KV_ID" 2>/dev/null; then
echo -e "${GREEN}RBAC role assigned successfully${NC}"
else
echo -e "${YELLOW}Role may already exist or there was an error${NC}"
fi
echo -e "\n${YELLOW}Verifying role assignment...${NC}"
az role assignment list \
--assignee "$OBJECT_ID" \
--scope "$KV_ID" \
--query "[].{role:roleDefinitionName, scope:scope}" -o table
else
echo -e "${YELLOW}Granting Access Policy permissions (get, list)...${NC}"
if az keyvault set-policy \
--name "$KV_NAME" \
--object-id "$OBJECT_ID" \
--secret-permissions get list; then
echo -e "${GREEN}Access policy set successfully${NC}"
else
echo -e "${RED}Failed to set access policy${NC}"
exit 1
fi
echo -e "\n${YELLOW}Verifying access policy...${NC}"
az keyvault show -n "$KV_NAME" \
--query "properties.accessPolicies[?objectId=='$OBJECT_ID'].{objectId:objectId, permissions:permissions.secrets}" -o table
fi
echo -e "\n${GREEN}=== Complete ===${NC}"
echo "Next steps:"
echo "1. Delete the failing pod to trigger a restart"
echo "2. Monitor pod status: kubectl get pod <pod-name> -w"