Vault-agent-injector fetches secret twice when using dynamic secret with environment variable on Kubernetes

Recently, I was trying to integrate Hashicorp Vault into our Kubernetes environment. Since our applications use environment variable to load Kuberbetes secret, I decided to use vault-agent-injector recommended by Vault official.

Follow the official instruction, Vault will inject an init container for creating original secret in specific path and a sidecar container to monitor the TTL of secret.

Then, I deployed Vault with vault-agent-injector, updated the deployment annotations and checked whether the secret is rendered correctly. However, I found that the number of leases created by the vault-agent-injector containers was twice the number of replicas in deployment.

From the log, it seemed that init container and sidecar container generated the below log, which means that both of them fetched secret from Vault and rendered secret in /vault/secrets/config.

2022-06-20T13:35:12.541Z [INFO] (runner) rendered "(dynamic)" => "/vault/secrets/config"

I found that the secret in Pod environment variable is originally fetched by init container and after the sidecar container is created, it will fetch a new secret again and update the secret in /vault/secrets/config. Since the two leases are different, the TTL of the secret in Pod environment variable will not be updated by sidecar container.

After reading the annotation document, I found what I need to do is to add annotation to share cache between init and sidecar container to prevent them from fetching the secret twice.

vault.hashicorp.com/agent-cache-enable - configures Vault Agent to enable caching. In Vault 1.7+ this annotation will also enable a Vault Agent persistent cache. This persistent cache will be shared between the init and sidecar container to reuse tokens and leases retrieved by the init container. Defaults to false.

After applying the annotation above, the leases number became the same as the replica number and our secret was being renewed normally.

I think that’s it. Thanks for reading and feel free to discuss with me in the comment.