In this article, we will look into common ways to secure secrets in a Kubernetes application and how to manage them in a GitOps workflow based on ArgoCD with the help of Sops. The problem is the following: your application depends on some secrets that you need to store securely and make available to your running application. You can address this requirement in two ways: You put these secrets in remote secret manager, for instance, or and you use the provided API or convenient tools like Vault AWS Secret Manager External Secrets You keep these secrets as vanilla Kubernetes objects, you commit those in your Git repository with your code but you take care of encrypting them with for example Secret sops This second solution has a clear advantage: you can provide your own GPG key and you don’t need to rely on a cloud provider or any external tools. If your goal is a multi-cloud strategy, it’s the way to go. If you are using ArgoCD to deploy our Kubernetes objects you may wonder . Let’s see what the ArgoCD documentation has to say. how to integrate Sops with ArgoCD ArgoCD Stance on Secrets Management makes it quite clear: ArgoCD documentation Argo CD is un-opinionated about how secrets are managed. There’s many ways to do it and there’s no one-size-fits-all solution. Basically, you are left to your own devices to make sops work with ArgoCD. In this article, we will take a look at how we can implement secret handling in an elegant, non-breaking way. Our Tools of Choice Let’s recap the tools we will use: Secrets will be encrypted and decrypted using sops and we will provide our own GPG key as the encryption key Our app is packaged using Helm Instead of dealing with sops directly, we will use Helm Secrets Helm Secrets is essentially a wrapper for Helm that encrypt and decrypt secrets on the fly for you. While no longer under heavy development, it’s still working really well. But the problem is that ArgoCD doesn’t know this plugin as it only comes with the basic Helm binary built-in. Let’s address this now. Creating our own Custom ArgoCD We will create a new ArgoCD docker image which contains exactly what we need: GPG which handle the encryption key Sops for encryption/decryption Helm Secrets plugin The Dockerfile will look like this: argoproj/argocd:v1. SOPS_VERSION= HELM_SECRETS_VERSION= SOPS_PGP_FP= SOPS_PGP_FP=${SOPS_PGP_FP} root argocd HELM_PLUGINS= FROM 7.6 ARG "v3.6.1" ARG "2.0.2" ARG "141B69EE206943BA9A64E691A00C9B1A7DCB6D07" ENV USER COPY helm-wrapper.sh /usr/ /bin/ local RUN apt-get update && \ apt-get install -y \ curl \ gpg && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \ curl -o /usr/ /bin/sops -L https://github.com/mozilla/sops/releases/download/ /sops- .linux && \ chmod +x /usr/ /bin/sops && \ /usr/ /bin && \ mv helm helm.bin && \ mv helm2 helm2.bin && \ mv helm-wrapper.sh helm && \ ln helm helm2 && \ chmod +x helm helm2 local ${SOPS_VERSION} ${SOPS_VERSION} local cd local # helm secrets plugin should be installed as user argocd or it won't be found USER RUN /usr/ /bin/helm.bin plugin install https://github.com/zendesk/helm-secrets --version local ${HELM_SECRETS_VERSION} ENV "/home/argocd/.local/share/helm/plugins/" A few things to note: First, we set up GPG and Sops Then we install the Helm Secrets plugin Finally, we move the ArgoCD default Helm binary as helm.bin and replace it with a wrapper script This wrapper script will look after the GPG key (you can mount it as a secret volume for example) and if found will import it. Then it will replace every call to helm with calls to helm secrets . Well, that’s in theory because Helm secrets does not support all Helm commands and is quite talkative so we need to alter the output for certain commands. GPG_KEY= [ -f ] gpg --quiet --import [ = ] || [ = ] || [ = ] || [ = ] || [ = ] out=$(helm.bin secrets ) code=$? [ -eq 0 ]; | sed -E 0 helm.bin #! /bin/sh '/home/argocd/gpg/gpg.asc' if ${GPG_KEY} then ${GPG_KEY} fi # helm secrets only supports a few helm commands if $1 "template" $1 "install" $1 "upgrade" $1 "lint" $1 "diff" then # Helm secrets add some useless outputs to every commands including template, namely # 'remove: <secret-path>.dec' for every decoded secrets. # As argocd use helm template output to compute the resources to apply, these outputs # will cause a parsing error from argocd, so we need to remove them. # We cannot use exec here as we need to pipe the output so we call helm in a subprocess and # handle the return code ourselves. $@ if $code then # printf insted of echo here because we really don't want any backslash character processing printf '%s\n' " " $out "/^removed '.+\.dec'$/d" exit else exit $code fi else # helm.bin is the original helm binary exec $@ fi Our ArgoCD image now understands Helm secrets without any additional configuration! Cool. Our Test Application Our test application is a Helm chart with encrypted secrets. Please note the file which is supposed to contain sensitive data. secrets.yaml testapp ├── Chart.yaml ├── charts ├── secrets.yaml ├── templates │ ├── NOTES.txt │ ├── _helpers.tpl │ ├── deployment.yaml │ ├── hpa.yaml │ ├── ingress.yaml │ ├── service.yaml │ └── serviceaccount.yaml └── values.yaml We encode the secrets with sops using our private key, the same one that is looked after by our Helm wrapper script: sops -i --encrypt testapp/secrets.yaml Then we push this Helm chart in our Git repository. The only thing left to do is to create the corresponding Application CRD to watch this new repository: apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: testapp namespace: argocd spec: project: default source: repoURL: git@gitlab.com:my/repo/charts.git targetRevision: master path: charts/testapp helm: releaseName: testapp valueFiles: - "secrets.yaml" destination: server: https://kubernetes.default.svc namespace: testapp syncPolicy: automated: {} syncOptions: - CreateNamespace=true After this, you should see your new application appears in the ArgoCD dashboard. Secrets have been decrypted under the hood using the provided GPG key and the app is working properly. kubectl apply -f From an ArgoCD standpoint, the Helm wrapper appears as the built-in Helm binary so any GUI functionalities related to Helm are still working as usual. ArgoCD Plugin as an Alternative Solution To be exhaustive, let’s mention a simpler solution to our problem. We could have used an . A plugin responsibility is to output some YAML that ArgoCD will then send to the Kubernetes API. ArgoCD plugin To make this work, you will still need a custom ArgoCD Dockerfile but you will not replace the Helm binary, only adding sops and Helm secrets. Then you will declare the plugin. The example below is an extract of the file using the : values.yaml ArgoCD Helm chart server: config: configManagementPlugins: | - name: helmSecrets init: command: ["gpg"] args: ["--import", "/home/argocd/gpg/gpg.asc" ] # is mounted as a kube secret generate: command: ["/bin/sh", "-c" ] args: ["helm secrets template $HELM_OPTS $RELEASE_NAME ."] It is the same philosophy: we import the GPG key when initializing the plugin and then we call helm secrets template with parameters from the environment to generates the expected YAML objects. To use the plugin in an Application, do it like this: piVersion: argoproj.io/v1alpha1 kind: Application metadata: name: testapp namespace: argocd spec: project: default source: repoURL: git@gitlab.com:my/repo/charts.git targetRevision: master path: charts/testapp plugin: name: helmSecrets env: - name: HELM_OPTS value: "secrets.yaml" - name: RELEASE_NAME value: "testapp" destination: server: https://kubernetes.default.svc namespace: testapp syncPolicy: automated: {} syncOptions: - CreateNamespace=true You should get the same result as with our previous solution but with one notable exception: ArgoCD cannot recognize your plugin is in fact Helm in disguise so any GUI functionalities related to Helm will not be available, like seeing the values and parameters. For this reason, our initial solution, albeit a little more complicated, is clearly superior. Concluding Thoughts With some quick adjustments, you can make your secrets handling tools work with ArgoCD. Even if ArgoCD doesn’t handle secrets by itself, it is really well thought since you can integrate your own tools quite easily. Also published on: https://medium.com/faun/handling-kubernetes-secrets-with-argocd-and-sops-650df91de173
