Let’s talk about Kubernetes security As Kubernetes continues to grow in adoption, it is important for us to know how to secure it. In a dynamic infrastructure platform such as Kubernetes, detecting and addressing threats is important but also challenging at the same time. Falco, the open source cloud native runtime security project, is one of the leading open source Kubernetes threat detection engines. Falco was created by Sysdig in 2016 and is the first runtime security project to join CNCF as an incubation-level project. Falco detects unexpected application behaviour and alerts on threats at runtime. Why is it difficult? Security is one of the harder challenges of running Kubernetes according to . The reason it is difficult is that there are multiple moving layers in a cloud native stack, hence operators may not focus on security early on. Another contributor is that some distributions of Kubernetes may not be secure by default, contrary to what operators assume. this analysis Prevention and Detection Information security is a process that moves through phases building and strengthening itself along the way. Security is a journey, not a destination. Although the Information Security process has multiple strategies and activities, we can group them all into three distinct phases - prevention, detection, and response. Preventive measures Preventive measures include having proper access control, authentication and authorization in place. No matter what level of protection a system may have, it will get compromised given a greater level of motivation and skill. There is no foolproof “silver bullet” security solution. A defense in-depth strategy should be deployed so when each layer fails, it fails safely to a known state and sounds an alarm. The most important element of this strategy is timely detection and notification of a compromise. Detective measures An example of detective measures includes sending logs from security and network devices like host-based intrusion detection systems (HIDS) and network intrusion detection systems (NIDS) to an SIEM and building rules to alert when suspicious activity is seen. According to the defense in-depth approach, we should be auditing and placing detection triggers on multiple layers of the technology stack, from looking at cloud audit logs (like Cloudtrail), the Kubernetes cluster, containers, application code, host OS and the kernel. We will take a look at the detective measures in this blog which will further help us to respond to these threats. Image credit: @leodido Setting Falco up on Kubernetes The most secure way to run Falco is to install Falco directly on the host system so that Falco is isolated from Kubernetes in the case of compromise. Then the Falco alerts can be consumed via read-only agents running in Kubernetes. Falco can also be run directly in Kubernetes if isolation is not a concern. We will be setting Falco up using Helm in this tutorial, but you should be aware of the pros and cons of the method you are choosing. Prerequisite: We need a working Kubernetes setup for this. You can use a cloud Kubernetes offering from AWS/GCP or set one up yourself locally using . We also require kubectl and to be installed on your client machine. Steps minikube Helm Let’s start with the installation of Falco. 1. Add the falcosecurity repo in Helm $ helm repo add falcosecurity https: $ helm repo update
Hang tight we grab the latest your chart repositories...
...Successfully got an update the chart repository
...Successfully got an update the chart repository
Update Complete. ⎈Happy Helming!⎈ //falcosecurity.github.io/charts while from from "falcosecurity" from "stable" 2. Installing the chart $ helm install falco falcosecurity/falco
NAME: falco
LAST DEPLOYED: Mon Nov : : NAMESPACE: STATUS: deployed
REVISION: TEST SUITE: None
NOTES:
Falco agents are spinning up on each node your cluster. After a few
seconds, they are going to start monitoring your containers looking security issues.
No further action should be required. 9 22 09 28 2020 default 1 in for 3. After a few seconds Falco should be running, to verify, you will see the pods you created in a moment. $ helm ls
$ kubectl get pods Setting up the environment We will need an environment to emulate the attacks and detect them. Let us set it up. We will be using Helm for this. Let’s add Helm’s stable repository and install chart from it. mysql-db $ helm repo add stable https: $ helm repo update
Hang tight we grab the latest your chart repositories...
...Successfully got an update the chart repository
...Successfully got an update the chart repository
Update Complete. ⎈Happy Helming!⎈ //charts.helm.sh/stable while from from "falcosecurity" from "stable" $ helm install mysql-db stable/mysql
NAME: mysql-db
LAST DEPLOYED: Mon Nov : : NAMESPACE: STATUS: deployed
REVISION: NOTES:
MySQL can be accessed via port on the following DNS name within your cluster:
mysql-db.default.svc.cluster.local
... 9 22 11 07 2020 default 1 3306 from Let us now monitor Falco logs: $ kubectl get pods
NAME                      READY  STATUS   RESTARTS  AGE
falco rr9r / Running m
mysql-db-d5dc6b85d hrm / Running m
ubuntu / Running m

$ kubectl logs -f falco rr9r # Replace your Falco pod name here -6 1 1 0 49 -77 1 1 0 47 1 1 0 43 -6 with Analyzing Falco’s logs The first steps when an attacker does after they gain access to a Kubernetes cluster is to enumerate available network hosts. An attacker will try to find their way around a seemingly new environment. They can try to use safe/less noisy commands to gain more insights from the cluster. Let us go through the steps an attacker might do and look at the corresponding detection signals from Falco. We will use the default ruleset which comes with Falco, these rules can be tuned as per your needs in your environment. You can find Falco’s default rules here: https://github.com/falcosecurity/falco/tree/master/rules Terminal shell in container Description: Detecting terminal shells being spawned in pods. We open a terminal shell on a running pod to replicate the scenario. $ kubectl exec -it mysql-db-d5dc6b85d hrm -- bash -il # Replace the mysql pod name you have -77 with Detection: : : : Notice A shell was spawned a container an attached terminal (user=root user_loginuid= k8s.ns= k8s.pod=mysql-db-d5dc6b85d hrm container= fc8155f9d1a shell=bash parent=runc cmdline=bash -il terminal= container_id= fc8155f9d1a image=mysql) k8s.ns= k8s.pod=mysql-db-d5dc6b85d hrm container= fc8155f9d1a 18 08 40.584075795 in with -1 default -77 3 34816 3 default -77 3 Contact Kubernetes API Server From Container Description: We run kube-recon, which is a tool which does initial reconnaissance in a Kubernetes environment. This rule detects attempts to contact the Kubernetes API Server from a container. $ kubectl run kuberecon --tty -i --image octarinesec/kube-recon
/ # ./kube-recon / / : : Testing K8S API permissions / / : : Your K8S API Server is configured properly / / : : Running Nmap on the discovered IPs / / : : Getting local ip address and subnet / / : : Trying to download EICAR file / / : : Downloaded EICAR successfully. No malware protection is place 2020 11 09 18 21 22 2020 11 09 18 21 23 2020 11 09 18 21 23 2020 11 09 18 21 23 2020 11 09 18 21 23 2020 11 09 18 21 23 in Detection: : : : Notice A shell was spawned a container an attached terminal (user=root user_loginuid= k8s.ns= <NA> container=4f63870599d0 shell=sh parent=<NA> cmdline=sh terminal=34816 container_id=4f63870599d0 image=octarinesec/kube-recon) k8s.ns=<NA> k8s.pod=<NA> container=4f63870599d0
18:21:22.657590195: Warning Docker or kubernetes client executed in container (user=root user_loginuid=-1 k8s.ns=default k8s.pod=kuberecon container=4f63870599d0 parent=kube-recon cmdline=kubectl get pods image=octarinesec/kube-recon:latest) k8s.ns=default k8s.pod=kuberecon container=4f63870599d0
18:21:22.723707794: Notice Unexpected connection to K8s API Server from container (command=kubectl get pods k8s.ns=default k8s.pod=kuberecon container=4f63870599d0 image=octarinesec/kube-recon:latest connection=172.17.0.5:56972->10.96.0.1:443) k8s.ns=default k8s.pod=kuberecon container=4f63870599d0 18 20 45.927730981 in with -1 k8s.pod= < > NA Checking if shell is a container environment (Change thread namespace) Description: amicontained is a tool to check whether the shell is a containerized environment. The rule detects an attempt to change a program/thread’s namespace. $ cd /tmp; curl -L -o amicontained https: Output: 
Container Runtime: docker
Has Namespaces:
  pid: user: AppArmor Profile: kernel
Capabilities:
  BOUNDING -> chown dac_override fowner fsetid kill setgid setuid setpcap net_bind_service net_raw sys_chroot mknod audit_write setfcap
Seccomp: disabled    
Blocked Syscalls ( ):
  MSGRCV SYSLOG SETSID VHANGUP PIVOT_ROOT ACCT SETTIMEOFDAY UMOUNT2 SWAPON SWAPOFF REBOOT SETHOSTNAME SETDOMAINNAME INIT_MODULE DELETE_MODULE LOOKUP_DCOOKIE KEXEC_LOAD OPEN_BY_HANDLE_AT FINIT_MODULE //github.com/genuinetools/amicontained/releases/download/v0.4.7/amicontained-linux-amd64; chmod 555 amicontained; ./amicontained true false 19 Detection: : : : Notice Namespace change (setns) by unexpected program (user=root user_loginuid= command=amicontained parent=bash k8s.ns= k8s.pod=kuberecon container=c6112967b4f2 container_id=c6112967b4f2 image=octarinesec/kube-recon:latest) k8s.ns= k8s.pod=kuberecon container=c6112967b4f2
Mon Nov : : : Falco internal: syscall event drop. system calls dropped last second. : : : Critical Falco internal: syscall event drop. system calls dropped last second. (ebpf_enabled= n_drops= n_drops_buffer= n_drops_bug= n_drops_pf= n_evts= ) 18 43 37.288344192 -1 default default 9 18 43 37 2020 9 in 18 43 37.970973376 9 in 0 9 0 0 9 15232 Next steps In this tutorial, we looked at the basics of Kubernetes security monitoring and how Falco allows us to use rules to achieve the detection of security issues. The default detection ruleset of Falco is enough to get yourself up and running, but you will most likely need to build on top of the default ruleset. We will explore building and tuning these rules in the upcoming posts on security. That’s a wrap folks! Do share your thoughts in the comment section below. Previously published at https://www.infracloud.io/blogs/introduction-kubernetes-security-falco/

InfraCloud

Timely

Kubernetes Security 101: Cloud Native Runtime Security with Falco

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Beginner's Guide To Telepresence: Develop Your K8s Application Locally

10 Strategic Technology Trends Businesses Should Plan for in 2021

The Noonification: Craig Wright is Satoshi Nakamoto (12/17/2023)

300 Scholarships Up For Grabs at The School of Cloud Computing Courtesy Udacity and SUSE

4 Best Cloud Deployment models

5 Tips to Be More Productive With Kubernetes

Beginner's Guide To Telepresence: Develop Your K8s Application Locally

10 Strategic Technology Trends Businesses Should Plan for in 2021

The Noonification: Craig Wright is Satoshi Nakamoto (12/17/2023)

300 Scholarships Up For Grabs at The School of Cloud Computing Courtesy Udacity and SUSE

4 Best Cloud Deployment models

5 Tips to Be More Productive With Kubernetes

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps