Detecting Threats and Securing the Kubernetes Environment with Falco

We are always looking for best practices while working with the Kubernetes cluster to secure our system. Also, we use 2FA's, Container Image Scanning and Network Policy to prevent intrusions. Sometimes, it can be hard to keep our system secure despite all these precautions. And when attacks happen, it's crucial to detect threats and respond as quickly as possible in order to secure our Kubernetes environment. In this blog post, we'll explore using open-source Falco to detect unexpected behaviour and alerts on threats at runtime.

What is Falco?

Falco is an open-source, first Cloud-Native Runtime Security project, IDS (Intrusion Detection Systems) and behavioural activity monitor tool. Falco was created by Sysdig in 2016. Falco helps us check Linux kernel, container, Kubernetes and other logs and raise alerts for unwanted usage. Falco detects runtime unexpected application behaviours at the kernel level and issues alerts about threats.

Falco can consume events from different sources and apply rules to these events to detect abnormal behaviours. One of the event sources is syscalls which detects with drivers. Currently, the Falco project has three different kinds of drivers such as the Kernel module, which is the default driver for Falco, eBPF probe and Userspace instrumentation. For detailed information, please visit here. 

The other event source is Kubernetes audit logs. Falco uses Kubernetes audit logs to capture threat findings and raises alerts on the results it receives. In this way, we can view who is logging into the cluster and what dangerous behaviour they are doing in the cluster.

(Source: https://sysdig.com/)

At the core of Falco is a list of rules. These rules govern all events in a Kubernetes cluster. Here are some events Falco checks:

• Container running in privileged mode

• A server process that creates a new child process

• Any resource that reads a sensitive file

• The starting of the new privileged pod

What are Rules?

A Falco rules file is a YAML file. A rule file contains three types of elements: 

• Rule

• Macro  

• List

Rules consist of key fields such as description, condition, output and priority. The condition field is a filtering expression that is applied against events to check whether they match the rule, under which alerts should be generated. It has another field named output, which contains the string that is sent with the alert.

Macros are reusable mini-rules. They are used to create rules quickly and predictably. Instead of redefining the sub-portions, we can define them as macro and use them in more than one condition. You will find an example of this below.

Lists are collections of items that you can include in rules, macros or even other lists.

Let’s explain the condition with an example. Let’s say we want to detect if any of our node.js containers run any processes which are not node.js binary. So, we’re going to write this:

 condition: evt.type=execve and k8s.deployment.name=my_node_app and proc.name!= node

Let’s explain how Falco reads each part of the condition here:

• evt.type=execve → If something is executing a program
• k8s.deployment.name=my_node_app → If the process is running in a container in my Kubernetes deployment named my_node_app
• proc.name!= node → If the running process name isn’t node (node.js binary files)

You can use many field classes to create conditions such as evt as in evt.type or k8s as in k8s.deployment.name. To check the list please visit here.

Here's another example of a condition that alerts whenever a bash shell is run inside a container successfully:

 - rule: shell_in_container
  desc: notice shell activity within a container
  condition: evt.type = execve and evt.dir=< and container.id != host and proc.name = bash
  output: shell in a container (user=%user.name container_id=%container.id container_name=%container.name shell=%proc.name 
parent=%proc.pname cmdline=%proc.cmdline)
  priority: WARNING 

Let’s take a look at another example with macros. Falco can hook the kube-api events. In this example, we can detect if someone is creating or modifying a config map that has some private credential inside rather than using it as a secret.

 - macro: contains_private_credentials
  condition: > 
   (ka.req.configmap.obj contains "aws_access_key_id" or
    ka.req.configmap.obj contains "aws_s3_key_id" or
    ka.req.configmap.obj contains "password")
- macro: configmap
  condition: ka.target.resouce=configmaps
- macro: modify
  condition: (ka.verb in (create,update,patch))
- rule: Create/modify Configmap with private credentials
  desc: Detect creating/modifying a configmap containing a private credential (aws key, password, etc.)
  condition: configmap and modify and contains_proivate_credentials
  output: K8s configmap with private credential (user=%ka.user.name 
         verb=%ka.verb name=%ka.req.configmap.name configmap=%ka.req.configmap.name config=%ka.req.configmap.obj)
  priority: WARNING
  source: k8s_audit
  tags: [k8s]

In this example, we used macros to simplify the condition of the rule.

As mentioned before, if any rule in Falco is violated, it triggers an alert. By default, Falco has 5 outputs for its events: stdout, file, gRPC, shell and HTTP. Even if they're convenient, we can quickly be limited to integrating Falco with other components. This is where Falcosidekick comes into play. Falcosidekick is a little daemon that extends that number of possible outputs. It manages a large variety of outputs with different purposes such as Slack, Teams and Discord for Chat, Datadog and Prometheus for Metrics, OpsGenie and PagerDuty for Alerting, etc. To see the whole list please visit here.

Now, Let’s Get Our Hands Dirty!

Installation

We can deploy Falco on a local machine, cloud, a managed Kubernetes cluster or a Kubernetes cluster such as K3s running on IoT & Edge computing. 

One of the easiest ways to install Falco is to use Helm. 

After making sure Helm is installed, let’s continue to install the Falco with notification-daemon Falcosidekick. We have a cluster with 2 nodes that we will use for the demo. 

Next, we need to add falcosecurity to the Helm repo, update and install it. We’ll use Slack as an output in this demo. So, please don't forget to change the Slack webhook URL while installing it.

Falco is applied once per node because it is deployed as a DaemonSet in the cluster. To check the status of Falco pods: 

We have successfully installed Falco.
Let’s Test Falco!
First, create an httpd pod with imperative command and run an exec command for that pod.

Now, let's take a look at the channel we set as incoming webhook on Slack.

As you can see, Falco caught the bash command we were running when using the exec command and informed us about the activities in our system. From now on, Falco will notify us if there is any "dangerous" behaviour in our system.

Falco is a great behavioural activity monitor tool for Kubernetes clusters. By using this tool, you can be instantly informed about what is happening in your cluster. In this article, we briefly talked about how a rule is created for Falco. Once you understand the logic of the rule, the limit will be your imagination.