Mastering the CKS Kubernetes Exam — 6/6 — Monitoring, Logging and Runtime Security

Rafael Medeiros
6 min readSep 17, 2024

--

As you prepare for the Certified Kubernetes Security Specialist (CKS) exam, it’s crucial to focus on securing your Kubernetes cluster effectively.

This is the last part of a 6-post series, showing all the exam’s domains and competencies.

Part 1 is here

part 2 is here

part 3 is here

part 4 is here

part 5 is here

Today’s competency is Monitoring, Logging and Runtime Security, which corresponds to 20% of the exam:

Table of Contents

1. Behavioral Analytics of Syscall, Process, and File Activities

2. Analyze Container Behavior with Falco

3. Ensuring Container Immutability at Runtime

4. Leveraging Audit Logs to Monitor Access

1. Behavioral Analytics of Syscall, Process, and File Activities

Recommended Lab:

https://killercoda.com/killer-shell-cks/scenario/syscall-activity-strace

Behavioral analytics in the context of system security involves monitoring and analyzing the behavior of processes at the syscall level. This helps in detecting anomalies and malicious activities by examining how processes interact with the system, including file and network activities.

Monitoring System Calls

System calls are the interface between user-space applications and the Linux kernel. By analyzing these calls, you can understand how a process interacts with the system.

  1. Identify the Pod and Container: First, get the name of the pod and identify the container you want to monitor.
  2. Execute strace in the Container: Use kubectl exec to run strace inside the container. Suppose you have a pod named my-pod and the container’s name is my-container:
kubectl exec -it my-pod -c my-container -- strace -p <PID>
  1. Replace <PID> with the process ID you want to trace. You can find the PID by listing the processes inside the container:
kubectl exec -it my-pod -c my-container -- ps aux

Trace System Calls of a Command in a Pod

To trace system calls for a command executed inside the pod, use:

kubectl exec -it my-pod -c my-container -- strace <command>

This command will show all system calls made by the ls command within the container:

kubectl exec -it my-pod -c my-container -- strace ls

These commands help in detecting unusual activities that might indicate a breach.

Exam tips:

During the exam, you might be asked to check a specific binary running on a pod, find forbidden syscalls and delete the offendind pods. Make sure that you know how to use strace well.

2. Analyze Container Behavior with Falco

Recommended lab:

Falco is an open-source runtime security tool designed to monitor and detect abnormal behavior in containerized environments, like those managed by Kubernetes. It works by analyzing system calls and kernel events to identify suspicious activities, which is particularly useful for securing containerized applications.

This is the tool that will be used in the exam.

Falco rules are typically configured in the falco_rules.yaml file, which is the main configuration file for Falco.

  • On a Local Installation: If you’re running Falco directly on a host (e.g., for testing or in a non-Kubernetes environment), you would usually find or place this file in /etc/falco/falco_rules.yaml.
  • In a Kubernetes Environment: When running Falco in Kubernetes, the configuration is often managed through a ConfigMap.

Example: Delete or rename shell history

Scenario: You want to be alerted when someone deleted the shell history:

- rule: Delete or rename shell history
desc: Detect shell history deletion
condition: >
(modify_shell_history or truncate_shell_history) and
not var_lib_docker_filepath and
not proc.name in (docker_binaries)
output: >
Shell history had been deleted or renamed (user=%user.name user_loginuid=%user.loginuid type=%evt.type command=%proc.cmdline fd.name=%fd.name name=%evt.arg.name path=%evt.arg.path oldpath=%evt.arg.oldpath %container.info)
priority:
WARNING
tags: [process, mitre_defense_evasion]

We can trigger it by cleaning the history:

cat /dev/null > ~/.bash_history

This is the output when the rule is triggered:

Exam tip:

You might be asked to change the output, so, ensure that you know falco parameters.

During the exam, you will be provided with falco documentation. This is the part that you are most interested on:

3. Ensuring Container Immutability at Runtime

In this section, we will explore how to ensure the immutability of containers at runtime.

While containers are designed to be immutable by default, it’s possible to perform in-place modifications.

Such modifications can be made in several ways, like copying files into the pod or accessing the container shell to make changes. For instance:

docker cp new-config.json myapp:/app/config/

Or:

kubectl exec -it myapp -- apt-get install vim

To prevent such updates and ensure immutability, you should configure the pod to prevent any write operations to its file system after it has started.

This can be achieved by setting the readOnlyRootFilesystem property to true in the security context.

Here’s how you can configure a python application to be immutable using readOnlyRootFilesystem=true in the manifest file python-app.yaml:

apiVersion: v1
kind: Pod
metadata:
labels:
app: python-app
name: python-app
spec:
containers:
- image: python:3.9
name: python-app
securityContext:
readOnlyRootFilesystem: true
volumeMounts:
- name: temp-volume
mountPath: /tmp
- name: logs-volume
mountPath: /var/log/python
volumes:
- name: temp-volume
emptyDir: {}
- name: logs-volume
emptyDir: {}

Create the resource from the python-app.yaml file using:

kubectl create -f python-app.yaml

To verify the immutability, try running:

kubectl exec -it python-app -- apt-get install vim

If the command fails, it confirms that the container is indeed immutable at runtime.

4. Leveraging Audit Logs to Monitor Access

Recommended Lab:

Audit logs are crucial for tracking access and ensuring compliance:

  • Enable Kubernetes Audit Logs: Configure audit logging in the Kubernetes API server.

vim /etc/kubernetes/manifests/kube-apiserver.yaml

# Example of audit log configuration in kube-apiserver
--audit-policy-file=/etc/kubernetes/audit-policy.yaml
--audit-log-path=/var/log/kubernetes/audit.log
--audit-log-maxage=30
--audit-log-maxbackup=5
--audit-log-maxsize=100

Let’s nowsee how to configure the audit-policy.yaml:

Each request can be tracked with a specific stage. The available stages are:

  • RequestReceived: The stage that captures events when the audit handler first gets the request, before it is passed down the handler chain.
  • ResponseStarted: The stage that occurs after the response headers are sent but before the response body is transmitted. This stage is applicable only for long-running requests.
  • ResponseComplete: The stage indicating that the response body has been fully sent and no further data will be transmitted.
  • Panic: The stage where events are logged when a panic occurs.

During the exam, you will be tasked to modify rules, such as the following:

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages: ["RequestReceived"]
rules:
- namespaces: ["prod-namespace"]
verbs: ["delete"]
resources:
- groups: ""
resources: ["pods"]
resourceNames: ["webapp-pod"]
level: RequestResponse

The omitStages field specifies which stages to exclude from audit events. In this case, events in the RequestReceived stage will not be generated.

The namespaces field is optional. If not specified, the policy will apply to objects across all namespaces as applicable.

The verbs field defines the types of operations on objects that the policy should match. This field is also optional; if omitted, all operations such as create, update, and delete will be logged by default.

The resources field specifies which Kubernetes objects to include in the audit, allowing for multiple types of objects as an array.

The level field can have four possible values:

  • None: No events will be logged for this rule. Therefore, if the webapp-pod is deleted in the prod-namespace, no events will be recorded.
  • Metadata: Only metadata such as timestamp, user, resources, and verbs will be logged if the event matches the rule.
  • Request: Logs include both metadata and the request body, providing more details than Metadata.
  • RequestResponse: Logs include metadata, the request body, and the response body, offering the most comprehensive details.

Exam tips:

Remember these flags and what each one does:
— audit-policy-file=/etc/kubernetes/audit-policy.yaml
— audit-log-path=/var/log/kubernetes/audit.log
— audit-log-maxage=30
— audit-log-maxbackup=5
— audit-log-maxsize=100

--

--

Rafael Medeiros
Rafael Medeiros

Written by Rafael Medeiros

DevOps Engineer | CNCF Kubestronaut | 3x Azure | Terraform Fanatic | Another IT Professional willing to help the community

No responses yet