Mastering the CKS Kubernetes Exam — 6/6 — Monitoring, Logging and Runtime Security
As you prepare for the Certified Kubernetes Security Specialist (CKS) exam, it’s crucial to focus on securing your Kubernetes cluster effectively.
This is the last part of a 6-post series, showing all the exam’s domains and competencies.
Today’s competency is Monitoring, Logging and Runtime Security, which corresponds to 20% of the exam:
Table of Contents
1. Behavioral Analytics of Syscall, Process, and File Activities
2. Analyze Container Behavior with Falco
1. Behavioral Analytics of Syscall, Process, and File Activities
Recommended Lab:
https://killercoda.com/killer-shell-cks/scenario/syscall-activity-strace
Behavioral analytics in the context of system security involves monitoring and analyzing the behavior of processes at the syscall level. This helps in detecting anomalies and malicious activities by examining how processes interact with the system, including file and network activities.
Monitoring System Calls
System calls are the interface between user-space applications and the Linux kernel. By analyzing these calls, you can understand how a process interacts with the system.
- Identify the Pod and Container: First, get the name of the pod and identify the container you want to monitor.
- Execute
strace
in the Container: Usekubectl exec
to runstrace
inside the container. Suppose you have a pod namedmy-pod
and the container’s name ismy-container
:
kubectl exec -it my-pod -c my-container -- strace -p <PID>
- Replace
<PID>
with the process ID you want to trace. You can find the PID by listing the processes inside the container:
kubectl exec -it my-pod -c my-container -- ps aux
Trace System Calls of a Command in a Pod
To trace system calls for a command executed inside the pod, use:
kubectl exec -it my-pod -c my-container -- strace <command>
This command will show all system calls made by the ls
command within the container:
kubectl exec -it my-pod -c my-container -- strace ls
These commands help in detecting unusual activities that might indicate a breach.
Exam tips:
During the exam, you might be asked to check a specific binary running on a pod, find forbidden syscalls and delete the offendind pods. Make sure that you know how to use strace well.
2. Analyze Container Behavior with Falco
Recommended lab:
Falco is an open-source runtime security tool designed to monitor and detect abnormal behavior in containerized environments, like those managed by Kubernetes. It works by analyzing system calls and kernel events to identify suspicious activities, which is particularly useful for securing containerized applications.
This is the tool that will be used in the exam.
Falco rules are typically configured in the falco_rules.yaml
file, which is the main configuration file for Falco.
- On a Local Installation: If you’re running Falco directly on a host (e.g., for testing or in a non-Kubernetes environment), you would usually find or place this file in
/etc/falco/falco_rules.yaml
. - In a Kubernetes Environment: When running Falco in Kubernetes, the configuration is often managed through a ConfigMap.
Example: Delete or rename shell history
Scenario: You want to be alerted when someone deleted the shell history:
- rule: Delete or rename shell history
desc: Detect shell history deletion
condition: >
(modify_shell_history or truncate_shell_history) and
not var_lib_docker_filepath and
not proc.name in (docker_binaries)
output: >
Shell history had been deleted or renamed (user=%user.name user_loginuid=%user.loginuid type=%evt.type command=%proc.cmdline fd.name=%fd.name name=%evt.arg.name path=%evt.arg.path oldpath=%evt.arg.oldpath %container.info)
priority:
WARNING
tags: [process, mitre_defense_evasion]
We can trigger it by cleaning the history:
cat /dev/null > ~/.bash_history
This is the output when the rule is triggered:
Exam tip:
You might be asked to change the output, so, ensure that you know falco parameters.
During the exam, you will be provided with falco documentation. This is the part that you are most interested on:
3. Ensuring Container Immutability at Runtime
In this section, we will explore how to ensure the immutability of containers at runtime.
While containers are designed to be immutable by default, it’s possible to perform in-place modifications.
Such modifications can be made in several ways, like copying files into the pod or accessing the container shell to make changes. For instance:
docker cp new-config.json myapp:/app/config/
Or:
kubectl exec -it myapp -- apt-get install vim
To prevent such updates and ensure immutability, you should configure the pod to prevent any write operations to its file system after it has started.
This can be achieved by setting the readOnlyRootFilesystem
property to true
in the security context.
Here’s how you can configure a python
application to be immutable using readOnlyRootFilesystem=true
in the manifest file python-app.yaml
:
apiVersion: v1
kind: Pod
metadata:
labels:
app: python-app
name: python-app
spec:
containers:
- image: python:3.9
name: python-app
securityContext:
readOnlyRootFilesystem: true
volumeMounts:
- name: temp-volume
mountPath: /tmp
- name: logs-volume
mountPath: /var/log/python
volumes:
- name: temp-volume
emptyDir: {}
- name: logs-volume
emptyDir: {}
Create the resource from the python-app.yaml
file using:
kubectl create -f python-app.yaml
To verify the immutability, try running:
kubectl exec -it python-app -- apt-get install vim
If the command fails, it confirms that the container is indeed immutable at runtime.
4. Leveraging Audit Logs to Monitor Access
Recommended Lab:
Audit logs are crucial for tracking access and ensuring compliance:
- Enable Kubernetes Audit Logs: Configure audit logging in the Kubernetes API server.
vim /etc/kubernetes/manifests/kube-apiserver.yaml
# Example of audit log configuration in kube-apiserver
--audit-policy-file=/etc/kubernetes/audit-policy.yaml
--audit-log-path=/var/log/kubernetes/audit.log
--audit-log-maxage=30
--audit-log-maxbackup=5
--audit-log-maxsize=100
Let’s nowsee how to configure the audit-policy.yaml:
Each request can be tracked with a specific stage. The available stages are:
- RequestReceived: The stage that captures events when the audit handler first gets the request, before it is passed down the handler chain.
- ResponseStarted: The stage that occurs after the response headers are sent but before the response body is transmitted. This stage is applicable only for long-running requests.
- ResponseComplete: The stage indicating that the response body has been fully sent and no further data will be transmitted.
- Panic: The stage where events are logged when a panic occurs.
During the exam, you will be tasked to modify rules, such as the following:
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages: ["RequestReceived"]
rules:
- namespaces: ["prod-namespace"]
verbs: ["delete"]
resources:
- groups: ""
resources: ["pods"]
resourceNames: ["webapp-pod"]
level: RequestResponse
The omitStages
field specifies which stages to exclude from audit events. In this case, events in the RequestReceived stage will not be generated.
The namespaces
field is optional. If not specified, the policy will apply to objects across all namespaces as applicable.
The verbs
field defines the types of operations on objects that the policy should match. This field is also optional; if omitted, all operations such as create, update, and delete will be logged by default.
The resources
field specifies which Kubernetes objects to include in the audit, allowing for multiple types of objects as an array.
The level
field can have four possible values:
- None: No events will be logged for this rule. Therefore, if the
webapp-pod
is deleted in theprod-namespace
, no events will be recorded. - Metadata: Only metadata such as timestamp, user, resources, and verbs will be logged if the event matches the rule.
- Request: Logs include both metadata and the request body, providing more details than Metadata.
- RequestResponse: Logs include metadata, the request body, and the response body, offering the most comprehensive details.
Exam tips:
Remember these flags and what each one does:
— audit-policy-file=/etc/kubernetes/audit-policy.yaml
— audit-log-path=/var/log/kubernetes/audit.log
— audit-log-maxage=30
— audit-log-maxbackup=5
— audit-log-maxsize=100
Conclusion
Good luck with your CKS exam prep!
This is the end of this series of posts, here you can find more resources to get prepared:
If you have any questions or need a hand with anything, just let me know. Let’s tackle this certification journey together!
Follow me on Linkedin