Detection: Kubernetes Process with Resource Ratio Anomalies

Description

The following analytic detects anomalous changes in resource utilization ratios for processes running on a Kubernetes node. It leverages process metrics collected via an OTEL collector and hostmetrics receiver, analyzed through Splunk Observability Cloud. The detection uses a lookup table containing average and standard deviation values for various resource ratios (e.g., CPU:memory, CPU:disk operations). Significant deviations from these baselines may indicate compromised processes, malicious activity, or misconfigurations. If confirmed malicious, this could signify a security breach, allowing attackers to manipulate workloads, potentially leading to data exfiltration or service disruption.

Search

 1
 2| mstats avg(process.*) as process.* where `kubernetes_metrics` by host.name k8s.cluster.name k8s.node.name process.executable.name span=10s 
 3| eval cpu:mem = 'process.cpu.utilization'/'process.memory.utilization' 
 4| eval cpu:disk = 'process.cpu.utilization'/'process.disk.operations' 
 5| eval mem:disk = 'process.memory.utilization'/'process.disk.operations' 
 6| eval cpu:threads = 'process.cpu.utilization'/'process.threads' 
 7| eval disk:threads = 'process.disk.operations'/'process.threads' 
 8| eval key = 'k8s.cluster.name' + ":" + 'host.name' + ":" + 'process.executable.name' 
 9| lookup k8s_process_resource_ratio_baseline key 
10| fillnull 
11| eval anomalies = "" 
12| foreach stdev_* [ eval anomalies =if( '<<MATCHSTR>>' > ('avg_<<MATCHSTR>>' + 4 * 'stdev_<<MATCHSTR>>'), anomalies + "<<MATCHSTR>> ratio higher than average by " + tostring(round(('<<MATCHSTR>>' - 'avg_<<MATCHSTR>>')/'stdev_<<MATCHSTR>>' ,2)) + " Standard Deviations. <<MATCHSTR>>=" + tostring('<<MATCHSTR>>') + " avg_<<MATCHSTR>>=" + tostring('avg_<<MATCHSTR>>') + " 'stdev_<<MATCHSTR>>'=" + tostring('stdev_<<MATCHSTR>>') + ", " , anomalies) ] 
13| eval anomalies = replace(anomalies, ",\s$", "") 
14| where anomalies!="" 
15| stats count values(anomalies) as anomalies by host.name k8s.cluster.name k8s.node.name process.executable.name 
16| where count > 5 
17| rename host.name as host 
18| `kubernetes_process_with_resource_ratio_anomalies_filter`

Data Source

Macros Used

Annotations

Default Configuration

This detection is configured by default in Splunk Enterprise Security to run with the following settings:

Implementation

To implement this detection, follow these steps:

• Deploy the OpenTelemetry Collector (OTEL) to your Kubernetes cluster.
• Enable the hostmetrics/process receiver in the OTEL configuration.
• Ensure that the process metrics, specifically Process.cpu.utilization and process.memory.utilization, are enabled.
• Install the Splunk Infrastructure Monitoring (SIM) add-on. (ref: https://splunkbase.splunk.com/app/5247)
• Configure the SIM add-on with your Observability Cloud Organization ID and Access Token.
• Set up the SIM modular input to ingest Process Metrics. Name this input "sim_process_metrics_to_metrics_index".
• In the SIM configuration, set the Organization ID to your Observability Cloud Organization ID.
• Set the Signal Flow Program to the following: data('process.threads').publish(label='A'); data('process.cpu.utilization').publish(label='B'); data('process.cpu.time').publish(label='C'); data('process.disk.io').publish(label='D'); data('process.memory.usage').publish(label='E'); data('process.memory.virtual').publish(label='F'); data('process.memory.utilization').publish(label='G'); data('process.cpu.utilization').publish(label='H'); data('process.disk.operations').publish(label='I'); data('process.handles').publish(label='J'); data('process.threads').publish(label='K')
• Set the Metric Resolution to 10000.
• Leave all other settings at their default values.
• Run the Search Baseline Of Kubernetes Container Network IO Ratio

Known False Positives

unknown

Associated Analytic Story

• Abnormal Kubernetes Behavior using Splunk Infrastructure Monitoring

Risk Based Analytics (RBA)

References

• https://github.com/signalfx/splunk-otel-collector-chart

Detection Testing

Replay any dataset to Splunk Enterprise by using our replay.py tool or the UI. Alternatively you can replay a dataset into a Splunk Attack Range

Source: GitHub | Version: 2