Detection: Ollama Long-Running Inference Requests

Description

Identifies repeated Ollama inference requests with long response times on /api/generate, /api/chat, and /v1/chat/completions. Long response time can result from model size, prompt length, host resources, or normal workload. GIN access logs do not include prompt content and cannot establish a prompt injection or jailbreak attempt, so use this as an exploratory performance signal.

 1`ollama_server`
 2"[GIN]"
 3
 4
 5| rex field=_raw "\
 6|\s+(?<raw_status>\d+)\s+\
 7|\s+(?<raw_latency>(?:\d+h)?(?:\d+m)?[\d\.]+(?:µs
 8|ms
 9|s))\s+\
10|\s+(?<raw_src>[^
11|]+?)\s+\
12|\s+(?<raw_method>\w+)\s+\"(?<raw_path>[^\"]+)\""
13
14
15| eval src=coalesce(src, trim(raw_src))
16
17| eval uri_path=mvindex(split(coalesce(uri_path, raw_path), "?"), 0)
18
19| eval http_method=coalesce(http_method, raw_method)
20
21| eval http_response_code=coalesce(http_response_code, raw_status)
22
23| rex field=raw_latency "^(?:(?<raw_latency_hours>\d+)h)?(?:(?<raw_latency_minutes>\d+)m)?(?<raw_latency_value>[\d\.]+)(?<raw_latency_unit>µs
24|ms
25|s)$"
26
27| eval response_time_ms=(coalesce(tonumber(raw_latency_hours), 0) * 3600000) + (coalesce(tonumber(raw_latency_minutes), 0) * 60000) + (tonumber(raw_latency_value) * case(raw_latency_unit="µs", 0.001, raw_latency_unit="ms", 1, raw_latency_unit="s", 1000))
28
29
30| where isnotnull(src) AND in(uri_path, "/api/generate", "/api/chat", "/v1/chat/completions") AND response_time_ms > 30000
31
32
33| eval response_time_seconds=response_time_ms/1000
34
35
36| bin _time span=10m
37
38
39| stats count as long_request_count
40        avg(response_time_seconds) as avg_response_time
41        max(response_time_seconds) as max_response_time
42        values(uri_path) as uri_path
43        values(http_response_code) as status_codes
44  by _time src host
45
46
47| where long_request_count > 1
48
49
50| eval avg_response_time=round(avg_response_time, 2)
51
52| eval max_response_time=round(max_response_time, 2)
53
54| eval severity=case(max_response_time > 55, "high", 1=1, "medium")
55
56| eval attack_type="Long-Running Inference Requests"
57
58
59| table _time host src uri_path long_request_count
60        avg_response_time max_response_time
61        status_codes severity attack_type
62
63
64| `ollama_long_running_inference_requests_filter`

Data Source

Name Platform Sourcetype Source
Ollama Server Other 'ollama:server' 'server.log'

Macros Used

Name Value
ollama_server (sourcetype="ollama:server")
ollama_long_running_inference_requests_filter search *
ollama_long_running_inference_requests_filter is an empty macro by default. It allows the user to filter out any results (false positives) without editing the SPL.

Annotations

- MITRE ATT&CK
+ Kill Chain Phases
+ NIST
+ CIS
- Threat Actors
ID Technique Tactic
DE.AE
CIS 10

Default Configuration

This detection is configured by default in Splunk Enterprise Security to run with the following settings:

Setting Value
Disabled true
Cron Schedule 0 * * * *
Earliest Time -70m@m
Latest Time -10m@m
Schedule Window auto
Creates Finding (Notable) No
Creates Intermediate Finding (Risk Event) Yes
Anomaly detections generate Intermediate Findings (Risk Events). They do not generate a Finding (Notable) directly.

Implementation

Ingest Ollama logs via Splunk TA-ollama add-on by configuring file monitoring inputs pointed to your Ollama server log directories (sourcetype: ollama:server), or enable HTTP Event Collector (HEC) for real-time API telemetry and prompt analytics (sourcetypes: ollama:api, ollama:prompts). CIM compatibility using the Web datamodel for standardized security detections.

Known False Positives

Legitimate complex queries requiring extensive model reasoning, large context windows processing substantial amounts of text, batch processing operations, or resource-constrained systems experiencing performance degradation may trigger this detection during normal operations.

Associated Analytic Story

Intermediate Findings

Message Entity Field Entity Type Risk Score
[$long_request_count$] long-running inference requests from [$src$] averaged [$avg_response_time$] seconds and reached [$max_response_time$] seconds. Investigate workload, model size, prompt length, and host capacity; access logs do not establish prompt injection or data exfiltration. src system 20

References

Detection Testing

Test Type Status Dataset Source Sourcetype
Validation ✅ Passing N/A N/A N/A
Unit ✅ Passing Dataset server.log ollama:server
Integration ✅ Passing Dataset server.log ollama:server

Replay any dataset to Splunk Enterprise by using our replay.py tool or the UI. Alternatively you can replay a dataset into a Splunk Attack Range


Source: GitHub |

Version: 1