Detection: Ollama Repeated Application Errors

Description

Detects repeated Ollama application error events as a possible availability issue. Ollama application logs do not reliably record process termination, so this search cannot confirm a service crash without operating system or service manager telemetry.

 1`ollama_server`
 2(level=ERROR OR level=FATAL)
 3
 4
 5| rex field=_raw "msg=\"(?<msg>[^\"]+)\""
 6
 7| rex field=_raw "exit_code=(?<exit_code>\d+)"
 8
 9
10| eval event_time=_time
11
12| bin _time span=5m
13
14
15| stats count as error_count
16        earliest(event_time) as first_seen
17        latest(event_time) as last_seen
18        values(msg) as error_messages
19        values(exit_code) as exit_codes
20        dc(msg) as unique_errors
21  by _time host
22
23
24| where error_count > 1
25
26
27| eval first_seen=strftime(first_seen, "%Y-%m-%d %H:%M:%S")
28
29| eval last_seen=strftime(last_seen, "%Y-%m-%d %H:%M:%S")
30
31| eval severity=case(error_count > 5, "critical", error_count > 2, "high", 1=1, "medium")
32
33| eval attack_type="Repeated Ollama Application Errors"
34
35
36| table first_seen last_seen host error_count
37        unique_errors error_messages severity attack_type
38
39
40| `ollama_repeated_application_errors_filter`

Data Source

Name Platform Sourcetype Source
Ollama Server Other 'ollama:server' 'server.log'

Macros Used

Name Value
ollama_server (sourcetype="ollama:server")
ollama_repeated_application_errors_filter search *
ollama_repeated_application_errors_filter is an empty macro by default. It allows the user to filter out any results (false positives) without editing the SPL.

Annotations

- MITRE ATT&CK
+ Kill Chain Phases
+ NIST
+ CIS
- Threat Actors
ID Technique Tactic
DE.AE
CIS 10

Default Configuration

This detection is configured by default in Splunk Enterprise Security to run with the following settings:

Setting Value
Disabled true
Cron Schedule 0 * * * *
Earliest Time -70m@m
Latest Time -10m@m
Schedule Window auto
Creates Finding (Notable) No
Creates Intermediate Finding (Risk Event) Yes
Anomaly detections generate Intermediate Findings (Risk Events). They do not generate a Finding (Notable) directly.

Implementation

Ingest Ollama logs via Splunk TA-ollama add-on by configuring file monitoring inputs pointed to your Ollama server log directories (sourcetype: ollama:server), or enable HTTP Event Collector (HEC) for real-time API telemetry and prompt analytics (sourcetypes: ollama:api, ollama:prompts). CIM compatibility using the Web datamodel for standardized security detections.

Known False Positives

Recoverable model load failures, temporary registry or network errors, resource constraints, and other repeated application errors may trigger this exploratory signal without causing a service outage.

Associated Analytic Story

Intermediate Findings

Message Entity Field Entity Type Risk Score
Repeated Ollama application errors detected on host [$host$] between [$first_seen$] and [$last_seen$]: [$error_count$] events and [$unique_errors$] unique error types. Correlate with service manager logs to determine whether availability was affected. Error messages: [$error_messages$]. host system 20

References

Detection Testing

Test Type Status Dataset Source Sourcetype
Validation ✅ Passing N/A N/A N/A
Unit ✅ Passing Dataset app.log ollama:server
Integration ✅ Passing Dataset app.log ollama:server

Replay any dataset to Splunk Enterprise by using our replay.py tool or the UI. Alternatively you can replay a dataset into a Splunk Attack Range


Source: GitHub |

Version: 1