Splunk Integration
Prerequisites
| Requirement | Details |
|---|---|
| Splunk instance | Splunk Cloud or Splunk Enterprise 8.x+ |
| Splunk role | Admin role for HEC token creation and saved search management |
| Splunk ES | Splunk Enterprise Security required for notable event mapping (optional) |
| StackFlow role | super_admin |
Step 1: Create HEC Token
The HTTP Event Collector (HEC) token allows StackFlow to forward logs and events to Splunk.
- In Splunk Web, go to Settings → Data Inputs → HTTP Event Collector → New Token
- Name it StackFlow Events
- Set the Source Type to
_json - Set the Default Index to your preferred index (e.g.,
itsm_events) - Click Review and Submit
- Copy the generated token value
HEC URL: For Splunk Cloud:
https://http-inputs-<your-stack>.splunkcloud.com:443/services/collector. For Splunk Enterprise: https://splunk-host:8088/services/collector. Enable HEC in Settings → Data Inputs → HTTP Event Collector → Global Settings if not already enabled.
Step 2: Configure Saved Search to Incident
Configure Splunk saved searches to trigger StackFlow incident creation via webhook alert actions:
- Create or edit a saved search in Splunk
- Under Alerts → Add Actions → Webhook, set the URL to:
https://your-instance.stackflow-tech.com/prod/api/webhooks/inbound/splunk - Set the payload to the StackFlow Splunk payload template:
{
"search_name": "$name$",
"result_count": "$job.resultCount$",
"search_uri": "$job.searchId$",
"severity": "$result.severity$",
"host": "$result.host$",
"source": "$result.source$",
"message": "$result.message$",
"stackflow_priority": "$result.stackflow_priority$",
"stackflow_assignment_group": "$result.stackflow_assignment_group$",
"timestamp": "$trigger_time$"
}
- Add a custom header:
X-StackFlow-Secret: YOUR_WEBHOOK_SECRET
Step 3: Notable Event Mapping
For organizations using Splunk Enterprise Security, StackFlow can consume Notable Events from the notable index and create ITSM records:
{
"notable_event_mapping": {
"search": "index=notable | search status=new earliest=-15m | table event_id, rule_name, severity, dest, src, owner",
"poll_interval_minutes": 5,
"severity_mapping": {
"critical": "P1",
"high": "P2",
"medium": "P3",
"low": "P4"
},
"field_mapping": {
"short_description": "rule_name",
"ci_identifier": "dest",
"source_ip": "src"
}
}
}
Step 4: Log Forwarding to Splunk
StackFlow can forward incident, change, and audit log data to Splunk via HEC for SIEM analysis:
# Verify HEC is accepting events
curl -k https://splunk-host:8088/services/collector/event -H "Authorization: Splunk YOUR_HEC_TOKEN" -H "Content-Type: application/json" -d '{"event": {"source": "stackflow", "type": "test", "message": "HEC connectivity test"}, "sourcetype": "_json"}'
# Expected: {"text":"Success","code":0}
Configure forwarded event types in Admin → Integration Hub → Splunk → Log Forwarding:
| Event Type | Splunk Index | Default |
|---|---|---|
| Incident lifecycle events | itsm_incidents | Enabled |
| Change events | itsm_changes | Enabled |
| User authentication events | itsm_auth | Enabled |
| AI query logs | itsm_ai | Disabled |
| API access logs | itsm_api | Disabled |
Step 5: Connect in StackFlow
- Navigate to Admin → Integration Hub → Splunk → Configure
- Enter the Splunk HEC URL and HEC token
- Configure the saved search webhook secret
- Configure Notable Event polling (if using Splunk ES)
- Enable and configure log forwarding
- Click Save & Test
Field Reference
| Field | Required | Description |
|---|---|---|
| HEC URL | Yes (if forwarding) | Splunk HEC endpoint URL |
| HEC Token | Yes (if forwarding) | Splunk HTTP Event Collector token |
| Webhook Secret | Yes (if receiving) | Shared secret for inbound Splunk alert webhook verification |
| Splunk REST URL | ES only | Splunk REST API URL for notable event polling |
| Splunk Username | ES only | Username for notable event polling API auth |
| Splunk Password | ES only | Password for notable event polling (stored in Secrets Manager) |
| Severity Mapping | No | Map Splunk severity levels to StackFlow priorities |
| Log Forwarding | No | Configure which event types to forward to Splunk |
Testing the Integration
- Click Test HEC Connection to verify Splunk HEC is reachable and the token is valid
- Manually send a test Splunk alert payload and verify an incident is created
- Create a StackFlow incident and verify the log event appears in Splunk within 60 seconds
# Test inbound Splunk webhook
curl -X POST https://your-instance.stackflow-tech.com/prod/api/webhooks/inbound/splunk -H "Content-Type: application/json" -H "X-StackFlow-Secret: YOUR_SECRET" -d '{"search_name":"High Error Rate","severity":"high","host":"app-server-01","message":"Error rate exceeded 10%","stackflow_priority":"P2","stackflow_assignment_group":"Platform Engineering"}'
Troubleshooting
| Issue | Cause | Fix |
|---|---|---|
| HEC returns 403 | Invalid HEC token | Re-generate HEC token in Splunk → Settings → HTTP Event Collector |
| HEC returns 400 (Invalid data format) | JSON payload not wrapped in event key | Ensure HEC payload has format {"event": {...}} |
| Saved search webhook not firing | Alert throttling in Splunk | Increase alert throttle window or disable throttling for the saved search |
| Notable events not polling | Splunk REST API credentials invalid | Verify username/password; check Splunk user has search role |
| Log forwarding delayed | SQS queue backing up | Check StackFlow-Events-Ingestion.fifo queue depth in CloudWatch |