Alerts can be created on Splunk searches, the easiest way is to use an existing search and apply the threshold to it, this can be done be selecting "Last 4 hours" (To search on raw un-summarized data)
Afterwards you can hover over the table/chart and select the small search icon to open the search in a new tab:
You can then add the where filter to limit on a field, in this example "Avg. Response Time" called art:
| where art > 15.0
The alert will trigger based on results found and the time range selected:
Afterwards select "Save As" -> "Alert"
Fill out Title, Description and when it should run. (Please note that Real-time alert type will occupy one CPU constantly)
Select Scheduled and run every hour, you will have to specify Cron tab if it has to run every X minute, for example: */15 * * * * for every 15 minutes.
(Enough CPU's will need to be available, searches can be skipped if limit for concurrent searches is reached, monitor skipped searches in the Splunk Monitor Console -> Search -> "Scheduler Activity: Instance" if you experience issues)
Trigger condition is when results are found, it can send email each time the search runs and there are found results or Throttle it so it only send each X hour.
Add Email action under "Trigger actions" and setup the recipients.
Note that you will need to have SMTP server setup correctly under Settings -> Server Settings -> Email settings, only way to test the SMTP server is to "Schedule PDF delivery" on dashboard.
Search examples:
| Alert | Splunk SPL search |
| Alert when average response time is above 15.0 seconds for Web data. | `getWebData` app_key="*" | stats avg(time) AS average_response_time_sec by _time, name | where average_response_time_sec > 15.0 |