Setting up integration between UXM (Splunk) and ServiceNow involves configuring both platforms to communicate effectively so that events and incidents detected in UXM/Splunk can be automatically sent to ServiceNow for further action or resolution.
Example
Endpoint has less than 28 GB left on system disk
UXM detects that Endpoint is having under 28 GB disk space left on OS disk and creates ServiceNow incident.
Results from UXM can be added to short description, comments, etc for details on endpoint status when incident was created.
Assignment group can be dynamically set based on Country/Region that endpoint machine is located in. (Recommended to use region/location from Active Directory OU to avoid that users traveling or on VPN is created in incorrect Assignment Group)
Clicking on Splunk Drilldown button will open the Endpoint details dashboard inside UXM.
Affected CI / Configuration Item can be set to Endpoint machine to allow ServiceNow team to lookup more info and see other incidents affecting endpoint machine.
After the disk is freed up UXM reports to ServiceNow that incident is solved and auto-closes the ServiceNow incident:
Configuration
ServiceNow configuration
Follow the guide to set and configure Splunks ServiceNow integration: https://docs.splunk.com/Documentation/AddOns/released/ServiceNow/ConfigureServiceNowtointegratewithSplunkEnterprise
The following app will be installed on ServiceNow and Splunk will be able to query the API and create/pull incients.
Mandatory fields will need to be configured/mapped correctly in the Splunk -> ServiceNow transform sets.
UXM configuration
Account
Splunk Add-on for ServiceNow needs to be installed. This needs to be downloaded from the following link: https://splunkbase.splunk.com/app/1928/
Open the Add-on and configure the ServiceNow service account using either Basic or OAth2 authentication:
Inputs
Incidents can be import to Splunk/UXM via input_name incident.
Alerts - UXM Alert Manager
The UXM Alert Manager can handle if no data is seen from Endpoint, Application, Team Room, etc.
Create notification that handles action on creation, auto resolve and closing of alert, actions can be Emails, Teams, Webhooks and SPL to integrate with custom commands from Splunk Add-ons.
Results from Search can be used in actions via $result.fieldName$ token, example on creating ServiceNow incident with results from search:
| makeresults 1
| eval ad_country = lower("$result.ad_country$")
| lookup servicenow_country_to_local_it_group_lookup country AS ad_country OUTPUT servicenow_assignment_group | fillnull value="ServiceDesk" servicenow_assignment_group
| eval account="servicenow_prd"
| eval scripted_endpoint="/api/now/table/x_splu2_splunk_ser_u_splunk_incident"
| eval category=""
| eval short_description="$result.threshold_text$"
| eval contact_type="Event"
| eval splunk_url="https://customername.uxmapp.com/en-US/app/uxmapp/endpoint_details?form.timesel.earliest=-7d%40d&form.timesel.latest=now&lookup_key=$result.context$&form.link=disks&sort=avg_cpu&form.programs_search=*&form.metricspan=1d&form.logname=*&form.event_source=*&form.event_level=*&form.event_id=*&form.rows=25&form.eventlog_search=*&form.devices_search=*"
| eval urgency="5"
| eval impact="5"
| eval subcategory=""
| eval state="1"
| eval comments="[code]<p>UXM has detected that machine $result.hostname$ has $result.free_space_left_gbytes$ GB left on OS Disk $result.mount_point$.</p><p>Please investigate and free up space. Use the Splunk Drilldown link in ServiceNow to see machine info in UXM.</p><p>Workstation with less than 30 GB of available space on their system disks can cause:</p><p><ul><li><b>Update Failures:</b> Windows upgrades to Windows 11 or other updates may fail to install, leaving the system vulnerable to security risks.</li><li><b>Performance Degradation:</b> The system may slow down significantly due to limited space for temporary files and virtual memory.</li><li><b>System Instability:</b> Applications and the operating system may crash or freeze due to insufficient space for necessary operations.</li><li><b>File Corruption:</b> Lack of space can lead to file corruption, especially if the system cannot properly save or manage files.</li></ul></p>[/code]"
| eval location=""
| eval correlation_id="$result.correlationID$"
| eval priority="5"
| eval assignment_group=servicenow_assignment_group
| eval ci_identifier="$result.hostname$"
| snowincidentstream
Create template that defined assigne, initial status, priority and when alert is auto resolved if no data is seen from Endpoint, Application, etc.
Enable Saved Searches for creating and closing incidents.
Alerts - ServiceNow Incident Integration
Alerts can be send to ServiceNow from Scheduled Saved Searches either as Incidents or Events, UXM needs to know the values for the following, which could be custom at customers:
- State
- Contact Type
- Assignment Group
- Category
- Subcategory
- Impact
- Urgency
- Priority
Correlation ID can be used to only create 1x incident and reopen/append to that incident if it already exists: