Overview
UXM is setup to handle 10.000+ Desktop agents and million of Web page requests per day.
The recommended architecture is to setup an Splunk Heavy-Forwarder with UXM (containing the IIS/RabbitMQ queue) and send data via HTTP Event Collector (HEC) to the indexers.
Setup Splunk indexers
Install the indexer app "uxmapp_indexer_YYYY.MM.DD.tar.gz" on the Splunk Indexers.
Activate HTTP Event Collector
Activate the HTTP Event Collector (HEC) on the indexers that should receive the UXM data.
This is done under Settings -> Data Inputs -> HTTP Event Collector -> Global Settings
Write down the FQDN/IP of the Indexer, if SSL is enabled and Port number (Default 8088), these settings will be used later when setting up the Heavy Forwarder.
Create a new HTTP Event Collector and call it "UXM - uxmapp", indexer acknowlegement has to be disabled.
Select:
- Source type: automatic
- App context: UXM Indexers (uxmapp_indexer) or (uxmapp for standalone environment)
- Indexes: Select the 4 indexes uxmapp_confidential, uxmapp_metrics, uxmapp_response, uxmapp_sessiondata
- Default Index: uxmapp_response
Press Preview and Submit, write down the token value, the settings will be used when configuring the Heavy Forwarder and Search Head.
Setup Splunk Search Head
The splunk search head contains dashboards and data models and is where the user analyses the UXM data.
Please note that multiple scheduled searches which creates summary indexes are created by the UXM app, these requires that you follow Splunk best practices and forwards all data from the Search Heads to the Indexers.
Install the following apps on the Search Head. You can skip the restart untill later.
- Search Head app: uxmapp_searchhead_YYYY.MM.DD.tar.gz
- Custom visualization: uxmapp_waterfall_YYYY.MM.DD.tar.gz
- Custom visualization: uxmapp_worldmap_YYYY.MM.DD.tar.gz
Goto Settings -> Data Inputs -> Scripts and enable the script setup/distributed_searchhead_000_setup_app.py. (The script creates default KVStores entries, Splunk roles and Splunk user that allows Heavy Forwarders to access the KVStore on the Search Head), it will auto disable when done.
You can also follow this guide to "Setup Search Head Manually" if you prefer to configure Splunk manually.
You can view the output of the script by running the following Splunk search:
index="_internal" source="*_setup_distributed_searchhead_000_setup_app.log"
Verify roles
There will be 2 new roles after the script has executed called: uxmapp_user and uxmapp_admin:
And a user called uxmapp_wsgi, reset the password for the user and disable that password change is required on next login, store the password it will be used later when setting up the Heavy Forwarder.
Setup/verify permissions for app
Goto Apps -> Manage Apps and click permissions on the uxmapp app.
Add read permissions for the newly created uxmapp_user and read+write permissions for the uxmapp_admin user.
Setup/verify UXM configuration
Open the UXM app, it will ask you to configure it, enter HTTP Event Collector Hostname and Token
Apply license and press save.
Enable Splunk batch processing scripts
Enable following Data Input script under Settings -> Data Input -> Scripts:
- check_license.py
- daily_maintenance.py
- task_generate_tags.py
- update_alert_event_summaries.py
- update_kvstores.py
- update_applications.py
- update_endpoint_groups.py
The Splunk Search Head needs to be restarted afterwards when all configuration is done.
Setup Heavy Forwarder
The Splunk Heavy Forwarder (HF) receives the data and processes it according to the configuration on the Splunk Search Head KVStores. It also respond with configuration to the UXM Desktop agents when they synchronize hourly.
IIS and RabbitMQ is needed to control the data retrieval and queuing to avoid overloading the HF or Splunk environment, because receiving data from Desktop endpoint and public websites requires a high number of TCP connections.
Setup RabbitMQ
Install newest version of RabbitMQ and Erlang - https://www.rabbitmq.com/install-windows.html
Open elevated command prompt as administrator and run commands below to configure new virtual host and user for UXM:
(Replace GeneratedRabbitMQPassword with own password, and save it for configuration in Splunk)
cd "C:\Program Files\RabbitMQ Server\rabbitmq_server-4.0.5\sbin" rabbitmq-plugins enable rabbitmq_management rabbitmqctl add_user uxmapp GeneratedRabbitMQPassword rabbitmqctl set_user_tags uxmapp monitoring rabbitmqctl add_vhost /uxmapp/ rabbitmqctl set_permissions -p /uxmapp/ uxmapp ".*" ".*" ".*" rabbitmqctl delete_user guest
Install app
Install the app "uxmapp_heavyforwarder_YYYY.MM.DD.tar.gz" on the Splunk Heavy Forwarder. You can skip the restart untill later.
Configure app
Open the UXM app, it will ask you to configure it, use same Agent Key as the Search Head, enter KVStore, HTTP Event Collector and RabbitMQ settings, leave rest of values as default and press save.
Storage path is for UXM Desktop agent log files and UXM Robot agent video, screenshot and log results, can be skipped.
Save the generated Agent Key for later when deploying the UXM Desktop agent to endpoints. See Deploying Desktop Agents
Save and restart Splunk.
Setup IIS
IIS is used to create web front for uWSGI data receiver.
We recommend for increased security that you setup HTTPs certificates or use an Reverse Proxy if data has to be received from outside the company network.
UXM Web agent and UXM Browser extensions requires that valid HTTPs certificate is configured, because data is send directly from the users browser using the same HTTP/HTTPs security that monitored website has. (Agents after 2022.08.01 sends browser data through UXM Desktop agent)
Used for | Splunk script | Reverse Proxy endpoint |
---|---|---|
Desktop/Robot agent data receiving |
bin\task_mq_consumer_pcagent.py reads from RabbitMQ queue. |
IIS / uWSGI adds to RabbitMQ queue |
Web agent data receiving | bin\task_mq_consumer_web.py reads from RabbitMQ queue. |
IIS / uWSGI adds to RabbitMQ queue Have to respond with following headers:
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, OPTIONS
Access-Control-Allow-Headers: origin, content-type, accept, LoginRequestCorrelationId
Content-Type: text/plain
|
Prerequisites: Install IIS role, this can be done from the Server Manager by clicking "Add roles and features"
Select Web Server (IIS) under Server Roles:
Press Next twice and select the following Web Server Role Service options
Ensure that CGI is checked under Application Development.
Setup data collection website
Open Internet Information Services Manager and remove the default site
Right-click on sites and select Add Website, give it following information:
Site name: UXM
Application pool: UXM
Physical path: C:\Program Files\Splunk\etc\apps\uxmapp\bin\wsgi
Binding: Port 80
Edit bindings and add HTTP's to receive data from UXM Web Agent or UXM Browser Extensions, this step can be skipped if you are offloading the HTTPs to an external load-balancer/reverse proxy.
Configure folder permissions
IIS/Python needs to be able to access the following folders, execute the commands in an elevated cmd prompt:
icacls "C:\Program Files\Splunk\etc\apps\mcg_uxm" /grant "IIS AppPool\UXM":(OI)(CI)(RX) /T icacls "C:\Program Files\Splunk\etc\apps\uxmapp" /grant "IIS AppPool\UXM":(OI)(CI)(RX) /T icacls "C:\Program Files\Splunk\etc\apps\search\lookups" /grant "IIS AppPool\UXM":(OI)(CI)(RX) /T icacls "C:\Program Files\Splunk\share" /grant "IIS AppPool\UXM":(RX) /T icacls "C:\Program Files\Splunk\etc\auth\splunk.secret" /grant "IIS AppPool\UXM":(R) icacls "C:\Program Files\Splunk\var\log" /grant "IIS AppPool\UXM":(OI)(CI)(R,W,M) /T
Unlock system.webServer/handlers
Open IIS Manager and go to root server and select Configuration Editor.
Select system.webServer/handlers and click "Unlock Section"
Install Python and wfastcgi module
Install newest python 3.13.x from https://www.python.org/downloads/windows/.
Check "Use admin privileges ..." and select "Customize installation".
Under Advanced Options select "Install for all users" and install under "C:\Python313". (Avoid long paths or whitespaces in the path)
"C:\Program Files\Splunk\etc\apps\uxmapp\bin\wsgi\web.config" will have to be modifed if using other path than "C:\Python313".
Open eleveated command prompt as administrator and execute: (Cryptography and CFFI is limited to older versions to support Splunks embedded OpenSSL 1.0.2 and Python 3.7)
"C:\Python313\Scripts\pip.exe" install wfastcgi
Collecting wfastcgi Using cached wfastcgi-3.0.0.tar.gz (14 kB) Using legacy 'setup.py install' for wfastcgi, since package 'wheel' is not installed. Installing collected packages: wfastcgi Running setup.py install for wfastcgi ... done Successfully installed wfastcgi-3.0.0
Enable wfastcgi in IIS, see https://pypi.org/project/wfastcgi/:
"C:\Python313\Scripts\wfastcgi-enable"
Open IIS Manager -> Server -> FastCGI Settings and edit the python item.
Set "Instance MaxRequests" to 1.000.000 and Max Instances to 4 to avoid that fastcgi recycles too often.
Verify that data collector url work
Open http://localhost/data/browser and validate that it returns "no get/post data received".
Enable Splunk batch processing scripts
Enable scripts for UXM Web and UXM Desktop agent data processing.
Goto Settings -> Data inputs -> Script and enable the "task_mq_consumer_pcagent.py consumer1" and "task_mq_consumer_web.py consumer1"
Standalone environment will have to enable the processing scripts defined under: https://support.mcg.dk/hc/en-us/articles/360004216951#scripts
Check for errors
Open the UXM app on the Heavy Forwarder, the default dashboard will show status on installation and report any errors detected.
PCAgent and Web consumer will show following info if everything works: