Upgrade on Windows (Distributed)

Follow the steps below to upgrade the UXM monitoring solution On-Premise, the software can be downloaded from here: Download server software.

Please note that older uxmapp installations calls app and indexes for mcg_uxm, newer installations uses uxmapp_ as prefix for app and indexes

Upgrade Splunk

Ensure Splunk are upgraded to newest version on both Search Heads, Indexers and the Heavy Forwarder where UXM is installed, Splunk 9, 9.1 and newer are supported by UXM.

Upgrade Indexer

No upgrades needed.

Upgrade Search Head

Cleanup old folder and apps

delete old deprecated visualization apps $SPLUNK_HOME\etc\apps\mcg_uxm_waterfall and $SPLUNK_HOME\etc\apps\mcg_uxm_worldmap. (is replaced with uxmapp_waterfall and uxmapp_worldmap)
```
rmdir /Q /S "C:\Program Files\Splunk\etc\apps\mcg_uxm_waterfall\"
rmdir /Q /S "C:\Program Files\Splunk\etc\apps\mcg_uxm_worldmap\"
```
Update with new apps

Goto Apps -> Manage Apps and upload the mcg_uxm_distributed_sh_onpremise_YYYY.MM.DD.tar.gz or uxmapp_sh_YYYY.MM.DD.tar.gz app file depending on app installed on you system.

Remember to click "Upgrade app checkbox" and click Upload, restart Splunk later when asked.

You can continue if error "Error connecting to /services/apps/local: The read operation timed out" occurs, normally app is uploaded and old app overridden successfully.

Upload new uxmapp_waterfall and uxmapp_worldmap visualization apps and restart Splunk later.
Update permissions for uxm roles

For mcg_uxm_admin or uxmapp_admin role add uxmapp_view_* capabilities if needed, allows role to see menu for configuring uxm and inspecting/opening in search on tables/charts.

For mcg_uxm_user or uxmapp_user ensure the following capabilities are enabled:

change_own_password
edit_own_objects
export_results_is_visible
list_all_objects
pattern_detect
rest_properties_get
run_collect
run_mcollect
schedule_search
search
uxmapp_view_* capabilities if needed
Clear Splunk JS/CSS caches

Open /en-US/_bump and click "Bump version" to clear JS/CSS caches.
Enable and delete old input scripts

Open Settings -> Data Inputs -> Scripts

Delete create_new_web_agents.py (if exists)
Delete task_tmp_fill_user_and_boottime.py (if exists, moved to update_kvstore.py)

Enable input scripts
- enable delete_old_files_and_packages.py (Deletes old unused app files and python dependencies, script will disable when done)
- when upgrading to 2023.12.13: distributed_searchhead_23q4_migration.py (Migrates uxmap KVStore collections with new required entries, script will disable when done)
- when upgrading to 2024.09.04: distributed_searchhead_24q1_migration.py (Assigns desktop profiles to SLA's and removes application processes with setup/tmp names, script will disable when done)
- enable check_license.py
- enable update_applications.py
- enable update_endpoint_groups.py
Update setup settings

Goto Apps -> Manage Apps and select "Set up" for mcg_uxm or uxmapp app to set up hostnames and encrypt passwords.

Enter HTTP Event collector hostname and token.

Change retention period for endpoints if needed and press Save.
Update monitor settings

Open uxmapp and select Administration -> UXM Desktop agent -> Monitors

when upgrading to 2023.12.13: Edit Endpoint monitor and enable new event fields for newer agents (keep old monitor events for older agents) and press save.
Overridden dashboards and settings

UXM distributes it's dashboards and settings through uxmapp\default, all changes performed to dashboards and configs inside Splunk is stored under uxmapp\local.

Please ensure that there are no local dashboards overrides our default provided dashboard under C:\Program Files\etc\apps\mcg_uxm\local\data\ui\views or C:\Program Files\etc\apps\uxmapp\local\data\ui\views.

Please compare with the dashboards under C:\Program Files\etc\apps\uxmapp\default\data\ui\views to see if the changes are required and should be kept, restart the splunk server if any changes is made to the files.

Delete the local dashboards that shouldn't be there
Restart the Splunk Search Head.

Upgrade Heavy Forwarder

The UXM Splunk App can be downloaded from here: Download server software

Please note that customers using app mcg_uxm will need to download and upgrade with the mcg_uxm app package.

Stop the IIS/WSGI windows service

Run an elevated command prompt and write iisreset /stop to avoid that IIS locks used files.
```
iisreset /stop
```
Stop Splunk processing scripts

Open Splunk and goto Settings -> Data Inputs -> Scripts and ensure that following UXM scripts is disabled to avoid that they lock files.
- "task_mq_consumer_pcagent.py"
- "task_mq_consumer_web.py".
Upgrade to Python 3.11 (If running older Python versions)

Run an elevated command prompt and uninstall the old wfastcgi python version if installed. (Skip if you are on Python 3.11 already)
```
"C:\Python39\Scripts\wfastcgi-disable.exe"
"C:\Python310\Scripts\wfastcgi-disable.exe"
```
Uninstall Python Launcher, 3.8, 3.9 or 3.10 and install newest Python following install guide Install_Python_and_wfastcgi_module

Edit IIS config file C:\Program Files\Splunk\etc\apps\uxmapp\bin\wsgi\web.config or C:\Program Files\Splunk\etc\apps\mcg_uxm\bin\wsgi\web.config and change Python path if it's incorrect

Open eleveated command prompt as administrator and execute:
```
"C:\Python311\Scripts\pip.exe" install wfastcgi six pycryptodome cryptography
```
Enable wfastcgi in IIS, see https://pypi.org/project/wfastcgi/:
```
"C:\Python311\Scripts\wfastcgi-enable"
```
Upgrade RabbitMQ 3.13.7
Uninstall old Erlang 22/23/24/25 version by removing it from program and features:

Upgrade to newest erlang 26 by downloading newest 26 release: https://github.com/erlang/otp/releases
Direct link: https://github.com/erlang/otp/releases/download/OTP-26.2.5.3/otp_win64_26.2.5.3.exe
Install with default options.

See compatibility matrix between erlang and rabbitmq here: https://www.rabbitmq.com/docs/which-erlang

Download newest 3.13.x release from https://github.com/rabbitmq/rabbitmq-server/releases
Direct Link: https://github.com/rabbitmq/rabbitmq-server/releases/download/v3.13.7/rabbitmq-server-3.13.7.exe
Run installer, confirm to uninstall old version and install 3.13.x with default settings.

Enable all features after upgrading by running command in elevated prompt:
```
cd "C:\Program Files\RabbitMQ Server\rabbitmq_server-3.13.7\sbin"
rabbitmqctl enable_feature_flag all
```
You can verify the version and that RabbitMQ runs by opening http://localhost:15672/ in a browser and logging in with the RabbitMQ username/password from $SPLUNK_HOME\etc\apps\mcg_uxm\local\setup.conf

Re-run RabbitMQ configuration if RabbitMQ has errors:
```
cd "C:\Program Files\RabbitMQ Server\rabbitmq_server-3.13.7\sbin"
rabbitmq-plugins enable rabbitmq_management
rabbitmqctl add_user uxmapp GeneratedRabbitMQPassword
rabbitmqctl set_user_tags uxmapp monitoring
rabbitmqctl add_vhost /uxmapp/
rabbitmqctl add_vhost /mcg_uxm/
rabbitmqctl set_permissions -p /uxmapp/ uxmapp ".*" ".*" ".*"
rabbitmqctl set_permissions -p /mcg_uxm/ uxmapp ".*" ".*" ".*"
rabbitmqctl delete_user guest
```
Encrypted passwords can be decrypted with: "\Program Files\Splunk\bin\splunk.exe" show-decrypted --value $7$57ENCRYPTED_PASSWORD)
Upload new Heavy Forwarder app

Upload the mcg_uxm_distributed_hf_onpremise_YYYY.MM.DD.tar.gz under Apps -> Manage Apps, postpone restarting Splunk.

Setup permissions to folders

Setup permissions to folders to allow IIS to decrypt encrypted passwords: See Configure_folder_permissions

icacls "C:\Program Files\Splunk\etc\apps\mcg_uxm" /grant "IIS AppPool\UXM":(OI)(CI)(RX) /T
icacls "C:\Program Files\Splunk\etc\apps\uxmapp" /grant "IIS AppPool\UXM":(OI)(CI)(RX) /T
icacls "C:\Program Files\Splunk\etc\apps\search\lookups" /grant "IIS AppPool\UXM":(OI)(CI)(RX) /T
icacls "C:\Program Files\Splunk\share" /grant "IIS AppPool\UXM":(RX) /T
icacls "C:\Program Files\Splunk\etc\auth\splunk.secret" /grant "IIS AppPool\UXM":(R)
icacls "C:\Program Files\Splunk\var\log" /grant "IIS AppPool\UXM":(OI)(CI)(R,W,M) /T

Clear Splunk JS/CSS caches

Open /en-US/_bump and click "Bump version" to clear JS/CSS caches.
Run setup in app to set hostnames and encrypt passwords

Open Manage Apps -> UXM -> Set up and enter the missing fields for KVStore hostname, HEC hostname and RabbitMQ hostname.

Ensure RabbitMQ user is new uxmapp or old mcg_uxm if RabbitMQ isn't upgraded.

Press Save in bottom, passwords will be encrypted using Splunk's key and UXM will redirect you to the Enterprise Dashboard.
Start Splunk processing scripts

Open Splunk and goto Settings -> Data Inputs -> Scripts and ensure that following scripts is enabled.
- enable delete_old_files_and_packages.py (Deletes old unused app files and python dependencies, script will disable when done)
- disable/enable task_mq_consumer_pcagent.py (Will be named consumer1 to consumer4)
- disable/enable task_mq_consumer_web.py (Will be named consumer1 to consumer4)
Restart Splunk

Restart "Splunkd Service" from the Windows Service Manager.
Start the IIS/WSGI windows service

Run an elevated command prompt and write iisreset /stop to avoid that IIS locks used files.
```
iisreset /start
```
Test that endpoints works

Open https://localhost/data/browser in a browser and validate that the HF works and returns "no get/post data received"
Check for errors

Open the UXM app on the Heavy Forwarder, the default dashboard will show status on installation and report any errors detected.

PCAgent and Web consumer will show following info if everything works: