Discovered
UXM triggered alerts that high number of crashes was occurring on endpoint and that endpoints was experiencing higher than normal CPU usage.
Actions
Technicians investigate which proceses that crashed and could see that it only occurred for msedgewebview2.exe on 6 of their 90 citrix servers.
Average CPU had increased from 20% to 70% since the 4th august due to msedgewebview2.exe crashing constantly, causing windows to launch a process dump through the process werfault.exe.
Solution
Technicians investigated where msedgewebview2.exe was launched from and discovered that it was Outlook Desktop app that launched 6x msedgewebview2.exe processes when Calendar appointments was opened up by their Citrix users.
Only 6 out of 90 citrix servers had the Edge WebView2 embedded browser installed, because Office 365 was pushing it out automatically via the Office 365 package. Ref: https://docs.microsoft.com/en-us/deployoffice/webview2-install
CodeIntegrity events was also seen in the EventLog due to Citrix hooking into the msedgewebview2.exe process.
XenDesktop/XenApp 7.9 and later utilizes Kernel APC Hooking as a replacement of AppInit_DLLs used in previous versions. All Citrix Hooking (including MfApHook.dll and MfApHook64.dll) was disabled by creating the following registry value and the issue disappeared, ref: https://support.citrix.com/article/CTX107825/how-to-disable-citrix-api-hooks-on-a-perapplication-basis:
Key: HKLM\SYSTEM\CurrentControlSet\services\CtxUvi Value Name: UviProcessExcludes Type: REG_SZ Value:msedgewebview2
The fix was pushed out after testing to the rest of the Citrix farm via GPO’s.