Task Manager output for ‘Top Process PowerShell script management pack’
Ever wish you had task manager output when you had a monitor go unhealthy? Following Kevin Holman’s lead to ‘Monitor Processes‘, the idea landed to build out the ‘Top Process PowerShell script’. This morphed into a management pack with Knowledge entries to better explain what is being done. Integrating Top Process into Health Explorer output as a recovery task helped provide another step before alerting. The idea started from the need to prove which Security tool(s) were causing the over-utilized compute spikes, causing non-responsive server(s). Thinking back to my UNIX days, we simply used top, vmstat, iostat, and other commands to identify problematic processes. Integrating PowerShell scripts into SCOM is part of the fun, then linking the obfuscated Security processes for the final output. From there, extrapolate into Azure Functions or Azure Logic apps, for additional functionality for cloud native monitoring.
Kevin Holman built a ‘ Monitor.Performance.ConsecSamples.ThenScript.TwoState.mpx fragment, beginning the logical journey. His fragment helped me start with a working model, taking processes and cores into consideration for true CPU usage on multi-core servers.
Kevin Holman Monitor performance then script fragment for PowerShell get-counter syntax
We need to see the processes, and their corresponding value, then build an output table (custom object). After gathering the processes, feed the TopProcesses array, lastly sorting the array for CPUValue
Top Process memory usage snippet
Next, we’ll want to see what applications/tools might be involved, including Active Client, IIS, monitoring, and EndPoint Management tools (keep things honest!).
Added the Security Processes into the mix
Then we build an output of the data so we can take the datasource (DS) or WriteAction (WA) into a scripted monitor/rule, or recovery tasks linked to various monitors. Even built a forked version in case of SAW/Red Forest, separating Tier0 monitoring from Tier1 (snippet below is NOT that pack)
snippet of manual tasks and recoveries that link to multiple monitors
Time to ‘create a Closed Alerts view’ for all users (versus an individual user workspace). Sometimes, we just need a different view. My thanks to Joe Kelly for his help documenting this!
Create Closed Alerts view
How to ‘create a Closed SCOM Alert’ view. Follow these steps:
Open the Operations Console and navigate to the Monitoring workspace.
Click on the “New” button in the toolbar and select “Alert View” from the dropdown menu.
In the “Create Alert View” wizard, give your view a name and select “Closed Alerts” as the criteria.
Click “Next” and select the columns you want to display in your view.
Click “Next” again and choose any grouping or sorting options you want to apply.
Click “Finish” to create your view.
SCOM Closed alert view
Once you have created your closed alert view, access the new view from the Monitoring tab. Customize the view further by right-clicking on it and selecting “Properties”. From there, you can add or remove columns, change the grouping or sorting, and apply filters to further refine the view.
Learn article here to help personalize views like ‘Create Closed Alerts view’
I’m ISO (in search of) the mythical single pane of glass. In my best Steve Irwin voice… Integrate SCOM and SolarWinds – We are strong together. To me, integration occurs everywhere, at home, in your job, anyone you work with. Contribute, don’t consume 🙂 Everyone is unique, including preferences, and past experiences. In my career, I’ve been lucky to administer both tools for Fortune 100 companies (and more tools) across my career. I hope this blog provides a way to use both tools to get the full value for the least cost!
Integration typewriter picture – stronger together – integrate SCOM and SolarWinds
Integration
The real meat of this is how to get the most for the least cost!
Let’s ‘Integrate SCOM and SolarWinds’ into our unicorn. To date, getting data from SolarWinds into SCOM allows for easiest PowerBI Apps/reports
In my experience, the MSI requires Solarwinds Support login to download the SolarWinds Management Pack for SCOM.msi. Downlload and configure article here.
SCOM view of SolarWinds data
3rd party options (free)
Cookdown vendor method to grab SolarWinds data into SCOM
Most times the Remedy application is old, deprecated, outside it’s service life, rarely integrated with monitoring. One customer’s PowerBI report shows the utility at a glance (see picture below)
PowerBI report referencing ITSM insights for teams, totals, volume
I think of My Big Fat Greek wedding to ‘Compare SolarWinds and SCOM’. The wedding reception, where the father says the root of his daughter, and son-in-law’s last names, are from the greek word for Orange, and Apple. “so in the end, we’re all fruits” We are the same but different, where diversity and inclusion is key. Everyone’s got a voice. Contribute, don’t consume 🙂
First, I’ve been lucky to administer both tools for Fortune 100 companies (and more tools). Second, I hope this blog provides some clarification of the strengths, weaknesses, and costs associated with both tools. Here’s hoping wordpress readers identify with my background – saving money, cutting coupons, looking for on-sale, buy one get one deals. Thirdly, while everyone’s past experiences may not be the same, cost is still a big factor. Lastly, proprietary tools, Security, and other requirements can make or break an implementation.
Here’s a link to a PPT built to ‘Compare SolarWinds and SCOM’ feature wise, that goes along with ‘My Big Fat Greek Wedding’ and the fruit. PPT title ‘better together’, is loaded with links and breaking out key capabilities.
Some items NOT covered in the PPT comparison
Example context – SAW/PAW/Red Forest
Both tools can store credentials within the application, obfuscated.
SCOM allows gMSA’s (managed service accounts) for key services including run as accounts. View the Monitoring Guys blog plug here for CJ, Scott, and Tyson’s contributions 😛
COST
SolarWinds small enterprise example
Windows Server, SQL licenses (no cost given)
Monitors Windows, Non-Windows, Microsoft products
Community of custom application monitoring
Renewal cost per year in 2020 $48K/year
Add HA for SQL Enterprise licenses is same, where SW HA/High availability is the SolarWinds cost, not compute licenses for Windows Server, SQL
***500 license SAM, VOIP, IPAM, NPM/NCM.
Redesigning licensing to unlimited (site license) was $344K
Wow! Site licenses cost considerably more.
Though for clarification, 500 licenses equates to 500 monitors targeted at 500 servers.
SolarWinds costs broken out by feature
Add unlimited VMAN, DPA, SCM, VNQM adds $256K
Migrate functionality to site license ($48K > $344K)
Adding SolarWinds features with site unlimited licenses
SCOM small enterprise example
Windows Server, SQL licenses (no cost given)
No license limitation for products/features used, community built solutions
Monitors Windows, Non-Windows, Microsoft products
Large community of custom application monitoring
No yearly support costs (included with Microsoft support agreement)
SQL Enterprise licenses is same, where SW HA/High availability is the SolarWinds cost, not compute licenses for Windows Server, SQL
ESX monitoring via NiCE VMWare 3rd party pay pack is $10K/year
OpsLogix Teams integration helps with NOC/NOSC/SOC integration
Including NiCE Oracle monitoring $10k/year
I’ll leave the cost comparisons to you.
Securing the Applications and web consoles
SolarWinds (SW)
Secure SW website search, Smart Cards post, 2FA/MFA/RSA post
NO DISA STIG for SolarWinds, so IIS, Windows Server, SQL, WebServer ALL apply
NOTE: I’ve NOT supported SolarWinds recently to see Security scans for other vulnerabilities and STIG settings (Windows Server, SQL, IIS, Network blog. STIG dashboard ‘how to’
Licensing
Licensing is a big differentiator cost wise
SolarWinds needs an EA for Windows Server, SQL licenses.
SCOM has been part of the EA (Enterprise agreement) for at least 15+ years (since SCOM2007, if not MOM2005). Windows Server license (now CPU based), SQL license, however NOT enterprise comes standard. One reason the System Center suite is successful might be this built-in licensing, as well as the feature depth and cost the tools provide.
Hardware requirements
In my experience interacting with customers, SolarWinds support recommends hardware configuration well above vendor recommendations. Support recommendations requesting high compute to provide memory level SQL speed and responsive web console. However, the compute is basically ESX host level compute in the realm of 128GB of memory per server, in High Availability (HA), meaning x4 – 2 servers for 2 sites.
Monitoring tools are rarely Tier1 Applications with respective Service Level Availability (SLA). Expectation alone presents a disparity, and false impression. People just see a tool and base on personal experience.
Ferrari vs. GMC Cyclone – fooled you eh
Is it really surprising if one is faster than the other?
What does your mind link to with the FIPS acronym? FIPS makes me think of the movie Greyhound where Tom Hanks says LT Flipper, instead of Fippler, all that said being ZERO to do with resolving ‘STIGs for SCOM FIPS compliance on Windows’
The biggest hurdle to ‘STIGs for SCOM FIPS compliance on Windows’, is obtaining the files. The current bundled SCOM ISO’s since 2012 SP1 do NOT contain the gacutil, and cryptography DLL files, to resolve STIG V-220942 (win10), V-226335 (Server 2012/2012R2), V-73701 (Server 2016), V-93511 (Server 2019), V-254480 (Server 2022). As much as we want to resolve FIPS ‘STIGS for SCOM FIPS compliance for Windows Server’, gotta start with the finding relevant files. My thanks to Nathan Gau, Tyson Paul, and Aakash Basavaraj, for their involvement and clarification.
Install DLL for STIGs for SCOM FIPS compliance on Windows
Time to mitigate!
Let’s begin to fix the SCOM Web Console role servers (possibly SQL SSRS and PowerBI Report Server included) for resolving multiple ‘STIGs for SCOM FIPS compliance for Windows Server’. Blog post applies to multiple STIG(s) including STIGs V-220942, V-226335, V-73701, V-93511, V-254480
Download files
Whether from blog download link, or if you have the old ISO’s to obtain the DLL, and server ISO for gacutil , or myvisualstudio.com link
Download SCOM ISO from my.visualstudio.com/Downloads?q=operations
If you downloaded from my.visualstudio.com, extract from ISO.
Copy files to IIS role servers (SCOM web console, SSRS, or PowerBI report Servers) to setup files for FIPS compliance.
Download the DLL to the SCOM default folder –
Best practice is SCOM Default folder on non-system disk @
HA HA HA, that’s so funny. An error I didn’t expect importing the latest SQL packs ‘Updating SQLServer packs to v7.2.0.0’
Quick public service announcement – remove the SQL Server Core Custom Monitoring pack before ‘updating SQLServer packs to v7.2.0.0’! Read to save time and frustration, before importing the packs, as the previous 7.0.42.0 pack isn’t upgradable to v7.2.0.0.
Much like the character from Seinfeld, finding out that the ‘V-237434 SCOM Web Console SSL Settings’ is NOT STIG Compliant (STIG’d), is just as tramatic as being hungry, and told ‘No soup for you!” With all the many STIG findings, here’s a quick and dirty way to resolve the finding.
Kevin Holman SCOM QuickStart guides for SCOM 2019, SCOM 2022
V-237434 SCOM Web Console SSL Settings
STIG V-237434 requires trusted CA SSL certificates. Previous July blog posts are related to the effort to secure the SCOM web console. The redirect post forces HTTPS, complimenting this STIG finding. As the STIG states, remediation verification that IIS web site binding is HTTPS, and remove HTTP.
Remediate SCOM servers with Web Console role
Assumption = SmartCards are used for authentication, this part is applicable, otherwise skip.
RDP to server, connect to IISManager
Expand IIS Server > Expand Sites > Expand Default Web Site
IIS Manager Default Web Site menu
Click on SSL Settings
If the menu is greyed out, follow the SCOM WebConsole settings blog to setup the SSL certificate. Once complete, proceed below.
Click on SSL Settings > Check box to ‘Require SSL’
If menu is NOT greyed out, click radio button to ‘Accept’ client certificates
Click Apply
IIS Manager, Default Web Site, SSL Settings default when NOT running SSL certificate and bindings
Click on Default Website on left hand pane
In the Actions Pane (right hand side), click on Restart to restart the IIS website
Restart IIS website from IIS manager actions pane
IIS Website bindings
Next pieces is to verify the SSL HTTPS binding is setup correctly. In case you got disconnected, or rebooted the server
RDP to server, connect to IISManager
Expand IIS Server > Expand Sites > Expand Default Web Site
In the Actions pane on the top right, click on Bindings
IISManager, Default Web Site, Actions Pane, Bindings to setup HTTPS and remove HTTPS
Kevin Holman’s QuickStart blog(s) for SCOM 2019, SCOM2022 setup default HTTP binding (i.e. NO SSL cert configured)
Default website, Bindings selection showing HTTP if following SCOM quick start
If HTTP ONLY, click the Add button
Change dropdown for Type to https
Enter Host Name
Click Select to choose the SSL cert
Click OK
Adding HTTPS Binding with server name, SSL cert drop down and selected
Verify SSL certificate added
IIS HTTPS Bindings with SSL cert
If you have the binding above, change your STIG CKL finding and document as NOT a finding, for V-237434 SCOM Web Console SSL Settings!
Authentication Authentication Authentication! SCOM Web Console authentication settings
SCOM Web Console authentication settings discussion! Let’s go through standard IIS authentication settings like disabling Anonymous Authentication, and enabling Windows Authentication, AD Client Certificate Authentication, and binding providers (Negotiate before NTLM). Ready to begin?! A shout out to Alden Hatten as we worked through this and resetting the Web Console run here recently, that brought up the urgency to document.
Kevin Holman’s SCOM QuickStart guides for SCOM 2019, 2022 (Including WebConsole default setup steps)
SCOM Web Console Authentication settings defaults
RDP to server with SA or Local admin level account
Go into IISManager > Expand the tree to then click on ‘Default Web Site’
Click on Authentication
IIS Manager output for ‘Default Web Site’
IISManager Default Authentication settings
SmartCard aka AD Client Certificate Authentication defaults
In IIS Manager for the server > Click on Authentication
Verify AD Client Certificate Authentication is added and enabled.
IIS Manager Authentication, with SmartCard or Client Certificate Authentication
Windows Authentication
Set Authentication Providers order
From IIS Manager > Expand Default Web Site
Click on Authentication > Click on Providers at the top right
If Negotiate is not on top, highlight, and click Move Up button > Click OK to set. Restart IIS to make setting take effect ( also use iisreset from command prompt or PowerShell )
NOTE: Anonymous Authentication should be disabled!
IIS Manager Authentication, Windows Authentication, Providers, Negotiate on top
If screenshot is your setup, close the Providers window
After reviewing these authentication settings, you should be one step closer to encrypted authentication.
Use this post when the SCOM WebConsole gets flagged for HTTP Redirect. The IIS configuration is pretty easy to set up. When your Security team contacts you to resolve VulnID 121040, the steps below should resolve the compliance finding. Use the Microsoft learn site for more details.
Add HTTP Redirect role from Server Manager
Time to Configure ‘SCOM WebConsole HTTP Redirect’
RDP to server, open Server Manager
Click on Manage on top right
Click Next on the ‘before you begin popup’
Server Manager splash screen
Click Next
Server Manager Role Installation Type popup wizard
Click Next
Server Manager Destination Manager screen
Expand the ‘Web Server’ drop down menu
Server Manager Roles
Expand Web Server drop down menu
Expand Common HTTP Features
Check box for HTTP Redirection
Server Manager Roles expanding Web Server for HTTP Redirect
Click Next
Server Manager HTTP Redirection check box selected
Click Next at the Features tab
Server Manager Features window
Click Install to install the feature
NOTE the checkbox to ‘Restart if required is NOT selected’
Most change processes don’t allow this on the fly (unplanned outage)
Server Manager Selections window
Wait while the feature(s) install
Click Close once complete
Server Manager feature install in progress
Setup Redirection in IIS Manager
Open IISManager
NOTE If IISManager was open before the feature was closed, exit and open IISManager again. IISManager refresh does NOT make HTTP Redirect reappear (even if restarting IIS service).
Click on your webServer > Double click on HTTP Redirect
IIS Manager with HTTP Redirect
IISManager HTTP Redirect Default splash screen
Check the ‘Redirect requests to this destination:’ check box
Enter the WebConsole URL for your installation.
NOTE SCOM default WebConsole URL is http://<webserverName>/OperationsManager
Check the two (2) boxes for Redirect behaviors
IISManager HTTP Redirect configuration screen
Click Apply
Recommend restart/reboot of server (off hours) to apply configuration before having Security team scan server.
Verify HTTP Redirect after reboot
After reboot, verify current settings (shown are default)
Click on ‘Default WebSite’ dropdown > Select HTTP Redirect
I attribute Kerberos AD delegation as the Navajo and Comanche helped allies in WW2, encrypted and encoded communication
Next on the list is to setup SCOM WebConsole settings for Kerberos AD Delegation. I attribute Kerberos AD delegation as the Navajo and Comanche helped allies in WW2, encrypted and encoded communication. Time to make the donuts! (to setup SCOM WebConsole settings for Kerberos AD Delegation)
If you’re improperly setup – you’ll flag on STIG configs V-243470, V-243478
Configure delegation on SCOM and/or PowerBI servers
Take the list of affected servers, to take action. Use the steps below to configure relevant SCOM or PowerBI servers.
Configure SCOM Web Console server
With domain administrator (DA or Tier0) rights, open the Active Directory Users and Computers MMC snap-in.
From ADUC > change ‘Find’ drop-down to Computers
In the Computer name text box, enter <SCOMWebConsoleServerName> and click search
Right click the server in the results box > Select Properties.
Select the Delegation tab.
Select Trust this computer for delegation to specified services only > Use any authentication protocol.
Under Services to which this account can present delegated credentials, select Add.
In the new dialog box, select Users or Computers.
Enter <SCOMWebConsoleServerName>, and then select OK.
Click the Add button to add services
Select the w3svc and www processes
Select OK.
ADUC SCOM Lab server choosing process
Verification of delegation settings
ADUC Delegation flags with SCOM MS processes selected.
Depending on replication times for the forest, wait and later reboot <SCOMWebConsoleServerName> to have settings take effect.
PowerBI Report Server
With domain administrator (DA or Tier0) rights, open the (ADUC) Active Directory Users and Computers MMC snap-in. NOTE: RSAT tools recommended to be installed on SCOM Management Server(s)
In the Search text box, enter PowerBI service account <Example can be SCOMDataAccessReader Account> and click search
Right-click the PowerBI service account <Example can be SCOMDataAccessReader Account>, select Properties.
Select the Delegation tab.
Select Trust this computer for delegation to specified services only > Use any authentication protocol.
Under Services to which this account can present delegated credentials, select Add.
In the new dialog box, select Users or Computers.
Enter the service account for the data source, and then select OK.
Select the SPN that you created for <PowerBI Report Server Name>
Select both as FQDN and the NetBIOS names are in the SPN
Select OK.
Back to ADUC (AD Users and Computers), change Find drop-down to Computers
Enter <PowerBI Report Server Name>, and click search
Right click the server in the results box > Select Properties.
Select the Delegation tab.
Select Trust this computer for delegation to specified services only > Use any authentication protocol.
Under Services to which this account can present delegated credentials, select Add.
In the new dialog box, select Users or Computers.
Enter <Example can be SCOMDataAccessReader Account>, and then select OK.
Click the Add button to add services
Select the HTTP process
ADUC Delegation Add Services > HTTP, WWW
Select OK.
ADUC Delegation Settings for http for PowerBI Report Server (PBIRS)
Manage Cookie Consent
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
