As a SME or team lead, ever need to know ‘Proactive Patching alerts’? i.e. What servers need patches applied, aren’t patching, or were missed? This pack builds on three (3) pillars – Health/Security/Compliance, enabling Cyber teams and more. This became an alternate option to a complex pack, with SSRS report, used by a customer to identify systems. The report was long, and had many blank lines/pages, which required a re-write. This pack started with the pending restart monitor directly from the AquilaWeb reboot pack logic. The logic helps SysAdmin/Domain Admin/NOC/NOSC/SOC teams to know when servers need reboots. This need is driven further due to multiple reboots (sometimes) required with Windows monthly updates, and Application updates. Used across multiple customers, this is the first pack enabling a proactive stance to answer the ‘Am I compliant’ question.
David Allen built the ‘Aquilaweb.Support.PendingReboot.Monitor.PendingReboot’ PowerShell monitor, to tell system owners when the pending restart flag was present. Some builds though, make system changes which repeatedly flip the registry key, causing many alerts. Also, downloading the Aquila pack is a trick, as TechNet was retired.
David provided a great idea, which was built upon. This gave rise to the question of, what if the server was not patched, or not rebooted in a period of time? With my Cyber hat on, this became the next piece of content to create. That gave rise to another question – do these scenarios need to reflect in health (monitor), or not (rule)? We’re all about choices, free will, so the pack is built with those options (rules disabled out of the box).
Pending restart monitor XML showing options
The pack is setup to alert with CBS application updates, SCCM/MECM/Config Mgr Endpoint Management updates, and Windows Updates. This has been my experience for the most accurate reflections of alerts on secure builds where Application/System Owner needs to take action.
Last Patch and Last Reboot monitor/rules in the download, are set to 45 days. Tune this value down, if patching occurs at the 30 day mark, increase if you need more time before alerts.
Last Patch Monitor reflecting number of days
Otherwise, download and import into your environment. Depending on your subscription/notification settings, the Proactive set of alerts are built upon the Windows Operating System class. If subscriptions include the class, the notifications are automatic to System/Application owners.
Task Manager output for ‘Top Process PowerShell script management pack’
Ever wish you had task manager output when you had a monitor go unhealthy? Following Kevin Holman’s lead to ‘Monitor Processes‘, the idea landed to build out the ‘Top Process PowerShell script’. This morphed into a management pack with Knowledge entries to better explain what is being done. Integrating Top Process into Health Explorer output as a recovery task helped provide another step before alerting. The idea started from the need to prove which Security tool(s) were causing the over-utilized compute spikes, causing non-responsive server(s). Thinking back to my UNIX days, we simply used top, vmstat, iostat, and other commands to identify problematic processes. Integrating PowerShell scripts into SCOM is part of the fun, then linking the obfuscated Security processes for the final output. From there, extrapolate into Azure Functions or Azure Logic apps, for additional functionality for cloud native monitoring.
Kevin Holman built a ‘ Monitor.Performance.ConsecSamples.ThenScript.TwoState.mpx fragment, beginning the logical journey. His fragment helped me start with a working model, taking processes and cores into consideration for true CPU usage on multi-core servers.
Kevin Holman Monitor performance then script fragment for PowerShell get-counter syntax
We need to see the processes, and their corresponding value, then build an output table (custom object). After gathering the processes, feed the TopProcesses array, lastly sorting the array for CPUValue
Top Process memory usage snippet
Next, we’ll want to see what applications/tools might be involved, including Active Client, IIS, monitoring, and EndPoint Management tools (keep things honest!).
Added the Security Processes into the mix
Then we build an output of the data so we can take the datasource (DS) or WriteAction (WA) into a scripted monitor/rule, or recovery tasks linked to various monitors. Even built a forked version in case of SAW/Red Forest, separating Tier0 monitoring from Tier1 (snippet below is NOT that pack)
snippet of manual tasks and recoveries that link to multiple monitors
Time to ‘create a Closed Alerts view’ for all users (versus an individual user workspace). Sometimes, we just need a different view. My thanks to Joe Kelly for his help documenting this!
Create Closed Alerts view
How to ‘create a Closed SCOM Alert’ view. Follow these steps:
Open the Operations Console and navigate to the Monitoring workspace.
Click on the “New” button in the toolbar and select “Alert View” from the dropdown menu.
In the “Create Alert View” wizard, give your view a name and select “Closed Alerts” as the criteria.
Click “Next” and select the columns you want to display in your view.
Click “Next” again and choose any grouping or sorting options you want to apply.
Click “Finish” to create your view.
SCOM Closed alert view
Once you have created your closed alert view, access the new view from the Monitoring tab. Customize the view further by right-clicking on it and selecting “Properties”. From there, you can add or remove columns, change the grouping or sorting, and apply filters to further refine the view.
Learn article here to help personalize views like ‘Create Closed Alerts view’
Ever need to build out a capability and the SQL query is your blocker? Use a SQL query Plan ‘howTo’ to figure out what’s taking query so long. My thanks to Dennis Zwahlen (a Data and AI CSA – LinkedIn ) helping me figure out what was causing a SCOM DW SQL query to render data VERY slowly!
Don’t get me wrong, the sheer volume of events is definitely part of the problem. Event rules are using expressions to further restrict collected event data.
SCOM DW Events ingested for DC Security Events when SIEM is a limit, and NOT using ACS feature. Will discuss the SCOM DW Event ingestion and additional XML authoring options to turn down the pressure.
Time to use the ‘SQL query Plan howto’ blog for SQL execution plan, to help to figure out why the DW Query takes so long. Using the execution plan, similar to SQL profiler, will provide insight to possibly speed up query, allowing PowerBI app/report rendering of data.
From SSMS > View > Add Display Estimated Execution Plan
From SSMS > View > Add Display Estimated Execution Plan
SQL execution plan starting from the left documenting SQL query
SQL query plan starting from the left documenting SQL query
Sort is taking 4.5 minutes in this example of the SQL execution plan visual. You can see moving right from the Join lines documents how SQL behaves, and how each piece affects overall execution.
SQL query plan starting moving right from the left documenting SQL query
Hope this helps for another diagnostic SQL step in your tool box!
VulnID 178852 – Vulnerable to hackers – SQL OLE DB Driver update required
Got another vulnerability pop up on the last scan. ‘Vuln 178852 OLE DB driver’ has vulnerabilities and needs updated. My experience links this NOT to ODBC vuln 175441, thereby related to added capabilities and drivers installed with SSMS v19. NOTE: OLE has a pre-req of the new Visual C++ Redistributable x86 and x64 bits. Let’s mitigate Vuln 178852 OLE DB driver update!
Quick outline of steps with Vuln 178852 OLE DB driver
Download the bits (and copy to repository and servers for install)
Once downloaded, copy the OLE DB Driver and VC Redistributable EXE’s for x64 and x86 to the affected servers. Search for OLE first, to assess OLE and Redistributable versions currently installed.
Assess ‘Vuln 178852 OLE DB driver’ updates on affected servers
Log into the server(s)
From Control Panel > Programs > Programs and Features > Search for ‘ole’ to see Redistributable versions
Check Control Panel for OLE DB Version
Check Redistributable version
From Control Panel > Programs > Programs and Features > Search for ‘Red’ to see Redistributable versions
From Control Panel > Programs > Programs and Features > Search for ‘Red’ to see Redistributable versions
If you don’t upgrade Visual C++ Redistributable first, you’ll get this setup error
Executing OLE DB Driver update pre-requisite error for Visual C++ Redistrubutable update
First, we have to install the Visual C++ updates to the server before we can update the driver.
From PowerShell (as admin) on affected servers
Go to saved directory for EXE and MSI files
PowerShell as admin > go to directory > run the EXE
Click the Check box to EULA ‘I agree’
At the Visual C++ Redistributable EULA splash screen
Check agree checkbox, then click Install button lower right
Visual C++ Redistributable EULA splash screen to check agree checkbox, then click on Install
Update installing
VC_Redistributable installing screenshot
Click Restart button (when in approved change window)
Click Restart when in change window to reboot server for Visual C++ update to apply
Restart server
Update VC_Redist.x86.exe
Second part, if applicable x86 library is installed, is to update.
Install next pre-req, if server contained both x86 and x64 bits for the ‘Vuln 178852 OLE DB driver’
From PowerShell (as admin) on affected servers:
Go to saved directory for EXE and MSI files
.\VC_redist.x86.exe
Powershell as admin window initiating the Visual C++ Redistributable x86 exe
Click the Check box to EULA ‘I agree’
At the Visual C++ Redistributable EULA splash screen
Check agree checkbox, then click Install button lower right
Click on ‘I agree’ checkbox, and click Install button to begin the x86 Visual C++ Redistributable update
Update installing
Screenshot installing the x86 Visual C++ Redistributable update
Update complete
Screenshot showing successful install of the x86 Visual C++ Redistributable update
Update MSOLEDB drivers
Third, assess first if you need x64 AND x86 drivers (my example is only x64)
Start by checking the Control Panel > Programs > Programs and Features > search for ole (and hit enter)
Control Panel > Programs > Programs and Features > searching for ole, showing old v18
From PowerShell (as admin) on affected servers
Go to saved directory for EXE and MSI files
Open MSI to begin install
PowerShell as Admin running the ole MSI install
Click Next if you get the ‘User Account Control’ (UAC) prompt to initiate MSI install
OLE MSI Install – User Account Control (UAC) prompt to initiate MSI install
Click Next
OLE MSI install, click Next
Click ‘I agree’ radio button and Click Next
OLE MSI Install, EULA splash screen to check ‘I Agree’ radio button and click Next
Next, on the OLE MSI install, click next to accept default features (just the driver install)
OLE MSI install, click next to accept default features (just the driver install)
Click Install to begin driver install
OLE MSI install, click install
OLE driver install completed, click Finish
OLE driver install completed, click Finish
Verify Control Panel for OLE driver install and version
Lastly, assess server and application requirements to verify if the old OLE driver is okay to remove from system to clear vulnerability. The old OLE driver on my system was installed the day I installed SSMS v19.x
Back to your Control Panel > Programs > Programs and Features window
Change search to OLE in the top right > hit enter
Click Delete on old version
On the Warning popup window, click continue
Control Panel view showing two OLE drivers, reflecting the newly installed, and the old version
At the UAC prompt, click Yes
OLE MSI Install – User Account Control (UAC) prompt to initiate MSI install
Once complete, verify Control Panel window
Control Panel > Programs > Programs and Features > searching for ole, showing old v18
Other documentation
Security Updates for Microsoft SQL Server OLE DB Driver (June … | Tenable®
What does your mind link to with the FIPS acronym? FIPS makes me think of the movie Greyhound where Tom Hanks says LT Flipper, instead of Fippler, all that said being ZERO to do with resolving ‘STIGs for SCOM FIPS compliance on Windows’
The biggest hurdle to ‘STIGs for SCOM FIPS compliance on Windows’, is obtaining the files. The current bundled SCOM ISO’s since 2012 SP1 do NOT contain the gacutil, and cryptography DLL files, to resolve STIG V-220942 (win10), V-226335 (Server 2012/2012R2), V-73701 (Server 2016), V-93511 (Server 2019), V-254480 (Server 2022). As much as we want to resolve FIPS ‘STIGS for SCOM FIPS compliance for Windows Server’, gotta start with the finding relevant files. My thanks to Nathan Gau, Tyson Paul, and Aakash Basavaraj, for their involvement and clarification.
Install DLL for STIGs for SCOM FIPS compliance on Windows
Time to mitigate!
Let’s begin to fix the SCOM Web Console role servers (possibly SQL SSRS and PowerBI Report Server included) for resolving multiple ‘STIGs for SCOM FIPS compliance for Windows Server’. Blog post applies to multiple STIG(s) including STIGs V-220942, V-226335, V-73701, V-93511, V-254480
Download files
Whether from blog download link, or if you have the old ISO’s to obtain the DLL, and server ISO for gacutil , or myvisualstudio.com link
Download SCOM ISO from my.visualstudio.com/Downloads?q=operations
If you downloaded from my.visualstudio.com, extract from ISO.
Copy files to IIS role servers (SCOM web console, SSRS, or PowerBI report Servers) to setup files for FIPS compliance.
Download the DLL to the SCOM default folder –
Best practice is SCOM Default folder on non-system disk @
HA HA HA, that’s so funny. An error I didn’t expect importing the latest SQL packs ‘Updating SQLServer packs to v7.2.0.0’
Quick public service announcement – remove the SQL Server Core Custom Monitoring pack before ‘updating SQLServer packs to v7.2.0.0’! Read to save time and frustration, before importing the packs, as the previous 7.0.42.0 pack isn’t upgradable to v7.2.0.0.
Much like the character from Seinfeld, finding out that the ‘V-237434 SCOM Web Console SSL Settings’ is NOT STIG Compliant (STIG’d), is just as tramatic as being hungry, and told ‘No soup for you!” With all the many STIG findings, here’s a quick and dirty way to resolve the finding.
Kevin Holman SCOM QuickStart guides for SCOM 2019, SCOM 2022
V-237434 SCOM Web Console SSL Settings
STIG V-237434 requires trusted CA SSL certificates. Previous July blog posts are related to the effort to secure the SCOM web console. The redirect post forces HTTPS, complimenting this STIG finding. As the STIG states, remediation verification that IIS web site binding is HTTPS, and remove HTTP.
Remediate SCOM servers with Web Console role
Assumption = SmartCards are used for authentication, this part is applicable, otherwise skip.
RDP to server, connect to IISManager
Expand IIS Server > Expand Sites > Expand Default Web Site
IIS Manager Default Web Site menu
Click on SSL Settings
If the menu is greyed out, follow the SCOM WebConsole settings blog to setup the SSL certificate. Once complete, proceed below.
Click on SSL Settings > Check box to ‘Require SSL’
If menu is NOT greyed out, click radio button to ‘Accept’ client certificates
Click Apply
IIS Manager, Default Web Site, SSL Settings default when NOT running SSL certificate and bindings
Click on Default Website on left hand pane
In the Actions Pane (right hand side), click on Restart to restart the IIS website
Restart IIS website from IIS manager actions pane
IIS Website bindings
Next pieces is to verify the SSL HTTPS binding is setup correctly. In case you got disconnected, or rebooted the server
RDP to server, connect to IISManager
Expand IIS Server > Expand Sites > Expand Default Web Site
In the Actions pane on the top right, click on Bindings
IISManager, Default Web Site, Actions Pane, Bindings to setup HTTPS and remove HTTPS
Kevin Holman’s QuickStart blog(s) for SCOM 2019, SCOM2022 setup default HTTP binding (i.e. NO SSL cert configured)
Default website, Bindings selection showing HTTP if following SCOM quick start
If HTTP ONLY, click the Add button
Change dropdown for Type to https
Enter Host Name
Click Select to choose the SSL cert
Click OK
Adding HTTPS Binding with server name, SSL cert drop down and selected
Verify SSL certificate added
IIS HTTPS Bindings with SSL cert
If you have the binding above, change your STIG CKL finding and document as NOT a finding, for V-237434 SCOM Web Console SSL Settings!
Use this post when the SCOM WebConsole gets flagged for HTTP Redirect. The IIS configuration is pretty easy to set up. When your Security team contacts you to resolve VulnID 121040, the steps below should resolve the compliance finding. Use the Microsoft learn site for more details.
Add HTTP Redirect role from Server Manager
Time to Configure ‘SCOM WebConsole HTTP Redirect’
RDP to server, open Server Manager
Click on Manage on top right
Click Next on the ‘before you begin popup’
Server Manager splash screen
Click Next
Server Manager Role Installation Type popup wizard
Click Next
Server Manager Destination Manager screen
Expand the ‘Web Server’ drop down menu
Server Manager Roles
Expand Web Server drop down menu
Expand Common HTTP Features
Check box for HTTP Redirection
Server Manager Roles expanding Web Server for HTTP Redirect
Click Next
Server Manager HTTP Redirection check box selected
Click Next at the Features tab
Server Manager Features window
Click Install to install the feature
NOTE the checkbox to ‘Restart if required is NOT selected’
Most change processes don’t allow this on the fly (unplanned outage)
Server Manager Selections window
Wait while the feature(s) install
Click Close once complete
Server Manager feature install in progress
Setup Redirection in IIS Manager
Open IISManager
NOTE If IISManager was open before the feature was closed, exit and open IISManager again. IISManager refresh does NOT make HTTP Redirect reappear (even if restarting IIS service).
Click on your webServer > Double click on HTTP Redirect
IIS Manager with HTTP Redirect
IISManager HTTP Redirect Default splash screen
Check the ‘Redirect requests to this destination:’ check box
Enter the WebConsole URL for your installation.
NOTE SCOM default WebConsole URL is http://<webserverName>/OperationsManager
Check the two (2) boxes for Redirect behaviors
IISManager HTTP Redirect configuration screen
Click Apply
Recommend restart/reboot of server (off hours) to apply configuration before having Security team scan server.
Verify HTTP Redirect after reboot
After reboot, verify current settings (shown are default)
Click on ‘Default WebSite’ dropdown > Select HTTP Redirect
I attribute Kerberos AD delegation as the Navajo and Comanche helped allies in WW2, encrypted and encoded communication
Next on the list is to setup SCOM WebConsole settings for Kerberos AD Delegation. I attribute Kerberos AD delegation as the Navajo and Comanche helped allies in WW2, encrypted and encoded communication. Time to make the donuts! (to setup SCOM WebConsole settings for Kerberos AD Delegation)
If you’re improperly setup – you’ll flag on STIG configs V-243470, V-243478
Configure delegation on SCOM and/or PowerBI servers
Take the list of affected servers, to take action. Use the steps below to configure relevant SCOM or PowerBI servers.
Configure SCOM Web Console server
With domain administrator (DA or Tier0) rights, open the Active Directory Users and Computers MMC snap-in.
From ADUC > change ‘Find’ drop-down to Computers
In the Computer name text box, enter <SCOMWebConsoleServerName> and click search
Right click the server in the results box > Select Properties.
Select the Delegation tab.
Select Trust this computer for delegation to specified services only > Use any authentication protocol.
Under Services to which this account can present delegated credentials, select Add.
In the new dialog box, select Users or Computers.
Enter <SCOMWebConsoleServerName>, and then select OK.
Click the Add button to add services
Select the w3svc and www processes
Select OK.
ADUC SCOM Lab server choosing process
Verification of delegation settings
ADUC Delegation flags with SCOM MS processes selected.
Depending on replication times for the forest, wait and later reboot <SCOMWebConsoleServerName> to have settings take effect.
PowerBI Report Server
With domain administrator (DA or Tier0) rights, open the (ADUC) Active Directory Users and Computers MMC snap-in. NOTE: RSAT tools recommended to be installed on SCOM Management Server(s)
In the Search text box, enter PowerBI service account <Example can be SCOMDataAccessReader Account> and click search
Right-click the PowerBI service account <Example can be SCOMDataAccessReader Account>, select Properties.
Select the Delegation tab.
Select Trust this computer for delegation to specified services only > Use any authentication protocol.
Under Services to which this account can present delegated credentials, select Add.
In the new dialog box, select Users or Computers.
Enter the service account for the data source, and then select OK.
Select the SPN that you created for <PowerBI Report Server Name>
Select both as FQDN and the NetBIOS names are in the SPN
Select OK.
Back to ADUC (AD Users and Computers), change Find drop-down to Computers
Enter <PowerBI Report Server Name>, and click search
Right click the server in the results box > Select Properties.
Select the Delegation tab.
Select Trust this computer for delegation to specified services only > Use any authentication protocol.
Under Services to which this account can present delegated credentials, select Add.
In the new dialog box, select Users or Computers.
Enter <Example can be SCOMDataAccessReader Account>, and then select OK.
Click the Add button to add services
Select the HTTP process
ADUC Delegation Add Services > HTTP, WWW
Select OK.
ADUC Delegation Settings for http for PowerBI Report Server (PBIRS)
Manage Cookie Consent
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
