DISA DOD SQL STIG vulnerabilities V-213902, V-213935
SCOM SECURITY Documentation
SCOM2019 https://kevinholman.com/2020/07/23/scom-2019-security-account-matrix/
Operational monitoring tools including System Center, Azure Monitor
SCOM2019 https://kevinholman.com/2020/07/23/scom-2019-security-account-matrix/

Rebranding central – MEM, EM, MECM, SCCM, Configuration manager, depending on the synonym, we’re referring to the same product. Tune the most common critical alerts per the health model to warning.
Background
Read Holman’s blog for more details.
Did you know – MCM discoveries are based on registry keys added with various role installs on windows servers. These registry keys are typically under this path: HKLM\SOFTWARE\Microsoft\SMS\Operations Management\Components
What capabilities does the ‘MCM addendum pack’ provide?
Quite simply, the pack provides warning severity overrides for common alerts, disable event collection rules.

Includes warning severity changes for the following rules and monitors:
BackupStatus.StatusMessage.Monitor
ReportingPoint.RoleAvailability.Monitor
SoftwareUpdatePoint.RoleAvailability.Monitor
SoftwareUpdatePointSync.AlertState.Monitor
ComponentServer.ComponentStoppedUnexpectedly.Event.Rule
SiteComponentManager – CanNotFindObjectInAD.Event.Rule, CouldNotAccessSiteSystem.Event.Rule
StateSystem.FailedToExecuteSummaryTask.Event.Rule
WsusConfigurationManager.FailedToConfigProxy.Event.Rule
Download Kevin Holman’s MCM pack from GitHub.
Download the Addendum here, to get alerts where manual intervention required.
Save packs
Enjoy some acronym humor and ‘who moved my cheese fun!’

Import into SCOM & Enjoy!
If you need more capabilities, reach out on the blog or GitHub.
Github repository here
SCCM management pack
Holman blog for MEM, EM, MCM, MECM, CM, ConfigMgr, Configuration Manager

One more admin process and workflow is to ‘update SCAP tools’ on servers. Many times overlooked, this can save many headaches with the newest version installed on servers.
Check the website here, to search for Win in SCAP tools, then download & Install

Navigation steps:
Control Panel > Programs > Programs and Features
In the search bar (top right) enter scap (and hit enter)
SCAP Control panel output showing multiple versions installed. Need to install latest application, then remove the old versions (in this case, all three!)

Extract files from ZIP
Copy folder to repository (my path example below)
Save SCAP zip and files to folder repository and on server to install SCAP on.

Take the defaults (unless you want the checker icon on desktop). Run SCAP application from PowerShell (as admin) window.
Open PowerShell as admin window
Example:
cd “D:\MonAdmin\STIGS\scc-5.7.2_Windows”; gci; .\SCC_5.7.2_Windows_Setup.exe
Hit enter to begin install

On the SCAP EULA radio button application install screen, click ‘I accept’ radio button and click Next.

Select Destination location (preferably on non-system disk), and click Next
Change path to non-system disk (like d:)

From the ‘Select Components’ window, click Next

Click Next on the Setup Start Menu folder window

On the SCAP select additional tasks install window, click Next

Click Install on ‘Ready to install’ popup screen

With the new SCAP tool Install window, click Finish to complete.

Remove old versions
Click Continue and go through removal prompts

With the Uninstall screen, click Yes to uninstall.

Click OK on uninstall

Verify control panel only has latest version installed. Close out Programs and Features window


Click on Start > start typing SCAP > Click on SCAP Compliance Checker
From the SCAP checker UAC prompt, click Yes to continue

Click OK to end the install

Run local scan to prove functionality.
Select STIG(s) in the middle pane > Click Start Scan

Recheck Windows Explorer for OpenSSL; look at file properties for version details. Interesting, NONE of these files have versions (openssl, x509 searches show nothing file version wise)

Ask the Security Admin to re-scan!
DOD Cyber Exchange https://public.cyber.mil/stigs/scap/

The ‘MSSQL Addendum pack’ wouldn’t be possible without Brandon Pires contributions. Brandon dealt with my many questions to better alert! If you need more background, check the ‘why addendum pack’ post.
The pack is based on the SQL engineering blog and program team making multiple updates per year for SQL monitoring. The addendum creates two groups for dev/test and notification/subscription modeling. Second, the overrides, man there are a bunch! aid consumption of real issues. Lastly, most environments should be SQL 2016+, as the 2012R2 EOL/EOSL is quickly approaching in October!

MSSQL group discoveries require updates to be applicable to environment
First, the Addendum pack requires the MSSQL packs MUST be installed. The addendum is based on the MSSQL 2016+ version agnostic is currently supported, as the 2012,2012R2 products are near end of support.
Find/Replace the variables as needed:
Example ##TESTSERVER##|##DEVSERVER##
Save file
Addendum pack contains discovery, monitor, and rule overrides to tune MSSQL to CSA (old PFE/CE/CSAe Microsoft Field engineer recommendations), to match the health model reducing critical ‘wake me up in the middle of the night’ alerts.

Download pack, and save to your environment
Import into SCOM
Enjoy!
MSSQL Engineering blog and old post here
SQL Releases TechCommunity here
Engineering team latest management pack, TechCommunity release v7.2.0.0
Import ‘gotcha’ importing new custom functionality blog

‘Why addendum packs’? What value can they bring to my customer? Kevin Holman started the Addendum thought process quite a while back. Added functionality to a core application/program/product. The first example of this pack naming convention is his SQL RunAs Addendum to simplify SQL monitoring. Let’s break down a number of examples how the SCOM community has built packs to better monitoring, and how I believe the addendum packs bring IT Ninja lessons from Microsoft experts monitoring to your environment.
Better monitoring from the experts, including customer examples for other ‘blind spots’ in monitoring. Blind spots consist of ‘not monitored’ pieces of infrastructure, from simply an event, ping, service, tcp port check, process, web site, scripted workflow, with the purpose to identify a problem.
The goal of monitoring is to:
Identify, self-heal, automatically run recovery or diagnostic workflows alert when manual intervention is required. Doesn’t matter what tool you use, they all do some portion of these steps.
The addendum packs do these things, adding a few differentiators.
Auto closure daily scripts (close rules/monitors)
Auto reports of problems (M-F 0600-0700 local, reflecting last 24-72 hours of open/closed alerts)
Employ count logic (x in y time)
Self-heal monitors with no new events
Adjust alert severities to health model
where critical (red) = outage, warning (yellow) = issue, informational reports or FYI’s
Capable of updating alerts (status, owner, ticketID+)
Tasks to run workflows on-demand
Recovery tasks – (i.e. service restart automation or TopProcess, Logical disk cleanup, MECM Client cache clean )
Integrate additional monitoring (like DFS replication queue script/alerts)
Synthetic checks for DNS and web applications
Web Availability and Transactional monitoring, ADFS, CRL, PowerShell Invoke-WebRequest, and more
Security and Compliance checks
Imagine I forgot something capability wise.
Stay tuned, as this builds into an even better outcome, quality data into ‘a single pane of glass’ of multiple tools within PowerBI.

Ever need to build out a capability and the SQL query is your blocker? Use a SQL query Plan ‘howTo’ to figure out what’s taking query so long. My thanks to Dennis Zwahlen (a Data and AI CSA – LinkedIn ) helping me figure out what was causing a SCOM DW SQL query to render data VERY slowly!
Don’t get me wrong, the sheer volume of events is definitely part of the problem. Event rules are using expressions to further restrict collected event data.
SCOM DW Events ingested for DC Security Events when SIEM is a limit, and NOT using ACS feature. Will discuss the SCOM DW Event ingestion and additional XML authoring options to turn down the pressure.
Time to use the ‘SQL query Plan howto’ blog for SQL execution plan, to help to figure out why the DW Query takes so long. Using the execution plan, similar to SQL profiler, will provide insight to possibly speed up query, allowing PowerBI app/report rendering of data.
From SSMS > View > Add Display Estimated Execution Plan

SQL execution plan starting from the left documenting SQL query

SQL query plan starting from the left documenting SQL query
Sort is taking 4.5 minutes in this example of the SQL execution plan visual. You can see moving right from the Join lines documents how SQL behaves, and how each piece affects overall execution.

Hope this helps for another diagnostic SQL step in your tool box!

Got another vulnerability pop up on the last scan. ‘Vuln 178852 OLE DB driver’ has vulnerabilities and needs updated. My experience links this NOT to ODBC vuln 175441, thereby related to added capabilities and drivers installed with SSMS v19. NOTE: OLE has a pre-req of the new Visual C++ Redistributable x86 and x64 bits. Let’s mitigate Vuln 178852 OLE DB driver update!
Download the bits (and copy to repository and servers for install)
Update VC_Redist.x64.exe (and subsequent VC_Redist.x86.exe)
Update MSOLEDB drivers (x64 and possibly x86)
Re-scan to validate remediated!
Download Microsoft OLE DB Driver for SQL Server – OLE DB Driver for SQL Server | Microsoft Learn
Latest supported Visual C++ Redistributable downloads | Microsoft Learn
https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170
Latest supported Visual C++ Redistributable downloads | Microsoft Learn
Once downloaded, copy the OLE DB Driver and VC Redistributable EXE’s for x64 and x86 to the affected servers. Search for OLE first, to assess OLE and Redistributable versions currently installed.
Log into the server(s)
From Control Panel > Programs > Programs and Features > Search for ‘ole’ to see Redistributable versions
Check Control Panel for OLE DB Version
Check Redistributable version
From Control Panel > Programs > Programs and Features > Search for ‘Red’ to see Redistributable versions

If you don’t upgrade Visual C++ Redistributable first, you’ll get this setup error

First, we have to install the Visual C++ updates to the server before we can update the driver.
From PowerShell (as admin) on affected servers
Go to saved directory for EXE and MSI files
PowerShell as admin > go to directory > run the EXE
Click the Check box to EULA ‘I agree’
At the Visual C++ Redistributable EULA splash screen
Check agree checkbox, then click Install button lower right

Update installing

Click Restart button (when in approved change window)

Restart server
Second part, if applicable x86 library is installed, is to update.
Install next pre-req, if server contained both x86 and x64 bits for the ‘Vuln 178852 OLE DB driver’
From PowerShell (as admin) on affected servers:
Go to saved directory for EXE and MSI files
.\VC_redist.x86.exe

Click the Check box to EULA ‘I agree’
At the Visual C++ Redistributable EULA splash screen
Check agree checkbox, then click Install button lower right

Update installing

Update complete

Third, assess first if you need x64 AND x86 drivers (my example is only x64)
Start by checking the Control Panel > Programs > Programs and Features > search for ole (and hit enter)

From PowerShell (as admin) on affected servers
Go to saved directory for EXE and MSI files
Open MSI to begin install

Click Next if you get the ‘User Account Control’ (UAC) prompt to initiate MSI install

Click Next

Click ‘I agree’ radio button and Click Next

Next, on the OLE MSI install, click next to accept default features (just the driver install)

Click Install to begin driver install

OLE driver install completed, click Finish

Lastly, assess server and application requirements to verify if the old OLE driver is okay to remove from system to clear vulnerability. The old OLE driver on my system was installed the day I installed SSMS v19.x
Back to your Control Panel > Programs > Programs and Features window
Change search to OLE in the top right > hit enter
Click Delete on old version
On the Warning popup window, click continue

At the UAC prompt, click Yes

Once complete, verify Control Panel window

Security Updates for Microsoft SQL Server OLE DB Driver (June … | Tenable®

Updated 4 Apr 2023 with Tyson’s feedback!
First, some background – the Snapshot Synchronization alert just tells you there was a SQL issue running the workflow.
Second, the Snapshot Synchronization alert from a health model perspective, is NOT a critical issue (outage). Create override severity to 1 (warning) to prevent false wake-up calls. I’ll get this to my GitHub repo shortly!
Start with Tyson’s SCOM Maintenance pack, and run the tasks

Alternative long steps
Login to server with SSMS installed –
Open SSMS > Connect to the SCOM OpsMgr DB > Click on New Query
***NOTE verify database dropdown shows Operations Manager!
Paste SQL query into the query textbox
Select WorkItemName, b.WorkItemStateName, ServerName, StartedDateTimeUtc, CompletedDateTimeUtc, DurationSeconds, ERRORMESSAGE
from cs.WorkItem a , cs.WorkItemState b
where a.WorkItemStateId= b.WorkItemStateId
and WorkItemName = ‘SnapshotSynchronization’
Screenshot

Example SQL Output

WorkItemName WorkItemStateName ServerName StartedDateTimeUtc CompletedDateTimeUtc DurationSeconds ERRORMESSAGE
SnapshotSynchronization Succeeded SCOMV01 2023-01-24 00:26:23.427 2023-01-24 00:27:46.100 83 NULL
SnapshotSynchronization Failed SCOMV02 2023-01-25 00:27:52.363 2023-01-25 00:28:07.520 15
SnapshotSynchronization Succeeded SCOMV00 2023-01-25 21:43:36.540 2023-01-25 21:45:07.947 91 NULL
SnapshotSynchronization Running SCOMV00 2023-01-25 21:45:32.227 NULL NULL NULL
Solution:
The jobs may show Succeeded by the time you login to SQL = EOJ (end of job)
If Failed is latest date/timestamp, re-run the task “Request Snapshot Synchronization” which can be found when we select “Management Configuration Service Group” in the below mentioned view.
View:
From Monitoring Tab > Click on Operations Manager folder > Click on Management Group Health widget > Highlight unhealthy state from Management Group Functions.
Click on the ‘Request Snapshot Synchronization’ task to execute the Stored Procedure “SnapshotSynchronizationForce” on the OpsMgr DB.
NOTE: There are two tasks with same name but with different targets i.e. ‘Management Configuration Service Group’ and ‘Management Configuration Services’
The other task can be found on below view after selecting the Management Server you want the Task to be executed on
View:
From Monitoring Tab > Expand Operations Manager folder > Expand Management Configuration Service folder > Click on Services State view
Create Override for the alert
To change snapshot monitor to warning
From SCOM Console > Authoring Tab
Expand ‘Management Pack Objects’ > Click on Monitors
In the ‘Look for:’ bar type Snapshot synchronization state and hit enter
Monitor name = Snapshot synchronization state
Right click on monitor > Overrides > Override the monitor > For all objects of class
Click checkbox for Severity > change Critical to Warning
Click Edit – add comment – i.e. date/time changing to warning
Select your override pack > Click OK
Click OK to execute change

Resources
Tyson’s MonitoringGuys blog for SCOM Maintenance pack – download here

This post is courtesy of Andres Naranjo
Fix SQL2017+ .NET assembly errors after moving DB’s to new SQL servers.
Scenario: Moved the SCOM 2019 databases from a SQL 2014 database engine to a SQL 2019 database engine. SQL ApplicationThe following error occurred when opening the SCOM admin console:
Operations Manager Event Log, Event ID 26317
Date: 10/22/2021 11:17:27 AM
Application: Operations Manager
Application Version: 10.19.10505.0
Severity: Error
Message:
An error occurred in the Microsoft .NET Framework while trying to load assembly id 65537. The server may be running out of resources, or the assembly may not be trusted. Run the query again, or check documentation to see how to solve the assembly trust issues. For more information about this error:
System.IO.FileLoadException: Could not load file or assembly ‘microsoft.enterprisemanagement.sql.userdefineddatatype, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null’ or one of its dependencies. An error relating to security occurred. (Exception from HRESULT: 0x8013150A)
System.IO.FileLoadException:
at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean forIntrospection)
at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection)
at System.Reflection.Assembly.Load(String assemblyString)
Fix SQL2017+ .NET assembly
In addition, Operations Manager event ID’s 26317 events document the error (also check SQL Application log, see Holman’s blog). Here is an example from the Operations Manager event log:

Cause:
Starting with SQL 2017, SQL restricts trusted managed assemblies.
See more details in Microsoft TechNet article here
First, ensure that SQL CLR execution is enabled with the following SQL query:
sp_configure @configname=clr_enabled, @configvalue=1
GO
RECONFIGURE
GO
NOTE: It is important to make sure the SQL Server Service is re-started after the query above.
Second, execute ‘add trusted’ stored procedure queries to mark both as trusted:
EXEC sp_add_trusted_assembly 0xFAC2A8ECA2BE6AD46FBB6EDFB53321240F4D98D199A5A28B4EB3BAD412BEC849B99018D9207CEA045D186CF67B8D06507EA33BFBF9A7A132DC0BB1D756F4F491
EXEC sp_add_trusted_assembly 0xEC312664052DE020D0F9631110AFB4DCDF14F477293E1C5DE8C42D3265F543C92FCF8BC1648FC28E9A0731B3E491BCF1D4A8EB838ED9F0B24AE19057BDDBF6EC
Verify assemblies are successfully registered as trusted run:
Select * from sys.trusted_assemblies
The output should look like this:
At this point, re-start the SCOM services System Center Data Access, and System Center Management Configuration, on all management servers, and re-launch the SCOM admin console to make sure everything is working properly.
From PowerShell (as Admin)
restart-service healthservice; restart-service omsdk; restart-service cshost
Leverage Invoke-Command
# Invoke-Command syntax is PoSH remoting is enabled
#
# Run on multiple servers
# From PowerShell on SCOM Mgmt server, where you have same credential/access
# Example 1
“server1”,”server2″| % {invoke-command $_ -scriptblock {$env:ComputerName; restart-service healthservice; restart-service omsdk; restart-service cshost; get-service healthservice; get-service omsdk; get-service cshost }}
# Example 2
# Restart healthservice on MS/Agent
“server1”,”server2″| % {invoke-command $_ -scriptblock {$env:ComputerName; restart-service healthservice; restart-service omsdk; restart-service cshost; get-service healthservice; get-service omsdk; get-service cshost }}

Updated 25 Feb 2023
Ever wish alerts were like a wad of cash?
The more you solve, the more you make!
How about performance counter data?
The SQL management packs are awesome for visualizations, and provide a bunch of data.
Tim McFadden pointed out SQL Performance counters https://www.scom2k7.com/crazy-db-performance-collection-rules-in-the-sql-mps/
His blog brings up SQL MP Disk Latency performance counters.
His blog got me thinking about SQL DB and DB file design, where multiple DB files are on the same Drive, causes duplicate performance counters (SCOM workflows) on the agent, and will typically be one of the culprits for HealthService restarts.
SQL MP creates performance counters (per DB file, group, instance, engine)
Start in the SCOM console
Click on the Reporting Tab
Click on the ‘System Center Core Monitoring Reports’ folder
Double click on the Data Volume by Management Pack

Select the timeframe (from, to)
Click Run

Reporting Data
I have 2 2016 DB’s and 1 2014 (SCVMM) database server monitored, and it’s 50% of my data volume!

Another example – had the DW shutdown for days

Did you know there are 60+ perf counter rules in 2012 alone, and nearly 200 in 2016?
How about an OFF pack, a management pack that turns off all the performance counter rules?
The monitors still exist for health, just no pretty performance graph, should you look.
Github repo link
Check out the Gallery post for download
Zip file contains