SQL STIG vulnerabilities V-213902, V-213935

Happy leap year, let’s talk Security and SQL STIG vulnerabilities V-213902, V-213935!
Happy Leap year
Happy Leap year

DISA DOD SQL STIG vulnerabilities V-213902, V-213935

SQL DBA team for RCC-C customer requesting documentation for exception, in light of vulnerabilities.
V-213902
V-213935
SCOM uses individual computer accounts in SQL for these findings
Holman documented this since 2012

SCOM SECURITY Documentation

SCOM2019 https://kevinholman.com/2020/07/23/scom-2019-security-account-matrix/

Both V-213902 AND V-213935 state same identification action.

Run this SQL Query on SCOM DB(s)
SELECT name
FROM sys.database_principals
WHERE type in (‘U’,’G’)
AND name LIKE ‘%$’
To remove users:
Run the following command for each user:
DROP USER [ IF EXISTS ] ;

V-213935 has a different identifier:

Launch PowerShell.
Execute the following code:
Note: <name> represents the username portion of the user. For example; if the user is “CONTOSO\user1$”, the username is “user1”.
([ADSISearcher]”(&(ObjectCategory=Computer)(Name=<name>))”).FindAll()
If no account information is returned, this is not a finding.
If account information is returned, this is a finding.

Tab delimited view –

Remove Computer Accounts DB SQL6-D0-000400 V-213902 CAT II Non-repudiation of actions taken is required in order to maintain data integrity. Examples of particular actions taken by individuals include creating information, sending a message, approving information (e.g., indicating concurrence or signing a contract), and receiving a message.
Remove Computer AccountsSQL6-D0-004200V-213935CAT IINon-repudiation of actions taken is required in order to maintain data integrity. Examples of particular actions taken by individuals include creating information, sending a message, approving information (e.g., indicating concurrence or signing a contract), and receiving a message.
Can provide one work-around to mitigate.
Awaiting CSS engagement for official mitigation from support and  SCOM PG.

MCM Addendum pack

The MCM addendum pack helps monitor MEM. See start menu folder structure for Endpoint Manager software.
The MCM addendum pack helps monitor MEM. See start menu folder structure for Endpoint Manager software.

Rebranding central – MEM, EM, MECM, SCCM, Configuration manager, depending on the synonym, we’re referring to the same product.  Tune the most common critical alerts per the health model to warning.

 

QUICK DOWNLOAD https://github.com/theKevinJustin/MCMAddendum/

Background

Read Holman’s blog for more details.

Did you know – MCM discoveries are based on registry keys added with various role installs on windows servers.  These registry keys are typically under this path:  HKLM\SOFTWARE\Microsoft\SMS\Operations Management\Components

 

What capabilities does the ‘MCM addendum pack’ provide?

Quite simply, the pack provides warning severity overrides for common alerts, disable event collection rules.

9 overrides for monitors and rules included in addendum.
9 overrides for monitors and rules included in addendum.

 

Includes warning severity changes for the following rules and monitors:

Monitors

BackupStatus.StatusMessage.Monitor

ReportingPoint.RoleAvailability.Monitor

SoftwareUpdatePoint.RoleAvailability.Monitor

SoftwareUpdatePointSync.AlertState.Monitor

Rules

ComponentServer.ComponentStoppedUnexpectedly.Event.Rule

SiteComponentManager – CanNotFindObjectInAD.Event.Rule, CouldNotAccessSiteSystem.Event.Rule

StateSystem.FailedToExecuteSummaryTask.Event.Rule

WsusConfigurationManager.FailedToConfigProxy.Event.Rule

 

 

Utilize the ‘MCM Addendum pack’

Download Kevin Holman’s MCM pack from GitHub.

Download the Addendum here, to get alerts where manual intervention required.

Save packs

 

Enjoy some acronym humor and ‘who moved my cheese fun!’

MECM PowerShell
MECM PowerShell

 

Import into SCOM & Enjoy!

 

If you need more capabilities, reach out on the blog or GitHub.

 

Documentation

Github repository here

SCCM management pack

Holman blog for MEM, EM, MCM, MECM, CM, ConfigMgr, Configuration Manager

Update SCAP tools

DISA Security Content Automation Protocol
DISA Security Content Automation Protocol

 

One more admin process and workflow is to ‘update SCAP tools’ on servers.  Many times overlooked, this can save many headaches with the newest version installed on servers.

 

 

Check DOD Cyber Exchange

Check the website  here, to search for Win in SCAP tools, then download & Install

SCAP tool download from DOD Cyber Exchange public website.
SCAP tool download from DOD Cyber Exchange public website.

 

Navigation steps:

Control Panel > Programs > Programs and Features

In the search bar (top right) enter scap (and hit enter)

 

SCAP Control panel output showing multiple versions installed.  Need to install latest application, then remove the old versions (in this case, all three!)

SCAP Control panel output showing multiple versions installed.
SCAP Control panel output showing multiple versions installed.

 

 

Install SCAP application

Extract files from ZIP

Copy folder to repository (my path example below)

Save SCAP zip and files to folder repository and on server to install SCAP on.

Save SCAP zip and files to folder repository and on server to install SCAP on.
Save SCAP zip and files to folder repository and on server to install SCAP on.

 

 

Run SCAP application

Take the defaults (unless you want the checker icon on desktop).  Run SCAP application from PowerShell (as admin) window.

Open PowerShell as admin window

 Example:

cd “D:\MonAdmin\STIGS\scc-5.7.2_Windows”; gci; .\SCC_5.7.2_Windows_Setup.exe

Hit enter to begin install

Run SCAP install from PowerShell (as admin) window.
Run SCAP install from PowerShell (as admin) window.

 

On the SCAP EULA radio button application install screen, click ‘I accept’ radio button and click Next.

SCAP EULA radio button application install screen.
SCAP EULA radio button application install screen.

 

Select Destination location (preferably on non-system disk), and click Next

Change path to non-system disk (like d:)

SCAP Destination Location Application install window.
SCAP Destination Location Application install window.

 

From the ‘Select Components’ window, click Next

SCAP Select Components application install window.
SCAP Select Components application install window.

 

Click Next on the Setup Start Menu folder window

SCAP Start Menu folder install window
SCAP Start Menu folder install window

 

On the SCAP select additional tasks install window, click Next 

SCAP select additional tasks install window
SCAP select additional tasks install window

 

Click Install on ‘Ready to install’ popup screen

SCAP Ready to Install popup screen.
SCAP Ready to Install popup screen.

 

 

With the new SCAP tool Install window, click Finish to complete.

SCAP tool install finished splash screen.
SCAP tool install finished splash screen.

 

 

Refresh Control Panel SCAP search

Remove old versions

Click Continue and go through removal prompts

SCAP control panel remove old version with prompt to continue.
SCAP control panel remove old version with prompt to continue.

 

With the Uninstall screen, click Yes to uninstall.

SCAP uninstall yes/no screen
SCAP uninstall yes/no screen

 

Click OK on uninstall

Old SCAP unistall completed.
Old SCAP unistall completed.

 

 

Check Control Panel for SCAP installs

Verify control panel only has latest version installed.  Close out Programs and Features window

Windows Control Panel, Programs and Features, SCAP search for new version install
Windows Control Panel, Programs and Features, SCAP search for new version install

 

 

Review SCC (SCAP Compliance Checker) Release Notes

SCAP release Notes details
SCAP release Notes details

 

Verify SCAP application functionality

Click on Start > start typing SCAP > Click on SCAP Compliance Checker

SCAP Compliance Checker

 

From the SCAP checker UAC prompt, click Yes to continue

SCAP checker UAC prompt, click Yes to continue
SCAP checker UAC prompt, click Yes to continue

 

Click OK to end the install

SCAN new features popup after install
SCAN new features popup after install

 

 

Run Local Scan

Run local scan to prove functionality.

Select STIG(s) in the middle pane > Click Start Scan

Run SCAP scan against server, choose your STIGs and Start Scan
Run SCAP scan against server, choose your STIGs and Start Scan

 

Verify SCAP tool modified files after installation

Recheck Windows Explorer for OpenSSL; look at file properties for version details.  Interesting, NONE of these files have versions (openssl, x509 searches show nothing file version wise)

Verify SCAP tool modified files after installation
Verify SCAP tool modified files after installation

 

Ask the Security Admin to re-scan!

 

 

Documentation/Links

DOD Cyber Exchange https://public.cyber.mil/stigs/scap/

MSSQL Addendum pack

 

Time to tune MSSQL alerts!
Time to tune MSSQL alerts!

The ‘MSSQL Addendum pack’ wouldn’t be possible without Brandon Pires contributions.  Brandon dealt with my many questions to better alert!  If you need more background, check the ‘why addendum pack’ post.

Quick Download(s)

2012+ https://github.com/theKevinJustin/MSSQLAddendum

 

Capabilities

The pack is based on the SQL engineering blog and program team making multiple updates per year for SQL monitoring.  The addendum creates two groups for dev/test and notification/subscription modeling.  Second, the overrides, man there are a bunch! aid consumption of real issues.   Lastly, most environments should be SQL 2016+, as the 2012R2 EOL/EOSL is quickly approaching in October!

MSSQL groups defined in the Addendum pack
MSSQL groups defined in the Addendum pack

MSSQL group discoveries require updates to be applicable to environment

 

Tailor addendum

First, the Addendum pack requires the MSSQL packs MUST be installed.  The addendum is based on the MSSQL 2016+ version agnostic is currently supported, as the 2012,2012R2 products are near end of support.

Find/Replace the variables as needed:

Example    ##TESTSERVER##|##DEVSERVER##

Save file

 

Overrides

Addendum pack contains discovery, monitor, and rule overrides to tune MSSQL to CSA (old PFE/CE/CSAe Microsoft Field engineer recommendations), to match the health model reducing critical ‘wake me up in the middle of the night’ alerts.

Partial snapshot of MSSQL overrides in the pack
Partial snapshot of MSSQL overrides in the pack

Import

Download pack, and save to your environment

Import into SCOM

Enjoy!

 

 

MSSQL Addendum references

MSSQL Engineering blog and old post here

SQL Releases TechCommunity here

Engineering team latest management pack, TechCommunity release v7.2.0.0

Import ‘gotcha’ importing new custom functionality blog

Why Addendum packs

IT Ninja required for improving monitoring hence 'Why addendum packs'
IT Ninja required for improving monitoring hence ‘Why addendum packs’

 

‘Why addendum packs’?  What value can they bring to my customer?  Kevin Holman started the Addendum thought process quite a while back.  Added functionality to a core application/program/product.  The first example of this pack naming convention is his SQL RunAs Addendum to simplify SQL monitoring.   Let’s break down a number of examples how the SCOM community has built packs to better monitoring, and how I believe the addendum packs bring IT Ninja lessons from Microsoft experts monitoring to your environment.

 

Why Addendum packs

Better monitoring from the experts, including customer examples for other ‘blind spots’ in monitoring.  Blind spots consist of ‘not monitored’ pieces of infrastructure, from simply an event, ping, service, tcp port check, process, web site, scripted workflow, with the purpose to identify a problem.

The goal of monitoring is to:

Identify, self-heal, automatically run recovery or diagnostic workflows alert when manual intervention is required.  Doesn’t matter what tool you use, they all do some portion of these steps.

 

The addendum packs do these things, adding a few differentiators.

Auto closure daily scripts (close rules/monitors)

Auto reports of problems (M-F 0600-0700 local, reflecting last 24-72 hours of open/closed alerts)

Employ count logic (x in y time)

Self-heal monitors with no new events

Adjust alert severities to health model

where critical (red) = outage, warning (yellow) = issue, informational reports or FYI’s

Capable of updating alerts (status, owner, ticketID+)

Tasks to run workflows on-demand

Recovery tasks – (i.e. service restart automation or TopProcess, Logical disk cleanup, MECM Client cache clean )

Integrate additional monitoring (like DFS replication queue script/alerts)

Synthetic checks for DNS and web applications

Web Availability and Transactional monitoring, ADFS, CRL, PowerShell Invoke-WebRequest, and more

Security and Compliance checks

 

Imagine I forgot something capability wise.

Stay tuned, as this builds into an even better outcome, quality data into ‘a single pane of glass’ of multiple tools within PowerBI.

SQL query Plan howto

SQL Query Plan - can't you do anything right?
SQL Query Plan – can’t you do anything right?

Ever need to build out a capability and the SQL query is your blocker?  Use a SQL query Plan ‘howTo’ to figure out what’s taking query so long.  My thanks to Dennis Zwahlen (a Data and AI CSA – LinkedIn ) helping me figure out what was causing a SCOM DW SQL query to render data VERY slowly!

 

Don’t get me wrong, the sheer volume of events is definitely part of the problem.   Event rules are using expressions to further restrict collected event data.

SCOM DW Events ingested for DC Security Events when SIEM is a limit, and NOT using ACS feature

SCOM DW Events ingested for DC Security Events when SIEM is a limit, and NOT using ACS feature.  Will discuss the SCOM DW Event ingestion and additional XML authoring options to turn down the pressure.

 

Time to use the ‘SQL query Plan howto’ blog for SQL execution plan, to help to figure out why the DW Query takes so long.  Using the execution plan, similar to SQL profiler, will provide insight to possibly speed up query, allowing PowerBI app/report rendering of data.

From SSMS > View > Add Display Estimated Execution Plan

From SSMS > View > Add Display Estimated Execution Plan
From SSMS > View > Add Display Estimated Execution Plan

 

SQL execution plan starting from the left documenting SQL query
SQL query plan starting from the left documenting SQL query

SQL query plan starting from the left documenting SQL query

Sort is taking 4.5 minutes in this example of the SQL execution plan visual.  You can see moving right from the Join lines documents how SQL behaves, and how each piece affects overall execution.

SQL query plan starting moving right from the left documenting SQL query
SQL query plan starting moving right from the left documenting SQL query

Hope this helps for another diagnostic SQL step in your tool box!

Vuln 178852 OLE DB driver

VulnID 178852 - Vulnerable to hackers - SQL OLE DB Driver update required
VulnID 178852 – Vulnerable to hackers – SQL OLE DB Driver update required

 

Got another vulnerability pop up on the last scan.  ‘Vuln 178852 OLE DB driver’ has vulnerabilities and needs updated.  My experience links this NOT to  ODBC vuln 175441, thereby related to added capabilities and drivers installed with SSMS v19NOTE: OLE has a pre-req of the new Visual C++ Redistributable x86 and x64 bits.  Let’s mitigate Vuln 178852 OLE DB driver update!

 

 

 

Quick outline of steps with Vuln 178852 OLE DB driver

Download the bits (and copy to repository and servers for install)

Update VC_Redist.x64.exe (and subsequent VC_Redist.x86.exe)

Update MSOLEDB drivers (x64 and possibly x86)

Re-scan to validate remediated!

 

 

Download the bits

Download Microsoft OLE DB Driver for SQL Server – OLE DB Driver for SQL Server | Microsoft Learn

https://learn.microsoft.com/en-us/sql/connect/oledb/download-oledb-driver-for-sql-server?view=sql-server-ver16

Latest supported Visual C++ Redistributable downloads | Microsoft Learn

https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170

Latest supported Visual C++ Redistributable downloads | Microsoft Learn

https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170#visual-studio-2015-2017-2019-and-2022

 Once downloaded, copy the OLE DB Driver and VC Redistributable EXE’s for x64 and x86 to the affected servers.  Search for OLE first, to assess OLE and Redistributable versions currently installed.

 

 

Assess ‘Vuln 178852 OLE DB driver’ updates on affected servers

Log into the server(s)

From Control Panel > Programs > Programs and Features > Search for ‘ole’ to see Redistributable versions

Check Control Panel for OLE DB Version

Check Control Panel for OLE DB Version

Check Redistributable version

From Control Panel > Programs > Programs and Features > Search for ‘Red’ to see Redistributable versions

From Control Panel > Programs > Programs and Features > Search for 'Red' to see Redistributable versions
From Control Panel > Programs > Programs and Features > Search for ‘Red’ to see Redistributable versions

 

 

If you don’t upgrade Visual C++ Redistributable first, you’ll get this setup error

Executing OLE DB Driver update pre-requisite error for Visual C++ Redistrubutable update
Executing OLE DB Driver update pre-requisite error for Visual C++ Redistrubutable update

 

 

Update VC_Redist.x64.exe (and subsequent VC_Redist.x86.exe

First, we have to install the Visual C++ updates to the server before we can update the driver.

From PowerShell (as admin) on affected servers

Go to saved directory for EXE and MSI files

PowerShell as admin > go to directory > run the EXE

PowerShell as admin > go to directory > run the EXE

  

Click the Check box to EULA ‘I agree’

At the Visual C++ Redistributable EULA splash screen

Check agree checkbox, then click Install button lower right

Visual C++ Redistributable EULA splash screen to check agree checkbox, then click on Install
Visual C++ Redistributable EULA splash screen to check agree checkbox, then click on Install

Update installing

VC_Redistributable installing screenshot
VC_Redistributable installing screenshot


Click Restart button (when in approved change window)

Click Restart when in change window to reboot server for Visual C++ update to apply
Click Restart when in change window to reboot server for Visual C++ update to apply

Restart server

 

 

Update VC_Redist.x86.exe

Second part, if applicable x86 library is installed, is to update.

Install next pre-req, if server contained both x86 and x64 bits for the ‘Vuln 178852 OLE DB driver’

 

From PowerShell (as admin) on affected servers:

Go to saved directory for EXE and MSI files

.\VC_redist.x86.exe

Powershell as admin window initiating the Visual C++ Redistributable x86 exe
Powershell as admin window initiating the Visual C++ Redistributable x86 exe

 

Click the Check box to EULA ‘I agree’

At the Visual C++ Redistributable EULA splash screen

Check agree checkbox, then click Install button lower right

Click on 'I agree' checkbox, and click Install button to begin the x86 Visual C++ Redistributable update
Click on ‘I agree’ checkbox, and click Install button to begin the x86 Visual C++ Redistributable update

 

Update installing

Screenshot installing the x86 Visual C++ Redistributable update
Screenshot installing the x86 Visual C++ Redistributable update

 

Update complete

Screenshot showing successful install of the x86 Visual C++ Redistributable update
Screenshot showing successful install of the x86 Visual C++ Redistributable update

 

 

 

Update MSOLEDB drivers

Third, assess first if you need x64 AND x86 drivers (my example is only x64)

Start by checking the Control Panel > Programs > Programs and Features > search for ole (and hit enter)

Control Panel > Programs > Programs and Features > searching for ole, showing old v18
Control Panel > Programs > Programs and Features > searching for ole, showing old v18

 

From PowerShell (as admin) on affected servers

Go to saved directory for EXE and MSI files

Open MSI to begin install

PowerShell as Admin running the ole MSI install
PowerShell as Admin running the ole MSI install

Click Next if you get the ‘User Account Control’ (UAC) prompt to initiate MSI install

OLE MSI Install - User Account Control (UAC) prompt to initiate MSI install
OLE MSI Install – User Account Control (UAC) prompt to initiate MSI install

Click Next

OLE MSI install, click Next
OLE MSI install, click Next

 

Click ‘I agree’ radio button and Click Next

OLE MSI Install, EULA splash screen to check 'I Agree' radio button and click Next
OLE MSI Install, EULA splash screen to check ‘I Agree’ radio button and click Next

 

Next, on the OLE MSI install, click next to accept default features (just the driver install)

OLE MSI install, click next to accept default features (just the driver install)
OLE MSI install, click next to accept default features (just the driver install)

 

Click Install to begin driver install

OLE MSI install, click install
OLE MSI install, click install

 

OLE driver install completed, click Finish

OLE driver install completed, click Finish
OLE driver install completed, click Finish

 

 

Verify Control Panel for OLE driver install and version

Lastly, assess server and application requirements to verify if the old OLE driver is okay to remove from system to clear vulnerability.  The old OLE driver on my system was installed the day I installed SSMS v19.x

Back to your Control Panel > Programs > Programs and Features window

Change search to OLE in the top right > hit enter

Click Delete on old version

On the Warning popup window, click continue

Control Panel view showing two OLE drivers, reflecting the newly installed, and the old version
Control Panel view showing two OLE drivers, reflecting the newly installed, and the old version

 

At the UAC prompt, click Yes

OLE MSI Install - User Account Control (UAC) prompt to initiate MSI install
OLE MSI Install – User Account Control (UAC) prompt to initiate MSI install

 

 Once complete, verify Control Panel window

Control Panel > Programs > Programs and Features > searching for ole, showing old v18
Control Panel > Programs > Programs and Features > searching for ole, showing old v18

 

 

 

 

Other documentation

Security Updates for Microsoft SQL Server OLE DB Driver (June … | Tenable®

https://www.tenable.com/plugins/nessus/178852

SCOM Snapshot Synchronization alerts

Synchronized Swimmers

Updated 4 Apr 2023 with Tyson’s feedback!

 

First, some background – the Snapshot Synchronization alert just tells you there was a SQL issue running the workflow.

Second, the Snapshot Synchronization alert from a health model perspective, is NOT a critical issue (outage).  Create override severity to 1 (warning) to prevent false wake-up calls.  I’ll get this to my GitHub repo shortly!

 

 

 

Let’s troubleshoot the alert

Start with Tyson’s SCOM Maintenance pack, and run the tasks

Tyson’s SCOM Maintenance pack tasks

 

Alternative long steps

Login to server with SSMS installed –

Open SSMS > Connect to the SCOM OpsMgr DB > Click on New Query

***NOTE verify database dropdown shows Operations Manager!

Paste SQL query into the query textbox

Select WorkItemName, b.WorkItemStateName, ServerName, StartedDateTimeUtc, CompletedDateTimeUtc, DurationSeconds, ERRORMESSAGE
from cs.WorkItem a , cs.WorkItemState b
where a.WorkItemStateId= b.WorkItemStateId
and WorkItemName = ‘SnapshotSynchronization’

Screenshot

Snapshot Synchronization alert query

 

 

 

Example SQL Output

SQLQueryExecution

WorkItemName WorkItemStateName ServerName StartedDateTimeUtc CompletedDateTimeUtc DurationSeconds ERRORMESSAGE
SnapshotSynchronization Succeeded SCOMV01 2023-01-24 00:26:23.427 2023-01-24 00:27:46.100 83 NULL
SnapshotSynchronization Failed SCOMV02 2023-01-25 00:27:52.363 2023-01-25 00:28:07.520 15
SnapshotSynchronization Succeeded SCOMV00 2023-01-25 21:43:36.540 2023-01-25 21:45:07.947 91 NULL
SnapshotSynchronization Running SCOMV00 2023-01-25 21:45:32.227 NULL NULL NULL

Solution:
The jobs may show Succeeded by the time you login to SQL = EOJ (end of job)

If Failed is latest date/timestamp, re-run the task “Request Snapshot Synchronization” which can be found when we select  “Management Configuration Service Group” in the below mentioned view.

View:

From Monitoring Tab > Click on Operations Manager folder > Click on Management Group Health widget > Highlight unhealthy state from Management Group Functions.

Click on the ‘Request Snapshot Synchronization’ task to execute the Stored Procedure “SnapshotSynchronizationForce” on the OpsMgr DB.

 

NOTE: There are two tasks with same name but with different targets i.e. ‘Management Configuration Service Group’ and ‘Management Configuration Services’

 

The other task can be found on below view after selecting the Management Server you want the Task to be executed on

View:

From Monitoring Tab > Expand Operations Manager folder > Expand Management Configuration Service folder > Click on Services State view

 

 

Create Override for the alert

To change snapshot monitor to warning

From SCOM Console > Authoring Tab

Expand ‘Management Pack Objects’ > Click on Monitors

In the ‘Look for:’ bar type Snapshot synchronization state and hit enter

Monitor name = Snapshot synchronization state

Right click on monitor > Overrides > Override the monitor > For all objects of class

Click checkbox for Severity > change Critical to Warning

Click Edit – add comment – i.e. date/time changing to warning

Select your override pack > Click OK

Click OK to execute change

Override snapshot monitor to warning

 

 

Resources

Tyson’s MonitoringGuys blog for SCOM Maintenance pack – download here

Link to TechNet article

 

Weird SQL issue from SCOM DB move

Fix SQL2017+ .NET assembly - Weird .NET error
BillMurry-ThatsWeird

This post is courtesy of Andres Naranjo

 

Fix SQL2017+ .NET assembly error

Weird SQL issue from SCOM DB move to new SQL servers

Fix SQL2017+ .NET assembly errors after moving DB’s to new SQL servers.

 

Scenario: Moved the SCOM 2019 databases from a SQL 2014 database engine to a SQL 2019 database engine.  SQL ApplicationThe following error occurred when opening the SCOM admin console:

 

Operations Manager Event Log, Event ID 26317

Date: 10/22/2021 11:17:27 AM

Application: Operations Manager

Application Version: 10.19.10505.0

Severity: Error

Message:

 

An error occurred in the Microsoft .NET Framework while trying to load assembly id 65537. The server may be running out of resources, or the assembly may not be trusted. Run the query again, or check documentation to see how to solve the assembly trust issues. For more information about this error:

System.IO.FileLoadException: Could not load file or assembly ‘microsoft.enterprisemanagement.sql.userdefineddatatype, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null’ or one of its dependencies. An error relating to security occurred. (Exception from HRESULT: 0x8013150A)

System.IO.FileLoadException:

at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)

at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)

at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean forIntrospection)

at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection)

at System.Reflection.Assembly.Load(String assemblyString)

 

Fix SQL2017+ .NET assembly

In addition, Operations Manager event ID’s 26317 events document the error (also check SQL Application log, see Holman’s blog).  Here is an example from the Operations Manager event log:

Screenshot of Operations Manager event log, EventID 26319
Operations Manager event log, EventID 26319

 

 

Weird SQL issue from SCOM DB move to new SQL servers

Cause:

Starting with SQL 2017, SQL restricts trusted managed assemblies.

See more details in Microsoft TechNet article here

First, ensure that SQL CLR execution is enabled with the following SQL query:

 

sp_configure @configname=clr_enabled, @configvalue=1
GO
RECONFIGURE
GO

 

NOTE: It is important to make sure the SQL Server Service is re-started after the query above.

 

Second, execute ‘add trusted’ stored procedure queries to mark both as trusted:

 

EXEC sp_add_trusted_assembly 0xFAC2A8ECA2BE6AD46FBB6EDFB53321240F4D98D199A5A28B4EB3BAD412BEC849B99018D9207CEA045D186CF67B8D06507EA33BFBF9A7A132DC0BB1D756F4F491

EXEC sp_add_trusted_assembly 0xEC312664052DE020D0F9631110AFB4DCDF14F477293E1C5DE8C42D3265F543C92FCF8BC1648FC28E9A0731B3E491BCF1D4A8EB838ED9F0B24AE19057BDDBF6EC

 

 

Verify assemblies are successfully registered as trusted run:

Select * from sys.trusted_assemblies

 

The output should look like this:

SQLTrustedAssemblies output from SSMS

 

At this point, re-start the SCOM services System Center Data Access, and System Center Management Configuration, on all management servers, and re-launch the SCOM admin console to make sure everything is working properly.

 

Quicker ways to start SCOM services

From PowerShell (as Admin)

restart-service healthservice; restart-service omsdk; restart-service cshost

 

Leverage Invoke-Command

# Invoke-Command syntax is PoSH remoting is enabled

#

# Run on multiple servers

# From PowerShell on SCOM Mgmt server, where you have same credential/access

# Example 1

“server1”,”server2″| % {invoke-command $_ -scriptblock {$env:ComputerName; restart-service healthservice; restart-service omsdk; restart-service cshost; get-service healthservice; get-service omsdk; get-service cshost }}

 

# Example 2

# Restart healthservice on MS/Agent

“server1”,”server2″| % {invoke-command $_ -scriptblock {$env:ComputerName; restart-service healthservice; restart-service omsdk; restart-service cshost; get-service healthservice; get-service omsdk; get-service cshost }}

SQL MP bloat

Updated 25 Feb 2023

 

Ever wish alerts were like a wad of cash?

The more you solve, the more you make!

 

How about performance counter data?

 

 

The SQL management packs are awesome for visualizations, and provide a bunch of data.

 

Tim McFadden pointed out SQL Performance counters https://www.scom2k7.com/crazy-db-performance-collection-rules-in-the-sql-mps/

His blog brings up SQL MP Disk Latency performance counters.

 

His blog got me thinking about SQL DB and DB file design, where multiple DB files are on the same Drive, causes duplicate performance counters (SCOM workflows) on the agent, and will typically be one of the culprits for HealthService restarts.

 

SQL MP creates performance counters (per DB file, group, instance, engine)

 

Let’s start with how I figured out why all my money goes into storage.

 

Start in the SCOM console

Click on the Reporting Tab

Click on the ‘System Center Core Monitoring Reports’ folder

Double click on the Data Volume by Management Pack

View of SCOM report from console reporting tab

Select the timeframe (from, to)

Click Run

Data Volume MP selected

 

Reporting Data

I have 2 2016 DB’s and 1 2014 (SCVMM) database server monitored, and it’s 50% of my data volume!

 

 

 

Another example – had the DW shutdown for days

Data volume of SQL after

 

Did you know there are 60+ perf counter rules in 2012 alone, and nearly 200 in 2016?

 

How about an OFF pack, a management pack that turns off all the performance counter rules?

The monitors still exist for health, just no pretty performance graph, should you look.

 

 

Github repo link

Check out the Gallery post for download

TechNet gallery download

 

Zip file contains

  1.  OFF MP’s for 2008,2012,2014,2016
  2. XLS sheets to allow you to go to the SQL team and ask them what performance counters they use