Data Integration with SQL2022

Integration - time to integrate data sources to data lake
Integration – time to integrate data sources to data lake

Ready for a single pane of glass?  Ready to have your insights in a common location?  Let’s discuss Data Integration with SQL2022.

 

Let’s start with some background on SQL2022 and similarly SQL2025, start with the learn site link.  SQL2022 by design is Azure enabled with multiple capabilities like ‘Bi-directional HA/DR to Azure SQL’ and ‘Azure Synapse Link’.  Basically, Synapse link is the key.

SQL2022 by design is Azure Enabled
SQL2022 by design is Azure Enabled

Utilize the PowerBI Cloud Service with today’s hybrid environments.  SQL2022 allows integration with other Azure capabilities like Azure Data factory/data lake, and Azure Synapse.  Another reason to upgrade SQL 2022, is design simplification.  However, PowerBI data gateway adds a potential break point (single point of failure).  While PowerBI data gateway centralizes all premise data to a central location.  In the same way, consolidating data sent to the cloud.  When PowerBI data gateway fails, insights and visualizations have stale data (i.e. data NOT transferred for a near real-time display).

 

Why SQL2022 then?

Connect insights and visualization to justify ‘Data Integration with SQL2022’ scenarios.

SQL2022 built in capability to Azure Synapse Analytics
SQL2022 built in capability to Azure Synapse Analytics

Use SQL2022 to configure SQL agent jobs which pull SQL scripts from your cloud environment.  DevOps and common Azure Storage repository are great advantages for speed of execution.

SQL STIGMonitor

End the STIG(ma)

Seriously, dream on!  End the STIGma is a good thing, but STIGs can be a burden.  Hit the easy button, if you’re not already using it.  Contact your SQL Data and AI Cloud Solutions Architect for the latest SQL STIG Monitor 2024 Q4 build!


 

Latest SQL STIG monitor 31 Oct 2024 release includes

DISA UPDATES – see link
MS SQL Server 2016 Instance STIG, V3R2:

(NOTE: DISA has been contacted to remove related CCI STIGID for AzureSQLDB that was overlooked: ASQL-00-010700)

POWERSHELL MODULE
Updated version to 1.23
Added STIGID parameter to Invoke-StigMonitor allowing granular control over STIGID scanning.

DATABASE CHANGES
Updated Checklist Templates for Q4 Revisions.
Updated Instance & Database STIG for Q4 benchmark date.
Script updates include:

CNTNMIXDB: Not A Finding if using Windows Auth
FORCENRYPT: NA if using Windows Auth
PWDCMPLX: Updated Finding to remove OS STIG reference
AZDBPERMISS: Revised script with new version.
DBPERMISS: Revised script with new version.
ENFCACCSS: Revised script with new version.
PSERRPERM: Revised script with new version.
UNQSVCACC: Removed code stripping out port number.
AZAUDITSTATE: Properly returns No Finding when audit setup is correct.
Fixed bug in vDocumentation view causing POAMs to not display custom comment in exported documentation.
Added usp_RemoveInstance stored procedure to easily clean up a specific Instance from StigMonitor that no longer exists.

DOCUMENTS
Updated checklist templates, Approvals scripts, and Documentation Templates for Q4 Revisions.
Removed Set-CEIPRegKeys.ps1, Set-FIPSCompliance.ps1, and Set-SqlRegKey.ps1 in favor of Module commands.
Updated InfoPage with new StigMonitor logo and text references.
Documentation updated with new examples of Invoke-StigMonitor STIGID parameter.
Updated documentation to add Azure DB Permission for MS_SecurityDefinitionReader.
Added DatabaseName to CSV Export of Export-StigDocumentation.

REPORTS
Updated Report banner to display new StigMonitor logo and latest report versions.
Removed Adhoc scanning to Policy Management Report in favor of Invoke-StigMonitor parameter.
Removed references to Sunset 2012 and 2014 STIGs.
Added AzureSQLMI for future use.
Combined NF and Approved in Total Findings summary
Reduced Recent Scans to latest 6.

Also please send us your feedback if you get a chance to check this out.
If you want to be added/removed from this, click here (Subscribe /Unsubscribe) or send us an email.

 

New SQL Updates

Updated SQL patches released in July
Updated SQL patches released in July

It’s that time again, time to update SQL.  Just in case your configuration management solution automatically add SQL updates, you can be prepared.  Secondly, if you have to tell the configuration management team to approve updates, patches, this will help jumpstart that process.  Either way, knowing about the updates helps you make decisions for your organization’s change process.  I believe ‘knowledge is power’, so power up and take away whatever you need to keep up to date.

 

 

Subset of the SQL product group released ‘new SQL updates’ in July

SQL2016SP3GDR Security Update – 9 July https://techcommunity.microsoft.com/t5/sql-server-blog/security-update-for-sql-server-2016-sp3-gdr/ba-p/4187396

SQL2017RTMCU31 Security Update – 9 July https://techcommunity.microsoft.com/t5/sql-server-blog/security-update-for-sql-server-2017-rtm-cu31/ba-p/4187385

SQL2019 RTM CU27 Security Update – 9 July https://techcommunity.microsoft.com/t5/sql-server-blog/security-update-for-sql-server-2019-rtm-cu27/ba-p/4187401

SQL2022 RTM CU13 Security Update – 9 July https://techcommunity.microsoft.com/t5/sql-server-blog/security-update-for-sql-server-2022-rtm-cu13/ba-p/4187356

SQL2022 RTM CU14 23 July https://techcommunity.microsoft.com/t5/sql-server-blog/cumulative-update-14-for-sql-server-2022-rtm/ba-p/4199659

 

 

Example New SQL Updates for SQL2022

Cumulative Update #14 for SQL Server 2022 RTM

The 14th cumulative update release for SQL Server 2022 RTM is now available for download at the Microsoft Downloads site. Please note that registration is no longer required to download Cumulative updates.
To learn more about the release or servicing model, please visit:

Starting with SQL Server 2017, we adopted a new modern servicing model. Please refer to our blog for more details on Modern Servicing Model for SQL Server

New SQL management pack

SQL Server Blog – New SQL Management pack released!

 

The blog posting the pack release fell through the cracks.  Released on 10 July, I’ve had some issues getting the updated MSI’s, but they’re live now.   I normally use the SQL Tech Community SQL releases site https://techcommunity.microsoft.com/t5/sql-server-blog/bg-p/SQLServer/label-name/SQLReleases

 

Don’t forget to look for SQL Security updates, (CU) Cumulative Updates, or (SP) Service Pack updates at the SQL releases link!  https://techcommunity.microsoft.com/t5/sql-server-blog/bg-p/SQLServer/label-name/SQLReleases

 

 

New SQL pack released

Microsoft System Center Management Pack for SQL Server enables the discovery and monitoring of SQL Server 2012, 2014, 2016, 2017, 2019, 2022, and upcoming versions.

Download link https://www.microsoft.com/en-us/download/details.aspx/?id=56203

Version:
7.6.5

File Name:
SQLServerMP.Windows.msi

SQLServerMP.CustomMonitoring.msi

SQLServerMP.Linux.msi

SQLServerMPWorkflowList.pdf <missing as of today>

Date Published:
7/10/2024

Functionality https://learn.microsoft.com/en-us/system-center/scom/sql-server-management-pack-changes-history?view=sc-om-2022

https://www.microsoft.com/en-us/download/details.aspx/?id=56203

June 2024 – 7.5.19.0 CTP

What’s New

  • Added new “Table Clustered Index Fragmentation” monitor that targets databases and checks for high fragmentation of clustered indexes
  • Added new “Property Bag” step in the custom monitor setup to extend the alert context with a property from the query result
  • Updated the “Product Version Compliance” monitor with the most recent version of public updates for the SQL Server
  • Reworked the “Long Running Queries” alert rule to improve security
  • Improved accessibility for the Summary Dashboard view and Monitoring Wizard template, including the following major changes:
    • implemented Keyboard Navigation using the A and D buttons on the tiles in the dashboard
    • added the ability for the screen reader to announce buttons and errors in the SQL Server wizard
    • redesigned dashboard list controls for greater accessibility

 

Pretty simple steps

Download and save to your SCOM server, or SCOM console connected machine

Navigate to the Administration tab

Expand Management Packs

Click on Installed Management packs

Click the Add drop-down, select the packs

Verify selections, and click Install button

Importing new SQL v7.6.5.0 packs into the SCOM Console

Click Close after import

v7.6.5.0 management packs are imported into the SCOM console

 

Enjoy!

 

SCOM MS TLS1.2 drivers

SCOM MS TLS1.2 drivers
SCOM MS TLS1.2 drivers

Courtesy of Brook Hudson, who provided clarification for encrypting SCOM data –

Question – Can we update the OLE DB Driver from 18.6.5 to 18.6.7 and the ODBC driver from 17.10.3 to 17.10.5.1 without breaking anything?

 

This configuration applies to SCOM2016 forward –

MS OLE DB Driver 18.6.7: https://go.microsoft.com/fwlink/?linkid=2242656

ODBC Driver 17.10.5.1: https://go.microsoft.com/fwlink/?linkid=2249004

 

 

I did NOT have success with this for SCOM2019 and SCOM2022 –

If the SQL endpoint is secured with encryption, then the following drivers can be used.

MS OLE DB Driver 19.3.2: https://aka.ms/downloadmsoledbsql

ODBC Driver 18.3.2.1: https://aka.ms/downloadmsodbcsql

If you want to use these newer drivers then SQL encryption is required, more information about enabling SQL Encryption: Configure SQL Server Database Engine for encryption – SQL Server | Microsoft Learnhttps://learn.microsoft.com/en-us/sql/database-engine/configure-windows/configure-sql-server-encryption?view=sql-server-ver15

 

The SQL team noted that the newer versions are defaulting Encrypt to be Yes/Mandatory. That is why the new drivers were having an issue. Setting up a certificate in the SQL endpoint would have allowed the connection to work:

Enable encrypted connections – SQL Server | Microsoft Docs

Certificate Management (SQL Server Configuration Manager) – SQL Server | Microsoft Docs

OLE DB Driver 19.0 for SQL Server Released – Microsoft Tech Community

ODBC Driver 18.0 for SQL Server Released – Microsoft Tech Community

 

IMPORTANT:

Update: Hotfixes released for ODBC and OLE DB drivers for SQL Server – Microsoft Community Hub

 

SQL SysMessages 18054 events

 

SCOM2016+ SQL SysMessages 18054 events
SCOM2016+ SQL SysMessages 18054 events

Whether you’re building a new SCOM2019, SCOM2022 environment or not, you might be missing these event details, and NOT even know!

 

It’s been a while for me, and I came across these, so posting for a fresh heads up!

Leverage Holman’s TXT files to keep your logging up to maximum potential!  Use the information below to resolve SCOM2016+ SQL SysMessages and 18054 events.

Holman GitHub download – contains SQL TXT files to run on OpsMgr & DW databases https://github.com/thekevinholman/SQLFix18054EventsSysmessages

 

The Github TXT files to download contain a clear scope of messages.

 

SQL messages excerpt

—————————————–
— MOMv3 messages are 77798xxxx —
—————————————–

———————————————–
— Discovery range: 77798-0000 to 77798-0049 —
———————————————–
— Managed type doesn’t exist.
EXECUTE sp_addmessage @msgnum = 777980000, @msgtext = N’The specified managed type doesn”t exist.’, @severity = 16, @lang = ‘us_english’, @with_log = false, @replace = ‘REPLACE’
GO

— Relationship type doesn’t exist.
EXECUTE sp_addmessage @msgnum = 777980001, @msgtext = N’The specified relationship type doesn”t exist.’, @severity = 16, @lang = ‘us_english’, @with_log = false, @replace = ‘REPLACE’
GO

— Source entity of the relationship doesn’t exist.
EXECUTE sp_addmessage @msgnum = 777980002, @msgtext = N’The specified relationship doesn”t have a valid source.’, @severity = 16, @lang = ‘us_english’, @with_log = false, @replace = ‘REPLACE’
GO

— Target entity of the relationship doesn’t exist.
EXECUTE sp_addmessage @msgnum = 777980003, @msgtext = N’The specified relationship doesn”t have a valid target.’, @severity = 16, @lang = ‘us_english’, @with_log = false, @replace = ‘REPLACE’
GO

— Discovery data from invalid managed entity is dropped.
EXECUTE sp_addmessage @msgnum = 777980004, @msgtext = N’Discovery data has been received from a rule targeted to a non-existent entity. The discovery data will be dropped.’, @severity = 16, @lang = ‘us_english’, @with_log = false, @replace = ‘REPLACE’
GO

— Invalid relationship rejected by cycle detection.
EXECUTE sp_addmessage @msgnum = 777980005, @msgtext = N’Relationship {%s} was rejected because it would cause a containment cycle; relationship source = ”%s” and target = ”%s”.’, @severity = 16, @lang = ‘us_english’, @with_log = false, @replace = ‘REPLACE’
GO

— Discovery data generated by invalid connector.
EXECUTE sp_addmessage @msgnum = 777980006, @msgtext = N’Discovery data generated by invalid connector:%s.’, @severity = 16, @lang = ‘us_english’, @with_log = false, @replace = ‘REPLACE’
GO

— Discovery data generated by invalid rule, task, discovery.
EXECUTE sp_addmessage @msgnum = 777980007, @msgtext = N’Discovery data generated by invalid discovery source. Id:%s.’, @severity = 16, @lang = ‘us_english’, @with_log = false, @replace = ‘REPLACE’
GO

Documentation links

Blog for 18054 events https://kevinholman.com/2017/08/27/scom-2016-event-18054-errors-in-the-sql-application-log/

SCOM 2016, 2019 and 2022: Event 18054 errors in the SQL application log

Alternatively, if AlwaysOn configuration, leverage Holman’s newer blog post – https://kevinholman.com/2022/10/02/scom-deployment-configuration-for-sql-always-on/

SCOM deployment configuration for SQL Always On

 

SQL STIG vulnerabilities V-213902, V-213935

Happy leap year, let’s talk Security and SQL STIG vulnerabilities V-213902, V-213935!
Happy Leap year
Happy Leap year

DISA DOD SQL STIG vulnerabilities V-213902, V-213935

SQL DBA team for RCC-C customer requesting documentation for exception, in light of vulnerabilities.
V-213902
V-213935
SCOM uses individual computer accounts in SQL for these findings
Holman documented this since 2012

SCOM SECURITY Documentation

SCOM2019 https://kevinholman.com/2020/07/23/scom-2019-security-account-matrix/

Both V-213902 AND V-213935 state same identification action.

Run this SQL Query on SCOM DB(s)
SELECT name
FROM sys.database_principals
WHERE type in (‘U’,’G’)
AND name LIKE ‘%$’
To remove users:
Run the following command for each user:
DROP USER [ IF EXISTS ] ;

V-213935 has a different identifier:

Launch PowerShell.
Execute the following code:
Note: <name> represents the username portion of the user. For example; if the user is “CONTOSO\user1$”, the username is “user1”.
([ADSISearcher]”(&(ObjectCategory=Computer)(Name=<name>))”).FindAll()
If no account information is returned, this is not a finding.
If account information is returned, this is a finding.

Tab delimited view –

Remove Computer Accounts DB SQL6-D0-000400 V-213902 CAT II Non-repudiation of actions taken is required in order to maintain data integrity. Examples of particular actions taken by individuals include creating information, sending a message, approving information (e.g., indicating concurrence or signing a contract), and receiving a message.
Remove Computer AccountsSQL6-D0-004200V-213935CAT IINon-repudiation of actions taken is required in order to maintain data integrity. Examples of particular actions taken by individuals include creating information, sending a message, approving information (e.g., indicating concurrence or signing a contract), and receiving a message.
Can provide one work-around to mitigate.
Awaiting CSS engagement for official mitigation from support and  SCOM PG.

MCM Addendum pack

The MCM addendum pack helps monitor MEM. See start menu folder structure for Endpoint Manager software.
The MCM addendum pack helps monitor MEM. See start menu folder structure for Endpoint Manager software.

Rebranding central – MEM, EM, MECM, SCCM, Configuration manager, depending on the synonym, we’re referring to the same product.  Tune the most common critical alerts per the health model to warning.

 

QUICK DOWNLOAD https://github.com/theKevinJustin/MCMAddendum/

Background

Read Holman’s blog for more details.

Did you know – MCM discoveries are based on registry keys added with various role installs on windows servers.  These registry keys are typically under this path:  HKLM\SOFTWARE\Microsoft\SMS\Operations Management\Components

 

What capabilities does the ‘MCM addendum pack’ provide?

Quite simply, the pack provides warning severity overrides for common alerts, disable event collection rules.

9 overrides for monitors and rules included in addendum.
9 overrides for monitors and rules included in addendum.

 

Includes warning severity changes for the following rules and monitors:

Monitors

BackupStatus.StatusMessage.Monitor

ReportingPoint.RoleAvailability.Monitor

SoftwareUpdatePoint.RoleAvailability.Monitor

SoftwareUpdatePointSync.AlertState.Monitor

Rules

ComponentServer.ComponentStoppedUnexpectedly.Event.Rule

SiteComponentManager – CanNotFindObjectInAD.Event.Rule, CouldNotAccessSiteSystem.Event.Rule

StateSystem.FailedToExecuteSummaryTask.Event.Rule

WsusConfigurationManager.FailedToConfigProxy.Event.Rule

 

 

Utilize the ‘MCM Addendum pack’

Download Kevin Holman’s MCM pack from GitHub.

Download the Addendum here, to get alerts where manual intervention required.

Save packs

 

Enjoy some acronym humor and ‘who moved my cheese fun!’

MECM PowerShell
MECM PowerShell

 

Import into SCOM & Enjoy!

 

If you need more capabilities, reach out on the blog or GitHub.

 

Documentation

Github repository here

SCCM management pack

Holman blog for MEM, EM, MCM, MECM, CM, ConfigMgr, Configuration Manager

MSSQL Addendum pack

 

Time to tune MSSQL alerts!
Time to tune MSSQL alerts!

The ‘MSSQL Addendum pack’ wouldn’t be possible without Brandon Pires contributions.  Brandon dealt with my many questions to better alert!  If you need more background, check the ‘why addendum pack’ post.

Quick Download(s)

2012+ https://github.com/theKevinJustin/MSSQLAddendum

 

Capabilities

The pack is based on the SQL engineering blog and program team making multiple updates per year for SQL monitoring.  The addendum creates two groups for dev/test and notification/subscription modeling.  Second, the overrides, man there are a bunch! aid consumption of real issues.   Lastly, most environments should be SQL 2016+, as the 2012R2 EOL/EOSL is quickly approaching in October!

MSSQL groups defined in the Addendum pack
MSSQL groups defined in the Addendum pack

MSSQL group discoveries require updates to be applicable to environment

 

Tailor addendum

First, the Addendum pack requires the MSSQL packs MUST be installed.  The addendum is based on the MSSQL 2016+ version agnostic is currently supported, as the 2012,2012R2 products are near end of support.

Find/Replace the variables as needed:

Example    ##TESTSERVER##|##DEVSERVER##

Save file

 

Overrides

Addendum pack contains discovery, monitor, and rule overrides to tune MSSQL to CSA (old PFE/CE/CSAe Microsoft Field engineer recommendations), to match the health model reducing critical ‘wake me up in the middle of the night’ alerts.

Partial snapshot of MSSQL overrides in the pack
Partial snapshot of MSSQL overrides in the pack

Import

Download pack, and save to your environment

Import into SCOM

Enjoy!

 

 

MSSQL Addendum references

MSSQL Engineering blog and old post here

SQL Releases TechCommunity here

Engineering team latest management pack, TechCommunity release v7.2.0.0

Import ‘gotcha’ importing new custom functionality blog