Data Integration with SQL2022

Integration - time to integrate data sources to data lake
Integration – time to integrate data sources to data lake

Ready for a single pane of glass?  Ready to have your insights in a common location?  Let’s discuss Data Integration with SQL2022.

 

Let’s start with some background on SQL2022 and similarly SQL2025, start with the learn site link.  SQL2022 by design is Azure enabled with multiple capabilities like ‘Bi-directional HA/DR to Azure SQL’ and ‘Azure Synapse Link’.  Basically, Synapse link is the key.

SQL2022 by design is Azure Enabled
SQL2022 by design is Azure Enabled

Utilize the PowerBI Cloud Service with today’s hybrid environments.  SQL2022 allows integration with other Azure capabilities like Azure Data factory/data lake, and Azure Synapse.  Another reason to upgrade SQL 2022, is design simplification.  However, PowerBI data gateway adds a potential break point (single point of failure).  While PowerBI data gateway centralizes all premise data to a central location.  In the same way, consolidating data sent to the cloud.  When PowerBI data gateway fails, insights and visualizations have stale data (i.e. data NOT transferred for a near real-time display).

 

Why SQL2022 then?

Connect insights and visualization to justify ‘Data Integration with SQL2022’ scenarios.

SQL2022 built in capability to Azure Synapse Analytics
SQL2022 built in capability to Azure Synapse Analytics

Use SQL2022 to configure SQL agent jobs which pull SQL scripts from your cloud environment.  DevOps and common Azure Storage repository are great advantages for speed of execution.

SQL STIGMonitor

End the STIG(ma)

Seriously, dream on!  End the STIGma is a good thing, but STIGs can be a burden.  Hit the easy button, if you’re not already using it.  Contact your SQL Data and AI Cloud Solutions Architect for the latest SQL STIG Monitor 2024 Q4 build!


 

Latest SQL STIG monitor 31 Oct 2024 release includes

DISA UPDATES – see link
MS SQL Server 2016 Instance STIG, V3R2:

(NOTE: DISA has been contacted to remove related CCI STIGID for AzureSQLDB that was overlooked: ASQL-00-010700)

POWERSHELL MODULE
Updated version to 1.23
Added STIGID parameter to Invoke-StigMonitor allowing granular control over STIGID scanning.

DATABASE CHANGES
Updated Checklist Templates for Q4 Revisions.
Updated Instance & Database STIG for Q4 benchmark date.
Script updates include:

CNTNMIXDB: Not A Finding if using Windows Auth
FORCENRYPT: NA if using Windows Auth
PWDCMPLX: Updated Finding to remove OS STIG reference
AZDBPERMISS: Revised script with new version.
DBPERMISS: Revised script with new version.
ENFCACCSS: Revised script with new version.
PSERRPERM: Revised script with new version.
UNQSVCACC: Removed code stripping out port number.
AZAUDITSTATE: Properly returns No Finding when audit setup is correct.
Fixed bug in vDocumentation view causing POAMs to not display custom comment in exported documentation.
Added usp_RemoveInstance stored procedure to easily clean up a specific Instance from StigMonitor that no longer exists.

DOCUMENTS
Updated checklist templates, Approvals scripts, and Documentation Templates for Q4 Revisions.
Removed Set-CEIPRegKeys.ps1, Set-FIPSCompliance.ps1, and Set-SqlRegKey.ps1 in favor of Module commands.
Updated InfoPage with new StigMonitor logo and text references.
Documentation updated with new examples of Invoke-StigMonitor STIGID parameter.
Updated documentation to add Azure DB Permission for MS_SecurityDefinitionReader.
Added DatabaseName to CSV Export of Export-StigDocumentation.

REPORTS
Updated Report banner to display new StigMonitor logo and latest report versions.
Removed Adhoc scanning to Policy Management Report in favor of Invoke-StigMonitor parameter.
Removed references to Sunset 2012 and 2014 STIGs.
Added AzureSQLMI for future use.
Combined NF and Approved in Total Findings summary
Reduced Recent Scans to latest 6.

Also please send us your feedback if you get a chance to check this out.
If you want to be added/removed from this, click here (Subscribe /Unsubscribe) or send us an email.

 

SCOM SSRS permissions

Microsoft SQL Server SSRS icon
Microsoft SQL Server SSRS icon

Let’s discuss SCOM SSRS permissions.  The SCOM Reporting role install really comes down to three (3) things – permissions, latest SSRS EXE downloaded (for install 2019, 2022), and ReportExtensions configuration.  I’ve hit some permission issues that need more ‘how to’ details.

 

Set SCOM Admins group permissions

Whether the permissions are set up as part of a group policy (GPO) or not, if these steps are missing, expect problems.

Verify that your SCOM Admins domain group is a local administrator on the SCOM servers (SSRS server in this case)

Right click on Start > Computer Management

Expand System Tools

Expand Local Users and Groups

Click on Groups

Double click on Administrators

Verify SCOM Admins group, or specific service/MSA accounts are listed

Computer Management with Administrators group properties documenting relevant members which include the SCOM Admins group, and any other SQL related service accounts.
Computer Management with Administrators group properties documenting relevant members which include the SCOM Admins group, and any other SQL related service accounts.

Click OK

 

 

Set SQL Instance permissions for SCOM Admins group

Reference Holman’s QuickStart > Install SCOM Reporting Role…

  • Log on using your domain user account that is a member of the OMAdmins group, and has “sysadmin” role level rights over the SQL instance.

RDP to server with SSMS that connects to SQL server

Connect to Database Engine

Expand instance , then expand Security folder, thirdly expand Logins folder

Right click on the SCOM Admins group and select properties

In the pop-up, click on SQL Server Role

Verify that sysAdmin

View of SSMS Database Engine showing SCOM Admins group SQL Server Role has sysAdmin
View of SSMS Database Engine showing SCOM Admins group SQL Server Role has sysAdmin

Follow similar steps if using a domain connected SVC/MSA account when configuration differs from Holman’s QuickStart template.

 

Additional troubleshooting from the SCOM install can be found in the user’s directory – C:\Users\<accountHere>\AppData\Local\SCOM\LOGS

 

Find additional details in the SQL install logs

C:\Program Files\Microsoft SQL Server\MSRS13.MSSQLSERVER\Reporting Services\LogFiles

NOTE that the Instance and version 'MSRS13.MSSQLSERVER' can change

 

 

Additional documentation and relevant links

The go-to reference is Holman’s QuickStart deployment guides for SCOM2019 forward list the how-to starting point.

Holman Quick Start links:

https://kevinholman.com/2022/05/01/scom-2022-quickstart-deployment-guide/

https://kevinholman.com/2019/03/14/scom-2019-quickstart-deployment-guide/

 

SSRS learn.microsoft.com site article https://learn.microsoft.com/en-us/troubleshoot/system-center/scom/cannot-deploy-operations-manager-reports

SSRS Error occurred when invoking the authorization extension https://learn.microsoft.com/en-us/answers/questions/266488/installing-scom-2019-reporting-ssrs-2019-error-an

SCOM SSRS ReportExtensions

For a smooth install, everything comes down to SCOM SSRS prerequisites.  The SCOM Reporting role install really comes down to three (3) things – permissions, latest SSRS EXE downloaded (for install 2019, 2022), and ReportExtensions configuration.  The go-to reference is Holman’s QuickStart deployment guides for SCOM2019 forward list the how-to starting point.  This post focuses on ReportExtensions configuration, where more ‘how to’ details are needed.

Quick Start links:

SCOM 2022 – QuickStart Deployment Guide

SCOM 2019 – QuickStart Deployment Guide

SSRS learn.microsoft.com site article https://learn.microsoft.com/en-us/troubleshoot/system-center/scom/cannot-deploy-operations-manager-reports

 

Configure Report Extensions via SSMS (GUI)

RDP to server with enabled account

Open SSMS that has connectivity to SSRS install/server

Change ‘Server type’ drop-down to Reporting Service

Change SSMS Server Type from Database Engine to Reporting Service
Change SSMS Server Type from Database Engine to Reporting Service

Click Connect

Right click on Server > Properties

In the Server Properties window, select the Advanced Tab

Click on the AllowedResourceExtensionsForUpload, and add *.*

Click OK

Screenshot of SSMS Connected to Reporting Service, expanding SSRS Properties > Advanced Tab > showing AllowedResourceExtensionsForUpload
Screenshot of SSMS Connected to Reporting Service, expanding SSRS Properties > Advanced Tab > showing AllowedResourceExtensionsForUpload

Don’t forget to restart SSRS to make changes take effect!

Once restarted, verify SVC/MSA account permissions, and begin SCOM Reporting role!

 

Configure Report Extensions via PowerShell

Testing learn article PowerShell for SSRS Defaults (pre-requisite for SCOM Reporting role with SSRS2017+ versus SSMS).   > Reporting Services

SSRS Note for ServiceAddress (SSRS URL) is other than localhost

On respective server, open PowerShell as Admin

Paste the following:

$ServiceAddress = ‘http://localhost

$ExtensionAdd = @(

                ‘*’

                ‘CustomConfiguration’

                ‘Report’

                ‘AvailabilityMonitor’

                ‘TopNApplications’

                ‘Settings’

                ‘License’

                ‘ServiceLevelTrackingSummary’

                ‘CustomPerformance’

                ‘MostCommonEvents’

                ‘PerformanceTop’

                ‘Detail’

                ‘DatabaseSettings’

                ‘ServiceLevelObjectiveDetail’

                ‘PerformanceDetail’

                ‘ConfigurationChange’

                ‘TopNErrorGroupsGrowth’

                ‘AvailabilityTime’

                ‘rpdl’

                ‘mp’

                ‘TopNErrorGroups’

                ‘Downtime’

                ‘TopNApplicationsGrowth’

                ‘DisplayStrings’

                ‘Space’

                ‘Override’

                ‘Performance’

                ‘AlertDetail’

                ‘ManagementPackODR’

                ‘AlertsPerDay’

                ‘EventTemplate’

                ‘ManagementGroup’

                ‘Alert’

                ‘EventAnalysis’

                ‘MostCommonAlerts’

                ‘Availability’

                ‘AlertLoggingLatency’

                ‘PerformanceTopInstance’

                ‘rdl’

                ‘PerformanceBySystem’

                ‘InstallUpdateScript’

                ‘PerformanceByUtilization’

                ‘DropScript’

)

Write-Output ‘Setting Allowed Resource Extensions for Upload’

$error.clear()

try

{

                $Uri = [System.Uri]”$ServiceAddress/ReportServer/ReportService2010.asmx”

                $Proxy = New-WebServiceProxy -Uri $Uri -UseDefaultCredential

                $Type = $Proxy.GetType().Namespace + ‘.Property’

                $Property = New-Object -TypeName $Type

                $Property.Name = ‘AllowedResourceExtensionsForUpload’

$ValueAdd = $ExtensionAdd | ForEach-Object -Process {

                                “*.$psItem”

                }

$Current = $Proxy.GetSystemProperties($Property)

                if ($Current)

    {

                $ValueCurrent = $Current.Value -split ‘,’

                $ValueSet = $ValueCurrent + $ValueAdd | Sort-Object -Unique

                }

                else

    {

        $ValueSet = $ValueAdd | Sort-Object -Unique

    }

$Property.Value = $ValueSet -join ‘,’

                $Proxy.SetSystemProperties($Property)

    Write-Output ‘  Successfully set property to: *.*’

}

catch

{

                Write-Warning “Failure occurred: $error”

}

Write-Output ‘Script completed!’

 

Successfully set property to: *.*
PS C:\Windows\system32> Write-Output ‘Script completed!’
Script completed!
PS C:\Windows\system32>

 

Don’t forget to restart SSRS.

Verify SVC/MSA account permissions, then begin SCOM Reporting role!

Enjoy!

New SQL Updates

Updated SQL patches released in July
Updated SQL patches released in July

It’s that time again, time to update SQL.  Just in case your configuration management solution automatically add SQL updates, you can be prepared.  Secondly, if you have to tell the configuration management team to approve updates, patches, this will help jumpstart that process.  Either way, knowing about the updates helps you make decisions for your organization’s change process.  I believe ‘knowledge is power’, so power up and take away whatever you need to keep up to date.

 

 

Subset of the SQL product group released ‘new SQL updates’ in July

SQL2016SP3GDR Security Update – 9 July https://techcommunity.microsoft.com/t5/sql-server-blog/security-update-for-sql-server-2016-sp3-gdr/ba-p/4187396

SQL2017RTMCU31 Security Update – 9 July https://techcommunity.microsoft.com/t5/sql-server-blog/security-update-for-sql-server-2017-rtm-cu31/ba-p/4187385

SQL2019 RTM CU27 Security Update – 9 July https://techcommunity.microsoft.com/t5/sql-server-blog/security-update-for-sql-server-2019-rtm-cu27/ba-p/4187401

SQL2022 RTM CU13 Security Update – 9 July https://techcommunity.microsoft.com/t5/sql-server-blog/security-update-for-sql-server-2022-rtm-cu13/ba-p/4187356

SQL2022 RTM CU14 23 July https://techcommunity.microsoft.com/t5/sql-server-blog/cumulative-update-14-for-sql-server-2022-rtm/ba-p/4199659

 

 

Example New SQL Updates for SQL2022

Cumulative Update #14 for SQL Server 2022 RTM

The 14th cumulative update release for SQL Server 2022 RTM is now available for download at the Microsoft Downloads site. Please note that registration is no longer required to download Cumulative updates.
To learn more about the release or servicing model, please visit:

Starting with SQL Server 2017, we adopted a new modern servicing model. Please refer to our blog for more details on Modern Servicing Model for SQL Server

New SQL management pack

SQL Server Blog – New SQL Management pack released!

 

The blog posting the pack release fell through the cracks.  Released on 10 July, I’ve had some issues getting the updated MSI’s, but they’re live now.   I normally use the SQL Tech Community SQL releases site https://techcommunity.microsoft.com/t5/sql-server-blog/bg-p/SQLServer/label-name/SQLReleases

 

Don’t forget to look for SQL Security updates, (CU) Cumulative Updates, or (SP) Service Pack updates at the SQL releases link!  https://techcommunity.microsoft.com/t5/sql-server-blog/bg-p/SQLServer/label-name/SQLReleases

 

 

New SQL pack released

Microsoft System Center Management Pack for SQL Server enables the discovery and monitoring of SQL Server 2012, 2014, 2016, 2017, 2019, 2022, and upcoming versions.

Download link https://www.microsoft.com/en-us/download/details.aspx/?id=56203

Version:
7.6.5

File Name:
SQLServerMP.Windows.msi

SQLServerMP.CustomMonitoring.msi

SQLServerMP.Linux.msi

SQLServerMPWorkflowList.pdf <missing as of today>

Date Published:
7/10/2024

Functionality https://learn.microsoft.com/en-us/system-center/scom/sql-server-management-pack-changes-history?view=sc-om-2022

https://www.microsoft.com/en-us/download/details.aspx/?id=56203

June 2024 – 7.5.19.0 CTP

What’s New

  • Added new “Table Clustered Index Fragmentation” monitor that targets databases and checks for high fragmentation of clustered indexes
  • Added new “Property Bag” step in the custom monitor setup to extend the alert context with a property from the query result
  • Updated the “Product Version Compliance” monitor with the most recent version of public updates for the SQL Server
  • Reworked the “Long Running Queries” alert rule to improve security
  • Improved accessibility for the Summary Dashboard view and Monitoring Wizard template, including the following major changes:
    • implemented Keyboard Navigation using the A and D buttons on the tiles in the dashboard
    • added the ability for the screen reader to announce buttons and errors in the SQL Server wizard
    • redesigned dashboard list controls for greater accessibility

 

Pretty simple steps

Download and save to your SCOM server, or SCOM console connected machine

Navigate to the Administration tab

Expand Management Packs

Click on Installed Management packs

Click the Add drop-down, select the packs

Verify selections, and click Install button

Importing new SQL v7.6.5.0 packs into the SCOM Console

Click Close after import

v7.6.5.0 management packs are imported into the SCOM console

 

Enjoy!

 

NOT monitored servers

Ever run into NOT monitored servers?
NOT monitored servers
NOT monitored servers
Unsure your experience, but recently, I’ve run across multiple occurrences where servers show up as ‘not monitored.  As a result, I’ve found three distinct sets of issues that might cause ‘not monitored’ status –
1) Orphaned Agent blog
This scenario usually entails deleting server from Managed Agents view in Administration tab, where alerts or other details may still exist.  The procedure leverages Holman’s orphaned agent blog (tried and true) post from years back to aid cleanup.
2) Do you have packs or connectors extending classes?
3) Rebuilding a server with the same name is a common server occurrence
Related to 1, Holman’s orphaned agent blog to be used to cleanup.
First, let’s test in the Lab.  Second, let’s talk about the express lane ‘easy button’.  Begin by deleting the Windows Computer orphaned object GUID.  Process the Windows Computer object (bottom), followed by the top two (2) are HealthService, and HealthServiceWatcher object properties (see three items highlighted).
16db02 properties
16db02 properties
Second piece, marking the Windows Computer GUID for deletion (IsDeleted=1) cleans up nearly ALL properties.  See the progress below, how this slight change makes short order of orphaned properties for ‘server’.
Windows Computer object marks all but SCVMM for deletion
Windows Computer object marks all but SCVMM for deletion
Third HealthService & SCVMM objects require manual deletion per GUID.
Note first screenshot shows health service properties marked ‘IsDeleted’ = 1 after manually processing each GUID.
HealthService marked for deletion
HealthService marked for deletion
Fourth, screenshot shows there the HealthServiceWatcher property is marked for deletion (IsDeleted=1)
HealthServiceWatcher marked for deletion
HealthServiceWatcher marked for deletion
If you have SCVMM, you will need to repeat for each of the SCVMM properties to clear out the orphans in the DB.
Why – the issue:
Typically, when servers are reimaged, i.e. NOT deleted from SCOM, there are two+ healthservice, HealthServiceWatcher, Windows Computer properties created for each image of example server.   Additional properties may show duplicated for any class discoveries that are common to the old and new image.
NOTE: Deleting the current agent may clean up objects for that instance of the discovered server, but NOT the old server image.

SCOM MS TLS1.2 drivers

SCOM MS TLS1.2 drivers
SCOM MS TLS1.2 drivers

Courtesy of Brook Hudson, who provided clarification for encrypting SCOM data –

Question – Can we update the OLE DB Driver from 18.6.5 to 18.6.7 and the ODBC driver from 17.10.3 to 17.10.5.1 without breaking anything?

 

This configuration applies to SCOM2016 forward –

MS OLE DB Driver 18.6.7: https://go.microsoft.com/fwlink/?linkid=2242656

ODBC Driver 17.10.5.1: https://go.microsoft.com/fwlink/?linkid=2249004

 

 

I did NOT have success with this for SCOM2019 and SCOM2022 –

If the SQL endpoint is secured with encryption, then the following drivers can be used.

MS OLE DB Driver 19.3.2: https://aka.ms/downloadmsoledbsql

ODBC Driver 18.3.2.1: https://aka.ms/downloadmsodbcsql

If you want to use these newer drivers then SQL encryption is required, more information about enabling SQL Encryption: Configure SQL Server Database Engine for encryption – SQL Server | Microsoft Learnhttps://learn.microsoft.com/en-us/sql/database-engine/configure-windows/configure-sql-server-encryption?view=sql-server-ver15

 

The SQL team noted that the newer versions are defaulting Encrypt to be Yes/Mandatory. That is why the new drivers were having an issue. Setting up a certificate in the SQL endpoint would have allowed the connection to work:

Enable encrypted connections – SQL Server | Microsoft Docs

Certificate Management (SQL Server Configuration Manager) – SQL Server | Microsoft Docs

OLE DB Driver 19.0 for SQL Server Released – Microsoft Tech Community

ODBC Driver 18.0 for SQL Server Released – Microsoft Tech Community

 

IMPORTANT:

Update: Hotfixes released for ODBC and OLE DB drivers for SQL Server – Microsoft Community Hub

 

SQL SysMessages 18054 events

 

SCOM2016+ SQL SysMessages 18054 events
SCOM2016+ SQL SysMessages 18054 events

Whether you’re building a new SCOM2019, SCOM2022 environment or not, you might be missing these event details, and NOT even know!

 

It’s been a while for me, and I came across these, so posting for a fresh heads up!

Leverage Holman’s TXT files to keep your logging up to maximum potential!  Use the information below to resolve SCOM2016+ SQL SysMessages and 18054 events.

Holman GitHub download – contains SQL TXT files to run on OpsMgr & DW databases https://github.com/thekevinholman/SQLFix18054EventsSysmessages

 

The Github TXT files to download contain a clear scope of messages.

 

SQL messages excerpt

—————————————–
— MOMv3 messages are 77798xxxx —
—————————————–

———————————————–
— Discovery range: 77798-0000 to 77798-0049 —
———————————————–
— Managed type doesn’t exist.
EXECUTE sp_addmessage @msgnum = 777980000, @msgtext = N’The specified managed type doesn”t exist.’, @severity = 16, @lang = ‘us_english’, @with_log = false, @replace = ‘REPLACE’
GO

— Relationship type doesn’t exist.
EXECUTE sp_addmessage @msgnum = 777980001, @msgtext = N’The specified relationship type doesn”t exist.’, @severity = 16, @lang = ‘us_english’, @with_log = false, @replace = ‘REPLACE’
GO

— Source entity of the relationship doesn’t exist.
EXECUTE sp_addmessage @msgnum = 777980002, @msgtext = N’The specified relationship doesn”t have a valid source.’, @severity = 16, @lang = ‘us_english’, @with_log = false, @replace = ‘REPLACE’
GO

— Target entity of the relationship doesn’t exist.
EXECUTE sp_addmessage @msgnum = 777980003, @msgtext = N’The specified relationship doesn”t have a valid target.’, @severity = 16, @lang = ‘us_english’, @with_log = false, @replace = ‘REPLACE’
GO

— Discovery data from invalid managed entity is dropped.
EXECUTE sp_addmessage @msgnum = 777980004, @msgtext = N’Discovery data has been received from a rule targeted to a non-existent entity. The discovery data will be dropped.’, @severity = 16, @lang = ‘us_english’, @with_log = false, @replace = ‘REPLACE’
GO

— Invalid relationship rejected by cycle detection.
EXECUTE sp_addmessage @msgnum = 777980005, @msgtext = N’Relationship {%s} was rejected because it would cause a containment cycle; relationship source = ”%s” and target = ”%s”.’, @severity = 16, @lang = ‘us_english’, @with_log = false, @replace = ‘REPLACE’
GO

— Discovery data generated by invalid connector.
EXECUTE sp_addmessage @msgnum = 777980006, @msgtext = N’Discovery data generated by invalid connector:%s.’, @severity = 16, @lang = ‘us_english’, @with_log = false, @replace = ‘REPLACE’
GO

— Discovery data generated by invalid rule, task, discovery.
EXECUTE sp_addmessage @msgnum = 777980007, @msgtext = N’Discovery data generated by invalid discovery source. Id:%s.’, @severity = 16, @lang = ‘us_english’, @with_log = false, @replace = ‘REPLACE’
GO

Documentation links

Blog for 18054 events https://kevinholman.com/2017/08/27/scom-2016-event-18054-errors-in-the-sql-application-log/

SCOM 2016, 2019 and 2022: Event 18054 errors in the SQL application log

Alternatively, if AlwaysOn configuration, leverage Holman’s newer blog post – https://kevinholman.com/2022/10/02/scom-deployment-configuration-for-sql-always-on/

SCOM deployment configuration for SQL Always On