OS Addendum updates

OS Addendum updates - Can I get your attention please!?
Can I get your attention please!?

 

More updates for your monitoring pleasure with OS addendum updates!

 

OS Addendum updates

Been busy in the monitoring ‘bat’ cave crafting up new ways to simplify things, automating recoveries, top process finds, STIG compliance, automatic services logic, and PowerShell transcription checks.

Download https://github.com/theKevinJustin/2016ServerAgnostic/

 

Automatic Services – monitors services with automatic service type, and includes built in recovery when monitor unhealthy

PowerShell transcription – be compliant for STIG V-257502 and V-257503, also includes Daily report (runs daily Monday-Friday)

Sets multiple overrides to reduce extraneous OS workflows that most customers never use.

 

Change/revision history

v1.0.7.6 13 Jan 2025 Updated EnableTranscription Report variables
v1.0.7.5 8 Nov 2024 Added additional services into excluded services, uncommented ServiceName variable
v1.0.7.4 7 Nov 2024 Updated ExcludedServices in AutomaticServices monitor DS
v1.0.7.3 21 Oct 2024 Logical Disk rollup disable, hourly monitor setting changes, event and perf collection rule disabled
v1.0.7.2 21 Oct 2024 Updated AutomaticServices DS/WA workflows
v1.0.7.1 18 Oct 2024 Updated Excluded Services array with additional services, updated AutomaticServices DS/WA workflows.
v1.0.7.0 17 Oct 2024 Updated AutomaticServices monitor timeout from 240 to 600 for OpsMgr event ID 22404
v1.0.6.8 16 Oct 2024 Updated Storport Timed Out Monitor from 5 to 10 events per hour
v1.0.6.8 11 Oct 2024 Additional EnableTranscription workflows – reports, report cleanup, transcription log cleanup
v1.0.6.7 11 Apr 2024 New weekly PowerShell Enable Transcription monitor for STIG V-257502 and V-257503, and new weekly report
v1.0.6.5 10 Apr 2024 Updated recovery logic for AutomaticServices DS/WA
v1.0.6.4 9 Apr 2024 Updated AutomaticServices DS/WA logic change, ServiceName variable change to Service

Updated NOSC Daily Tasks

Keep your head up

Updated NOSC Daily Tasks with more insights, whether NOC/NOSC, or SCOM Admin related, check out the GitHub for the pack and change/revision history.

Keep your head up!   I find this is always a positive message to look up, not down. Leverage new key insights and download the pack from my GitHub repo – Proactive NOSC Daily Tasks link

 

Updated NOSC Daily Tasks Summary

Latest round adds simplification of SCOM agent workflow errors, adding the offending computer with the SCOMAdmin DailySummary alert details.

Offending alert examples from multiple customers

MSSQL on Windows: SQL Server has failed to allocate sufficient memory to run

Alert generation was temporarily suspended due to too many alerts (event ID 5399)

PowerShell was dropped

Expression Filter Module Failed Initialization (group regex errors)

The November pack updates add TicketID field to the SCOMAdmin, Daily Summary, Logical Disk report, and Alert updates reports.  This is invaluable when integrating service management (ITSM) system events/alerts/incidents into your monitoring.  Lastly, visibility into created incidents is key to business issues (see the AlertUpdates workflows).

Details

NOSC Management pack provides summary report alerts of key insights including: Expiring certificates, Logical Disk alerts, Pending reboots, System Admin summary, and SCOM admin reports including long-running scripts, script errors, SCOM errors, and alert updates report.

Blog https://kevinjustin.com/blog/2023/08/15/proactive-daily-reports/

Change History

v1.0.5.7  13 Jan 2025 Updated SCOMAlerts report details with format-table properties from select
v1.0.5.6  15 Nov 2024 SCOMAdmin and Daily Summary, Logical disk report changes
v1.0.5.4  12 Nov 2024 AlertUpdates report and various logging changes
v1.0.5.3   5 Nov 2024 Enabled AlertUpdate rules
v1.0.5.2  30 Oct 2024 Daily Summary and SCOMAlerts report updates
v1.0.5.1  17 Oct 2024 Added Operations Manager Event ID's 22402, 22406
v1.0.5.0   4 Jan 2024 Resolution State logic improvements for large environments
v1.0.4.9  21 Dec 2023 WhiteSpace, newline, return updates, Expiring Certs report moved back 1 hour
v1.0.4.8  20 Dec 2023 Updated all Get-SCOMAlert queries to use -ResolutionState (0..254) for performance increase over where-object
v1.0.4.7  18 Dec 2023 Updated Expiring Certs DS/WA, whitespace code check
v1.0.4.6  30 Nov 2023 Removed debug detail from DS/WA which showed in Health Explorer pane

Alert on DWDataRP output

Holman's blog for DWDataRP is one way to Alert on DWDataRP findings
Holman’s blog for DWDataRP is one way to Alert on DWDataRP findings

I want to alert on DWDataRP output!  While everyone’s familiar with Holman’s SCOM SQL queries blog, read below to configure a new way to maintain data warehouse integrity and retention.

 

Data warehouse audits are included in monitoring platform checks.  For those new to monitoring, basically DWDataRP analyzes SCOM Data Warehouse issues for alert/event/performance/state retention.  One administration option is to utilize the SCOM Core Monitoring Addendum pack to run DWDataRP.  Another option is to run DWDataRP via Holman’s blog, or recently with Blake Drumm’s GUI tool.

 

Using the SCOM Core Monitoring addendum pack pre-configures a number of overrides, as well as adding DWDataRP monitor/rule options.  Consequently, the SCOM action account needs to have additional permissions on SCOM SQL servers where the OperationsManagerDW
resides.

Configure SCOM management server action account to alert on DWDataRP output

Example uses lab environment SVC.SCOM.PBIreader
Substitute the SCOM action account above for the SCOM data warehouse (OperationsManagerDW) databases on their respective SCOM management group(s).

Give SCOM Action account necessary rights
Update SVC account rights
Set and verify SVC account has Server role public

Click on User Mapping > select OperationsManagerDW database
Verify Default Schema shows DBO
Under Database Role Membership
Select db_datareader AND db_owner
Click OK



Verification
Reach out to SCOM team to verify execution
From SCOM, RDP to one of the management servers
Click on Start > Right click on Windows PowerShell
Click on More > click on Select PowerShell
Click on More > Click on 'Run as a different user'
Open PowerShell > right click > Run as a different user
Open PowerShell > right click > Run as a different user
On the Windows Security pop-up > Click on 'Use a different account'
Type the action account username and password
Click OK

Click Use a different account in the 'Run As different user' popup
Click Use a different account in the ‘Run As different user’ popup
Paste in the following commands, and verify output
cd "##YourPathtoDWDATARP.EXE##"
# cd D:\MonAdmin\TOOLS\DWDataRP"

# Check events
$Command = '.\dwdatarp.exe -s 16DB02 -d OperationsManagerDW -ds "Event Data
Set"'
$EventDataSet = Invoke-Expression $Command
$EventDataSet
$EventDataSet[2]
$LLineSplit = $EventDataSet[2].Split("(")
$EventDBPercent = $LLineSplit[1].Split("%")
$EventDBPercent[0]
DWDataRP PowerShell event output
DWDataRP PowerShell event output
Example PowerShell output when SVC Account cannot execute DWDataRP
PS C:\monadmin\tools\dwdatarp> whoami
testlab\svc.scom.pbireader
PS C:\monadmin\tools\dwdatarp> .\dwdatarp.exe -s 16db02 -d OperationsManagerDW -ds
Event
Dataset name
Aggregation name Max Age Current Size, Kb
----------------------------------------------------------------------------

Data Integration with SQL2022

Integration - time to integrate data sources to data lake
Integration – time to integrate data sources to data lake

Ready for a single pane of glass?  Ready to have your insights in a common location?  Let’s discuss Data Integration with SQL2022.

 

Let’s start with some background on SQL2022 and similarly SQL2025, start with the learn site link.  SQL2022 by design is Azure enabled with multiple capabilities like ‘Bi-directional HA/DR to Azure SQL’ and ‘Azure Synapse Link’.  Basically, Synapse link is the key.

SQL2022 by design is Azure Enabled
SQL2022 by design is Azure Enabled

Utilize the PowerBI Cloud Service with today’s hybrid environments.  SQL2022 allows integration with other Azure capabilities like Azure Data factory/data lake, and Azure Synapse.  Another reason to upgrade SQL 2022, is design simplification.  However, PowerBI data gateway adds a potential break point (single point of failure).  While PowerBI data gateway centralizes all premise data to a central location.  In the same way, consolidating data sent to the cloud.  When PowerBI data gateway fails, insights and visualizations have stale data (i.e. data NOT transferred for a near real-time display).

 

Why SQL2022 then?

Connect insights and visualization to justify ‘Data Integration with SQL2022’ scenarios.

SQL2022 built in capability to Azure Synapse Analytics
SQL2022 built in capability to Azure Synapse Analytics

Use SQL2022 to configure SQL agent jobs which pull SQL scripts from your cloud environment.  DevOps and common Azure Storage repository are great advantages for speed of execution.

SQL STIGMonitor

End the STIG(ma)

Seriously, dream on!  End the STIGma is a good thing, but STIGs can be a burden.  Hit the easy button, if you’re not already using it.  Contact your SQL Data and AI Cloud Solutions Architect for the latest SQL STIG Monitor 2024 Q4 build!


 

Latest SQL STIG monitor 31 Oct 2024 release includes

DISA UPDATES – see link
MS SQL Server 2016 Instance STIG, V3R2:

(NOTE: DISA has been contacted to remove related CCI STIGID for AzureSQLDB that was overlooked: ASQL-00-010700)

POWERSHELL MODULE
Updated version to 1.23
Added STIGID parameter to Invoke-StigMonitor allowing granular control over STIGID scanning.

DATABASE CHANGES
Updated Checklist Templates for Q4 Revisions.
Updated Instance & Database STIG for Q4 benchmark date.
Script updates include:

CNTNMIXDB: Not A Finding if using Windows Auth
FORCENRYPT: NA if using Windows Auth
PWDCMPLX: Updated Finding to remove OS STIG reference
AZDBPERMISS: Revised script with new version.
DBPERMISS: Revised script with new version.
ENFCACCSS: Revised script with new version.
PSERRPERM: Revised script with new version.
UNQSVCACC: Removed code stripping out port number.
AZAUDITSTATE: Properly returns No Finding when audit setup is correct.
Fixed bug in vDocumentation view causing POAMs to not display custom comment in exported documentation.
Added usp_RemoveInstance stored procedure to easily clean up a specific Instance from StigMonitor that no longer exists.

DOCUMENTS
Updated checklist templates, Approvals scripts, and Documentation Templates for Q4 Revisions.
Removed Set-CEIPRegKeys.ps1, Set-FIPSCompliance.ps1, and Set-SqlRegKey.ps1 in favor of Module commands.
Updated InfoPage with new StigMonitor logo and text references.
Documentation updated with new examples of Invoke-StigMonitor STIGID parameter.
Updated documentation to add Azure DB Permission for MS_SecurityDefinitionReader.
Added DatabaseName to CSV Export of Export-StigDocumentation.

REPORTS
Updated Report banner to display new StigMonitor logo and latest report versions.
Removed Adhoc scanning to Policy Management Report in favor of Invoke-StigMonitor parameter.
Removed references to Sunset 2012 and 2014 STIGs.
Added AzureSQLMI for future use.
Combined NF and Approved in Total Findings summary
Reduced Recent Scans to latest 6.

Also please send us your feedback if you get a chance to check this out.
If you want to be added/removed from this, click here (Subscribe /Unsubscribe) or send us an email.

 

SCOM SSRS permissions

Microsoft SQL Server SSRS icon
Microsoft SQL Server SSRS icon

Let’s discuss SCOM SSRS permissions.  The SCOM Reporting role install really comes down to three (3) things – permissions, latest SSRS EXE downloaded (for install 2019, 2022), and ReportExtensions configuration.  I’ve hit some permission issues that need more ‘how to’ details.

 

Set SCOM Admins group permissions

Whether the permissions are set up as part of a group policy (GPO) or not, if these steps are missing, expect problems.

Verify that your SCOM Admins domain group is a local administrator on the SCOM servers (SSRS server in this case)

Right click on Start > Computer Management

Expand System Tools

Expand Local Users and Groups

Click on Groups

Double click on Administrators

Verify SCOM Admins group, or specific service/MSA accounts are listed

Computer Management with Administrators group properties documenting relevant members which include the SCOM Admins group, and any other SQL related service accounts.
Computer Management with Administrators group properties documenting relevant members which include the SCOM Admins group, and any other SQL related service accounts.

Click OK

 

 

Set SQL Instance permissions for SCOM Admins group

Reference Holman’s QuickStart > Install SCOM Reporting Role…

  • Log on using your domain user account that is a member of the OMAdmins group, and has “sysadmin” role level rights over the SQL instance.

RDP to server with SSMS that connects to SQL server

Connect to Database Engine

Expand instance , then expand Security folder, thirdly expand Logins folder

Right click on the SCOM Admins group and select properties

In the pop-up, click on SQL Server Role

Verify that sysAdmin

View of SSMS Database Engine showing SCOM Admins group SQL Server Role has sysAdmin
View of SSMS Database Engine showing SCOM Admins group SQL Server Role has sysAdmin

Follow similar steps if using a domain connected SVC/MSA account when configuration differs from Holman’s QuickStart template.

 

Additional troubleshooting from the SCOM install can be found in the user’s directory – C:\Users\<accountHere>\AppData\Local\SCOM\LOGS

 

Find additional details in the SQL install logs

C:\Program Files\Microsoft SQL Server\MSRS13.MSSQLSERVER\Reporting Services\LogFiles

NOTE that the Instance and version 'MSRS13.MSSQLSERVER' can change

 

 

Additional documentation and relevant links

The go-to reference is Holman’s QuickStart deployment guides for SCOM2019 forward list the how-to starting point.

Holman Quick Start links:

https://kevinholman.com/2022/05/01/scom-2022-quickstart-deployment-guide/

https://kevinholman.com/2019/03/14/scom-2019-quickstart-deployment-guide/

 

SSRS learn.microsoft.com site article https://learn.microsoft.com/en-us/troubleshoot/system-center/scom/cannot-deploy-operations-manager-reports

SSRS Error occurred when invoking the authorization extension https://learn.microsoft.com/en-us/answers/questions/266488/installing-scom-2019-reporting-ssrs-2019-error-an

SCOM SSRS ReportExtensions

For a smooth install, everything comes down to SCOM SSRS prerequisites.  The SCOM Reporting role install really comes down to three (3) things – permissions, latest SSRS EXE downloaded (for install 2019, 2022), and ReportExtensions configuration.  The go-to reference is Holman’s QuickStart deployment guides for SCOM2019 forward list the how-to starting point.  This post focuses on ReportExtensions configuration, where more ‘how to’ details are needed.

Quick Start links:

SCOM 2022 – QuickStart Deployment Guide

SCOM 2019 – QuickStart Deployment Guide

SSRS learn.microsoft.com site article https://learn.microsoft.com/en-us/troubleshoot/system-center/scom/cannot-deploy-operations-manager-reports

 

Configure Report Extensions via SSMS (GUI)

RDP to server with enabled account

Open SSMS that has connectivity to SSRS install/server

Change ‘Server type’ drop-down to Reporting Service

Change SSMS Server Type from Database Engine to Reporting Service
Change SSMS Server Type from Database Engine to Reporting Service

Click Connect

Right click on Server > Properties

In the Server Properties window, select the Advanced Tab

Click on the AllowedResourceExtensionsForUpload, and add *.*

Click OK

Screenshot of SSMS Connected to Reporting Service, expanding SSRS Properties > Advanced Tab > showing AllowedResourceExtensionsForUpload
Screenshot of SSMS Connected to Reporting Service, expanding SSRS Properties > Advanced Tab > showing AllowedResourceExtensionsForUpload

Don’t forget to restart SSRS to make changes take effect!

Once restarted, verify SVC/MSA account permissions, and begin SCOM Reporting role!

 

Configure Report Extensions via PowerShell

Testing learn article PowerShell for SSRS Defaults (pre-requisite for SCOM Reporting role with SSRS2017+ versus SSMS).   > Reporting Services

SSRS Note for ServiceAddress (SSRS URL) is other than localhost

On respective server, open PowerShell as Admin

Paste the following:

$ServiceAddress = ‘http://localhost

$ExtensionAdd = @(

                ‘*’

                ‘CustomConfiguration’

                ‘Report’

                ‘AvailabilityMonitor’

                ‘TopNApplications’

                ‘Settings’

                ‘License’

                ‘ServiceLevelTrackingSummary’

                ‘CustomPerformance’

                ‘MostCommonEvents’

                ‘PerformanceTop’

                ‘Detail’

                ‘DatabaseSettings’

                ‘ServiceLevelObjectiveDetail’

                ‘PerformanceDetail’

                ‘ConfigurationChange’

                ‘TopNErrorGroupsGrowth’

                ‘AvailabilityTime’

                ‘rpdl’

                ‘mp’

                ‘TopNErrorGroups’

                ‘Downtime’

                ‘TopNApplicationsGrowth’

                ‘DisplayStrings’

                ‘Space’

                ‘Override’

                ‘Performance’

                ‘AlertDetail’

                ‘ManagementPackODR’

                ‘AlertsPerDay’

                ‘EventTemplate’

                ‘ManagementGroup’

                ‘Alert’

                ‘EventAnalysis’

                ‘MostCommonAlerts’

                ‘Availability’

                ‘AlertLoggingLatency’

                ‘PerformanceTopInstance’

                ‘rdl’

                ‘PerformanceBySystem’

                ‘InstallUpdateScript’

                ‘PerformanceByUtilization’

                ‘DropScript’

)

Write-Output ‘Setting Allowed Resource Extensions for Upload’

$error.clear()

try

{

                $Uri = [System.Uri]”$ServiceAddress/ReportServer/ReportService2010.asmx”

                $Proxy = New-WebServiceProxy -Uri $Uri -UseDefaultCredential

                $Type = $Proxy.GetType().Namespace + ‘.Property’

                $Property = New-Object -TypeName $Type

                $Property.Name = ‘AllowedResourceExtensionsForUpload’

$ValueAdd = $ExtensionAdd | ForEach-Object -Process {

                                “*.$psItem”

                }

$Current = $Proxy.GetSystemProperties($Property)

                if ($Current)

    {

                $ValueCurrent = $Current.Value -split ‘,’

                $ValueSet = $ValueCurrent + $ValueAdd | Sort-Object -Unique

                }

                else

    {

        $ValueSet = $ValueAdd | Sort-Object -Unique

    }

$Property.Value = $ValueSet -join ‘,’

                $Proxy.SetSystemProperties($Property)

    Write-Output ‘  Successfully set property to: *.*’

}

catch

{

                Write-Warning “Failure occurred: $error”

}

Write-Output ‘Script completed!’

 

Successfully set property to: *.*
PS C:\Windows\system32> Write-Output ‘Script completed!’
Script completed!
PS C:\Windows\system32>

 

Don’t forget to restart SSRS.

Verify SVC/MSA account permissions, then begin SCOM Reporting role!

Enjoy!

DNS Scavenging alerts

DNS Scavenging how it works

Need DNS Scavenging alerts, to see what’s cleaned up, or that scavenging failed?  Download the DNS Addendum pack from my GitHub repo https://github.com/theKevinJustin/DNSAddendumAgnostic

Latest revision first includes a EventID 2502 monitor for scavenging failed.  Second, the monitor has count logic (setup to alert with 2 events in 30 minutes).  Third, EventID 2501 rule details scavenging totals.  Lastly, built a weekly report to summarize the scavenging alerts (cliff notes!).

 

 

Some quick ‘how-to’ setup DNS scavenging

Example of RegKey showing that Scavenging is setup – note Scavenging Interval key

 

Example of AD integrated DNS setup with 21 day scavenging interval, and prompts to configure (click OK twice)

DNS Scavenging setup on AD integrated DNS server

 

Import management pack, and run DNS scavenging.

 

Verify scavenging alerts

SCOM Monitoring Tab > Active Alerts > ‘Look for:’ scavenging

Example output

 

Additional SCOM PowerShell commands

Run PowerShell commands from the SCOM management server (MS)

$DNSAlerts = get-scomalert -name "*Scavenging*"
$DNSAlerts
$DNSAlerts | format-table PrincipalName,TimeRaised,Description -auto -wrap

 

Example Output

PS C:\Users\scomadmin> $DNSAlerts = get-scomalert -name “*Scavenging*”

PS C:\Users\scomadmin> $DNSAlerts

 

Severity     Priority   Name                                                                        TimeRaised

——–     ——–   —-                                                                        ———-

Warning      Normal     Windows DNS Event 2502 Scavenging Failed monitor addendum alert             8/19/2024 2:02:3…

Warning      Normal     Windows DNS Event 2502 Scavenging Failed monitor addendum alert             8/19/2024 1:07:0…

Information  Normal     Proactive DailyTasks DNSAlerts Scavenging Summary Report Alert              8/19/2024 10:11:…

 

 

DNS alerts formatted

PS C:\Users\scomadmin> $DNSAlerts | format-table PrincipalName,TimeRaised,Description -auto -wrap

 

PrincipalName    TimeRaised            Description

————-    ———-            ———–

DC02.testlab.net 8/19/2024 2:02:32 PM  Windows DNS Event 2502 Scavenging Failed monitor alert 1 alert in 15 minutes

Event Description:

The DNS server has completed a scavenging cycle but no nodes were visited.

Possible causes of this condition include:

The next scavenging cycle is scheduled to run in 168 hours.

 

Learn articles for more details https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/dns-scavenging-setup

SCOM Agent Maintenance

Wrench for SCOM agent maintenance
Wrench for SCOM agent maintenance

When we talk about best practices for monitoring, this will typically include (SLA) Service Level Availability.  SLA is an important piece in your environment, as uptime and happy customers come with a high SLA.  There are some cases where IT Teams do work on demand.  On-demand work is outside of a standard change window, a scheduled change.  Typically this is outside configuration management tools, responsible to update software (applications/packages), machines, drivers, compliance settings, and more.  In the one-off, non-scheduled maintenance or recovery, try leveraging ‘SCOM Agent Maintenance’ PowerShell commands on SCOM agents.

 

SCOM Agent maintenance PowerShell commands

cd “C:\Program Files\Microsoft Monitoring Agent\Agent”

Import-module .\MaintenanceMode.dll

Start-SCOMAgentMaintenanceMode -Duration 10 -Reason PlannedOther

 

# Verify

# If messages show with current timestamp, Agent objects are in maintenance.

 

get-eventlog -LogName “Operations Manager” -newest 50 | ? { $_.Message -like “Suspending monitoring*”  } | ft TimeGenerated,Message -autosize

 

TimeGenerated        Message

————-        ——-

6/25/2020 8:37:57 AM Suspending monitoring for instance “modeldev” with id:”{F9E45AA4-7DF7-C1F1-70C9-5D76C9F2725C}” …

6/25/2020 8:37:57 AM Suspending monitoring for instance “C:” with id:”{ED00048A-7DDC-D4BE-901D-D64DA281B7C6}” as the…

6/25/2020 8:37:57 AM Suspending monitoring for instance “central_log” with id:”{EA619D69-D1CC-3B19-D93C-2E3FCD1409AE…

 

PS C:\Program Files\Microsoft Monitoring Agent\Agent> get-eventlog -LogName “Operations Manager” -newest 25 | ? { $_.Message -like “Resuming monitoring*”  } | ft TimeGenerated,Message -autosize

 

343998 Jun 25 08:50  Information HealthService          1073743040 Resuming monitoring for instance “modeldev” wit…

343997 Jun 25 08:50  Information HealthService          1073743040 Resuming monitoring for instance “C:” with id:”…

343996 Jun 25 08:50  Information HealthService          1073743040 Resuming monitoring for instance “central_log” …

343995 Jun 25 08:50  Information HealthService          1073743040 Resuming monitoring for instance “dnmll05s1.UNE…