Alert on DWDataRP output

Holman's blog for DWDataRP is one way to Alert on DWDataRP findings
Holman’s blog for DWDataRP is one way to Alert on DWDataRP findings

I want to alert on DWDataRP output!  While everyone’s familiar with Holman’s SCOM SQL queries blog, read below to configure a new way to maintain data warehouse integrity and retention.

 

Data warehouse audits are included in monitoring platform checks.  For those new to monitoring, basically DWDataRP analyzes SCOM Data Warehouse issues for alert/event/performance/state retention.  One administration option is to utilize the SCOM Core Monitoring Addendum pack to run DWDataRP.  Another option is to run DWDataRP via Holman’s blog, or recently with Blake Drumm’s GUI tool.

 

Using the SCOM Core Monitoring addendum pack pre-configures a number of overrides, as well as adding DWDataRP monitor/rule options.  Consequently, the SCOM action account needs to have additional permissions on SCOM SQL servers where the OperationsManagerDW
resides.

Configure SCOM management server action account to alert on DWDataRP output

Example uses lab environment SVC.SCOM.PBIreader
Substitute the SCOM action account above for the SCOM data warehouse (OperationsManagerDW) databases on their respective SCOM management group(s).

Give SCOM Action account necessary rights
Update SVC account rights
Set and verify SVC account has Server role public

Click on User Mapping > select OperationsManagerDW database
Verify Default Schema shows DBO
Under Database Role Membership
Select db_datareader AND db_owner
Click OK



Verification
Reach out to SCOM team to verify execution
From SCOM, RDP to one of the management servers
Click on Start > Right click on Windows PowerShell
Click on More > click on Select PowerShell
Click on More > Click on 'Run as a different user'
Open PowerShell > right click > Run as a different user
Open PowerShell > right click > Run as a different user
On the Windows Security pop-up > Click on 'Use a different account'
Type the action account username and password
Click OK

Click Use a different account in the 'Run As different user' popup
Click Use a different account in the ‘Run As different user’ popup
Paste in the following commands, and verify output
cd "##YourPathtoDWDATARP.EXE##"
# cd D:\MonAdmin\TOOLS\DWDataRP"

# Check events
$Command = '.\dwdatarp.exe -s 16DB02 -d OperationsManagerDW -ds "Event Data
Set"'
$EventDataSet = Invoke-Expression $Command
$EventDataSet
$EventDataSet[2]
$LLineSplit = $EventDataSet[2].Split("(")
$EventDBPercent = $LLineSplit[1].Split("%")
$EventDBPercent[0]
DWDataRP PowerShell event output
DWDataRP PowerShell event output
Example PowerShell output when SVC Account cannot execute DWDataRP
PS C:\monadmin\tools\dwdatarp> whoami
testlab\svc.scom.pbireader
PS C:\monadmin\tools\dwdatarp> .\dwdatarp.exe -s 16db02 -d OperationsManagerDW -ds
Event
Dataset name
Aggregation name Max Age Current Size, Kb
----------------------------------------------------------------------------

DNS Scavenging alerts

DNS Scavenging how it works

Need DNS Scavenging alerts, to see what’s cleaned up, or that scavenging failed?  Download the DNS Addendum pack from my GitHub repo https://github.com/theKevinJustin/DNSAddendumAgnostic

Latest revision first includes a EventID 2502 monitor for scavenging failed.  Second, the monitor has count logic (setup to alert with 2 events in 30 minutes).  Third, EventID 2501 rule details scavenging totals.  Lastly, built a weekly report to summarize the scavenging alerts (cliff notes!).

 

 

Some quick ‘how-to’ setup DNS scavenging

Example of RegKey showing that Scavenging is setup – note Scavenging Interval key

 

Example of AD integrated DNS setup with 21 day scavenging interval, and prompts to configure (click OK twice)

DNS Scavenging setup on AD integrated DNS server

 

Import management pack, and run DNS scavenging.

 

Verify scavenging alerts

SCOM Monitoring Tab > Active Alerts > ‘Look for:’ scavenging

Example output

 

Additional SCOM PowerShell commands

Run PowerShell commands from the SCOM management server (MS)

$DNSAlerts = get-scomalert -name "*Scavenging*"
$DNSAlerts
$DNSAlerts | format-table PrincipalName,TimeRaised,Description -auto -wrap

 

Example Output

PS C:\Users\scomadmin> $DNSAlerts = get-scomalert -name “*Scavenging*”

PS C:\Users\scomadmin> $DNSAlerts

 

Severity     Priority   Name                                                                        TimeRaised

——–     ——–   —-                                                                        ———-

Warning      Normal     Windows DNS Event 2502 Scavenging Failed monitor addendum alert             8/19/2024 2:02:3…

Warning      Normal     Windows DNS Event 2502 Scavenging Failed monitor addendum alert             8/19/2024 1:07:0…

Information  Normal     Proactive DailyTasks DNSAlerts Scavenging Summary Report Alert              8/19/2024 10:11:…

 

 

DNS alerts formatted

PS C:\Users\scomadmin> $DNSAlerts | format-table PrincipalName,TimeRaised,Description -auto -wrap

 

PrincipalName    TimeRaised            Description

————-    ———-            ———–

DC02.testlab.net 8/19/2024 2:02:32 PM  Windows DNS Event 2502 Scavenging Failed monitor alert 1 alert in 15 minutes

Event Description:

The DNS server has completed a scavenging cycle but no nodes were visited.

Possible causes of this condition include:

The next scavenging cycle is scheduled to run in 168 hours.

 

Learn articles for more details https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/dns-scavenging-setup

New SQL Updates

Updated SQL patches released in July
Updated SQL patches released in July

It’s that time again, time to update SQL.  Just in case your configuration management solution automatically add SQL updates, you can be prepared.  Secondly, if you have to tell the configuration management team to approve updates, patches, this will help jumpstart that process.  Either way, knowing about the updates helps you make decisions for your organization’s change process.  I believe ‘knowledge is power’, so power up and take away whatever you need to keep up to date.

 

 

Subset of the SQL product group released ‘new SQL updates’ in July

SQL2016SP3GDR Security Update – 9 July https://techcommunity.microsoft.com/t5/sql-server-blog/security-update-for-sql-server-2016-sp3-gdr/ba-p/4187396

SQL2017RTMCU31 Security Update – 9 July https://techcommunity.microsoft.com/t5/sql-server-blog/security-update-for-sql-server-2017-rtm-cu31/ba-p/4187385

SQL2019 RTM CU27 Security Update – 9 July https://techcommunity.microsoft.com/t5/sql-server-blog/security-update-for-sql-server-2019-rtm-cu27/ba-p/4187401

SQL2022 RTM CU13 Security Update – 9 July https://techcommunity.microsoft.com/t5/sql-server-blog/security-update-for-sql-server-2022-rtm-cu13/ba-p/4187356

SQL2022 RTM CU14 23 July https://techcommunity.microsoft.com/t5/sql-server-blog/cumulative-update-14-for-sql-server-2022-rtm/ba-p/4199659

 

 

Example New SQL Updates for SQL2022

Cumulative Update #14 for SQL Server 2022 RTM

The 14th cumulative update release for SQL Server 2022 RTM is now available for download at the Microsoft Downloads site. Please note that registration is no longer required to download Cumulative updates.
To learn more about the release or servicing model, please visit:

Starting with SQL Server 2017, we adopted a new modern servicing model. Please refer to our blog for more details on Modern Servicing Model for SQL Server

New SQL management pack

SQL Server Blog – New SQL Management pack released!

 

The blog posting the pack release fell through the cracks.  Released on 10 July, I’ve had some issues getting the updated MSI’s, but they’re live now.   I normally use the SQL Tech Community SQL releases site https://techcommunity.microsoft.com/t5/sql-server-blog/bg-p/SQLServer/label-name/SQLReleases

 

Don’t forget to look for SQL Security updates, (CU) Cumulative Updates, or (SP) Service Pack updates at the SQL releases link!  https://techcommunity.microsoft.com/t5/sql-server-blog/bg-p/SQLServer/label-name/SQLReleases

 

 

New SQL pack released

Microsoft System Center Management Pack for SQL Server enables the discovery and monitoring of SQL Server 2012, 2014, 2016, 2017, 2019, 2022, and upcoming versions.

Download link https://www.microsoft.com/en-us/download/details.aspx/?id=56203

Version:
7.6.5

File Name:
SQLServerMP.Windows.msi

SQLServerMP.CustomMonitoring.msi

SQLServerMP.Linux.msi

SQLServerMPWorkflowList.pdf <missing as of today>

Date Published:
7/10/2024

Functionality https://learn.microsoft.com/en-us/system-center/scom/sql-server-management-pack-changes-history?view=sc-om-2022

https://www.microsoft.com/en-us/download/details.aspx/?id=56203

June 2024 – 7.5.19.0 CTP

What’s New

  • Added new “Table Clustered Index Fragmentation” monitor that targets databases and checks for high fragmentation of clustered indexes
  • Added new “Property Bag” step in the custom monitor setup to extend the alert context with a property from the query result
  • Updated the “Product Version Compliance” monitor with the most recent version of public updates for the SQL Server
  • Reworked the “Long Running Queries” alert rule to improve security
  • Improved accessibility for the Summary Dashboard view and Monitoring Wizard template, including the following major changes:
    • implemented Keyboard Navigation using the A and D buttons on the tiles in the dashboard
    • added the ability for the screen reader to announce buttons and errors in the SQL Server wizard
    • redesigned dashboard list controls for greater accessibility

 

Pretty simple steps

Download and save to your SCOM server, or SCOM console connected machine

Navigate to the Administration tab

Expand Management Packs

Click on Installed Management packs

Click the Add drop-down, select the packs

Verify selections, and click Install button

Importing new SQL v7.6.5.0 packs into the SCOM Console

Click Close after import

v7.6.5.0 management packs are imported into the SCOM console

 

Enjoy!

 

Create ServiceNow incidents from SCOM

Create ServiceNow incidents from SCOM
Create ServiceNow incidents from SCOM

Been working to ‘Create ServiceNow incidents from SCOM’ with Joe Kelly (Joe’s LinkedIn), Steven Brown (Steven’s LinkedIn), and Tim Fields (Tim’s LinkedIn).  Over the past few months, we’ve found quite a few methods available, some free, some with cost.  All methods are built on ServiceNow (SNow) REST commands.  The resulting PowerShell scripts can do most of the heavy lifting for other Tools like SolarWinds, vCenter, vRealize,  Zabbix, SAP manager, Kafka, Nagios, HP OpenView.

 

Last month’s blog showcased the ServiceNow API/Connector for SCOM

ServiceNow Connector for SCOM

 

Depending on your customer use cases, this may not be the preferred choice.

One SNOW flavor is like NetCool, where you inject ALL SCOM alerts into ITSM tool (i.e. SNOW events).  Another flavor allows you to create a group of objects which will inject SNOW events.

What if you only want incidents for a small subset, and don’t want to wait on SNOW team?

We got that covered.

What if SNOW team wants events only, how can you create an incident?

We got that covered.

 

ServiceNow connector options:

Inject SNOW incident from SCOM (direct) https://github.com/theKevinJustin/New-SNOWIncidents

Inject SNOW event (requires SNOW alert rules to process alert to incident) https://github.com/theKevinJustin/New-SNowEvent

SNOW SCOM connector

    1. Send all SCOM alerts to SNOW as events
    2. Send specific SCOM group of objects to SNOW as events

3rd party options

      1. OpsLogix SCOM ServiceNow connector (costs) https://www.opslogix.com/servicenow-incident-connector
      2. Cookdown SNOW incident connector https://www.cookdown.com/blog/how-to-build-a-servicenow-incident-connector-for-scom download MP https://www.cookdown.com/servicenow-monitoring-mp#download
      3. Tim McFadden SCOM ServiceNow connector https://www.scom2k7.com/introducing-the-scom-to-servicenow-connector/ (unsure if Tim still supports this)

 

Stay tuned…

Next steps are to build out the SCOM command execution channel using Holman’s blog https://kevinholman.com/2021/08/25/what-account-will-command-channel-notifications-run-as-in-scom/

ServiceNow Connector for SCOM

ServiceNow SNow Connector for SCOM
ServiceNow SNow Connector for SCOM

Tim Fields (see Tim’s LinkedIn) and I were recently working on ServiceNow (SNow) REST commands with a customer.  Little did we know the SNow API/connector was released in February.  We’ve been waiting for months for the release.  It was also nice to find that many other tools were mapped.  Tools like SolarWinds, vCenter, vRealize,  Zabbix, SAP manager, Kafka, Nagios, HP OpenView, and even email, SNMP traps were already done.  Not that I didn’t figure this existed, just wasn’t part of multiple customer’s request.  The SNow connectors are typically installed on SNow MID server(s).  In SCOM’s case, the connector uses SCOM DLL’s, a configuration file including name, IP, ID with access.

 

Depending on use cases, you have some options to what best fits your needs.

  1. SNow connector only works for a subset of SCOM alerts
    • Out of the Box (OotB) built connector. SCOM SNow connector assumes you have SNow MID server(s) installed and configured.
    • This is better than NetCool connector, as everything was sent to ITSM tool.
    • Connector allows SCOM group to customize objects for alert/incident creation
    • Note:

The default binding rules that contain SCOM as the external source, that applies to IT alerts and Metric Intelligence raw data, are the following SCOM Management Packs:

All OS Management Packs

MS SQL Server

IIS

 

Example

If bi-directional is configured, the bi-directional exchange of values to-and-from the external event source is enabled.

These scenarios describe the default bi-directional functionality for SCOM connectors:

  • When an alert is resolved in SCOM, it is auto-closed in ServiceNow. However, it is updated irrespective of the bi-directional feature because during each collection cycle, all alert changes are updated.
  • When an alert is manually closed in ServiceNow, it is auto-closed in SCOM. If the alert state is changed to Reopen, SCOM is also updated.
  • When an incident is created and associated to an alert in ServiceNow, SCOM receives the incident number as a ticket ID. However, the state of the incident is not available on SCOM. Therefore when the incident is resolved in ServiceNow, SCOM is not updated as the incident number remains the same. When the alert is associated with a new incident, the new incident number is updated in SCOM.

Vendor Documentation

Connector https://docs.servicenow.com/bundle/utah-it-operations-management/page/product/event-management/task/t_EMConfigureSCOMConnector.html

Limit to SCOM group https://docs.servicenow.com/bundle/washingtondc-it-operations-management/page/product/event-management/task/t_EMAssignRoleSCOMGroup.html

Holman Configure SCOM connector https://kevinholman.com/2021/08/25/what-account-will-command-channel-notifications-run-as-in-scom/

NOT monitored servers

Ever run into NOT monitored servers?
NOT monitored servers
NOT monitored servers
Unsure your experience, but recently, I’ve run across multiple occurrences where servers show up as ‘not monitored.  As a result, I’ve found three distinct sets of issues that might cause ‘not monitored’ status –
1) Orphaned Agent blog
This scenario usually entails deleting server from Managed Agents view in Administration tab, where alerts or other details may still exist.  The procedure leverages Holman’s orphaned agent blog (tried and true) post from years back to aid cleanup.
2) Do you have packs or connectors extending classes?
3) Rebuilding a server with the same name is a common server occurrence
Related to 1, Holman’s orphaned agent blog to be used to cleanup.
First, let’s test in the Lab.  Second, let’s talk about the express lane ‘easy button’.  Begin by deleting the Windows Computer orphaned object GUID.  Process the Windows Computer object (bottom), followed by the top two (2) are HealthService, and HealthServiceWatcher object properties (see three items highlighted).
16db02 properties
16db02 properties
Second piece, marking the Windows Computer GUID for deletion (IsDeleted=1) cleans up nearly ALL properties.  See the progress below, how this slight change makes short order of orphaned properties for ‘server’.
Windows Computer object marks all but SCVMM for deletion
Windows Computer object marks all but SCVMM for deletion
Third HealthService & SCVMM objects require manual deletion per GUID.
Note first screenshot shows health service properties marked ‘IsDeleted’ = 1 after manually processing each GUID.
HealthService marked for deletion
HealthService marked for deletion
Fourth, screenshot shows there the HealthServiceWatcher property is marked for deletion (IsDeleted=1)
HealthServiceWatcher marked for deletion
HealthServiceWatcher marked for deletion
If you have SCVMM, you will need to repeat for each of the SCVMM properties to clear out the orphans in the DB.
Why – the issue:
Typically, when servers are reimaged, i.e. NOT deleted from SCOM, there are two+ healthservice, HealthServiceWatcher, Windows Computer properties created for each image of example server.   Additional properties may show duplicated for any class discoveries that are common to the old and new image.
NOTE: Deleting the current agent may clean up objects for that instance of the discovered server, but NOT the old server image.

File Services Addendum

File Services is all about sharing.
File Services is all about sharing.

‘File Services Addendum’, named Microsoft Windows Server FileServices 2016 Addendum, adds replication health/backlog script, seed and group classes, replication/service monitors, recovery tasks, and overrides to tune monitored environment.

 

Quick Download HTTPS://GITHUB.COM/THEKEVINJUSTIN/FILESERVICESADDENDUM

 

Overview of File Services monitoring

Addendum assumes the file services version agnostic version 10. pack is installed.

Looking at XML file in Notepad++, the pack references are what packs the workflows refer to (other management packs).  Kevin Holman taught building backwards compatibility with MP authoring.  Backwards compatibility allows SCOM2012+ import without errors.  To take this one step further, the v10.0.0.0 file services packs referenced represent the version agnostic packs.

NOTE: File Services Addendum references may need updates if the whole file services management packs are NOT installed.

References screenshot

File Services Addendum references may need updates if the whole file services management packs are NOT installed.

 

Addendum logic

Capabilities

Daily report and close automation, on-demand tasks for reports

DFS backlog script errors

SmSvc, DFSN, DFSR service recovery and rule alerts (from Holman fragments library)

DFS replication backlog watcher, script, alerts

 

Notepad++ screenshot

Addendum rules, tasks, monitors, recoveries.

 

Next, we look at the group/class discoveries

Update the Class/Group discoveries for DFS servers or script install paths for replication script.
Update the Class/Group discoveries for DFS servers or script install paths for replication script.

Update Class/Group discoveries for DFS servers or script install paths for replication script.

Find and replace FilePath and ##DFSServerNamingConvention## variable.

Save file and Import

 

 

Documentation

Kevin Holman MP authoring with fragments https://kevinholman.com/2019/01/17/mp-authoring-with-fragments-introducing-combo-fragments/

Kevin Holman MP fragment library https://github.com/thekevinholman/FragmentLibrary

Addendum GitHub Repository HTTPS://GITHUB.COM/THEKEVINJUSTIN/FILESERVICESADDENDUM

OS Addendum packs

OS Addendum packs for Windows Server from 2012 forward
OS Addendum packs for Windows Server from 2012 forward

Download the ‘OS Addendum packs’ for new capabilities contains Event count logic monitor type, Disk cleanup, Group Policy, self-healing/reset monitors, as well as ‘eventLog full’ logic and reports.  Additional monitors reduce alert noise.  Examples of common alert scenarios are: StorPort storage errors, Group Policy 1096 identification and rebuild.  Disk Cleanup & EventLog service recovery, which includes Event Log file expansion and rollover.

 

Quick DownloadS

https://github.com/theKevinJustin/2012OSAddendum

https://github.com/theKevinJustin/2016ServerAgnostic

 

 

Tune ‘OS Addendum packs’ as needed

Update logical disk paths and retentions.  The default report contains quite a few common checks, including root folders broken out by path, highest to lowest GB’.  The workflow is scalable to add additional application paths, as well as file retention timeframes.  Workflow runs on a weekly basis to cleanup/archive log files, paths.  See Disk cleanup logic blog for more details.

' OS Addendum packs' contains Logical disk breakdown of root folders to list paths were files stored, highest to lowest
‘ OS Addendum packs’ contains Logical disk breakdown of root folders to list paths were files stored, highest to lowest

 

UpdateStorPortCountForRepeatedStorageErrors

StorPort storage errors typically cut lots of alerts with storage reads/writes.  The ‘count’ monitors decrease the alerts, and the daily alert report consolidates the warning alerts (critical by default).  If you’re seeing these alerts, the default should decrease overall alerts to near zero.  Tune as  needed for disk alerts, by updating MatchCount or TimerWait in Seconds (the x events in y time piece of the monitor logic)

Update StorPort Count for Repeated Storage read/write errors
Update StorPort Count for Repeated Storage read/write errors

Save file(s) and import

Disk cleanup logic

Logical disk cleanup, most times is harder vs. smarter manual intervention required, why not smarter vs. harder?
Logical disk cleanup, most times is harder vs. smarter manual intervention required, why not smarter vs. harder?

‘Disk cleanup logic’ traditionally follows manual intervention.  Why would you want harder and manual?  This article will present options to clean up system and non-system disks, by leveraging largest root folder, API’s and more.  This is one step in the OS Addendum pack that needs explanation and can be tailored to applications where admins have regular manual cleanup actions.

 

Breakdown of Disk cleanup

We want to check system disks and non-system disks for different scenarios.  Figure out Disk free space, user profiles, largest folder on root of disk, IIS cleanup, and MECM/SCCM client cache clear API.  Second, utilize different behaviors depending on PowerShell version, application log(s) cleanup, and expand drive alerts when NO space after cleanup action.

Disk Free space

# Check Disk free space
#=====================
if ($Driveletter -eq “C” )
{
$CFreeSpace = gwmi win32_logicaldisk | ? { $_.DeviceID -eq “C:” }
$CFreeSpace.DeviceID
$CFreeSpace
  $DeviceDriveLetter = $CFreeSpace.DeviceID
$DeviceDriveLetter
# Check folder size after cleanup
#==========================
$BeforeSize = (Get-ChildItem “$DeviceDriveLetter” -Recurse | Measure-Object -Property Length -Sum ).sum
$Before = [math]::Round($BeforeSize/1GB,2)
    $DiskFreeSpace = [pscustomobject]@{
DeviceID = $DeviceDriveLetter
Size = [math]::Round($CFreeSpace.Size/1GB,2)
FreeSpace = [math]::Round($CFreeSpace.FreeSpace/1GB,2)
       }
$DiskFreeSpace
$SoftwareDistribution = (gci C:\windows\SoftwareDistribution | measure length -s).sum / 1Mb
# Debug
$SoftwareDistribution

 

Check Software Distribution for ConfigMgr/SCCM/MECM client

Checking software distribution path was an item for discussion where the folder was larger than 3GB, stemming from customer and field engineers  recommendations.

 

If ($DiskFreeSpace.FreeSpace -lt 15 )
{
# Audit Software Distribution
#==================================
If ( $SoftwareDistribution -lt “3000” )
{
Write-host “NO SME/SystemOwner/SysAdmin/Server Action required”
}

If ( $SoftwareDistribution -gt “3000” )
{
Write-host “SME/SystemOwner/SysAdmin/Server Action required, stopping Windows Update service, removing SoftwareDIstribution folder and restarting”
Get-Service -Name wuauserv | Stop-Service
Remove-Item -Path C:\Windows\SoftwareDistribution -Recurse
Get-Service -Name wuauserv | Start-Service
Write-host “Windows Update wuauserv service restarted after SoftwareDistribution directory removed”
}

 

Cleanup Application log folders

The nice part of this is you can reuse this by changing the path and deletion actions to tailor to customer environment.  The script comes in handy for VEEAM, SQL, IIS instances and log directory on multiple drives.

 

# Cleanup IIS log files
#=====================
#if ( Test-Path C:\inetpub\logs\LogFiles\W3SVC1 )
#{
## Years older than
#$HowOld = [DateTime]::Now.AddYears(-1)
#$RecentUse = [DateTime]::Now.AddDays(-90)
## Path to root folder
#$Path = “C:\inetpub\logs\LogFiles\W3SVC1\*.log”
## Deletion task
#get-childitem $Path -Recurse -Depth 1 -EA SilentlyContinue | where { $_.lastAccesstime -lt $RecentUse -and $_.CreationTime -lt $HowOld -and $_.LastWriteTime -lt $RecentUse } | remove-item -force -verbose
#}

Sample report alert output

Sample system disk cleanup report alert
Sample system disk cleanup report alert

Documentation

CleanMgr https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/cleanmgr

Delete client cache the right way https://sccm-zone.com/deleting-the-sccm-cache-the-right-way-3c1de8dc4b48

MECM client cache cleanup PowerShell https://learn.microsoft.com/en-us/powershell/module/configurationmanager/invoke-cmclientaction?view=sccm-ps

SCCM Client Cache cleanup https://rzander.azurewebsites.net/sccm-config-item-to-cleanup-ccmcache/

Stack Overflow disk cleanup https://stackoverflow.com/questions/28852786/automate-process-of-disk-cleanup-cleanmgr-exe-without-user-intervention