January addendum updates

Fast and Furious (sarcasm and humor)
Fast and Furious (sarcasm and humor)

January addendum updates for multiple management packs

First, the biggest change item for large enterprise environments included a change in syntax for get-SCOMAlert
Example
get-scomalert -ResolutionState (0..254) -Name “##stringhere##*”
get-scomalert -ResolutionState 255 -Name “##stringhere##*”
Second, another change with the repo’s was a ‘whitespace audit’ encoded characters, or ‘data concealment’.  See AT&T link CyberSecurity Link
Third, after whitespace we focused on script/workflow efficiencies seen in large enterprise environments.  While Efforts began in December, the workflow efficiencies sprint resulted in two sets of improvements.
Fast and Efficient
Fast and Efficient
1) Added ‘Reset Monitors Script base code’ $Age variable
What does this mean?
Simply put $Age allows admins to define monitor age before resetting.
The default is 1 (day), but can be specified in the script to tailor to requirements.
Example
$Age = [DateTime](Get-Date).AddDays(-1)
2) Beyond incorporating $Age into the reset monitor logic, the packs utilize logic for a much faster runtime (~90%+).
What does this mean?
Updated logic quickly gathers unhealthy monitor objects, by leveraging ‘Get-SCOMManagementPack‘ and then ‘Get-SCOMClass‘, before passing to ‘Get-SCOMClassInstance‘.
Example PowerShell
## Grab the MP, get the Monitors and Rules from the MP, then grab all alerts found inside the Monitors/Rules
$SCOMCoreMP = Get-SCOMManagementPack -DisplayName “Microsoft Windows Server DNS Monitoring”
# Get classes – Examples –
$Monitoring = $SCOMCoreMP
# DNS pack naming
$DNSClasses = @(Get-SCOMClass -ManagementPack $Monitoring; )
$DNSClass = $DNSClasses | sort -property Name -uniq

Repo’s updated in January

January addendum updates include:
ADCS, ADDS, DNS, DFS/File Services, IIS, SCCM pack for MECM/MEM/MCM monitoring, Operating Systems, Proactive NOSC Daily Tasks, and Tangible ProV application monitoring.

Links below to GitHub repositories (repo’s)

Hear ye hear ye

Hear ye hear ye - see the nice warm toasty updated packs
Hear ye hear ye – see the nice warm toasty updated packs

See the nice warm toasty updated packs

Fresh off the press, right to your door, just in time for that gift for your special someone!  Time for new updates to keep you ever-green’d, up to date, fixes, etc.  ;-P

 

Holman updated his SCOM.Management pack for SCOM2022 UR2

Github https://github.com/thekevinholman/SCOM.Management

 

Addendum packs updated

Multiple packs with multiple updates.  Removed debug detail for DS/WA (Data Source/Write Action workflows) Health Explorer outputs, simplified mgmt pack recovery tasks for single WA script.

Active Directory Certificate Services (ADCS) version agnostic 2016+ addendum https://github.com/theKevinJustin/ADCS2016-Addendum 2012 here  See the blog post for capabilities here https://kevinjustin.com/blog/2023/08/18/adcs-addendum-packs/

Active Directory Domain Services addendum https://github.com/theKevinJustin/ADDSAddendumAgnostic

See the blog post for capabilities here https://kevinjustin.com/blog/2023/08/18/adds-addendum-pack/

Active Directory Federation Services addendum https://github.com/theKevinJustin/ADFSAddendum See the blog post for capabilities here https://kevinjustin.com/blog/2023/08/18/adfs-addendum-pack/

FileServices Agnostic addendum https://github.com/theKevinJustin/FileServicesAddendum See the blog post for capabilities here https://kevinjustin.com/blog/2023/08/31/file-services-addendum/

MCM/MEM/MECM/SCCM Configuration Manager addendum https://github.com/theKevinJustin/MECMSCCMAddendum See the blog post for capabilities here https://kevinjustin.com/blog/2023/08/30/mecm-sccm-addendum-pack/

PKI certificate monitoring addendum https://github.com/theKevinJustin/PKIAddendum See the blog post for capabilities here https://kevinjustin.com/blog/2023/08/24/pki-addendum-pack/

Proactive NOSC DailyTasks reports addendum https://github.com/theKevinJustin/ProactiveNOSCDailyTasks See the blog post for capabilities here https://kevinjustin.com/blog/2023/08/15/proactive-daily-reports/

SCOM Core addendum https://github.com/theKevinJustin/SCOMCoreAddendum See the blog post for capabilities here https://kevinjustin.com/blog/2023/08/30/scomcore-addendum-pack/

Top Process workflows tied to monitors in Tier1 https://github.com/theKevinJustin/TopProcessTier1 See the blog post for capabilities here https://kevinjustin.com/blog/2023/08/15/proactive-daily-reports/

Tier0 https://github.com/theKevinJustin/TopProcess

Windows Server 2012/2012R2 Operating System Addendum https://github.com/theKevinJustin/2012OSAddendum See the blog post for capabilities here https://kevinjustin.com/blog/2023/08/28/os-addendum-packs/

Windows Server 2016+ version agnostic Operating System Addendum https://github.com/theKevinJustin/2016ServerAgnostic See the blog post for capabilities here https://kevinjustin.com/blog/2023/08/28/os-addendum-packs/

 

Enjoy!

Updated DNS2012R2 Addendum

DNS2012R2 addendum pack updated!

Updated DNS2012R2 Addendum overrides.  Learned a few new things with Overrides workspace views, and why Authoring pane > Management pack Objects > Overrides may not load.

 

 

When your management pack has improper overrides, expect the loading icon.  This may be caused due to overrides, whether error is with target, class/rule/monitor.

Console Overrides Loading
Console Overrides Loading

 

Sometimes, an Object of class error gets your hopes up (pointing at a non-existent object).

Object of Class error
Object of Class error

 

Example when Overrides loads properly

When Authoring Tab Overrides view loads successfully.
When Authoring Tab Overrides view loads successfully.

 

 

If Overrides view will not load, try creating a workspace view for Overrides.

Navigation Steps:

From SCOM Console

Click on My Workspace

Right Click  > New > Overrides Summary View

Create Workspace Overrides View
Create Workspace Overrides View

 

Select checkbox ‘with a specific override management pack’ checkbox, then the ‘specific’ link to choose management pack(s).

Select Specific Override management pack(s)
Select Specific Override management pack(s)

 

Choose unsealed management pack(s) with overrides

Can select all – OR pick a few to see what loads without errors

Click OK

If you get the loading screen and error, now begins the pack analysis.

OverridesViewFailsToLoad
OverridesViewFailsToLoad

Clicking on the ‘Show’ link points to a non-existent object

Microsoft.EnterpriseManagement.Common.ObjectNotFoundException: An object of class ManagementPackClass with ID 76e2559c-aaf4-b1ec-60cf-d40ab4102fbc was not found.

 

How did I know that?

Run get-SCOMClassInstance command from PowerShell or Operations Manager shell

Example output of ‘get-SCOMClassInstance -ID “76e2559c-aaf4-b1ec-60cf-d40ab4102fbc” ‘

Get-SCOMClassInstance output of the GUID listed in the console error.
Get-SCOMClassInstance output of the GUID listed in the console error.

 

Work on the Overrides of the affected XML packs, and Import.

Once corrected, the Workspace view loads successfully, finite!

Overrides Workspace view of addendum packs

 

 

Documentation

My Workspace https://learn.microsoft.com/en-us/system-center/scom/manage-web-console-my-workspace?view=sc-om-2022

Monitoring workspace https://learn.microsoft.com/en-us/system-center/scom/manage-using-monitoring-workspace?view=sc-om-2022

NiCE VMware addendum

'NiCE VMware addendum' enhances VMware monitoring, tuning alerts to 'manual intervention' required alerting. 
‘NiCE VMware addendum’ enhances VMware monitoring, tuning alerts to ‘manual intervention’ required alerting.

‘NiCE VMware addendum’ enhances VMware monitoring, tuning alerts to ‘manual intervention’ required alerting. The NiCE folks have been around for some time as a trusted Microsoft partner, creating additional monitoring functionality across Microsoft products.  Having completed a number of projects implementing the VMware pack, it’s time to share the configuration and alert report capabilities.

 

Quick Download HTTPS://GITHUB.COM/THEKEVINJUSTIN/NICEVMWAREADDENDUM/

Changes to Nice vmware pack

Key breakdown of VMware ESX environment monitoring

NiCE VMware monitoring features for ESX, vSphere, vSAN environments
NiCE VMware monitoring features for ESX, vSphere, vSAN environments

 

Adjustments to vendor pack to further the mantra ‘alert when manual intervention required’.

Set monitor alerts to multiple samples over an hour (i.e. compute and performance of ESX environment)

Reports by team (requires regular expression updates for environment servers owned by each team)

Monitor reset logic, and service monitorType (count logic for X failures over Y time, before alert)

Overrides to change vendor pack provided discoveries, rules, monitors

Remove alert noise for unmanaged objects in ESX environment

 

Customize pack for environment

Customize the ‘NiCE VMware addendum’ pack for specific environment. This means updating group discoveries, and GUIDs for group specific overrides.  Further updates are required to update server naming conventions for team virtualization reports.

Classes/groups created for pack

VMware classes included for additional customization.

Discoveries

Breakout of Discoveries that need pattern updates to match

Find/Replace ##ESXHostDataStoreNamingConventions## with names to exclude

Example of regular expressions for multiple customers

VMware Group Seed Classes defined in the addendum.

 

Update disable guest machine alerts

Disable guest machines in ESX environment to disable alerts.

Find ##ESXGuestServersDiskUsageNamingConventions##

Replace with relevant guest naming conventions

 

Example template/guest/virtual machine names typically disabled

Update discovery to disable alerts on object names of virtual machines in ESX environment.

 

Service MonitorType

Service MonitorType adds Samples and Intervals to alert after consecutive failures (x failures in y minutes then alert )

VMware service MonitorType defined in the addendum.

Rules, Monitors, Recoveries

List of workflows used to troubleshoot/resolve problems

VMware addendum rules, VMTools monitor, and recovery components included.

 

 

Documentation

NiCE VMware management pack https://www.nice.de/nice-vmware-mp/

 

IIS addendum packs

IIS addendum packs to tune IIS from 2012 forward.

IIS addendum packs to tune IIS from 2012 forward.’IIS addendum packs’ to tune IIS from 2012 forward.  The GitHub repository has two packs 2012/2016+ (version agnostic pack).  This includes an IIS enabled group, Daily report and cleanup DataSource and WriteAction (tasks), as well as a regular expression to set up the IIS enabled group.  The IIS enabled group is to enable IIS monitoring on servers IIS monitoring is needed.

 

 

Customize for environment

Update addendums to server naming conventions for enabled IIS monitoring.  Read below to better understand addendum functionality.

First, the addendums include class/group, datasource and write action alert reports and automated alert closure workflows, as well as event count logic/reset monitorType.

Addendum includes class/group, datasource and write action workflows for alert reports and automated alert closure, as well as event count logic reset monitorType.

 

Second, the group discovery, find/replace the pattern to various application/web server naming conventions where IIS monitoring IS wanted.

Third, the version agnostic has overrides to disable most perf and rule alerts.  Can provide OFF packs to turn off performance counter collection rules, to keep both the OperationsManager, and OperationsManagerDW databases cleaner, thereby faster with less data.

IIS2012 overrides
IIS2012 overrides

Lastly, once addendum updated, save file, move to SCOM MS, and import.

Enjoy the ‘IIS addendum packs’ for how few alerts, perhaps life changing?! (sarcasm)

 

 

Documentation

Download Addendum packs https://github.com/theKevinJustin/IISAddendums

IIS2012 SCOM Management pack download https://www.microsoft.com/en-us/download/details.aspx?id=34767

IIS2016+ SCOM management pack download https://www.microsoft.com/en-us/download/details.aspx?id=54445

SCOMCore Addendum pack

SCOMCore Addendum pack - having a strong core makes bigger gains
SCOMCore Addendum pack – having a strong core makes bigger gains

Time to configure the Microsoft System Center Core Monitoring pack per health model and best practice.  That’s where the SCOMCore Addendum pack comes in. Addendum adds High Agent Handle count group, daily report and alert closure automation, and rule/monitor overrides.  Some assembly required – update the discovery pattern for offending high handle counts, and high handle count group ContextInstance GUID after import.

 

Quick Download: https://github.com/theKevinJustin/SCOMCoreAddendum

 

 

Background:

While High Agent  Handle count was more an issue before the x365 platform migrated UC, SharePoint, and email (i.e. Lync/Skype, SharePoint, Exchange on prem) went to the cloud.  This is still seen where cloud scalability options and virtualization/storage limitations exist.  Example typically is an over-utilized virtual machine in hybrid/IaaS/premise scenarios.  Kevin Holman caught this performance issue years back, creating monitoring alerts pack and blog.  In case you’re on SCOM jeopardy, the LAW/OMS/Microsoft Monitoring Agent/SCOM agent has a built-in health check.  The built-in health check restarts service when Handle Count or memory of the HealthService (aka Microsoft Monitoring Agent service) ran too hot per SCOM PG.   SCOM agent restarts caused config churn, and high compute, as workflows re-ran after the service restarted.

 

 

Assess agent restarts

Begin by verifying if you have Kevin Holman’s pack for SCOM agent restarts  downloaded and installed, which sets memory/handle count informational alerts https://github.com/thekevinholman/SCOM.AgentThresholds

Validate pack installed

Verify SCOM Agent Thresholds pack installed.
Verify SCOM Agent Thresholds pack installed.

 

 

Configure addendum for environment

Download and Install ‘SCOMCore Addendum pack’ here

Open saved XML in notepad or Notepad++ (your favorite XML editor here!)

Update the regular expression pattern line for offending servers in the

Update the pattern for the high agent handle count group for any offenders.

 

Figure out the group GUID for the high agent handle count

From PowerShell on SCOM management server, run:

Get-SCOMClassInstance -DisplayName “Proactive High Agent Handle Count servers” | fl DisplayName,ID

 

Find/Replace GUID

PowerShell GUID check.

 

Save file and Import > enjoy less alerts!

 

 

Documentation:

Kevin Holman blog on SCOM agent restarts

Holman’s pack for SCOM agent restarts and setting memory/handle count alerts https://github.com/thekevinholman/SCOM.AgentThresholds

Addendum download https://github.com/theKevinJustin/SCOMCoreAddendum

DHCP Addendum pack

Leverage DHCP addendum to tune DHCP subnet monitoring.
Leverage DHCP addendum to tune DHCP subnet monitoring.

Leverage the ‘DHCP Addendum pack’.  Why?  DHCP manages IP ranges, particularly customer facing issues like VPN connectivity, VDI/AVD/appliance devices, as well as client workstations/laptops/GFE’s.  The DHCP management pack alerts when a subnet is nearing zero available IP’s before you have an outage.  This article will help you understand how the addendum’s new capabilities tune DNS monitoring to best practice.

 

QUICK DOWNLOAD(S)

2016+ HTTPS://GITHUB.COM/THEKEVINJUSTIN/DCHPAGNOSTIC

 

What capabilities does the ‘DHCP Addendum pack’ provide?

Two groups, one DHCP server group, and DHCP subscription group to configure notifications to SME for DHCP related classes

Overrides for common alerts, disable event collection rules

 

 

Utilize the DHCP Addendum

Download the DHCP Addendum on GitHub, to get alerts where manual intervention required.

 

Update XML

The pack greatly decreases alerts, and the XML authoring is an easy feat.  After you import the pack, find/replace is required for two pieces.

  • Discovery group regular expressions (RegEx)

##DHCPServerRegEx##

Find ##DHCPServerRegEx## and replace with your DNS server expressions.

Example server names: 12dc01, 19dc01,19dc02,19dc03, etc.

RegEx = (?i)12dc0|19dc0

Using PowerShell on SCOM management server (MS) to determine group GUIDs for replace in Overrides/Discoveries.
Using PowerShell on SCOM management server (MS) to determine group GUIDs for replace in Overrides/Discoveries.

 

  • Update group GUIDs, after installing this pack.

Find/replace the GUIDs, as they are unique to every SCOM management group, hard coding the group ID GUID is not possible.  We will be running Get-SCOMClassInstance to determine the group GUID’s applicable in the management group.

 

From PowerShell, on your SCOM management server, run get-SCOMClassInstance commands for the two groups added.

get-scomclassinstance -DisplayName “Microsoft Windows DHCP 2016+ Servers” | ft Id

get-scomclassinstance -DisplayName “Microsoft Windows DHCP 2016+ subscription components” | ft Id

 

Example

Leveraging Notepad++ to find/replace the group GUID with SCOM environment specific GUIDs

Find/Replace the GUID in the pack with the ID from the output above.

Example leveraging Notepad++ to find/replace the group GUID with SCOM environment specific GUIDs

Save pack

Import into SCOM & Enjoy!

OS Addendum packs

OS Addendum packs for Windows Server from 2012 forward
OS Addendum packs for Windows Server from 2012 forward

Download the ‘OS Addendum packs’ for new capabilities contains Event count logic monitor type, Disk cleanup, Group Policy, self-healing/reset monitors, as well as ‘eventLog full’ logic and reports.  Additional monitors reduce alert noise.  Examples of common alert scenarios are: StorPort storage errors, Group Policy 1096 identification and rebuild.  Disk Cleanup & EventLog service recovery, which includes Event Log file expansion and rollover.

 

Quick DownloadS

https://github.com/theKevinJustin/2012OSAddendum

https://github.com/theKevinJustin/2016ServerAgnostic

 

 

Tune ‘OS Addendum packs’ as needed

Update logical disk paths and retentions.  The default report contains quite a few common checks, including root folders broken out by path, highest to lowest GB’.  The workflow is scalable to add additional application paths, as well as file retention timeframes.  Workflow runs on a weekly basis to cleanup/archive log files, paths.  See Disk cleanup logic blog for more details.

' OS Addendum packs' contains Logical disk breakdown of root folders to list paths were files stored, highest to lowest
‘ OS Addendum packs’ contains Logical disk breakdown of root folders to list paths were files stored, highest to lowest

 

UpdateStorPortCountForRepeatedStorageErrors

StorPort storage errors typically cut lots of alerts with storage reads/writes.  The ‘count’ monitors decrease the alerts, and the daily alert report consolidates the warning alerts (critical by default).  If you’re seeing these alerts, the default should decrease overall alerts to near zero.  Tune as  needed for disk alerts, by updating MatchCount or TimerWait in Seconds (the x events in y time piece of the monitor logic)

Update StorPort Count for Repeated Storage read/write errors
Update StorPort Count for Repeated Storage read/write errors

Save file(s) and import

DNS2012R2 Addendum pack

Still running Server2012R2 servers with AD DCs with AD integrated DNS?
Still running Server2012R2 servers with AD DCs with AD integrated DNS?

In case you’re still running Windows Server 2012R2, here’s the ‘DNS2012R2 Addendum pack’ giving the same functionality as the version agnostic 2016+ addendum.  Why?  DNS is a translation method to convert names to IP’s.  Can you imagine if we wanted to connect to google via IP?  The number of workflows in the SCOM DNS pack (built by the DNS Product Group) makes for an astounding number of workflows running on your DC every minute.  Forward and reverse lookups are a good check, verifying DNS is functioning.  In a complex environment with 100’s of zones, SCOM becomes a utilization culprit for a DC’s primary missions – authenticate and resolve.  This article will help you understand how the pack will add new capabilities and tune DNS monitoring to best practice.

 

Quick Download HTTPS://GITHUB.COM/THEKEVINJUSTIN/DNSADDENDUM2012R2/

 

 

What capabilities does the ‘DNS Addendum pack’ provide?

Count logic monitors (i.e. x events in y time, and self heal)

Daily summary report of DNS alerts broken out

Daily alert closure workflow to close out DNS rules/monitor

DNS service(s) recovery automation

Synthetic internal/external nslookup monitor (scoped to PDC emulators versus ALL DNS servers

WMI validation alert recovery to prevent false positive alerts with weird one off scenarios – one example: Security tools randomly block WMI access.

 

Download the ‘DNS2012R2 Addendum pack’ on GitHub to improve AD Integrated (ADI) DNS monitoring on Windows Server 2016+ (version agnostic).

Save and Import pack, then update XML for group GUIDs

 

 

Update XML

First, update XML with the GUIDs from your management group.  Second, map the group DisplayName to find/replace the GUID for each group.

Get-SCOMClassInstance output for DNS2012R2 groups
Get-SCOMClassInstance output for DNS2012R2 groups

 

Third, using Notepad++ highlight the ContextInstance GUID and hit Control-H, and paste the group GUID then click Replace All.

Using Notepad++ highlight the ContextInstance GUID and hit Control-H, and paste the group GUID then click Replace All.
Using Notepad++ highlight the ContextInstance GUID and hit Control-H, and paste the group GUID then click Replace All.

Fourth – Rinse and repeat for the other three groups.

Lastly, save file, move to SCOM MS, and import!

 

Documentation and links

DNS Pack download

DNS2012R2 addendum blog including updates

GitHub Repository https://github.com/theKevinJustin/DNSAddendum2012R2/

 

MSSQL Addendum pack

 

Time to tune MSSQL alerts!
Time to tune MSSQL alerts!

The ‘MSSQL Addendum pack’ wouldn’t be possible without Brandon Pires contributions.  Brandon dealt with my many questions to better alert!  If you need more background, check the ‘why addendum pack’ post.

Quick Download(s)

2012+ https://github.com/theKevinJustin/MSSQLAddendum

 

Capabilities

The pack is based on the SQL engineering blog and program team making multiple updates per year for SQL monitoring.  The addendum creates two groups for dev/test and notification/subscription modeling.  Second, the overrides, man there are a bunch! aid consumption of real issues.   Lastly, most environments should be SQL 2016+, as the 2012R2 EOL/EOSL is quickly approaching in October!

MSSQL groups defined in the Addendum pack
MSSQL groups defined in the Addendum pack

MSSQL group discoveries require updates to be applicable to environment

 

Tailor addendum

First, the Addendum pack requires the MSSQL packs MUST be installed.  The addendum is based on the MSSQL 2016+ version agnostic is currently supported, as the 2012,2012R2 products are near end of support.

Find/Replace the variables as needed:

Example    ##TESTSERVER##|##DEVSERVER##

Save file

 

Overrides

Addendum pack contains discovery, monitor, and rule overrides to tune MSSQL to CSA (old PFE/CE/CSAe Microsoft Field engineer recommendations), to match the health model reducing critical ‘wake me up in the middle of the night’ alerts.

Partial snapshot of MSSQL overrides in the pack
Partial snapshot of MSSQL overrides in the pack

Import

Download pack, and save to your environment

Import into SCOM

Enjoy!

 

 

MSSQL Addendum references

MSSQL Engineering blog and old post here

SQL Releases TechCommunity here

Engineering team latest management pack, TechCommunity release v7.2.0.0

Import ‘gotcha’ importing new custom functionality blog