January addendum updates

Fast and Furious (sarcasm and humor)
Fast and Furious (sarcasm and humor)

January addendum updates for multiple management packs

First, the biggest change item for large enterprise environments included a change in syntax for get-SCOMAlert
Example
get-scomalert -ResolutionState (0..254) -Name “##stringhere##*”
get-scomalert -ResolutionState 255 -Name “##stringhere##*”
Second, another change with the repo’s was a ‘whitespace audit’ encoded characters, or ‘data concealment’.  See AT&T link CyberSecurity Link
Third, after whitespace we focused on script/workflow efficiencies seen in large enterprise environments.  While Efforts began in December, the workflow efficiencies sprint resulted in two sets of improvements.
Fast and Efficient
Fast and Efficient
1) Added ‘Reset Monitors Script base code’ $Age variable
What does this mean?
Simply put $Age allows admins to define monitor age before resetting.
The default is 1 (day), but can be specified in the script to tailor to requirements.
Example
$Age = [DateTime](Get-Date).AddDays(-1)
2) Beyond incorporating $Age into the reset monitor logic, the packs utilize logic for a much faster runtime (~90%+).
What does this mean?
Updated logic quickly gathers unhealthy monitor objects, by leveraging ‘Get-SCOMManagementPack‘ and then ‘Get-SCOMClass‘, before passing to ‘Get-SCOMClassInstance‘.
Example PowerShell
## Grab the MP, get the Monitors and Rules from the MP, then grab all alerts found inside the Monitors/Rules
$SCOMCoreMP = Get-SCOMManagementPack -DisplayName “Microsoft Windows Server DNS Monitoring”
# Get classes – Examples –
$Monitoring = $SCOMCoreMP
# DNS pack naming
$DNSClasses = @(Get-SCOMClass -ManagementPack $Monitoring; )
$DNSClass = $DNSClasses | sort -property Name -uniq

Repo’s updated in January

January addendum updates include:
ADCS, ADDS, DNS, DFS/File Services, IIS, SCCM pack for MECM/MEM/MCM monitoring, Operating Systems, Proactive NOSC Daily Tasks, and Tangible ProV application monitoring.

Links below to GitHub repositories (repo’s)

NiCE VMware addendum

'NiCE VMware addendum' enhances VMware monitoring, tuning alerts to 'manual intervention' required alerting. 
‘NiCE VMware addendum’ enhances VMware monitoring, tuning alerts to ‘manual intervention’ required alerting.

‘NiCE VMware addendum’ enhances VMware monitoring, tuning alerts to ‘manual intervention’ required alerting. The NiCE folks have been around for some time as a trusted Microsoft partner, creating additional monitoring functionality across Microsoft products.  Having completed a number of projects implementing the VMware pack, it’s time to share the configuration and alert report capabilities.

 

Quick Download HTTPS://GITHUB.COM/THEKEVINJUSTIN/NICEVMWAREADDENDUM/

Changes to Nice vmware pack

Key breakdown of VMware ESX environment monitoring

NiCE VMware monitoring features for ESX, vSphere, vSAN environments
NiCE VMware monitoring features for ESX, vSphere, vSAN environments

 

Adjustments to vendor pack to further the mantra ‘alert when manual intervention required’.

Set monitor alerts to multiple samples over an hour (i.e. compute and performance of ESX environment)

Reports by team (requires regular expression updates for environment servers owned by each team)

Monitor reset logic, and service monitorType (count logic for X failures over Y time, before alert)

Overrides to change vendor pack provided discoveries, rules, monitors

Remove alert noise for unmanaged objects in ESX environment

 

Customize pack for environment

Customize the ‘NiCE VMware addendum’ pack for specific environment. This means updating group discoveries, and GUIDs for group specific overrides.  Further updates are required to update server naming conventions for team virtualization reports.

Classes/groups created for pack

VMware classes included for additional customization.

Discoveries

Breakout of Discoveries that need pattern updates to match

Find/Replace ##ESXHostDataStoreNamingConventions## with names to exclude

Example of regular expressions for multiple customers

VMware Group Seed Classes defined in the addendum.

 

Update disable guest machine alerts

Disable guest machines in ESX environment to disable alerts.

Find ##ESXGuestServersDiskUsageNamingConventions##

Replace with relevant guest naming conventions

 

Example template/guest/virtual machine names typically disabled

Update discovery to disable alerts on object names of virtual machines in ESX environment.

 

Service MonitorType

Service MonitorType adds Samples and Intervals to alert after consecutive failures (x failures in y minutes then alert )

VMware service MonitorType defined in the addendum.

Rules, Monitors, Recoveries

List of workflows used to troubleshoot/resolve problems

VMware addendum rules, VMTools monitor, and recovery components included.

 

 

Documentation

NiCE VMware management pack https://www.nice.de/nice-vmware-mp/

 

MSSQL Addendum pack

 

Time to tune MSSQL alerts!
Time to tune MSSQL alerts!

The ‘MSSQL Addendum pack’ wouldn’t be possible without Brandon Pires contributions.  Brandon dealt with my many questions to better alert!  If you need more background, check the ‘why addendum pack’ post.

Quick Download(s)

2012+ https://github.com/theKevinJustin/MSSQLAddendum

 

Capabilities

The pack is based on the SQL engineering blog and program team making multiple updates per year for SQL monitoring.  The addendum creates two groups for dev/test and notification/subscription modeling.  Second, the overrides, man there are a bunch! aid consumption of real issues.   Lastly, most environments should be SQL 2016+, as the 2012R2 EOL/EOSL is quickly approaching in October!

MSSQL groups defined in the Addendum pack
MSSQL groups defined in the Addendum pack

MSSQL group discoveries require updates to be applicable to environment

 

Tailor addendum

First, the Addendum pack requires the MSSQL packs MUST be installed.  The addendum is based on the MSSQL 2016+ version agnostic is currently supported, as the 2012,2012R2 products are near end of support.

Find/Replace the variables as needed:

Example    ##TESTSERVER##|##DEVSERVER##

Save file

 

Overrides

Addendum pack contains discovery, monitor, and rule overrides to tune MSSQL to CSA (old PFE/CE/CSAe Microsoft Field engineer recommendations), to match the health model reducing critical ‘wake me up in the middle of the night’ alerts.

Partial snapshot of MSSQL overrides in the pack
Partial snapshot of MSSQL overrides in the pack

Import

Download pack, and save to your environment

Import into SCOM

Enjoy!

 

 

MSSQL Addendum references

MSSQL Engineering blog and old post here

SQL Releases TechCommunity here

Engineering team latest management pack, TechCommunity release v7.2.0.0

Import ‘gotcha’ importing new custom functionality blog

Why Addendum packs

IT Ninja required for improving monitoring hence 'Why addendum packs'
IT Ninja required for improving monitoring hence ‘Why addendum packs’

 

‘Why addendum packs’?  What value can they bring to my customer?  Kevin Holman started the Addendum thought process quite a while back.  Added functionality to a core application/program/product.  The first example of this pack naming convention is his SQL RunAs Addendum to simplify SQL monitoring.   Let’s break down a number of examples how the SCOM community has built packs to better monitoring, and how I believe the addendum packs bring IT Ninja lessons from Microsoft experts monitoring to your environment.

 

Why Addendum packs

Better monitoring from the experts, including customer examples for other ‘blind spots’ in monitoring.  Blind spots consist of ‘not monitored’ pieces of infrastructure, from simply an event, ping, service, tcp port check, process, web site, scripted workflow, with the purpose to identify a problem.

The goal of monitoring is to:

Identify, self-heal, automatically run recovery or diagnostic workflows alert when manual intervention is required.  Doesn’t matter what tool you use, they all do some portion of these steps.

 

The addendum packs do these things, adding a few differentiators.

Auto closure daily scripts (close rules/monitors)

Auto reports of problems (M-F 0600-0700 local, reflecting last 24-72 hours of open/closed alerts)

Employ count logic (x in y time)

Self-heal monitors with no new events

Adjust alert severities to health model

where critical (red) = outage, warning (yellow) = issue, informational reports or FYI’s

Capable of updating alerts (status, owner, ticketID+)

Tasks to run workflows on-demand

Recovery tasks – (i.e. service restart automation or TopProcess, Logical disk cleanup, MECM Client cache clean )

Integrate additional monitoring (like DFS replication queue script/alerts)

Synthetic checks for DNS and web applications

Web Availability and Transactional monitoring, ADFS, CRL, PowerShell Invoke-WebRequest, and more

Security and Compliance checks

 

Imagine I forgot something capability wise.

Stay tuned, as this builds into an even better outcome, quality data into ‘a single pane of glass’ of multiple tools within PowerBI.

Detected malicious verification code error

'detected malicious verification code' errors
Detected malicious verification code when verifying element – ever run into this scenario while authoring?

 

 

Ever run into the ‘detected malicious verification code’ error while authoring?  I ran into the malicious verification error authoring, and couldn’t find any content for this error while authoring a pack.

 

Watch your copy/paste’s with additional monitoring changes to prevent ‘detected malicious verification code’ errors

In my authoring example, I received the ‘detected malicious verification code error’ after adding Rules, Datasources, and WriteActions (including tasks).  I was copying and pasting DataSources (DS) and WriteActions (WA), thought I had it all.  Uploaded > got the error, and GRR!   Hopefully this will help others authoring to know what to check to get the management pack uploaded.

 

Simply put – Watch out for typo’s to avoid ‘detected malicious verification code’ errors!

I stumbled across a few websites, but nothing really pointed out to what caused the ‘detected malicious verification code error’ when uploading a management pack.  First, check monitor and rules to verify the DS/WA are called correctly (no errors in file names.  Check the Tasks as well as DisplayStrings, to make sure everything matches.

 

Error Seen when uploading Management pack from SCOM Console GUI regarding ‘detected malicious verification code’ error

<ManagementPackNameHere> Reports could not be imported.

 

If any management packs in the Import list are dependent on this management

pack, the installation of the dependent management packs will fail.

 

Verification failed with 1 errors:

——————————————————-

Error 1:

Found error in

2|<ManagementPackNameHere>|1.0.0.6|<ManagementPackNameHere>|

| with message:

Detected malicious verification code when verifying element of type

Microsoft.EnterpriseManagement.Configuration.ManagementPackRule with inner

exception: System.Collections.Generic.KeyNotFoundException: The given key

was not present in the dictionary.

at System.ThrowHelper.ThrowKeyNotFoundException()

at System.Collections.Generic.Dictionary`2.get_Item(TKey key)

at

Microsoft.EnterpriseManagement.Configuration.ManagementPackRule.VerifyDataTy

pes(Dictionary`2 moduletypes)

at

Microsoft.EnterpriseManagement.Configuration.ManagementPackRule.Verify(Verif

icationContext context)

at

Microsoft.EnterpriseManagement.Configuration.Verification.VerificationEngine

.VerifyCollectionItems(Object context)


 

Additional links

Detected malicious verification code when verifying element

Forum https://social.technet.microsoft.com/Forums/en-US/ac50ae14-882a-4788-a8e4-6a975c498a29/detected-malicious-verification-code-when-verifying-element-of-type

Caution using Tags/Notes extending classes

Caution
Caution

 

Please take ‘caution using Tags/Notes extending classes’.  Please read below if you use Tags/Notes on SCOM classes.  Ran across examples where SCOM Class Properties were used for tags that used the .Notes field on various classes, causing orphaned properties, NOT removed from OperationsManager database.

 

 

Background

The Microsoft.Windows.Computer Class (insert class here) is updated using Tim McFadden’s blog.  This can cause issues with orphaned classes in the database because it is not currently handled as part of the stored procedure (i.e. the Notes property classes do not get marked for deletion).

 

First, identify which classes have Notes property.  Start from Management Server (MS) via PowerShell.   See attached TXT for additional examples to check and add/remove Notes Property on additional windows classes.

 

 

Set Notes property for Windows Operating System server

Second, we need to see how to set and clear the value, in order to clean up the Operations Manager database, to remove the orphaned instances.  The example below sets the value for one (1) server to ‘Production’.

 

$WOS = Get-SCOMClass -name Microsoft.Windows.OperatingSystem | get-SCOMClassInstance | where-object -property Path -eq “16db01.testlab.net”
$WOS.'[System.ConfigItem].Notes’

$WOS.'[System.ConfigItem].Notes’.Value = “Production”

$WOS.Overwrite()

$WOS = Get-SCOMClass -name Microsoft.Windows.OperatingSystem | get-SCOMClassInstance | where-object -property Path -eq “16db01.testlab.net”

$WOS.'[System.ConfigItem].Notes’

 

 

Example Output

PS C:\Users\scomadmin> $WOS.'[System.ConfigItem].Notes’.Value = “Production”
PS C:\Users\scomadmin> $WOS.Overwrite()
PS C:\Users\scomadmin> $WOS = Get-SCOMClass -name Microsoft.Windows.OperatingSystem | get-SCOMClassInstance | where-object -property Path -eq “16db01.testlab.net”
PS C:\Users\scomadmin> $WOS.'[System.ConfigItem].Notes’

PropertyAccessRights : Unknown
Parent : Microsoft Windows Server 2016 Standard
Type : Notes
Value : Production
Id : 00000000-0000-0000-0000-000000000000
ManagementGroup : SCOM2016
ManagementGroupId : e39f5f53-9fbb-9d7f-4bfe-5f0324630ae5

 

 

Set Notes property to NULL

$WOS.'[System.ConfigItem].Notes’.Value = $null
$WOS.Overwrite()

$WOS = Get-SCOMClass -name Microsoft.Windows.OperatingSystem | get-SCOMClassInstance | where-object -property Path -eq “16db01.testlab.net”

Verify Notes value

$WOS = Get-SCOMClass -name Microsoft.Windows.OperatingSystem | get-SCOMClassInstance | where-object -property Path -eq “16db01.testlab.net”
$WOS.'[System.ConfigItem].Notes’

 

 

Example Output
PS C:\Users\scomadmin> $WOS = Get-SCOMClass -name Microsoft.Windows.OperatingSystem | get-SCOMClassInstance | where-object -property Path -eq “16db01.testlab.net”
PS C:\Users\scomadmin> $WOS.'[System.ConfigItem].Notes’

PropertyAccessRights : Unknown
Parent : Microsoft Windows Server 2016 Standard
Type : Notes
Value : (null)
Id : 00000000-0000-0000-0000-000000000000
ManagementGroup : SCOM2016
ManagementGroupId : e39f5f53-9fbb-9d7f-4bfe-5f0324630ae5

 

Have a happy Holiday!

Good luck, hopefully this scenario isn’t something that impacted the monitoring environment!

ADCS – Active Directory Certificate Services Addendum pack

Time to talk Certificates!
Certificate of Achievement

 

Hello again, it’s time to talk about ADCS – Active Directory Certificate Services Addendum!

 

First, I’d like to call out Bob Williams and Vance Cozier for their help and expertise!

SCOM-ADCS-Addendum download

 

 

Background

ADCS is Active Directory Certificate Services, or what we would know as a Certificate Authority.  The goal was to improve the pack, because the focus is on how important certificates are to a modern enterprise.  Let’s begin the Active Directory Certificate Services Addendum pack review.

Collaboration

In this paragraph, let’s talk through the Certificate Services packs for 2016+, and how we as Microsoft consultants, and field engineers, recommend changes to the pack.  First, for some background, the collaboration process gets a better result improving Microsoft products.   Second, the collaboration result can vary.  Third, collaboration input can be based on customer input, or field engineer experience.  Most importantly, this is how we ‘would have liked’ the pack to work.

 

AD Certificate Services Monitoring

The Certificate services pack alerts on events/services.  Therefore, the pack does NOT monitor the SCEP URL.  For instance, a transaction web monitor was added.   The collaboration effort was focused on improving the ADCS pack, resulting in the creation of the Active Directory Certificate Services Addendum and customizations packs.

 

Download File

Let’s delve into the download file

SCOM-ADCS-Addendum download

 

Review file contents

  • Download.txt (in case you need to find it later!)
  • Version.Info.txt (MP version history, what was added & when)
  • XLS MP export of rules/monitors
  • ADCS Addendum & Customizations packs

 

References

Configuring Certificate Services docs site

ADCS download

Management Pack wiki

SQL on Windows Addendum pack

It’s spring time; time to tune the SQL carb!

 

Carbs are way less easy to find these days, but I’ve been busy tuning the SQL agnostic pack (MSSQL on Windows).

 

Tuning the SQL Agnostic pack would be far less successful without expert help.  My thanks to Brandon Pires – MCS SQL Consultant who helped provide a SQL DBA perspective.   Brandon’s LinkedIn profile

 

Always grab an expert, and for SQL, it’s a DBA.  If you’re new to SCOM, most product teams provide their management packs.  SCOM PFE’s build addendum packs to improve a pack (from our perspective).  Addendum packs make the a pack stronger, for an improved customer experience.  I’m not complaining at what the pack delivers.  The SQL Team is awesome for taking user feedback and making improvements quarterly!

 

Background:

Initially this journey started out with Tim McFadden disabling the duplicate rules/monitors in the SQL MP’s (here).

After talking with Tim and Kevin H, I set out to clean up the SQL version specific packs to remove bloat by creating the version specific OFF packs.  The OFF packs disabled the plethora of SQL performance counters (see MP bloat blog here).

With the SQL Agnostic packs (thank God!), I wanted to deliver an addendum pack to tune the SQL alerts/health for what SQL PFE/Consultants recommended for an improved out of the box experience (OoBE).

 

 

MP Version history
v1.0.0.0 24 Feb 2020 Override to enable SQL Monitoring
v1.0.0.1 24 Feb 2020 Override pack cleanup to human readable format
v1.0.0.2  2 Mar 2020 Overrides for severities and SQL CPU samples
v1.0.0.3  2 Mar 2020 Overrides for SQL rules for warning
v1.0.0.4  4 Mar 2020 Completed overrides for SQL warning rules

v1.0.0.5  1 Apr 2020 Updated rules for backup failures when customer uses Netbackup vs. SQL agent/scheduled tasks

v1.0.0.6  9 Apr 2020 Created groups for seed discovery Test/Dev and Prod; excluded EXPRESS, disabled Securables monitor

v1.0.0.7 15 Apr 2020 Updated pack name to include ‘SQL Server’.

Updated AddendumGroupGUIDUpdate to include RegEx pattern replace
AddendumGroupGUIDUpdate will version pack to v1.0.0.7 for group GUID and regex changes

 

 

Please feel free to download the zip file, which includes the XLS for review of what was updated.

My website download

 

 

Additional References

The Agnostic OFF Pack to turn off the performance rules (found here)

The old SQL version specific OFF packs for the performance counters can be found here.

TechNet Gallery download here

 

Updated Skype for Business 2015 Addendum pack

Continuing work with Nick Wood on the Skype pack for additional operational features.

Previously Blogged about this July 2018, and continue to make improvements

The TechNet gallery bundle is updated with new functionality.

Skype KHI addendum

Pack gathers the Skype KHI performance counters

Packets * Discards performance rules where greater than 100 discards are seen on NIC’s,

Monitoring Tab folder/performance view

Skype Custom Overrides

Includes common overrides for noisy monitors/rules.

Install SCVMM management packs from VMM Server

Time for some automation

Ever have to upgrade SCVMM packs every time a new Update Release (UR) comes out?

Copy the files off from the VMM server to your SCOM MS, install.

How long does that take?

Try this script out – assuming you have a login on the VMM Server

TechNet Gallery post here

# Set up some variables

$UR=”UR5″

$VMMServer = “16VMM01”

# Set up your path, this example is monadmin\backup

$date = Get-Date -UFormat “%Y-%m-%d”

# Set up backup path

$backupPath = “C:\monadmin\backup”

$backupDrive = “C:”

# Create some functions

Watch them roll, let PowerShell do your work!

UR6 packs

SCOM management packs backed up

Check out the SCOM Console Admin tab for updates!