addendum Archives - Page 2 of 3 - Kevin Justin's Blog

ADFS Addendum pack

Do you associate StarTrek when the word federation is used inside of federation services (ADFS)?

To begin, the ‘ADFS addendum pack’ needs acknowledgement of the contributors who dealt with my many questions to better alert on AD issues! My thanks to Jason Windisch for his help and expertise with Active Directory Federation Services (ADFS). If you need more background, check the ‘why addendum pack’ post. BTW, what do you associate with the word – Federation?

Quick Download(s)

2016+ https://github.com/theKevinJustin/ADFSAddendum

Overview of capabilities

The Active Directory Federation Services ‘ADFS Addendum pack’ configures ADFS group of related classes for notification/subscription modeling. Second, the rules, service monitors, tasks, service recovery, alert cleanup, and summary reports aid consumption of real issues. Third, if you have ADFS2012R2, I have an addendum pack, but coordination necessary to get the ADFS management packs MSI (not currently available). Lastly, most environments should be 2016+, as the EOL/EOSL is quickly approaching in October!

ADFS Addendum pack creates ADFS Group AND discovery requiring server names applicable to environment.

ADFS Group discovery requires server names applicable to environment

Tailoring the pack(s) to your environment

First, the Active Directory Federation Services management packs MUST be installed for the ‘ADFS Addendum pack’ to load. 2016+ agnostic is currently supported, as the 2012,2012R2 products are near end of support.

Find/Replace the variables as needed

##ADFSSERVERNAME1##|##ADFSSERVERNAME1##|##LAB##

Save file

Workflows

First, the DataSources (DS) and WriteActions (WA) clean up alerts, create daily reports, where the WA are the on-demand tasks versions.

Data source (DS) scheduled workflows run weekdays between 0600-0700 local SCOM management server local time. The summary and team reports (run during this time) summarize key insights. NOTE: the Monday report gathers the last 72 hours, so administrators get a ‘what happened over the weekend’ view. Tuesday-Friday reports are past 24 hours. Lastly, the group policy report summarizing unique GPUpdate error output.

Monitoring

Addendum pack rules schedule data source execution, add on-demand tasks. The service monitor, and Recovery tasks add service recovery automation to bring us to the ‘manual intervention required’ alerting. There are a few monitor/rule overrides to match the health model.

Import

Download updated ‘ADFS addendum pack’ and save to your environment

Import into SCOM

Enjoy!

Documentation

ADFS 2016+ management pack download

Why Addendum packs

‘Why addendum packs’? What value can they bring to my customer? Kevin Holman started the Addendum thought process quite a while back. Added functionality to a core application/program/product. The first example of this pack naming convention is his SQL RunAs Addendum to simplify SQL monitoring. Let’s break down a number of examples how the SCOM community has built packs to better monitoring, and how I believe the addendum packs bring IT Ninja lessons from Microsoft experts monitoring to your environment.

Why Addendum packs

Better monitoring from the experts, including customer examples for other ‘blind spots’ in monitoring. Blind spots consist of ‘not monitored’ pieces of infrastructure, from simply an event, ping, service, tcp port check, process, web site, scripted workflow, with the purpose to identify a problem.

The goal of monitoring is to:

Identify, self-heal, automatically run recovery or diagnostic workflows alert when manual intervention is required. Doesn’t matter what tool you use, they all do some portion of these steps.

The addendum packs do these things, adding a few differentiators.

Auto closure daily scripts (close rules/monitors)

Auto reports of problems (M-F 0600-0700 local, reflecting last 24-72 hours of open/closed alerts)

Employ count logic (x in y time)

Self-heal monitors with no new events

Adjust alert severities to health model

where critical (red) = outage, warning (yellow) = issue, informational reports or FYI’s

Capable of updating alerts (status, owner, ticketID+)

Tasks to run workflows on-demand

Recovery tasks – (i.e. service restart automation or TopProcess, Logical disk cleanup, MECM Client cache clean )

Integrate additional monitoring (like DFS replication queue script/alerts)

Synthetic checks for DNS and web applications

Web Availability and Transactional monitoring, ADFS, CRL, PowerShell Invoke-WebRequest, and more

Security and Compliance checks

Imagine I forgot something capability wise.

Stay tuned, as this builds into an even better outcome, quality data into ‘a single pane of glass’ of multiple tools within PowerBI.

AD Application monitoring

Data from StarTrek the next generation - Mr. Tricorder makes me laugh! — Data from StarTrek the next generation – Mr. Tricorder makes me laugh!

‘AD Application monitoring’ > web synthetics, artificial users > android what image comes to mind? Is it a person, or a thing from a Sci-Fi movie? Perhaps Bishop from Aliens, Data from Star Trek. What does ‘AD Application monitoring’ consist of? Currently that means a CRL validity check, and ADFS web synthetic (proving that ADFS is responding). My thanks to Jason Windisch CSA, for the supplied PowerShell!

Quick Download https://github.com/theKevinJustin/ADApplications/

Tailoring the pack to your environment

The purpose of the pack is to add scheduled workflow that acts like the user, identifies if the CRL’s are about to expire. Most times, monitoring stops at ICMP ping. Most times, there’s still an outage, as the network, and servers are responding. The next layer is IIS, Apache, etc. Sometimes the network team gets involved, checking a base IIS URL is configured. Most outages aren’t network, nor IIS wasn’t running. This is why we focus on the web application responding. Does the multi-prong tactical attack make sense?

This pack delivers on-demand tasks, daily reports, and rules/monitors to reflect health. Customize the watcher node, some URL’s, save, and import into SCOM! The purpose

Assign watcher node(s)

Assign a watcher node by creating a registry key.

What does that mean? Watcher nodes are needed to provide user perspective.

Multiple site example

Issue: Users from sites 1,2,3 are having problems accessing web pages. To understand a user in site 2, leverage a server in site 2 to initiate the web request (invoke-webRequest in PowerShell).

Why: Differentiate user experience (per site). Answer the ‘did you know’ – is the application responding from this site/perspective.

Unfortunately, the watcher node concept eludes most administrators. Mastering ‘user perspective’ makes for an invaluable aid moving from reactive ‘fire fighting’ to proactively being told before users. Hopefully this explains the power where monitoring imitates user interactions for key web applications.

How: Create registry key on whatever servers you want to initiate web monitor

From PowerShell (as Admin), or Command Prompt (as admin)

reg add “HKLM\SOFTWARE\ADApplications\WatcherNode”

AD Applications regedit registry key validation

Example of XML snippet from AD Applications management pack

AD Applications Watcher Node - create specific registry key — AD Applications Watcher Node – create specific registry key

Set up CRL Validity check and ADFS synthetic

Next, configure the URL’s for the customer environment for the ‘AD Application monitoring’ management pack.

Update AD Applications module types for monitor/rules for CRL and ADFS synthetics

Configure the CRL validity check array

From your favorite XML editor (notepad++ pictured)

Find/Replace ##FQDN##, ##CRLstring##, numbers to customer environment

CRL Validity check, create your array length as needed for customer environment

Configure the ADFS synthetic request(s)

From your favorite XML editor (notepad++ pictured)

Find/Replace $server, ##FederationFQDN##, if necessary, update ADFS URL string if different (the /adfs/ls/idpiniatedsignon.aspx portion) to customer environment

Update ADFS URL for invoke-webRequest, ADFS default URL in specified example

Save pack

Import and enjoy!

Documentation

URLGenie for advanced website monitoring

PowerShell invoke-webRequest

Addendum logic blog

Proactive Daily Reports

Proactive Analyst Reports as a new way to ingest key insights from SCOM

As a SME or team lead, ever need to know a key insight for the enclave? Let’s talk about the ‘Proactive Daily Reports’ pack. This provides you some built-in reports on what transpired in an enclave. Building again on the Health pillar, we can simplify what owners need to see. Creating a PowerShell script was a simpler alternative to a complex SSRS report that often broke due to patching, and not following best practices. The pack shows a simpler way to bring key insights to owners for Pending Reboots, Expiring PKI certificates, Logical Disk alerts, System Admin summary, and SCOM admin reports including long-running scripts, script errors, SCOM errors, and alert updates report.

Quick Download: https://github.com/theKevinJustin/ProactiveNOSCDailyTasks

Testing the Proactive Daily reports

Let’s start with some example reports – examples for expiring certificates, Logical Disk, Pending Reboot, System Admin summary, and SCOM admin reports including long-running scripts, script errors, SCOM errors, and alert updates report.

Expiring Certs –

About to expire certificates

Logical disk alerts –

Shows Server, drive, and % full data

Pending Reboots

Alerts of servers pending restart, not patched, not rebooted

Pending reboot report lists servers pending restart, not patched, not rebooted alerts

System Admin summary

This is really a consolidation of multiple insights:

Server performance issues
Open ITSM/Remedy tickets
Unhealthy Agents
Pending Reboot, Not Rebooted, Not patched
Disabled/Unhealthy/MaintenanceMode, Repeatedly down Agents
Logical Disk free space alerts
Expiring certificates
AD DC (ADDS) critical alerts
DNS alerts
Group Policy issues

SysAdmin daily summary report example alert

SCOM admin reports

Admin reports have a few separate alert reports, including long-running scripts, script errors, SCOM errors, and alert updates report.

SCOM Admin alerts report example of common SCOM problems

Long running scripts

ScriptErrors showing key SCOM connectivity issues

SCOM Admin script errors to help diagnose report script syntax errors

Useful links

Other blog posts for addendum management packs and capabilities –

https://kevinjustin.com/blog/2023/08/15/proactive-patching-alerts/
https://kevinjustin.com/blog/2023/08/14/top-process-powershell-script/
https://kevinjustin.com/blog/2023/08/15/proactive-daily-reports/

https://kevinjustin.com/blog/2023/08/08/create-closed-alerts-view/

Proactive Patching alerts

As a SME or team lead, ever need to know ‘Proactive Patching alerts’? i.e. What servers need patches applied, aren’t patching, or were missed? This pack builds on three (3) pillars – Health/Security/Compliance, enabling Cyber teams and more. This became an alternate option to a complex pack, with SSRS report, used by a customer to identify systems. The report was long, and had many blank lines/pages, which required a re-write. This pack started with the pending restart monitor directly from the AquilaWeb reboot pack logic. The logic helps SysAdmin/Domain Admin/NOC/NOSC/SOC teams to know when servers need reboots. This need is driven further due to multiple reboots (sometimes) required with Windows monthly updates, and Application updates. Used across multiple customers, this is the first pack enabling a proactive stance to answer the ‘Am I compliant’ question.

Quick Download: https://github.com/theKevinJustin/ProactivePatchUptimeReboot/

Testing the Proactive Patch alerts

David Allen built the ‘Aquilaweb.Support.PendingReboot.Monitor.PendingReboot’ PowerShell monitor, to tell system owners when the pending restart flag was present. Some builds though, make system changes which repeatedly flip the registry key, causing many alerts. Also, downloading the Aquila pack is a trick, as TechNet was retired.

David provided a great idea, which was built upon. This gave rise to the question of, what if the server was not patched, or not rebooted in a period of time? With my Cyber hat on, this became the next piece of content to create. That gave rise to another question – do these scenarios need to reflect in health (monitor), or not (rule)? We’re all about choices, free will, so the pack is built with those options (rules disabled out of the box).

Pending restart monitor XML showing options

The pack is setup to alert with CBS application updates, SCCM/MECM/Config Mgr Endpoint Management updates, and Windows Updates. This has been my experience for the most accurate reflections of alerts on secure builds where Application/System Owner needs to take action.

Last Patch and Last Reboot monitor/rules in the download, are set to 45 days. Tune this value down, if patching occurs at the 30 day mark, increase if you need more time before alerts.

Last Patch Monitor reflecting number of days

Otherwise, download and import into your environment. Depending on your subscription/notification settings, the Proactive set of alerts are built upon the Windows Operating System class. If subscriptions include the class, the notifications are automatic to System/Application owners.

Useful links

David Allen blog

Addendum, what does it mean blog

ADCS – Active Directory Certificate Services Addendum pack

Time to talk Certificates! — Certificate of Achievement

Hello again, it’s time to talk about ADCS – Active Directory Certificate Services Addendum!

First, I’d like to call out Bob Williams and Vance Cozier for their help and expertise!

SCOM-ADCS-Addendum download

Background

ADCS is Active Directory Certificate Services, or what we would know as a Certificate Authority. The goal was to improve the pack, because the focus is on how important certificates are to a modern enterprise. Let’s begin the Active Directory Certificate Services Addendum pack review.

Collaboration

In this paragraph, let’s talk through the Certificate Services packs for 2016+, and how we as Microsoft consultants, and field engineers, recommend changes to the pack. First, for some background, the collaboration process gets a better result improving Microsoft products. Second, the collaboration result can vary. Third, collaboration input can be based on customer input, or field engineer experience. Most importantly, this is how we ‘would have liked’ the pack to work.

AD Certificate Services Monitoring

The Certificate services pack alerts on events/services. Therefore, the pack does NOT monitor the SCEP URL. For instance, a transaction web monitor was added. The collaboration effort was focused on improving the ADCS pack, resulting in the creation of the Active Directory Certificate Services Addendum and customizations packs.

Download File

Let’s delve into the download file

SCOM-ADCS-Addendum download

Review file contents

Download.txt (in case you need to find it later!)
Version.Info.txt (MP version history, what was added & when)
XLS MP export of rules/monitors
ADCS Addendum & Customizations packs

References

Configuring Certificate Services docs site

ADCS download

Management Pack wiki

SQL on Windows Addendum pack

It’s spring time; time to tune the SQL carb!

Carbs are way less easy to find these days, but I’ve been busy tuning the SQL agnostic pack (MSSQL on Windows).

Tuning the SQL Agnostic pack would be far less successful without expert help. My thanks to Brandon Pires – MCS SQL Consultant who helped provide a SQL DBA perspective. Brandon’s LinkedIn profile

Always grab an expert, and for SQL, it’s a DBA. If you’re new to SCOM, most product teams provide their management packs. SCOM PFE’s build addendum packs to improve a pack (from our perspective). Addendum packs make the a pack stronger, for an improved customer experience. I’m not complaining at what the pack delivers. The SQL Team is awesome for taking user feedback and making improvements quarterly!

Background:

Initially this journey started out with Tim McFadden disabling the duplicate rules/monitors in the SQL MP’s (here).

After talking with Tim and Kevin H, I set out to clean up the SQL version specific packs to remove bloat by creating the version specific OFF packs. The OFF packs disabled the plethora of SQL performance counters (see MP bloat blog here).

With the SQL Agnostic packs (thank God!), I wanted to deliver an addendum pack to tune the SQL alerts/health for what SQL PFE/Consultants recommended for an improved out of the box experience (OoBE).

MP Version history
v1.0.0.0 24 Feb 2020 Override to enable SQL Monitoring
v1.0.0.1 24 Feb 2020 Override pack cleanup to human readable format
v1.0.0.2 2 Mar 2020 Overrides for severities and SQL CPU samples
v1.0.0.3 2 Mar 2020 Overrides for SQL rules for warning
v1.0.0.4 4 Mar 2020 Completed overrides for SQL warning rules

v1.0.0.5 1 Apr 2020 Updated rules for backup failures when customer uses Netbackup vs. SQL agent/scheduled tasks

v1.0.0.6 9 Apr 2020 Created groups for seed discovery Test/Dev and Prod; excluded EXPRESS, disabled Securables monitor

v1.0.0.7 15 Apr 2020 Updated pack name to include ‘SQL Server’.

Updated AddendumGroupGUIDUpdate to include RegEx pattern replace
AddendumGroupGUIDUpdate will version pack to v1.0.0.7 for group GUID and regex changes

Please feel free to download the zip file, which includes the XLS for review of what was updated.

My website download

Additional References

The Agnostic OFF Pack to turn off the performance rules (found here)

The old SQL version specific OFF packs for the performance counters can be found here.

TechNet Gallery download here

Workflow Manager Addendum MP for SQL Aliases

A SQL Alias is kinda like wearing disguise glasses…

From a security perspective, you can make things difficult for attackers by specifying a SQL alias and different port for SQL.

Symptom – discovery fails for WFM pack

Trying to monitor and figure out what the real database name, instance, etc. can be a challenge.

A couple of years ago, I was able to find an example for one customer where the registry key shed light on the alias.

The workflow manager management pack has a DataSourceModuleType “Microsoft.WorkflowManager.Addendum.v1.WFCommandExecuterDataSource”, where this change successfully retrieved the sql server name.

This datasource uses the PowerShell script (WorkflowPSDiscovery.ps1)

This function was changed in one example

# Get computer name from splitted dataSource
function GetPrincipalName {
param(
$ADDomain,
$ss
)

#$ssWithoutPort = $ss[0].split(‘,’)
#if (-not $ssWithoutPort[0].Contains(‘.’))
#{
# $ssWithoutPort[0] = $ssWithoutPort[0] + “.” + $ADDomain.Name
#}
#$principalName = $ssWithoutPort[0]

$key = ‘HKLM:\SOFTWARE\Microsoft\MSSQLServer\Client\ConnectTo’
$sqlfromalias = (Get-ItemProperty -Path $key -Name $ss).$ss
$sqlserverstr = $sqlfromalias.Split(‘,’)
$sqlserver = $sqlserverstr[1]
$principalName = $sqlserver

return $principalName
}

Ran into this discovery issue a second time, and the function didn’t solve the failure.

Real quick – a shout out and my thanks to Chuck Hughes and Mike Sadoff, for their time and testing this more robust discovery method.

Added logic to fix the assumed InstanceName ($instname) – Most likely why my first function worked (configuration had default SQL instance name of MSSQLSERVER )

Added GetSqlAlias function to help decode the disguise

Gallery download here

Don’t forget to override the original workflow manager discovery!

Microsoft.WorkflowManager.v1.Addendum.WFPSDiscovery

Updated Skype for Business 2015 (premise) Addendum MP

Updated and completed for Company Knowledge!

Many thanks to Nick Wood for his help deciphering user impact for these alerts.

Reach out to Nick on LinkedIn

This has been an arduous effort to complete the Skype alerts and components.

Gallery Download

It’s taken a few steps along the way, to get all the content delivered.

To read the whole Skype Addendum journey, please read these additional blog posts
June blog
January blog

The initial Addendum pack with just service recoveries and Azure overrides

Old pack https://gallery.technet.microsoft.com/Skype-for-Business-2015-b005f49f
This download disabled Azure, set service recovery tasks

The new Gallery download contains the following:

Skype SCOM Alerts.xls
Microsoft.LS.2015.Monitoring.ComponentAndUser.Addendum.xml
Microsoft.SystemCenter.Notifications.Internal.xml
Skype.for.Business.Server.Management.Pack.Alert.Grooming.xml

NOTE The Skype.for.Business.Server.Management.Pack.Install.txt file contains the information as well

#############################################################
#
# Breakdown of files
#
#############################################################

#
# Skype SCOM Alerts.xls
# Skype SCOM Alerts XLS is is MP Export excerpt formatted for XLS workbook
#
# Feel free to search this file

# Column D is ‘Escalate to Who’
# This has values as SCOM Engineering, Messaging Ops, Telephony, Messaging Engineering
# Column E is impact
# This has values: P3-P5, *Email
# Column F is the Display String for the monitor
# Column H has the User Impact, Cause, and Troubleshooting steps

#
# Microsoft.LS.2015.Monitoring.ComponentAndUser.Addendum.xml
# Addendum management pack sets up company knowledge tab for each Skype monitor, with actionable troubleshooting steps.
#

#
# Microsoft.SystemCenter.Notifications.Internal.xml
#

# Backup the current MP first, and merge if you are adding this to your environment!
# Use this pack cautiously, as it will replace existing Channels, subscribers, subscriptions.

# On Management server, open PowerShell window as Admin
#
# cd <path>
# Example

cd $HOME/desktop
Get-SCOMManagementPack -Name *Notification* | export-SCOMmanagementpack -path “C:\”

Copy-item .\Microsoft.SystemCenter.Notifications.Internal.xml .\Original-<CompanyName>-Microsoft.SystemCenter.Notifications.Internal.xml

# Save the bundled Notifications pack to the same path

# YES it’s that important, the file can eliminate any alerts leaving SCOM!
#
# Save file to local drive

# Follow MP Fragment authoring if you need to merge existing Notifications with Skype pack
# https://kevinholman.com/2016/06/04/authoring-management-packs-the-fast-and-easy-way-using-visual-studio/
#

#
# Skype.for.Business.Server.Management.Pack.Alert.Grooming.xml
#
# This file is to set alerts per the XLS, making warnings for P4,P5, email, and P3 for Critical

#
#############################################################

Skype for Business 2015 (premise) Addendum MP

Ever try to figure out a Skype alert, for which server in the pool(s) is failing?

While maybe not the clearest to find root cause, the Skype pack brings a bunch of functionality, including synthetic transactions.

I was lucky enough to collaborate with Nick Wood, Skype PFE, to help provide more detail, troubleshooting, impact on what is critical versus warning.

What the addendum pack brings

Do you think 656 monitors can all be critical?

Sets up service restart recovery tasks for all Skype services

Company Knowledge tab for troubleshooting/user impact

Gallery Download

Here is a visual of our Skype efforts for integrating troubleshooting details into SCOM console.

NOTE: Company Knowledge tab would be accessible from the alert as well

Company Knowledge

SCOM Console, Authoring tab, Dispatcher Queue monitor

Highlight monitor, right click, choose properties

Click on ‘Company Knowledge’ tab

Incorporated the XLS into SCOM under Company Knowledge for additional information on user impact, causes, and troubleshooting (under resolutions)