MECM/SCCM Addendum pack

The 'MECM/SCCM Addendum pack' encompasses Endpoint Management which as of late, has taken on a number of names over the past few years.
The ‘MECM/SCCM Addendum pack’ encompasses Endpoint Management which as of late, has taken on a number of names over the past few years.

The ‘MECM/SCCM Addendum pack’ started from administrators and field engineers’ inputs on actionable/manual intervention required alerts.  While Endpoint Management has taken on a number of names over the past few years, monitoring the platform functionality has stayed pretty much the same.  The underlying application infrastructure is based on registry key discovery of installed roles.

 

 

Quick Download https://github.com/theKevinJustin/MCMAddendum

 

 

Tailor the addendum for environment

Add monitoring for MECM servers per health model through daily team report, alert cleanup, custom groups to address subscription objects, servers, custom disk and client cache cleanup workflows, and lastly service restart automation.

Quick overview

The classes and DataSource/WriteAction alert reports require updates to target server naming convention(s).  The alert report is most effective this way, only giving the administrator/AppOwner alerts relevant to owned/supported servers.   Why – make the changes most effective, i.e. alert when manual intervention required.

Workflows, classes, and MonitorType

Addendum Classes, DataSource, WriteAction, and Unit MonitorType to build out 'manual intervention required alerting'.

 

Update Discovery to find/replace hashtags

Leveraging Kevin Holman’s MP fragment find/replace common variables notated by the ##variable##, we begin by updating the ##MECMServerNamingConvention## with a regular expression of the servers involved with Configuration Management.

Subscription group example of relevant classes for subscription notifications for Application/System Owners.

Second, we update the disk specific alerts if drives fill, where different amounts of space is required to alert before application/server crashes, different than the OS Logical Disk full composite alerts for % and MB free alerts.  These disk specific updates allowing administrator to get unique alerts for common disk full scenarios.

Disk specific updates allowing administrator to get unique alerts for common disk full scenarios.

 

Third, update MECM Group discoveries for various regular expressions.

MECM Group discovery updates for various regular expressions.

 

Lastly, review MECM Rules, Tasks, Monitor and Overrides for pack functionality.

MECM Rules, Tasks, Monitor and Overrides for pack functionality.

 

After updating relevant pieces, save file, move to SCOM MS, and Import.

My customers have loved this, hopefully this experience is shared!

 

Documentation

Kevin Holman MP fragments

Endpoint Management https://learn.microsoft.com/en-us/mem/endpoint-manager-overview

Microsoft System Center 2012 Configuration Manager Monitoring 5.0.8239.1010
Download https://systemcenter.wiki/?GetCategory=System+Center+2012+Configuration+Manager

Trellix Agent pack

Time to monitor the 'Trellix agent' pack
Time to monitor the ‘Trellix agent’ pack

Trellix bought McAfee, and rebranded, but the service, application, registry keys, etc. have not yet changed.  Many times, the pack fills in the gaps that the admin misses.  Examples when Application services crash or become non-responsive, or just adding the capability to summarize issues seen in a daily alert report.

 

Quick Download: https://github.com/theKevinJustin/TrellixAgentMonitoring

 

Did you know?

System Event ID 7031 is logged for each application/service when the process has issues?

Trellix agent services have a monitor alert when System Event Log, EventID 7031 events have the agent services in the event description.

Trellix agent services have a monitor alert when System Event Log, EventID 7031 events have the agent services in the event description.

 

Second, my own spin for Application monitoring starts with the mantra ‘smarter vs. harder.  Besides dynamic discovery based on registry key, adding the Service MonitorType gives additional monitorign flexibility adding Samples and Intervals to decrease false positive alerts.  Simply put – count logic – x failures in y time before alerting.

Service MonitorType adds Samples and Intervals to decrease false positive alerts.

Service MonitorType adds Samples and Intervals to decrease false positive alerts.

Third, the pack adds Trellix Agent rules, monitors, on-demand report task, and recovery scripts build out the manual intervention required alert action mantra.

Trellix Agent rules, monitors, on-demand report task, and recovery scripts build out the manual intervention required alert action mantra.
Trellix Agent rules, monitors, on-demand report task, and recovery scripts build out the manual intervention required alert action mantra.

 

 

Optional – Configure addendum for environment

Download and Install ‘Trellix Agent pack’ here

Open saved XML in notepad or Notepad++ (your favorite XML editor here!)

Update the regular expression pattern line for McAfee server group

Update Trellix Server group to match enclave/server naming convention

Save file and Import > enjoy less alerts!

 

 

Documentation

Addendum download https://github.com/theKevinJustin/TrellixAgentMonitoring

SCOMCore Addendum pack

SCOMCore Addendum pack - having a strong core makes bigger gains
SCOMCore Addendum pack – having a strong core makes bigger gains

Time to configure the Microsoft System Center Core Monitoring pack per health model and best practice.  That’s where the SCOMCore Addendum pack comes in. Addendum adds High Agent Handle count group, daily report and alert closure automation, and rule/monitor overrides.  Some assembly required – update the discovery pattern for offending high handle counts, and high handle count group ContextInstance GUID after import.

 

Quick Download: https://github.com/theKevinJustin/SCOMCoreAddendum

 

 

Background:

While High Agent  Handle count was more an issue before the x365 platform migrated UC, SharePoint, and email (i.e. Lync/Skype, SharePoint, Exchange on prem) went to the cloud.  This is still seen where cloud scalability options and virtualization/storage limitations exist.  Example typically is an over-utilized virtual machine in hybrid/IaaS/premise scenarios.  Kevin Holman caught this performance issue years back, creating monitoring alerts pack and blog.  In case you’re on SCOM jeopardy, the LAW/OMS/Microsoft Monitoring Agent/SCOM agent has a built-in health check.  The built-in health check restarts service when Handle Count or memory of the HealthService (aka Microsoft Monitoring Agent service) ran too hot per SCOM PG.   SCOM agent restarts caused config churn, and high compute, as workflows re-ran after the service restarted.

 

 

Assess agent restarts

Begin by verifying if you have Kevin Holman’s pack for SCOM agent restarts  downloaded and installed, which sets memory/handle count informational alerts https://github.com/thekevinholman/SCOM.AgentThresholds

Validate pack installed

Verify SCOM Agent Thresholds pack installed.
Verify SCOM Agent Thresholds pack installed.

 

 

Configure addendum for environment

Download and Install ‘SCOMCore Addendum pack’ here

Open saved XML in notepad or Notepad++ (your favorite XML editor here!)

Update the regular expression pattern line for offending servers in the

Update the pattern for the high agent handle count group for any offenders.

 

Figure out the group GUID for the high agent handle count

From PowerShell on SCOM management server, run:

Get-SCOMClassInstance -DisplayName “Proactive High Agent Handle Count servers” | fl DisplayName,ID

 

Find/Replace GUID

PowerShell GUID check.

 

Save file and Import > enjoy less alerts!

 

 

Documentation:

Kevin Holman blog on SCOM agent restarts

Holman’s pack for SCOM agent restarts and setting memory/handle count alerts https://github.com/thekevinholman/SCOM.AgentThresholds

Addendum download https://github.com/theKevinJustin/SCOMCoreAddendum

MCM Addendum pack

The MCM addendum pack helps monitor MEM. See start menu folder structure for Endpoint Manager software.
The MCM addendum pack helps monitor MEM. See start menu folder structure for Endpoint Manager software.

Rebranding central – MEM, EM, MECM, SCCM, Configuration manager, depending on the synonym, we’re referring to the same product.  Tune the most common critical alerts per the health model to warning.

 

QUICK DOWNLOAD https://github.com/theKevinJustin/MCMAddendum/

Background

Read Holman’s blog for more details.

Did you know – MCM discoveries are based on registry keys added with various role installs on windows servers.  These registry keys are typically under this path:  HKLM\SOFTWARE\Microsoft\SMS\Operations Management\Components

 

What capabilities does the ‘MCM addendum pack’ provide?

Quite simply, the pack provides warning severity overrides for common alerts, disable event collection rules.

9 overrides for monitors and rules included in addendum.
9 overrides for monitors and rules included in addendum.

 

Includes warning severity changes for the following rules and monitors:

Monitors

BackupStatus.StatusMessage.Monitor

ReportingPoint.RoleAvailability.Monitor

SoftwareUpdatePoint.RoleAvailability.Monitor

SoftwareUpdatePointSync.AlertState.Monitor

Rules

ComponentServer.ComponentStoppedUnexpectedly.Event.Rule

SiteComponentManager – CanNotFindObjectInAD.Event.Rule, CouldNotAccessSiteSystem.Event.Rule

StateSystem.FailedToExecuteSummaryTask.Event.Rule

WsusConfigurationManager.FailedToConfigProxy.Event.Rule

 

 

Utilize the ‘MCM Addendum pack’

Download Kevin Holman’s MCM pack from GitHub.

Download the Addendum here, to get alerts where manual intervention required.

Save packs

 

Enjoy some acronym humor and ‘who moved my cheese fun!’

MECM PowerShell
MECM PowerShell

 

Import into SCOM & Enjoy!

 

If you need more capabilities, reach out on the blog or GitHub.

 

Documentation

Github repository here

SCCM management pack

Holman blog for MEM, EM, MCM, MECM, CM, ConfigMgr, Configuration Manager

DHCP Addendum pack

Leverage DHCP addendum to tune DHCP subnet monitoring.
Leverage DHCP addendum to tune DHCP subnet monitoring.

Leverage the ‘DHCP Addendum pack’.  Why?  DHCP manages IP ranges, particularly customer facing issues like VPN connectivity, VDI/AVD/appliance devices, as well as client workstations/laptops/GFE’s.  The DHCP management pack alerts when a subnet is nearing zero available IP’s before you have an outage.  This article will help you understand how the addendum’s new capabilities tune DNS monitoring to best practice.

 

QUICK DOWNLOAD(S)

2016+ HTTPS://GITHUB.COM/THEKEVINJUSTIN/DCHPAGNOSTIC

 

What capabilities does the ‘DHCP Addendum pack’ provide?

Two groups, one DHCP server group, and DHCP subscription group to configure notifications to SME for DHCP related classes

Overrides for common alerts, disable event collection rules

 

 

Utilize the DHCP Addendum

Download the DHCP Addendum on GitHub, to get alerts where manual intervention required.

 

Update XML

The pack greatly decreases alerts, and the XML authoring is an easy feat.  After you import the pack, find/replace is required for two pieces.

  • Discovery group regular expressions (RegEx)

##DHCPServerRegEx##

Find ##DHCPServerRegEx## and replace with your DNS server expressions.

Example server names: 12dc01, 19dc01,19dc02,19dc03, etc.

RegEx = (?i)12dc0|19dc0

Using PowerShell on SCOM management server (MS) to determine group GUIDs for replace in Overrides/Discoveries.
Using PowerShell on SCOM management server (MS) to determine group GUIDs for replace in Overrides/Discoveries.

 

  • Update group GUIDs, after installing this pack.

Find/replace the GUIDs, as they are unique to every SCOM management group, hard coding the group ID GUID is not possible.  We will be running Get-SCOMClassInstance to determine the group GUID’s applicable in the management group.

 

From PowerShell, on your SCOM management server, run get-SCOMClassInstance commands for the two groups added.

get-scomclassinstance -DisplayName “Microsoft Windows DHCP 2016+ Servers” | ft Id

get-scomclassinstance -DisplayName “Microsoft Windows DHCP 2016+ subscription components” | ft Id

 

Example

Leveraging Notepad++ to find/replace the group GUID with SCOM environment specific GUIDs

Find/Replace the GUID in the pack with the ID from the output above.

Example leveraging Notepad++ to find/replace the group GUID with SCOM environment specific GUIDs

Save pack

Import into SCOM & Enjoy!

OS Addendum packs

OS Addendum packs for Windows Server from 2012 forward
OS Addendum packs for Windows Server from 2012 forward

Download the ‘OS Addendum packs’ for new capabilities contains Event count logic monitor type, Disk cleanup, Group Policy, self-healing/reset monitors, as well as ‘eventLog full’ logic and reports.  Additional monitors reduce alert noise.  Examples of common alert scenarios are: StorPort storage errors, Group Policy 1096 identification and rebuild.  Disk Cleanup & EventLog service recovery, which includes Event Log file expansion and rollover.

 

Quick DownloadS

https://github.com/theKevinJustin/2012OSAddendum

https://github.com/theKevinJustin/2016ServerAgnostic

 

 

Tune ‘OS Addendum packs’ as needed

Update logical disk paths and retentions.  The default report contains quite a few common checks, including root folders broken out by path, highest to lowest GB’.  The workflow is scalable to add additional application paths, as well as file retention timeframes.  Workflow runs on a weekly basis to cleanup/archive log files, paths.  See Disk cleanup logic blog for more details.

' OS Addendum packs' contains Logical disk breakdown of root folders to list paths were files stored, highest to lowest
‘ OS Addendum packs’ contains Logical disk breakdown of root folders to list paths were files stored, highest to lowest

 

UpdateStorPortCountForRepeatedStorageErrors

StorPort storage errors typically cut lots of alerts with storage reads/writes.  The ‘count’ monitors decrease the alerts, and the daily alert report consolidates the warning alerts (critical by default).  If you’re seeing these alerts, the default should decrease overall alerts to near zero.  Tune as  needed for disk alerts, by updating MatchCount or TimerWait in Seconds (the x events in y time piece of the monitor logic)

Update StorPort Count for Repeated Storage read/write errors
Update StorPort Count for Repeated Storage read/write errors

Save file(s) and import

AD insight reports

Need to audit AD? Use AD insight reports pack!
Need to audit AD? Use AD insight reports pack!

Download the ‘AD insights pack’ for new capabilities to audit users, svc/MSA accounts, password last set, expiring, last login AD insights. Includes AD group audit alert capability.

 

Quick Download https://github.com/theKevinJustin/ADInsights/

 

 

AD audit

Time to provide key ‘AD insight reports’ into users and groups.  Delve into different AD audit capabilities for users and groups.  The pack also gathers DC Security events (rules), and lastly, on demand tasks for reports.

 

The question is what determines a problem?

Every domain admin has a different experience and perspective, whether cyber (hack) focused or not.  Audit standards differ, from HIPAA, SOX, CCRI, STIG, etc.

Pack examples:

Users – service account naming conventions, password change frequency, expired date/time configured.

Groups – Choose your OU structure to audit WA in DA, SA in DA, WA in SA etc.

NOTE: Take caution on the OU group audit, to limit the output, as events have a size limitation

 

Configure ‘AD insight reports’

Now we can configure the user pack for applicable standards, like password age, last set, or AppOwners.  The AppOwners is an array, so you can add whatever Application, system owners/teams in your organization.  The password datasource (DS) rule runs weekly.

Configure the Password Time, last set, month, week and AppOwners to build out actionable svc/msa accounts failing audit artifacts.
Configure the Password Time, last set, month, week and AppOwners to build out actionable svc/msa accounts failing audit artifacts.

 

Break out the regular expressions of whatever accounts each team uses, to tailor relevant data into the report alert.  Find/Replace (Control-H) might be more effective, as the DS/WA repeat the logic for the on-demand task report, vs. the rule and monitor.

App Owner relevant service accounts by SamAccountName
App Owner relevant service accounts by SamAccountName

 

Update patterns ID naming conventions

Tailor account names to environment to match ingested DC Security events.

Tailor the DC Security Events to account naming conventions
Tailor the DC Security Events to account naming conventions.

 

Configure OU to environment

Configure OU structure to audit based on domain canonical names, groups, DC, etc.

AD Group audit example
AD Group audit example

 

Save file(s) and import

Disk cleanup logic

Logical disk cleanup, most times is harder vs. smarter manual intervention required, why not smarter vs. harder?
Logical disk cleanup, most times is harder vs. smarter manual intervention required, why not smarter vs. harder?

‘Disk cleanup logic’ traditionally follows manual intervention.  Why would you want harder and manual?  This article will present options to clean up system and non-system disks, by leveraging largest root folder, API’s and more.  This is one step in the OS Addendum pack that needs explanation and can be tailored to applications where admins have regular manual cleanup actions.

 

Breakdown of Disk cleanup

We want to check system disks and non-system disks for different scenarios.  Figure out Disk free space, user profiles, largest folder on root of disk, IIS cleanup, and MECM/SCCM client cache clear API.  Second, utilize different behaviors depending on PowerShell version, application log(s) cleanup, and expand drive alerts when NO space after cleanup action.

Disk Free space

# Check Disk free space
#=====================
if ($Driveletter -eq “C” )
{
$CFreeSpace = gwmi win32_logicaldisk | ? { $_.DeviceID -eq “C:” }
$CFreeSpace.DeviceID
$CFreeSpace
  $DeviceDriveLetter = $CFreeSpace.DeviceID
$DeviceDriveLetter
# Check folder size after cleanup
#==========================
$BeforeSize = (Get-ChildItem “$DeviceDriveLetter” -Recurse | Measure-Object -Property Length -Sum ).sum
$Before = [math]::Round($BeforeSize/1GB,2)
    $DiskFreeSpace = [pscustomobject]@{
DeviceID = $DeviceDriveLetter
Size = [math]::Round($CFreeSpace.Size/1GB,2)
FreeSpace = [math]::Round($CFreeSpace.FreeSpace/1GB,2)
       }
$DiskFreeSpace
$SoftwareDistribution = (gci C:\windows\SoftwareDistribution | measure length -s).sum / 1Mb
# Debug
$SoftwareDistribution

 

Check Software Distribution for ConfigMgr/SCCM/MECM client

Checking software distribution path was an item for discussion where the folder was larger than 3GB, stemming from customer and field engineers  recommendations.

 

If ($DiskFreeSpace.FreeSpace -lt 15 )
{
# Audit Software Distribution
#==================================
If ( $SoftwareDistribution -lt “3000” )
{
Write-host “NO SME/SystemOwner/SysAdmin/Server Action required”
}

If ( $SoftwareDistribution -gt “3000” )
{
Write-host “SME/SystemOwner/SysAdmin/Server Action required, stopping Windows Update service, removing SoftwareDIstribution folder and restarting”
Get-Service -Name wuauserv | Stop-Service
Remove-Item -Path C:\Windows\SoftwareDistribution -Recurse
Get-Service -Name wuauserv | Start-Service
Write-host “Windows Update wuauserv service restarted after SoftwareDistribution directory removed”
}

 

Cleanup Application log folders

The nice part of this is you can reuse this by changing the path and deletion actions to tailor to customer environment.  The script comes in handy for VEEAM, SQL, IIS instances and log directory on multiple drives.

 

# Cleanup IIS log files
#=====================
#if ( Test-Path C:\inetpub\logs\LogFiles\W3SVC1 )
#{
## Years older than
#$HowOld = [DateTime]::Now.AddYears(-1)
#$RecentUse = [DateTime]::Now.AddDays(-90)
## Path to root folder
#$Path = “C:\inetpub\logs\LogFiles\W3SVC1\*.log”
## Deletion task
#get-childitem $Path -Recurse -Depth 1 -EA SilentlyContinue | where { $_.lastAccesstime -lt $RecentUse -and $_.CreationTime -lt $HowOld -and $_.LastWriteTime -lt $RecentUse } | remove-item -force -verbose
#}

Sample report alert output

Sample system disk cleanup report alert
Sample system disk cleanup report alert

Documentation

CleanMgr https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/cleanmgr

Delete client cache the right way https://sccm-zone.com/deleting-the-sccm-cache-the-right-way-3c1de8dc4b48

MECM client cache cleanup PowerShell https://learn.microsoft.com/en-us/powershell/module/configurationmanager/invoke-cmclientaction?view=sccm-ps

SCCM Client Cache cleanup https://rzander.azurewebsites.net/sccm-config-item-to-cleanup-ccmcache/

Stack Overflow disk cleanup https://stackoverflow.com/questions/28852786/automate-process-of-disk-cleanup-cleanmgr-exe-without-user-intervention

PKI Addendum pack

The PKI addendum pack monitors PKI certificate hierarchy. Certificates can be a challenge, where we want to change focus to WHEN manual intervention is required.
The PKI addendum pack monitors PKI certificate hierarchy. Certificates can be a challenge, where we want to change focus to WHEN manual intervention is required.

The ‘PKI Addendum pack’ is a tricky pack, due to certificate hierarchy.  The decisions included are part of the three pillars – health, Security, Compliance, as well as alerting WHEN manual intervention required.

 

 

QUICK DOWNLOAD https://github.com/theKevinJustin/PKIAddendum

 

 

The PKI pack provides discoveries of the server certificate stores to then analyze certificates for validity, chain, and expiration.  The v1.4.3.0 release adds some task logic and script changes that helps discover more stores, trusted root, etc.

WHAT CAPABILITIES DOES THE ‘PKI ADDENDUM PACK’ PROVIDE?

Set timeframe for certificate per organizational standards.

Break out different expiration alerts based on internal/external certificate, or by AD Client Certificate enrollment templates (to build out the manual intervention required scenario when alerts are generated).

Create groups breaking out application self-signed, PKI certificates.

Separate RDP Auth, Domain Controller, Computer, and OCSP certificates.

 

If this sounds interesting, and you want to dabble in XML authoring…

Download the pack from GitHub to improve PKI monitoring on Windows Servers.

 

Additional screenshots of addendum components

Addendum pack creates multiple groups to break out various types of certificates that have different decisions/behaviors requiring unique timing

Groups

Addendum pack created groups to help admins get to the 'manual intervention' required alerting goal.
Addendum pack created groups to help admins get to the ‘manual intervention’ required alerting goal.

 

Discoveries

Leverage dynamic groups based on server name and EnhancedKeyUsageList (EKU) list

PKI dynamic group discoveries
PKI dynamic group discoveries

 

Overrides

Change PKI pack default discoveries, lifetime threshold expirations and more

Override PKI pack defaults
Override PKI pack defaults

 

DOCUMENTATION AND LINKS

Addendum requires the PKI Certificate MP release v1.4.3.0 download

Bob’s TopQuore blog

DNS2012R2 Addendum pack

Still running Server2012R2 servers with AD DCs with AD integrated DNS?
Still running Server2012R2 servers with AD DCs with AD integrated DNS?

In case you’re still running Windows Server 2012R2, here’s the ‘DNS2012R2 Addendum pack’ giving the same functionality as the version agnostic 2016+ addendum.  Why?  DNS is a translation method to convert names to IP’s.  Can you imagine if we wanted to connect to google via IP?  The number of workflows in the SCOM DNS pack (built by the DNS Product Group) makes for an astounding number of workflows running on your DC every minute.  Forward and reverse lookups are a good check, verifying DNS is functioning.  In a complex environment with 100’s of zones, SCOM becomes a utilization culprit for a DC’s primary missions – authenticate and resolve.  This article will help you understand how the pack will add new capabilities and tune DNS monitoring to best practice.

 

Quick Download HTTPS://GITHUB.COM/THEKEVINJUSTIN/DNSADDENDUM2012R2/

 

 

What capabilities does the ‘DNS Addendum pack’ provide?

Count logic monitors (i.e. x events in y time, and self heal)

Daily summary report of DNS alerts broken out

Daily alert closure workflow to close out DNS rules/monitor

DNS service(s) recovery automation

Synthetic internal/external nslookup monitor (scoped to PDC emulators versus ALL DNS servers

WMI validation alert recovery to prevent false positive alerts with weird one off scenarios – one example: Security tools randomly block WMI access.

 

Download the ‘DNS2012R2 Addendum pack’ on GitHub to improve AD Integrated (ADI) DNS monitoring on Windows Server 2016+ (version agnostic).

Save and Import pack, then update XML for group GUIDs

 

 

Update XML

First, update XML with the GUIDs from your management group.  Second, map the group DisplayName to find/replace the GUID for each group.

Get-SCOMClassInstance output for DNS2012R2 groups
Get-SCOMClassInstance output for DNS2012R2 groups

 

Third, using Notepad++ highlight the ContextInstance GUID and hit Control-H, and paste the group GUID then click Replace All.

Using Notepad++ highlight the ContextInstance GUID and hit Control-H, and paste the group GUID then click Replace All.
Using Notepad++ highlight the ContextInstance GUID and hit Control-H, and paste the group GUID then click Replace All.

Fourth – Rinse and repeat for the other three groups.

Lastly, save file, move to SCOM MS, and import!

 

Documentation and links

DNS Pack download

DNS2012R2 addendum blog including updates

GitHub Repository https://github.com/theKevinJustin/DNSAddendum2012R2/