Create ServiceNow incidents from SCOM

Create ServiceNow incidents from SCOM
Create ServiceNow incidents from SCOM

Been working to ‘Create ServiceNow incidents from SCOM’ with Joe Kelly (Joe’s LinkedIn), Steven Brown (Steven’s LinkedIn), and Tim Fields (Tim’s LinkedIn).  Over the past few months, we’ve found quite a few methods available, some free, some with cost.  All methods are built on ServiceNow (SNow) REST commands.  The resulting PowerShell scripts can do most of the heavy lifting for other Tools like SolarWinds, vCenter, vRealize,  Zabbix, SAP manager, Kafka, Nagios, HP OpenView.

 

Last month’s blog showcased the ServiceNow API/Connector for SCOM

ServiceNow Connector for SCOM

 

Depending on your customer use cases, this may not be the preferred choice.

One SNOW flavor is like NetCool, where you inject ALL SCOM alerts into ITSM tool (i.e. SNOW events).  Another flavor allows you to create a group of objects which will inject SNOW events.

What if you only want incidents for a small subset, and don’t want to wait on SNOW team?

We got that covered.

What if SNOW team wants events only, how can you create an incident?

We got that covered.

 

ServiceNow connector options:

Inject SNOW incident from SCOM (direct) https://github.com/theKevinJustin/New-SNOWIncidents

Inject SNOW event (requires SNOW alert rules to process alert to incident) https://github.com/theKevinJustin/New-SNowEvent

SNOW SCOM connector

    1. Send all SCOM alerts to SNOW as events
    2. Send specific SCOM group of objects to SNOW as events

3rd party options

      1. OpsLogix SCOM ServiceNow connector (costs) https://www.opslogix.com/servicenow-incident-connector
      2. Cookdown SNOW incident connector https://www.cookdown.com/blog/how-to-build-a-servicenow-incident-connector-for-scom download MP https://www.cookdown.com/servicenow-monitoring-mp#download
      3. Tim McFadden SCOM ServiceNow connector https://www.scom2k7.com/introducing-the-scom-to-servicenow-connector/ (unsure if Tim still supports this)

 

Stay tuned…

Next steps are to build out the SCOM command execution channel using Holman’s blog https://kevinholman.com/2021/08/25/what-account-will-command-channel-notifications-run-as-in-scom/

ServiceNow Event integration

ServiceNow Event integration
ServiceNow Event integration
Time to integrate your Monitoring tools to ITSM tool.  First, this blog post documents ‘ServiceNow Event integration’.  Second, let’s explain the common acronym in my experience is SNOW/SNow.  Third, some background – ServiceNow has been around for some time as an Information Technology Service Management (ITSM), and discovery tool.  As a SaaS solution, companies can purchase a subscription and integrate tools via RESTAPI to create/update/close events or incidents.
First, let’s begin to discuss SCOM notification methods.  SCOM2022 adds a new capability with Teams integration.  Second, most people are familiar with notification methods leveraging Email (html or not), perhaps SMS, but not so much command channel, calling some script in shell, PowerShell, etc.  Generally, the command channel is basically a post processing script capability to execute notifications.  Third, example tools where command channel might be used – BMC BEM (BMC Event Manager), BMC Remedy, xMatters, DerDack; SNOW integration within SCOM, using notification channels.  Lastly, SaaS solutions (vendors like xMatters, and ServiceNow) allow RESTAPI crafted requests to take actions.
SNOW prerequisites
1) ServiceNow User/Password (or API key)
2) SNOW RESTAPI PowerShell needs to securely access credentials
For the Incident PowerShell, we store Credentials within Windows Credential Manager
3) Network connectivity to SaaS provider (use PowerShell test-netconnection from SCOM MS to test connectivity over whatever port(s) vendor requires.
4) ServiceNow CallerID GUID
5) Production and Test URL’s (also required for network connectivity tests)
6) Access to SNOW UI to verify required fields and values for the script parameters.
Update incident script and begin testing.
Download script from GitHub repo https://github.com/theKevinJustin/New-SNowEvent/
Download script, and copy to monitoring repository
Copy to SCOM management servers (MS)
NOTE Path, to run from management server
Update script, with pre-reqs above –
Credential Manager stored ID
For more detail, look at parameter examples below to verify UI.
Update with customer/ServiceNow SNOW subscription specific values:
##CallerID##
##CUSTOMER##    (customize SNOW short_description)
##TEAM##    (customize SNOW short_description)$Channel = “Direct”
$ServiceNowURL=”https://##SERVICENOWURL##/api/now/table/em_event”
$CallerID = “##CallerID##”
# if proxy is used, uncomment and replace with Proxy URL
#$Proxy = “##Proxy##”
# Test New-SNOWEvent.ps1
# Depending on how you want to randomly choose an alert to create SNOW event
Lab example
$Alerts = get-scomalert -resolutionstate 0 | where { $_.Name -like “System Center*” }
Gather Critical, New alerts
$Alerts = get-scomalert -ResolutionState 0 -severity 2
Debug for warning alerts
$Alerts = get-scomalert -ResolutionState 0 -severity 1
# Debug
$Alerts[0] | fl ID,Name,Description,Severity,MonitoringObjectDisplayName
.\New-SNOWEvent.ps1 -AlertName $Alerts[0].Name -AlertID $Alerts[0].ID -Impact 4 -Urgency 4 -Priority 3 -AssignmentGroup “System Admin” -BusinessService “System Management” -Category Support -SubCategory Repair -Channel Direct
Example output
PS C:\Users\scomadmin\Desktop> .\New-SNOWIncident.ps1 -AlertName $Alert.Name -AlertID $Alert.ID -Impact 4 -Urgency 4 -Priority 3 -AssignmentGroup “System Admin” -BusinessService “System Management” -Category Support -SubCategory Repair -Channel Direct
TEST ServiceNow URL specified.
CredentialManager PoSH Module Installed, ModuleBase = C:\Program Files\WindowsPowerShell\Modules\CredentialManager\2.0
The System Center Management Health Service 5E04F804-8B71-6EB6-0101-DCBB58022498 running on host 16DB02.testlab.net and s
erving management group with id {E39F5F53-9FBB-9D7F-4BFE-5F0324630AE5} is not healthy. Some system rules failed to load.
16DB02
Warning
impact 4
urgency 4
priority 3
ServiceNow Credential NOT stored on server

ServiceNow Incident Integration

ServiceNow Incident Integration
ServiceNow Incident Integration
Time to integrate your Monitoring tools to ITSM tool.  First, this blog post documents ‘ServiceNow Incident integration’.  Second, let’s explain the common acronym in my experience is SNOW/SNow.  Third, some background – ServiceNow has been around for some time as an Information Technology Service Management (ITSM), and discovery tool.  As a SaaS solution, companies can purchase a subscription and integrate tools via RESTAPI to create/update/close events or incidents.
First, let’s begin to discuss SCOM notification methods.  SCOM2022 adds a new capability with Teams integration.  Second, most people are familiar with notification methods leveraging Email (html or not), perhaps SMS, but not so much command channel, calling some script in shell, PowerShell, etc.  Generally, the command channel is basically a post processing script capability to execute notifications.  Third, example tools where command channel might be used – BMC BEM (BMC Event Manager), BMC Remedy, xMatters, DerDack; SNOW integration within SCOM, using notification channels.  Lastly, SaaS solutions (vendors like xMatters, and ServiceNow) allow RESTAPI crafted requests to take actions.
SNOW prerequisites
1) ServiceNow User/Password (or API key)
2) SNOW RESTAPI PowerShell needs to securely access credentials
For the Incident PowerShell, we store Credentials within Windows Credential Manager
3) Network connectivity to SaaS provider (use test-netconnection from SCOM MS to test connectivity over whatever port(s) vendor requires.
ServiceNow CallerID GUID
4) Production and Test URL’s (also required for network connectivity tests)
5) Access to SNOW UI to verify required fields and values for the script parameters.
Update incident script and begin testing.
Download script, and copy to monitoring repository
Copy to SCOM management servers (MS)
NOTE Path, to run from management server
Update script, with pre-reqs above –
Credential Manager stored ID
For more detail, look at parameter examples below to verify UI.
Update with customer/ServiceNow SNOW subscription specific values:
##Company##     (customize SNOW short_description)
##Team##     (customize SNOW short_description)$Channel = “Direct”
$ServiceNowURL=”https://##ServiceNowURL##/api/now/table/incident”
#$Proxy = “##CustomerProxyURL##”
$CallerID = “##GUID##”
# Test New-SNOWIncident.ps1
# Depending on how you want to randomly choose an incident
Lab example
$Alerts = get-scomalert -resolutionstate 0 | where { $_.Name -like “System Center*” }
Gather Critical, New alerts
$Alerts = get-scomalert -ResolutionState 0 -severity 2
Debug for warning alerts
$Alerts = get-scomalert -ResolutionState 0 -severity 1
# Debug
$Alerts[0] | fl ID,Name,Description,Severity,MonitoringObjectDisplayName
.\New-SNOWIncident.ps1 -AlertName $Alerts[0].Name -AlertID $Alerts[0].ID -Impact 4 -Urgency 4 -Priority 3 -AssignmentGroup “System Admin” -BusinessService “System Management” -Category Support -SubCategory Repair -Channel Direct
Example output
PS C:\Users\scomadmin\Desktop> .\New-SNOWIncident.ps1 -AlertName $Alert.Name -AlertID $Alert.ID -Impact 4 -Urgency 4 -Priority 3 -AssignmentGroup “System Admin” -BusinessService “System Management” -Category Support -SubCategory Repair -Channel Direct
TEST ServiceNow URL specified.
CredentialManager PoSH Module Installed, ModuleBase = C:\Program Files\WindowsPowerShell\Modules\CredentialManager\2.0
The System Center Management Health Service 5E04F804-8B71-6EB6-0101-DCBB58022498 running on host 16DB02.testlab.net and s
erving management group with id {E39F5F53-9FBB-9D7F-4BFE-5F0324630AE5} is not healthy. Some system rules failed to load.
16DB02
Warning
impact 4
urgency 4
priority 3
ServiceNow Credential NOT stored on server

Configure SCOM agent from PowerShell

Configure SCOM agent from PowerShell
Configure SCOM agent from PowerShell

 

Ever have a need to configure a SCOM agent outside the SCOM Console GUI using SCOM.Management pack?   Tyson’s ‘SCOM Helper PowerShell module’ on monitoringguys.com may assist if you can update PowerShell and SnapIn’s.  If Tyson’s solution may not be an applicable solution in a locked down environment, the blog is a PowerShell script to manage agent configuration.

 

A shout out to Aris for his watchful eye testing and playing with this!

 

Configure SCOM agent from PowerShell

GitHub Download  https://github.com/theKevinJustin/SCOMAgentInstallApplication/

First, let’s begin the conversation.  Leverage the PowerShell script to configure SCOM/MOM agent per domain(s).  Whether DMZ forest(s), or multiple forests with SCOM gateways, the script will enable configuration management.  Alternatively, add script as a compliance check to the Configuration Management tool, or as a post install step to configure SCOM Agent (MomAgent.MSI) via application or package.  Third, utilize the PowerShell script where agent upgrade is a pre-req to resolve ‘a really old SCOM agent version’.   Lastly, the script also tests network connectivity, and verifies if configuration is requested/loaded.

 

It all starts with this object from the command:

$SCOMAgent = New-Object -ComObject AgentConfigManager.MgmtSvcCfg

 

Use the PowerShell script to:

Clean up extraneous management groups

Determine domain

Check network connectivity via test-netconnection

Reset healthservice to expedite agent configuration

 

 

Background

Because Kevin Holman blogged this almost 10 years ago in VB, and I’m pretty rusty in VB these days.  Check out Kevin’s blog titled ‘Using the agent scripting objects on a SCOM 2012 agent‘.

 

Documentation

Download https://github.com/theKevinJustin/SCOMAgentInstallApplication/

Kevin Holman’s blog for VB commands

Using the agent scripting objects on a SCOM 2012 agent

The agent configuration objects in MSDN http://msdn.microsoft.com/en-us/library/hh328967.aspx

 

Improving SCOM Monitor reset logic

Faster - Improving SCOM Monitor reset logic
Faster – Improving SCOM Monitor reset logic

 

My thanks to Aris Somatis for his deep dive reviewing the packs with me, particularly new use cases.  The PowerShell below builds on Scott Murr’s initial TechNet published logic from years back.  Consequently, the reset logic provides a ‘manual intervention required’ alerting/monitoring system.

 

 

Improving SCOM monitor reset logic

Calling the reset method has been a game changer for my customers – including operators, system and application owners!

Background

Scott’s reset logic, from SCOM2012, helped administrators reset unhealthy monitors where alerts may have been closed.  Because Scott leveraged the ResetMonitoringState method, the community gained a way to keep true health.  Additionally, many administrators and engineers built custom management packs to provide solutions.  Second, the addendum packs blog brought in more options – best practices, lessons from the field (and customers), and health model accurate alerting for what was really broken in the environment.  Third,    addressing ‘gaps’ or ‘blind spots’ from product teams.   As a result of NEW monitoring, the packs may include: rules/monitors, datasource/writeAction (DS/WA) workflows, recovery tasks and automation, count logic monitors, overrides, discoveries, and groups.    Thirdly, to take monitoring to the next level.  To top that off, with very little/NO cost compared to competitors!

PowerShell code

Aris’s Age use case takes this even further.  Using monitor age allows further analysis to dial down ‘monitor reset’ to object is X days old.  Comparatively, the 24-72 hour setup default is used in the addendums, so Age provides a second option.  Third option can rely on SCOM’s built-in cleanup, but that’s typically 14-30 days.   Overall, flexibility is a good thing.

 

# Specify age variable for your environment

$Age = [DateTime](Get-Date).AddDays(-7)

 

PowerShell code snippet

First, the reset logic can pivot on the age requirement.  Then, adjust the Age variable per requirements.  Third, figure out which method applies to gather a unique list of classes, whether by partial string(s), or by management pack name(s).

 

Set age variable (how long ‘OLD’ monitors might be stale and need reset)

# Example sets $Age variable to 7 days ago (-7)

$Age = [DateTime](Get-Date).AddDays(-7)

 

 

Unpack two different ways to gather classes for monitors to reset

# When common string name exists in all classes

Example DFS/FileServices packs all have one of the three strings:

# DFS pack naming

$DFSClasses = @(Get-SCOMClass -Name “*FileServices*”; Get-SCOMClass -Name “*FileServer*”; Get-SCOMClass -Name “*DFS*” )

$DFSClass = $DFSClasses | sort -property Name -uniq

# Debug

$DFSClass.Count

 

# Get AD classes – Microsoft.Windows.Server.AD.2016.Discovery, Microsoft.Windows.Server.AD.Library

$ADLibrary = Get-SCOMManagementPack -name “Microsoft.Windows.Server.AD.Class.Library”

#get-scomclass -ManagementPack $ADLibrary

$ADMonitoring = Get-SCOMManagementPack -name “Microsoft.Windows.Server.AD.2016.Monitoring”

#get-scomclass -ManagementPack $ADMonitoring | fl DisplayName,Name,ID

$ADDiscovery = Get-SCOMManagementPack -name Microsoft.Windows.Server.AD.2016.Discovery

#get-scomclass -ManagementPack $ADDiscovery | fl DisplayName,Name,ID

# ADDS pack naming

$ADDSClasses = @(Get-SCOMClass -ManagementPack $ADLibrary; Get-SCOMClass -ManagementPack $ADDiscovery; )

# NOTE Excluded AD Monitoring pack as NO classes existed

$ADDSClass = $ADDSClasses | sort -property Name -uniq

 

# Debug count of unique classes

$ADDSClass.Count

 

 

Reset monitor PowerShell screenshot

Download from GitHub https://github.com/theKevinJustin/SCOMMonitorReset

SCOM Reset PowerShell
SCOM Reset PowerShell

 

Example PowerShell on HealthService resets

NOTE debug logic enabled

0
Found 0 unhealthy monitors for class Microsoft.SystemCenter.HealthServicesGroup
1
Found 1 unhealthy monitors for class Microsoft.SystemCenter.HealthServiceWatcher

Resetting Health State on ' + Microsoft.SystemCenter.HealthServiceWatcher:Microsoft.SystemCenter.AgentWatchersGroup;5e0
4f804-8b71-6eb6-0101-dcbb58022498 + '

Guid
----
0218d239-3d37-f9b1-75d2-6d52c2c7c0c1


Documentation/Sources

Building on Scott’s idea – (retired links)
Original post https://sc.scomurr.com/scom-2012-monitor-reset-cleaning-up-the-environment/
TechNet gallery download https://gallery.technet.microsoft.com/SCOM-2012-Batch-reset-63a17534
Alternate link https://gallery.technet.microsoft.com/scriptcenter/Auto-reset-script-for-d8b775ca

Delete PKI certificates

Don't you wish this were the certificates we worked with!
Don’t you wish this were the certificates we worked with!

 

Man, I wish I had a few of these certificates in my PKI infrastructure (portfolio)!  Ever need to identify an expired certificate, and or delete the certificate?  Depending on UAC, AppLocker, and other settings, the delete portion may require server logon.

 

 

Identify an expired certificate

Few ways to identify an expired certificate on servers.

Via RDP session to server

RDP to server, open MMC > Add Plug In > Certificates > for Computer

RDP to server, MMC, Certificates Plug In, Computer, expand certificate store to find expired certificate.
RDP to server, MMC, Certificates Plug In, Computer, expand certificate store to find expired certificate.

 

Via PowerShell

GCI Cert:\LocalMachine\*

The certificate store file path will vary the above command.

NOTE the SuperUser blog post will help decipher the folder name

 

Via ‘smart’ PowerShell across multiple machines

Courtesy of Vance Cozier!

“server1”,”server2″| % {invoke-command $_ -scriptblock {$env:ComputerName; GCI Cert:\LocalMachine\* }}

 

Via Monitoring for expired certificates

Using SCOM Certificate Monitoring or PKI monitoring management packs

Identify Expired certificates

Console navigation steps:

From SCOM console > Monitoring Tab

Expand PKI folder > Expand Certificates and CRLs folder

Click on ‘Certificates – Expired’ state view

SCOM Expired PKI certificates state view.
SCOM Expired PKI certificates state view.

 

Did you know

Run PowerShell commands from Holman’s SCOM Management pack (blog)- github download

Run PowerShell from Monitoring tool

From SCOM console > Monitoring Tab > SCOM Management folder > SCOM Agents

Highlight agent(s)

From Tasks Pane > click on ‘Execute any PowerShell’ task

SCOM Management pack, SCOM Agents, highlight agent(s), click on Task Pane > Execute any PowerShell task.
SCOM Management pack, SCOM Agents, highlight agent(s), click on Task Pane > Execute any PowerShell task.

On the pop-up window, click Override

Adjust timeout to 70 (seconds)

Override command line

Run the following commands (various Certificate store examples provided for removing expired certificates)

Personal Certificates folder

Get-ChildItem Cert:\LocalMachine\My | ? { (( $_.Subject -like “*GlobalSign*” ) -OR ( $_.Issuer -like “*GlobalSign*” )) -and $_.NotAfter -lt (get-date) } | fl Subject,Issuer

Trusted Root folder

Get-ChildItem Cert:\LocalMachine\Root | ? { ( $_.Subject -like “*GlobalSign*” ) -OR ( $_.Issuer -like “*GlobalSign*” ) } | fl  Subject,Issuer

Third Party Root

Get-ChildItem Cert:\LocalMachine\AuthRoot | ? { (( $_.Subject -like “*GlobalSign*” ) -OR ( $_.Issuer -like “*GlobalSign*” )) -and $_.NotAfter -lt (get-date) } | fl Subject,Issuer

Run PowerShell task.
Run PowerShell task.

Click OK button to close Overrides window

Click OK to run task

View task output

PS C:\Users\admin> Get-ChildItem Cert:\LocalMachine\Root | ? { ( $_.Subject -like “*GlobalSign*” ) -OR ( $_.Issuer

like “*GlobalSign*” ) } | fl Subject,Issuer

 

Subject : CN=GlobalSign Root CA, OU=Root CA, O=GlobalSign nv-sa, C=BE

Issuer  : CN=GlobalSign Root CA, OU=Root CA, O=GlobalSign nv-sa, C=BE

Subject : CN=GlobalSign, O=GlobalSign, OU=GlobalSign Root CA – R2

Issuer  : CN=GlobalSign, O=GlobalSign, OU=GlobalSign Root CA – R2

 

 

Once task completes, go back to the Certificates state view > highlight the certificate > Click the Rediscover Certificates task

SCOM PKI Certificate state view
SCOM PKI Certificate state view

 

 

 

Delete PKI certificates

After identifying correct certificate and folder in question that requires deletion:

    1. Verify backup of certificate
    2. Test delete via PowerShell window from SCOM

Get-ChildItem Cert:\LocalMachine\AuthRoot | ? { ( $_.Subject -like “*GlobalSign*” ) -OR ( $_.Isser -like “*GlobalSign*” ) } | Remove-Item -DeleteKey

  1. Verify Task output shows certificate deleted
  2. Refresh MMC GUI > Certificates plug in, and folder to verify certificate deleted
  3. Go back to SCOM Console Monitoring Tab
  4. Expand PKI folder
  5. Expand Certificates and CRLs folder
  6. Click on Certificates – Expired state view
  7. In Tasks pane on right, click on Rediscover certificates task, Click OK to run
  8. Open Health Explorer and reset monitor to clear alert

Alternatively, leverage PowerShell reset logic on SCOM MS, use blog

 

 

Documentation

SCOM Certificate monitoring TechCommunity blog https://techcommunity.microsoft.com/t5/system-center-blog/scom-management-pack-for-certificate-monitoring/ba-p/3619299

Download SCOM Certificate monitoring pack https://www.microsoft.com/en-us/download/details.aspx?id=104858

Long-standing PKI certificate pack and blog https://blog.topqore.com/new-version-pki-certificates-monitoring-pack-for-scom/

Identify certificate stores to windows folders https://superuser.com/questions/217719/what-are-the-windows-system-certificate-stores

STIG finding V-69223 https://www.stigviewer.com/stig/zos_tss/2016-06-30/finding/V-69223

IIS addendum packs

IIS addendum packs to tune IIS from 2012 forward.

IIS addendum packs to tune IIS from 2012 forward.’IIS addendum packs’ to tune IIS from 2012 forward.  The GitHub repository has two packs 2012/2016+ (version agnostic pack).  This includes an IIS enabled group, Daily report and cleanup DataSource and WriteAction (tasks), as well as a regular expression to set up the IIS enabled group.  The IIS enabled group is to enable IIS monitoring on servers IIS monitoring is needed.

 

 

Customize for environment

Update addendums to server naming conventions for enabled IIS monitoring.  Read below to better understand addendum functionality.

First, the addendums include class/group, datasource and write action alert reports and automated alert closure workflows, as well as event count logic/reset monitorType.

Addendum includes class/group, datasource and write action workflows for alert reports and automated alert closure, as well as event count logic reset monitorType.

 

Second, the group discovery, find/replace the pattern to various application/web server naming conventions where IIS monitoring IS wanted.

Third, the version agnostic has overrides to disable most perf and rule alerts.  Can provide OFF packs to turn off performance counter collection rules, to keep both the OperationsManager, and OperationsManagerDW databases cleaner, thereby faster with less data.

IIS2012 overrides
IIS2012 overrides

Lastly, once addendum updated, save file, move to SCOM MS, and import.

Enjoy the ‘IIS addendum packs’ for how few alerts, perhaps life changing?! (sarcasm)

 

 

Documentation

Download Addendum packs https://github.com/theKevinJustin/IISAddendums

IIS2012 SCOM Management pack download https://www.microsoft.com/en-us/download/details.aspx?id=34767

IIS2016+ SCOM management pack download https://www.microsoft.com/en-us/download/details.aspx?id=54445

Proactive Daily Reports

Proactive Analyst Reports as a new way to ingest key insights from SCOM
Proactive Analyst Reports as a new way to ingest key insights from SCOM

As a SME or team lead, ever need to know a key insight for the enclave?  Let’s talk about the ‘Proactive Daily Reports’ pack.  This provides you some built-in reports on what transpired in an enclave.  Building again on the Health pillar, we can simplify what owners need to see.  Creating a PowerShell script was a simpler alternative to a complex SSRS report that often broke due to patching, and not following best practices.  The pack shows a simpler way to bring key insights to owners for Pending Reboots, Expiring PKI certificates, Logical Disk alerts, System Admin summary, and SCOM admin reports including long-running scripts, script errors, SCOM errors, and alert updates report.

 

Quick Download: https://github.com/theKevinJustin/ProactiveNOSCDailyTasks

Testing the Proactive Daily reports

Let’s start with some example reports – examples for expiring certificates, Logical Disk, Pending Reboot, System Admin summary, and SCOM admin reports including long-running scripts, script errors, SCOM errors, and alert updates report.

 

Expiring Certs –

About to expire certificates

Expiring PKI certificates reports
Expiring PKI certificates reports

 

Logical disk alerts –

Shows Server, drive, and % full data

Logical disk alerts report, showing zero for the past 72 hours (over a weekend)
Logical disk alerts report, showing zero for the past 72 hours (over a weekend)

 

Pending Reboots

Alerts of servers pending restart, not patched, not rebooted

Pending reboot report lists servers pending restart, not patched, not rebooted alerts
Pending reboot report lists servers pending restart, not patched, not rebooted alerts

 

System Admin summary

This is really a consolidation of multiple insights:

Server performance issues
Open ITSM/Remedy tickets
Unhealthy Agents
Pending Reboot, Not Rebooted, Not patched
Disabled/Unhealthy/MaintenanceMode, Repeatedly down Agents
Logical Disk free space alerts
Expiring certificates
AD DC (ADDS) critical alerts
DNS alerts
Group Policy issues

SysAdmin daily summary report example alert
SysAdmin daily summary report example alert

 

SCOM admin reports

Admin reports have a few separate alert reports, including long-running scripts, script errors, SCOM errors, and alert updates report.

SCOM Admin alerts report example of common SCOM problems
SCOM Admin alerts report example of common SCOM problems

 

Long running scripts

SCOM Admin long running scripts alerts report example of longrunning report workflows to help tune run-time
SCOM Admin long running scripts alerts report example of long-running report workflows to help tune run-time

 

ScriptErrors showing key SCOM connectivity issuesSCOM Admin script errors to help diagnose report script syntax errors

SCOM Admin script errors to help diagnose report script syntax errors

Useful links

Other blog posts for addendum management packs and capabilities –

https://kevinjustin.com/blog/2023/08/15/proactive-patching-alerts/
https://kevinjustin.com/blog/2023/08/14/top-process-powershell-script/
https://kevinjustin.com/blog/2023/08/15/proactive-daily-reports/

https://kevinjustin.com/blog/2023/08/08/create-closed-alerts-view/

 

Proactive Patching alerts

Proactive
proactive scrabble tiles

As a SME or team lead, ever need to know ‘Proactive Patching alerts’?  i.e. What servers need patches applied, aren’t patching, or were missed?  This pack builds on three (3) pillars – Health/Security/Compliance, enabling Cyber teams and more.  This became an alternate option to a complex pack, with SSRS report, used by a customer to identify systems.  The report was long, and had many blank lines/pages, which required a re-write.  This pack started with the pending restart monitor directly from the AquilaWeb reboot pack logic.  The logic helps SysAdmin/Domain Admin/NOC/NOSC/SOC teams to know when servers need reboots.  This need is driven further due to multiple reboots (sometimes) required with Windows monthly updates, and Application updates.  Used across multiple customers, this is the first pack enabling a proactive stance to answer the ‘Am I compliant’ question.

 

Quick Download: https://github.com/theKevinJustin/ProactivePatchUptimeReboot/

Testing the Proactive Patch alerts

David Allen built the ‘Aquilaweb.Support.PendingReboot.Monitor.PendingReboot’ PowerShell monitor, to tell system owners when the pending restart flag was present.  Some builds though, make system changes which repeatedly flip the registry key, causing many alerts.  Also, downloading the Aquila pack is a trick, as TechNet was retired.

 

David provided a great idea, which was built upon.  This gave rise to the question of, what if the server was not patched, or not rebooted in a period of time?   With my Cyber hat on, this became the next piece of content to create.  That gave rise to another question – do these scenarios need to reflect in health (monitor), or not (rule)?   We’re all about choices, free will, so the pack is built with those options (rules disabled out of the box).

Pending restart monitor XML showing options
Pending restart monitor XML showing options

The pack is setup to alert with CBS application updates, SCCM/MECM/Config Mgr Endpoint Management updates, and Windows Updates.  This has been my experience for the most accurate reflections of alerts on secure builds where Application/System Owner needs to take action.

Last Patch and Last Reboot monitor/rules in the download, are set to 45 days.  Tune this value down, if patching occurs at the 30 day mark, increase if you need more time before alerts.

Last Patch Monitor reflecting number of days
Last Patch Monitor reflecting number of days

 

Otherwise, download and import into your environment.  Depending on your subscription/notification settings, the Proactive set of alerts are built upon the Windows Operating System class.  If subscriptions include the class, the notifications are automatic to System/Application owners.

 

Useful links

David Allen blog

Addendum, what does it mean blog

Check your delegation settings

 

Sometimes as the monitoring admin, you may be responsible to secure your servers, being told from Security/Cyber Teams about new vulnerabilities.  The vulnerabilities may be from Tanium, ACAS, Tenable or other security tools.   This article is how to help you secure related SCOM web console, and SSRS reporting sites against Unconstrained Delegation vulnerabilities CVE-2020-17049, also AD STIG V-36435.

 

First we need to identify IF this is a true finding.

Typically this comes from Server/SystemsAdmin with domain admin access:

From PowerShell run:

Get-ADComputer -LDAPFilter
“(userAccountControl:1.2.840.113556.1.4.803:=524288)”

After identifying SCOM servers with unconstrained delegation (scope of blog post is focused on SCOM in a hybrid scenario (prem/cloud), we need to resolve.

With domain administrator rights, open the Active Directory Users and Computers MMC snap-in.

In ADUC, change Find drop-down to Computers
In the Computer name text box, enter <SCOMServer>  and click search

Right click the server in the results box > Select Properties.

Select the Delegation tab.

ADUC view of lab server delegation setting

 

Select Trust this computer for delegation to specified services only > Use any authentication protocol.

Under Services to which this account can present delegated credentials, select Add.

In the new dialog box, select Users or Computers.

Enter <SCOMServer>, and then select OK.

Click the Add button to add services

Select the w3svc and www processes

Select OK.

ADUC GUI adding services for delegation on SCOM server

Once set in AD, reboot server.  Running ‘gpupdate /force’ may not apply AD changes to the server object.

After reboot, reach out to SCOM Admins to test webconsole authentication

From edge browser, go to SCOM web console, typically @ https://<SCOMServer>/OperationsManager

On the Monitoring tab, click on Active Directory dashboard on left

Verify authentication works

 

Documentation

Pentestlab – Detecting Unconstrained Delegation Exposures in AD Environment

Petri.com find and block unconstrained delegation

Learn.Microsoft.com unconstrained kerberos article

Explanatory documents on what/why

Remove Unconstrained Kerberos Delegation