Been working to ‘Create ServiceNow incidents from SCOM’ with Joe Kelly (Joe’s LinkedIn), Steven Brown (Steven’s LinkedIn), and Tim Fields (Tim’s LinkedIn). Over the past few months, we’ve found quite a few methods available, some free, some with cost. All methods are built on ServiceNow (SNow) REST commands. The resulting PowerShell scripts can do most of the heavy lifting for other Tools like SolarWinds, vCenter, vRealize, Zabbix, SAP manager, Kafka, Nagios, HP OpenView.
Last month’s blog showcased the ServiceNow API/Connector for SCOM
Depending on your customer use cases, this may not be the preferred choice.
One SNOW flavor is like NetCool, where you inject ALL SCOM alerts into ITSM tool (i.e. SNOW events). Another flavor allows you to create a group of objects which will inject SNOW events.
What if you only want incidents for a small subset, and don’t want to wait on SNOW team?
We got that covered.
What if SNOW team wants events only, how can you create an incident?
Time to integrate your Monitoring tools to ITSM tool. First, this blog post documents ‘ServiceNow Event integration’. Second, let’s explain the common acronym in my experience is SNOW/SNow. Third, some background – ServiceNow has been around for some time as an Information Technology Service Management (ITSM), and discovery tool. As a SaaS solution, companies can purchase a subscription and integrate tools via RESTAPI to create/update/close events or incidents.
First, let’s begin to discuss SCOM notification methods. SCOM2022 adds a new capability with Teams integration. Second, most people are familiar with notification methods leveraging Email (html or not), perhaps SMS, but not so much command channel, calling some script in shell, PowerShell, etc. Generally, the command channel is basically a post processing script capability to execute notifications. Third, example tools where command channel might be used – BMC BEM (BMC Event Manager), BMC Remedy, xMatters, DerDack; SNOW integration within SCOM, using notification channels. Lastly, SaaS solutions (vendors like xMatters, and ServiceNow) allow RESTAPI crafted requests to take actions.
SNOW prerequisites
1) ServiceNow User/Password (or API key)
2) SNOW RESTAPI PowerShell needs to securely access credentials
For the Incident PowerShell, we store Credentials within Windows Credential Manager
3) Network connectivity to SaaS provider (use PowerShell test-netconnection from SCOM MS to test connectivity over whatever port(s) vendor requires.
4) ServiceNow CallerID GUID
5) Production and Test URL’s (also required for network connectivity tests)
6) Access to SNOW UI to verify required fields and values for the script parameters.
Time to integrate your Monitoring tools to ITSM tool. First, this blog post documents ‘ServiceNow Incident integration’. Second, let’s explain the common acronym in my experience is SNOW/SNow. Third, some background – ServiceNow has been around for some time as an Information Technology Service Management (ITSM), and discovery tool. As a SaaS solution, companies can purchase a subscription and integrate tools via RESTAPI to create/update/close events or incidents.
First, let’s begin to discuss SCOM notification methods. SCOM2022 adds a new capability with Teams integration. Second, most people are familiar with notification methods leveraging Email (html or not), perhaps SMS, but not so much command channel, calling some script in shell, PowerShell, etc. Generally, the command channel is basically a post processing script capability to execute notifications. Third, example tools where command channel might be used – BMC BEM (BMC Event Manager), BMC Remedy, xMatters, DerDack; SNOW integration within SCOM, using notification channels. Lastly, SaaS solutions (vendors like xMatters, and ServiceNow) allow RESTAPI crafted requests to take actions.
SNOW prerequisites
1) ServiceNow User/Password (or API key)
2) SNOW RESTAPI PowerShell needs to securely access credentials
For the Incident PowerShell, we store Credentials within Windows Credential Manager
3) Network connectivity to SaaS provider (use test-netconnection from SCOM MS to test connectivity over whatever port(s) vendor requires.
ServiceNow CallerID GUID
4) Production and Test URL’s (also required for network connectivity tests)
5) Access to SNOW UI to verify required fields and values for the script parameters.
Ever have a need to configure a SCOM agent outside the SCOM Console GUI using SCOM.Management pack? Tyson’s ‘SCOM Helper PowerShell module’ on monitoringguys.com may assist if you can update PowerShell and SnapIn’s. If Tyson’s solution may not be an applicable solution in a locked down environment, the blog is a PowerShell script to manage agent configuration.
A shout out to Aris for his watchful eye testing and playing with this!
First, let’s begin the conversation. Leverage the PowerShell script to configure SCOM/MOM agent per domain(s). Whether DMZ forest(s), or multiple forests with SCOM gateways, the script will enable configuration management. Alternatively, add script as a compliance check to the Configuration Management tool, or as a post install step to configure SCOM Agent (MomAgent.MSI) via application or package. Third, utilize the PowerShell script where agent upgrade is a pre-req to resolve ‘a really old SCOM agent version’. Lastly, the script also tests network connectivity, and verifies if configuration is requested/loaded.
My thanks to Aris Somatis for his deep dive reviewing the packs with me, particularly new use cases. The PowerShell below builds on Scott Murr’s initial TechNet published logic from years back. Consequently, the reset logic provides a ‘manual intervention required’ alerting/monitoring system.
Improving SCOM monitor reset logic
Calling the reset method has been a game changer for my customers – including operators, system and application owners!
Background
Scott’s reset logic, from SCOM2012, helped administrators reset unhealthy monitors where alerts may have been closed. Because Scott leveraged the ResetMonitoringState method, the community gained a way to keep true health. Additionally, many administrators and engineers built custom management packs to provide solutions. Second, the addendum packs blog brought in more options – best practices, lessons from the field (and customers), and health model accurate alerting for what was really broken in the environment. Third, addressing ‘gaps’ or ‘blind spots’ from product teams. As a result of NEW monitoring, the packs may include: rules/monitors, datasource/writeAction (DS/WA) workflows, recovery tasks and automation, count logic monitors, overrides, discoveries, and groups. Thirdly, to take monitoring to the next level. To top that off, with very little/NO cost compared to competitors!
PowerShell code
Aris’s Age use case takes this even further. Using monitor age allows further analysis to dial down ‘monitor reset’ to object is X days old. Comparatively, the 24-72 hour setup default is used in the addendums, so Age provides a second option. Third option can rely on SCOM’s built-in cleanup, but that’s typically 14-30 days. Overall, flexibility is a good thing.
# Specify age variable for your environment
$Age = [DateTime](Get-Date).AddDays(-7)
PowerShell code snippet
First, the reset logic can pivot on the age requirement. Then, adjust the Age variable per requirements. Third, figure out which method applies to gather a unique list of classes, whether by partial string(s), or by management pack name(s).
Set age variable (how long ‘OLD’ monitors might be stale and need reset)
# Example sets $Age variable to 7 days ago (-7)
$Age = [DateTime](Get-Date).AddDays(-7)
Unpack two different ways to gather classes for monitors to reset
# When common string name exists in all classes
Example DFS/FileServices packs all have one of the three strings:
0
Found 0 unhealthy monitors for class Microsoft.SystemCenter.HealthServicesGroup
1
Found 1 unhealthy monitors for class Microsoft.SystemCenter.HealthServiceWatcher
Resetting Health State on ' + Microsoft.SystemCenter.HealthServiceWatcher:Microsoft.SystemCenter.AgentWatchersGroup;5e0
4f804-8b71-6eb6-0101-dcbb58022498 + '
Guid
----
0218d239-3d37-f9b1-75d2-6d52c2c7c0c1
Don’t you wish this were the certificates we worked with!
Man, I wish I had a few of these certificates in my PKI infrastructure (portfolio)! Ever need to identify an expired certificate, and or delete the certificate? Depending on UAC, AppLocker, and other settings, the delete portion may require server logon.
Identify an expired certificate
Few ways to identify an expired certificate on servers.
Via RDP session to server
RDP to server, open MMC > Add Plug In > Certificates > for Computer
RDP to server, MMC, Certificates Plug In, Computer, expand certificate store to find expired certificate.
Via PowerShell
GCI Cert:\LocalMachine\*
The certificate store file path will vary the above command.
NOTE the SuperUser blog post will help decipher the folder name
IIS addendum packs to tune IIS from 2012 forward.’IIS addendum packs’ to tune IIS from 2012 forward. The GitHub repository has two packs 2012/2016+ (version agnostic pack). This includes an IIS enabled group, Daily report and cleanup DataSource and WriteAction (tasks), as well as a regular expression to set up the IIS enabled group. The IIS enabled group is to enable IIS monitoring on servers IIS monitoring is needed.
Customize for environment
Update addendums to server naming conventions for enabled IIS monitoring. Read below to better understand addendum functionality.
First, the addendums include class/group, datasource and write action alert reports and automated alert closure workflows, as well as event count logic/reset monitorType.
Second, the group discovery, find/replace the pattern to various application/web server naming conventions where IIS monitoring IS wanted.
Third, the version agnostic has overrides to disable most perf and rule alerts. Can provide OFF packs to turn off performance counter collection rules, to keep both the OperationsManager, and OperationsManagerDW databases cleaner, thereby faster with less data.
IIS2012 overrides
Lastly, once addendum updated, save file, move to SCOM MS, and import.
Enjoy the ‘IIS addendum packs’ for how few alerts, perhaps life changing?! (sarcasm)
Proactive Analyst Reports as a new way to ingest key insights from SCOM
As a SME or team lead, ever need to know a key insight for the enclave? Let’s talk about the ‘Proactive Daily Reports’ pack. This provides you some built-in reports on what transpired in an enclave. Building again on the Health pillar, we can simplify what owners need to see. Creating a PowerShell script was a simpler alternative to a complex SSRS report that often broke due to patching, and not following best practices. The pack shows a simpler way to bring key insights to owners for Pending Reboots, Expiring PKI certificates, Logical Disk alerts, System Admin summary, and SCOM admin reports including long-running scripts, script errors, SCOM errors, and alert updates report.
Let’s start with some example reports – examples for expiring certificates, Logical Disk, Pending Reboot, System Admin summary, and SCOM admin reports including long-running scripts, script errors, SCOM errors, and alert updates report.
Expiring Certs –
About to expire certificates
Expiring PKI certificates reports
Logical disk alerts –
Shows Server, drive, and % full data
Logical disk alerts report, showing zero for the past 72 hours (over a weekend)
Alerts of servers pending restart, not patched, not rebooted
Pending reboot report lists servers pending restart, not patched, not rebooted alerts
System Admin summary
This is really a consolidation of multiple insights:
Server performance issues
Open ITSM/Remedy tickets
Unhealthy Agents
Pending Reboot, Not Rebooted, Not patched
Disabled/Unhealthy/MaintenanceMode, Repeatedly down Agents
Logical Disk free space alerts
Expiring certificates
AD DC (ADDS) critical alerts
DNS alerts
Group Policy issues
SysAdmin daily summary report example alert
SCOM admin reports
Admin reports have a few separate alert reports, including long-running scripts, script errors, SCOM errors, and alert updates report.
SCOM Admin alerts report example of common SCOM problems
Long running scripts
SCOM Admin long running scripts alerts report example of long-running report workflows to help tune run-time
ScriptErrors showing key SCOM connectivity issues
SCOM Admin script errors to help diagnose report script syntax errors
Useful links
Other blog posts for addendum management packs and capabilities –
As a SME or team lead, ever need to know ‘Proactive Patching alerts’? i.e. What servers need patches applied, aren’t patching, or were missed? This pack builds on three (3) pillars – Health/Security/Compliance, enabling Cyber teams and more. This became an alternate option to a complex pack, with SSRS report, used by a customer to identify systems. The report was long, and had many blank lines/pages, which required a re-write. This pack started with the pending restart monitor directly from the AquilaWeb reboot pack logic. The logic helps SysAdmin/Domain Admin/NOC/NOSC/SOC teams to know when servers need reboots. This need is driven further due to multiple reboots (sometimes) required with Windows monthly updates, and Application updates. Used across multiple customers, this is the first pack enabling a proactive stance to answer the ‘Am I compliant’ question.
David Allen built the ‘Aquilaweb.Support.PendingReboot.Monitor.PendingReboot’ PowerShell monitor, to tell system owners when the pending restart flag was present. Some builds though, make system changes which repeatedly flip the registry key, causing many alerts. Also, downloading the Aquila pack is a trick, as TechNet was retired.
David provided a great idea, which was built upon. This gave rise to the question of, what if the server was not patched, or not rebooted in a period of time? With my Cyber hat on, this became the next piece of content to create. That gave rise to another question – do these scenarios need to reflect in health (monitor), or not (rule)? We’re all about choices, free will, so the pack is built with those options (rules disabled out of the box).
Pending restart monitor XML showing options
The pack is setup to alert with CBS application updates, SCCM/MECM/Config Mgr Endpoint Management updates, and Windows Updates. This has been my experience for the most accurate reflections of alerts on secure builds where Application/System Owner needs to take action.
Last Patch and Last Reboot monitor/rules in the download, are set to 45 days. Tune this value down, if patching occurs at the 30 day mark, increase if you need more time before alerts.
Last Patch Monitor reflecting number of days
Otherwise, download and import into your environment. Depending on your subscription/notification settings, the Proactive set of alerts are built upon the Windows Operating System class. If subscriptions include the class, the notifications are automatic to System/Application owners.
Sometimes as the monitoring admin, you may be responsible to secure your servers, being told from Security/Cyber Teams about new vulnerabilities. The vulnerabilities may be from Tanium, ACAS, Tenable or other security tools. This article is how to help you secure related SCOM web console, and SSRS reporting sites against Unconstrained Delegation vulnerabilities CVE-2020-17049, also AD STIG V-36435.
First we need to identify IF this is a true finding.
Typically this comes from Server/SystemsAdmin with domain admin access:
After identifying SCOM servers with unconstrained delegation (scope of blog post is focused on SCOM in a hybrid scenario (prem/cloud), we need to resolve.
With domain administrator rights, open the Active Directory Users and Computers MMC snap-in.
In ADUC, change Find drop-down to Computers
In the Computer name text box, enter <SCOMServer> and click search
Right click the server in the results box > Select Properties.
Select the Delegation tab.
ADUC view of lab server delegation setting
Select Trust this computer for delegation to specified services only > Use any authentication protocol.
Under Services to which this account can present delegated credentials, select Add.
In the new dialog box, select Users or Computers.
Enter <SCOMServer>, and then select OK.
Click the Add button to add services
Select the w3svc and www processes
Select OK.
ADUC GUI adding services for delegation on SCOM server
Once set in AD, reboot server. Running ‘gpupdate /force’ may not apply AD changes to the server object.
After reboot, reach out to SCOM Admins to test webconsole authentication
From edge browser, go to SCOM web console, typically @ https://<SCOMServer>/OperationsManager
On the Monitoring tab, click on Active Directory dashboard on left
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.