Latest revision first includes a EventID 2502 monitor for scavenging failed. Second, the monitor has count logic (setup to alert with 2 events in 30 minutes). Third, EventID 2501 rule details scavenging totals. Lastly, built a weekly report to summarize the scavenging alerts (cliff notes!).
Some quick ‘how-to’ setup DNS scavenging
Example of RegKey showing that Scavenging is setup – note Scavenging Interval key
Example of AD integrated DNS setup with 21 day scavenging interval, and prompts to configure (click OK twice)
Import management pack, and run DNS scavenging.
Verify scavenging alerts
SCOM Monitoring Tab > Active Alerts > ‘Look for:’ scavenging
Example output
Additional SCOM PowerShell commands
Run PowerShell commands from the SCOM management server (MS)
Time to integrate your Monitoring tools to ITSM tool. First, this blog post documents ‘ServiceNow Event integration’. Second, let’s explain the common acronym in my experience is SNOW/SNow. Third, some background – ServiceNow has been around for some time as an Information Technology Service Management (ITSM), and discovery tool. As a SaaS solution, companies can purchase a subscription and integrate tools via RESTAPI to create/update/close events or incidents.
First, let’s begin to discuss SCOM notification methods. SCOM2022 adds a new capability with Teams integration. Second, most people are familiar with notification methods leveraging Email (html or not), perhaps SMS, but not so much command channel, calling some script in shell, PowerShell, etc. Generally, the command channel is basically a post processing script capability to execute notifications. Third, example tools where command channel might be used – BMC BEM (BMC Event Manager), BMC Remedy, xMatters, DerDack; SNOW integration within SCOM, using notification channels. Lastly, SaaS solutions (vendors like xMatters, and ServiceNow) allow RESTAPI crafted requests to take actions.
SNOW prerequisites
1) ServiceNow User/Password (or API key)
2) SNOW RESTAPI PowerShell needs to securely access credentials
For the Incident PowerShell, we store Credentials within Windows Credential Manager
3) Network connectivity to SaaS provider (use PowerShell test-netconnection from SCOM MS to test connectivity over whatever port(s) vendor requires.
4) ServiceNow CallerID GUID
5) Production and Test URL’s (also required for network connectivity tests)
6) Access to SNOW UI to verify required fields and values for the script parameters.
Do your SCOM users need to know if a server is in scheduled maintenance? This came about as Aris asked questions.
First, let’s discuss specific maintenance mode and maintenance schedule scenarios users might ask. Second, determining IF scheduled maintenance enabled, running, about to run. Third, how does another user know when scheduled maintenance ends, allowing action and decision point to add/extend server maintenance. Fourth, whenever scheduled maintenance entered by one user, is NOT automatically seen by other roles. While product guidance states ‘maintenance schedules be added by someone in SCOM admin group’, self-service users still need visibility. Lastly, can we figure out a way to answer these questions. Given these points, users to be able to see server maintenance details. Also, can solution adhere to best practice ‘no alerts during planned maintenance’.
Second, another change with the repo’s was a ‘whitespace audit’ encoded characters, or ‘data concealment’. See AT&T link CyberSecurity Link
Third, after whitespace we focused on script/workflow efficiencies seen in large enterprise environments. While Efforts began in December, the workflow efficiencies sprint resulted in two sets of improvements.
1) Added ‘Reset Monitors Script base code’ $Age variable
What does this mean?
Simply put $Age allows admins to define monitor age before resetting.
The default is 1 (day), but can be specified in the script to tailor to requirements.
Example
$Age = [DateTime](Get-Date).AddDays(-1)
2) Beyond incorporating $Age into the reset monitor logic, the packs utilize logic for a much faster runtime (~90%+).
What does this mean?
Updated logic quickly gathers unhealthy monitor objects, by leveraging ‘Get-SCOMManagementPack‘ and then ‘Get-SCOMClass‘, before passing to ‘Get-SCOMClassInstance‘.
Example PowerShell
## Grab the MP, get the Monitors and Rules from the MP, then grab all alerts found inside the Monitors/Rules
$SCOMCoreMP = Get-SCOMManagementPack -DisplayName “Microsoft Windows Server DNS Monitoring”
Use the Tangible SCOM management pack to monitor logins and ProV application registration issues. First, the management pack configures Seed class discovery. Second, the pack includes rules/monitors for Tangible ProV software. Third, rules and monitors for 2802 ‘Could not validate product key’ and 4402 ‘Could not validate the contents of user logon request context: AS-REQ contains an invalid or unknown username type’ events. Fourth, the service monitor, which uses Kevin Holman’s fragment library for service recovery scripts/rules. Fifth, scheduled and on-demand daily reports for audit and record keeping purposes. Lastly, alert cleanup logic, to reduce admin burden and overhead.
Reference the Tangible vendor’s website – Tangible ProV application website
NOTE: This may not apply for everyone, as the ProV application ‘Auto-provisions Active Directory user accounts for visitors or new employees whenever they want to work from one of your PCs.’
The Daily report piece of the pack makes things easier answering ‘what happened in the last 24-72 hours’ question. Gathers open/closed insights and organizes alerts.
Screenshot of the daily report
Report example of insights (in text)
Open ProV alerts = 13
Since last report run: #———————– Total ProV alerts = 23 Auto-closed monitors = 22 Auto-closed rules = 0
My thanks to Aris Somatis for his deep dive reviewing the packs with me, particularly new use cases. The PowerShell below builds on Scott Murr’s initial TechNet published logic from years back. Consequently, the reset logic provides a ‘manual intervention required’ alerting/monitoring system.
Improving SCOM monitor reset logic
Calling the reset method has been a game changer for my customers – including operators, system and application owners!
Background
Scott’s reset logic, from SCOM2012, helped administrators reset unhealthy monitors where alerts may have been closed. Because Scott leveraged the ResetMonitoringState method, the community gained a way to keep true health. Additionally, many administrators and engineers built custom management packs to provide solutions. Second, the addendum packs blog brought in more options – best practices, lessons from the field (and customers), and health model accurate alerting for what was really broken in the environment. Third, addressing ‘gaps’ or ‘blind spots’ from product teams. As a result of NEW monitoring, the packs may include: rules/monitors, datasource/writeAction (DS/WA) workflows, recovery tasks and automation, count logic monitors, overrides, discoveries, and groups. Thirdly, to take monitoring to the next level. To top that off, with very little/NO cost compared to competitors!
PowerShell code
Aris’s Age use case takes this even further. Using monitor age allows further analysis to dial down ‘monitor reset’ to object is X days old. Comparatively, the 24-72 hour setup default is used in the addendums, so Age provides a second option. Third option can rely on SCOM’s built-in cleanup, but that’s typically 14-30 days. Overall, flexibility is a good thing.
# Specify age variable for your environment
$Age = [DateTime](Get-Date).AddDays(-7)
PowerShell code snippet
First, the reset logic can pivot on the age requirement. Then, adjust the Age variable per requirements. Third, figure out which method applies to gather a unique list of classes, whether by partial string(s), or by management pack name(s).
Set age variable (how long ‘OLD’ monitors might be stale and need reset)
# Example sets $Age variable to 7 days ago (-7)
$Age = [DateTime](Get-Date).AddDays(-7)
Unpack two different ways to gather classes for monitors to reset
# When common string name exists in all classes
Example DFS/FileServices packs all have one of the three strings:
0
Found 0 unhealthy monitors for class Microsoft.SystemCenter.HealthServicesGroup
1
Found 1 unhealthy monitors for class Microsoft.SystemCenter.HealthServiceWatcher
Resetting Health State on ' + Microsoft.SystemCenter.HealthServiceWatcher:Microsoft.SystemCenter.AgentWatchersGroup;5e0
4f804-8b71-6eb6-0101-dcbb58022498 + '
Guid
----
0218d239-3d37-f9b1-75d2-6d52c2c7c0c1
Fresh off the press, right to your door, just in time for that gift for your special someone! Time for new updates to keep you ever-green’d, up to date, fixes, etc. ;-P
Holman updated his SCOM.Management pack for SCOM2022 UR2
Multiple packs with multiple updates. Removed debug detail for DS/WA (Data Source/Write Action workflows) Health Explorer outputs, simplified mgmt pack recovery tasks for single WA script.
Updated DNS2012R2 Addendum overrides. Learned a few new things with Overrides workspace views, and why Authoring pane > Management pack Objects > Overrides may not load.
When your management pack has improper overrides, expect the loading icon. This may be caused due to overrides, whether error is with target, class/rule/monitor.
Sometimes, an Object of class error gets your hopes up (pointing at a non-existent object).
Example when Overrides loads properly
If Overrides view will not load, try creating a workspace view for Overrides.
Navigation Steps:
From SCOM Console
Click on My Workspace
Right Click > New > Overrides Summary View
Select checkbox ‘with a specific override management pack’ checkbox, then the ‘specific’ link to choose management pack(s).
Choose unsealed management pack(s) with overrides
Can select all – OR pick a few to see what loads without errors
Click OK
If you get the loading screen and error, now begins the pack analysis.
Clicking on the ‘Show’ link points to a non-existent object
Microsoft.EnterpriseManagement.Common.ObjectNotFoundException: An object of class ManagementPackClass with ID 76e2559c-aaf4-b1ec-60cf-d40ab4102fbc was not found.
How did I know that?
Run get-SCOMClassInstance command from PowerShell or Operations Manager shell
Example output of ‘get-SCOMClassInstance -ID “76e2559c-aaf4-b1ec-60cf-d40ab4102fbc” ‘
Work on the Overrides of the affected XML packs, and Import.
Once corrected, the Workspace view loads successfully, finite!
‘NiCE VMware addendum’ enhances VMware monitoring, tuning alerts to ‘manual intervention’ required alerting. The NiCE folks have been around for some time as a trusted Microsoft partner, creating additional monitoring functionality across Microsoft products. Having completed a number of projects implementing the VMware pack, it’s time to share the configuration and alert report capabilities.
Key breakdown of VMware ESX environment monitoring
Adjustments to vendor pack to further the mantra ‘alert when manual intervention required’.
Set monitor alerts to multiple samples over an hour (i.e. compute and performance of ESX environment)
Reports by team (requires regular expression updates for environment servers owned by each team)
Monitor reset logic, and service monitorType (count logic for X failures over Y time, before alert)
Overrides to change vendor pack provided discoveries, rules, monitors
Remove alert noise for unmanaged objects in ESX environment
Customize pack for environment
Customize the ‘NiCE VMware addendum’ pack for specific environment. This means updating group discoveries, and GUIDs for group specific overrides. Further updates are required to update server naming conventions for team virtualization reports.
Classes/groups created for pack
Discoveries
Breakout of Discoveries that need pattern updates to match
Find/Replace ##ESXHostDataStoreNamingConventions## with names to exclude
Example of regular expressions for multiple customers
Update disable guest machine alerts
Disable guest machines in ESX environment to disable alerts.
‘File Services Addendum’, named Microsoft Windows Server FileServices 2016 Addendum, adds replication health/backlog script, seed and group classes, replication/service monitors, recovery tasks, and overrides to tune monitored environment.
Addendum assumes the file services version agnostic version 10. pack is installed.
Looking at XML file in Notepad++, the pack references are what packs the workflows refer to (other management packs). Kevin Holman taught building backwards compatibility with MP authoring. Backwards compatibility allows SCOM2012+ import without errors. To take this one step further, the v10.0.0.0 file services packs referenced represent the version agnostic packs.
NOTE: File Services Addendum references may need updates if the whole file services management packs are NOT installed.
References screenshot
Addendum logic
Capabilities
Daily report and close automation, on-demand tasks for reports
DFS backlog script errors
SmSvc, DFSN, DFSR service recovery and rule alerts (from Holman fragments library)
DFS replication backlog watcher, script, alerts
Notepad++ screenshot
Next, we look at the group/class discoveries
Update Class/Group discoveries for DFS servers or script install paths for replication script.
Find and replace FilePath and ##DFSServerNamingConvention## variable.
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.