A car mechanic uses battery jumper cables to charge a dead battery.
Do you feel like a mechanic having to jump start the agent configuration like a dead car battery? Assuming the Agent is already installed, you can configure the SCOM agent via PowerShell. Even better when you can PowerShell remote to multiple systems. I hope the PowerShell commands below help you master PowerShell to configure the SCOM side of the MMA agent (house).
powershell
/*
# Find/replace variables to your environment like Kevin Holman’s fragments!
##SCOMMGMTGROUP1##
##SCOMMGMTGROUP2##
##SCOMMGMTSERVER1##
##SCOMMGMTSERVER2##
#
*/
My PowerShell variables to reset SCOM monitors, includes my Addendum and the core – DNS example provided below (thank you Andrew!)
## Grab the MP, get the Monitors and Rules from the MP, then grab all alerts found inside the Monitors/Rules
$SCOMCoreMP = Get-SCOMManagementPack -DisplayName “Microsoft Windows Server 2016 and 1709+ DNS Monitoring” $SCOMAddendumMP = Get-SCOMManagementPack -DisplayName “Microsoft Windows Server 2016 DNS Monitoring Addendum”
First, some background – the Snapshot Synchronization alert just tells you there was a SQL issue running the workflow.
Second, the Snapshot Synchronization alert from a health model perspective, is NOT a critical issue (outage). Create override severity to 1 (warning) to prevent false wake-up calls. I’ll get this to my GitHub repo shortly!
Let’s troubleshoot the alert
Login to server with SSMS installed –
Open SSMS > Connect to the SCOM OpsMgr DB > Click on New Query
Select WorkItemName, b.WorkItemStateName, ServerName, StartedDateTimeUtc, CompletedDateTimeUtc, DurationSeconds, ERRORMESSAGE
from cs.WorkItem a , cs.WorkItemState b
where a.WorkItemStateId= b.WorkItemStateId
and WorkItemName = ‘SnapshotSynchronization’
Solution:
The jobs may show Succeeded by the time you login to SQL = EOJ (end of job)
If Failed is latest date/timestamp, re-run the task “Request Snapshot Synchronization” which can be found when we select “Management Configuration Service Group” in the below mentioned view.
View:
From Monitoring Tab > Click on Operations Manager folder > Click on Management Group Health widget > Highlight unhealthy state from Management Group Functions.
Click on the ‘Request Snapshot Synchronization’ task to execute the Stored Procedure “SnapshotSynchronizationForce” on the OpsMgr DB.
NOTE: There are two tasks with same name but with different targets i.e. ‘Management Configuration Service Group’ and ‘Management Configuration Services’
The other task can be found on below view after selecting the Management Server you want the Task to be executed on
View:
From Monitoring Tab > Expand Operations Manager folder > Expand Management Configuration Service folder > Click on Services State view
Create Override for the alert
To change snapshot monitor to warning
From SCOM Console > Authoring Tab
Expand ‘Management Pack Objects’ > Click on Monitors
In the ‘Look for:’ bar type Snapshot synchronization state and hit enter
Monitor name = Snapshot synchronization state
Right click on monitor > Overrides > Override the monitor > For all objects of class
Click checkbox for Severity > change Critical to Warning
Click Edit – add comment – i.e. date/time changing to warning
Like Meguiar’s cleaner wax to your car’s finish, this post will help utilize cleaner PowerShell to help reset monitors and rules
Cleaner PowerShell supplied by Andrew Bradley that’s helped simplify the PowerShell code included resetting/closing monitors and rules via a method call. Hard to believe I’ve been quiet on the blog for the past year, as I’ve been working on SCOM management pack addendums. The ‘cleaner PowerShell’ is being integrated into the various addendums.
have been helpful with many customers, by building out better ways to monitor, clean up alerts, and create daily reports. The Addendum packs add report key insights for many 1P (1st party) Microsoft authored management packs.
Methods
Cleaner PowerShell to help reset monitors and rules
## Grab the MP, get the Monitors and Rules from the MP, then grab all alerts found inside the Monitors/Rules
$SCOMCoreMP = Get-SCOMManagementPack -DisplayName “System Center Core Monitoring”
## Grab the MP, get the Monitors and Rules from the MP, then grab all alerts found inside the Monitors/Rules
$SCOMCoreMP = Get-SCOMManagementPack -DisplayName “Microsoft Windows Server 2016 and 1709+ DNS Monitoring”
$SCOMCoreRules = $SCOMCoreMP.GetRules()
$SCOMCoreMonitors = $SCOMCoreMP.GetMonitors()
Adding parameters to datasource/probeaction moduletypes
This post is adding parameters to datasource (DS) or probeaction (PA) moduletypes. Sorry, found this draft from last year that I never published. 🙁 I’m in the ‘missing functionality’ boat. Some would say I’m a dreamer, a good system admin, a car guy who has different ideas than the manufacturer, or something altogether different — you decide 🙂 Hope this blog post helps monitoring experts that author more functionality than what was delivered. Specifically adding parameters to datasource/probeaction moduletype NOT delivered in the OotB functionality?!
Adding parameters to datasource/probeaction moduletypes
First – What is needed
Second – Verify dependencies required for a workflow
Third – Build on example ‘datasource’
Fourth – Configure Monitor/Rule to use Datasource/ProbeAction
Let’s go through step by step through ‘adding parameters to datasource/probeaction moduletypes’ to customize a data source. The datasource requirements are to include/verify the following parameters” TimeOut,TimeOutInMS,MatchCount,SampleCount (match/sample count are intended for rules/monitors)
Pre-reqs (what’s needed for a ModuleType to function)
Working Script – PowerShell/BASH/Perl/SH/KSH
ScriptArgs required at runtime
Other Configuration, or Overrideable Parameters
Using configured parameters properly
Verify ProbeActions (PA) inside DS have relevant parameters
Easiest way to summarize adding a configuration parameter
Must be added to Configuration, OverrideableParameters,ModuleImplementation,
When taking an Out of the box’ OotB’ moduletype to modify, where parameter(s) MUST be used in UnitMonitorType,Rule,Monitor
Quick background for MatchCount/SampleCount:
When adding parameters to datasource/probeaction moduletypes, it’s good to know why this is part of the conversation to be added to monitoring design/implementation.
MatchCount comes in handy for repeated failures BEFORE alerting (count 5 events before alerting)
SampleCount comes in handy for counting number of failed workflows BEFORE alerting (run workflow 6 times failing before alerting)
Example Unix.ShellCommand.Invoke.Script DataSource
Requirement = Add MatchCount/SampleCount (or TimeOut to the PA ProbeAction)
Unseal, and open Microsoft.Unix.ShellCommand.Library.xml in NotePad++, VStudio, (or your favorite XML editor)
Screenshot of default Microsoft.Unix.ShellCommand.Invoke.DataSource
TimeOut and TimeOutinMS are baked in. We begin by adding MatchCount/SampleCount
Adding MatchCount/SampleCount for Configuration, OverrideableParameters, and Module Implementation for DS/PA
How to add MatchCount/SampleCount syntax
Adding MatchCount/SampleCount for Configuration, OverrideableParameters, and Module Implementation for DS/PA
NOTE – sometimes you don’t find an example!
This part gets complicated – how far down the rabbit hole do you need the parameters?
Does the DS workflow only need the respective parameters?
Do you have to add to the corresponding PA’s called in the workflow?
Add MatchCount/SampleCount to OverrideableParameters (if you want capability to override)
<OverrideableParameter ID=”MatchCount” Selector=”$Config/MatchCount$” ParameterType=”int” />
<OverrideableParameter ID=”SampleCount” Selector=”$Config/SampleCount$” ParameterType=”int” />
Add MatchCount/SampleCount to DS MemberModule
<MatchCount>$Config/MatchCount$</MatchCount>
<SampleCount>$Config/SampleCount$</SampleCount>
Add MatchCount/SampleCount to PA Configuration
<xsd:element name=”MatchCount” type=”xsd:unsignedInt” maxOccurs=”1″ minOccurs=”0″ xmlns:xsd=”http://www.w3.org/2001/XMLSchema” />
<xsd:element name=”SampleCount” type=”xsd:unsignedInt” maxOccurs=”1″ minOccurs=”0″ xmlns:xsd=”http://www.w3.org/2001/XMLSchema” />
Add MatchCount/SampleCount to PA MemberModule
<MatchCount>$Config/MatchCount$</MatchCount>
<SampleCount>$Config/SampleCount$</SampleCount>
Unix.ShellCommand.Invoke.Script
Alternate example for monitors, the SQL Windows Replication mgmt pack has a good UnitMonitor/UnitMonitorType example – Microsoft.SQLServer.Replication.Windows.Monitoring.xml
Find example by searching unsealed management pack repository (use Tyson’s SCOMHelper PowerShell module to unseal mp/mpb’s to facilitate a better unsealed mp search) https://monitoringguys.com/2019/11/12/scomhelper/
Detected malicious verification code when verifying element – ever run into this scenario while authoring?
Ever run into the ‘detected malicious verification code’ error while authoring? I ran into the malicious verification error authoring, and couldn’t find any content for this error while authoring a pack.
Watch your copy/paste’s with additional monitoring changes to prevent ‘detected malicious verification code’ errors
In my authoring example, I received the ‘detected malicious verification code error’ after adding Rules, Datasources, and WriteActions (including tasks). I was copying and pasting DataSources (DS) and WriteActions (WA), thought I had it all. Uploaded > got the error, and GRR! Hopefully this will help others authoring to know what to check to get the management pack uploaded.
Simply put – Watch out for typo’s to avoid ‘detected malicious verification code’ errors!
I stumbled across a few websites, but nothing really pointed out to what caused the ‘detected malicious verification code error’ when uploading a management pack. First, check monitor and rules to verify the DS/WA are called correctly (no errors in file names. Check the Tasks as well as DisplayStrings, to make sure everything matches.
Error Seen when uploading Management pack from SCOM Console GUI regarding ‘detected malicious verification code’ error
<ManagementPackNameHere> Reports could not be imported.
If any management packs in the Import list are dependent on this management
pack, the installation of the dependent management packs will fail.
Please take ‘caution using Tags/Notes extending classes’. Please read below if you use Tags/Notes on SCOM classes. Ran across examples where SCOM Class Properties were used for tags that used the .Notes field on various classes, causing orphaned properties, NOT removed from OperationsManager database.
Background
The Microsoft.Windows.Computer Class (insert class here) is updated using Tim McFadden’s blog. This can cause issues with orphaned classes in the database because it is not currently handled as part of the stored procedure (i.e. the Notes property classes do not get marked for deletion).
First, identify which classes have Notes property. Start from Management Server (MS) via PowerShell. See attached TXT for additional examples to check and add/remove Notes Property on additional windows classes.
Set Notes property for Windows Operating System server
Second, we need to see how to set and clear the value, in order to clean up the Operations Manager database, to remove the orphaned instances. The example below sets the value for one (1) server to ‘Production’.
PropertyAccessRights : Unknown
Parent : Microsoft Windows Server 2016 Standard
Type : Notes
Value : Production
Id : 00000000-0000-0000-0000-000000000000
ManagementGroup : SCOM2016
ManagementGroupId : e39f5f53-9fbb-9d7f-4bfe-5f0324630ae5
PropertyAccessRights : Unknown
Parent : Microsoft Windows Server 2016 Standard
Type : Notes Value : (null)
Id : 00000000-0000-0000-0000-000000000000
ManagementGroup : SCOM2016
ManagementGroupId : e39f5f53-9fbb-9d7f-4bfe-5f0324630ae5
Have a happy Holiday!
Good luck, hopefully this scenario isn’t something that impacted the monitoring environment!
Ever get ‘command not found’ errors when calling a command on a machine? Many times, these errors are related to what is defined on said machine. So with monitoring tools like SCOM, ALA, Azure Automation, BMC Patrol, the ID used in monitoring rely on filepaths defined on the local server (holds true for Windows/UNIX). Because sometimes even ls, awk, dir, etc. if their various bin directory filepaths are NOT specified as a security hardening measure. The result of STIG/Security hardening is ALL scripts/commands require a fully qualified filepath.
Fully qualifying command paths holds true for Windows and UNIX, from generic OS commands, AND also application specific files (including an executable). Updates are required if you want to supply the short name command. Add the full filepath to PATH= statement. The alternative is to fully qualify in your SCOM mgmt. pack, so the command will run regardless of user, as long as the path is correct.
Check for specified shell
First, let’s check UNIX to see what shell is specified for user(s).
Second, log into your UNIX server, and check files type: ls -al .* | more
Use ls -al | more to see what PATH files are in the user directory
Third, another option with less output
example: ls -al .*profile
What’s my path? Use command ls -al .*profile to find which profile(s) exist
Fourth, Look for the shell defined for the user account
On my server, SCOM user is bash shell (but I do NOT have a .bash_profile, only a .profile (also note NO .ksh_profile) ) Knowing what profiles are configured for user account will help define what is inherited from the OS, (automatically included). Leverage when calling commands in your management packs for custom rules/monitors.
In conclusion, if executable is NOT in the filepath variable, you have two ways to resolve the issue:
Create a .bash_profile
Call bash/ksh shell in your script or command line: bash; <commandhere>
To check path:
UNIX $PATH vs. Windows $ENV:path
UNIX example – ‘echo $PATH’ from UNIX ssh session/logon
UNIX what’s my path? Use echo $PATH
Windows PowerShell example
What’s my path? Windows PowerShell example of $PATH
Here’s my .profile that sets up SCOM user (only /bin shown)
First, use SCOM helper for advanced administration (a shameless plug to Tyson & MonitoringGuys blog!)
Me as Picard being hit by SCOM helper
In case you didn’t know, as I may be the last off the airplane, it’s time to talk about SCOMHelper. Things you forget, like using a power drill, versus manual screwdriver, all because you’re familiar with the old trusty screwdriver. Man, I think I upgraded past the power drill with a cord, to a lithium-ion powered impact drill with SCOMHelper.
Time to use SCOM helper for advanced administration, and get it installed!
Pretty simple install. I would recommend adding to monitoring team server, jumpserver, from SCOM MS, or console enabled, Operations Manager plug-in installed machines.
Open PowerShell (as admininstrator)
Type: Install-Module -Name SCOMHelper
Example output
PS C:\> Install-Module -Name SCOMHelper
Untrusted repository
You are installing the modules from an untrusted repository. If you trust this repository, change its
InstallationPolicy value by running the Set-PSRepository cmdlet. Are you sure you want to install the modules from
‘PSGallery’?
[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is “N”): A
If SCOMHelper is already installed
Open PowerShell (as admininstrator)
Type: Install-Module -Name SCOMHelper -Force
(You may get the untrusted repository message if you answered Y (yes) or N (no)
Example output
PS C:\> Install-Module -Name SCOMHelper -Force
PS C:\>
Install SCOM Helper -force
Just one example –
Use the Unseal-SCOMMP to unseal your MP and MPB’s for your SCOM mgmt pack repository
Have fun playing!
If you’re administering management groups, I can pretty much guarantee you will need these:
Export-SCOMEffectiveMonitoringConfigurationReport
Export-SCOMOverrides
Get-SCOMHealthCheckOpsConfig
Get-SCOMRunningWorkflows
New-SCOMClassGraph
Remove-SCOMObsoleteReferenceFromMPFile
Show-SCOMPropertyBag
I’m off to unseal my repository, so my Notepad++ searches can find more examples for new authoring functionality.
It’s that time to figure out the ConfigMgr SMS role alerts – If you are monitoring your SCCM/MECM environment, then you get role failure alerts. Many times, the Operations Helpdesk, NOSC, NOC, SOC, etc. will get alerts when various roles fail on the Configuration Manager platform. The common ask is why, what do you see, etc. Much like Exchange, ConfigMgr internalizes the checks that are seen in the console as registry keys or events documenting said degraded component/feature. Helping the MECM administrator understand the failure is key to decoding how to notify administrator, and when the helpdesk needs to act on ‘ConfigMgr SMS role alerts’.
Example – MECM/SCCM looks at replication probe action state $Config/RoleName$
Example MECM Service Monitor for role alerts
The role check is based on a variable of the RoleName in a registry key that the application updates.
MECM Monitor Config
This is the origin of ConfigMgr SMS role alerts
HKLM:SOFTWARE\Microsoft\SMS\Operations Management\SMS Server Role\$Config/RoleName$\Availability State
Decoder ring:
1 is critical state
2,3,4 are warning states
If more details are needed, download SCCM/MECM Management Pack for SCOM here
Use Tyson’s SCOM Helper pack to unseal, and inspect XML.
Once you know the origin of the ConfigMgr SMS role alerts, you can begin tuning the MECM alerts to your environment. Understanding role alerts will help both teams understand MECM application health. First, use MECM application health to trend alerts/outages. Second, leverage maintenance mode schedules, or MM scripts to NOT monitor for common administration tasks. From my experience, the alerts are the result of MECM Admins maintaining the application – common actions like building application/package lists, cleanup actions, site maintenance, backups, etc. Lastly, set up a subscription to notify after the tuning discussion. See my blog on building a subscription for more details.
