Deciding ‘Event Collection vs. Alert’ rule

Question example of two cartoon people discussing something. Both have thought bubble cartoons looming overhead.

Ever run through an event log scenario deciding ‘event collection vs. alert rule’ is the way to filter out the needle from the haystack? There’s a few ways to do this with Monitoring tools. If you’re cloud centric, a KQL query (assuming you’re collecting the event logs, if you’re using Operations Manager (SCOM), there’s a few ways to consume the events. SCOM ACS is basically a DB for collecting Security events, and typically is an unused feature in SCOM by most customers. Kevin Holman’s had many blog posts for ACS, testing the filter, as well as a management pack (MP) fragment (blog here, GitHub fragment library here).

Let’s walk through criteria deciding ‘event collection vs. alert rule’:

Do the event(s) happen often? If so, how often?
Can you filter the event description to limit the amount of gathered event?
Do you need match count or samples before action required? (i.e. count x events in y time)
Is there a regulatory or compliance requirement to collect every event?
Is this something you want to visualize with PowerBI?
For better visualizations, would the EventID help view/sort data in a tabular output? i.e. Think PowerShell property) as well as TimeRaised/TimeGenerated, and Event Description

Example – DC Security events

When there is a regulatory requirement to collect events, we need to decide ‘event collection vs. alert rule, and IF we can filter for specific pieces of the event. Holman has examples of alert parameters, and dynamic data, which are very useful to get the needles out of the haystacks. Depending on your goals, use event parameters, or leverage CustomFields in the alert to build required fields.

Depending on the requirements, event collection is useful to collect related EventID’s with RegularExpressions. Use Event rules WHEN action is required. Leverage Regular expressions help filter what we collect (via event collection or alert rule. By extension, utilize CustomFields in the alerts to help the presentation or SQL query towards a PowerBI report.

Let’s talk about regular expressions examples for rules (or monitors)

MatchesRegularExpression

<Expression>
<RegExExpression>
<ValueExpression>
<XPathQuery Type=”String”>EventDescription</XPathQuery>
</ValueExpression>
<Operator>MatchesRegularExpression</Operator>
<Pattern>^(Security ID:.*admin*)|^(Security ID:.*[des]a*)$</Pattern>
</RegExExpression>
</Expression>

<Expression>
<RegExExpression>
<ValueExpression>
<XPathQuery Type=”UnsignedInteger”>EventDisplayNumber</XPathQuery>
</ValueExpression>
<Operator>MatchesMOM2005BooleanRegularExpression</Operator>
<Pattern>^(4625|4740)$</Pattern>
</RegExExpression>
</Expression>

Contains example

<Expression>
<RegExExpression>
<ValueExpression>
<XPathQuery Type=”String”>EventDescription</XPathQuery>
</ValueExpression>
<Operator>ContainsSubstring</Operator>
<Pattern>Proactive DailyTasks ADDS Monitors close automation for</Pattern>
</RegExExpression>
</Expression>

<Expression>
<RegExExpression>
<ValueExpression>
<XPathQuery Type=”String”>Params/Param[2]</XPathQuery>
</ValueExpression>
<Operator>ContainsSubstring</Operator>
<Pattern>dnsserver</Pattern>
</RegExExpression>
</Expression>

DoesNotContain example

<Expression>
<RegExExpression>
<ValueExpression>
<XPathQuery Type=”String”>EventDescription</XPathQuery>
</ValueExpression>
<Operator>DoesNotContainSubstring</Operator>
<Pattern>None</Pattern>
</RegExExpression>
</Expression>

Holman MP Fragment example of specific EventID:

<Rule ID=”Rule.StateChangeAlerts” Enabled=”true” Target=”SCOMMagementServer.Class” ConfirmDelivery=”true” Remotable=”true” Priority=”Normal” DiscardLevel=”100″>
<Category>EventCollection</Category>
<DataSources>
<DataSource ID=”DS” TypeID=”Windows!Microsoft.Windows.EventCollector”>
<ComputerName>$Target/Host/Property[Type=”Windows!Microsoft.Windows.Computer”]/NetworkName$</ComputerName>
<LogName>TestAPP</LogName>
<AllowProxying>false</AllowProxying>
<Expression>
<And>
<Expression>
<SimpleExpression>
<ValueExpression>
<XPathQuery Type=”UnsignedInteger”>EventDisplayNumber</XPathQuery>
</ValueExpression>
<Operator>Equal</Operator>
<ValueExpression>
<Value Type=”UnsignedInteger”>600</Value>
</ValueExpression>
</SimpleExpression>
</Expression>
<Expression>
<SimpleExpression>
<ValueExpression>
<XPathQuery Type=”String”>PublisherName</XPathQuery>
</ValueExpression>
<Operator>Equal</Operator>
<ValueExpression>
<Value Type=”String”>APP Test Log Monitoring</Value>
</ValueExpression>
</SimpleExpression>
</Expression>
</And>
</Expression>
</DataSource>
</DataSources>
<WriteActions>
<WriteAction ID=”CollectToDB” TypeID=”SC!Microsoft.SystemCenter.CollectEvent” />
<WriteAction ID=”CollectToDW” TypeID=”SCDW!Microsoft.SystemCenter.DataWarehouse.PublishEventData” />
</WriteActions>
</Rule>

Lastly, let’s talk about the use of CustomFields to add additional data to the alert, but NOT in the event description (Holman’s blog here)

For the tabular view of alert data (from PowerShell as with SQL query of Alerts view, we might need to display the data, such as EventDisplayNumber, TimeRaised, Message, (alternate is Parameters, or UnformattedDescription). Additionally, check alert output details, from the SCOM MS in PowerShell via get-SCOMAlert -name “MonitorDisplayNameHere” | fl | more

Leverage Custom Fields to add

EventID $Data/EventDisplayNumber$

Event Category $Data/EventCategory$

Happy Authoring!

Additional links

How to collect events – but not ALL the events?

https://learn.microsoft.com/en-us/answers/questions/69667/scom-event-collection-rule

Positive SSL by Comodo SSL

February 25, 2023March 7, 2023

Check your delegation settings

Sometimes as the monitoring admin, you may be responsible to secure your servers, being told from Security/Cyber Teams about new vulnerabilities. The vulnerabilities may be from Tanium, ACAS, Tenable or other security tools. This article is how to help you secure related SCOM web console, and SSRS reporting sites against Unconstrained Delegation vulnerabilities CVE-2020-17049, also AD STIG V-36435.

First we need to identify IF this is a true finding.

Typically this comes from Server/SystemsAdmin with domain admin access:

From PowerShell run:

Get-ADComputer -LDAPFilter
“(userAccountControl:1.2.840.113556.1.4.803:=524288)”

After identifying SCOM servers with unconstrained delegation (scope of blog post is focused on SCOM in a hybrid scenario (prem/cloud), we need to resolve.

With domain administrator rights, open the Active Directory Users and Computers MMC snap-in.

In ADUC, change Find drop-down to Computers
In the Computer name text box, enter <SCOMServer> and click search

Right click the server in the results box > Select Properties.

Select the Delegation tab.

ADUC view of lab server delegation setting

Select Trust this computer for delegation to specified services only > Use any authentication protocol.

Under Services to which this account can present delegated credentials, select Add.

In the new dialog box, select Users or Computers.

Enter <SCOMServer>, and then select OK.

Click the Add button to add services

Select the w3svc and www processes

Select OK.

ADUC GUI adding services for delegation on SCOM server

Once set in AD, reboot server. Running ‘gpupdate /force’ may not apply AD changes to the server object.

After reboot, reach out to SCOM Admins to test webconsole authentication

From edge browser, go to SCOM web console, typically @ https://<SCOMServer>/OperationsManager

On the Monitoring tab, click on Active Directory dashboard on left

Verify authentication works

Documentation

Pentestlab – Detecting Unconstrained Delegation Exposures in AD Environment

Petri.com find and block unconstrained delegation

Learn.Microsoft.com unconstrained kerberos article

Explanatory documents on what/why

Remove Unconstrained Kerberos Delegation

February 1, 2023February 2, 2023

Configure MMA agent via PowerShell

A car mechanic uses battery jumper cables to charge a dead battery.

Do you feel like a mechanic having to jump start the agent configuration like a dead car battery? Assuming the Agent is already installed, you can configure the SCOM agent via PowerShell. Even better when you can PowerShell remote to multiple systems. I hope the PowerShell commands below help you master PowerShell to configure the SCOM side of the MMA agent (house).

/*
# Find/replace variables to your environment like Kevin Holman’s fragments!
##SCOMMGMTGROUP1##
##SCOMMGMTGROUP2##
##SCOMMGMTSERVER1##
##SCOMMGMTSERVER2##
#
*/

$SCOMAgent = New-Object -ComObject AgentConfigManager.MgmtSvcCfg
$SCOMAgent.GetManagementGroup(“##SCOMMGMTGROUP1##”);$SCOMAgent.GetManagementGroup(“##SCOMMGMTGROUP2##”)

# If mgmt groups are incorrectly set
$SCOMAgent.RemoveManagementGroup(“##SCOMMGMTGROUP1##”)
$SCOMAgent.RemoveManagementGroup(“##SCOMMGMTGROUP2##”)

restart-service healthservice

# Domain
$SCOMAgent.AddManagementGroup(“##SCOMMGMTGROUP1##”,”##SCOMMGMTSERVER1##”,5723)

# Verify agent config
$SCOMAgent.GetManagementGroup(“##SCOMMGMTGROUP1##”)
# If you have a second management group

$SCOMAgent.GetManagementGroup(“##SCOMMGMTGROUP2##”)

# Restart and test connectivity
restart-service healthservice

# Check connectivity
test-netconnection -port 5723 -computername ##SCOMMGMTSERVER1##

February 1, 2023

SCOM Monitor reset logic

Ever want to reset SCOM monitors, and wish it was just a simple Reset Button for unhealthy monitors?

I’ve been using Scott Murr’s TechNet gallery loop to maintain my alerts, and ensure monitors are healthy for all my management packs.

The blurb I put in my DS/WA scripts to reset SCOM monitors. I build on Andrew’s methods I didn’t realize (just think much uglier code!)

Cleaner PowerShell to help reset monitors and rules

My PowerShell variables to reset SCOM monitors, includes my Addendum and the core – DNS example provided below (thank you Andrew!)

## Grab the MP, get the Monitors and Rules from the MP, then grab all alerts found inside the Monitors/Rules

$SCOMCoreMP = Get-SCOMManagementPack -DisplayName “Microsoft Windows Server 2016 and 1709+ DNS Monitoring”
$SCOMAddendumMP = Get-SCOMManagementPack -DisplayName “Microsoft Windows Server 2016 DNS Monitoring Addendum”

$SCOMCoreRules = $SCOMCoreMP.GetRules()
$SCOMCoreMonitors = $SCOMCoreMP.GetMonitors()
$SCOMAddendumRules = $SCOMAddendumMP.GetRules()
$SCOMAddendumMonitors = $SCOMAddendumMP.GetMonitors()

$SCOMCoreReportAlerts = Get-SCOMAlert | ? { ($_.Name -in $SCOMCoreRules.DisplayName) -or ($_.Name -in $SCOMCoreMonitors.DisplayName) }
$SCOMCoreReportAlerts.Count
$SCOMAddendumReportAlerts = Get-SCOMAlert | ? { ($_.Name -in $SCOMAddendumRules.DisplayName) -or ($_.Name -in $SCOMAddendumMonitors.DisplayName) }
$SCOMAddendumReportAlerts.Count

$SCOMOpenReportAlerts = $SCOMAddendumReportAlerts | ? { ( $_.ResolutionState -ne “255” ) }
$SCOMOpenReportAlerts.Count
$SCOMOpenAddendumReportAlerts = $SCOMAddendumReportAlerts | ? { ( $_.ResolutionState -ne “255” ) }
$SCOMOpenAddendumReportAlerts.Count

$SCOMCoreRuleAlerts = Get-SCOMAlert | ? { ( $_.Name -in $SCOMCoreMonitors.DisplayName) -AND ( $_.ResolutionState -ne “255” ) }
$SCOMCoreRuleAlerts.Count
$SCOMAddendumRuleAlerts = Get-SCOMAlert | ? { ( $_.Name -in $SCOMAddendumRules.DisplayName) -AND ( $_.ResolutionState -ne “255” ) }
$SCOMAddendumRuleAlerts.Count

$SCOMCoreMonitorAlerts = Get-SCOMAlert | ? { ($_.Name -in $SCOMCoreMonitors.DisplayName ) -AND ( $_.ResolutionState -ne “255” ) }
$SCOMCoreMonitorAlerts.Count
$SCOMAddendumMonitorAlerts = Get-SCOMAlert | ? { ($_.Name -in $SCOMAddendumMonitors.DisplayName ) -AND ( $_.ResolutionState -ne “255” ) }
$SCOMAddendumMonitorAlerts.Count

$AutoClosed = $SCOMCoreMonitorAlerts.Count + $SCOMCoreRuleAlerts.Count + $SCOMAddendumMonitorAlerts.Count + $SCOMAddendumRuleAlerts.Count
$Test = $SCOMCoreReportAlerts.Count + $SCOMAddendumReportAlerts.Count
$OpenAlerts = $SCOMOpenReportAlerts.Count + $SCOMOpenAddendumReportAlerts.Count
$ResetMonitors = $SCOMCoreMonitors + $SCOMAddendumMonitors
$MonitorAlerts = $SCOMCoreMonitorAlerts.Count + $SCOMAddendumMonitorAlerts.Count

#
# If Cleanup needed, array of report monitors

# Reset Monitors Script
# Put ps1 in mgmtpacks folder
# https://sc.scomurr.com/scom-2012-monitor-reset-cleaning-up-the-environment/
# Download
# https://gallery.technet.microsoft.com/SCOM-2012-Batch-reset-63a17534

#Alternate
#https://gallery.technet.microsoft.com/scriptcenter/Auto-reset-script-for-d8b775ca

if ( $MonitorAlerts -gt 0 )
{
foreach ( $MonitorDisplayName in $ResetMonitors.DisplayName )
{
$Monitors = @( Get-SCOMMonitor -displayname $MonitorDisplayName )

# Set up monitor objects to reset
foreach ($Monitor in $Monitors)
{
$MonitorClass = Get-SCOMClass -Id $Monitor.Target.Id
$ActiveMonitors = Get-SCOMClassInstance -Class $MonitorClass | ? { ($_.healthstate -ne ‘Success’) -AND ( $_.healthstate -ne ‘Uninitialized’) -AND ($_.IsAvailable -eq $true) }
write-host “Found” + $ActiveMonitors.Count + “active monitors.”
if ( $ActiveMonitors -ne $null)
{
foreach ($ActiveMonitor in $ActiveMonitors)
{
write-host ” Resetting Health State on ‘” +$ActiveMonitor.FullName + “‘”
$ActiveMonitor.ResetMonitoringState($Monitor.ID)
}
}
}
}
}

February 1, 2023February 1, 2023

SCOM Snapshot Synchronization alerts

First, some background – the Snapshot Synchronization alert just tells you there was a SQL issue running the workflow.

Second, the Snapshot Synchronization alert from a health model perspective, is NOT a critical issue (outage). Create override severity to 1 (warning) to prevent false wake-up calls. I’ll get this to my GitHub repo shortly!

Let’s troubleshoot the alert

Open SSMS > Connect to the SCOM OpsMgr DB > Click on New Query

***NOTE verify database dropdown shows Operations Manager!

Paste SQL query into the query textbox

Select WorkItemName, b.WorkItemStateName, ServerName, StartedDateTimeUtc, CompletedDateTimeUtc, DurationSeconds, ERRORMESSAGE
from cs.WorkItem a , cs.WorkItemState b
where a.WorkItemStateId= b.WorkItemStateId
and WorkItemName = ‘SnapshotSynchronization’

Screenshot

Example SQL Output

WorkItemName WorkItemStateName ServerName StartedDateTimeUtc CompletedDateTimeUtc DurationSeconds ERRORMESSAGE
SnapshotSynchronization Succeeded SCOMV01 2023-01-24 00:26:23.427 2023-01-24 00:27:46.100 83 NULL
SnapshotSynchronization Failed SCOMV02 2023-01-25 00:27:52.363 2023-01-25 00:28:07.520 15
SnapshotSynchronization Succeeded SCOMV00 2023-01-25 21:43:36.540 2023-01-25 21:45:07.947 91 NULL
SnapshotSynchronization Running SCOMV00 2023-01-25 21:45:32.227 NULL NULL NULL

Solution:
The jobs may show Succeeded by the time you login to SQL = EOJ (end of job)

If Failed is latest date/timestamp, re-run the task “Request Snapshot Synchronization” which can be found when we select “Management Configuration Service Group” in the below mentioned view.

View:

From Monitoring Tab > Click on Operations Manager folder > Click on Management Group Health widget > Highlight unhealthy state from Management Group Functions.

Click on the ‘Request Snapshot Synchronization’ task to execute the Stored Procedure “SnapshotSynchronizationForce” on the OpsMgr DB.

NOTE: There are two tasks with same name but with different targets i.e. ‘Management Configuration Service Group’ and ‘Management Configuration Services’

The other task can be found on below view after selecting the Management Server you want the Task to be executed on

View:

From Monitoring Tab > Expand Operations Manager folder > Expand Management Configuration Service folder > Click on Services State view

Create Override for the alert

To change snapshot monitor to warning

From SCOM Console > Authoring Tab

Expand ‘Management Pack Objects’ > Click on Monitors

In the ‘Look for:’ bar type Snapshot synchronization state and hit enter

Monitor name = Snapshot synchronization state

Right click on monitor > Overrides > Override the monitor > For all objects of class

Click checkbox for Severity > change Critical to Warning

Click Edit – add comment – i.e. date/time changing to warning

Select your override pack > Click OK

Click OK to execute change

Resources

Link to TechNet article

January 31, 2023

Cleaner PowerShell to help reset monitors and rules

Cleaner PowerShell supplied by Andrew Bradley that’s helped simplify the PowerShell code included resetting/closing monitors and rules via a method call. Hard to believe I’ve been quiet on the blog for the past year, as I’ve been working on SCOM management pack addendums. The ‘cleaner PowerShell’ is being integrated into the various addendums.

have been helpful with many customers, by building out better ways to monitor, clean up alerts, and create daily reports. The Addendum packs add report key insights for many 1P (1st party) Microsoft authored management packs.

Methods

Cleaner PowerShell to help reset monitors and rules

## Grab the MP, get the Monitors and Rules from the MP, then grab all alerts found inside the Monitors/Rules

$SCOMCoreMP = Get-SCOMManagementPack -DisplayName “System Center Core Monitoring”

$SCOMCoreRules = $SCOMCoreMP.GetRules()

$SCOMCoreMonitors = $SCOMCoreMP.GetMonitors()

$SCOMCoreReportAlerts = Get-SCOMAlert | ? { ($_.Name -in $SCOMCoreRules.DisplayName) -or ($_.Name -in $SCOMCoreMonitors.DisplayName) }

Example for DNS management pack

## Grab the MP, get the Monitors and Rules from the MP, then grab all alerts found inside the Monitors/Rules
$SCOMCoreMP = Get-SCOMManagementPack -DisplayName “Microsoft Windows Server 2016 and 1709+ DNS Monitoring”
$SCOMCoreRules = $SCOMCoreMP.GetRules()
$SCOMCoreMonitors = $SCOMCoreMP.GetMonitors()

$SCOMCoreReportAlerts = Get-SCOMAlert | ? { ($_.Name -in $SCOMCoreRules.DisplayName) -or ($_.Name -in $SCOMCoreMonitors.DisplayName) }
$SCOMCoreReportAlerts.Count

$SCOMOpenReportAlerts = $SCOMCoreReportAlerts | ? { ( $_.ResolutionState -ne “255” ) }
$SCOMOpenReportAlerts.Count

# Open alerts

$SCOMCoreRuleAlerts = Get-SCOMAlert | ? { ( $_.Name -in $SCOMCoreRules.DisplayName) -AND ( $_.ResolutionState -ne “255” ) }
$SCOMCoreRuleAlerts.Count

$SCOMCoreMonitorAlerts = Get-SCOMAlert | ? { ($_.Name -in $SCOMCoreMonitors.DisplayName ) -AND ( $_.ResolutionState -ne “255” ) }
$SCOMCoreMonitorAlerts.Count

December 19, 2022December 19, 2022

Adding parameters to datasource/probeaction moduletypes

This post is adding parameters to datasource (DS) or probeaction (PA) moduletypes. Sorry, found this draft from last year that I never published. 🙁 I’m in the ‘missing functionality’ boat. Some would say I’m a dreamer, a good system admin, a car guy who has different ideas than the manufacturer, or something altogether different — you decide 🙂 Hope this blog post helps monitoring experts that author more functionality than what was delivered. Specifically adding parameters to datasource/probeaction moduletype NOT delivered in the OotB functionality?!

Adding parameters to datasource/probeaction moduletypes
First – What is needed
Second – Verify dependencies required for a workflow
Third – Build on example ‘datasource’
Fourth – Configure Monitor/Rule to use Datasource/ProbeAction

Let’s go through step by step through ‘adding parameters to datasource/probeaction moduletypes’ to customize a data source. The datasource requirements are to include/verify the following parameters” TimeOut,TimeOutInMS,MatchCount,SampleCount (match/sample count are intended for rules/monitors)

Pre-reqs (what’s needed for a ModuleType to function)

Working Script – PowerShell/BASH/Perl/SH/KSH
ScriptArgs required at runtime
Other Configuration, or Overrideable Parameters
Using configured parameters properly
Verify ProbeActions (PA) inside DS have relevant parameters

Easiest way to summarize adding a configuration parameter
Must be added to Configuration, OverrideableParameters,ModuleImplementation,
When taking an Out of the box’ OotB’ moduletype to modify, where parameter(s) MUST be used in UnitMonitorType,Rule,Monitor

Quick background for MatchCount/SampleCount:
When adding parameters to datasource/probeaction moduletypes, it’s good to know why this is part of the conversation to be added to monitoring design/implementation.

MatchCount comes in handy for repeated failures BEFORE alerting (count 5 events before alerting)
SampleCount comes in handy for counting number of failed workflows BEFORE alerting (run workflow 6 times failing before alerting)

Example Unix.ShellCommand.Invoke.Script DataSource
Requirement = Add MatchCount/SampleCount (or TimeOut to the PA ProbeAction)

Download

Unseal, and open Microsoft.Unix.ShellCommand.Library.xml in NotePad++, VStudio, (or your favorite XML editor)

Screenshot of default Microsoft.Unix.ShellCommand.Invoke.DataSource
TimeOut and TimeOutinMS are baked in. We begin by adding MatchCount/SampleCount

How to add MatchCount/SampleCount syntax

Adding MatchCount/SampleCount for Configuration, OverrideableParameters, and Module Implementation for DS/PA

NOTE – sometimes you don’t find an example!

This part gets complicated – how far down the rabbit hole do you need the parameters?
Does the DS workflow only need the respective parameters?
Do you have to add to the corresponding PA’s called in the workflow?

Starting simple, add to DS

Add MatchCount/SampleCount to DS Configuration
<xsd:element name=”MatchCount” type=”xsd:unsignedInt” maxOccurs=”1″ minOccurs=”0″ xmlns:xsd=”http://www.w3.org/2001/XMLSchema” />
<xsd:element name=”SampleCount” type=”xsd:unsignedInt” maxOccurs=”1″ minOccurs=”0″ xmlns:xsd=”http://www.w3.org/2001/XMLSchema” />

Add MatchCount/SampleCount to OverrideableParameters (if you want capability to override)
<OverrideableParameter ID=”MatchCount” Selector=”$Config/MatchCount$” ParameterType=”int” />
<OverrideableParameter ID=”SampleCount” Selector=”$Config/SampleCount$” ParameterType=”int” />

Add MatchCount/SampleCount to DS MemberModule
<MatchCount>$Config/MatchCount$</MatchCount>
<SampleCount>$Config/SampleCount$</SampleCount>

Add MatchCount/SampleCount to PA Configuration
<xsd:element name=”MatchCount” type=”xsd:unsignedInt” maxOccurs=”1″ minOccurs=”0″ xmlns:xsd=”http://www.w3.org/2001/XMLSchema” />
<xsd:element name=”SampleCount” type=”xsd:unsignedInt” maxOccurs=”1″ minOccurs=”0″ xmlns:xsd=”http://www.w3.org/2001/XMLSchema” />

Add MatchCount/SampleCount to PA MemberModule
<MatchCount>$Config/MatchCount$</MatchCount>
<SampleCount>$Config/SampleCount$</SampleCount>

Unix.ShellCommand.Invoke.Script
Alternate example for monitors, the SQL Windows Replication mgmt pack has a good UnitMonitor/UnitMonitorType example – Microsoft.SQLServer.Replication.Windows.Monitoring.xml

References

Kevin Holman has a good example for changing frequency and MatchCount here
https://kevinholman.com/2017/08/12/creating-a-scom-service-monitor-that-allows-overrides-for-interval-frequency-and-samples/

Find example by searching unsealed management pack repository (use Tyson’s SCOMHelper PowerShell module to unseal mp/mpb’s to facilitate a better unsealed mp search) https://monitoringguys.com/2019/11/12/scomhelper/

December 14, 2021December 19, 2022

Detected malicious verification code error

Ever run into the ‘detected malicious verification code’ error while authoring? I ran into the malicious verification error authoring, and couldn’t find any content for this error while authoring a pack.

Watch your copy/paste’s with additional monitoring changes to prevent ‘detected malicious verification code’ errors

In my authoring example, I received the ‘detected malicious verification code error’ after adding Rules, Datasources, and WriteActions (including tasks). I was copying and pasting DataSources (DS) and WriteActions (WA), thought I had it all. Uploaded > got the error, and GRR! Hopefully this will help others authoring to know what to check to get the management pack uploaded.

Simply put – Watch out for typo’s to avoid ‘detected malicious verification code’ errors!

I stumbled across a few websites, but nothing really pointed out to what caused the ‘detected malicious verification code error’ when uploading a management pack. First, check monitor and rules to verify the DS/WA are called correctly (no errors in file names. Check the Tasks as well as DisplayStrings, to make sure everything matches.

Error Seen when uploading Management pack from SCOM Console GUI regarding ‘detected malicious verification code’ error

<ManagementPackNameHere> Reports could not be imported.

If any management packs in the Import list are dependent on this management

pack, the installation of the dependent management packs will fail.

Verification failed with 1 errors:

——————————————————-

Error 1:

Found error in

2|<ManagementPackNameHere>|1.0.0.6|<ManagementPackNameHere>|

| with message:

Detected malicious verification code when verifying element of type

Microsoft.EnterpriseManagement.Configuration.ManagementPackRule with inner

exception: System.Collections.Generic.KeyNotFoundException: The given key

was not present in the dictionary.

at System.ThrowHelper.ThrowKeyNotFoundException()

at System.Collections.Generic.Dictionary`2.get_Item(TKey key)

Microsoft.EnterpriseManagement.Configuration.ManagementPackRule.VerifyDataTy

pes(Dictionary`2 moduletypes)

Microsoft.EnterpriseManagement.Configuration.ManagementPackRule.Verify(Verif

icationContext context)

Microsoft.EnterpriseManagement.Configuration.Verification.VerificationEngine

.VerifyCollectionItems(Object context)

Additional links

Detected malicious verification code when verifying element

Forum https://social.technet.microsoft.com/Forums/en-US/ac50ae14-882a-4788-a8e4-6a975c498a29/detected-malicious-verification-code-when-verifying-element-of-type

December 14, 2021December 14, 2021

Caution using Tags/Notes extending classes

Please take ‘caution using Tags/Notes extending classes’. Please read below if you use Tags/Notes on SCOM classes. Ran across examples where SCOM Class Properties were used for tags that used the .Notes field on various classes, causing orphaned properties, NOT removed from OperationsManager database.

Background

The Microsoft.Windows.Computer Class (insert class here) is updated using Tim McFadden’s blog. This can cause issues with orphaned classes in the database because it is not currently handled as part of the stored procedure (i.e. the Notes property classes do not get marked for deletion).

First, identify which classes have Notes property. Start from Management Server (MS) via PowerShell. See attached TXT for additional examples to check and add/remove Notes Property on additional windows classes.

Set Notes property for Windows Operating System server

Second, we need to see how to set and clear the value, in order to clean up the Operations Manager database, to remove the orphaned instances. The example below sets the value for one (1) server to ‘Production’.

$WOS = Get-SCOMClass -name Microsoft.Windows.OperatingSystem | get-SCOMClassInstance | where-object -property Path -eq “16db01.testlab.net”
$WOS.'[System.ConfigItem].Notes’

$WOS.'[System.ConfigItem].Notes’.Value = “Production”

$WOS.Overwrite()

$WOS = Get-SCOMClass -name Microsoft.Windows.OperatingSystem | get-SCOMClassInstance | where-object -property Path -eq “16db01.testlab.net”

$WOS.'[System.ConfigItem].Notes’

Example Output

PS C:\Users\scomadmin> $WOS.'[System.ConfigItem].Notes’.Value = “Production”
PS C:\Users\scomadmin> $WOS.Overwrite()
PS C:\Users\scomadmin> $WOS = Get-SCOMClass -name Microsoft.Windows.OperatingSystem | get-SCOMClassInstance | where-object -property Path -eq “16db01.testlab.net”
PS C:\Users\scomadmin> $WOS.'[System.ConfigItem].Notes’

PropertyAccessRights : Unknown
Parent : Microsoft Windows Server 2016 Standard
Type : Notes
Value : Production
Id : 00000000-0000-0000-0000-000000000000
ManagementGroup : SCOM2016
ManagementGroupId : e39f5f53-9fbb-9d7f-4bfe-5f0324630ae5

Set Notes property to NULL

$WOS.'[System.ConfigItem].Notes’.Value = $null
$WOS.Overwrite()

$WOS = Get-SCOMClass -name Microsoft.Windows.OperatingSystem | get-SCOMClassInstance | where-object -property Path -eq “16db01.testlab.net”

Verify Notes value

$WOS = Get-SCOMClass -name Microsoft.Windows.OperatingSystem | get-SCOMClassInstance | where-object -property Path -eq “16db01.testlab.net”
$WOS.'[System.ConfigItem].Notes’

Example Output
PS C:\Users\scomadmin> $WOS = Get-SCOMClass -name Microsoft.Windows.OperatingSystem | get-SCOMClassInstance | where-object -property Path -eq “16db01.testlab.net”
PS C:\Users\scomadmin> $WOS.'[System.ConfigItem].Notes’

PropertyAccessRights : Unknown
Parent : Microsoft Windows Server 2016 Standard
Type : Notes
Value : (null)
Id : 00000000-0000-0000-0000-000000000000
ManagementGroup : SCOM2016
ManagementGroupId : e39f5f53-9fbb-9d7f-4bfe-5f0324630ae5

Have a happy Holiday!

Good luck, hopefully this scenario isn’t something that impacted the monitoring environment!

November 18, 2021November 18, 2021

What’s my path again?

Say What

What’s my path again? Why did my command fail?

Ever get ‘command not found’ errors when calling a command on a machine? Many times, these errors are related to what is defined on said machine. So with monitoring tools like SCOM, ALA, Azure Automation, BMC Patrol, the ID used in monitoring rely on filepaths defined on the local server (holds true for Windows/UNIX). Because sometimes even ls, awk, dir, etc. if their various bin directory filepaths are NOT specified as a security hardening measure. The result of STIG/Security hardening is ALL scripts/commands require a fully qualified filepath.

Fully qualifying command paths holds true for Windows and UNIX, from generic OS commands, AND also application specific files (including an executable). Updates are required if you want to supply the short name command. Add the full filepath to PATH= statement. The alternative is to fully qualify in your SCOM mgmt. pack, so the command will run regardless of user, as long as the path is correct.

Check for specified shell

First, let’s check UNIX to see what shell is specified for user(s).

Second, log into your UNIX server, and check files type: ls -al .* | more

Use ls -al | more to see what PATH files are in the user directory

Third, another option with less output

example: ls -al .*profile

What’s my path? Use command ls -al .*profile to find which profile(s) exist

Fourth, Look for the shell defined for the user account

On my server, SCOM user is bash shell (but I do NOT have a .bash_profile, only a .profile (also note NO .ksh_profile) ) Knowing what profiles are configured for user account will help define what is inherited from the OS, (automatically included). Leverage when calling commands in your management packs for custom rules/monitors.

In conclusion, if executable is NOT in the filepath variable, you have two ways to resolve the issue:

Create a .bash_profile
Call bash/ksh shell in your script or command line: bash; <commandhere>

To check path:

UNIX $PATH vs. Windows $ENV:path

UNIX example – ‘echo $PATH’ from UNIX ssh session/logon

UNIX what’s my path? Use echo $PATH

Windows PowerShell example

What’s my path? Windows PowerShell example of $PATH

Here’s my .profile that sets up SCOM user (only /bin shown)

What’s my path? Use UNIX .profile to find PATH

Here’s a UNIX .profile example:

https://www.unix.com/unix-for-dummies-questions-and-answers/21995-basic-profile-setup.html

Example

set PATH=$PATH:/usr/homes/myhome/sqlldr:/appl/oracle/product/9.2.0/bin