Ever get ‘command not found’ errors when calling a command on a machine? Many times, these errors are related to what is defined on said machine. So with monitoring tools like SCOM, ALA, Azure Automation, BMC Patrol, the ID used in monitoring rely on filepaths defined on the local server (holds true for Windows/UNIX). Because sometimes even ls, awk, dir, etc. if their various bin directory filepaths are NOT specified as a security hardening measure. The result of STIG/Security hardening is ALL scripts/commands require a fully qualified filepath.
Fully qualifying command paths holds true for Windows and UNIX, from generic OS commands, AND also application specific files (including an executable). Updates are required if you want to supply the short name command. Add the full filepath to PATH= statement. The alternative is to fully qualify in your SCOM mgmt. pack, so the command will run regardless of user, as long as the path is correct.
Check for specified shell
First, let’s check UNIX to see what shell is specified for user(s).
Second, log into your UNIX server, and check files type: ls -al .* | more
Third, another option with less output
example: ls -al .*profile
Fourth, Look for the shell defined for the user account
On my server, SCOM user is bash shell (but I do NOT have a .bash_profile, only a .profile (also note NO .ksh_profile) ) Knowing what profiles are configured for user account will help define what is inherited from the OS, (automatically included). Leverage when calling commands in your management packs for custom rules/monitors.
In conclusion, if executable is NOT in the filepath variable, you have two ways to resolve the issue:
Create a .bash_profile
Call bash/ksh shell in your script or command line: bash; <commandhere>
To check path:
UNIX $PATH vs. Windows $ENV:path
UNIX example – ‘echo $PATH’ from UNIX ssh session/logon
Windows PowerShell example
Here’s my .profile that sets up SCOM user (only /bin shown)
First, use SCOM helper for advanced administration (a shameless plug to Tyson & MonitoringGuys blog!)
In case you didn’t know, as I may be the last off the airplane, it’s time to talk about SCOMHelper. Things you forget, like using a power drill, versus manual screwdriver, all because you’re familiar with the old trusty screwdriver. Man, I think I upgraded past the power drill with a cord, to a lithium-ion powered impact drill with SCOMHelper.
Open PowerShell (as admininstrator)
Type: Install-Module -Name SCOMHelper
PS C:\> Install-Module -Name SCOMHelper
You are installing the modules from an untrusted repository. If you trust this repository, change its
InstallationPolicy value by running the Set-PSRepository cmdlet. Are you sure you want to install the modules from
[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is “N”): A
If SCOMHelper is already installed
Open PowerShell (as admininstrator)
Type: Install-Module -Name SCOMHelper -Force
(You may get the untrusted repository message if you answered Y (yes) or N (no)
PS C:\> Install-Module -Name SCOMHelper -Force
Just one example –
Use the Unseal-SCOMMP to unseal your MP and MPB’s for your SCOM mgmt pack repository
Have fun playing!
If you’re administering management groups, I can pretty much guarantee you will need these:
I’m off to unseal my repository, so my Notepad++ searches can find more examples for new authoring functionality.
It’s that time to figure out the ConfigMgr SMS role alerts – If you are monitoring your SCCM/MECM environment, then you get role failure alerts. Many times, the Operations Helpdesk, NOSC, NOC, SOC, etc. will get alerts when various roles fail on the Configuration Manager platform. The common ask is why, what do you see, etc. Much like Exchange, ConfigMgr internalizes the checks that are seen in the console as registry keys or events documenting said degraded component/feature. Helping the MECM administrator understand the failure is key to decoding how to notify administrator, and when the helpdesk needs to act on ‘ConfigMgr SMS role alerts’.
Example – MECM/SCCM looks at replication probe action state $Config/RoleName$
The role check is based on a variable of the RoleName in a registry key that the application updates.
This is the origin of ConfigMgr SMS role alerts
HKLM:SOFTWARE\Microsoft\SMS\Operations Management\SMS Server Role\$Config/RoleName$\Availability State
1 is critical state
2,3,4 are warning states
If more details are needed, download SCCM/MECM Management Pack for SCOM here
Use Tyson’s SCOM Helper pack to unseal, and inspect XML.
Once you know the origin of the ConfigMgr SMS role alerts, you can begin tuning the MECM alerts to your environment. Understanding role alerts will help both teams understand MECM application health. First, use MECM application health to trend alerts/outages. Second, leverage maintenance mode schedules, or MM scripts to NOT monitor for common administration tasks. From my experience, the alerts are the result of MECM Admins maintaining the application – common actions like building application/package lists, cleanup actions, site maintenance, backups, etc. Lastly, set up a subscription to notify after the tuning discussion. See my blog on building a subscription for more details.
Weird SQL issue from SCOM DB move to new SQL servers
Fix SQL2017+ .NET assembly errors after moving DB’s to new SQL servers.
Scenario: Moved the SCOM 2019 databases from a SQL 2014 database engine to a SQL 2019 database engine. SQL ApplicationThe following error occurred when opening the SCOM admin console:
Operations Manager Event Log, Event ID 26317
Date: 10/22/2021 11:17:27 AM
Application: Operations Manager
Application Version: 10.19.10505.0
An error occurred in the Microsoft .NET Framework while trying to load assembly id 65537. The server may be running out of resources, or theassembly may not be trusted. Run the query again, or check documentation to see how to solve the assembly trust issues. For more information about this error:
System.IO.FileLoadException: Could not load file or assembly ‘microsoft.enterprisemanagement.sql.userdefineddatatype, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null’ or one of its dependencies. An error relating to security occurred. (Exceptionfrom HRESULT: 0x8013150A)
Verify assemblies are successfully registered as trusted run:
Select * from sys.trusted_assemblies
The output should look like this:
At this point, re-start the SCOM services System Center Data Access, and System Center Management Configuration, on all management servers, and re-launch the SCOM admin console to make sure everything is working properly.
SCOM hotfix released for WebConsole/APM on SCOM2012R2 and above, time for another SCOM shot! Don’t forget your vaccination card 🙂
Let’s get started. Time to fix the vulnerability for ‘SCOM hotfix released for WebConsole/APM on SCOM2012R2 and above’. Read the support article, and assess what versions you have in your sandbox and production. Once assessed, it’s time to test/implement/verify the fix applied.
Use Get-WinEvent to use XML and filters from event viewer
The Tip or Trick part of this – leverage your Event Viewer Filter as a query to use with get-WinEvent
Credit for this tip comes from Andrew Blumhardt!
See below for examples to ‘use Get-WinEvent to use XML and filters from event viewer’
Navigating via Event Viewer:
Hop onto your favorite server, or connect to another server via Event Viewer
Go to the Event Log > Click Filter Current Log
Build out your filter (i.e. choose specific Event Sources, exclude events, include severities, timeframe (start/end), etc.)
Switch to the XML tab (and note you can edit your query further!)
You can copy the query from the Event Viewer into your Get-WinEvent syntax
$query = @”
<QueryList> <Query Id=”0″ Path=”Application”> <Select Path=”Application”>*[System[Provider[@Name=’Microsoft.SystemCenter.VirtualMachineManager.2012.Monitor.UserRoleQuotaUsageMonitor’ or @Name=’Microsoft.SystemCenter.VirtualMachineManager.2012.Report.ServiceUsageCollection’ or @Name=’Microsoft.SystemCenter.VirtualMachineManager.2012.Report.VMUsageCollection’ or @Name=’Microsoft.SystemCenter.VirtualMachineManager.2016.EnableCredSSPClient’ or @Name=’Microsoft.SystemCenter.VirtualMachineManager.2016.Monitor.UserRoleQuotaUsageMonitor’ or @Name=’Microsoft.SystemCenter.VirtualMachineManager.2016.Report.ServiceUsageCollection’ or @Name=’Microsoft.SystemCenter.VirtualMachineManager.2016.Report.VMUsageCollection’] and (Level=2 or Level=3) and (EventID=25933)]]</Select> </Query> </QueryList>
Get-WinEvent -FilterXml $query
Grab System Event Log, Event ID 5827 (NetLogon denied events)
# Change Drive letter if you hopefully installed SCOM on D: drive (non-system drive)
copy “C:\Program Files\Microsoft System Center 2016\Operations Manager\WebConsole\MonitoringView\bin\Microsoft.EnterpriseManagement.OperationsManager.MonitoringViews.dll” “C:\Program Files\Microsoft System Center 2016\Operations Manager\WebConsole\MonitoringView\bin\Microsoft.EnterpriseManagement.OperationsManager.MonitoringViews-old.dll”
# Replace DLL
copy “C:\MonAdmin\Microsoft*.dll” “C:\Program Files\Microsoft System Center 2016\Operations Mana ger\WebConsole\MonitoringView\bin”
cd “D:\Program Files\Microsoft System Center 2016\Operations Manager\WebConsole\MonitoringView \bin”
Back again, I’m going to ‘Identify orphaned agent properties’. For instance, does an agent still show up under Windows Computer, or more classes, like Windows Operating System? Typically we have handled this by using Holman’s purge blog.
First, my thanks to Kevin H, Mihai S from the SCOM PG, & Premier Support CSS, for their help. Let’s begin the ‘Identify orphaned agent properties’ discussion with ‘how’. First, how do you get an orphaned property? Second, how to you resolve?
Some example scenarios
Server rebuilt with same name. New agent runs discovery, and creates new set of GUID’s in the database.
The Monitoring Tab > Windows Computer view contains unhealthy <gray> server objects. Upon further inspection, the server does NOT show up in the Administration > Agent Managed view.
Custom management pack authoring extends the Windows Computer class, or others (via SDK or PowerShell)
‘Identify and resolve’ orphaned agent properties
Check for COMMIT or Overrides in management packs
PG recommended looking at Windows Computer extended class properties, and Connector Framework discoveries.