November 2021 - Kevin Justin's Blog

What’s my path again?

Say What

What’s my path again? Why did my command fail?

Ever get ‘command not found’ errors when calling a command on a machine? Many times, these errors are related to what is defined on said machine. So with monitoring tools like SCOM, ALA, Azure Automation, BMC Patrol, the ID used in monitoring rely on filepaths defined on the local server (holds true for Windows/UNIX). Because sometimes even ls, awk, dir, etc. if their various bin directory filepaths are NOT specified as a security hardening measure. The result of STIG/Security hardening is ALL scripts/commands require a fully qualified filepath.

Fully qualifying command paths holds true for Windows and UNIX, from generic OS commands, AND also application specific files (including an executable). Updates are required if you want to supply the short name command. Add the full filepath to PATH= statement. The alternative is to fully qualify in your SCOM mgmt. pack, so the command will run regardless of user, as long as the path is correct.

Check for specified shell

First, let’s check UNIX to see what shell is specified for user(s).

Second, log into your UNIX server, and check files type: ls -al .* | more

Use ls -al | more to see what PATH files are in the user directory

Third, another option with less output

example: ls -al .*profile

What’s my path? Use command ls -al .*profile to find which profile(s) exist

Fourth, Look for the shell defined for the user account

On my server, SCOM user is bash shell (but I do NOT have a .bash_profile, only a .profile (also note NO .ksh_profile) ) Knowing what profiles are configured for user account will help define what is inherited from the OS, (automatically included). Leverage when calling commands in your management packs for custom rules/monitors.

In conclusion, if executable is NOT in the filepath variable, you have two ways to resolve the issue:

Create a .bash_profile
Call bash/ksh shell in your script or command line: bash; <commandhere>

To check path:

UNIX $PATH vs. Windows $ENV:path

UNIX example – ‘echo $PATH’ from UNIX ssh session/logon

UNIX what’s my path? Use echo $PATH

Windows PowerShell example

What’s my path? Windows PowerShell example of $PATH

Here’s my .profile that sets up SCOM user (only /bin shown)

What’s my path? Use UNIX .profile to find PATH

Here’s a UNIX .profile example:

https://www.unix.com/unix-for-dummies-questions-and-answers/21995-basic-profile-setup.html

Example

set PATH=$PATH:/usr/homes/myhome/sqlldr:/appl/oracle/product/9.2.0/bin

Use SCOM helper for advanced administration

VoteForPedro — Vote for Pedro and SCOM helper

First, use SCOM helper for advanced administration (a shameless plug to Tyson & MonitoringGuys blog!)

In case you didn’t know, as I may be the last off the airplane, it’s time to talk about SCOMHelper. Things you forget, like using a power drill, versus manual screwdriver, all because you’re familiar with the old trusty screwdriver. Man, I think I upgraded past the power drill with a cord, to a lithium-ion powered impact drill with SCOMHelper.

Blog https://monitoringguys.com/2019/11/12/scomhelper/
Download https://www.powershellgallery.com/packages/SCOMHelper

Install SCOMHelper

Time to use SCOM helper for advanced administration, and get it installed!

Pretty simple install. I would recommend adding to monitoring team server, jumpserver, from SCOM MS, or console enabled, Operations Manager plug-in installed machines.

From PowerShell (as admin)

Install-Module -Name SCOMHelper

SCOM helper functions
Clear-SCOMCache Compare-String Deploy-SCOMAgent Export-SCOMEffectiveMonitoringConfigurationReport Export-SCOMEventsToCSV Export-SCOMKnowledge Export-SCOMOverrides Fast-Ping Get-SCOMAlertKnowledge Get-SCOMClassInfo Get-SCOMHealthCheckOpsConfig Get-SCOMMPFileInfo Get-SCOMRunAsAccountName Get-SCOMRunAsProfilesAccounts Get-SCOMRunningWorkflows Get-StringHash New-SCOMClassGraph New-SCOMComputerGroup Ping-AllHosts Remove-SCOMObsoleteReferenceFromMPFile Set-SCOMMPAliases Show-SCOMModules Show-SCOMPropertyBag Test-Port Unseal-SCOMMP Update-SCOMComputerGroup Start-SCOMOverrideTool

Open PowerShell (as admininstrator)
Type: Install-Module -Name SCOMHelper

Example output
PS C:\> Install-Module -Name SCOMHelper

Untrusted repository
You are installing the modules from an untrusted repository. If you trust this repository, change its
InstallationPolicy value by running the Set-PSRepository cmdlet. Are you sure you want to install the modules from
‘PSGallery’?
[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is “N”): A

If SCOMHelper is already installed
Open PowerShell (as admininstrator)
Type: Install-Module -Name SCOMHelper -Force

(You may get the untrusted repository message if you answered Y (yes) or N (no)

Example output
PS C:\> Install-Module -Name SCOMHelper -Force
PS C:\>

Just one example –

Use the Unseal-SCOMMP to unseal your MP and MPB’s for your SCOM mgmt pack repository

Have fun playing!

If you’re administering management groups, I can pretty much guarantee you will need these:

Export-SCOMEffectiveMonitoringConfigurationReport

Export-SCOMOverrides

Get-SCOMHealthCheckOpsConfig

Get-SCOMRunningWorkflows

New-SCOMClassGraph

Remove-SCOMObsoleteReferenceFromMPFile

Show-SCOMPropertyBag

I’m off to unseal my repository, so my Notepad++ searches can find more examples for new authoring functionality.

ConfigMgr SMS role alerts

Microsoft Endpoint Configuration Manager

It’s that time to figure out the ConfigMgr SMS role alerts – If you are monitoring your SCCM/MECM environment, then you get role failure alerts. Many times, the Operations Helpdesk, NOSC, NOC, SOC, etc. will get alerts when various roles fail on the Configuration Manager platform. The common ask is why, what do you see, etc. Much like Exchange, ConfigMgr internalizes the checks that are seen in the console as registry keys or events documenting said degraded component/feature. Helping the MECM administrator understand the failure is key to decoding how to notify administrator, and when the helpdesk needs to act on ‘ConfigMgr SMS role alerts’.

Example – MECM/SCCM looks at replication probe action state $Config/RoleName$

Example MECM Service Monitor for role alerts

The role check is based on a variable of the RoleName in a registry key that the application updates.

This is the origin of ConfigMgr SMS role alerts

HKLM:SOFTWARE\Microsoft\SMS\Operations Management\SMS Server Role\$Config/RoleName$\Availability State

Decoder ring:

1 is critical state

2,3,4 are warning states

If more details are needed, download SCCM/MECM Management Pack for SCOM here

Use Tyson’s SCOM Helper pack to unseal, and inspect XML.

Once you know the origin of the ConfigMgr SMS role alerts, you can begin tuning the MECM alerts to your environment. Understanding role alerts will help both teams understand MECM application health. First, use MECM application health to trend alerts/outages. Second, leverage maintenance mode schedules, or MM scripts to NOT monitor for common administration tasks. From my experience, the alerts are the result of MECM Admins maintaining the application – common actions like building application/package lists, cleanup actions, site maintenance, backups, etc. Lastly, set up a subscription to notify after the tuning discussion. See my blog on building a subscription for more details.

Weird SQL issue from SCOM DB move

Fix SQL2017+ .NET assembly - Weird .NET error — BillMurry-ThatsWeird

This post is courtesy of Andres Naranjo

Fix SQL2017+ .NET assembly error

Weird SQL issue from SCOM DB move to new SQL servers

Fix SQL2017+ .NET assembly errors after moving DB’s to new SQL servers.

Scenario: Moved the SCOM 2019 databases from a SQL 2014 database engine to a SQL 2019 database engine. SQL ApplicationThe following error occurred when opening the SCOM admin console:

Operations Manager Event Log, Event ID 26317

Date: 10/22/2021 11:17:27 AM

Application: Operations Manager

Application Version: 10.19.10505.0

Severity: Error

Message:

An error occurred in the Microsoft .NET Framework while trying to load assembly id 65537. The server may be running out of resources, or the assembly may not be trusted. Run the query again, or check documentation to see how to solve the assembly trust issues. For more information about this error:

System.IO.FileLoadException: Could not load file or assembly ‘microsoft.enterprisemanagement.sql.userdefineddatatype, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null’ or one of its dependencies. An error relating to security occurred. (Exception from HRESULT: 0x8013150A)

System.IO.FileLoadException:

at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)

at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)

at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean forIntrospection)

at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection)

at System.Reflection.Assembly.Load(String assemblyString)

Fix SQL2017+ .NET assembly

In addition, Operations Manager event ID’s 26317 events document the error (also check SQL Application log, see Holman’s blog). Here is an example from the Operations Manager event log:

Screenshot of Operations Manager event log, EventID 26319 — Operations Manager event log, EventID 26319

Weird SQL issue from SCOM DB move to new SQL servers

Cause:

Starting with SQL 2017, SQL restricts trusted managed assemblies.

See more details in Microsoft TechNet article here

First, ensure that SQL CLR execution is enabled with the following SQL query:

sp_configure @configname=clr_enabled, @configvalue=1
GO
RECONFIGURE
GO

NOTE: It is important to make sure the SQL Server Service is re-started after the query above.

Second, execute ‘add trusted’ stored procedure queries to mark both as trusted:

EXEC sp_add_trusted_assembly 0xFAC2A8ECA2BE6AD46FBB6EDFB53321240F4D98D199A5A28B4EB3BAD412BEC849B99018D9207CEA045D186CF67B8D06507EA33BFBF9A7A132DC0BB1D756F4F491

EXEC sp_add_trusted_assembly 0xEC312664052DE020D0F9631110AFB4DCDF14F477293E1C5DE8C42D3265F543C92FCF8BC1648FC28E9A0731B3E491BCF1D4A8EB838ED9F0B24AE19057BDDBF6EC

Verify assemblies are successfully registered as trusted run:

Select * from sys.trusted_assemblies

The output should look like this:

At this point, re-start the SCOM services System Center Data Access, and System Center Management Configuration, on all management servers, and re-launch the SCOM admin console to make sure everything is working properly.

Quicker ways to start SCOM services

From PowerShell (as Admin)

restart-service healthservice; restart-service omsdk; restart-service cshost

Leverage Invoke-Command

# Invoke-Command syntax is PoSH remoting is enabled

# Run on multiple servers

# From PowerShell on SCOM Mgmt server, where you have same credential/access

# Example 1

“server1”,”server2″| % {invoke-command $_ -scriptblock {$env:ComputerName; restart-service healthservice; restart-service omsdk; restart-service cshost; get-service healthservice; get-service omsdk; get-service cshost }}

# Example 2

# Restart healthservice on MS/Agent