Test SNOW script

Time to ‘Test SNOW script’ for event or incident injection. As long as the prerequisites are verified, to include network connectivity, URL, ID, Password, etc., we’re ready to go!

Once the CredentialManager piece has been completed, by the same token you can begin testing the script. Testing can begin, whether to the SCOM Admin SA account, or to the SCOM Notifications account, or even hard coding the values into your PowerShell session.

Begin script testing

The testing leverages that you’ve downloaded various integration scripts first, then being saved on SCOM MS (management servers). The following blog posts, GitHub repo’s will set up multiple methods to test from PowerShell (command line) as SA or SVC accounts.

Create new SCOM Command Channels

LogAlert.ps1 from Kevin Holman’s blog https://kevinholman.com/2021/08/25/what-account-will-command-channel-notifications-run-as-in-scom/

New-SNowEvent GitHub repository https://github.com/theKevinJustin/New-SNowEvent

New-SNowIncident.ps1 GitHub repository https://github.com/theKevinJustin/New-SNOWIncidents

Once files are saved, copy to SCOM MS (noting path for later tests).

Preferred choice is non-system disk (i.e. D:)

Lab example is C:\MonAdmin\Scripts

RDP to server with SVC or SCOM Admin SA account

Open PowerShell as administrator

Change directory to script directory on SCOM MS

cd \ ; cd MonAdmin/scripts

Inventory saved files via dir (gci):

gci New-SNOW[ie]*.ps1

gci LogAlert.ps1

Grab Critical alert from $Alerts array

$Alerts = get-scomalert -ResolutionState 0 -severity 2

# Test warning events with Severity 1

# $Alerts = get-scomalert -ResolutionState 0 -severity 1

$AlertID = $Alerts[0].ID

$AlertName = $Alerts[0].Name

$AssignmentGroup = “JustinTime Infra”

PowerShell SCOM alert variables assumed for script to run from PowerShell

Test script supplying variables

.\New-SNowEvent.ps1 -AlertID $AlertID -AlertName $AlertName -AssignmentGroup $AssignmentGroup -Team SYM

.\New-SNowIncident.ps1 -AlertID $AlertID -AlertName $AlertName -AssignmentGroup $AssignmentGroup -Team SYM

New-SNOWEventPowerSHellOutput for REST Event injection.

Verify SCOM alert updated for ServiceNow REST injection

Check SCOM console/web console for SCOM alert updates to ResolutionState, TicketID, Owner fields, where TEAM = SYM, and Assignment Group = JustinTime Infra specified

SCOM Monitoring tab Active Alerts output view showing Owner, ResolutionState, and TicketId fields updated.

Be aware of issues

Indicator of Certificate/trust issue

Invoke-RestMethod error seen when organization cert not installed, making server sending REST injection NOT trusted.

Indicator when SNOW alert rule not configured or matching – excessive retry’s. Also note output shows summary of tests, ServiceNow SNOW detail, and SCOM alert updates.

New-SNOWEvent.ps1 failures when SNOW Alert rule not matching or created.

Logging to Operations Manager Event Log for addtional troubleshooting or debug. Unless otherwise updated, the script logs to the ‘Operations Manager’ event log, EventID = 710-712

Single Starting event indicates failed pre-requisite (pre-req NOT met)

SCOM OperationsManager events logging integration script status, as well as whoami and additional debug.

Additional information

ServiceNow Event integration

ServiceNow Incident Integration

Setup SCOM Notifications account

With domain joined machine, use a separate notification services svc account for notifications. SCOM is typically leverages MSAA, or even local system, depending on the accounts used when building out SCOM. Kevin Holman did an excellent job blogging this here

Verify SCOM notification account

Verify and ‘Setup SCOM Notifications account’ to separate notifications outside typical SCOM service SVC account functionality. Also, separating allows CredentialManager to secure, encrypt, and store credentials used by the notification account. Time to verify!

RDP to SCOM MS using notification account.

Open SCOM Console

Click on Administration tab

Expand Run As Configuration

Click on Accounts

Search for notification

Double click on Notifications account

Click on Credentials tab

Verify account being used, in light of CredentialManager piece storing SNOW ID and account.

NOTE Account should be part of SCOM Admins AD group

SCOM console view of Notifications SVC Account

SCOM Notifications Event Log troubleshooting

Knowing the notifications account will aid with SNOW integration scripts, as well as help log whoami, ‘run as’ logging to the ‘Operations Manager’ event log. The specific test and event or incident scripts leverage EventID’s 710-712. 710 for LogAlert.ps1, 711 for New-SNowEvent.ps1, and 712 for New-SNowIncident.ps1.

Single Starting event indicates failed pre-requisite (pre-req NOT met)

Additional SCOM Notifications documentation

Learn articles

SCOM2019 https://learn.microsoft.com/en-us/system-center/scom/manage-notifications-create-configure?view=sc-om-2019

SCOM2022 https://learn.microsoft.com/en-us/system-center/scom/manage-notifications-create-configure?view=sc-om-2022

Additional blog posts explaining ServiceNow connectivity

Options https://kevinjustin.com/blog/2024/05/24/lots-of-options-to-create-servicenow-incidents-through-powershell-scripts-connectors-and-3rd-party-vendors/

PreReqs https://kevinjustin.com/blog/2024/05/30/snow-rest-integration-prerequisites/
Network and Credential https://kevinjustin.com/blog/2024/05/30/snow-rest-connectivity/

Event Integration https://kevinjustin.com/blog/2024/03/27/servicenow-event-integration/

Incident Integration https://kevinjustin.com/blog/2024/03/27/servicenow-incident-integration/

ServiceNow Connector https://kevinjustin.com/blog/2024/04/30/servicenow-connector-for-scom/

SNOW REST connectivity

Verify ServiceNow test/production environment connectivity. Are you ready to begin the checks for ‘SNOW REST connectivity’?

Verify the following pre-requisites are met before proceeding:

Proxy

TEST/PROD ServiceNow (SNow) URL(s)

Password

High level outline:

Verify network connectivity via proxy, using SNOW URL’s, ID, password

Verify CredentialManager Snapin installed

Setup credentials in CredentialManager

Verify network connectivity via proxy, using SNOW URL’s, ID, password

Determine the SCOM notification account being used, see blog for details.

RDP to SCOM MS using notification account.

Open PowerShell as administrator

Verify network connectivity pasting command into PowerShell window

Example SNOW URL JustinTime.servicenowservices.com

Test-NetConnection -ComputerName JustinTime.servicenowservices.com -port 443

Test-NetConnection -ComputerName JustinTime.servicenowservices.com -port 443 -proxy ##PROXY##

Successful connectivity

PS C:\MonAdmin\SCRIPTS> Test-NetConnection -ComputerName servicenowservices.com -port 443

ComputerName : servicenowservices.com

RemoteAddress : 199.91.136.115

RemotePort : 443

InterfaceAlias : Ethernet 2

SourceAddress : 10.0.0.30

TcpTestSucceeded : True

Verify CredentialManager Snapin is installed

Best practice – Encrypt credentials on SCOM MS to prevent cleartext in scripts. To begin, this includes details to verify SnapIn, verify credentials stored, store credentials.

Verify CredentialManager Snapin is installed

RDP to SCOM MS server(s) as SCOM Notifications SVC account with SA access

Open PowerShell as administrator

Paste command(s) into PowerShell window to test network connectivity to SNOW environments

# Verify Credential Manager snap in installed

$CredMgrModuleBase = Get-Module -Name CredentialManager

if ( $Null -ne $CredMgrModuleBase.ModuleBase )

{ write-host -f yellow “CredentialManager PoSH Module Installed, ModuleBase = $($CredMgrModuleBase.ModuleBase)” }

if ( $Null -eq $CredMgrModuleBase.ModuleBase )

{ write-host -f red “CredentialManager PoSH Module NOT Installed” }

CredentialManager snapin PowerShell output

Verify Stored Credentials on server

First, verify any credentials stored on server, specific to ServiceNow or not. Second, we begin to use the Get-StoredCredential command. Third, we will setup the credential for REST integration, lastly verifying credential.

Setup Credentials for SNOW RESTAPI

RDP to SCOM MS server(s) as SCOM Notifications SVC account with SA access

Open PowerShell as administrator

Paste command(s) into PowerShell window to test network connectivity to SNOW environments

If no output, there are no credentials stored under the RDP login.

Get-StoredCredential

Commands specific to ServiceNow to verify credentials exist

$Credential = Get-StoredCredential -Target “ServiceNowCredential”

$Credential = Get-StoredCredential -Target “svc_rest_scom”

CredentialManager Get-StoredCredential Output

Alternate command to verify stored credential

Get-StoredCredential -Target “ServiceNowCredential”

Setup Credentials for SNOW RESTAPI

RDP to SCOM MS server(s) as SCOM Notifications SVC account with SA access

Open PowerShell as administrator

Paste the following commands to create stored credential:

New-StoredCredential -Target “ServiceNowCredential” -UserName “svc_rest_scom” -Password “##Password##” -Persist ‘LocalMachine’

Verify credential being used by SCOM

$Credential = Get-StoredCredential -Target “ServiceNowCredential”

Therefore, network connectivity verified. Lastly, encrypted credentials are stored in the SCOM notifications SVC account. Continue the integration!

SNOW REST integration prerequisites

Did you know –

These PowerShell scripts allow organizations to specify which alerts/events that need to go to ITSM tool.

Does not matter to the tool, whether Broadcom (Spectrum/DXOI), SolarWinds, MECM/MEM/MCM, SCOM

Pretty much ANY tool that can leverage PowerShell scripts and/or REST calls can utilize this script.

The following ServiceNow ‘SNOW REST integration prerequisites’ are required before proceeding.

TEST/PROD ServiceNow (SNow) URL(s)

Password

Incident short_description naming convention

ServiceNow SNOW Alert rule (to make events create incidents)

ServiceNow SNOW Incidents require additional variables to match ServiceNow selections

SNOW Incident short_description field is the title of the incident

Pre-define this in the SNOW REST Event/incident injection, to meet organizational naming conventions

Description can be additional details about the issue to be investigated, resolved.

SNOW Incident short_description, description fields

Examples of short_description titles

# Setup SNOW Event Name standard

Example SNOWAlertName

$SNOWAlertName = “<Org> <Team> SCOM Test Event – $Alert”

Example SNOWAlertName

$SNOWAlertName = “<Team> <ORG> SCOM Event – $AlertName”

Example SNOWAlertName

$SNOWAlertName = “<Team> <ORG> SCOM $AlertName”

Example SNOWAlertName

$SNOWAlertName = “##CUSTOMER## ##TEAM## SCOM Event – $AlertName”

Example SNOWAlertName

$SNOWAlertName = “##TEAM## ##CUSTOMER##: SCOM – $AlertName”

Example SNOWAlertName

$SNOWAlertName = “##TEAM## ##CUSTOMER##: SPECTRUM – $AlertName”

Example SNOWAlertName

$SNOWAlertName = “##TEAM## ##CUSTOMER##: SOLARWINDS – $AlertName”

SNOW Incident fields

Direct Incident REST injection requires additional fields, such as caller, business_service, category, subcategory, channel, impact, urgency, priority, assignment_group

Additional information

REST/RESTAPI https://www.techtarget.com/searchapparchitecture/definition/RESTful-API

SNOW Utah Connector https://docs.servicenow.com/bundle/utah-it-operations-management/page/product/event-management/task/t_EMConfigureSCOMConnector.html

Additional blogs showing scope and options using ServiceNow for ITSM tool

https://kevinjustin.com/blog/2024/03/27/servicenow-event-integration/

https://kevinjustin.com/blog/2024/03/27/servicenow-incident-integration/

https://kevinjustin.com/blog/2024/04/30/servicenow-connector-for-scom/

https://kevinjustin.com/blog/2024/05/24/lots-of-options-to-create-servicenow-incidents-through-powershell-scripts-connectors-and-3rd-party-vendors/

NOT monitored servers

Ever run into NOT monitored servers?

Unsure your experience, but recently, I’ve run across multiple occurrences where servers show up as ‘not monitored. As a result, I’ve found three distinct sets of issues that might cause ‘not monitored’ status –

1) Orphaned Agent blog

https://kevinjustin.com/blog/2020/07/02/identify-orphaned-agent-properties/

This scenario usually entails deleting server from Managed Agents view in Administration tab, where alerts or other details may still exist. The procedure leverages Holman’s orphaned agent blog (tried and true) post from years back to aid cleanup.

2) Do you have packs or connectors extending classes?

https://kevinjustin.com/blog/2021/12/14/caution-using-tags-notes-extending-classes/

3) Rebuilding a server with the same name is a common server occurrence

Related to 1, Holman’s orphaned agent blog to be used to cleanup.

First, let’s test in the Lab. Second, let’s talk about the express lane ‘easy button’. Begin by deleting the Windows Computer orphaned object GUID. Process the Windows Computer object (bottom), followed by the top two (2) are HealthService, and HealthServiceWatcher object properties (see three items highlighted).

Second piece, marking the Windows Computer GUID for deletion (IsDeleted=1) cleans up nearly ALL properties. See the progress below, how this slight change makes short order of orphaned properties for ‘server’.

Windows Computer object marks all but SCVMM for deletion

Third HealthService & SCVMM objects require manual deletion per GUID.

Note first screenshot shows health service properties marked ‘IsDeleted’ = 1 after manually processing each GUID.

Fourth, screenshot shows there the HealthServiceWatcher property is marked for deletion (IsDeleted=1)

HealthServiceWatcher marked for deletion

If you have SCVMM, you will need to repeat for each of the SCVMM properties to clear out the orphans in the DB.

Why – the issue:

Typically, when servers are reimaged, i.e. NOT deleted from SCOM, there are two+ healthservice, HealthServiceWatcher, Windows Computer properties created for each image of example server. Additional properties may show duplicated for any class discoveries that are common to the old and new image.

NOTE: Deleting the current agent may clean up objects for that instance of the discovered server, but NOT the old server image.

ServiceNow Incident Integration

Time to integrate your Monitoring tools to ITSM tool. First, this blog post documents ‘ServiceNow Incident integration’. Second, let’s explain the common acronym in my experience is SNOW/SNow. Third, some background – ServiceNow has been around for some time as an Information Technology Service Management (ITSM), and discovery tool. As a SaaS solution, companies can purchase a subscription and integrate tools via RESTAPI to create/update/close events or incidents.

First, let’s begin to discuss SCOM notification methods. SCOM2022 adds a new capability with Teams integration. Second, most people are familiar with notification methods leveraging Email (html or not), perhaps SMS, but not so much command channel, calling some script in shell, PowerShell, etc. Generally, the command channel is basically a post processing script capability to execute notifications. Third, example tools where command channel might be used – BMC BEM (BMC Event Manager), BMC Remedy, xMatters, DerDack; SNOW integration within SCOM, using notification channels. Lastly, SaaS solutions (vendors like xMatters, and ServiceNow) allow RESTAPI crafted requests to take actions.

SNOW prerequisites

1) ServiceNow User/Password (or API key)

2) SNOW RESTAPI PowerShell needs to securely access credentials

For the Incident PowerShell, we store Credentials within Windows Credential Manager

3) Network connectivity to SaaS provider (use test-netconnection from SCOM MS to test connectivity over whatever port(s) vendor requires.

ServiceNow CallerID GUID

4) Production and Test URL’s (also required for network connectivity tests)

5) Access to SNOW UI to verify required fields and values for the script parameters.

Update incident script and begin testing.

Download script from GitHub repo https://github.com/theKevinJustin/SCOM-SNOWIncidents

Download script, and copy to monitoring repository

Copy to SCOM management servers (MS)

NOTE Path, to run from management server

Update script, with pre-reqs above –

Credential Manager stored ID

For more detail, look at parameter examples below to verify UI.

Update with customer/ServiceNow SNOW subscription specific values:

##Company## (customize SNOW short_description)

##Team## (customize SNOW short_description)$Channel = “Direct”
$ServiceNowURL=”https://##ServiceNowURL##/api/now/table/incident”
#$Proxy = “##CustomerProxyURL##”
$CallerID = “##GUID##”

# Test New-SNOWIncident.ps1

# Depending on how you want to randomly choose an incident

Lab example

$Alerts = get-scomalert -resolutionstate 0 | where { $_.Name -like “System Center*” }

Gather Critical, New alerts

$Alerts = get-scomalert -ResolutionState 0 -severity 2

Debug for warning alerts

$Alerts = get-scomalert -ResolutionState 0 -severity 1

# Debug

$Alerts[0] | fl ID,Name,Description,Severity,MonitoringObjectDisplayName

.\New-SNOWIncident.ps1 -AlertName $Alerts[0].Name -AlertID $Alerts[0].ID -Impact 4 -Urgency 4 -Priority 3 -AssignmentGroup “System Admin” -BusinessService “System Management” -Category Support -SubCategory Repair -Channel Direct

Example output

PS C:\Users\scomadmin\Desktop> .\New-SNOWIncident.ps1 -AlertName $Alert.Name -AlertID $Alert.ID -Impact 4 -Urgency 4 -Priority 3 -AssignmentGroup “System Admin” -BusinessService “System Management” -Category Support -SubCategory Repair -Channel Direct

TEST ServiceNow URL specified.

CredentialManager PoSH Module Installed, ModuleBase = C:\Program Files\WindowsPowerShell\Modules\CredentialManager\2.0

The System Center Management Health Service 5E04F804-8B71-6EB6-0101-DCBB58022498 running on host 16DB02.testlab.net and s

erving management group with id {E39F5F53-9FBB-9D7F-4BFE-5F0324630AE5} is not healthy. Some system rules failed to load.

16DB02

Warning

impact 4

urgency 4

priority 3

ServiceNow Credential NOT stored on server

SCOM maintenance schedules

Do your SCOM users need to know if a server is in scheduled maintenance? This came about as Aris asked questions.

First, let’s discuss specific maintenance mode and maintenance schedule scenarios users might ask. Second, determining IF scheduled maintenance enabled, running, about to run. Third, how does another user know when scheduled maintenance ends, allowing action and decision point to add/extend server maintenance. Fourth, whenever scheduled maintenance entered by one user, is NOT automatically seen by other roles. While product guidance states ‘maintenance schedules be added by someone in SCOM admin group’, self-service users still need visibility. Lastly, can we figure out a way to answer these questions. Given these points, users to be able to see server maintenance details. Also, can solution adhere to best practice ‘no alerts during planned maintenance’.

From PowerShell on SCOM MS

Get-SCOMMaintenanceScheduleList

$ScheduleList = Get-SCOMMaintenanceScheduleList

$ScheduleList.ID

$ScheduleList.ScheduleID.Guid

foreach ( $ID in $ScheduleList.ScheduleID)

{

$Schedule = get-SCOMMaintenanceSchedule -ID $ID

# $Schedule.MonitoringObjects ;

(get-scomclassInstance -id $Schedule.MonitoringObjects.Guid).DisplayName

# Debug endtime

$Schedule | ft User,ActiveStartTime,ActiveEndDate,ScheduledEndTime

}

Example Output

Workflows:

Scheduled Maintenance report task

Maintenance mode report – what’s about to end maintenance mode.

Obviously, expect both workflows into the ‘Proactive NOSC DailyTasks’ pack. GitHub repo https://github.com/theKevinJustin/ProactiveNOSCDailyTasks

Documentation:

Learn says maintenance schedules be added by someone in SCOM admin group – https://learn.microsoft.com/en-us/system-center/scom/manage-maintenance-mode-overview?view=sc-om-2019

SCOM 2022 Get-SCOMMaintenanceSchedule List syntax https://learn.microsoft.com/en-us/powershell/module/operationsmanager/get-scommaintenanceschedulelist?view=systemcenter-ps-2022 2019 here

Install SCOM agent MSI via command line

Shout out to AT and Aris for his help with this!

Install SCOM agent MSI via command line and roll into a ‘SCOM agent’ MECM application package.

Why an application/package? Many times, servers are built, and the IT Enterprise doesn’t find out device is NOT monitored until it’s involved in an outage. Proceed to add to your Configuration management tool will provide a consistent configuration to avoid the ‘NOT monitored’ configuration.

While there are a few options to build out a MCM/MEM/MECM/SCCM application/package to install the SCOM agent on managed windows machines. In August, I posted Neal’s updates to install/configure the SCOM agent. Neal’s scenario works great for a new environment, and it’s simple PowerShell. Also, if Neal’s scenario with PowerShell is NOT your preference, try the batch file below. Another key point, the script has logic for multiple forests and SCOM gateways (to connect said forests). Basically test the command line script, then roll into a MCM/MEM/MECM/SCCM application/package. Have the SCOM agent added to task sequence which consequently includes the SCOM agent in the default build.

Install SCOM agent command line script

First – NOTE each line has no hidden line feed/carriage returns

Second – go through the variables before testing

Variables to review/replace:

##SCOMManagementGroup## # SCOM Management group, seen in SCOM console on top left corner

SCOMManagementServer## # SCOM management Server in primary domain
##SCOMManagementServerGW## # SCOM first gateway
##SCOMManagementServerGW2## # Second gateway
##Domain## as forest FQDN
##Domain2## as forest FQDN

Determine domain –

‘systeminfo | findstr /i /c:”domain” | findstr /i /c:”##Domain##”

Example Lab output

C:\> systeminfo | findstr /i /c:”domain” | findstr /i /c:”testlab”
Domain: testlab.net

Third – Find/Replace ##hashtagged## variables and test away!

@echo off

cls

REM This script command line is for NEW installs of the agent (nothing prior)
REM Replace ##SCOMManagementGroup## SCOM Management group, seen in SCOM console on top left corner

ECHO Installing SCOM 2019 Windows Agent…

echo Determining domain….

REM Determine if machine is in DMZ….

systeminfo | findstr /i /c:”domain” | findstr /i /c:”DMZ” && (

echo Machine found to be in DMZ…installing for that enclave and management server: ##Domain2##”

echo.

%WinDir%\System32\msiexec.exe /i %~DP0MOMAgent.msi /qn USE_SETTINGS_FROM_AD=0 USE_MANUALLY_SPECIFIED_SETTINGS=1 MANAGEMENT_GROUP=##SCOMManagementGroup## MANAGEMENT_SERVER_DNS=##SCOMManagementServerGW## SECURE_PORT=5723 ACTIONS_USE_COMPUTER_ACCOUNT=1 NOAPM=1 AcceptEndUserLicenseAgreement=1

echo.

GOTO Logging)

REM Determine if machine is on domain….

systeminfo | findstr /i /c:”domain” | findstr /i /c:”##Domain##” && (

echo Machine found to be in ##Domain##…installing for that enclave and management server: ##Domain##

echo.

%WinDir%\System32\msiexec.exe /i %~DP0MOMAgent.msi /qn USE_SETTINGS_FROM_AD=0 USE_MANUALLY_SPECIFIED_SETTINGS=1 MANAGEMENT_GROUP=##SCOMManagementGroup## MANAGEMENT_SERVER_DNS=##SCOMManagementServer## SECURE_PORT=5723 ACTIONS_USE_COMPUTER_ACCOUNT=1 NOAPM=1 AcceptEndUserLicenseAgreement=1

echo.

GOTO Logging)

REM Neither DMZ or domain joined found above…look for alternate domain(s)

REM These all share this same management server: FOREST2.FQDN and Gateway server

%WinDir%\System32\msiexec.exe /i %~DP0MOMAgent.msi /qn USE_SETTINGS_FROM_AD=0 USE_MANUALLY_SPECIFIED_SETTINGS=1 MANAGEMENT_GROUP=##SCOMManagementGroup## MANAGEMENT_SERVER_DNS=##SCOMManagementServerGW2## SECURE_PORT=5723 ACTIONS_USE_COMPUTER_ACCOUNT=1 NOAPM=1 AcceptEndUserLicenseAgreement=1

:Logging

ECHO Installing SCOM 2019 Windows Agent Completed…>> C:\Windows\Logs\SCOM2019Agent.log

time /t >> C:\Windows\Logs\SCOM2019Agent.log

date /t >> C:\Windows\Logs\SCOM2019Agent.log

Lastly enjoy! In summary, I hope the ‘Install SCOM agent MSI via command line’ script helps get the agent deployed faster, and more consistent!

Delete PKI certificates

Don't you wish this were the certificates we worked with! — Don’t you wish this were the certificates we worked with!

Man, I wish I had a few of these certificates in my PKI infrastructure (portfolio)! Ever need to identify an expired certificate, and or delete the certificate? Depending on UAC, AppLocker, and other settings, the delete portion may require server logon.

Identify an expired certificate

Few ways to identify an expired certificate on servers.

Via RDP session to server

RDP to server, open MMC > Add Plug In > Certificates > for Computer

RDP to server, MMC, Certificates Plug In, Computer, expand certificate store to find expired certificate.

Via PowerShell

GCI Cert:\LocalMachine\*

The certificate store file path will vary the above command.

NOTE the SuperUser blog post will help decipher the folder name

Via ‘smart’ PowerShell across multiple machines

Courtesy of Vance Cozier!

“server1”,”server2″| % {invoke-command $_ -scriptblock {$env:ComputerName; GCI Cert:\LocalMachine\* }}

Via Monitoring for expired certificates

Using SCOM Certificate Monitoring or PKI monitoring management packs

Identify Expired certificates

Console navigation steps:

From SCOM console > Monitoring Tab

Expand PKI folder > Expand Certificates and CRLs folder

Click on ‘Certificates – Expired’ state view

SCOM Expired PKI certificates state view.

Did you know

Run PowerShell commands from Holman’s SCOM Management pack (blog)- github download

Run PowerShell from Monitoring tool

From SCOM console > Monitoring Tab > SCOM Management folder > SCOM Agents

Highlight agent(s)

From Tasks Pane > click on ‘Execute any PowerShell’ task

On the pop-up window, click Override

Adjust timeout to 70 (seconds)

Override command line

Run the following commands (various Certificate store examples provided for removing expired certificates)

Personal Certificates folder

Get-ChildItem Cert:\LocalMachine\My | ? { (( $_.Subject -like “*GlobalSign*” ) -OR ( $_.Issuer -like “*GlobalSign*” )) -and $_.NotAfter -lt (get-date) } | fl Subject,Issuer

Trusted Root folder

Get-ChildItem Cert:\LocalMachine\Root | ? { ( $_.Subject -like “*GlobalSign*” ) -OR ( $_.Issuer -like “*GlobalSign*” ) } | fl Subject,Issuer

Third Party Root

Get-ChildItem Cert:\LocalMachine\AuthRoot | ? { (( $_.Subject -like “*GlobalSign*” ) -OR ( $_.Issuer -like “*GlobalSign*” )) -and $_.NotAfter -lt (get-date) } | fl Subject,Issuer

Click OK button to close Overrides window

Click OK to run task

View task output

PS C:\Users\admin> Get-ChildItem Cert:\LocalMachine\Root | ? { ( $_.Subject -like “*GlobalSign*” ) -OR ( $_.Issuer

like “*GlobalSign*” ) } | fl Subject,Issuer

Subject : CN=GlobalSign Root CA, OU=Root CA, O=GlobalSign nv-sa, C=BE

Issuer : CN=GlobalSign Root CA, OU=Root CA, O=GlobalSign nv-sa, C=BE

Subject : CN=GlobalSign, O=GlobalSign, OU=GlobalSign Root CA – R2

Issuer : CN=GlobalSign, O=GlobalSign, OU=GlobalSign Root CA – R2

Once task completes, go back to the Certificates state view > highlight the certificate > Click the Rediscover Certificates task

Delete PKI certificates

After identifying correct certificate and folder in question that requires deletion:

1. Verify backup of certificate
2. Test delete via PowerShell window from SCOM

Get-ChildItem Cert:\LocalMachine\AuthRoot | ? { ( $_.Subject -like “*GlobalSign*” ) -OR ( $_.Isser -like “*GlobalSign*” ) } | Remove-Item -DeleteKey

Verify Task output shows certificate deleted
Refresh MMC GUI > Certificates plug in, and folder to verify certificate deleted
Go back to SCOM Console Monitoring Tab
Expand PKI folder
Expand Certificates and CRLs folder
Click on Certificates – Expired state view
In Tasks pane on right, click on Rediscover certificates task, Click OK to run
Open Health Explorer and reset monitor to clear alert

Alternatively, leverage PowerShell reset logic on SCOM MS, use blog

Documentation

SCOM Certificate monitoring TechCommunity blog https://techcommunity.microsoft.com/t5/system-center-blog/scom-management-pack-for-certificate-monitoring/ba-p/3619299

Download SCOM Certificate monitoring pack https://www.microsoft.com/en-us/download/details.aspx?id=104858

Long-standing PKI certificate pack and blog https://blog.topqore.com/new-version-pki-certificates-monitoring-pack-for-scom/

Identify certificate stores to windows folders https://superuser.com/questions/217719/what-are-the-windows-system-certificate-stores

STIG finding V-69223 https://www.stigviewer.com/stig/zos_tss/2016-06-30/finding/V-69223

Updated DNS2012R2 Addendum

Updated DNS2012R2 Addendum overrides. Learned a few new things with Overrides workspace views, and why Authoring pane > Management pack Objects > Overrides may not load.

When your management pack has improper overrides, expect the loading icon. This may be caused due to overrides, whether error is with target, class/rule/monitor.

Sometimes, an Object of class error gets your hopes up (pointing at a non-existent object).

Example when Overrides loads properly

When Authoring Tab Overrides view loads successfully.

If Overrides view will not load, try creating a workspace view for Overrides.

Navigation Steps:

From SCOM Console

Click on My Workspace

Right Click > New > Overrides Summary View

Select checkbox ‘with a specific override management pack’ checkbox, then the ‘specific’ link to choose management pack(s).

Select Specific Override management pack(s)

Choose unsealed management pack(s) with overrides

Can select all – OR pick a few to see what loads without errors

Click OK

If you get the loading screen and error, now begins the pack analysis.

Clicking on the ‘Show’ link points to a non-existent object

Microsoft.EnterpriseManagement.Common.ObjectNotFoundException: An object of class ManagementPackClass with ID 76e2559c-aaf4-b1ec-60cf-d40ab4102fbc was not found.

How did I know that?

Run get-SCOMClassInstance command from PowerShell or Operations Manager shell

Example output of ‘get-SCOMClassInstance -ID “76e2559c-aaf4-b1ec-60cf-d40ab4102fbc” ‘

Get-SCOMClassInstance output of the GUID listed in the console error.

Work on the Overrides of the affected XML packs, and Import.

Once corrected, the Workspace view loads successfully, finite!

Overrides Workspace view of addendum packs

Documentation

My Workspace https://learn.microsoft.com/en-us/system-center/scom/manage-web-console-my-workspace?view=sc-om-2022

Monitoring workspace https://learn.microsoft.com/en-us/system-center/scom/manage-using-monitoring-workspace?view=sc-om-2022