SCOMCore Addendum – having a strong core makes bigger gains
Updated SCOMCore addendum pack now contains DWDataRP integration, and additional overrides since the last pack posted in 2023. There’s been a lot of updates made since the last update to GitHub. Github Link https://github.com/theKevinJustin/SCOMCoreAddendum
Holman’s blog for DWDataRP is one way to Alert on DWDataRP findings
I want to alert on DWDataRP output! While everyone’s familiar with Holman’s SCOM SQL queries blog, read below to configure a new way to maintain data warehouse integrity and retention.
Data warehouse audits are included in monitoring platform checks. For those new to monitoring, basically DWDataRP analyzes SCOM Data Warehouse issues for alert/event/performance/state retention. One administration option is to utilize the SCOM Core Monitoring Addendum pack to run DWDataRP. Another option is to run DWDataRP via Holman’s blog, or recently with Blake Drumm’s GUI tool.
Using the SCOM Core Monitoring addendum pack pre-configures a number of overrides, as well as adding DWDataRP monitor/rule options. Consequently, the SCOM action account needs to have additional permissions on SCOM SQL servers where the OperationsManagerDW
resides.
Configure SCOM management server action account to alert on DWDataRP output
Example uses lab environment SVC.SCOM.PBIreader
Substitute the SCOM action account above for the SCOM data warehouse (OperationsManagerDW) databases on their respective SCOM management group(s).
Give SCOM Action account necessary rights
Update SVC account rights
Set and verify SVC account has Server role public
Click on User Mapping > select OperationsManagerDW database
Verify Default Schema shows DBO
Under Database Role Membership
Select db_datareader AND db_owner
Click OK
Verification
Reach out to SCOM team to verify execution
From SCOM, RDP to one of the management servers
Click on Start > Right click on Windows PowerShell
Click on More > click on Select PowerShell
Click on More > Click on 'Run as a different user'
Open PowerShell > right click > Run as a different user
On the Windows Security pop-up > Click on 'Use a different account'
Type the action account username and password
Click OK
Click Use a different account in the ‘Run As different user’ popup
Paste in the following commands, and verify output
cd "##YourPathtoDWDATARP.EXE##"
# cd D:\MonAdmin\TOOLS\DWDataRP"
# Check events
$Command = '.\dwdatarp.exe -s 16DB02 -d OperationsManagerDW -ds "Event Data
Set"'
$EventDataSet = Invoke-Expression $Command
$EventDataSet
$EventDataSet[2]
$LLineSplit = $EventDataSet[2].Split("(")
$EventDBPercent = $LLineSplit[1].Split("%")
$EventDBPercent[0]
DWDataRP PowerShell event output
Example PowerShell output when SVC Account cannot execute DWDataRP
PS C:\monadmin\tools\dwdatarp> whoami
testlab\svc.scom.pbireader
PS C:\monadmin\tools\dwdatarp> .\dwdatarp.exe -s 16db02 -d OperationsManagerDW -ds
Event
Dataset name
Aggregation name Max Age Current Size, Kb
----------------------------------------------------------------------------
For a smooth install, everything comes down to SCOM SSRS prerequisites. The SCOM Reporting role install really comes down to three (3) things – permissions, latest SSRS EXE downloaded (for install 2019, 2022), and ReportExtensions configuration. The go-to reference is Holman’s QuickStart deployment guides for SCOM2019 forward list the how-to starting point. This post focuses on ReportExtensions configuration, where more ‘how to’ details are needed.
Depending on requirements, creating multiple subscriptions within SCOM to leverage subscriber/channels required. Selecting rules/monitors, and resolution state conditions to help Application teams get incidents for key issues requiring intervention. NOTE Depending on what was command channels were created for various AssignmentGroup(s) and Team(s) within the organization.
Configure channel to execute logAlert.ps1 command channel to verify SCOM outputs
SCOM Navigation steps:
Click on Administration Tab > Notifications > Channels
Click New
Name
TEST Holman’s Command Channel
Description
C:\MonAdmin\Scripts\LogAlert.ps1 Utilize LogAlert.ps1 example from Holman’s blog. Specific Subscription details: +CRITERIA = ALL Alerts +RESOLUTIONSTATE = NEW (0) +SUBSCRIBER = CHANNEL SCOM Command Channel Subscriber via POWERSHELL +CHANNEL Test LogAlert.ps1 SCOM Command Channel
Setup and use Holman’s script execution channel blog to test what account SCOM uses for notifications
NOTE Use these steps to create multiple command channels, as the AssignmentGroup and Team may differ depending on Application Owners
SCOM Navigation steps:
Click on Administration Tab > Notifications > Channels
Click New
Name
TEST SNOW Event Creation
Description
C:\MonAdmin\Scripts\New-SNowEvent.ps1 Outputs 711 Events into Operations Manager event log.
Specific Subscription details: +CRITERIA = ALL Alerts +SUBSCRIBER = CHANNEL New-SNowEvent.ps1 via POWERSHELL +CHANNEL ServiceNow SNOW Event Creation Channel
New-SNOWEvent.ps1 command channel creates ServiceNow SNOW events for alerts and incidents.
This channel will also update the SCOM alert TicketID, Owner, ResolutionState to modify SCOM alert with SNOW information, or information passed in SNOW event.
NOTE Use these steps to create multiple command channels, as the AssignmentGroup and Team may differ depending on Application Owners
SCOM Navigation steps:
Click on Administration Tab > Notifications > Channels
Click New
Name
TEST SNOW Event Creation
Description
C:\MonAdmin\Scripts\New-SNowEvent.ps1 Outputs 711 Events into Operations Manager event log.
Specific Subscription details: +CRITERIA = ALL Alerts +SUBSCRIBER = CHANNEL New-SNowEvent.ps1 via POWERSHELL +CHANNEL ServiceNow SNOW Event Creation Channel
New-SNOWEvent.ps1 command channel creates ServiceNow SNOW events for alerts and incidents.
This channel will also update the SCOM alert TicketID, Owner, ResolutionState to modify SCOM alert with SNOW information, or information passed in SNOW event.
Read the ‘Configure SCOM Subscribers’ blog to build out the SNOW subscribers for multiple PowerShell command channels. Create subscribers according to design requirements.
CHANNEL New-SNowEvent.ps1 via POWERSHELL
Follow the screenshots and fill in the wizard per the steps below.
Time to update SCOM, specifically to ‘create SCOM Command Channels’, then subscribers, and subscriptions. Depending on requirements, create multiple channels within SCOM.
Save .ps1 file(s) to SCOM MS
LogAlert.ps1 to verify SCOM notification account
New-SNowEvent.ps1 to inject events into ServiceNow
New-SNowIncident.ps1 to inject incidents into ServiceNow
Save LogAlert.ps1
Create Command channel script and save to SCOM MS(s)
SNOW Event command channel injects ServiceNow SNOW events, with logic to check for alert, incident, then update the SCOM alert TicketID, Owner, ResolutionState based on runtime.
SCOM Navigation steps:
Click on Administration Tab > Notifications > Channels
Click New
Name
ServiceNow SNOW Event Creation Channel
Use New-SNowEvent.ps1 to create SCOM subscription that creates SNOW incidents, then updates SCOM alert with TicketID, Owner, Resolution State for SCOM alert.
This command channel helps admins determine variables possible to pass to PowerShell script(s).
SCOM Navigation steps:
Click on Administration Tab > Notifications > Channels
Click New
Name
ServiceNow SNOW Incident creation channel
Use New-SNowIncident.ps1 to create SCOM subscription that creates SNOW incidents, then updates SCOM alert with TicketID, Owner, Resolution State for SCOM alert.
Time to ‘Test SNOW script’ for event or incident injection. As long as the prerequisites are verified, to include network connectivity, URL, ID, Password, etc., we’re ready to go!
Once the CredentialManager piece has been completed, by the same token you can begin testing the script. Testing can begin, whether to the SCOM Admin SA account, or to the SCOM Notifications account, or even hard coding the values into your PowerShell session.
Begin script testing
The testing leverages that you’ve downloaded various integration scripts first, then being saved on SCOM MS (management servers). The following blog posts, GitHub repo’s will set up multiple methods to test from PowerShell (command line) as SA or SVC accounts.
New-SNOWEventPowerSHellOutput for REST Event injection.
Verify SCOM alert updated for ServiceNow REST injection
Check SCOM console/web console for SCOM alert updates to ResolutionState, TicketID, Owner fields, where TEAM = SYM, and Assignment Group = JustinTime Infra specified
SCOM Monitoring tab Active Alerts output view showing Owner, ResolutionState, and TicketId fields updated.
Be aware of issues
Indicator of Certificate/trust issue
Invoke-RestMethod error seen when organization cert not installed, making server sending REST injection NOT trusted.
Indicator when SNOW alert rule not configured or matching – excessive retry’s. Also note output shows summary of tests, ServiceNow SNOW detail, and SCOM alert updates.
New-SNOWEvent.ps1 failures when SNOW Alert rule not matching or created.
Logging to Operations Manager Event Log for addtional troubleshooting or debug. Unless otherwise updated, the script logs to the ‘Operations Manager’ event log, EventID = 710-712
Single Starting event indicates failed pre-requisite (pre-req NOT met)
SCOM OperationsManager events logging integration script status, as well as whoami and additional debug.
With domain joined machine, use a separate notification services svc account for notifications. SCOM is typically leverages MSAA, or even local system, depending on the accounts used when building out SCOM. Kevin Holman did an excellent job blogging this here
Verify SCOM notification account
Verify and ‘Setup SCOM Notifications account’ to separate notifications outside typical SCOM service SVC account functionality. Also, separating allows CredentialManager to secure, encrypt, and store credentials used by the notification account. Time to verify!
RDP to SCOM MS using notification account.
Open SCOM Console
Click on Administration tab
Expand Run As Configuration
Click on Accounts
Search for notification
Double click on Notifications account
Click on Credentials tab
Verify account being used, in light of CredentialManager piece storing SNOW ID and account.
NOTE Account should be part of SCOM Admins AD group
SCOM console view of Notifications SVC Account
SCOM Notifications Event Log troubleshooting
Knowing the notifications account will aid with SNOW integration scripts, as well as help log whoami, ‘run as’ logging to the ‘Operations Manager’ event log. The specific test and event or incident scripts leverage EventID’s 710-712. 710 for LogAlert.ps1, 711 for New-SNowEvent.ps1, and 712 for New-SNowIncident.ps1.
Single Starting event indicates failed pre-requisite (pre-req NOT met)
SCOM OperationsManager events logging integration script status, as well as whoami and additional debug.
Best practice – Encrypt credentials on SCOM MS to prevent cleartext in scripts. To begin, this includes details to verify SnapIn, verify credentials stored, store credentials.
Verify CredentialManager Snapin is installed
RDP to SCOM MS server(s) as SCOM Notifications SVC account with SA access
Open PowerShell as administrator
Paste command(s) into PowerShell window to test network connectivity to SNOW environments
{ write-host -f red “CredentialManager PoSH Module NOT Installed” }
CredentialManager snapin PowerShell output
Verify Stored Credentials on server
First, verify any credentials stored on server, specific to ServiceNow or not. Second, we begin to use the Get-StoredCredential command. Third, we will setup the credential for REST integration, lastly verifying credential.
Setup Credentials for SNOW RESTAPI
RDP to SCOM MS server(s) as SCOM Notifications SVC account with SA access
Open PowerShell as administrator
Paste command(s) into PowerShell window to test network connectivity to SNOW environments
If no output, there are no credentials stored under the RDP login.
Get-StoredCredential
Commands specific to ServiceNow to verify credentials exist
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
![dir gci New-SNow[ie]*.ps1](https://kevinjustin.com/blog/wp-content/uploads/2024/05/New-SNOWie.png)
