SCOM WebConsole settings for Kerberos AD Delegation

I attribute Kerberos AD delegation as the Navajo and Comanche helped allies in WW2, encrypted and encoded communication

Next on the list is to setup SCOM WebConsole settings for Kerberos AD Delegation. I attribute Kerberos AD delegation as the Navajo and Comanche helped allies in WW2, encrypted and encoded communication. Time to make the donuts! (to setup SCOM WebConsole settings for Kerberos AD Delegation)

If you’re improperly setup – you’ll flag on STIG configs V-243470, V-243478

Documentation

https://www.sentinelone.com/blog/detecting-unconstrained-delegation-exposure

https://pentestlab.blog/2022/03/21/unconstrained-delegation/

Outline

Assess affected unconstrained delegation servers in environment

Configure delegation on SCOM and/or PowerBI servers

Assess affected unconstrained delegation servers in environment

From a computer, with ADUC, and RSAT feature installed, search for relevant account(s) used (Read Only RO access displayed below).

Alternatively, from PowerShell > run this command to see affected servers (much wider list, unless you add a where clause)

Get-ADComputer -LDAPFilter

“(userAccountControl:1.2.840.113556.1.4.803:=524288)”

Configure delegation on SCOM and/or PowerBI servers

Take the list of affected servers, to take action. Use the steps below to configure relevant SCOM or PowerBI servers.

Configure SCOM Web Console server
With domain administrator (DA or Tier0) rights, open the Active Directory Users and Computers MMC snap-in.

From ADUC > change ‘Find’ drop-down to Computers

In the Computer name text box, enter <SCOMWebConsoleServerName> and click search

Right click the server in the results box > Select Properties.

Select the Delegation tab.

Select Trust this computer for delegation to specified services only > Use any authentication protocol.

Under Services to which this account can present delegated credentials, select Add.

In the new dialog box, select Users or Computers.

Enter <SCOMWebConsoleServerName>, and then select OK.

Click the Add button to add services

Select the w3svc and www processes

Select OK.

ADUC SCOM Lab server choosing process

Verification of delegation settings

ADUC Delegation flags with SCOM MS processes selected.

Depending on replication times for the forest, wait and later reboot <SCOMWebConsoleServerName> to have settings take effect.

PowerBI Report Server

With domain administrator (DA or Tier0) rights, open the (ADUC) Active Directory Users and Computers MMC snap-in. NOTE: RSAT tools recommended to be installed on SCOM Management Server(s)

In the Search text box, enter PowerBI service account <Example can be SCOMDataAccessReader Account> and click search

Right-click the PowerBI service account <Example can be SCOMDataAccessReader Account>, select Properties.

Select the Delegation tab.

Select Trust this computer for delegation to specified services only > Use any authentication protocol.

Under Services to which this account can present delegated credentials, select Add.

In the new dialog box, select Users or Computers.

Enter the service account for the data source, and then select OK.

Select the SPN that you created for <PowerBI Report Server Name>

Select both as FQDN and the NetBIOS names are in the SPN

Select OK.

Back to ADUC (AD Users and Computers), change Find drop-down to Computers

Enter <PowerBI Report Server Name>, and click search

Right click the server in the results box > Select Properties.

Select the Delegation tab.

Select Trust this computer for delegation to specified services only > Use any authentication protocol.

Under Services to which this account can present delegated credentials, select Add.

In the new dialog box, select Users or Computers.

Enter <Example can be SCOMDataAccessReader Account>, and then select OK.

Click the Add button to add services

Select the HTTP process

ADUC Delegation Add Services > HTTP, WWW

Select OK.

ADUC Delegation Settings for http for PowerBI Report Server (PBIRS)

Security – ODBC Vuln 175441

Time to make the doughnuts again, new Security ODBC Vuln 175441 that needs to be mitigated. Not sure if you ever saw the commercials, but this is where my mind goes sarcastic humor and all. Whether you’re using ACAS/Tenable/Nessus for security scans, this may show up with your SCOM servers (MS, DB), and PowerBI Report Servers.

Let’s get started to upgrade ODBC

Action: Security scan shows a new ODBC Vuln 175441, that may impact SCOM or PowerBI Report Server talking with SQL servers.

Start with some documentation, to understand what and why…

Tenable/Nessus Link to vulnerability

Download ODBC v18 here, v17 here

Outline of mitigation steps

What servers are vulnerable

Mitigate vulnerability on affected servers

Verify server Control Panel shows update

Have Security run additional scan to verify resolved

What servers are vulnerable?

We’re focused on the ‘Security – ODBC Vuln 175441’

Begin by looking at your Security scanning tool output (PowerBI report pictured). I am also showcasing the PowerBI report, as this streamlines what the Security Admin has to provide when System Administrators (sysAdmin) reach out for debug/details.

In my case, I wanted to see what servers are impacted. The PowerBI Report has a built-in ‘Deep Dive’ tab to see the details from the scan/check. Click on the Deep Dive Tab, enter the PlugIn ID (175441 for ODBC) and hit enter. This breaks out what servers are vulnerable. Assess what servers are yours (my output simplified to show what I own with SCOM and PowerBI 🙂 Looking at the ‘NetBIOS Name’ column. Alternatively, the admin typically has the scan tool email XLS files.

Access your ACAS/Tenable/Nessus scan deep dive tab (or PowerBI Report) to see how many systems are vulnerable.

Mitigate vulnerability on affected servers

Download ODBC v18 here , v17 here

Save to share or common path to put file on affected server(s).

Once moved, login to affected server(s), typically RDP with Local Administrator equivalent admin ID

Open Windows Explorer > Copy ODBC MSI to server

Open PowerShell (as Admin) window > Go to path > Run ODBCMSI

Now the ODBC popup window for install

Note the screenshots and progress prompts

Click ‘I accept’ radio button and then click ‘Next’

Click Next to move beyond the ODBC features screen

Click on Install

Watch progress bar (maybe 1-2 minutes)

Click Finished

Once the MSI installer window closes, it’s time to verify server Control Panel.

Verify server Control Panel shows update

Click on Start > Control Panel > Programs > Programs and Features

In the top right search bar, type ‘ODBC’ and hit enter to filter results.

Snapshot of Control Panel before

Control Panel with ODBC as the search string

Snapshot of Control Panel after

Hit F5 to refresh screen output

The one question is if version 17 has to be removed to clear vulnerability. Ran into this scenario with Java, as the update left old versions.

I typically reboot the server to reinitialize server to assess any impacts, as well as boot on the new drivers. For this instance, I coordinated my July server updates were installed to simplify my admin (as both require reboot!)

Have Security run additional scan to verify resolved

Typically SME has scheduled scans that run weekly, and can run scans on-demand. Depending on urgency, you can decide whether or not waiting is relevant.

Enjoy!

Microsoft links

Learn article here

Download ODBC v18 here, v17 here

Detected malicious verification code error

Ever run into the ‘detected malicious verification code’ error while authoring? I ran into the malicious verification error authoring, and couldn’t find any content for this error while authoring a pack.

Watch your copy/paste’s with additional monitoring changes to prevent ‘detected malicious verification code’ errors

In my authoring example, I received the ‘detected malicious verification code error’ after adding Rules, Datasources, and WriteActions (including tasks). I was copying and pasting DataSources (DS) and WriteActions (WA), thought I had it all. Uploaded > got the error, and GRR! Hopefully this will help others authoring to know what to check to get the management pack uploaded.

Simply put – Watch out for typo’s to avoid ‘detected malicious verification code’ errors!

I stumbled across a few websites, but nothing really pointed out to what caused the ‘detected malicious verification code error’ when uploading a management pack. First, check monitor and rules to verify the DS/WA are called correctly (no errors in file names. Check the Tasks as well as DisplayStrings, to make sure everything matches.

Error Seen when uploading Management pack from SCOM Console GUI regarding ‘detected malicious verification code’ error

<ManagementPackNameHere> Reports could not be imported.

If any management packs in the Import list are dependent on this management

pack, the installation of the dependent management packs will fail.

Verification failed with 1 errors:

——————————————————-

Error 1:

Found error in

2|<ManagementPackNameHere>|1.0.0.6|<ManagementPackNameHere>|

| with message:

Detected malicious verification code when verifying element of type

Microsoft.EnterpriseManagement.Configuration.ManagementPackRule with inner

exception: System.Collections.Generic.KeyNotFoundException: The given key

was not present in the dictionary.

at System.ThrowHelper.ThrowKeyNotFoundException()

at System.Collections.Generic.Dictionary`2.get_Item(TKey key)

Microsoft.EnterpriseManagement.Configuration.ManagementPackRule.VerifyDataTy

pes(Dictionary`2 moduletypes)

Microsoft.EnterpriseManagement.Configuration.ManagementPackRule.Verify(Verif

icationContext context)

Microsoft.EnterpriseManagement.Configuration.Verification.VerificationEngine

.VerifyCollectionItems(Object context)

Additional links

Detected malicious verification code when verifying element

Forum https://social.technet.microsoft.com/Forums/en-US/ac50ae14-882a-4788-a8e4-6a975c498a29/detected-malicious-verification-code-when-verifying-element-of-type

Caution using Tags/Notes extending classes

Please take ‘caution using Tags/Notes extending classes’. Please read below if you use Tags/Notes on SCOM classes. Ran across examples where SCOM Class Properties were used for tags that used the .Notes field on various classes, causing orphaned properties, NOT removed from OperationsManager database.

Background

The Microsoft.Windows.Computer Class (insert class here) is updated using Tim McFadden’s blog. This can cause issues with orphaned classes in the database because it is not currently handled as part of the stored procedure (i.e. the Notes property classes do not get marked for deletion).

First, identify which classes have Notes property. Start from Management Server (MS) via PowerShell. See attached TXT for additional examples to check and add/remove Notes Property on additional windows classes.

Set Notes property for Windows Operating System server

Second, we need to see how to set and clear the value, in order to clean up the Operations Manager database, to remove the orphaned instances. The example below sets the value for one (1) server to ‘Production’.

$WOS = Get-SCOMClass -name Microsoft.Windows.OperatingSystem | get-SCOMClassInstance | where-object -property Path -eq “16db01.testlab.net”
$WOS.'[System.ConfigItem].Notes’

$WOS.'[System.ConfigItem].Notes’.Value = “Production”

$WOS.Overwrite()

$WOS = Get-SCOMClass -name Microsoft.Windows.OperatingSystem | get-SCOMClassInstance | where-object -property Path -eq “16db01.testlab.net”

$WOS.'[System.ConfigItem].Notes’

Example Output

PS C:\Users\scomadmin> $WOS.'[System.ConfigItem].Notes’.Value = “Production”
PS C:\Users\scomadmin> $WOS.Overwrite()
PS C:\Users\scomadmin> $WOS = Get-SCOMClass -name Microsoft.Windows.OperatingSystem | get-SCOMClassInstance | where-object -property Path -eq “16db01.testlab.net”
PS C:\Users\scomadmin> $WOS.'[System.ConfigItem].Notes’

PropertyAccessRights : Unknown
Parent : Microsoft Windows Server 2016 Standard
Type : Notes
Value : Production
Id : 00000000-0000-0000-0000-000000000000
ManagementGroup : SCOM2016
ManagementGroupId : e39f5f53-9fbb-9d7f-4bfe-5f0324630ae5

Set Notes property to NULL

$WOS.'[System.ConfigItem].Notes’.Value = $null
$WOS.Overwrite()

$WOS = Get-SCOMClass -name Microsoft.Windows.OperatingSystem | get-SCOMClassInstance | where-object -property Path -eq “16db01.testlab.net”

Verify Notes value

$WOS = Get-SCOMClass -name Microsoft.Windows.OperatingSystem | get-SCOMClassInstance | where-object -property Path -eq “16db01.testlab.net”
$WOS.'[System.ConfigItem].Notes’

Example Output
PS C:\Users\scomadmin> $WOS = Get-SCOMClass -name Microsoft.Windows.OperatingSystem | get-SCOMClassInstance | where-object -property Path -eq “16db01.testlab.net”
PS C:\Users\scomadmin> $WOS.'[System.ConfigItem].Notes’

PropertyAccessRights : Unknown
Parent : Microsoft Windows Server 2016 Standard
Type : Notes
Value : (null)
Id : 00000000-0000-0000-0000-000000000000
ManagementGroup : SCOM2016
ManagementGroupId : e39f5f53-9fbb-9d7f-4bfe-5f0324630ae5

Have a happy Holiday!

Good luck, hopefully this scenario isn’t something that impacted the monitoring environment!

Parse Events via PowerShell into table

Parsing Events via PowerShell into table — Optometrist eye testing equipment picture

Parse Events via PowerShell into table. Ever have need to parse an event, and grab a field from the event description, then perform some action after that?

Here’s some PowerShell that may help you first to create a table, then setup columns, gather data, then parse what you need, and run a command to then output to the table

# Create Table for alerts

$Table = @()

$Table = $null

$Table = New-Object System.Data.DataTable “Failed Hosts List”

$Col1 = New-Object System.Data.DataColumn Host

$Col2 = New-Object System.Data.DataColumn IPAddress

$Table.Columns.Add($Col1)

$Table.Columns.Add($Col2)

$Alert20046 = Get-WinEvent -FilterHashtable @{LogName=’Operations Manager’;

ID=’20046′;}

$Alerts20046 = $Alert20046.Message

$Alerts20046.count

$Alerts20046uniq = $Alerts20046 | sort -uniq

$Alerts20046uniq.count

# $DeniedUniq = $Denied20046 | Sort-Object -Uniq

# $ServersDenied = @()

foreach ( $server in $DeniedUniq)

{

$Name = nslookup $server

foreach ($server in $Name)

{

# Add to Table

# $Name.Split(“:”)[6]

# $Name.Split(“:”)[8]

$row = $Table.NewRow()

$row.Host = $Name.Split(“:”)[6]

$row.IPAddress = $Name.Split(“:”)[8]

$Table.Rows.Add($row)

}

Identify orphaned agent properties

Back again, I’m going to ‘Identify orphaned agent properties’. For instance, does an agent still show up under Windows Computer, or more classes, like Windows Operating System? Typically we have handled this by using Holman’s purge blog.

Deleting and Purging data from the SCOM Database

First, my thanks to Kevin H, Mihai S from the SCOM PG, & Premier Support CSS, for their help. Let’s begin the ‘Identify orphaned agent properties’ discussion with ‘how’. First, how do you get an orphaned property? Second, how to you resolve?

Some example scenarios

1. Server rebuilt with same name. New agent runs discovery, and creates new set of GUID’s in the database.
2. The Monitoring Tab > Windows Computer view contains unhealthy <gray> server objects. Upon further inspection, the server does NOT show up in the Administration > Agent Managed view.
3. Custom management pack authoring extends the Windows Computer class, or others (via SDK or PowerShell)

‘Identify and resolve’ orphaned agent properties

1. Check for COMMIT or Overrides in management packs

PG recommended looking at Windows Computer extended class properties, and Connector Framework discoveries.

Microsoft.EnterpriseManagement.ConnectorFramework.IncrementalDiscoveryData.Commit()

Microsoft.EnterpriseManagement.ConnectorFramework.IncrementalDiscoveryData.Override()

Search for the ConnectorFramework

Search management packs (MP) via SCOM OpsDB (OperationsManager Database)

1. Login to your SCOM OpsDB > New Query

select MPName, convert(xml, MPXML)

from ManagementPack

where

MPXML like ‘%Commit(%’ or

MPXML like ‘%Override(%’

Export management pack output or snag it/snippet screenshot

Example Snapshot from SQL query

SQL Query output of Management Pack output with Commit or Override — SQL query of MP Commit or Override pack matches

FYI – mgmt packs above use %Commit(%, but not the connectorFramework

Correct discoveries that use ConnectorFramework

Replace Discoveries

Update discoveries that contain:

New-Object Microsoft.EnterpriseManagement.ConnectorFramework.IncrementalDiscoveryData.Commit()

New-Object Microsoft.EnterpriseManagement.ConnectorFramework.IncrementalDiscoveryData.Override()

Replace with:

New-Object -comObject MOM.ScriptAPI for discovery

Test discoveries that use Remove method

Microsoft.EnterpriseManagement.ConnectorFramework.IncrementalDiscoveryData.Remove()

Example management pack discovery script

Contains

$discovery = New-Object Microsoft.EnterpriseManagement.ConnectorFramework.IncrementalDiscoveryData

$discovery.RemoveInternal($Instance,$ClassInstance.GetClasses()[0])

$discovery.Commit($mg) <– This is the offender that causes the orphans

}

SCOM 1801 New Features (previously 1711 Technical Preview)

What does the new SCOM bring?!

HTML5 Web Dashboards! here

What’s New https://docs.microsoft.com/en-us/system-center/scom/what-is-new-1801?view=sc-om-1801

The SCOM Team published a 5 set blog post on Web Console https://blogs.technet.microsoft.com/momteam/2018/02/12/new-scom-web-console-blog-series-post1/

The new SCOM version also gives visibility into

Management Packs Updates and Recommendations

Operations Manager Products (view SCOM topology)

Partner Solutions

For now I’ll focus on the last two

Operations Manager Products

Click on Administration Tab

Expand Operations Manager Products

Topology features are pretty neat.

Whether you’re new to SCOM, or have interesting Server naming conventions, or someone built the environment and changed jobs or left the company…

At least you can easily find out how the SCOM environment is setup

Partner Solutions

Cool!

Silect MP Author, Comtrade, Nutanix, Veeam, Infront, ClearPointe, Backbone

OMS/Advisor Event ID 55002

This article is written for the Gateway CommunicationSecurityException event

At first I thought maybe this was TLS1.2 enabling, but backed off the change, the events kept pouring in every 5 minutes.

Tried to reconfigure the OMS/Advisor environment, and voila! Error resolved

Let’s go through the steps to re-configure the Operations Management Suite (OMS) in SCOM

Reconfigure OMS

From the SCOM Console, click on Administration tab
Expand Operations Management Suite (Advisor on 2012R2)
Click on the Connection
On the center pane, click on Re-configure Operations Management Suite

5. Add any trusted sites to IE if there are pop-ups

I had 2 missing websites

Secure.aadcdn.microsoftonline-p.com

az416426.vo.msecnd.net

( I hit Previous and next to verify the wizard would pass with the hopes the attempt would retry)

6. Exit the Reconfigure wizard to get a retry (then the second website popped up as an untrusted site)

7. Enter credentials to your OMS environment

Connection to OMS successful

Click Next twice

Reconfigure success

Click Close

Verify Event Log

Verify Operations Manager Event Log has no new events (this check runs every 5 minutes by default)

get-eventlog -logname “Operations Manager” | ? { $_.EventID -match 55002 } | select-object -last 2

Event ID 55002 from Operations Manager Event Log

Log Name:      Operations Manager
Source:        Advisor
Date:          12/11/2017 2:15:20 PM
Event ID:      55002
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      16MS01.testlab.net
Description:
Failed to synchronize the latest Management Package information from Advisor Cloud service. Wait for the next cycle to retry. Reason: Microsoft.SystemCenter.Advisor.Common.WebService.GatewayCommunicationSecurityException: Message security was invalid for the connection with web service when performing Get Intelligence Packs with client specified versions —> System.ServiceModel.Security.MessageSecurityException: An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail. —> System.ServiceModel.FaultException: ID3242: The security token could not be authenticated or authorized.
— End of inner exception stack trace —

Server stack trace:
at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.ProcessReply(Message reply, SecurityProtocolCorrelationState correlationState, TimeSpan timeout)
at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Request(Message message, TimeSpan timeout)

Exception rethrown at [0]:
at System.ServiceModel.Security.IssuanceTokenProviderBase`1.DoNegotiation(TimeSpan timeout)
at System.ServiceModel.Security.IssuanceTokenProviderBase`1.GetTokenCore(TimeSpan timeout)
at System.IdentityModel.Selectors.SecurityTokenProvider.GetToken(TimeSpan timeout)
at System.ServiceModel.Security.Tokens.IssuedSecurityTokenProvider.GetTokenCore(TimeSpan timeout)
at System.IdentityModel.Selectors.SecurityTokenProvider.GetToken(TimeSpan timeout)
at System.ServiceModel.Security.SecurityProtocol.TryGetSupportingTokens(SecurityProtocolFactory factory, EndpointAddress target, Uri via, Message message, TimeSpan timeout, Boolean isBlockingCall, IList`1& supportingTokens)
at System.ServiceModel.Security.TransportSecurityProtocol.SecureOutgoingMessageAtInitiator(Message& message, String actor, TimeSpan timeout)
at System.ServiceModel.Security.TransportSecurityProtocol.SecureOutgoingMessage(Message& message, TimeSpan timeout)
at System.ServiceModel.Security.SecurityProtocol.SecureOutgoingMessage(Message& message, TimeSpan timeout, SecurityProtocolCorrelationState correlationState)
at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [1]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Microsoft.AttachedServices.WebService.IIntelligenceService.GetIntelligencePacksInfo(ClientProperties clientProperties)
at Microsoft.SystemCenter.Advisor.Core.WebService.WebServiceCallHelper.CallWebService[T](Func`1 webServiceCall, String webServiceDescription)
— End of inner exception stack trace —
at Microsoft.SystemCenter.Advisor.Core.WebService.IntelligenceServiceClient.CallWebServiceWithRetry[T](Func`2 function)
at Microsoft.SystemCenter.Advisor.Core.WebService.IntelligenceServiceClient.GetIntelligencePacksInfo(ClientProperties clientProperties)
at Microsoft.SystemCenter.Advisor.Core.IntelligencePackWriteAction.UpdateIntelligencePacks()