Check your delegation settings

 

Sometimes as the monitoring admin, you may be responsible to secure your servers, being told from Security/Cyber Teams about new vulnerabilities.  The vulnerabilities may be from Tanium, ACAS, Tenable or other security tools.   This article is how to help you secure related SCOM web console, and SSRS reporting sites against Unconstrained Delegation vulnerabilities CVE-2020-17049, also AD STIG V-36435.

 

First we need to identify IF this is a true finding.

Typically this comes from Server/SystemsAdmin with domain admin access:

From PowerShell run:

Get-ADComputer -LDAPFilter
“(userAccountControl:1.2.840.113556.1.4.803:=524288)”

After identifying SCOM servers with unconstrained delegation (scope of blog post is focused on SCOM in a hybrid scenario (prem/cloud), we need to resolve.

With domain administrator rights, open the Active Directory Users and Computers MMC snap-in.

In ADUC, change Find drop-down to Computers
In the Computer name text box, enter <SCOMServer>  and click search

Right click the server in the results box > Select Properties.

Select the Delegation tab.

ADUC view of lab server delegation setting

 

Select Trust this computer for delegation to specified services only > Use any authentication protocol.

Under Services to which this account can present delegated credentials, select Add.

In the new dialog box, select Users or Computers.

Enter <SCOMServer>, and then select OK.

Click the Add button to add services

Select the w3svc and www processes

Select OK.

ADUC GUI adding services for delegation on SCOM server

Once set in AD, reboot server.  Running ‘gpupdate /force’ may not apply AD changes to the server object.

After reboot, reach out to SCOM Admins to test webconsole authentication

From edge browser, go to SCOM web console, typically @ https://<SCOMServer>/OperationsManager

On the Monitoring tab, click on Active Directory dashboard on left

Verify authentication works

 

Documentation

Pentestlab – Detecting Unconstrained Delegation Exposures in AD Environment

Petri.com find and block unconstrained delegation

Learn.Microsoft.com unconstrained kerberos article

Explanatory documents on what/why

Remove Unconstrained Kerberos Delegation

 

Configure MMA agent via PowerShell

A car mechanic uses battery jumper cables to charge a dead battery.

 

Do you feel like a mechanic having to jump start the agent configuration like a dead car battery?   Assuming the Agent is already installed, you can configure the SCOM agent via PowerShell.  Even better when you can PowerShell remote to multiple systems.  I hope the PowerShell commands below help you master PowerShell to configure the SCOM side of the MMA agent (house).

 

powershell

/*
# Find/replace variables to your environment like Kevin Holman’s fragments!
##SCOMMGMTGROUP1##
##SCOMMGMTGROUP2##
##SCOMMGMTSERVER1##
##SCOMMGMTSERVER2##
#
*/

$SCOMAgent = New-Object -ComObject AgentConfigManager.MgmtSvcCfg
$SCOMAgent.GetManagementGroup(“##SCOMMGMTGROUP1##”);$SCOMAgent.GetManagementGroup(“##SCOMMGMTGROUP2##”)

# If mgmt groups are incorrectly set
$SCOMAgent.RemoveManagementGroup(“##SCOMMGMTGROUP1##”)
$SCOMAgent.RemoveManagementGroup(“##SCOMMGMTGROUP2##”)

restart-service healthservice

# Domain
$SCOMAgent.AddManagementGroup(“##SCOMMGMTGROUP1##”,”##SCOMMGMTSERVER1##”,5723)

# Verify agent config
$SCOMAgent.GetManagementGroup(“##SCOMMGMTGROUP1##”)
# If you have a second management group

$SCOMAgent.GetManagementGroup(“##SCOMMGMTGROUP2##”)

# Restart and test connectivity
restart-service healthservice

# Check connectivity
test-netconnection -port 5723 -computername ##SCOMMGMTSERVER1##

 

 

WebConsole APM hotfix for SCOM2012R2 and above

Vaccination Record - SCOM hotfix released for WebConsole/APM on SCOM2012R2 and above
Vaccination Record

SCOM hotfix released for WebConsole/APM on SCOM2012R2 and above, time for another SCOM shot!  Don’t forget your vaccination card 🙂

 

Let’s get started.  Time to fix the vulnerability for ‘SCOM hotfix released for WebConsole/APM on SCOM2012R2 and above’.  Read the support article, and assess what versions you have in your sandbox and production.  Once assessed, it’s time to test/implement/verify the fix applied.

 

Support article

https://support.microsoft.com/en-us/topic/update-for-idor-vulnerability-in-system-center-operations-manager-kb5006871-0e3a513a-ad80-4830-8984-2fc5a40ee7f7

 

 

SCOM WebConsole Hotfix links

(support.microsoft.com articles)

Specific support article for SCOM2019 UR3 Hotfix

SCOM2019 UR3 Hotfix support.microsoft.com article link

Specific support article for SCOM2016 UR10 Hotfix

SCOM2016 UR10 Hotfix support.microsoft.com article link

Specific support article for SCOM2012R2 UR14 Hotfix

SCOM2016 UR10 Hotfix support.microsoft.com article link

 

# Download (same EXE has all 3 SCOM versions)

https://download.microsoft.com/download/3/e/e/3eec1274-64d5-4285-84b9-c50800eb2dd2/KB5006871.EXE

 

 

Hotfix updates two paths on SCOM management server with the WebConsole role

Paths updated

(don’t forget to add File Version property to your display)

NOTE Drive letter depends on where you installed SCOM (typically D:)

 

SCOM2019 paths

D:\Program Files\Microsoft System Center\Operations Manager\WebConsole\AppDiagnostics\Web\bin

D:\Program Files\Microsoft System Center\Operations Manager\WebConsole\AppDiagnostics\AppAdvisor\Web\Bin

SCOM2016 paths

D:\Program Files\Microsoft System Center 2016\Operations Manager\WebConsole\AppDiagnostics\Web\bin

D:\Program Files\Microsoft System Center 2016\Operations Manager\WebConsole\AppDiagnostics\AppAdvisor\Web\Bin

 

Screenshot of paths

AppDiagnostics File Path - SCOM hotfix released for WebConsole/APM on SCOM2012R2 and above
AppDiagnostics File Path
AppDiagnostics AppAdvisor File Path - SCOM hotfix released for WebConsole/APM on SCOM2012R2 and above
AppDiagnostics AppAdvisor File Path

 

Just in case you forgot how to add properties in Windows Explorer…

In the columns (Name, Date modified, etc,) right click > More

Add file property - SCOM hotfix released for WebConsole/APM on SCOM2012R2 and above
Add file property

Hit F to move down to the F named details > hit check box for ‘File Version’ or click on File Version and hit space bar

Click on OK

Add file property File Version - SCOM hotfix released for WebConsole/APM on SCOM2012R2 and above
Add file property File Version

 

Sort by ‘Date Modified’ Column

Verify File Version - SCOM hotfix released for WebConsole/APM on SCOM2012R2 and above
Verify File Version

 

File versions AFTER installing hotfix

Depending on which SCOM version you’re running, the path stays pretty much the same, and you want to verify that files were updated for the ‘SCOM hotfix released for WebConsole/APM’

SCOM2019

UR3 = 10.19.10505.0 > Hotfix file version = 10.19.10550.0

SCOM2016

UR10 = 7.2.12324 > Hotfix file version = 7.2.12335.0

Standard UR10 files are 8.0.10918.0

 

Voila > SCOM hotfix complete

Notify your Security team you’ve patched, because sometimes the scanner software isn’t accurately updated (where Security needs to open a case with their vendor!)

 

Complete:  Patched environment for ‘SCOM WebConsole/APM on SCOM2012R2 and above’

Mining Windows Event Log

Mining Ore from the Windows Event Log and finding a way to make it portable

 

Use Get-WinEvent to use XML and filters from event viewer, to mine an event, including examples for a specific string, from a specific event, in a specific event log?

 

 

Hopefully this post will help with a few tips to simplify monitoring for events, whether in AzMon, SCOM, or via PowerShell.

 

 

Let’s start with the Dr Scripto blog post from quite a while ago –

https://devblogs.microsoft.com/scripting/data-mine-the-windows-event-log-by-using-powershell-and-xml/

 

Not sure how many people use get-WinEvent, but this is one tool in PowerShell that can help an admin parse the XML side of an event.

 

Example 1

Query Application Event Log for Severity, Event, and Event Data contains lync.exe

$query = @”

<QueryList>

  <Query Id=”0″ Path=”Application”>

    <Select Path=”Application”>*[System[Provider[@Name=’Application Hang’]

    and (Level=2) and (EventID=1002)]]

    and *[EventData[Data=’lync.exe’]]</Select>

  </Query>

</QueryList>

“@

Get-WinEvent -FilterXml $query

 

PowerShell output

Use Get-WinEvent to use XML and filters from event viewer
Lync.exe event example output

 

 

 

Use Get-WinEvent to use XML and filters from event viewer

The Tip or Trick part of this – leverage your Event Viewer Filter as a query to use with get-WinEvent

Credit for this tip comes from Andrew Blumhardt!

See below for examples to ‘use Get-WinEvent to use XML and filters from event viewer’

 

Navigating via Event Viewer:

Hop onto your favorite server, or connect to another server via Event Viewer

Go to the Event Log > Click Filter Current Log

Build out your filter (i.e. choose specific Event Sources, exclude events, include severities, timeframe (start/end), etc.)

Use Get-WinEvent to use XML and filters from event viewer
SCVMM Application Log Event ID 25933

Switch to the XML tab (and note you can edit your query further!)

SCVMM query example screenshot
Event Viewer filter XML tab

You can copy the query from the Event Viewer into your Get-WinEvent syntax

$query = @”

<QueryList>
<Query Id=”0″ Path=”Application”>
<Select Path=”Application”>*[System[Provider[@Name=’Microsoft.SystemCenter.VirtualMachineManager.2012.Monitor.UserRoleQuotaUsageMonitor’ or @Name=’Microsoft.SystemCenter.VirtualMachineManager.2012.Report.ServiceUsageCollection’ or @Name=’Microsoft.SystemCenter.VirtualMachineManager.2012.Report.VMUsageCollection’ or @Name=’Microsoft.SystemCenter.VirtualMachineManager.2016.EnableCredSSPClient’ or @Name=’Microsoft.SystemCenter.VirtualMachineManager.2016.Monitor.UserRoleQuotaUsageMonitor’ or @Name=’Microsoft.SystemCenter.VirtualMachineManager.2016.Report.ServiceUsageCollection’ or @Name=’Microsoft.SystemCenter.VirtualMachineManager.2016.Report.VMUsageCollection’] and (Level=2 or Level=3) and (EventID=25933)]]</Select>
</Query>
</QueryList>

“@

Get-WinEvent -FilterXml $query

 

PowerShell output

Use Get-WinEvent to use XML and filters from event viewer
SCVMM query example screenshot

 

 

 

 

Example 3

Grab System Event Log, Event ID 5827  (NetLogon denied events)

get-WinEvent -FilterHashtable @{LogName=’System’; ID=’5827′;}

 

PowerShell output

Use Get-WinEvent to use XML and filters from event viewer
get-WinEvent filter by logname and event ID

 

 

Documentation:

Get-WinEvent https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.diagnostics/get-winevent?view=powershell-7.1

MSFT DevBlogs https://devblogs.microsoft.com/scripting/data-mine-the-windows-event-log-by-using-powershell-and-xml/

Don’t forget python as pre-req for agent install

Grocery List, items to get and notes
Grocery List

 

Hey guys, don’t forget python as pre-req for agent install!  Came across this again, where the docs site doesn’t mention python-ctypes as pre-req for agent install.  Let’s flip to GitHub for the agent.  GitHub lists the python pre-req here.  Otherwise, it’s Openssl 1.1.0 is only supported on x86_64 platforms (64-bit).

 

 

Let’s begin by starting with a Linux server.  I’ve used Ubuntu in my lab, specifically, Ubuntu v16.04.

Login and check if you have latest, or have the package installed (Debian Linux)

Regular user run   ‘sudo apt-get install python-ctypes

Super user/root     ‘apt-get install python-ctypes

 

screen capture of python install

 

As of 1 July, v1.13.7.0 is current (latest) 64 bit OMS for Linux agent released.

 

 

 

 

 

References

GitHub link https://github.com/Microsoft/OMS-Agent-for-Linux

GitHub Agent Download (AzMon/ALA/OMS/SCOM agent for Linux ) https://github.com/microsoft/OMS-Agent-for-Linux/releases/download/OMSAgent_v1.13.7-0/omsagent-1.13.7-0.universal.x64.sh

Python requirements https://github.com/Microsoft/OMS-Agent-for-Linux#python-requrements

Install guide https://github.com/Microsoft/OMS-Agent-for-Linux#azure-install-guide

Troubleshooting Service Map pack

 

 

 

Updated 14 Mar 2019

 

If you get these exceptions like me, the issue has been raised, with a deliverable targeted for SCOM2019UR1.

Disable the rule to reduce noise.

 

 

Are you using Service Map Management pack, and getting errors?

 

This alert is based on the 46651/46652 event ID in the Operations Manager event log

From SCOM Console > Authoring Tab > Management Pack Objects > Rules

Search in ‘Look for:’ bar GenericException (yes no space in between)

 

Rule

 

 

Rule Details

 

To enable debug on the MS

 

For collecting logs, please do the following:

  • Create folders “c:\Debug\ext\”
  • Now, Wait for an hour(which is the default time interval set in the rule for running service map api).
  • You will see some log files created in that folder “ext”. Please share the same in email.

 

The file showed up after the alerts, and listed debug INFO and WARN lines, and the time stamps match up to the generic exception rules.

 

Stay tuned for more information, I have been trying to get more answers on the exception

{WARN} [12:35:20.966] [ScomUtils] failed to export XML for Management Pack: System.NullReferenceException: Object reference not set to an instance of an object.

   at ScomBridge.ScomUtils.WritePackXmlToFile(ManagementPack pack, String filename)

 

 

Gather Log Analytics/MMA agent version

Had some questions come up from the community to check the Log Analytics agent version.

Depending on how you are setup, the SCOM Integration makes this easy with Holman’s blog for the agent management pack.

If you have admin right in Operations Manager console then you can check this directly from SCOM server:

If you are an admin in SCOM, you can check from MS

$Server = “DC01.yourlabnamehere.net”
(Get-SCOMAgent -Name $ServerName).Version

Alternatively, from server registry:

(Get-ItemProperty “HKLM:\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\setup”)

# Just the Agent version variable

(Get-ItemProperty “HKLM:\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\setup”).AgentVersion

Log Analytics

Kusto query

// Servers and Versions

Heartbeat
| project Computer,Version

// Specific version

Heartbeat
| where Version == “8.0.10918.0”
| project Computer,Version

// Summarize by Version

Heartbeat
| summarize by Version

If you’re visual

From the Portal > Log Analytics > workspace > Workspace Summary > Agent Health

Scroll right to agent version

Monitor

Monitor > Overview > Agent Health Assessment

Scroll right to agent version

Azure Log Analytics for Windows Telemetry data

 

 

I blogged about this last year here

 

 

As best practice, the Upgrade Analytics script checks for far more than just injecting the workspace key and telemetry value.

 

 

FYI – This could also be managed in an SCCM Compliance setting.

Paul Fitzgerald – Platform PFE blogged about a non SCCM method here

 

 

Assess requirements for environment:

 

Barebones configuration requires Commercial ID, allow telemetry, and level of telemetry data to send

Optional – Create key for IEDataOptIn

Send data to Application Insights

Customer proxy setup

 

 

Script has 11 parameters specified, not all are needed (excerpt below from script)

Param(
# run mode (Deployment or Pilot)
[Parameter(Mandatory=$true, Position=1)]
[string]$runMode,

# File share to store logs
[Parameter(Mandatory=$true, Position=2)]
[string]$logPath,

# Commercial ID provided to you
[Parameter(Mandatory=$true, Position=3)]
[string]$commercialIDValue,

# logMode == 0 log to console only
# logMode == 1 log to file and console
# logMode == 2 log to file only
[Parameter(Mandatory=$true, Position=4)]
[string]$logMode,

#To enable IE data, set AllowIEData=IEDataOptIn and set IEOptInLevel
[Parameter(Position=5)]
[string]$AllowIEData,

#IEOptInLevel = 0 Internet Explorer data collection is disabled
#IEOptInLevel = 1 Data collection is enabled for sites in the Local intranet + Trusted sites + Machine local zones
#IEOptInLevel = 2 Data collection is enabled for sites in the Internet + Restricted sites zones
#IEOptInLevel = 3 Data collection is enabled for all sites
[Parameter(Position=6)]
[string]$IEOptInLevel,

[Parameter(Position=7)]
[string]$AppInsightsOptIn,

[Parameter(Position=8)]
[string]$NoOfAppraiserRetries = 30,

[Parameter(Position=9)]
[string]$ClientProxy = “Direct”,

[Parameter(Position=10)]
[int]$HKCUProxyEnable,

[Parameter(Position=11)]
[string]$HKCUProxyServer

 

 

 

Simple method to update machines to send Windows telemetry data:

 

 

PowerShell script

From PowerShell as Administrator

Set-Location HKLM:

 

$registryPath = “HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies”

$Name = “DataCollection”

$Name2 = “AllowTelemetry”

$CommercialID = “00000000-0000-0000-0000-000000000000”

$value = “2”  # Values from 0-3 accepted

$vIEDataOptInPath = “HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\DataCollection”

$IEOptInLevel = “2”  # Values from 0-3 accepted

 

If ( (Test-Path $registryPath\$Name) ) { write-host -f green “Registry keys already exist” }

If ( ! (Test-Path $registryPath\$Name) )

{

New-ItemProperty -Path $registryPath -Name $name

New-ItemProperty -Path $registryPath -Name $CommercialID

New-ItemProperty -Path $vIEDataOptInPath -Name IEDataOptIn -Type DWord -Value $IEOptInLevel

New-ItemProperty -Path $registryPath\$Name -Name $name2 -Value $value `

    -PropertyType DWORD -Force | Out-Null

Write-host -f green “Registry keys added for Telemetry”

}

 

 

 

 

References

Configure telemetry

Get Started link

Win 7,8 Opt in link

Test fire any event on any server from any application

Golden Oldies – always popular (tools vs music)

Old Holman blog that’s still relevant, even more powerful than EventLog Explorer

Basically anyone who wants to test fire events off a SCOM MP should use this tool.

Event Create, write-eventlog all have limitations (certain event sources that can be used to create events, or event ID number limitations)

First, download the 2007 R2 Admin ResKit here

MomTeam blog reference

Double click the downloaded MSI

I prefer to move extracted files under my SCOM tools/Management pack directory structure under MonAdmin (Monitoring Admin)

Copy extracted files to gold depot

Move to gold depot – SCOM \ tools \ <toolname here>

Go into the MPEventAnalyzer directory

Run the exe

MP Event Analyzer

Click on Investigate Event Sources Tab (bottom middle)

Don’t forget you can use the search bar (where I typed apm)

For my example, double click on APM Agent

Search Events on right hand pane

Check checkbox to select the 1319 APM event for configuration error (right hand pane)

Click the ‘Add selected events to execution list’

Once event verified in bottom box, click the green box to fire selected event(s)

Verify event in Event Viewer

Validate Management Pack

Stay tuned… this did not complete the validation process.

Re-learn an old but still relevant tool – EventLog Explorer

 

Sometimes we forget about tools that can make things easier.

 

Time to talk about EventLog Explorer.

 

Need to repro and test events for an installed program, to see what SCOM will handle?

Read this old mom team blog, courtesy of Kevin Holman blog

 

 

I wanted to try it to test fire some events, had a use case where we needed to test Skype events from the SCOM MP

 

Testing on my SCOM 2016 Management server

 

Download file, run EventLog Explorer

The Paste icon next to the X is ‘Add to Execution List’ and fills out the bottom pane

The Green Arrow is ‘go’ or execute (similar to PowerShell ISE)

 

Navigate through the Event Log and Event Source on the left hand pane

Mark events with the checkbox  

 

Add to Execution

 

Verify events added to bottom pane

(see my test yesterday for fired, and not fired events from today)

 

 

 

Click Green box with white arrow to fire events, and check Event Viewer

 

 

Yesterday’s test

 

 

 

Today’s test

 

 

Verify alerting occurred as expected!