Tag: scom 2016

Resolve HSTS vulnerability CVEs on IIS10

IIS Error 500 – Don’t let a vulnerability cause downtime with your SCOM web console

 

This article will help resolve security HSTS vulnerability CVEs on IIS10.  The steps apply to Windows Server 2016+, to help resolve multiple vulnerabilities, including CVE-2023-23915 CVE-2023-23914 CVE-2017-7789.   There are a few ways to configure IIS, and the blog post will show how to set up HTTP response, and HTTP redirect for the SCOM web console role’d server(s).

 

Setting HSTS on IIS10 to resolve with Server2016 1609

Open PowerShell window as Admin
cd c:\windows\winsxs
gci wow64_microsoft-windows-iis-shared* | ft Name

Example aim for latest directory
NOTE bottom entry based on software versioning

Example output
PS C:\windows\winsxs> gci wow64_microsoft-windows-iis-shared* | ft Name

Name
—-
wow64_microsoft-windows-iis-sharedlibraries_31bf3856ad364e35_10.0.14393.0_none_48b28891ffe5bdae
wow64_microsoft-windows-iis-sharedlibraries_31bf3856ad364e35_10.0.14393.1613_none_90c5a57843ef1621
wow64_microsoft-windows-iis-sharedlibraries_31bf3856ad364e35_10.0.14393.5246_none_90f3a94643cc33e1

# AppCMD lines
.\appcmd.exe set config -section:system.applicationHost/sites “/[name=’Default Web Site’].hsts.enabled:True” /commit:apphost
.\appcmd.exe set config -section:system.applicationHost/sites “/[name=’Default Web Site’].hsts.max-age:31536000” /commit:apphost
.\appcmd.exe set config -section:system.applicationHost/sites “/[name=’Default Web Site’].hsts.includeSubDomains:True” /commit:apphost
.\appcmd.exe set config -section:system.applicationHost/sites “/[name=’Default Web Site’].hsts.redirectHttpToHttps:True” /commit:apphost

 

 

For Server2016 1709 and greater

To add the HSTS Header, follow the steps below:

Open IIS manager.
Select your site.
Open HTTP Response Headers option.
Click on Add in the Actions section.
In the Add Custom HTTP Response Header dialog, add the following values:
Name: Strict-Transport-Security
Value: max-age=31536000; includeSubDomains; preload
Or directly in web.config as below under system.webServer:

<httpProtocol>
<customHeaders>
<add name=”Strict-Transport-Security” value=”max-age=31536000; includeSubDomains; preload” />
</customHeaders>
</httpProtocol>

NOTE iisreset may be required to restart IIS and apply settings

 

 

Verify HTTP Response Headers

From IIS10 (IIS Manager) > click on ‘Default Web Site’ > HTTP Response Headers

Verify Strict-Transport-Security blurb matches

HSTS IIS10 HTTP Response Headers screenshot verifying settings applied

 

 

Set HTTP Redirect

Now to set the HTTP redirect, to prevent denial of service (DoS) attacks.

From IIS10 (IIS Manager) > Expand ‘Default Web Site’ > HTTP Redirect

Screenshot

Default Web Site HTTP Redirect to SCOM web console URL

 

 

From IIS10 (IIS Manager) > Expand ‘Default Web Site’ > go through each Application to set HTTP redirect

Screenshot

Set HSTS HTTP Redirect on other web applications
Set HSTS HTTP Redirect on other web applications

 

Test your web console URL to verify components

 

 

References

NIST CVE-2023-23915 CVE-2023-23914

Mitre CVE-2017-7789

Blog link https://inthetechpit.com/2019/07/17/add-strict-transport-security-hsts-response-header-to-iis-hosted-site/

 

Configure MMA agent via PowerShell

A car mechanic uses battery jumper cables to charge a dead battery.

 

Do you feel like a mechanic having to jump start the agent configuration like a dead car battery?   Assuming the Agent is already installed, you can configure the SCOM agent via PowerShell.  Even better when you can PowerShell remote to multiple systems.  I hope the PowerShell commands below help you master PowerShell to configure the SCOM side of the MMA agent (house).

 

powershell

/*
# Find/replace variables to your environment like Kevin Holman’s fragments!
##SCOMMGMTGROUP1##
##SCOMMGMTGROUP2##
##SCOMMGMTSERVER1##
##SCOMMGMTSERVER2##
#
*/

$SCOMAgent = New-Object -ComObject AgentConfigManager.MgmtSvcCfg
$SCOMAgent.GetManagementGroup(“##SCOMMGMTGROUP1##”);$SCOMAgent.GetManagementGroup(“##SCOMMGMTGROUP2##”)

# If mgmt groups are incorrectly set
$SCOMAgent.RemoveManagementGroup(“##SCOMMGMTGROUP1##”)
$SCOMAgent.RemoveManagementGroup(“##SCOMMGMTGROUP2##”)

restart-service healthservice

# Domain
$SCOMAgent.AddManagementGroup(“##SCOMMGMTGROUP1##”,”##SCOMMGMTSERVER1##”,5723)

# Verify agent config
$SCOMAgent.GetManagementGroup(“##SCOMMGMTGROUP1##”)
# If you have a second management group

$SCOMAgent.GetManagementGroup(“##SCOMMGMTGROUP2##”)

# Restart and test connectivity
restart-service healthservice

# Check connectivity
test-netconnection -port 5723 -computername ##SCOMMGMTSERVER1##

 

 

SCOM 2016 web console hot fix released

Burglar stealing a monitor
Security hotfix for SCOM 2016 web console released before your information is stolen

SCOM 2016 web console hot fix

Security teams may be contacting you for CVE-2020-1331 vulnerability on the 2016 web console.  In my example, the Tenable scanner listed ALL SCOM management group servers – under SCOM2016/2019).

NOTE KB does not install on server, so does not show up under ‘Installed Updates’

 

 

Background

HotFix DLL comes with a readme to replace the DLL for the SCOM 2016 WebConsole role

If you don’t already know this, the roles each get their own directory on your SCOM server

Security scanners run scripts to help validate if system is vulnerable.  It is possible that the scanner is just looking for some string for the install of SCOM, NOT the actual role that is vulnerable.

 

SCOM 2016 typically installs @ (‘\Program Files\Microsoft System Center 2016’)

SCOM 2019 typically installs @ (‘\Program Files\Microsoft System Center’)

 

 

Identify SCOM roles

Open PowerShell window to identify roles

cd “D:\Program Files\Microsoft System Center 2016\Operations Manager”

 

 

Resolve Web Console vulnerability

High level steps

Download the KB here

Execute KB

Copy dll and readme file

Backup DLL and replace

Reboot server

Contact Security Team to re-scan server

 

Mitigate vulnerability

Download the KB here

Extract downloaded the KB

Click Run to extract, and list extraction path

Click Run for Security Warning pop-up
Click Run for Security Warning pop-up

Copy Windows Explorer Path you want to extract to, and paste in the path

Example

S:\MonAdmin\MSDN images\SCOM\2016\WebConsole HotFix

Enter path to extract Hot Fix
Extract Hot Fix

 

Copy current DLL & replace with hotfix DLL

Open PowerShell window (as admin)

# Backup DLL

# Change Drive letter if you hopefully installed SCOM on D: drive (non-system drive)

copy “C:\Program Files\Microsoft System Center 2016\Operations Manager\WebConsole\MonitoringView\bin\Microsoft.EnterpriseManagement.OperationsManager.MonitoringViews.dll”  “C:\Program Files\Microsoft System Center 2016\Operations Manager\WebConsole\MonitoringView\bin\Microsoft.EnterpriseManagement.OperationsManager.MonitoringViews-old.dll”

# Replace DLL

copy “C:\MonAdmin\Microsoft*.dll” “C:\Program Files\Microsoft System Center 2016\Operations Mana
ger\WebConsole\MonitoringView\bin”

 

# Verify

cd “D:\Program Files\Microsoft System Center 2016\Operations Manager\WebConsole\MonitoringView \bin”

gci Microsoft.EnterpriseManagement.Operations*.dll

 

Sample screenshot from Windows Explorer view of Bin directory for replaced DLL – Same size, only timestamp changes July Page 4

Windows Explorer window showing DLL's
Windows Explorer window showing DLL’s

Reboot server

 

Test WebConsole functionality

Verify from SCOM Console > Administration Tab > Settings > Web

Find Console URL
Find Console URL

Example

http://16ms01/OperationsManager

 

Contact Security Team to re-scan SCOM asset(s)

 

 

References

CVE-2020-1331 https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-1331

 

Microsoft Support article

https://support.microsoft.com/en-us/help/4566040/prevent-javascript-injection-in-operations-manager-2016-web-console

Tenable forum post https://community.tenable.com/s/question/0D53a000074LGapCAG/plugin-137369-security-updatesfor-microsoft-system-center-operations-manager

 

Windows Server 2016 vuln found in Security scans

FYI – came across this today with a customer where Security scans SCOM servers.

 

Please note this is NOT a SCOM issue or vulnerability, and SCOM uses TLS1.2 just fine.

 

Found CVE-2017-8529 vulnerability on a SCOM server, so though this a good idea to communicate to the larger audience, in case Security finds vulnerabilities, based on customer 2016 server hardening.

CVE-2017-8529 details:

The remote Windows host is missing security update KB4022715 or a Registry key to prevent the host against CVE-2017-8529. It is, therefore, affected by an information disclosure vulnerability exists in Microsoft browsers in the scripting engines due to improper handling of objects in memory. An unauthenticated, remote attacker can exploit this, by convincing a user to visit a specially crafted website, to disclose files on a user’s computer.

The easiest way I found to update the server was via these two registry keys (32 bit and 64bit keys below)

# KB4022715
# Add registry key
reg add “HKLM\SOFTWARE\Microsoft\Internet Explorer\Main\FeatureControl\FEATURE_ENABLE_PRINT_INFO_DISCLOSURE_FIX” /v “iexplore.exe” /t REG_DWORD /d 0 /f
reg add “HKLM\SOFTWARE\WOW6432Node\Microsoft\Internet Explorer\Main\FeatureControl\FEATURE_ENABLE_PRINT_INFO_DISCLOSURE_FIX” /v “iexplore.exe” /t REG_DWORD /d 0 /f

References

https://community.tenable.com/s/article/CVE-2017-8529-Plugins-returning-in-scan-results-Not-a-false-positive

https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2017-8529

Installing and configuring the MMA agent via Command line

Command prompt

 

GUI install option, see blog

PowerShell Agent configuration, see blog

Updated 1 Feb 2023

Pre-reqs to build out an install script/package

MMA agent executable

ALA Workspace ID

ALA Workspace Primary Key

 

 

Download MMA agent

Click on Windows Servers from Connected Sources to download Windows Agent

Click on Linux Servers from Connected Sources to download Linux Agent

 

 

 

 

Obtain WorkspaceID

From the Azure Portal (https://portal.azure.com)

Click on Log Analytics, <your subscription >

Click on Advanced Settings

My view defaulted to Connected Sources > Windows Servers

 

Save the workspace ID and workspace key to notepad/OneNote for later

 

 

 

 

 

Build out command line for setup file

(optionally to include in Application Deployment package)

 

Grab pre-reqs above: (saved from above to build the command line)

Exe/msi file

Workspace ID

Workspace key

 

Craft out your command line (MECM super installer code updated by Neal Smith

SCOM MECM Agent Package Installer Command Line

The setup.exe or MSI command line parameters to pass are:

MMA-specific optionsNotes
NOAPM=1Optional parameter. Installs the agent without .NET Application Performance Monitoring.
ADD_OPINSIGHTS_WORKSPACE1 = Configure the agent to report to a workspace
OPINSIGHTS_WORKSPACE_IDWorkspace Id (guid) for the workspace to add
OPINSIGHTS_WORKSPACE_KEYWorkspace key used to initially authenticate with the workspace
OPINSIGHTS_WORKSPACE_AZURE_CLOUD_TYPESpecify the cloud environment where the workspace is located

0 = Azure commercial cloud (default)

1 = Azure Government

OPINSIGHTS_PROXY_URLURI for the proxy to use
OPINSIGHTS_PROXY_USERNAMEUsername to access an authenticated proxy
OPINSIGHTS_PROXY_PASSWORDPassword to access an authenticated proxy

Example:

setup.exe /qn NOAPM=1 ADD_OPINSIGHTS_WORKSPACE=1 OPINSIGHTS_WORKSPACE_AZURE_CLOUD_TYPE=0 OPINSIGHTS_WORKSPACE_ID=<your workspace id> OPINSIGHTS_WORKSPACE_KEY=<your workspace key> AcceptEndUserLicenseAgreement=1

 

 

 

Other helpful links

Docs site https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-quick-collect-windows-computer

Daniel Orneling Blog https://blog.orneling.se/2017/01/installing-oms-agent-with-powershell/

TechNet gallery https://gallery.technet.microsoft.com/scriptcenter/Install-OMS-Agent-with-2c9c99ab

Service Map for SCOM

 

Ever compare your work to an amusement park?

Every business application compares to a ride, roller coaster, or even a kiddie ride.

Anyone ever ask you directions to that ride, or more technical based questions like ‘what’ communication makes up that business application?

 

 

In comes Service Map to save the day!

 

 

Last year I blogged about setting up Service Map with OMS/Log Analytics, but I didn’t get the feature installed for SCOM.

December blog on how to set up OMS/Log Analytics

 

It’s basically the SCOM Agent (MMA) and a Dependency Agent (think old Blue Stripe agent)

 

Excited to see the new Service Map to hit public preview, hoping by September

 

 

Check out the blog series

Planning and PreReqs blog
Install and configure MMA agent blog
Dependency agent blog

Set up Azure Service Principal blog
Set up SCOM Management Group blog

 

 

Installing and configuring the MMA agent

 

Maybe the MMA agent is like Venom?
Proof I’ve watched too many a Marvel movie…

 

An existential moment perhaps, but the MMA agent can be a bunch of strings stuck from one place to another, monitoring whatever its told to do.

 

 

 

If you are running SCOM2016 or above, the MMA agent is built-in with Log Analytics, just configure your workspace

 

 

 

 

Download and Install MMA agent

SCOM 2012R2 agent does not have MMA, so download MMA agent from Log Analytics workspace

Azure Portal > Log Analytics > Subscription > Advanced Settings

Click on Windows Servers from Connected Sources to download Windows Agent

Click on Linux Servers from Connected Sources to download Linux Agent

 

 

From the Azure Portal (https://ms.portal.azure.com)

Click on Log Analytics, <your subscription >

Click on Advanced Settings

My view defaulted to Connected Sources > Windows Servers

 

Save the workspace ID and workspace key to notepad/OneNote for later

 

 

 

< Assuming the MMA agent is installed with Log Analytics capability >

 

 

Update MMA Agent with Workspace ID and Key

From MMA agent, update the OMS Workspace with the GUID copied to notepad

 

Click on Start > Control Panel, System and Security > Microsoft Monitoring Agent

Click on Azure Log Analytics (OMS) tab on MMA agent

Click Add

 

Add Workspace ID and Key to agent

Click OK

Click OK again on MMA properties

 

Look for the healthy green checkbox’d circle

 

Troubleshooting Errors in the Operations Manager Event Logs

Blog posts – Verify, 55002

 

 

 

 

 

 

 

 

What ID’s is SCOM using

Ever need to audit what ID’s SCOM is using?

Maybe you have to figure out how someone else setup SCOM.

Did they set up SCOM as recommended for best practices with different AD accounts per role?

 

If the ID’s are not logged during install, it’s a little more difficult to figure out what ID was used.

  • Domain Account for ALL services,
  • Enter in the unique DOMAIN\OMAA, DOMAIN\OMDAS, DOMAIN\OMREAD, DOMAIN\OMWRITE

 

Try these PowerShell commands to find what SCOM is using.

 

ON MS (from PowerShell (don’t need admin unless you’re restarting services)

$Services = ( Get-WmiObject -Class Win32_Service )

$Services | ? { $_.Name -eq “OMSDK” -OR $_.Name -eq “cshost” -OR $_.Name -eq “HealthService” } |

ft name,Startname,StartMode

 

 

 

ON SCOM DB’s, Reporting (from PowerShell (don’t need admin unless you’re restarting services)

$Services = ( Get-WmiObject -Class Win32_Service )

$Services | ? { $_.DisplayName -like “*SQL*” } | ft name,Startname,StartMode

 

 

Source https://blogs.technet.microsoft.com/heyscriptingguy/2012/02/15/the-scripting-wife-uses-powershell-to-find-service-accounts/

 

 

Active Directory 2012-2016 Addendum packs updated

Man time flies!

 

Thought I’d share some new functionality for AD DS (Active Directory Domain Services)

 

Ran across some customer errors with AD Event ID 1084, which exists in the old 8321 pack, but not in the v10.x pack.

Well, if you get these errors, your DC isn’t replicating, and most likely will need to be rebuilt.

 

Gallery download

 

Broke out the packs to separate the Recovery Tasks in their own pack, versus added functionality in the addendum.

Figured better to send packs NOT sealed, so that meant 2 packs,

WYSIWYG (wizzy-wig acronym)

 

What this means

v1.0.0.1 pack had just the AD DS Service Recovery Tasks

v1.0.0.2 pack has a Service Recovery Tasks pack, and the Addendum pack

What I think is cool is that the Addendum pack contains 2 rules, simple rule event (enabled by default), and also a PowerShell rule.

 

Rule Figured out how to simply look for criteria, count it, and alert on it.

We always look for alert suppression, some of the sliding/counting monitors are too much.

 

Starting with Holman’s alerting rule fragment, we can create more powerful combinations than just a single symptom.

Using Variations of the get-date command, we can actually specify how far back to look, to count for alerts.

Easier method to count events, to figure out an alert threshold.

 

From the rule in the Addendum pack

# Check blog for more detail https://blogs.technet.microsoft.com/heyscriptingguy/2015/01/21/adding-and-subtracting-dates-with-powershell/
# If you want this in other time increments – AddHours, AddSeconds, AddMilliseconds
#
$LastCheck = (Get-Date).AddMinutes(-65)

[int]$TempCount = (get-eventlog -logName “Directory Service” -Source “NTDS Replication” -InstanceID 1084 -Message “*8451 The replication operation encountered a database error*” -After $LastCheck).Count

IF ($TempCount -ge 1)
{
$Result = “BAD”
$Message = “The number of 1084 Replication Database error events was greater than 1”
}
ELSE
{
$Result = “GOOD”
}

 

Maybe we need multiple event ID’s, or search multiple event logs… you decide, and let me know.

 

SQL MP bloat

Updated 25 Feb 2023

 

Ever wish alerts were like a wad of cash?

The more you solve, the more you make!

 

How about performance counter data?

 

 

The SQL management packs are awesome for visualizations, and provide a bunch of data.

 

Tim McFadden pointed out SQL Performance counters https://www.scom2k7.com/crazy-db-performance-collection-rules-in-the-sql-mps/

His blog brings up SQL MP Disk Latency performance counters.

 

His blog got me thinking about SQL DB and DB file design, where multiple DB files are on the same Drive, causes duplicate performance counters (SCOM workflows) on the agent, and will typically be one of the culprits for HealthService restarts.

 

SQL MP creates performance counters (per DB file, group, instance, engine)

 

Let’s start with how I figured out why all my money goes into storage.

 

Start in the SCOM console

Click on the Reporting Tab

Click on the ‘System Center Core Monitoring Reports’ folder

Double click on the Data Volume by Management Pack

View of SCOM report from console reporting tab

Select the timeframe (from, to)

Click Run

Data Volume MP selected

 

Reporting Data

I have 2 2016 DB’s and 1 2014 (SCVMM) database server monitored, and it’s 50% of my data volume!

 

 

 

Another example – had the DW shutdown for days

Data volume of SQL after

 

Did you know there are 60+ perf counter rules in 2012 alone, and nearly 200 in 2016?

 

How about an OFF pack, a management pack that turns off all the performance counter rules?

The monitors still exist for health, just no pretty performance graph, should you look.

 

 

Github repo link

Check out the Gallery post for download

TechNet gallery download

 

Zip file contains

  1.  OFF MP’s for 2008,2012,2014,2016
  2. XLS sheets to allow you to go to the SQL team and ask them what performance counters they use