AD Application monitoring

Data from StarTrek the next generation - Mr. Tricorder makes me laugh!
Data from StarTrek the next generation – Mr. Tricorder makes me laugh!

‘AD Application monitoring’ > web synthetics, artificial users > android what image comes to mind?  Is it a person, or a thing from a Sci-Fi movie? Perhaps Bishop from Aliens, Data from Star Trek.  What does ‘AD Application monitoring’ consist of?  Currently that means a CRL validity check, and ADFS web synthetic (proving that ADFS is responding).  My thanks to Jason Windisch CSA, for the supplied PowerShell!

 

Quick Download https://github.com/theKevinJustin/ADApplications/

Tailoring the pack to your environment

The purpose of the pack is to add scheduled workflow that acts like the user, identifies if the CRL’s are about to expire.  Most times, monitoring stops at ICMP ping.  Most times, there’s still an outage, as the network, and servers are responding.  The next layer is IIS, Apache, etc.  Sometimes the network team gets involved, checking a base IIS URL is configured.  Most outages aren’t network, nor IIS wasn’t running.  This is why we focus on the web application responding.  Does the multi-prong tactical attack make sense?

This pack delivers on-demand tasks, daily reports, and rules/monitors to reflect health.  Customize the watcher node, some URL’s, save, and import into SCOM!  The purpose

 

Assign watcher node(s)

Assign a watcher node by creating a registry key.

What does that mean?   Watcher nodes are needed to provide user perspective.

 

Multiple site example

Issue:  Users from sites 1,2,3 are having problems accessing web pages.  To understand a user in site 2, leverage a server in site 2 to initiate the web request (invoke-webRequest in PowerShell).

Why:  Differentiate user experience (per site).  Answer the ‘did you know’ – is the application responding from this site/perspective.

Unfortunately, the watcher node concept eludes most administrators.  Mastering ‘user perspective’ makes for an invaluable aid moving from reactive ‘fire fighting’ to proactively being told before users.   Hopefully this explains the power where monitoring imitates user interactions for key web applications.

How:  Create registry key on whatever servers you want to initiate web monitor

From PowerShell (as Admin), or Command Prompt (as admin)

reg add “HKLM\SOFTWARE\ADApplications\WatcherNode”

 

 

AD Applications regedit registry key validation
AD Applications regedit registry key validation

 

Example of XML snippet from AD Applications management pack

AD Applications Watcher Node - create specific registry key
AD Applications Watcher Node – create specific registry key

 

 

Set up CRL Validity check and ADFS synthetic

Next, configure the URL’s for the customer environment for the ‘AD Application monitoring’ management pack.

Update AD Applications module types for monitor/rules for CRL and ADFS synthetics

Update AD Applications module types for monitor/rules for CRL and ADFS synthetics

Configure the CRL validity check array

From your favorite XML editor (notepad++ pictured)

Find/Replace ##FQDN##, ##CRLstring##, numbers to customer environment

CRL Validity check, create your array length as needed for customer environment
CRL Validity check, create your array length as needed for customer environment

 

Configure the ADFS synthetic request(s)

From your favorite XML editor (notepad++ pictured)

Find/Replace $server, ##FederationFQDN##, if necessary, update ADFS URL string if different (the /adfs/ls/idpiniatedsignon.aspx portion) to customer environment

Update ADFS URL for invoke-webRequest, ADFS default URL in specified example
Update ADFS URL for invoke-webRequest, ADFS default URL in specified example

Save pack

Import and enjoy!

 

Documentation

URLGenie for advanced website monitoring

PowerShell invoke-webRequest

Addendum logic blog

SCOM Web Console Authentication settings

Authentication Authentication Authentication! SCOM Web Console authentication settings
Authentication Authentication Authentication! SCOM Web Console authentication settings

 

SCOM Web Console authentication settings discussion!  Let’s go through standard IIS authentication settings like disabling Anonymous Authentication, and enabling Windows Authentication, AD Client Certificate Authentication, and binding providers (Negotiate before NTLM).  Ready to begin?!  A shout out to Alden Hatten as we worked through this and resetting the Web Console run here recently, that brought up the urgency to document.

 

 

Vendor documentation

Learn.Microsoft.Com link, SCOM2019 link

SCOM TechCommunity post for context

Kevin Holman’s SCOM QuickStart guides for SCOM 2019, 2022 (Including WebConsole default setup steps)

 

 

SCOM Web Console Authentication settings defaults

RDP to server with SA or Local admin level account

Go into IISManager > Expand the tree to then click on ‘Default Web Site’

Click on Authentication

IIS Manager output for ‘Default Web Site’

IISManager Default Authentication settings
IISManager Default Authentication settings

 

 

SmartCard aka AD Client Certificate Authentication defaults

In IIS Manager for the server > Click on Authentication

Verify AD Client Certificate Authentication is added and enabled.

IIS Manager Authentication, with SmartCard or Client Certificate Authentication
IIS Manager Authentication, with SmartCard or Client Certificate Authentication

 

 

Windows Authentication

Set Authentication Providers order

From IIS Manager > Expand Default Web Site

Click on Authentication > Click on Providers at the top right

If Negotiate is not on top, highlight, and click Move Up button > Click OK to set.   Restart IIS to make setting take effect ( also use iisreset from command prompt or PowerShell )

NOTE: Anonymous Authentication should be disabled!

IIS Manager Authentication, Windows Authentication, Providers, Negotiate on top
IIS Manager Authentication, Windows Authentication, Providers, Negotiate on top

If screenshot is your setup, close the Providers window

 

After reviewing these authentication settings, you should be one step closer to encrypted authentication.

Enjoy!

Check your delegation settings

 

Sometimes as the monitoring admin, you may be responsible to secure your servers, being told from Security/Cyber Teams about new vulnerabilities.  The vulnerabilities may be from Tanium, ACAS, Tenable or other security tools.   This article is how to help you secure related SCOM web console, and SSRS reporting sites against Unconstrained Delegation vulnerabilities CVE-2020-17049, also AD STIG V-36435.

 

First we need to identify IF this is a true finding.

Typically this comes from Server/SystemsAdmin with domain admin access:

From PowerShell run:

Get-ADComputer -LDAPFilter
“(userAccountControl:1.2.840.113556.1.4.803:=524288)”

After identifying SCOM servers with unconstrained delegation (scope of blog post is focused on SCOM in a hybrid scenario (prem/cloud), we need to resolve.

With domain administrator rights, open the Active Directory Users and Computers MMC snap-in.

In ADUC, change Find drop-down to Computers
In the Computer name text box, enter <SCOMServer>  and click search

Right click the server in the results box > Select Properties.

Select the Delegation tab.

ADUC view of lab server delegation setting

 

Select Trust this computer for delegation to specified services only > Use any authentication protocol.

Under Services to which this account can present delegated credentials, select Add.

In the new dialog box, select Users or Computers.

Enter <SCOMServer>, and then select OK.

Click the Add button to add services

Select the w3svc and www processes

Select OK.

ADUC GUI adding services for delegation on SCOM server

Once set in AD, reboot server.  Running ‘gpupdate /force’ may not apply AD changes to the server object.

After reboot, reach out to SCOM Admins to test webconsole authentication

From edge browser, go to SCOM web console, typically @ https://<SCOMServer>/OperationsManager

On the Monitoring tab, click on Active Directory dashboard on left

Verify authentication works

 

Documentation

Pentestlab – Detecting Unconstrained Delegation Exposures in AD Environment

Petri.com find and block unconstrained delegation

Learn.Microsoft.com unconstrained kerberos article

Explanatory documents on what/why

Remove Unconstrained Kerberos Delegation

 

OMS/Advisor Event ID 55002

 

This article is written for the Gateway CommunicationSecurityException event

At first I thought maybe this was TLS1.2 enabling, but backed off the change, the events kept pouring in every 5 minutes.

Tried to reconfigure the OMS/Advisor environment, and voila! Error resolved

 

Let’s go through the steps to re-configure the Operations Management Suite (OMS) in SCOM

 

Reconfigure OMS

  1. From the SCOM Console, click on Administration tab
  2. Expand Operations Management Suite (Advisor on 2012R2)
  3. Click on the Connection
  4. On the center pane, click on Re-configure Operations Management Suite

 

5. Add any trusted sites to IE if there are pop-ups

I had 2 missing websites

Secure.aadcdn.microsoftonline-p.com

az416426.vo.msecnd.net

( I hit Previous and next to verify the wizard would pass with the hopes the attempt would retry)

6. Exit the Reconfigure wizard to get a retry (then the second website popped up as an untrusted site)

7. Enter credentials to your OMS environment

 

Connection to OMS successful

 

Click Next twice

Reconfigure success

Click Close

 

Verify Event Log

Verify Operations Manager Event Log has no new events (this check runs every 5 minutes by default)

get-eventlog -logname “Operations Manager” | ? { $_.EventID -match 55002 } | select-object -last 2

 

 

Event ID 55002 from Operations Manager Event Log

Log Name:      Operations Manager
Source:        Advisor
Date:          12/11/2017 2:15:20 PM
Event ID:      55002
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      16MS01.testlab.net
Description:
Failed to synchronize the latest Management Package information from Advisor Cloud service. Wait for the next cycle to retry. Reason: Microsoft.SystemCenter.Advisor.Common.WebService.GatewayCommunicationSecurityException: Message security was invalid for the connection with web service when performing Get Intelligence Packs with client specified versions —> System.ServiceModel.Security.MessageSecurityException: An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail. —> System.ServiceModel.FaultException: ID3242: The security token could not be authenticated or authorized.
— End of inner exception stack trace —

Server stack trace:
at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.ProcessReply(Message reply, SecurityProtocolCorrelationState correlationState, TimeSpan timeout)
at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Request(Message message, TimeSpan timeout)

Exception rethrown at [0]:
at System.ServiceModel.Security.IssuanceTokenProviderBase`1.DoNegotiation(TimeSpan timeout)
at System.ServiceModel.Security.IssuanceTokenProviderBase`1.GetTokenCore(TimeSpan timeout)
at System.IdentityModel.Selectors.SecurityTokenProvider.GetToken(TimeSpan timeout)
at System.ServiceModel.Security.Tokens.IssuedSecurityTokenProvider.GetTokenCore(TimeSpan timeout)
at System.IdentityModel.Selectors.SecurityTokenProvider.GetToken(TimeSpan timeout)
at System.ServiceModel.Security.SecurityProtocol.TryGetSupportingTokens(SecurityProtocolFactory factory, EndpointAddress target, Uri via, Message message, TimeSpan timeout, Boolean isBlockingCall, IList`1& supportingTokens)
at System.ServiceModel.Security.TransportSecurityProtocol.SecureOutgoingMessageAtInitiator(Message& message, String actor, TimeSpan timeout)
at System.ServiceModel.Security.TransportSecurityProtocol.SecureOutgoingMessage(Message& message, TimeSpan timeout)
at System.ServiceModel.Security.SecurityProtocol.SecureOutgoingMessage(Message& message, TimeSpan timeout, SecurityProtocolCorrelationState correlationState)
at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [1]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Microsoft.AttachedServices.WebService.IIntelligenceService.GetIntelligencePacksInfo(ClientProperties clientProperties)
at Microsoft.SystemCenter.Advisor.Core.WebService.WebServiceCallHelper.CallWebService[T](Func`1 webServiceCall, String webServiceDescription)
— End of inner exception stack trace —
at Microsoft.SystemCenter.Advisor.Core.WebService.IntelligenceServiceClient.CallWebServiceWithRetry[T](Func`2 function)
at Microsoft.SystemCenter.Advisor.Core.WebService.IntelligenceServiceClient.GetIntelligencePacksInfo(ClientProperties clientProperties)
at Microsoft.SystemCenter.Advisor.Core.IntelligencePackWriteAction.UpdateIntelligencePacks()

Why not to use Local System for your core SCOM accounts

say-what-logo1

Stay with me here, this is for the SCOM management group installation

 

So first, let’s research and figure out what the experts are doing, and what the install guides exist.

Researching expert published documentation helps us understand the options, and we can dive into some of the reasons why.

 

SCOM Security

scom-kh-securityblogcapture

(KH blog )

 

 

SQLRights and roles

scom-sql_accountrightsmapping

(KH blog here to download the XLS (applies for 2012,2016 as well)

 

Experts separate out the various functions into dedicated ID’s

 

The reason for multiple ID’s is to lower the risk (less vulnerability if one ID is locked out, expired, disabled)

— If you use one ID for all SCOM functions, and something happens to the ID, your SCOM environment stops working.

— There’s always some associated risk with either scenario for LocalSystem or ID’s (decrypt RunAs ID’s UK blog )

If that is a concern, here is some great advice from Kevin Holman

  1. Control who has access to SCOM
  2. Control who has access to the servers using RunAs accounts, that are monitored by SCOM.

If you have lost control of local admin on a server, you are compromised, and I am not sure how gaining access to a RunAs account is no worse in some sense.

By the way – this is the entire reason “more secure” was introduced in SCOM 2007R2, to limit distribution of credentials only to servers that required it, to limit the potential for a local attack.

 

Another option (not recommended) is using Local System

— Cannot login to system to verify access concerns (quite honestly is why someone might sanction this approach)

— Scripts run as local system can be terminated, allowing a command window with Local System access

— Depending on which services LocalSystem is used, this could grant elevated privileges (like a Domain Controller DC)

localsystemaccount

— If ‘Local System’ was used for the core SCOM environment, a change made to the Local Security Policy, or group policy can break the environment.

Local Security Policy snapshot

localsystem-localsecuritypolicy

Security Options

localsystem-securityoptions-localsecuritypolicy

Group Policy

Locking down protocol blog here

gpo-snapshot-technetwebsite

 

 

 

Hope this helps you decide the ‘how to’ set up your environment

 

 

Related documentation

2019 Kevin Holman Deployment Guide

2016 Kevin Holman Deployment Guide

2012 R2 Kevin Holman Deployment Guide

Planning 2019 SCOM deployment Guide

Planning 2016 SCOM deployment Guide

2012 Technet Deployment Guide

2007R2 Technet Deployment Guide