SCHANNEL event logging

First, my thanks to Bhuvnesh Kumar for his help!

 

Time to figure out what’s going on behind the curtain!

 

 

Are you seeing System Event Log, Event ID 36871 events?

 

Why does this matter?

 

Depending on OS versions and patches, the TLS Cipher Suites may not match on the various SCOM servers.

  1. If you’re setting up TLS1.2, you need the SCOM servers to talk
  2. The bad part, is this isn’t logged much on the GW but log more often on MS
  3. Sometimes the 36871 events come with 36874, but in my experience they occur after Event Logging is enabled.

 

The unanswered question is “why are we seeing the 36871 events?”

 

In my example, the events only happened once a day, roughly 24 hours

 

Event Viewer

 

Are events related to the Cipher Suite, or is it a MP trying to run the old SQLOLEDB method?

 

This article will focus on verifying Cipher Suite on a server

See this article for MP analysis for SQL methods

 

 

 

SCHANNEL event logging setup

 

From Holman’s blog

DecimalDescription
0Do not log
1Log Error messages
2Log Warnings
3Log Error and Warning messages
4Log Informational and Success events
5Log Error, Informational and Success events
6Log Warnings, Informational and Success events
7Log Everything (Warnings, Errors, Informational and Success events

 

I’d recommend setting it to 3 to see errors and warnings, or 7 to see everything.

Remember to set this back to 1 when done resolving any issues.

 

Add

From Command Prompt or PowerShell (as administrator)

reg add “HKLM\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL” /v “EventLogging” /t REG_DWORD /d 7 /f

Disable

reg delete “HKLM\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL” /v “EventLogging”

Verification

reg query “HKLM\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL” /v “EventLogging”

 

PowerShell verification

RegEdit Verification

Time to reboot!

 

 

Verify SCHANNEL events

Look at the System Event log, and filter for 36880 and 36874 events for clues

 

36880 provides Cipher Suite details

 

Event ID 36874 definitely describes the scenario

 

The easy answer to solve the cipher suite is to ask – is this server patched with latest security and .NET patches?

After all this, in my example, we confirmed that simple step was assumed, and inaccurate.

 

 

 

 

References
36871 event https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-R2-and-2012/dn786445(v=ws.11)

SCHANNEL events https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-R2-and-2012/dn786445(v=ws.11)

SChannel error codes https://docs.microsoft.com/en-us/windows/win32/secauthn/schannel-error-codes-for-tls-and-ssl-alerts

SChannel events https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-R2-and-2012/dn786445(v=ws.11)

SSL errors https://www.experts-exchange.com/questions/28996780/event-id-36871-Schannel.html

Troubleshooting https://docs.microsoft.com/en-us/iis/troubleshoot/security-issues/troubleshooting-ssl-related-issues-server-certificate

Setting up OMS Capacity and Performance

Setting up OMS Capacity and Performance
Setting up OMS Capacity and Performance

 

Update 18 Dec 2023 – Solution retired in 2021 with OMS sunset.  

https://github.com/uglide/azure-content/blob/master/articles/log-analytics/log-analytics-add-solutions.md Repository archived by the owner on Feb 1, 2021. It is now read-only.

 

 

Do you know what your HyperV hosts are doing?

Not a HyperV fan, there’s a VMWare solution also here

 

Documentation https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-capacity

https://github.com/uglide/azure-content/blob/master/articles/log-analytics/log-analytics-capacity.md

 

Capacity dashboard

Capacity and performance preview summary
Capacity and performance preview summary

Details

OMS dashboard
OMS dashboard

 

 

Setting up OMS Capacity and Performance

Already have the dashboard setup?  Perhaps this will help troubleshoot

Do you have network connectivity, or is a proxy required?

 

Troubleshooting dashboard

Firewall https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-proxy-firewall
Windows Agents https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-windows-agents

 

Verify Operations Manager event log on local agent, then filter for error events and/or EventID 4506.  Look for dates/times to see when events started.

Example Event ID 4506 details the Capacity and Performance Solution, citing ‘Microsoft.IntelligencePacks.CapacityPerformance.Collector’.

Operations Manager Event Log, Event ID 4506 examples
Operations Manager Event Log, Event ID 4506 examples

 

Additional options

  1. Search LAW (Log Analytics workspace) logs

https://github.com/uglide/azure-content/blob/master/articles/log-analytics/log-analytics-log-searches.md

OMS Log search screenshot

 

2. Verify no proxy is set up (unless your network requires this)

OMSAgent proxy setting
OMSAgent proxy setting

 

3. 4506’s result from too many workflows sending data from MS to DB’s (OpsMgr and DW).  Additionally, 4506 events can be communication issues from MS to DB server(s).   Lastly, use TLS1.2 configuration as a best practice to enforce encryption from MS to SQL communication.  Beyond encryption, TLS may be a culprit if AlwaysOn or SQL clusters are involved, particularly as the SCOM console connections fail as SDK cannot talk with SQL side.  See Kevin Holman’s blog for additional TLS1.2 information and setup.

TLS blog https://kevinholman.com/2018/05/06/implementing-tls-1-2-enforcement-with-scom/

 

Documentation

Learn article https://learn.microsoft.com/en-us/answers/questions/212007/scom-errors-no-data-in-summary-performance-dashboa
TechNet blog https://social.technet.microsoft.com/Forums/ie/en-US/10b38121-b0e1-43ec-bf3a-d22ae9ef0220/event-4506-data-was-dropped-due-to-too-much-outstanding-data-in-rule
MS RMSe https://www.system-center.me/opsmgr/event-4506-and-new-root-management-server-rms-management-server-ms/

Setting up OMS Service Map solution

hmmmm

Ever wonder what happened to BlueStripe?

Anyone else have experience using it with SCOM?

If you weren’t aware, Microsoft bought Blue Stripe back in 2015 link

 

Looks like BlueStripe FactFinder is now Service Map in Azure

Documentation here

 

Service Map is very easy to add and get value from right away with OMS

Download agent

You have two choices:

  1. Choose from Docs.Microsoft.com documentation above, or from your OMS environmentdocsagentdownload
  2. From your OMS workspace, add the Service Map solution

Click on Home icon in top left hand corner

omshome

Click on Service Map pane

Click on Download Agent link as appropriate for Windows or Linux

Save file and install on your server(s)

oms-initialscreen

 

Windows Server Installation

Execute the MSI file downloaded from OMS (NOTE may prompt with UAC prompt)

Click ‘I Agree’

servicemapinstall

Watch the Install

servicemapinstalling

Click Finish

servicemapinstallcomplete

Now go back to OMS and look for updates (mine was that fast!)

servicemapsolution

Click on the Service Map pane to see more detail

servicemapdetail

To add additional machines is basically the same, just choose add machines

oms-addmachines

 

In case you caught that I have two (2) of the same named machines, it’s because I have that server set up for OMS separately.  Yes, it’s my lab, so I’m not following the best practice.

servicemapsolutionwclients

Enjoy!

Building a subscription

subscribe-and-save

 

Let’s talk notifications for a minute.

Everyone complains that a tool is noisy for alerts (typically emails).

Why not find a way to limit what you receive, and eliminate, the noise.

Sure, there’s alert tuning, but there are a ton of built-in options with Subscriptions in SCOM.

 

howto

Let’s Start by talking about now a subscription is built in SCOM.

Step 1 – An owner (or ‘subscriber’) is needed

This can be an email address, group name or variable you may want to pass to a command line for a destination (e.g. support team/NOC/POC)

Step 2 – A channel is needed (simply put, a way to get the data out of SCOM)

This can be SMTP (email), or a custom executable to a ticketing system, NetCool, BMC True Sight, xMatters, Derdack, to name a few.

Step 3 – Criteria to send to an owner (details)

Time to set up a subscription, and learn as we go!

Do you have the necessary 3 parts (subscriber, channel)

Do you have a destination/subscriber already set up?

Yes, see go to Channel

No, follow the subscriber blog here

Do you have a channel set up

Yes, see go to Subscription

No, follow the Channel blog here

 

Do you have a naming convention for the subscription parts?

The Subscription name needs to be intuitive, i.e. Application Name, Team Name, Company Name (depending on the environment)

Process an Application’s alerts

Example     ‘BizTalk alerts’

If BizTalk alerts needed to go to different teams

‘BizTalk DEV Alerts’ or ‘BizTalk PROD Alerts’

or if Criteria is involved ‘BizTalk Performance Alerts’

or if alerts need to route to another company ‘Contoso BizTalk alerts’

Capitalize what needs emphasis so in the Subscriptions view (make searches or sorts easier and more intuitive)

Making sense where I’m going with this?

 

Criteria can influence the name

CLASS, MONITOR, RULE, SEVERITY, GROUP, RESOLUTION STATE

To me the value comes in with the Description field in a subscription.

Adding relevant detail here makes life easier when followed, to know what the subscription is doing.

Try this model for the Subscription Description

CRITERIA

SUBSCRIBER

COMMENTS

Example

+MONITOR = Health Service Heartbeat Failure +SEVERITY = Warning/Critical +RESOLUTION STATE NOT equals 255 +SUBSCRIBERS = GROUP Server Admins via Email +Comments: Created 2016-02-12 for SCOM Agent tuning

 

Time to set up a subscription

Subscription Summary Healthservice Watcher subscription to alert on any NEW Healthservice Heartbeat failures

Name     SCOM HealthService Watcher

Description

+MONITOR = Health Service Heartbeat Failure +SEVERITY = Warning/Critical +RESOLUTION STATE NOT equals 255 +SUBSCRIBERS = GROUP Server Admins via Email +Comments: Created 2016-02-12 for SCOM Agent tuning

 

Criteria

Notify on all alerts where

created by Health Service Heartbeat Failure rules or monitors (e.g., sources)

and of a Warning or Critical severity

and with Not Equals 255 resolution state

 

Subscribers

GROUP Server Admins via eMail

 

Channels

SMTP Channel

Basic Admin ‘How-to’ Series

443053-royalty-free-rf-clip-art-illustration-of-a-cartoon-businessman-carrying-a-heavy-manual

This is a series of blog posts to help with SCOM best practices, and things that make SCOM easier to administer.

 

Associate MPX files in Notepad++ blog

Backup management packs via PowerShell blog

Get to know your monitor blog

Load Test MP with Report blog

Load Test MP Fragments blog

Maintenance Mode PowerShell blog

Manage DB storage with DWdataRP blog

Managing Subscriptions blog

PowerShell Rule/Monitor/PerfCounter MP and Fragments blog

Registry Key discovery MP Fragment clarification blog

Run As PowerShell monitor fragment blog

Sealing Management packs with 2012R2 and 2016 blog

Subscriptions blog

Subscription Set up Guide blog

Uncommon MP Fragments blog

Verifying Overrides blog

 

Best Practices

Agent Management pack KH Blog

Enable proxy as a default KH blog

How to be heard blog

Manage alerts/events/performance KH Blog

Office Analytics (find where all the time goes) blog

Optimize SQL blog

Recommended Registry tweaks KH blog

SCOM Agent Version Addendum KH blog

Set SCOM Agent to remotely managed KH Blog

SQL Engineering Blog

SYSTEM CENTER 2016 Operations Manager – Anti-Virus Exclusions blog

Update VMM MP’s for SCOM when SCVMM patched blog

 

Tools

MP Viewer blog

Download Notepad++ here

Kevin Holman blog on extracting scripts from MP’s using Transform tool from codeplex

Test fire events using EventLog Explorer here

Alternate tool to fire any events here