Month: November 2017

Ruling out SCOM as the cause of SCHANNEL events

 

Ruling out SCOM notifications as the cause of SCHANNEL events

 

 

Still getting SCHANNEL error events and want to rule out SCOM

Management pack SQL events https://kevinjustin.com/blog/2017/11/08/sql-native-client-for-tls1-2/

SCHANNEL ciphers debugged https://kevinjustin.com/blog/2017/11/08/schannel-event-logging/

 

What command Channels are setup for notifications?

 

 

Validate Subscriptions aren’t the cause for email/text

Exchange 2013 and above typically use S/MIME to digitally sign/encrypt messages

 

Email communication can cause System 36871 events https://support.microsoft.com/en-us/help/305088/schannel-error-message-36871-when-receiving-an-ehlo-smtp-command

Do the events correlate with emailed alerts?
Tracing Notifications http://blog.scomskills.com/enable-tracing-of-the-notification-component-om07/

 

SCOM ETL traces

Run traces on suspect MS

2012R2 MS (adjust drive letter according to drive SCOM install)
cd “D:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\Tools”
2012R2 GW (adjust drive letter according to drive SCOM install)
cd “C:\Program Files\System Center Operations Manager\Gateway\Tools”
2016 MS
cd ‘C:\Program Files\Microsoft System Center 2016\Operations Manager\Server\Tools\’

# Stop Tracing
 .\StopTracing.cmd
# Clean up old files
remove-item C:\windows\Logs\OpsMgrTrace\*

 

# Start Traces

StartTracing.cmd VER

TraceLogSM.exe -stop TracingGuidsNative

TraceLogSM.exe -stop TracingGuidsUI

 

# Wait until notification fires and validate if 36871 SCHANNEL event ID is logged

# Stop and format the trace
 .\StopTracing.cmd
 .\FormatTracing.cmd

# Review txt files from C:\windows\Logs\OpsMgrTrace

 

 

SCHANNEL event logging

First, my thanks to Bhuvnesh Kumar for his help!

 

Time to figure out what’s going on behind the curtain!

 

 

Are you seeing System Event Log, Event ID 36871 events?

 

Why does this matter?

 

Depending on OS versions and patches, the TLS Cipher Suites may not match on the various SCOM servers.

  1. If you’re setting up TLS1.2, you need the SCOM servers to talk
  2. The bad part, is this isn’t logged much on the GW but log more often on MS
  3. Sometimes the 36871 events come with 36874, but in my experience they occur after Event Logging is enabled.

 

The unanswered question is “why are we seeing the 36871 events?”

 

In my example, the events only happened once a day, roughly 24 hours

 

Event Viewer

 

Are events related to the Cipher Suite, or is it a MP trying to run the old SQLOLEDB method?

 

This article will focus on verifying Cipher Suite on a server

See this article for MP analysis for SQL methods

 

 

 

SCHANNEL event logging setup

 

From Holman’s blog

DecimalDescription
0Do not log
1Log Error messages
2Log Warnings
3Log Error and Warning messages
4Log Informational and Success events
5Log Error, Informational and Success events
6Log Warnings, Informational and Success events
7Log Everything (Warnings, Errors, Informational and Success events

 

I’d recommend setting it to 3 to see errors and warnings, or 7 to see everything.

Remember to set this back to 1 when done resolving any issues.

 

Add

From Command Prompt or PowerShell (as administrator)

reg add “HKLM\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL” /v “EventLogging” /t REG_DWORD /d 7 /f

Disable

reg delete “HKLM\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL” /v “EventLogging”

Verification

reg query “HKLM\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL” /v “EventLogging”

 

PowerShell verification

RegEdit Verification

Time to reboot!

 

 

Verify SCHANNEL events

Look at the System Event log, and filter for 36880 and 36874 events for clues

 

36880 provides Cipher Suite details

 

Event ID 36874 definitely describes the scenario

 

The easy answer to solve the cipher suite is to ask – is this server patched with latest security and .NET patches?

After all this, in my example, we confirmed that simple step was assumed, and inaccurate.

 

 

 

 

References
36871 event https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-R2-and-2012/dn786445(v=ws.11)

SCHANNEL events https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-R2-and-2012/dn786445(v=ws.11)

SChannel error codes https://docs.microsoft.com/en-us/windows/win32/secauthn/schannel-error-codes-for-tls-and-ssl-alerts

SChannel events https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-R2-and-2012/dn786445(v=ws.11)

SSL errors https://www.experts-exchange.com/questions/28996780/event-id-36871-Schannel.html

Troubleshooting https://docs.microsoft.com/en-us/iis/troubleshoot/security-issues/troubleshooting-ssl-related-issues-server-certificate

SQL native client for TLS1.2

Ever try to talk to someone when language is a barrier?

 

Sure, we can run an app, or search our phrase to pronounce, but it’s so much better when we can communicate seamlessly.

 

Post TLS1.2 for SCOM

Let’s talk SQL

Part of TLS1.2 is updating SQL Native Client to talk using a secure client that uses TLS1.2

That means a different executable should be called.

 

Why is that important in SCOM?

Maybe you have management packs that connect to SQL or run external commands.

 

 

On MS, there are multiple clues for various errors on Management Packs that use SSL or talk to SQL via a non-TLS method.  NOTE this may mean that the SQL DB that management pack is connecting to may need the same pre-req SQL updates to a TLS 1.2 enabled version.

  1. Do you have custom SQL queries being run, CMDB get’s, OLE DB Data Source checks?
  2. Any Event ID 1401 or 11854 events in the Operations Manager Event log?
    1. These events identify management pack scripts creating SCHANNEL events
      a. Event ID 1401 event example

 

 

Cause

SQLOLEDB connection strings will cause 36871 Sytem Log events

 

Example (TLS1.0)
sConnectString = “PROVIDER=SQLOLEDB;DATA SOURCE=<databaseServerFQDN>;DATABASE=MSSQLSERVER;trusted_connection=yes”
 SQLNCLI11 driver for TLS1.2 connection strings

Example (TLS1.2)
 sConnectString = “Provider=SQLNCLI11;DATA SOURCE=<databaseServerFQDN>;DATABASE=MSSQLSERVER;trusted_connection=yes”
 

 

Identify
Look for management packs with SQLOLEDB as the Connect string to reduce 36871 SCHANNEL events

In Windows Explorer, use the Advanced Options dropdown to select File Contents
In the Search bar (top right), enter SQLOLEDB (example shows SQLNCLI11)
NOTE SQL Discovery group pack IS compliant

 

 

In Windows Explorer, use the Advanced Options dropdown to select File Contents
In the Search bar (top right), enter SQLNCLI11

 

 

Additional offenders
HP Topology MP
SQL 2005 discovery MP (discontinued)
SQL Addendum MP’s (will work to update these with Holman)
SharePoint Foundation server (v15.0.4557.1000)
PRE TLS Microsoft.SystemCenter.2007

 

Resolution
Unseal (if necessary), update connection string, and reimport management packs
If Sealed vendor MP, request new MP via support Incident (and/or UserVoice if Microsoft sourced pack)
If Vendor will not release MP’s, accept risk with the logged errors, update MP, or remove from SCOM

 

 

 

Getting started with OMS Update Compliance

Do you already have Upgrade Readiness or Device Health deployed in OMS?

If not, read this blog

 

Need to know more about Windows 10 patch compliance and don’t want to access Config Man (SCCM)?

 

 

Update Compliance is the answer!

 

It’s just as simple as adding the OMS Update Compliance Solution

Click on the Shopping bag (on left hand pane)

Scroll right to Update Compliance

 

Click Add (this will be Add not View, if you don’t already have the solution loaded)

 

Voila! (time elapsed as the solution gathers data every 12 hours )