Lync 2013 Addendum Management Pack

Continuing the Addendum tradition 🙂 Lync couldn’t be forgotten.

 

To understand options and methods available on the Server and SCOM, re-read the Active Directory Addendum blog

 

 

Lync 2013

Now that we understand the methods available, let’s get to the Addendum.

 

 

The Addendum pack has 32 Recovery Tasks for Lync Service Monitors.

 

The recoveries cover the following services:

Access Edge, CMS Master, File Transfer Agent, Lync Backup Service, Push Notification Service, Replica Replicator Agent, Online Telephony Conferencing, Audio Video Conferencing, BI Data Collector, Conferencing Attendant, Conferencing Announcement, Application Sharing, Persistent Chat, Persistenc Chat Compliance, Centralized Logging Service Agent, Call Park, Web Conferencing, Web Conferencing Edge, IM Conferencing, Legal Intercept Service, Log Retention Service, Audio Video Edge, Mediation, Audio Video Authentication, Bandwidth Policy Service Authentication, Bandwidth Policy Service Core, Server Response Group, Front End Service, World Wide Web Publishing, XMPP Translating Gateway, XMPP Translating Gateway Proxy.

The recovery tasks verify service state, start ‘not running’ services, and include the option to recalculate health.

 

 

My goal is automation that helps anyone work smarter versus harder, with the goal to avoid being woke up at 2am just to restart a service.

 

Gallery Download          https://gallery.technet.microsoft.com/Lync-2013-Addendum-2a92aa00

Skype for Business 2015 (SfB) Addendum Management Pack

 

 

Continuing the Addendum tradition 🙂 Skype was next on the list.

 

To understand options and methods available on the Server and SCOM, re-read the Active Directory Addendum blog

 

 

Skype for Business 2015 (SfB)

Now that we understand the methods available, let’s get to the Addendum.

This Skype Addendum MP adds Recovery Tasks to the Skype for Business 2015 Service Monitors.

The recovery tasks verify service state, start ‘not running’ services, and recalculate health.

36 services monitored, with 36 recovery tasks.

The recovery tasks verify service state, start ‘not running’ services, and include the option to recalculate health.

 

 

My goal is automation that helps anyone work smarter versus harder, with the goal to avoid being woke up at 2am just to restart a service.

 

Gallery Download      https://gallery.technet.microsoft.com/Skype-for-Business-2015-b005f49f

 

Active Directory 2012-2016 Addendum Management Pack

A Post-it note is like an addendum, no?

 

 

As an Operations engineer, how many times do you get notified for a service restart?

 

Did you know about Service Recovery actions, or SCOM Recovery Tasks?

 

Why didn’t the SCOM Recovery tasks get added to many of the common Microsoft Applications?

 

 

Hopefully today, we can discuss some actions to help limit the amount of manual rework required to resolve service issues.

 

Let’s explain the basics

  1. Windows Servers have a Recovery tab in the Services.msc menu.
  2. Does your monitoring tool allow for recovery actions?

 

 

To implement recovery actions, here’s an example of the Services Recovery Tab

Here’s an example of the SCOM agent service

          NOTE 3 failures spaced 1 minute apart to restart the service

 

 

Let’s take it one step further, and add a restart to the service from another tool (insert your monitoring tool here).

 

In SCOM, taking an action after identifying the problem can be handled different ways

  • Services are related to Health, which are typically found as monitors, and to apply restart automation falls into Recovery Tasks.

 

  • In Monitors as a ‘Recovery Task’, or in Rules as a response

  • Rule Response

 

 

 

 

Active Directory Domain Services (AD DS)

Now that we understand the methods available, let’s get to the Addendum.

The Active Directory Domain Services Addendum MP will add Recovery tasks to AD DS Service Monitors.

NOTE: This is for the newer v10.0.x.y management packs that support AD DS 2012-2016

 

Specifically, the Pack has 12 Recovery tasks for DFS, NTDS, DFSR, IsmServ, KDC, NetLogon, NTFRS, W32Time, Group Policy, DNS Client, ADWS, and DNS.

 

The recovery tasks verify service state, start ‘not running’ services, and include the option to recalculate health.

 

 

My goal is automation that helps anyone work smarter versus harder, with the goal to avoid being woke up at 2am just to restart a service.

 

Gallery Download      https://gallery.technet.microsoft.com/SCOM-AD-Directory-Addendum-22d0473a

 

SCHANNEL event logging

First, my thanks to Bhuvnesh Kumar for his help!

 

Time to figure out what’s going on behind the curtain!

 

 

Are you seeing System Event Log, Event ID 36871 events?

 

Why does this matter?

 

Depending on OS versions and patches, the TLS Cipher Suites may not match on the various SCOM servers.

  1. If you’re setting up TLS1.2, you need the SCOM servers to talk
  2. The bad part, is this isn’t logged much on the GW but log more often on MS
  3. Sometimes the 36871 events come with 36874, but in my experience they occur after Event Logging is enabled.

 

The unanswered question is “why are we seeing the 36871 events?”

 

In my example, the events only happened once a day, roughly 24 hours

 

Event Viewer

 

Are events related to the Cipher Suite, or is it a MP trying to run the old SQLOLEDB method?

 

This article will focus on verifying Cipher Suite on a server

See this article for MP analysis for SQL methods

 

 

 

SCHANNEL event logging setup

 

From Holman’s blog

DecimalDescription
0Do not log
1Log Error messages
2Log Warnings
3Log Error and Warning messages
4Log Informational and Success events
5Log Error, Informational and Success events
6Log Warnings, Informational and Success events
7Log Everything (Warnings, Errors, Informational and Success events

 

I’d recommend setting it to 3 to see errors and warnings, or 7 to see everything.

Remember to set this back to 1 when done resolving any issues.

 

Add

From Command Prompt or PowerShell (as administrator)

reg add “HKLM\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL” /v “EventLogging” /t REG_DWORD /d 7 /f

Disable

reg delete “HKLM\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL” /v “EventLogging”

Verification

reg query “HKLM\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL” /v “EventLogging”

 

PowerShell verification

RegEdit Verification

Time to reboot!

 

 

Verify SCHANNEL events

Look at the System Event log, and filter for 36880 and 36874 events for clues

 

36880 provides Cipher Suite details

 

Event ID 36874 definitely describes the scenario

 

The easy answer to solve the cipher suite is to ask – is this server patched with latest security and .NET patches?

After all this, in my example, we confirmed that simple step was assumed, and inaccurate.

 

 

 

 

References
36871 event https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-R2-and-2012/dn786445(v=ws.11)

SCHANNEL events https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-R2-and-2012/dn786445(v=ws.11)

SChannel error codes https://docs.microsoft.com/en-us/windows/win32/secauthn/schannel-error-codes-for-tls-and-ssl-alerts

SChannel events https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-R2-and-2012/dn786445(v=ws.11)

SSL errors https://www.experts-exchange.com/questions/28996780/event-id-36871-Schannel.html

Troubleshooting https://docs.microsoft.com/en-us/iis/troubleshoot/security-issues/troubleshooting-ssl-related-issues-server-certificate

SQL native client for TLS1.2

Ever try to talk to someone when language is a barrier?

 

Sure, we can run an app, or search our phrase to pronounce, but it’s so much better when we can communicate seamlessly.

 

Post TLS1.2 for SCOM

Let’s talk SQL

Part of TLS1.2 is updating SQL Native Client to talk using a secure client that uses TLS1.2

That means a different executable should be called.

 

Why is that important in SCOM?

Maybe you have management packs that connect to SQL or run external commands.

 

 

On MS, there are multiple clues for various errors on Management Packs that use SSL or talk to SQL via a non-TLS method.  NOTE this may mean that the SQL DB that management pack is connecting to may need the same pre-req SQL updates to a TLS 1.2 enabled version.

  1. Do you have custom SQL queries being run, CMDB get’s, OLE DB Data Source checks?
  2. Any Event ID 1401 or 11854 events in the Operations Manager Event log?
    1. These events identify management pack scripts creating SCHANNEL events
      a. Event ID 1401 event example

 

 

Cause

SQLOLEDB connection strings will cause 36871 Sytem Log events

 

Example (TLS1.0)
sConnectString = “PROVIDER=SQLOLEDB;DATA SOURCE=<databaseServerFQDN>;DATABASE=MSSQLSERVER;trusted_connection=yes”
 SQLNCLI11 driver for TLS1.2 connection strings

Example (TLS1.2)
 sConnectString = “Provider=SQLNCLI11;DATA SOURCE=<databaseServerFQDN>;DATABASE=MSSQLSERVER;trusted_connection=yes”
 

 

Identify
Look for management packs with SQLOLEDB as the Connect string to reduce 36871 SCHANNEL events

In Windows Explorer, use the Advanced Options dropdown to select File Contents
In the Search bar (top right), enter SQLOLEDB (example shows SQLNCLI11)
NOTE SQL Discovery group pack IS compliant

 

 

In Windows Explorer, use the Advanced Options dropdown to select File Contents
In the Search bar (top right), enter SQLNCLI11

 

 

Additional offenders
HP Topology MP
SQL 2005 discovery MP (discontinued)
SQL Addendum MP’s (will work to update these with Holman)
SharePoint Foundation server (v15.0.4557.1000)
PRE TLS Microsoft.SystemCenter.2007

 

Resolution
Unseal (if necessary), update connection string, and reimport management packs
If Sealed vendor MP, request new MP via support Incident (and/or UserVoice if Microsoft sourced pack)
If Vendor will not release MP’s, accept risk with the logged errors, update MP, or remove from SCOM

 

 

 

Set up SCOM 2016 for TLS1.2

Security bugging you about SCOM using TLS1.0 ?

 

Have questions on the TLS1.2 Protocol Support Deployment guide link?

If using ACS, please review ACS steps to configure from the guide above

 

It’s time to update SCOM 2016 to TLS1.2!

 

Pre-requisites

.Net and SQL native client, ODBC must be updated to TLS1.2 compliant version

HTTPS Endpoints must be CA signed certificates using SHA1 or SHA2

 

 

 

Ensure .Net version 4.6 is installed on all SC components

Determine which .Net is installed https://docs.microsoft.com/en-us/dotnet/framework/migrationguide/how-to-determine-which-versions-are-installed

From PowerShell (run as admin is NOT required)

Get-ChildItem ‘HKLM:\SOFTWARE\Microsoft\NET Framework Setup\NDP’ -recurse | Get-ItemProperty -name Version,Release -EA 0 | Where { $_.PSChildName -match ‘^(?!S)\p{L}’} | Select PSChildName, Version, Release

 

Above commands from StackOverFlow article

Guide to .Net versions and dependencies https://docs.microsoft.com/enus/dotnet/framework/migration-guide/versions-and-dependencies

 

Sample output from win2k8R2 sp1 server (and same from 2016 server)

 

 

 

 

SQL Server updates

Install the required SQL server update supporting TLS1.2

From PowerShell as Administrator
Invoke-Sqlcmd -Query “SELECT @@VERSION;” -QueryTimeout 3

Example Output
PS C:\Windows\system32> Invoke-Sqlcmd -Query “SELECT @@VERSION;” -QueryTimeout 3
Column1

——

Microsoft SQL Server 2016 (RTM-GDR) (KB3210111) – 13.0.1728.2 (X64) …

OR

Microsoft SQL Server 2008 R2 (SP2) – 10.50.4000.0 (X64)

 

Compare to SQL matrix to download and install appropriate version
TLS 1.2 SQL Support https://support.microsoft.com/en-in/help/3135244/tls-1.2-support-for-microsoft-sql-server
NOTE Verify you are running a compliant cumulative update (CU), you will need the patch (SQL2016 natively supports TLS1.2)
SQL Server 2008R2 SP2 is NOT supported for TLS1.2  https://blogs.msdn.microsoft.com/sqlreleaseservices/tls-1-2-support-for-sql-server-2008-2008-r2-2012and-2014/

 

Install the required SQL Native Client
FYI – SQL 2016 uses the SQL 2012 Native client
Download link https://www.microsoft.com/en-us/download/details.aspx?id=50402

 

SQL Native client 11.0 should be installed on ALL MS and SQL servers (SQL 2008-2016)

From PowerShell as Administrator
get-odbcdriver -name “SQL Server Native Client*”

 

Example Output

 

 

From Control Panel, Programs and Features, Installed Programs

 

Stop SQL Server and SQL Server agent services
Stop-service SQLSERVERAGENT

Stop-service MSSQLSERVER

 

Install SQL Native Client MSI

Double click on SQL Native Client MSI file to begin installation

Click on Yes to begin installation

Click Next on the Installer window

 

Click I accept radio button

Click Next

 

Click Next on Feature Selection

 

Click Install

 

Click Yes on User Account Control (UAC) prompt

 

Stop SQL Server and SQL Server agent (if they restarted)

 

Watch installer status

 

Click Finish when complete

 

 

 

Verify SQL Native Client Verification

Verify SQL services are running
Stop SQL Server and SQL Server agent services From PowerShell as Admin
Get-service SQLSERVERAGENT

Get-service MSSQLSERVER

From PowerShell as Admin If necessary, start SQL Server and SQL Server agent services
Start-service SQLSERVERAGENT

Start-service MSSQLSERVER
Verify Installer completed
In Event Viewer, Windows Logs, Application look for event 11728

 

From PowerShell

Get-EventLog -LogName Application | ? { $_.InstanceId -eq 11728 }

Rinse and Repeat for other MS and SQL servers in environment

 

 

Install ODBC on all Management Servers

 

For SCOM & SM, ODBC 11.0 or ODBC 13.0 should be installed on all MS and SQL servers

 

Verify ODBC v11 for server win2k8R2

From Control Panel

Click on Programs

Click on Programs and Features

Search for ODBC

 

Verify ODBC v13 for Server 2016

Verify version from PowerShell (run as administrator NOT required)
get-odbcdriver -name “ODBC Driver * SQL Server”

 

Output

Download and install appropriate version

11.0: https://www.microsoft.com/en-us/download/details.aspx?id=36434 (Version 2.0.5543.11)
13.0: https://www.microsoft.com/en-us/download/details.aspx?id=50420
Verify Installer completed
In Event Viewer, Windows Logs, Application look for event 11728

 

From PowerShell

Get-EventLog -LogName Application | ? { $_.InstanceId -eq 11728 } | ? { $_.Message -like “*Microsoft ODBC*”

 

Output

 

NOTE Please make sure servers are patched with latest Monthly Rollup Updates

Had issue where KB3080079 was NOT installed on server.  Patch applied to Win7, Server 2008,2008R2

From Powershell

get-hotfix -id KB3080079

 

Output

 

 

 

Install SCOM 2016 UR4 update

See Kevin Holman’s UR4 install blog https://blogs.technet.microsoft.com/kevinholman/2017/10/28/ur4-for-scom-2016-step-by-step/

 

Time to enable TLS1.2 Secure Channel messages on MS and SQL server (gateway if installed in your environment)

See Gallery for add/query/remove registry keys

 

Add SCHANNEL path for TLS

$ProtocolList       = @(“SSL 2.0″,”SSL 3.0″,”TLS 1.0”, “TLS 1.1”, “TLS 1.2”)
$ProtocolSubKeyList = @(“Client”, “Server”)
$DisabledByDefault = “DisabledByDefault”
$Enabled = “Enabled”
$registryPath = “HKLM:\\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\”

foreach($Protocol in $ProtocolList)
{
    Write-Host ” In 1st For loop”
foreach($key in $ProtocolSubKeyList)
{
$currentRegPath = $registryPath + $Protocol + “\” + $key
Write-Host ” Current Registry Path $currentRegPath”

if(!(Test-Path $currentRegPath))
{
    Write-Host “creating the registry”
New-Item -Path $currentRegPath -Force | out-Null
}
if($Protocol -eq “TLS 1.2”)
{
    Write-Host “Working for TLS 1.2”
New-ItemProperty -Path $currentRegPath -Name $DisabledByDefault -Value “0” -PropertyType DWORD -Force | Out-Null
New-ItemProperty -Path $currentRegPath -Name $Enabled -Value “1” -PropertyType DWORD -Force | Out-Null

}
else
{
    Write-Host “Working for other protocol”
New-ItemProperty -Path $currentRegPath -Name $DisabledByDefault -Value “1” -PropertyType DWORD -Force | Out-Null
New-ItemProperty -Path $currentRegPath -Name $Enabled -Value “0” -PropertyType DWORD -Force | Out-Null
}
}
}

 

# Tighten up the .NET Framework
$NetRegistryPath = “HKLM:\SOFTWARE\Microsoft\.NETFramework\v4.0.30319”
 New-ItemProperty -Path $NetRegistryPath -Name “SchUseStrongCrypto” -Value “1” -PropertyType DWORD -Force | Out-Null

$NetRegistryPath = “HKLM:\SOFTWARE\WOW6432Node\Microsoft\.NETFramework\v4.0.30319”
 New-ItemProperty -Path $NetRegistryPath -Name “SchUseStrongCrypto” -Value “1” -PropertyType DWORD -Force | Out-Null

 

Restart servers

 

 

Verify SCOM Console for alerts and connectivity

 

Adding UNIX agents via PowerShell

First, a shout out to Vanessa Bruwer @VanessaBruwer and Tyson Paul for their help!

 

Feel like I was pounding rocks, and had a great find! 🙂

…How to add UNIX agents manually via command line

 

 

 

Required

1. Unix Agent action account and agent maintenance account ID and passwords
2. Unix Resource Pool name (use get-SCOMResourcePool)

Don’t confuse the WSMAN login and use your MSAA ID

BTW, cmdlets exist with 2012R2 and 2016

 

 

From MS running PowerShell as admin

$MyPool = Get-SCOMResourcePool “UNIX/Linux Monitoring Resource Pool”
$SSHCredential = Get-SCXSSHCredential -UserName scom -ElevationType sudo
$WSCredential = Get-Credential scom

# Using MSAA account this fails

$DiscResult = Invoke-SCXDiscovery -Name “ubuntu.testlab.net” -ResourcePool $MyPool -WSManCredential $WSCredential -SSHCredential $SSHCredential

# Alternative Discovery for Network IP range
$DiscResult = Invoke-SCXDiscovery -IPRange 192.168.1.50,192.168.1.75  -ResourcePool $MyPool -WSManCredential $WSCredential -SSHCredential $SSHCredential

 

$DiscResult |fl -property *

 

$installResult = Install-SCXAgent -DiscoveryResult $DiscResult -Verbose
$installResult | fl -property *

 

 

Using MSAA account this fails

 

Using SCOM Agent Maintenance Account

 

Console verified

 

Resources
Old https://blogs.msdn.microsoft.com/scxplat/2009/12/11/cross-platform-powershell-scripts-released/
2012R2 (tested on my 2016 lab) https://operatingquadrant.com/2012/12/06/using-powershell-for-automated-unixlinux-agent-discovery/
PoSH cmdlet reference https://docs.microsoft.com/en-us/previous-versions/system-center/powershell/system-center-2012-r2/hh545212(v=sc.20)

 

 

 

 

Sealing SCOM MP’s

 

Sealing MP’s

This is an updated version of Kevin Holman’s blog, and Jonathan Almquist’s blog for SCOM2012R2 and 2016

 

First why seal?

If you seal the MP – we will be able to use the classes/groups created for overrides in any other override MP.

Unsealed MP – any overrides you use for classes/groups will be forced into this same MP.

 

 

If you don’t have Visual Studio 2013 and above with VSAE, or have other requirements, you will need to download the SDK to get the SN.exe utility

 

Download SDK

Win2008 & R2 SDK No longer available for Download

Win2012 & R2 SDK Download

Win10 SDK Download https://developer.microsoft.com/en-US/windows/downloads/windows-10-sdk

Release blog https://blogs.windows.com/buildingapps/2017/05/11/windows-10-sdk-preview-build-16190-released/

NOTE Install path to go grab the sn.exe file

 

 

Install SDK

Copy file to the local machine

Open PowerShell window as administrator

cd $HOME/desktop

.\sdksetup.exe          # .\winsdksetup.exe for Server 2016/win10

 

 

Verify SN.exe is found after SDK install completes

Server 2008 – sn.exe located in C:\Program Files\Microsoft SDKs\Windows\v6.1\Bin\x64

Server 2012 – sn.exe located in C:\Program Files (x86)\Microsoft SDKs\Windows\v8.1A\bin\NETFX 4.5.1 Tools

Server 2016 and Win 10 – sn.exe located in C:\Program Files (x86)\Microsoft SDKs\Windows\v8.0A\bin\NETFX 4.0 Tools

 

 

 

Create the MPSeal folders

Repository where you want to keep the MPSeal.exe, SNK files, and related sealed packs for any MP sealed
C:\monadmin\MPSeal
C:\monadmin\MPSeal\unsealed
C:\monadmin\MPSeal\sealed # Reference sealed MP’s
C:\monadmin\MPSeal\key
C:\monadmin\MPSeal\output

PowerShell as Admin commands to create repository

new-item -itemtype directory -path c:\monadmin\
new-item -itemtype directory -path c:\monadmin\MPSeal
new-item -itemtype directory -path c:\monadmin\MPSeal\unsealed
new-item -itemtype directory -path c:\monadmin\MPSeal\sealed
new-item -itemtype directory -path c:\monadmin\MPSeal\key
new-item -itemtype directory -path c:\monadmin\MPSeal\output

 

 

Copy MPSeal utility from Support directory on SCOM ISO

On ISO, copy mpseal* from ISO SupportTools\AMD64 directory to c:\monadmin\MPSeal

 

 

 

Let’s get the Key file generated and start sealing MP’s!

 

Create SNK files
Note SN.exe only needs to be run once to create the SNK file
***Critical note – you need to keep a backup of this key… because it will be required for making updates to this MP in the future, re-sealing, and keeping the ability to upgrade the existing MP in production.

 

sn -k <yourDomainNameHere>.snk

Sample syntax from win2k8 server

Copy this SNK file to c:\monadmin\MPSeal\key

 

 

Copy Referenced MP’s
This is a good opportunity to add the MP’s referenced in the ISO, UR updates, and/or RTM folder when installing SCOM, Unix MP’s, etc.

Copy sealed MP’s to c:\monadmin\MPSeal\sealed

 

 

Seal MP

MPSeal.exe c:\monadmin\mpseal\unsealed\<mpNameHere>.xml /I “c:\monadmin\mpseal\sealed” /Keyfile “c:\monadmin\mpseal\key\PairKey.snk” /Company “CompanyName” /Outdir “c:\monadmin\mpseal\output”

 

 

References
How to Seal MP https://docs.microsoft.com/en-us/previous-versions/system-center/system-center-2012-R2/hh457550(v=sc.12)