Tag: OMS

Service Map SCOM pack errors and events

Running Service Map SCOM management pack and getting errors?

 

 

 

Gotta love holidays

Good family time

Not at work if we’re lucky.

When you come back, do you have to go investigate some new/weird errors?

 

 

This was one of those holidays for me 🙂

 

 

 

Figured I’d document SCOM errors, indicate what Event Sources, event ID ranges that aid troubleshooting.

 

Event Source = MS ServiceMap OMS

Event ID range = 46649-46652

 

Long story short, the root cause for my case, my azure workspace was disabled (fun part with a lab is trying to see how much you can do before it disables!)

 

Digging in my inbox, found this over the weekend

Email subject: Your services were disabled because you reached your spending limit

 

 

SCOM Alerts seen:

 

Service Map Unknown Exception

 

SCOM Console alert example

 

Cause:    May point to Network Connectivity, proxy, or subscription disabled

REST request failed, so did name resolution (may indicate DNS issues)

 

Rule details

Operations Manager Event Log

Event Source = MS ServiceMap OMS

Event ID 46651

 

Operations Manager Event log

 

 

 

No Machines Alert

Rule Name = Microsoft System Center ServiceMap No Machines Alert

Event Source = MS ServiceMap OMS

Event ID = 46652

Event ID also seen is 46649 – Error in getting machine details

 

SCOM Console alert

 

 

 

 

Event ID 46649

 

 

 

 

 

Set up Azure Service Principal

 

Azure Service principal is like a Mech ID that does work for you behind the scenes

Stack Overflow states it plainly

An Azure service principal is a security identity used by user-created apps, services, and automation tools to access specific Azure resources.
Docs site defines it as a Security identity object
We will need the AAD Tenant ID, Application ID (service principal, and Password (key)

AAD Tenant ID

 

For Service Map, the Tenant ID is the Azure Active Directory, Directory ID

 

From Azure Portal

Select Azure Active Directory > Properties > Directory ID in the Azure portal

See Docs site link

Save this to notepad, somewhere for safe keeping – password safe

Tenant ID

This is where you setup the Service Principal for an application
Azure Active Directory is NOT required
From Azure Portal
Click on Azure Active Directory
Click on Properties
Copy the Directory ID
From OMS
Click on Overview, Settings
Click on Accounts, Manage Users
Copy the Tenant ID
Once you have the Directory ID copied to notepad, you need to set up an App registration

App Registration ID

From Azure Portal
Click Azure Active Directory
Click App Registrations
Click + New application registration
Create name and URL
My example is ‘ServiceMap-App’ with my domain
Click Create
 
Click Settings
Click Keys
Recommend setting 2 keys, and save to notepad, and somewhere secure
I did 1 year and 2 year keys
Enter name for Description, Duration box, and click Save
Value will be displayed
Copy the value

PLEASE!!!!

Don’t exit without grabbing the keys!  You will have to delete the App-Registration and start over
After creation, copy the values from Notepad for Tenant ID, Application ID, and keys

 

Service Map for SCOM

 

Ever compare your work to an amusement park?

Every business application compares to a ride, roller coaster, or even a kiddie ride.

Anyone ever ask you directions to that ride, or more technical based questions like ‘what’ communication makes up that business application?

 

 

In comes Service Map to save the day!

 

 

Last year I blogged about setting up Service Map with OMS/Log Analytics, but I didn’t get the feature installed for SCOM.

December blog on how to set up OMS/Log Analytics

 

It’s basically the SCOM Agent (MMA) and a Dependency Agent (think old Blue Stripe agent)

 

Excited to see the new Service Map to hit public preview, hoping by September

 

 

Check out the blog series

Planning and PreReqs blog
Install and configure MMA agent blog
Dependency agent blog

Set up Azure Service Principal blog
Set up SCOM Management Group blog

 

 

Installing and configuring the MMA agent

 

Maybe the MMA agent is like Venom?
Proof I’ve watched too many a Marvel movie…

 

An existential moment perhaps, but the MMA agent can be a bunch of strings stuck from one place to another, monitoring whatever its told to do.

 

 

 

If you are running SCOM2016 or above, the MMA agent is built-in with Log Analytics, just configure your workspace

 

 

 

 

Download and Install MMA agent

SCOM 2012R2 agent does not have MMA, so download MMA agent from Log Analytics workspace

Azure Portal > Log Analytics > Subscription > Advanced Settings

Click on Windows Servers from Connected Sources to download Windows Agent

Click on Linux Servers from Connected Sources to download Linux Agent

 

 

From the Azure Portal (https://ms.portal.azure.com)

Click on Log Analytics, <your subscription >

Click on Advanced Settings

My view defaulted to Connected Sources > Windows Servers

 

Save the workspace ID and workspace key to notepad/OneNote for later

 

 

 

< Assuming the MMA agent is installed with Log Analytics capability >

 

 

Update MMA Agent with Workspace ID and Key

From MMA agent, update the OMS Workspace with the GUID copied to notepad

 

Click on Start > Control Panel, System and Security > Microsoft Monitoring Agent

Click on Azure Log Analytics (OMS) tab on MMA agent

Click Add

 

Add Workspace ID and Key to agent

Click OK

Click OK again on MMA properties

 

Look for the healthy green checkbox’d circle

 

Troubleshooting Errors in the Operations Manager Event Logs

Blog posts – Verify, 55002

 

 

 

 

 

 

 

 

Azure Log Analytics Service Map Planning and Pre-reqs

My grandfather said two things:

An ounce of prevention is worth a pound of manure

Death and taxes are part of life

 

Planning out a deployment is a good thing.

My best friend would say “No one plans to fail, they just fail to plan”

 

 

This will be a multi-part blog – breaking out the high level steps, and my experience getting the solution set up.

 

What do we need for Service Map?

  • Azure connectivity
    • Setup Log Analytics workspace on MMA/SCOM agent article
    • Troubleshooting onboarding issues KB,
      • Check for Events in Operations Manager event logs blog
  • Computers in scope for visualization
    • What computers (Windows or Linux)
    • Pricing FAQ
  • Dependency agent installed on computers
  • Azure Service Principal
    • (think of it as an SSH shared key ID/password for Azure Apps to communicate)
    • Docs article

 

High level steps

  1. Overview blog
  2. Install the MMA agent blog
  3. Install the dependency agent blog
  4. Configure Azure Service Principal blog
  5. Configure Service Map on SCOM blog

 

OMS Heartbeat failures and creating alerts

 

Feel like you have a ton of data, but lack insights?

 

 

Would like to thank the Product team to clarify how to do this with Kusto (new OMS Query language)

 

If you use OMS and need to verify the most recent data collection

Heartbeat | summarize max(TimeGenerated)

If you want to check a specific machine you can run this one:  

Heartbeat | where Computer==”contosovm” | summarize max(TimeGenerated)

If you want to build an alert based on it you can write something like:

Heartbeat | where Computer==”contosovm” | summarize m=max(TimeGenerated) | where m < ago(15m)

The last query will return result only if you have heartbeat missing for more than 15 minutes.

 

If you need additional information on OMS query syntax, check out Antoni’s blog

https://blogs.technet.microsoft.com/antoni/2018/02/03/operations-management-suite-101-log-analytics-queries-101/

 

Kusto site http://kusto/

Importing OMS views

Perusing the TechNet Gallery looking at other’s contributions, I came across some cool new OMS views by Cameron Fuller and Marin Frankovic 

 

What came next kept me puzzled for a few minutes… Where do I go to export/import views in OMS?

 

 

How to add views to OMS

From OMS,

Click the + sign for View Designer

 

Click Import (can only select one view at a time)

Click Browse at bottom

 

Select OMSview file to import

Click Open

 

Verify Overview Tile

Click Save

 

 

Click Home icon to return

 

 

Monitor Sensitive User

Custom Views https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-view-designer

Download https://gallery.technet.microsoft.com/Monitor-sensitive-user-and-5a721d64

Server Performance

                Blog http://blogs.catapultsystems.com/cfuller/archive/2017/11/28/updating-the-server-and-client-performance-solution-to-the-new-query-language/

                Download https://gallery.technet.microsoft.com/scriptcenter/Server-Performance-3d767ab1

 

 

OMS/Advisor Event ID 55002

 

This article is written for the Gateway CommunicationSecurityException event

At first I thought maybe this was TLS1.2 enabling, but backed off the change, the events kept pouring in every 5 minutes.

Tried to reconfigure the OMS/Advisor environment, and voila! Error resolved

 

Let’s go through the steps to re-configure the Operations Management Suite (OMS) in SCOM

 

Reconfigure OMS

  1. From the SCOM Console, click on Administration tab
  2. Expand Operations Management Suite (Advisor on 2012R2)
  3. Click on the Connection
  4. On the center pane, click on Re-configure Operations Management Suite

 

5. Add any trusted sites to IE if there are pop-ups

I had 2 missing websites

Secure.aadcdn.microsoftonline-p.com

az416426.vo.msecnd.net

( I hit Previous and next to verify the wizard would pass with the hopes the attempt would retry)

6. Exit the Reconfigure wizard to get a retry (then the second website popped up as an untrusted site)

7. Enter credentials to your OMS environment

 

Connection to OMS successful

 

Click Next twice

Reconfigure success

Click Close

 

Verify Event Log

Verify Operations Manager Event Log has no new events (this check runs every 5 minutes by default)

get-eventlog -logname “Operations Manager” | ? { $_.EventID -match 55002 } | select-object -last 2

 

 

Event ID 55002 from Operations Manager Event Log

Log Name:      Operations Manager
Source:        Advisor
Date:          12/11/2017 2:15:20 PM
Event ID:      55002
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      16MS01.testlab.net
Description:
Failed to synchronize the latest Management Package information from Advisor Cloud service. Wait for the next cycle to retry. Reason: Microsoft.SystemCenter.Advisor.Common.WebService.GatewayCommunicationSecurityException: Message security was invalid for the connection with web service when performing Get Intelligence Packs with client specified versions —> System.ServiceModel.Security.MessageSecurityException: An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail. —> System.ServiceModel.FaultException: ID3242: The security token could not be authenticated or authorized.
— End of inner exception stack trace —

Server stack trace:
at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.ProcessReply(Message reply, SecurityProtocolCorrelationState correlationState, TimeSpan timeout)
at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Request(Message message, TimeSpan timeout)

Exception rethrown at [0]:
at System.ServiceModel.Security.IssuanceTokenProviderBase`1.DoNegotiation(TimeSpan timeout)
at System.ServiceModel.Security.IssuanceTokenProviderBase`1.GetTokenCore(TimeSpan timeout)
at System.IdentityModel.Selectors.SecurityTokenProvider.GetToken(TimeSpan timeout)
at System.ServiceModel.Security.Tokens.IssuedSecurityTokenProvider.GetTokenCore(TimeSpan timeout)
at System.IdentityModel.Selectors.SecurityTokenProvider.GetToken(TimeSpan timeout)
at System.ServiceModel.Security.SecurityProtocol.TryGetSupportingTokens(SecurityProtocolFactory factory, EndpointAddress target, Uri via, Message message, TimeSpan timeout, Boolean isBlockingCall, IList`1& supportingTokens)
at System.ServiceModel.Security.TransportSecurityProtocol.SecureOutgoingMessageAtInitiator(Message& message, String actor, TimeSpan timeout)
at System.ServiceModel.Security.TransportSecurityProtocol.SecureOutgoingMessage(Message& message, TimeSpan timeout)
at System.ServiceModel.Security.SecurityProtocol.SecureOutgoingMessage(Message& message, TimeSpan timeout, SecurityProtocolCorrelationState correlationState)
at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [1]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Microsoft.AttachedServices.WebService.IIntelligenceService.GetIntelligencePacksInfo(ClientProperties clientProperties)
at Microsoft.SystemCenter.Advisor.Core.WebService.WebServiceCallHelper.CallWebService[T](Func`1 webServiceCall, String webServiceDescription)
— End of inner exception stack trace —
at Microsoft.SystemCenter.Advisor.Core.WebService.IntelligenceServiceClient.CallWebServiceWithRetry[T](Func`2 function)
at Microsoft.SystemCenter.Advisor.Core.WebService.IntelligenceServiceClient.GetIntelligencePacksInfo(ClientProperties clientProperties)
at Microsoft.SystemCenter.Advisor.Core.IntelligencePackWriteAction.UpdateIntelligencePacks()

Getting started with OMS Update Compliance

Do you already have Upgrade Readiness or Device Health deployed in OMS?

If not, read this blog

 

Need to know more about Windows 10 patch compliance and don’t want to access Config Man (SCCM)?

 

 

Update Compliance is the answer!

 

It’s just as simple as adding the OMS Update Compliance Solution

Click on the Shopping bag (on left hand pane)

Scroll right to Update Compliance

 

Click Add (this will be Add not View, if you don’t already have the solution loaded)

 

Voila! (time elapsed as the solution gathers data every 12 hours )

Get started with OMS Device Health

Anyone need telemetry data for win10 computers?

https://docs.microsoft.com/en-us/windows/deployment/update/device-health-get-started

 

Want the info with better reports and less overhead?

This easily replaces SCOM Agentless Exception Monitoring

 

OMS is technically free, why not get insights into client side problems?

 

 

Overview

Validate Telemetry Setting

Get CommercialID from OMS

Configure Deployment Script

Run Deployment Script

Verify OMS

 

 

Check Win10 Telemetry setting

Configure Telemetry Data link

 

FYI – Telemetry level can be managed via SCCM/MDM/Intune and/or GPO

 

Enhanced Telemetry (2) sends less data (not full crash dumps like Full)

The normal upload range for the Enhanced telemetry level is between 239 KB – 348 KB per day, per device.

 

Settings Explained

 

 

 

Verify Telemetry setting

My default Win10 setting was 3 based on setup wizard options

 

 

 

Retrieve CommercialID from OMS

Go to Settings (Cog at the top right hand corner)

Then Click on Connected Sources, Windows Telemetry

Copy the Commercial ID Key

 

 

 

 

Set up Deployment Script

Download the Deployment Script link

In my lab example, save script to Win10 client in C:\UpgradeAnalytics

 

Update the Deployment RunConfig.bat file

From Docs.Microsoft.com:

The Pilot folder contains advanced logging that can help troubleshoot issues and is intended to be run from an elevated command prompt.

 

Edit RunConfig.bat in Notepad, add your Commercial ID into the ‘set commercialIDValue’ line

Change the logPath as well if you have a preferred logging location

 

Run script and verify Registry keys

Set up command window as system

Don’t forget psexec from sysinternals tool

psexec -s cmd.exe

cd UpgradeAnalytics\Deployment

runConfig.bat

 

Example output

 

Verify Registry

Registry key paths depending on how these are set with SCCM/MDM/Intune vs. GPO

        $vCommercialIDPath = “HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\DataCollection”
        $GPOCommercialIDPath = “HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection”

 

 

 

Add Device Health Solution to OMS

Add Device Health as part of the Windows Analytics suite

NOTE Windows Analytics suite includes Upgrade Readiness and Update Compliance

 

 

Wait 2 days and see what shows up as devices check in

 

Clicking on Device Health pane

 

Added Bonus – once you configure the deployment script, the other two Windows Analytics tools are ready for consumption – Upgrade Readiness and Update Compliance

 

 

Requirements

OMS subscription

Win10 clients have HTTPS access to Microsoft hosts (see Endpoints in Configure Telemetry link below)

 

 

References

Windows Analytics link
Upgrade Readiness link
Upgrade Readiness Script V2 link
Upgrade Readiness Script Original link
Configure Telemetry link