Tag: agent

SCOM Agent Maintenance

Wrench for SCOM agent maintenance
Wrench for SCOM agent maintenance

When we talk about best practices for monitoring, this will typically include (SLA) Service Level Availability.  SLA is an important piece in your environment, as uptime and happy customers come with a high SLA.  There are some cases where IT Teams do work on demand.  On-demand work is outside of a standard change window, a scheduled change.  Typically this is outside configuration management tools, responsible to update software (applications/packages), machines, drivers, compliance settings, and more.  In the one-off, non-scheduled maintenance or recovery, try leveraging ‘SCOM Agent Maintenance’ PowerShell commands on SCOM agents.

 

SCOM Agent maintenance PowerShell commands

cd “C:\Program Files\Microsoft Monitoring Agent\Agent”

Import-module .\MaintenanceMode.dll

Start-SCOMAgentMaintenanceMode -Duration 10 -Reason PlannedOther

 

# Verify

# If messages show with current timestamp, Agent objects are in maintenance.

 

get-eventlog -LogName “Operations Manager” -newest 50 | ? { $_.Message -like “Suspending monitoring*”  } | ft TimeGenerated,Message -autosize

 

TimeGenerated        Message

————-        ——-

6/25/2020 8:37:57 AM Suspending monitoring for instance “modeldev” with id:”{F9E45AA4-7DF7-C1F1-70C9-5D76C9F2725C}” …

6/25/2020 8:37:57 AM Suspending monitoring for instance “C:” with id:”{ED00048A-7DDC-D4BE-901D-D64DA281B7C6}” as the…

6/25/2020 8:37:57 AM Suspending monitoring for instance “central_log” with id:”{EA619D69-D1CC-3B19-D93C-2E3FCD1409AE…

 

PS C:\Program Files\Microsoft Monitoring Agent\Agent> get-eventlog -LogName “Operations Manager” -newest 25 | ? { $_.Message -like “Resuming monitoring*”  } | ft TimeGenerated,Message -autosize

 

343998 Jun 25 08:50  Information HealthService          1073743040 Resuming monitoring for instance “modeldev” wit…

343997 Jun 25 08:50  Information HealthService          1073743040 Resuming monitoring for instance “C:” with id:”…

343996 Jun 25 08:50  Information HealthService          1073743040 Resuming monitoring for instance “central_log” …

343995 Jun 25 08:50  Information HealthService          1073743040 Resuming monitoring for instance “dnmll05s1.UNE…

Configure MMA agent via PowerShell

A car mechanic uses battery jumper cables to charge a dead battery.

 

Do you feel like a mechanic having to jump start the agent configuration like a dead car battery?   Assuming the Agent is already installed, you can configure the SCOM agent via PowerShell.  Even better when you can PowerShell remote to multiple systems.  I hope the PowerShell commands below help you master PowerShell to configure the SCOM side of the MMA agent (house).

 

powershell

/*
# Find/replace variables to your environment like Kevin Holman’s fragments!
##SCOMMGMTGROUP1##
##SCOMMGMTGROUP2##
##SCOMMGMTSERVER1##
##SCOMMGMTSERVER2##
#
*/

$SCOMAgent = New-Object -ComObject AgentConfigManager.MgmtSvcCfg
$SCOMAgent.GetManagementGroup(“##SCOMMGMTGROUP1##”);$SCOMAgent.GetManagementGroup(“##SCOMMGMTGROUP2##”)

# If mgmt groups are incorrectly set
$SCOMAgent.RemoveManagementGroup(“##SCOMMGMTGROUP1##”)
$SCOMAgent.RemoveManagementGroup(“##SCOMMGMTGROUP2##”)

restart-service healthservice

# Domain
$SCOMAgent.AddManagementGroup(“##SCOMMGMTGROUP1##”,”##SCOMMGMTSERVER1##”,5723)

# Verify agent config
$SCOMAgent.GetManagementGroup(“##SCOMMGMTGROUP1##”)
# If you have a second management group

$SCOMAgent.GetManagementGroup(“##SCOMMGMTGROUP2##”)

# Restart and test connectivity
restart-service healthservice

# Check connectivity
test-netconnection -port 5723 -computername ##SCOMMGMTSERVER1##

 

 

Need to find the command UNIX pack runs for perf counter

Magnifying Glass

 

 

Have you ever needed to find the command UNIX pack runs for perf counter?   Say the processor time value doesn’t match what the Unix admin may be saying SCOM is showing.

 

Many times you can look at the SCOM management pack, and those commands trace back to the UNIX library.

 

Background:  The SCOM management server runs many of the cross-plat/xplat workflows to the UNIX agent through WinRM.

 

Agenda
  1. Unseal SCOM UNIX management pack to obtain URI
  2. Understand command line options from UNIX/Linux side, and how to view the output
  3. Enumerate command line
  4. Test Command line from SCOM MS

 

 

 

Unseal SCOM UNIX management pack

The screenshot below is unsealing the Solaris10 pack to XML, and then viewing/searching to show the processor reference.

Solaris 10 processor rules

NOTE that’s a URI, not a script

 

 

How UNIX admin may supply processor output

Example – Unix admin typically uses vmstat or iostat.

 

The screenshot uses ‘vmstat 2 10‘ – a snapshot every 2 second intervals, 10 times

vmstat output

 

We can discuss the vmstat output, but it shows way more than just processor (ready queue, swap, user, system, and cpu %) to help figure out which operating system component is the problem.

 

 

Enumerate command line test

How do we test the command line syntax, to verify what SCOM pulls when running the rule?

For example, we need to make the URI actionable from the management pack.  What is needed to make a usable command?

 

Grab the URI from the pack –

http://schemas.microsoft.com/wbem/wscim/1/cim-schema/2/SCX_ProcessorStatisticalInformation?__cimnamespace=root/scx

 

Because we know the URI, we now build out the syntax with WinRM

winrm enumerate http://schemas.microsoft.com/wbem/wscim/1/cim-schema/2/SCX_ProcessorStatisticalInformation?__cimnamespace=root/scx -auth:basic -remote:https://<servername>:1270 -username:<scomID, not necessarily root> -skipCACheck -skipCNCheck -skiprevocationcheck –encoding:utf-8

 

 

Test WinRM command from SCOM MS

For instance, we want to test the WinRM command from the MS to the UNIX server

winrm enumerate http://schemas.microsoft.com/wbem/wscim/1/cim-schema/2/SCX_ProcessorStatisticalInformation?__cimnamespace=root/scx -auth:basic -remote:https://ubuntu:1270 -username:scom -skipCACheck -skipCNCheck -skiprevocationcheck –encoding:utf-8

 

Example output

SCX_ProcessorStatisticalInformation
InstanceID = null
Caption = Processor information
Description = CPU usage statistics
ElementName = null
Name = 0
IsAggregate = FALSE
PercentIdleTime = 99
PercentUserTime = 0
PercentNiceTime = 0
PercentPrivilegedTime = 0
PercentInterruptTime = 0
PercentDPCTime = 0
PercentProcessorTime = 1
PercentIOWaitTime = 0

SCX_ProcessorStatisticalInformation
InstanceID = null
Caption = Processor information
Description = CPU usage statistics
ElementName = null
Name = _Total
IsAggregate = TRUE
PercentIdleTime = 99
PercentUserTime = 0
PercentNiceTime = 0
PercentPrivilegedTime = 0
PercentInterruptTime = 0
PercentDPCTime = 0
PercentProcessorTime = 1
PercentIOWaitTime = 0

 

Additional references for WinRM syntax and troubleshooting

Warren’s blog

Docs site

Use Unix MP’s for shell commands

 

Re-learn an old but still relevant tool – EventLog Explorer

 

Sometimes we forget about tools that can make things easier.

 

Time to talk about EventLog Explorer.

 

Need to repro and test events for an installed program, to see what SCOM will handle?

Read this old mom team blog, courtesy of Kevin Holman blog

 

 

I wanted to try it to test fire some events, had a use case where we needed to test Skype events from the SCOM MP

 

Testing on my SCOM 2016 Management server

 

Download file, run EventLog Explorer

The Paste icon next to the X is ‘Add to Execution List’ and fills out the bottom pane

The Green Arrow is ‘go’ or execute (similar to PowerShell ISE)

 

Navigate through the Event Log and Event Source on the left hand pane

Mark events with the checkbox  

 

Add to Execution

 

Verify events added to bottom pane

(see my test yesterday for fired, and not fired events from today)

 

 

 

Click Green box with white arrow to fire events, and check Event Viewer

 

 

Yesterday’s test

 

 

 

Today’s test

 

 

Verify alerting occurred as expected!

Installing and configuring the MMA agent via Command line

Command prompt

 

GUI install option, see blog

PowerShell Agent configuration, see blog

Updated 1 Feb 2023

Pre-reqs to build out an install script/package

MMA agent executable

ALA Workspace ID

ALA Workspace Primary Key

 

 

Download MMA agent

Click on Windows Servers from Connected Sources to download Windows Agent

Click on Linux Servers from Connected Sources to download Linux Agent

 

 

 

 

Obtain WorkspaceID

From the Azure Portal (https://portal.azure.com)

Click on Log Analytics, <your subscription >

Click on Advanced Settings

My view defaulted to Connected Sources > Windows Servers

 

Save the workspace ID and workspace key to notepad/OneNote for later

 

 

 

 

 

Build out command line for setup file

(optionally to include in Application Deployment package)

 

Grab pre-reqs above: (saved from above to build the command line)

Exe/msi file

Workspace ID

Workspace key

 

Craft out your command line (MECM super installer code updated by Neal Smith

SCOM MECM Agent Package Installer Command Line

The setup.exe or MSI command line parameters to pass are:

MMA-specific optionsNotes
NOAPM=1Optional parameter. Installs the agent without .NET Application Performance Monitoring.
ADD_OPINSIGHTS_WORKSPACE1 = Configure the agent to report to a workspace
OPINSIGHTS_WORKSPACE_IDWorkspace Id (guid) for the workspace to add
OPINSIGHTS_WORKSPACE_KEYWorkspace key used to initially authenticate with the workspace
OPINSIGHTS_WORKSPACE_AZURE_CLOUD_TYPESpecify the cloud environment where the workspace is located

0 = Azure commercial cloud (default)

1 = Azure Government

OPINSIGHTS_PROXY_URLURI for the proxy to use
OPINSIGHTS_PROXY_USERNAMEUsername to access an authenticated proxy
OPINSIGHTS_PROXY_PASSWORDPassword to access an authenticated proxy

Example:

setup.exe /qn NOAPM=1 ADD_OPINSIGHTS_WORKSPACE=1 OPINSIGHTS_WORKSPACE_AZURE_CLOUD_TYPE=0 OPINSIGHTS_WORKSPACE_ID=<your workspace id> OPINSIGHTS_WORKSPACE_KEY=<your workspace key> AcceptEndUserLicenseAgreement=1

 

 

 

Other helpful links

Docs site https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-quick-collect-windows-computer

Daniel Orneling Blog https://blog.orneling.se/2017/01/installing-oms-agent-with-powershell/

TechNet gallery https://gallery.technet.microsoft.com/scriptcenter/Install-OMS-Agent-with-2c9c99ab

Adding UNIX agents via PowerShell

First, a shout out to Vanessa Bruwer @VanessaBruwer and Tyson Paul for their help!

 

Feel like I was pounding rocks, and had a great find! 🙂

…How to add UNIX agents manually via command line

 

 

 

Required

1. Unix Agent action account and agent maintenance account ID and passwords
2. Unix Resource Pool name (use get-SCOMResourcePool)

Don’t confuse the WSMAN login and use your MSAA ID

BTW, cmdlets exist with 2012R2 and 2016

 

 

From MS running PowerShell as admin

$MyPool = Get-SCOMResourcePool “UNIX/Linux Monitoring Resource Pool”
$SSHCredential = Get-SCXSSHCredential -UserName scom -ElevationType sudo
$WSCredential = Get-Credential scom

# Using MSAA account this fails

$DiscResult = Invoke-SCXDiscovery -Name “ubuntu.testlab.net” -ResourcePool $MyPool -WSManCredential $WSCredential -SSHCredential $SSHCredential

# Alternative Discovery for Network IP range
$DiscResult = Invoke-SCXDiscovery -IPRange 192.168.1.50,192.168.1.75  -ResourcePool $MyPool -WSManCredential $WSCredential -SSHCredential $SSHCredential

 

$DiscResult |fl -property *

 

$installResult = Install-SCXAgent -DiscoveryResult $DiscResult -Verbose
$installResult | fl -property *

 

 

Using MSAA account this fails

 

Using SCOM Agent Maintenance Account

 

Console verified

 

Resources
Old https://blogs.msdn.microsoft.com/scxplat/2009/12/11/cross-platform-powershell-scripts-released/
2012R2 (tested on my 2016 lab) https://operatingquadrant.com/2012/12/06/using-powershell-for-automated-unixlinux-agent-discovery/
PoSH cmdlet reference https://docs.microsoft.com/en-us/previous-versions/system-center/powershell/system-center-2012-r2/hh545212(v=sc.20)

 

 

 

 

Verify OMS Managed Computers

Ever wondered what objects are setup for OMS?

 

Maybe you’ve seen lots of errors on servers you don’t expect ?

It’s possible someone chose a group or nearly all managed computers in your SCOM environment.

 

How do we verify, or change what computers send data to OMS from SCOM?

1) Look for a group
In SCOM console, monitoring tab

Look for the ‘advisor’ group
Maybe someone put a group in there

2) Verify OMS members

In the SCOM console, Administration tab
Click on Managed Computers
See middle pane for what is currently set up

 

Update OMS Managed computers

In the SCOM console, Administration tab
Click on Managed Computers
See middle pane for what is currently set up

Click the ‘Add a computer/group’ link on the tasks pane (right side)

Add computers or groups

Add keyword, click search, highlight and click Add

Click OK when done updating members

 

Optionally, highlight the member, click delete

 

Verify the Advisor MP’s on computer

Go to server (added or removed)

If added, look for 1201 events in the Operations Manager Log

If removed, look for 1204 events in the Operations Manager Log

 

Enjoy!!