Exchange Message Queue Monitoring

Q ?

Hopefully something else popped into your mind, not Star Trek – The Next Generation

 

 

How about monitoring a message queue?

 

 

This will help you monitor specifically Exchange 2013-2016 queue changes (state or message count beyond a value)

 

Just specify the queue and encapsulate into a monitor

OR… download the MP Fragment here

############################################################

# Test Queues

#Lab environment

#$queue1 = “Submission”
#$queue2 = “Submission2”

$queue1 = “CompanyName365-mail-onmicrosoft-com.mail.protection.outlook.com”
              $queue2 = “clustered.testlab.net”

##################################################################
# 1. How to monitor the status of the queues below (when the status is “retry” they need alert)

$SPEvent = ( get-queue | where { $_.NextHopDomain -like “$queue1” } | where { $_.Status -like “Retry” } | measure-object )
$SPEvent2 = ( get-queue | where { $_.NextHopDomain -like “$queue2” } | where { $_.Status -like “Retry” } | measure-object )

#
# 2. To get an alert when a message in any of these queues is stuck for > 5minutes
#

$SPEventStuck = ( get-queue | where { $_.NextHopDomain -like $queue1 } | where { $_.Status -like “Suspended” } | measure-object )
$SPEventStuck2 = ( get-queue | where { $_.NextHopDomain -like $queue2 } | where { $_.Status -like “Suspended” } | measure-object )

# 3. Also need alert and exact count when the message count in the above queue goes beyond 1000

$SPEventMC = ( get-queue | where { $_.NextHopDomain -like $queue1 } | where { $_.MessageCount -gt 1000 } | measure-object )
$MCQ1 = ( get-queue | where { $_.NextHopDomain -like $queue1 } )
$MQ1 = ($MCQ1.MessageCount)

$SPEventMC2 = ( get-queue | where { $_.NextHopDomain -like $queue2 } | where { $_.MessageCount -gt 1000 } | measure-object )
$MCQ2 = ( get-queue | where { $_.NextHopDomain -like $queue2 } )
$MQ2 = ($MCQ2.MessageCount)

 

 

Have fun!

Exchange 2013 monitoring Addendum

This is a good source for additional Exchange Server 2013+ monitoring, brought to my attention from Exchange PFE Dave Groll!

 

See Volkan Coskun’s blog post here

Update 30 Nov 2018

NOTE this content is no longer available – posted Volkun’s custom extension pack on TechNet Gallery here

 

 

This will help provide additional visibility to individual objects like Mailbox Databases or Transport Queues that are otherwise hidden in the health sets.

NOTE my Lab environment is Exchange 2016

PowerShell Monitor Fragment with Run As

Stop!

 

Ever need to run a PowerShell command (or script) as a specific ID?

Maybe you need to know when the command fails to catch degraded application health?

 

Let’s work into the scenario with a MP Fragment

 

Download the latest fragments here

Load Test MP fragment

Shout to Tyson Paul for his initial MP with 2016, catch his blog here!

 

This should help speed up building the MP if you have Visual Studio 2013 or 2015 with Visual Studio Authoring Extensions (VSAE).  Read Kevin Holman’s blog if this is new

 

Download the latest fragments here

 

Quicker method To build the MP as MP fragment, update the following:

  1. Import MP Fragment into Visual Studio for MP
  2. Replace variables in fragment
    1.   ##CompanyID##
    2.   ##AppName## – LoadTesting
    3.   ##ClassID## – WindowsServer
    4.   ##OSVersion## – 2012 = 6.2, 2012R2 = 2012.R2, 2016 = 100
    5.   ##MPReferenceID## – MWS2M for 2012, MWS2RM for 2012R2
    6.   ##RuleGUID##
  3. Obtain Rule names for Override Targets and Report rule GUID’s for the report parameters section

          get-scomrule | ? { $_.DisplayName -like “System Processor Queue Length*” } | fl ID,DisplayName,Name

          get-scomrule | ? { $_.DisplayName -like “Current Disk Queue Length*” } | fl ID,DisplayName,Name

          get-scomrule | ? { $_.DisplayName -like “Current Disk Queue*” } | fl ID,DisplayName,Name

          get-scomrule | ? { $_.DisplayName -like “Current Dis*k Queue*” } | fl ID,DisplayName,Name

          get-scomrule | ? { $_.DisplayName -like “*Current Disk Queue*” } | fl ID,DisplayName,Name

          get-scomrule | ? { $_.DisplayName -like “*Average Disk Seconds Per Transfer*” } | fl ID,DisplayName,Name

          get-scomrule | ? { $_.DisplayName -like “*Logical Disk Idle Time*” } | fl ID,DisplayName,Name

          get-scomrule | ? { $_.DisplayName -like “*Processor Time Total*” } | fl ID,DisplayName,Name

          get-scomrule | ? { $_.DisplayName -like “*Memory Available Megabytes*” } | fl ID,DisplayName,Name

          get-scomrule | ? { $_.DisplayName -like “*Network Adapter Bytes Total*” } | fl ID,DisplayName,Name

          get-scomrule | ? { $_.DisplayName -like “*Memory Pages per Second*” } | fl ID,DisplayName,Name

          get-scomrule | ? { $_.DisplayName -like “*System Processor Queue Length*” } | fl ID,DisplayName,Name

  1. Save and Import MP into environment
  2. Get Report parameter value for group ID

          get-ScomGroup | ? { $_.DisplayName -like “*Load Testing Group*” } | fl ID,DisplayName

  1. Update ##TargetGroupID##, MP version
    1. Save MP, and import into environment
  2. From SCOM Console, Authoring Tab
    1. Update group with explicit members
    2. Verify Group members
  3. From SCOM Console, Monitoring Tab
    1. Verify Performance view has performance counters
  4. From SCOM Console Reporting Tab (this may take a few minutes to push report to Reporting server)
    1. Open report and run
    2. Export data for analysis

Have a lot of SCOM subscriptions?

You logged into SCOM because someone said they didn’t get a notification

Looked at subscriptions and it’s blank

 

Do you know if you backed up your subscriptions?

 

Hopefully, you read this and back up your management packs

Hope you read this blog, and it saved your life!

 

For those who love the Console UI

You can manually back up the MP

From the SCOM console

Click on the Administration Tab

Click on Management Packs (2012), or Installed Management Packs (2016)

Type ‘notification’ in the ‘Look for:’ bar

Hit Enter

Highlight the ‘Notifications Internal Library’ pack

Click on Export Management Pack

Name your path

Hit OK

 

For simple PowerShell

 

# Backup Management packs to C drive

# Set up your path, this example is monadmin\backup

$date = Get-Date -UFormat “%Y-%m-%d”

c:

cd monadmin\backup

new-item -itemtype directory -path c:\monadmin\backup\$date

cd $date

Get-SCOMManagementPack -Name *Notification* | Export-SCOMManagementPack -Path “C:\monadmin\backup\$date”

 

SQL Engineering Blog

hmmmm

Ever wonder when a SQL MP is published?

Wonder no longer, look for the SQL Engineering Blog!

 

New https://techcommunity.microsoft.com/t5/SQL-Server/bg-p/SQLServer/label-name/SQLReleases

Old – redirect in effect https://blogs.msdn.microsoft.com/sqlreleaseservices/

 

 

Verifying Custom MP overrides are valid when updating sealed MP’s

kidraisedhand

I will raise my hand when asked if I prefer Notepad++ for looking at XML (because I can shrink the sections I’m not concerned about).

 

Using Notepad++ (works best for color and concatenation of XML or quotes in case of syntax errors when editing)

 

Open Overrides management pack (XML)

Click on the (-) for Manifest

Click on the dash (-) for RelationshipTypes

Click on the dash (-) for each Discovery (if it exists)

simplifyingdiscoveryview

 

Verify targets exist in MP’s to be updated

Scroll to the right to view the Targets of your Override management pack

simplifyingview

 

If changes were overrides, look at the Monitor or Rule and verify this is in the pack to be updated

overridesmonitor

 

To understand which MP is being referenced, look at the example – Windows3!

Scroll to the top of your MP and click on the (+) plus sign to expand manifest

mpreference1

 

NOTE Windows3 is the server 2016 Monitoring MP

decodingmpreference

 

Verify your monitor/rule name still exists, and your Override should still apply

In Server Overrides MP, look at the Monitor= section for the Monitor name

verifyoverridemonitorhighlight

 

Go to the Windows Server 2008 Monitoring MP and look for that monitor

There is no monitor for 2008

notepadmonitortypes

 

Alternatively, you can look at the SCOM console as well (if MP is installed)

There is NO 2008 Memory Pages per second monitor

 scomconsolemonitorverify

 

Now to remove the override in our MP

In Notepad++, highlight the MonitorConfigurationOverride section, and delete

monitoroverridehighlight

Rinse and Repeat

Increment the version number and import MP when finished validating overrides.

 

 

Troubleshoot Office 365 SCOM MP Run As account

Run As Account

The Office 365 Run As account is used for Proxy access for an HTTPS connection from SCOM MS to Office 365 portal endpoint.

Must be a domain account, not an Azure account (particularly if they ‘re not the same tenant or AAD associated

Service Accounts are recommended to prevent impact should an employee leave

 

SCOM uses a domain account (example scom_action ID)

Verify that ID is in Azure tenant (contact your Azure Administrator if you don’t have access )

o365applicationazureidverify

To follow best practice, update the Run As account with the service account

o365applicationscomrunascredential

 

Verify Run As account

On SCOM console that there are no Operations Manager event log 7000 events for the ‘run as’ configured ID

Remote Desktop to SCOM MS Server

Verify if the ‘run as’ ID has a valid password

Look in the Operations Manager Event Log for Event ID 7000

Click on Find

Type in the user’s ID from the ‘run as’ account in SCOM

If no entries found, then ID is successfully authenticating against the domain

If errors found, correct ID/Password

Create a new subscription in SCOM to use the auto credentials option

NOTE New subscription may take 5-10 minutes to populate health data

From SCOM console

Click on Administration

Click on the Office 365 wizard

Click Add Subscription

o365applicationscomaddsubscription

 

Add Subscription Name

Click Next

o365applicationscomaddsubscriptionname

SCOM UI will prompt for Azure login

o365applicationscomazureauth

Enter ID and password

Click Sign in to authenticate

 

Click on Monitoring Tab

Click on Office 365 folder

Click on Office 365 Monitoring Dashboard

Verify state on the subscription in question

o365applicationscomnewsubscriptiondashboard

Verify SCOM ID used in O365 Subscription in Azure Portal

Verify SCOM ID used in O365 Subscription in Azure Portal

In Azure Portal

Verify the Application exists ( Azure tenant shows as SCOM O365MP )

o365applicationpermissions

NOTE In the right hand pane the Office 365 Management API’s has Application Permissions, and cannot be selected

o365application-requestpermissionsclean

Click Back to the Settings window

Click on Owners

o365applicationnoownersclean

NOTE NO owners show in this view

Click Add +

In the Add owner window, type the ID

Hit Select to add the user account (This example is the SCOM Service account)

o365application-scomidadded

Have user test