Data Integration with SQL2022

Integration - time to integrate data sources to data lake
Integration – time to integrate data sources to data lake

Ready for a single pane of glass?  Ready to have your insights in a common location?  Let’s discuss Data Integration with SQL2022.

 

Let’s start with some background on SQL2022 and similarly SQL2025, start with the learn site link.  SQL2022 by design is Azure enabled with multiple capabilities like ‘Bi-directional HA/DR to Azure SQL’ and ‘Azure Synapse Link’.  Basically, Synapse link is the key.

SQL2022 by design is Azure Enabled
SQL2022 by design is Azure Enabled

Utilize the PowerBI Cloud Service with today’s hybrid environments.  SQL2022 allows integration with other Azure capabilities like Azure Data factory/data lake, and Azure Synapse.  Another reason to upgrade SQL 2022, is design simplification.  However, PowerBI data gateway adds a potential break point (single point of failure).  While PowerBI data gateway centralizes all premise data to a central location.  In the same way, consolidating data sent to the cloud.  When PowerBI data gateway fails, insights and visualizations have stale data (i.e. data NOT transferred for a near real-time display).

 

Why SQL2022 then?

Connect insights and visualization to justify ‘Data Integration with SQL2022’ scenarios.

SQL2022 built in capability to Azure Synapse Analytics
SQL2022 built in capability to Azure Synapse Analytics

Use SQL2022 to configure SQL agent jobs which pull SQL scripts from your cloud environment.  DevOps and common Azure Storage repository are great advantages for speed of execution.

Configure MMA agent via PowerShell

A car mechanic uses battery jumper cables to charge a dead battery.

 

Do you feel like a mechanic having to jump start the agent configuration like a dead car battery?   Assuming the Agent is already installed, you can configure the SCOM agent via PowerShell.  Even better when you can PowerShell remote to multiple systems.  I hope the PowerShell commands below help you master PowerShell to configure the SCOM side of the MMA agent (house).

 

powershell

/*
# Find/replace variables to your environment like Kevin Holman’s fragments!
##SCOMMGMTGROUP1##
##SCOMMGMTGROUP2##
##SCOMMGMTSERVER1##
##SCOMMGMTSERVER2##
#
*/

$SCOMAgent = New-Object -ComObject AgentConfigManager.MgmtSvcCfg
$SCOMAgent.GetManagementGroup(“##SCOMMGMTGROUP1##”);$SCOMAgent.GetManagementGroup(“##SCOMMGMTGROUP2##”)

# If mgmt groups are incorrectly set
$SCOMAgent.RemoveManagementGroup(“##SCOMMGMTGROUP1##”)
$SCOMAgent.RemoveManagementGroup(“##SCOMMGMTGROUP2##”)

restart-service healthservice

# Domain
$SCOMAgent.AddManagementGroup(“##SCOMMGMTGROUP1##”,”##SCOMMGMTSERVER1##”,5723)

# Verify agent config
$SCOMAgent.GetManagementGroup(“##SCOMMGMTGROUP1##”)
# If you have a second management group

$SCOMAgent.GetManagementGroup(“##SCOMMGMTGROUP2##”)

# Restart and test connectivity
restart-service healthservice

# Check connectivity
test-netconnection -port 5723 -computername ##SCOMMGMTSERVER1##

 

 

WebConsole APM hotfix for SCOM2012R2 and above

Vaccination Record - SCOM hotfix released for WebConsole/APM on SCOM2012R2 and above
Vaccination Record

SCOM hotfix released for WebConsole/APM on SCOM2012R2 and above, time for another SCOM shot!  Don’t forget your vaccination card 🙂

 

Let’s get started.  Time to fix the vulnerability for ‘SCOM hotfix released for WebConsole/APM on SCOM2012R2 and above’.  Read the support article, and assess what versions you have in your sandbox and production.  Once assessed, it’s time to test/implement/verify the fix applied.

 

Support article

https://support.microsoft.com/en-us/topic/update-for-idor-vulnerability-in-system-center-operations-manager-kb5006871-0e3a513a-ad80-4830-8984-2fc5a40ee7f7

 

 

SCOM WebConsole Hotfix links

(support.microsoft.com articles)

Specific support article for SCOM2019 UR3 Hotfix

SCOM2019 UR3 Hotfix support.microsoft.com article link

Specific support article for SCOM2016 UR10 Hotfix

SCOM2016 UR10 Hotfix support.microsoft.com article link

Specific support article for SCOM2012R2 UR14 Hotfix

SCOM2016 UR10 Hotfix support.microsoft.com article link

 

# Download (same EXE has all 3 SCOM versions)

https://download.microsoft.com/download/3/e/e/3eec1274-64d5-4285-84b9-c50800eb2dd2/KB5006871.EXE

 

 

Hotfix updates two paths on SCOM management server with the WebConsole role

Paths updated

(don’t forget to add File Version property to your display)

NOTE Drive letter depends on where you installed SCOM (typically D:)

 

SCOM2019 paths

D:\Program Files\Microsoft System Center\Operations Manager\WebConsole\AppDiagnostics\Web\bin

D:\Program Files\Microsoft System Center\Operations Manager\WebConsole\AppDiagnostics\AppAdvisor\Web\Bin

SCOM2016 paths

D:\Program Files\Microsoft System Center 2016\Operations Manager\WebConsole\AppDiagnostics\Web\bin

D:\Program Files\Microsoft System Center 2016\Operations Manager\WebConsole\AppDiagnostics\AppAdvisor\Web\Bin

 

Screenshot of paths

AppDiagnostics File Path - SCOM hotfix released for WebConsole/APM on SCOM2012R2 and above
AppDiagnostics File Path
AppDiagnostics AppAdvisor File Path - SCOM hotfix released for WebConsole/APM on SCOM2012R2 and above
AppDiagnostics AppAdvisor File Path

 

Just in case you forgot how to add properties in Windows Explorer…

In the columns (Name, Date modified, etc,) right click > More

Add file property - SCOM hotfix released for WebConsole/APM on SCOM2012R2 and above
Add file property

Hit F to move down to the F named details > hit check box for ‘File Version’ or click on File Version and hit space bar

Click on OK

Add file property File Version - SCOM hotfix released for WebConsole/APM on SCOM2012R2 and above
Add file property File Version

 

Sort by ‘Date Modified’ Column

Verify File Version - SCOM hotfix released for WebConsole/APM on SCOM2012R2 and above
Verify File Version

 

File versions AFTER installing hotfix

Depending on which SCOM version you’re running, the path stays pretty much the same, and you want to verify that files were updated for the ‘SCOM hotfix released for WebConsole/APM’

SCOM2019

UR3 = 10.19.10505.0 > Hotfix file version = 10.19.10550.0

SCOM2016

UR10 = 7.2.12324 > Hotfix file version = 7.2.12335.0

Standard UR10 files are 8.0.10918.0

 

Voila > SCOM hotfix complete

Notify your Security team you’ve patched, because sometimes the scanner software isn’t accurately updated (where Security needs to open a case with their vendor!)

 

Complete:  Patched environment for ‘SCOM WebConsole/APM on SCOM2012R2 and above’

Mining Windows Event Log

Mining Ore from the Windows Event Log and finding a way to make it portable

 

Use Get-WinEvent to use XML and filters from event viewer, to mine an event, including examples for a specific string, from a specific event, in a specific event log?

 

 

Hopefully this post will help with a few tips to simplify monitoring for events, whether in AzMon, SCOM, or via PowerShell.

 

 

Let’s start with the Dr Scripto blog post from quite a while ago –

https://devblogs.microsoft.com/scripting/data-mine-the-windows-event-log-by-using-powershell-and-xml/

 

Not sure how many people use get-WinEvent, but this is one tool in PowerShell that can help an admin parse the XML side of an event.

 

Example 1

Query Application Event Log for Severity, Event, and Event Data contains lync.exe

$query = @”

<QueryList>

  <Query Id=”0″ Path=”Application”>

    <Select Path=”Application”>*[System[Provider[@Name=’Application Hang’]

    and (Level=2) and (EventID=1002)]]

    and *[EventData[Data=’lync.exe’]]</Select>

  </Query>

</QueryList>

“@

Get-WinEvent -FilterXml $query

 

PowerShell output

Use Get-WinEvent to use XML and filters from event viewer
Lync.exe event example output

 

 

 

Use Get-WinEvent to use XML and filters from event viewer

The Tip or Trick part of this – leverage your Event Viewer Filter as a query to use with get-WinEvent

Credit for this tip comes from Andrew Blumhardt!

See below for examples to ‘use Get-WinEvent to use XML and filters from event viewer’

 

Navigating via Event Viewer:

Hop onto your favorite server, or connect to another server via Event Viewer

Go to the Event Log > Click Filter Current Log

Build out your filter (i.e. choose specific Event Sources, exclude events, include severities, timeframe (start/end), etc.)

Use Get-WinEvent to use XML and filters from event viewer
SCVMM Application Log Event ID 25933

Switch to the XML tab (and note you can edit your query further!)

SCVMM query example screenshot
Event Viewer filter XML tab

You can copy the query from the Event Viewer into your Get-WinEvent syntax

$query = @”

<QueryList>
<Query Id=”0″ Path=”Application”>
<Select Path=”Application”>*[System[Provider[@Name=’Microsoft.SystemCenter.VirtualMachineManager.2012.Monitor.UserRoleQuotaUsageMonitor’ or @Name=’Microsoft.SystemCenter.VirtualMachineManager.2012.Report.ServiceUsageCollection’ or @Name=’Microsoft.SystemCenter.VirtualMachineManager.2012.Report.VMUsageCollection’ or @Name=’Microsoft.SystemCenter.VirtualMachineManager.2016.EnableCredSSPClient’ or @Name=’Microsoft.SystemCenter.VirtualMachineManager.2016.Monitor.UserRoleQuotaUsageMonitor’ or @Name=’Microsoft.SystemCenter.VirtualMachineManager.2016.Report.ServiceUsageCollection’ or @Name=’Microsoft.SystemCenter.VirtualMachineManager.2016.Report.VMUsageCollection’] and (Level=2 or Level=3) and (EventID=25933)]]</Select>
</Query>
</QueryList>

“@

Get-WinEvent -FilterXml $query

 

PowerShell output

Use Get-WinEvent to use XML and filters from event viewer
SCVMM query example screenshot

 

 

 

 

Example 3

Grab System Event Log, Event ID 5827  (NetLogon denied events)

get-WinEvent -FilterHashtable @{LogName=’System’; ID=’5827′;}

 

PowerShell output

Use Get-WinEvent to use XML and filters from event viewer
get-WinEvent filter by logname and event ID

 

 

Documentation:

Get-WinEvent https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.diagnostics/get-winevent?view=powershell-7.1

MSFT DevBlogs https://devblogs.microsoft.com/scripting/data-mine-the-windows-event-log-by-using-powershell-and-xml/

Troubleshooting Service Map pack

 

 

 

Updated 14 Mar 2019

 

If you get these exceptions like me, the issue has been raised, with a deliverable targeted for SCOM2019UR1.

Disable the rule to reduce noise.

 

 

Are you using Service Map Management pack, and getting errors?

 

This alert is based on the 46651/46652 event ID in the Operations Manager event log

From SCOM Console > Authoring Tab > Management Pack Objects > Rules

Search in ‘Look for:’ bar GenericException (yes no space in between)

 

Rule

 

 

Rule Details

 

To enable debug on the MS

 

For collecting logs, please do the following:

  • Create folders “c:\Debug\ext\”
  • Now, Wait for an hour(which is the default time interval set in the rule for running service map api).
  • You will see some log files created in that folder “ext”. Please share the same in email.

 

The file showed up after the alerts, and listed debug INFO and WARN lines, and the time stamps match up to the generic exception rules.

 

Stay tuned for more information, I have been trying to get more answers on the exception

{WARN} [12:35:20.966] [ScomUtils] failed to export XML for Management Pack: System.NullReferenceException: Object reference not set to an instance of an object.

   at ScomBridge.ScomUtils.WritePackXmlToFile(ManagementPack pack, String filename)

 

 

Gather Log Analytics/MMA agent version

Had some questions come up from the community to check the Log Analytics agent version.

Depending on how you are setup, the SCOM Integration makes this easy with Holman’s blog for the agent management pack.

If you have admin right in Operations Manager console then you can check this directly from SCOM server:

If you are an admin in SCOM, you can check from MS

$Server = “DC01.yourlabnamehere.net”
(Get-SCOMAgent -Name $ServerName).Version

Alternatively, from server registry:

(Get-ItemProperty “HKLM:\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\setup”)

# Just the Agent version variable

(Get-ItemProperty “HKLM:\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\setup”).AgentVersion

Log Analytics

Kusto query

// Servers and Versions

Heartbeat
| project Computer,Version

// Specific version

Heartbeat
| where Version == “8.0.10918.0”
| project Computer,Version

// Summarize by Version

Heartbeat
| summarize by Version

If you’re visual

From the Portal > Log Analytics > workspace > Workspace Summary > Agent Health

Scroll right to agent version

Monitor

Monitor > Overview > Agent Health Assessment

Scroll right to agent version

Azure Log Analytics for Windows Telemetry data

 

 

I blogged about this last year here

 

 

As best practice, the Upgrade Analytics script checks for far more than just injecting the workspace key and telemetry value.

 

 

FYI – This could also be managed in an SCCM Compliance setting.

Paul Fitzgerald – Platform PFE blogged about a non SCCM method here

 

 

Assess requirements for environment:

 

Barebones configuration requires Commercial ID, allow telemetry, and level of telemetry data to send

Optional – Create key for IEDataOptIn

Send data to Application Insights

Customer proxy setup

 

 

Script has 11 parameters specified, not all are needed (excerpt below from script)

Param(
# run mode (Deployment or Pilot)
[Parameter(Mandatory=$true, Position=1)]
[string]$runMode,

# File share to store logs
[Parameter(Mandatory=$true, Position=2)]
[string]$logPath,

# Commercial ID provided to you
[Parameter(Mandatory=$true, Position=3)]
[string]$commercialIDValue,

# logMode == 0 log to console only
# logMode == 1 log to file and console
# logMode == 2 log to file only
[Parameter(Mandatory=$true, Position=4)]
[string]$logMode,

#To enable IE data, set AllowIEData=IEDataOptIn and set IEOptInLevel
[Parameter(Position=5)]
[string]$AllowIEData,

#IEOptInLevel = 0 Internet Explorer data collection is disabled
#IEOptInLevel = 1 Data collection is enabled for sites in the Local intranet + Trusted sites + Machine local zones
#IEOptInLevel = 2 Data collection is enabled for sites in the Internet + Restricted sites zones
#IEOptInLevel = 3 Data collection is enabled for all sites
[Parameter(Position=6)]
[string]$IEOptInLevel,

[Parameter(Position=7)]
[string]$AppInsightsOptIn,

[Parameter(Position=8)]
[string]$NoOfAppraiserRetries = 30,

[Parameter(Position=9)]
[string]$ClientProxy = “Direct”,

[Parameter(Position=10)]
[int]$HKCUProxyEnable,

[Parameter(Position=11)]
[string]$HKCUProxyServer

 

 

 

Simple method to update machines to send Windows telemetry data:

 

 

PowerShell script

From PowerShell as Administrator

Set-Location HKLM:

 

$registryPath = “HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies”

$Name = “DataCollection”

$Name2 = “AllowTelemetry”

$CommercialID = “00000000-0000-0000-0000-000000000000”

$value = “2”  # Values from 0-3 accepted

$vIEDataOptInPath = “HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\DataCollection”

$IEOptInLevel = “2”  # Values from 0-3 accepted

 

If ( (Test-Path $registryPath\$Name) ) { write-host -f green “Registry keys already exist” }

If ( ! (Test-Path $registryPath\$Name) )

{

New-ItemProperty -Path $registryPath -Name $name

New-ItemProperty -Path $registryPath -Name $CommercialID

New-ItemProperty -Path $vIEDataOptInPath -Name IEDataOptIn -Type DWord -Value $IEOptInLevel

New-ItemProperty -Path $registryPath\$Name -Name $name2 -Value $value `

    -PropertyType DWORD -Force | Out-Null

Write-host -f green “Registry keys added for Telemetry”

}

 

 

 

 

References

Configure telemetry

Get Started link

Win 7,8 Opt in link

Test fire any event on any server from any application

Golden Oldies – always popular (tools vs music)

Old Holman blog that’s still relevant, even more powerful than EventLog Explorer

Basically anyone who wants to test fire events off a SCOM MP should use this tool.

Event Create, write-eventlog all have limitations (certain event sources that can be used to create events, or event ID number limitations)

First, download the 2007 R2 Admin ResKit here

MomTeam blog reference

Double click the downloaded MSI

I prefer to move extracted files under my SCOM tools/Management pack directory structure under MonAdmin (Monitoring Admin)

Copy extracted files to gold depot

Move to gold depot – SCOM \ tools \ <toolname here>

Go into the MPEventAnalyzer directory

Run the exe

MP Event Analyzer

Click on Investigate Event Sources Tab (bottom middle)

Don’t forget you can use the search bar (where I typed apm)

For my example, double click on APM Agent

Search Events on right hand pane

Check checkbox to select the 1319 APM event for configuration error (right hand pane)

Click the ‘Add selected events to execution list’

Once event verified in bottom box, click the green box to fire selected event(s)

Verify event in Event Viewer

Validate Management Pack

Stay tuned… this did not complete the validation process.

Adding Management Solutions in Azure

Decoder ring applies!

 

OMS is Log Analytics is Azure Management Solutions.

 

 

 

Do you want to add solutions to your Azure subscription?

Pre-packaged visuals and insights on your data, whether azure or hybrid.

 

 

 

Adding Management Solutions

Login to the Azure Portal

Click on All Services

Type ‘solutions’, hit enter

Click star icon to favorite Solutions

 

 

Drag Solutions higher in your preferences (wasn’t in above screenshot)

 

 

Click Solutions

 

 

 

 

Click + to Add

Click on Security and Compliance

 

 

Click Create

 

 

Don’t forget solutions require MMA agents connected to a workspace to render any data/insights!

 

 

 

 

References

The Docs article lists how to use the management solutions

 

MMA Agent and SCOM Agent version numbers


 

FYI – Updated 24 June 2022

 

What are the MMA Agent and SCOM Agent version numbers?

This idea sprung from a discussion with Sr. PFE Brian Barrington, and it got me wondering…See below for more details on OMS/MMA, and SCOM agent versions, as well as how to verify agent from PowerShell.

 

 

FYI – If you’re running a SCOM agent, 2016 or above, various Log Analytics solutions may have pre-reqs.

The Content Dev team under Brian Wren added this to the docs.microsoft.com site

SCOM 2022     https://docs.microsoft.com/en-us/system-center/scom/release-build-versions?view=sc-om-2022

SCOM 2019     https://docs.microsoft.com/en-us/system-center/scom/release-build-versions?view=sc-om-2019

SCOM 2016      https://docs.microsoft.com/en-us/system-center/scom/release-build-versions?view=sc-om-2016

 

 

Azure Monitor Agent

AMA (Azure Monitor Agent)/ALA/OMS/MMA Agent can run on Windows/Linux operations systems.  Name has changed over the years, where AMA (Azure Monitor Agent) will be the name going forward for the cloud based offer.  See docs article here.

This also has been updated on the Docs site

https://docs.microsoft.com/en-us/azure/azure-monitor/agents/azure-monitor-agent-extension-versions

Download installer files here

Review what operating systems are covered here

Previously known as Windows OMS/ALA/MMA agent

Unfortunately, there’s no github repo that I’ve found.

Examples:

As of 6 Sep 2018, MMA agent = 8.0.11103.0

As of 17 Oct 2018, MMA agent = 8.0.11136.0

Skipping forward to 2020, the MMA agent is 10.20.18040.0

[!WARNING] The Log Analytics agents are on a deprecation path and will no longer be supported after August 31, 2024.

 

 

OMS Gateway

Older product published in 2016 – Download link here

OMS Gateway requires Microsoft Monitoring Agent (MMA)

(agent version – 8.0.10900.0 or later)

Simple English, that means SCOM2016 RTM agent or above

 

 

 

OMSAgent for xPlat

OMS-Agent-for-Linux

(Linux/Universal Linux)

Sep 16, 2021      OMSAgent_v1.13.40-0
Mar 08, 2021      OMSAgent_v1.13.35-0
Nov 16, 2020      OMSAgent_v1.13.33-0
Support for Red Hat Enterprise Linux 8, CentOS 8, Oracle 8, Ubuntu 20.04, SLES…
Nov 14, 2019       OMSAgent_v1.12.15-0
Jun 17, 2019      OMSAgent_v1.11.0-9
Apr 23, 2019      OMSAgent_v1.10.0-1
Feb 12, 2019      OMSAgent_v1.9.0-0
Nov 05, 2018     OMSAgent_v1.8.1.256
Oct 30, 2018      OMSAgent_1.8.0-256
Sep 03, 2018      OMSAgent_v1.6.1.3

 

 

Windows SCOM Agent Version numbers 

SCOM2016 

Build NumberKBRelease DateDescriptionStep-by-Step
8.0.10918.0EvaluateOct 2016SCOM 2016 RTMLink
8.0.10931.0KB3190029Feb 2017SCOM 2016 Update Rollup 1Link
8.0.10949.0KB3209591March 2017SCOM 2016 Update Rollup 2Link
8.0.10970.0KB4016126May 2017SCOM 2016 Update Rollup 3Link
8.0.10977.0KB4024941Oct 2017SCOM 2016 Update Rollup 4Link
8.0.10990.0KB4090987April 2018SCOM 2016 Update Rollup 5None
8.0.11004.0KB4459897Oct 2018SCOM 2016 Update Rollup 6Link
8.0.11025.0KB4492182April 2019SCOM 2016 Update Rollup 7Link
8.0.11037.0KB4514877Sept 2019SCOM 2016 Update Rollup 8Link
8.0.11049.0KB4546986April 2020SCOM 2016 Update Rollup 9Link
8.0.11000.0KB4580254Dec 2020SCOM 2016 Update Rollup 10Link
7.2.12335.0KB5006871Oct 2021SCOM 2016 Update Rollup 10 HotfixLink

SCOM1801
8.0.13053.0 RTM

SCOM1807

8.0.13067.0      General Availability release

 

SCOM2019

Build NumberKBRelease DateDescriptionStep-by-Step
10.19.10050.0EvaluateMarch 2019SCOM 2019 RTMLink
10.19.10311.0KB4533415Feb 2020SCOM 2019 Update Rollup 1Link
10.19.10407.0KB4558752Sept 2020SCOM 2019 Update Rollup 2Link
10.19.10505.0KB4594078March 2021SCOM 2019 UR3Link
10.19.10550.0KB5006871Oct 2021SCOM 2019 UR3 HotfixLink

 

 

  • @Larry LeBlanc – thank you for the SCOM Agent version updates!

 

Verify what version is installed

Via SCOM – use Holman’s Agent Version Addendum management pack

 

If you don’t have SCOM

From PowerShell

$Agent = get-itemproperty -path “HKLM:\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Setup”

$Agent.CurrentVersion

 

 

 

 

 

 

Resources

SCOM Agent Version Addendum pack https://kevinholman.com/2017/02/26/scom-agent-version-addendum-management-pack/

SCOM Agent build numbers https://social.technet.microsoft.com/wiki/contents/articles/34312.system-center-operation-manager-momscom-list-of-build-numbers.aspx

Linux Agent can be downloaded from GitHub – github.com/Microsoft/OMS-Agent-for-Linux