Best Practice Archives - Page 5 of 10

DNS2012R2 Addendum pack

Still running Server2012R2 servers with AD DCs with AD integrated DNS?

In case you’re still running Windows Server 2012R2, here’s the ‘DNS2012R2 Addendum pack’ giving the same functionality as the version agnostic 2016+ addendum. Why? DNS is a translation method to convert names to IP’s. Can you imagine if we wanted to connect to google via IP? The number of workflows in the SCOM DNS pack (built by the DNS Product Group) makes for an astounding number of workflows running on your DC every minute. Forward and reverse lookups are a good check, verifying DNS is functioning. In a complex environment with 100’s of zones, SCOM becomes a utilization culprit for a DC’s primary missions – authenticate and resolve. This article will help you understand how the pack will add new capabilities and tune DNS monitoring to best practice.

Quick Download HTTPS://GITHUB.COM/THEKEVINJUSTIN/DNSADDENDUM2012R2/

What capabilities does the ‘DNS Addendum pack’ provide?

Count logic monitors (i.e. x events in y time, and self heal)

Daily summary report of DNS alerts broken out

Daily alert closure workflow to close out DNS rules/monitor

DNS service(s) recovery automation

Synthetic internal/external nslookup monitor (scoped to PDC emulators versus ALL DNS servers

WMI validation alert recovery to prevent false positive alerts with weird one off scenarios – one example: Security tools randomly block WMI access.

Download the ‘DNS2012R2 Addendum pack’ on GitHub to improve AD Integrated (ADI) DNS monitoring on Windows Server 2016+ (version agnostic).

Save and Import pack, then update XML for group GUIDs

Update XML

First, update XML with the GUIDs from your management group. Second, map the group DisplayName to find/replace the GUID for each group.

Get-SCOMClassInstance output for DNS2012R2 groups

Third, using Notepad++ highlight the ContextInstance GUID and hit Control-H, and paste the group GUID then click Replace All.

Fourth – Rinse and repeat for the other three groups.

Lastly, save file, move to SCOM MS, and import!

Documentation and links

DNS Pack download

DNS2012R2 addendum blog including updates

GitHub Repository https://github.com/theKevinJustin/DNSAddendum2012R2/

DNS Addendum pack

nslookup to find out IP to name or name to IP resolution.

Simply put: Leverage the ‘DNS Addendum pack’. Why? DNS is a translation method to convert names to IP’s. Can you imagine if we wanted to connect to google via IP? The amount of workflows in the SCOM DNS pack (built by the DNS Product Group) makes for an astounding number of workflows running on your DC every minute. Forward and reverse lookups are a good check, verifying DNS is functioning. In a complex environment with 100’s of zones, SCOM becomes a utilization culprit for a DC’s primary missions – authenticate and resolve. This article will help you understand how the pack will add new capabilities and tune DNS monitoring to best practice.

QUICK DOWNLOAD(S)

2016+ https://github.com/theKevinJustin/DNSAddendumAgnostic

What capabilities does the ‘DNS Addendum pack’ provide?

Count logic monitors (i.e. x events in y time, and self heal)

Daily summary report of DNS alerts broken out

DNS service(s) recovery automation

Daily alert closure workflow to close out DNS rules/monitor

Synthetic internal/external nslookup monitor (scoped to PDC emulators versus ALL DNS servers

WMI validation alert recovery to prevent false positive alerts with weird one off scenarios – one example: Security tools randomly block WMI access.

Download the DNS Addendum on GitHub and the PDF install guide, to improve AD Integrated (ADI) DNS monitoring on Windows Server 2016+ (version agnostic).

XML authoring

The pack greatly decreases alerts, workflows on your AD integrated DNS servers, and the XML authoring is an easy feat. After you import the pack, find/replace is required for two pieces.

Group GUIDs update, after installing this pack.

Find/replace the GUIDs, as they are unique to every SCOM management group, hard coding the group ID GUID is not possible.

From PowerShell, on your SCOM management server, run these commands (after DNS Addendum installed)

Use get-scomclassinstance -DisplayName “GroupNameHere” | ft Id

Find/Replace the GUID in the pack with the ID from the output above.

Discovery group regular expressions (RegEx)

##DNSServerRegEx##

Find ##DNSServerRegEx## and replace with your DNS server expressions.

Example server names: 16dns01, 19dc01,16dns02,19dc02,19dc03, etc.

RegEx = (?i)16dns0|19dc0

DNS Group discovery example of RegEx for find/replace

Save and Import & Enjoy!

Update SCAP tools

DISA Security Content Automation Protocol

One more admin process and workflow is to ‘update SCAP tools’ on servers. Many times overlooked, this can save many headaches with the newest version installed on servers.

Check DOD Cyber Exchange

Check the website here, to search for Win in SCAP tools, then download & Install

SCAP tool download from DOD Cyber Exchange public website.

Navigation steps:

Control Panel > Programs > Programs and Features

In the search bar (top right) enter scap (and hit enter)

SCAP Control panel output showing multiple versions installed. Need to install latest application, then remove the old versions (in this case, all three!)

Install SCAP application

Extract files from ZIP

Copy folder to repository (my path example below)

Save SCAP zip and files to folder repository and on server to install SCAP on.

Run SCAP application

Take the defaults (unless you want the checker icon on desktop). Run SCAP application from PowerShell (as admin) window.

Open PowerShell as admin window

Example:

cd “D:\MonAdmin\STIGS\scc-5.7.2_Windows”; gci; .\SCC_5.7.2_Windows_Setup.exe

Hit enter to begin install

Run SCAP install from PowerShell (as admin) window.

On the SCAP EULA radio button application install screen, click ‘I accept’ radio button and click Next.

Select Destination location (preferably on non-system disk), and click Next

Change path to non-system disk (like d:)

SCAP Destination Location Application install window.

From the ‘Select Components’ window, click Next

SCAP Select Components application install window.

Click Next on the Setup Start Menu folder window

On the SCAP select additional tasks install window, click Next

Click Install on ‘Ready to install’ popup screen

With the new SCAP tool Install window, click Finish to complete.

SCAP tool install finished splash screen.

Refresh Control Panel SCAP search

Remove old versions

Click Continue and go through removal prompts

SCAP control panel remove old version with prompt to continue.

With the Uninstall screen, click Yes to uninstall.

Click OK on uninstall

Check Control Panel for SCAP installs

Verify control panel only has latest version installed. Close out Programs and Features window

Windows Control Panel, Programs and Features, SCAP search for new version install

Review SCC (SCAP Compliance Checker) Release Notes

Verify SCAP application functionality

Click on Start > start typing SCAP > Click on SCAP Compliance Checker

From the SCAP checker UAC prompt, click Yes to continue

Click OK to end the install

Run Local Scan

Run local scan to prove functionality.

Select STIG(s) in the middle pane > Click Start Scan

Run SCAP scan against server, choose your STIGs and Start Scan

Verify SCAP tool modified files after installation

Recheck Windows Explorer for OpenSSL; look at file properties for version details. Interesting, NONE of these files have versions (openssl, x509 searches show nothing file version wise)

Ask the Security Admin to re-scan!

Documentation/Links

DOD Cyber Exchange https://public.cyber.mil/stigs/scap/

VMwareTools OpenSSL vulnerabilities

Extra Extra read all about it, VMwareTools OpenSSL vulnerabilities!

Update VMwareTools to solve OpenSSL vulnerabilities CVE-2023-3446, CVE-2023-2975. The ‘VMwareTools OpenSSL vulnerabilities’ showed up two (2) weeks ago, but it took about a week for the update to post. Latest Tenable scan article shows OpenSSL update to v3.0.10 required for VMware Tools.

Update VMwareTools

Start with the Security scan and the plugin ID to mitigate ‘Tenable Scan output of OpenSSL PlugIn ID documenting problems’

Talk with your security team to identify the offending path for guidance on which application might be the culprit. The diagnostic/debug details can be a lifesaver!

Snippet of Tenable OpenSSL path from scan diagnostic of OpenSSL vulnerabilities

Newer version of VMwareTools required to fix OpenSSL vulnerabilities.

Originally, no VMwareTools update posted

VmWare tools v12.6 resolves CVE-2023-3446, CVE-2023-2975. Hopefully your virtualization team uses an Endpoint Manager to manage server configurations, and they have an application/package wrapper to install VMwareTools without this being a manual process

Either way, you’ll have to download the update download link

VmWare tools v12.6 has OpenSSL update to resolve CVE-2023-3446, CVE-2023-2975

Follow VMware’s knowledge base (KB) ‘how to’ article ‘how to’ install VMWare Tools

I typically link this with the monthly patches, where a single reboot puts all the patches into a fresh boot (applying the configuration).

Documentation/Links

Tenable article OpenSSL 3.0.0 < 3.0.10 Multiple Vulnerabilities | Tenable®

VMware KB article ‘how to’ install VMWare Tools

Security Content Automation Protocol (SCAP) download

How to install OpenSSL in windows 10? – Stack Overflow

MSSQL Addendum pack

Time to tune MSSQL alerts!

The ‘MSSQL Addendum pack’ wouldn’t be possible without Brandon Pires contributions. Brandon dealt with my many questions to better alert! If you need more background, check the ‘why addendum pack’ post.

Quick Download(s)

2012+ https://github.com/theKevinJustin/MSSQLAddendum

Capabilities

The pack is based on the SQL engineering blog and program team making multiple updates per year for SQL monitoring. The addendum creates two groups for dev/test and notification/subscription modeling. Second, the overrides, man there are a bunch! aid consumption of real issues. Lastly, most environments should be SQL 2016+, as the 2012R2 EOL/EOSL is quickly approaching in October!

MSSQL groups defined in the Addendum pack

MSSQL group discoveries require updates to be applicable to environment

Tailor addendum

First, the Addendum pack requires the MSSQL packs MUST be installed. The addendum is based on the MSSQL 2016+ version agnostic is currently supported, as the 2012,2012R2 products are near end of support.

Find/Replace the variables as needed:

Example ##TESTSERVER##|##DEVSERVER##

Save file

Overrides

Addendum pack contains discovery, monitor, and rule overrides to tune MSSQL to CSA (old PFE/CE/CSAe Microsoft Field engineer recommendations), to match the health model reducing critical ‘wake me up in the middle of the night’ alerts.

Import

Download pack, and save to your environment

Import into SCOM

Enjoy!

MSSQL Addendum references

MSSQL Engineering blog and old post here

SQL Releases TechCommunity here

Engineering team latest management pack, TechCommunity release v7.2.0.0

Import ‘gotcha’ importing new custom functionality blog

ADFS Addendum pack

Do you associate StarTrek when the word federation is used inside of federation services (ADFS)?

To begin, the ‘ADFS addendum pack’ needs acknowledgement of the contributors who dealt with my many questions to better alert on AD issues! My thanks to Jason Windisch for his help and expertise with Active Directory Federation Services (ADFS). If you need more background, check the ‘why addendum pack’ post. BTW, what do you associate with the word – Federation?

Quick Download(s)

2016+ https://github.com/theKevinJustin/ADFSAddendum

Overview of capabilities

The Active Directory Federation Services ‘ADFS Addendum pack’ configures ADFS group of related classes for notification/subscription modeling. Second, the rules, service monitors, tasks, service recovery, alert cleanup, and summary reports aid consumption of real issues. Third, if you have ADFS2012R2, I have an addendum pack, but coordination necessary to get the ADFS management packs MSI (not currently available). Lastly, most environments should be 2016+, as the EOL/EOSL is quickly approaching in October!

ADFS Addendum pack creates ADFS Group AND discovery requiring server names applicable to environment.

ADFS Group discovery requires server names applicable to environment

Tailoring the pack(s) to your environment

First, the Active Directory Federation Services management packs MUST be installed for the ‘ADFS Addendum pack’ to load. 2016+ agnostic is currently supported, as the 2012,2012R2 products are near end of support.

Find/Replace the variables as needed

##ADFSSERVERNAME1##|##ADFSSERVERNAME1##|##LAB##

Save file

Workflows

First, the DataSources (DS) and WriteActions (WA) clean up alerts, create daily reports, where the WA are the on-demand tasks versions.

Data source (DS) scheduled workflows run weekdays between 0600-0700 local SCOM management server local time. The summary and team reports (run during this time) summarize key insights. NOTE: the Monday report gathers the last 72 hours, so administrators get a ‘what happened over the weekend’ view. Tuesday-Friday reports are past 24 hours. Lastly, the group policy report summarizing unique GPUpdate error output.

Monitoring

Addendum pack rules schedule data source execution, add on-demand tasks. The service monitor, and Recovery tasks add service recovery automation to bring us to the ‘manual intervention required’ alerting. There are a few monitor/rule overrides to match the health model.

Import

Download updated ‘ADFS addendum pack’ and save to your environment

Import into SCOM

Enjoy!

Documentation

ADFS 2016+ management pack download

ADCS Addendum packs

ADCS 'gift' certificate - don't we all wish! — ADCS ‘gift’ certificate – don’t we all wish!

If only certificates were all gift certificates! The ‘ADCS Addendum packs’ disables noisy rules, adds OCSP seed, OCSP responder and OCSP group (classes). Recovery and service monitoring and nCipher event are the main highlights reducing alerts for ADCS 2012,2012R2,2016+. My thanks to Bob Williams CSA, for the assist!

Quick Download(s)

2012 HTTPS://GITHUB.COM/THEKEVINJUSTIN/ADCS2012QAddendum

2012R2 HTTPS://GITHUB.COM/THEKEVINJUSTIN/ADCS2012R2ADDENDUM/

2016+ https://github.com/theKevinJustin/ADCS2016-Addendum

Overview of addendum capabilities

Remember the why addendum packs for guiding purpose, transform!

The ADCS Addendum packs discover OCSP (seed class), and OCSP responder registry keys installed on monitored servers.

Group discovery tailors OCSP classes, for subscription or alert tuning.

OCSP server group can be used for subscription, or alert tuning (depending on class targets)

Monitors and service recoveries keep OCSP services monitored, and only alert when manual intervention is required.

OCSP service, certsvc monitors and service recovery automations built in

Tailoring the pack(s) to your environment

First, you must have at least ONE (1) set of ADCS Active Directory Certificate Services management packs so the ‘ADCS Addendum pack’ will load. The three versions currently supported have addendums, hopefully 2012,2012R2 are planned to be decommissioned in the short term.

Second, if you don’t have OCSP in your environment, download, and then import into your environment –

ELSE

Update the ‘OCSP Responder’ server name(s) for the group regular expressions.

In your favorite XML editor (mine is Notepad++), open the addendum pack(s), and find/replace for the following strings:

CAServer##

CERTIFICATESERVERS##

Save pack

Import and enjoy!

Documentation

ADCS 2016+ version agnostic pack download

ADCS 2012/2012R2 management packs download

Why Addendum packs

‘Why addendum packs’? What value can they bring to my customer? Kevin Holman started the Addendum thought process quite a while back. Added functionality to a core application/program/product. The first example of this pack naming convention is his SQL RunAs Addendum to simplify SQL monitoring. Let’s break down a number of examples how the SCOM community has built packs to better monitoring, and how I believe the addendum packs bring IT Ninja lessons from Microsoft experts monitoring to your environment.

Why Addendum packs

Better monitoring from the experts, including customer examples for other ‘blind spots’ in monitoring. Blind spots consist of ‘not monitored’ pieces of infrastructure, from simply an event, ping, service, tcp port check, process, web site, scripted workflow, with the purpose to identify a problem.

The goal of monitoring is to:

Identify, self-heal, automatically run recovery or diagnostic workflows alert when manual intervention is required. Doesn’t matter what tool you use, they all do some portion of these steps.

The addendum packs do these things, adding a few differentiators.

Auto closure daily scripts (close rules/monitors)

Auto reports of problems (M-F 0600-0700 local, reflecting last 24-72 hours of open/closed alerts)

Employ count logic (x in y time)

Self-heal monitors with no new events

Adjust alert severities to health model

where critical (red) = outage, warning (yellow) = issue, informational reports or FYI’s

Capable of updating alerts (status, owner, ticketID+)

Tasks to run workflows on-demand

Recovery tasks – (i.e. service restart automation or TopProcess, Logical disk cleanup, MECM Client cache clean )

Integrate additional monitoring (like DFS replication queue script/alerts)

Synthetic checks for DNS and web applications

Web Availability and Transactional monitoring, ADFS, CRL, PowerShell Invoke-WebRequest, and more

Security and Compliance checks

Imagine I forgot something capability wise.

Stay tuned, as this builds into an even better outcome, quality data into ‘a single pane of glass’ of multiple tools within PowerBI.

AD Application monitoring

Data from StarTrek the next generation - Mr. Tricorder makes me laugh! — Data from StarTrek the next generation – Mr. Tricorder makes me laugh!

‘AD Application monitoring’ > web synthetics, artificial users > android what image comes to mind? Is it a person, or a thing from a Sci-Fi movie? Perhaps Bishop from Aliens, Data from Star Trek. What does ‘AD Application monitoring’ consist of? Currently that means a CRL validity check, and ADFS web synthetic (proving that ADFS is responding). My thanks to Jason Windisch CSA, for the supplied PowerShell!

Quick Download https://github.com/theKevinJustin/ADApplications/

Tailoring the pack to your environment

The purpose of the pack is to add scheduled workflow that acts like the user, identifies if the CRL’s are about to expire. Most times, monitoring stops at ICMP ping. Most times, there’s still an outage, as the network, and servers are responding. The next layer is IIS, Apache, etc. Sometimes the network team gets involved, checking a base IIS URL is configured. Most outages aren’t network, nor IIS wasn’t running. This is why we focus on the web application responding. Does the multi-prong tactical attack make sense?

This pack delivers on-demand tasks, daily reports, and rules/monitors to reflect health. Customize the watcher node, some URL’s, save, and import into SCOM! The purpose

Assign watcher node(s)

Assign a watcher node by creating a registry key.

What does that mean? Watcher nodes are needed to provide user perspective.

Multiple site example

Issue: Users from sites 1,2,3 are having problems accessing web pages. To understand a user in site 2, leverage a server in site 2 to initiate the web request (invoke-webRequest in PowerShell).

Why: Differentiate user experience (per site). Answer the ‘did you know’ – is the application responding from this site/perspective.

Unfortunately, the watcher node concept eludes most administrators. Mastering ‘user perspective’ makes for an invaluable aid moving from reactive ‘fire fighting’ to proactively being told before users. Hopefully this explains the power where monitoring imitates user interactions for key web applications.

How: Create registry key on whatever servers you want to initiate web monitor

From PowerShell (as Admin), or Command Prompt (as admin)

reg add “HKLM\SOFTWARE\ADApplications\WatcherNode”

AD Applications regedit registry key validation

Example of XML snippet from AD Applications management pack

AD Applications Watcher Node - create specific registry key — AD Applications Watcher Node – create specific registry key

Set up CRL Validity check and ADFS synthetic

Next, configure the URL’s for the customer environment for the ‘AD Application monitoring’ management pack.

Update AD Applications module types for monitor/rules for CRL and ADFS synthetics

Configure the CRL validity check array

From your favorite XML editor (notepad++ pictured)

Find/Replace ##FQDN##, ##CRLstring##, numbers to customer environment

CRL Validity check, create your array length as needed for customer environment

Configure the ADFS synthetic request(s)

From your favorite XML editor (notepad++ pictured)

Find/Replace $server, ##FederationFQDN##, if necessary, update ADFS URL string if different (the /adfs/ls/idpiniatedsignon.aspx portion) to customer environment

Update ADFS URL for invoke-webRequest, ADFS default URL in specified example

Save pack

Import and enjoy!

Documentation

URLGenie for advanced website monitoring

PowerShell invoke-webRequest

Addendum logic blog

Proactive Daily Reports

Proactive Analyst Reports as a new way to ingest key insights from SCOM

As a SME or team lead, ever need to know a key insight for the enclave? Let’s talk about the ‘Proactive Daily Reports’ pack. This provides you some built-in reports on what transpired in an enclave. Building again on the Health pillar, we can simplify what owners need to see. Creating a PowerShell script was a simpler alternative to a complex SSRS report that often broke due to patching, and not following best practices. The pack shows a simpler way to bring key insights to owners for Pending Reboots, Expiring PKI certificates, Logical Disk alerts, System Admin summary, and SCOM admin reports including long-running scripts, script errors, SCOM errors, and alert updates report.

Quick Download: https://github.com/theKevinJustin/ProactiveNOSCDailyTasks

Testing the Proactive Daily reports

Let’s start with some example reports – examples for expiring certificates, Logical Disk, Pending Reboot, System Admin summary, and SCOM admin reports including long-running scripts, script errors, SCOM errors, and alert updates report.

Expiring Certs –

About to expire certificates

Logical disk alerts –

Shows Server, drive, and % full data

Pending Reboots

Alerts of servers pending restart, not patched, not rebooted

Pending reboot report lists servers pending restart, not patched, not rebooted alerts

System Admin summary

This is really a consolidation of multiple insights:

Server performance issues
Open ITSM/Remedy tickets
Unhealthy Agents
Pending Reboot, Not Rebooted, Not patched
Disabled/Unhealthy/MaintenanceMode, Repeatedly down Agents
Logical Disk free space alerts
Expiring certificates
AD DC (ADDS) critical alerts
DNS alerts
Group Policy issues

SysAdmin daily summary report example alert

SCOM admin reports

Admin reports have a few separate alert reports, including long-running scripts, script errors, SCOM errors, and alert updates report.

SCOM Admin alerts report example of common SCOM problems

Long running scripts

ScriptErrors showing key SCOM connectivity issues

SCOM Admin script errors to help diagnose report script syntax errors

Useful links

Other blog posts for addendum management packs and capabilities –

https://kevinjustin.com/blog/2023/08/15/proactive-patching-alerts/
https://kevinjustin.com/blog/2023/08/14/top-process-powershell-script/
https://kevinjustin.com/blog/2023/08/15/proactive-daily-reports/

https://kevinjustin.com/blog/2023/08/08/create-closed-alerts-view/