Administration Archives - Page 6 of 18

DHCP Addendum pack

Leverage DHCP addendum to tune DHCP subnet monitoring.

Leverage the ‘DHCP Addendum pack’. Why? DHCP manages IP ranges, particularly customer facing issues like VPN connectivity, VDI/AVD/appliance devices, as well as client workstations/laptops/GFE’s. The DHCP management pack alerts when a subnet is nearing zero available IP’s before you have an outage. This article will help you understand how the addendum’s new capabilities tune DNS monitoring to best practice.

QUICK DOWNLOAD(S)

2016+ HTTPS://GITHUB.COM/THEKEVINJUSTIN/DCHPAGNOSTIC

What capabilities does the ‘DHCP Addendum pack’ provide?

Two groups, one DHCP server group, and DHCP subscription group to configure notifications to SME for DHCP related classes

Overrides for common alerts, disable event collection rules

Utilize the DHCP Addendum

Download the DHCP Addendum on GitHub, to get alerts where manual intervention required.

Update XML

The pack greatly decreases alerts, and the XML authoring is an easy feat. After you import the pack, find/replace is required for two pieces.

Discovery group regular expressions (RegEx)

##DHCPServerRegEx##

Find ##DHCPServerRegEx## and replace with your DNS server expressions.

Example server names: 12dc01, 19dc01,19dc02,19dc03, etc.

RegEx = (?i)12dc0|19dc0

Using PowerShell on SCOM management server (MS) to determine group GUIDs for replace in Overrides/Discoveries.

Update group GUIDs, after installing this pack.

Find/replace the GUIDs, as they are unique to every SCOM management group, hard coding the group ID GUID is not possible. We will be running Get-SCOMClassInstance to determine the group GUID’s applicable in the management group.

From PowerShell, on your SCOM management server, run get-SCOMClassInstance commands for the two groups added.

get-scomclassinstance -DisplayName “Microsoft Windows DHCP 2016+ Servers” | ft Id

get-scomclassinstance -DisplayName “Microsoft Windows DHCP 2016+ subscription components” | ft Id

Example

Leveraging Notepad++ to find/replace the group GUID with SCOM environment specific GUIDs

Find/Replace the GUID in the pack with the ID from the output above.

Save pack

Import into SCOM & Enjoy!

OS Addendum packs

Download the ‘OS Addendum packs’ for new capabilities contains Event count logic monitor type, Disk cleanup, Group Policy, self-healing/reset monitors, as well as ‘eventLog full’ logic and reports. Additional monitors reduce alert noise. Examples of common alert scenarios are: StorPort storage errors, Group Policy 1096 identification and rebuild. Disk Cleanup & EventLog service recovery, which includes Event Log file expansion and rollover.

Quick DownloadS

https://github.com/theKevinJustin/2012OSAddendum

https://github.com/theKevinJustin/2016ServerAgnostic

Tune ‘OS Addendum packs’ as needed

Update logical disk paths and retentions. The default report contains quite a few common checks, including root folders broken out by path, highest to lowest GB’. The workflow is scalable to add additional application paths, as well as file retention timeframes. Workflow runs on a weekly basis to cleanup/archive log files, paths. See Disk cleanup logic blog for more details.

' OS Addendum packs' contains Logical disk breakdown of root folders to list paths were files stored, highest to lowest — ‘ OS Addendum packs’ contains Logical disk breakdown of root folders to list paths were files stored, highest to lowest

UpdateStorPortCountForRepeatedStorageErrors

StorPort storage errors typically cut lots of alerts with storage reads/writes. The ‘count’ monitors decrease the alerts, and the daily alert report consolidates the warning alerts (critical by default). If you’re seeing these alerts, the default should decrease overall alerts to near zero. Tune as needed for disk alerts, by updating MatchCount or TimerWait in Seconds (the x events in y time piece of the monitor logic)

Update StorPort Count for Repeated Storage read/write errors

Save file(s) and import

AD insight reports

Download the ‘AD insights pack’ for new capabilities to audit users, svc/MSA accounts, password last set, expiring, last login AD insights. Includes AD group audit alert capability.

Quick Download https://github.com/theKevinJustin/ADInsights/

AD audit

Time to provide key ‘AD insight reports’ into users and groups. Delve into different AD audit capabilities for users and groups. The pack also gathers DC Security events (rules), and lastly, on demand tasks for reports.

The question is what determines a problem?

Every domain admin has a different experience and perspective, whether cyber (hack) focused or not. Audit standards differ, from HIPAA, SOX, CCRI, STIG, etc.

Pack examples:

Users – service account naming conventions, password change frequency, expired date/time configured.

Groups – Choose your OU structure to audit WA in DA, SA in DA, WA in SA etc.

NOTE: Take caution on the OU group audit, to limit the output, as events have a size limitation

Configure ‘AD insight reports’

Now we can configure the user pack for applicable standards, like password age, last set, or AppOwners. The AppOwners is an array, so you can add whatever Application, system owners/teams in your organization. The password datasource (DS) rule runs weekly.

Configure the Password Time, last set, month, week and AppOwners to build out actionable svc/msa accounts failing audit artifacts.

Break out the regular expressions of whatever accounts each team uses, to tailor relevant data into the report alert. Find/Replace (Control-H) might be more effective, as the DS/WA repeat the logic for the on-demand task report, vs. the rule and monitor.

App Owner relevant service accounts by SamAccountName

Update patterns ID naming conventions

Tailor account names to environment to match ingested DC Security events.

Tailor the DC Security Events to account naming conventions.

Configure OU to environment

Configure OU structure to audit based on domain canonical names, groups, DC, etc.

Save file(s) and import

Disk cleanup logic

Logical disk cleanup, most times is harder vs. smarter manual intervention required, why not smarter vs. harder?

‘Disk cleanup logic’ traditionally follows manual intervention. Why would you want harder and manual? This article will present options to clean up system and non-system disks, by leveraging largest root folder, API’s and more. This is one step in the OS Addendum pack that needs explanation and can be tailored to applications where admins have regular manual cleanup actions.

Breakdown of Disk cleanup

We want to check system disks and non-system disks for different scenarios. Figure out Disk free space, user profiles, largest folder on root of disk, IIS cleanup, and MECM/SCCM client cache clear API. Second, utilize different behaviors depending on PowerShell version, application log(s) cleanup, and expand drive alerts when NO space after cleanup action.

Disk Free space

# Check Disk free space

#=====================

if ($Driveletter -eq “C” )

{

$CFreeSpace = gwmi win32_logicaldisk | ? { $_.DeviceID -eq “C:” }

$CFreeSpace.DeviceID

$CFreeSpace

$DeviceDriveLetter = $CFreeSpace.DeviceID

$DeviceDriveLetter

# Check folder size after cleanup

#==========================

$BeforeSize = (Get-ChildItem “$DeviceDriveLetter” -Recurse | Measure-Object -Property Length -Sum ).sum

$Before = [math]::Round($BeforeSize/1GB,2)

$DiskFreeSpace = [pscustomobject]@{

DeviceID = $DeviceDriveLetter

Size = [math]::Round($CFreeSpace.Size/1GB,2)

FreeSpace = [math]::Round($CFreeSpace.FreeSpace/1GB,2)

}

$DiskFreeSpace

$SoftwareDistribution = (gci C:\windows\SoftwareDistribution | measure length -s).sum / 1Mb

# Debug

$SoftwareDistribution

Check Software Distribution for ConfigMgr/SCCM/MECM client

Checking software distribution path was an item for discussion where the folder was larger than 3GB, stemming from customer and field engineers recommendations.

If ($DiskFreeSpace.FreeSpace -lt 15 )
{
# Audit Software Distribution
#==================================
If ( $SoftwareDistribution -lt “3000” )
{
Write-host “NO SME/SystemOwner/SysAdmin/Server Action required”
}

If ( $SoftwareDistribution -gt “3000” )
{
Write-host “SME/SystemOwner/SysAdmin/Server Action required, stopping Windows Update service, removing SoftwareDIstribution folder and restarting”
Get-Service -Name wuauserv | Stop-Service
Remove-Item -Path C:\Windows\SoftwareDistribution -Recurse
Get-Service -Name wuauserv | Start-Service
Write-host “Windows Update wuauserv service restarted after SoftwareDistribution directory removed”
}

Cleanup Application log folders

The nice part of this is you can reuse this by changing the path and deletion actions to tailor to customer environment. The script comes in handy for VEEAM, SQL, IIS instances and log directory on multiple drives.

# Cleanup IIS log files

#=====================

#if ( Test-Path C:\inetpub\logs\LogFiles\W3SVC1 )

## Years older than

#$HowOld = [DateTime]::Now.AddYears(-1)

#$RecentUse = [DateTime]::Now.AddDays(-90)

## Path to root folder

#$Path = “C:\inetpub\logs\LogFiles\W3SVC1\*.log”

## Deletion task

#get-childitem $Path -Recurse -Depth 1 -EA SilentlyContinue | where { $_.lastAccesstime -lt $RecentUse -and $_.CreationTime -lt $HowOld -and $_.LastWriteTime -lt $RecentUse } | remove-item -force -verbose

Sample report alert output

Documentation

CleanMgr https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/cleanmgr

Delete client cache the right way https://sccm-zone.com/deleting-the-sccm-cache-the-right-way-3c1de8dc4b48

MECM client cache cleanup PowerShell https://learn.microsoft.com/en-us/powershell/module/configurationmanager/invoke-cmclientaction?view=sccm-ps

SCCM Client Cache cleanup https://rzander.azurewebsites.net/sccm-config-item-to-cleanup-ccmcache/

Stack Overflow disk cleanup https://stackoverflow.com/questions/28852786/automate-process-of-disk-cleanup-cleanmgr-exe-without-user-intervention

PKI Addendum pack

The ‘PKI Addendum pack’ is a tricky pack, due to certificate hierarchy. The decisions included are part of the three pillars – health, Security, Compliance, as well as alerting WHEN manual intervention required.

QUICK DOWNLOAD https://github.com/theKevinJustin/PKIAddendum

The PKI pack provides discoveries of the server certificate stores to then analyze certificates for validity, chain, and expiration. The v1.4.3.0 release adds some task logic and script changes that helps discover more stores, trusted root, etc.

WHAT CAPABILITIES DOES THE ‘PKI ADDENDUM PACK’ PROVIDE?

Set timeframe for certificate per organizational standards.

Break out different expiration alerts based on internal/external certificate, or by AD Client Certificate enrollment templates (to build out the manual intervention required scenario when alerts are generated).

Create groups breaking out application self-signed, PKI certificates.

Separate RDP Auth, Domain Controller, Computer, and OCSP certificates.

If this sounds interesting, and you want to dabble in XML authoring…

Download the pack from GitHub to improve PKI monitoring on Windows Servers.

Additional screenshots of addendum components

Addendum pack creates multiple groups to break out various types of certificates that have different decisions/behaviors requiring unique timing

Groups

Discoveries

Leverage dynamic groups based on server name and EnhancedKeyUsageList (EKU) list

Overrides

Change PKI pack default discoveries, lifetime threshold expirations and more

DOCUMENTATION AND LINKS

Addendum requires the PKI Certificate MP release v1.4.3.0 download

Bob’s TopQuore blog

SCOM agent application

Stop using the SCOM console to push agents. Move this to Endpoint Manager as application/package and task sequence to install and configure on every server built.

Shout out to Neal Smith, for his help simplifying the ‘SCOM agent application’ install per ConfigMgr/MECM best practice! Stop using the SCOM console to push agents. Move this to Endpoint Manager as application/package and task sequence to install and configure on every server built. Need a MECM package (Application) for the SCOM agent. Leverage a best practice, be more secure, include in task sequence, and automate manual install.

Easy button wrapper

Why? Helpdesk and server admin teams don’t have access to Tier0 devices. After no access, the Manual process (instruction steps get missed), then server is not monitored, becoming an outage resolution task/follow-up.

Use the below thread = ‘easy button’ to package SCOM agent when MECM administrator has availability to add SCOM agent to SCCM task sequence.

MECM/SCCM Application/package script :

Using Powershell.exe:
###############Startscript
“SCOM Super Installer”
start-transcript -path “c:\windows\ccm\logs\SCOMSUPERINSTALLER.log”

##ONLY THIS SECTION NEEDS TO BE EDITED, replace different domain FQDNS and the gateway/management server(s), management groups

# Leverage find/replace for the ##something## variables

# Provide SCOM Gateway or SCOM management server MS for $SCOMGATEWAYFQDN variable.

# Include SCOM Mgmt Group Name for ##SCOMMGMTGroupName##

Switch ($domain) {
“##DomainFQDN1##” {$SCOMManagementGroup=”##SCOMMGMTGroupName##”;$SCOMGATEWAYFQDN=”##SCOMServerName##”}
“##DomainFQDN2##” {$SCOMManagementGroup=”##SCOMMGMTGroupName##”;$SCOMGATEWAYFQDN=”##SCOMServerName##”}
}
########ONLY THIS SECTION NEEDS TO BE EDITED

“Install”
$commonArgs=”.\MOMAgent.msi /qn USE_SETTINGS_FROM_AD=0 USE_MANUALLY_SPECIFIED_SETTINGS=1 SECURE_PORT=5723 ACTIONS_USE_COMPUTER_ACCOUNT=1 NOAPM=1 AcceptEndUserLicenseAgreement=1 MANAGEMENT_SERVER_DNS=$SCOMGATEWAYFQDN MANAGEMENT_GROUP=$SCOMManagementGroup”

start-process MSIEXEC.exe -ArgumentList $commonArgs -wait -nonewwindow
stop-transcript
###############Endscript

Lastly, after SCOM agent added to Endpoint Manager, monitoring new servers should be a no-brainer. One less manual step having ‘SCOM agent application’ as part of the task sequence.

Enjoy!

Documentation

SCOM2022 install agent manually, 2019 here

DNS Addendum pack

nslookup to find out IP to name or name to IP resolution.

Simply put: Leverage the ‘DNS Addendum pack’. Why? DNS is a translation method to convert names to IP’s. Can you imagine if we wanted to connect to google via IP? The amount of workflows in the SCOM DNS pack (built by the DNS Product Group) makes for an astounding number of workflows running on your DC every minute. Forward and reverse lookups are a good check, verifying DNS is functioning. In a complex environment with 100’s of zones, SCOM becomes a utilization culprit for a DC’s primary missions – authenticate and resolve. This article will help you understand how the pack will add new capabilities and tune DNS monitoring to best practice.

QUICK DOWNLOAD(S)

2016+ https://github.com/theKevinJustin/DNSAddendumAgnostic

What capabilities does the ‘DNS Addendum pack’ provide?

Count logic monitors (i.e. x events in y time, and self heal)

Daily summary report of DNS alerts broken out

DNS service(s) recovery automation

Daily alert closure workflow to close out DNS rules/monitor

Synthetic internal/external nslookup monitor (scoped to PDC emulators versus ALL DNS servers

WMI validation alert recovery to prevent false positive alerts with weird one off scenarios – one example: Security tools randomly block WMI access.

Download the DNS Addendum on GitHub and the PDF install guide, to improve AD Integrated (ADI) DNS monitoring on Windows Server 2016+ (version agnostic).

XML authoring

The pack greatly decreases alerts, workflows on your AD integrated DNS servers, and the XML authoring is an easy feat. After you import the pack, find/replace is required for two pieces.

Group GUIDs update, after installing this pack.

Find/replace the GUIDs, as they are unique to every SCOM management group, hard coding the group ID GUID is not possible.

From PowerShell, on your SCOM management server, run these commands (after DNS Addendum installed)

Use get-scomclassinstance -DisplayName “GroupNameHere” | ft Id

Find/Replace the GUID in the pack with the ID from the output above.

Discovery group regular expressions (RegEx)

##DNSServerRegEx##

Find ##DNSServerRegEx## and replace with your DNS server expressions.

Example server names: 16dns01, 19dc01,16dns02,19dc02,19dc03, etc.

RegEx = (?i)16dns0|19dc0

DNS Group discovery example of RegEx for find/replace

Save and Import & Enjoy!

Update SCAP tools

DISA Security Content Automation Protocol

One more admin process and workflow is to ‘update SCAP tools’ on servers. Many times overlooked, this can save many headaches with the newest version installed on servers.

Check DOD Cyber Exchange

Check the website here, to search for Win in SCAP tools, then download & Install

SCAP tool download from DOD Cyber Exchange public website.

Navigation steps:

Control Panel > Programs > Programs and Features

In the search bar (top right) enter scap (and hit enter)

SCAP Control panel output showing multiple versions installed. Need to install latest application, then remove the old versions (in this case, all three!)

Install SCAP application

Extract files from ZIP

Copy folder to repository (my path example below)

Save SCAP zip and files to folder repository and on server to install SCAP on.

Run SCAP application

Take the defaults (unless you want the checker icon on desktop). Run SCAP application from PowerShell (as admin) window.

Open PowerShell as admin window

Example:

cd “D:\MonAdmin\STIGS\scc-5.7.2_Windows”; gci; .\SCC_5.7.2_Windows_Setup.exe

Hit enter to begin install

Run SCAP install from PowerShell (as admin) window.

On the SCAP EULA radio button application install screen, click ‘I accept’ radio button and click Next.

Select Destination location (preferably on non-system disk), and click Next

Change path to non-system disk (like d:)

SCAP Destination Location Application install window.

From the ‘Select Components’ window, click Next

SCAP Select Components application install window.

Click Next on the Setup Start Menu folder window

On the SCAP select additional tasks install window, click Next

Click Install on ‘Ready to install’ popup screen

With the new SCAP tool Install window, click Finish to complete.

SCAP tool install finished splash screen.

Refresh Control Panel SCAP search

Remove old versions

Click Continue and go through removal prompts

SCAP control panel remove old version with prompt to continue.

With the Uninstall screen, click Yes to uninstall.

Click OK on uninstall

Check Control Panel for SCAP installs

Verify control panel only has latest version installed. Close out Programs and Features window

Windows Control Panel, Programs and Features, SCAP search for new version install

Review SCC (SCAP Compliance Checker) Release Notes

Verify SCAP application functionality

Click on Start > start typing SCAP > Click on SCAP Compliance Checker

From the SCAP checker UAC prompt, click Yes to continue

Click OK to end the install

Run Local Scan

Run local scan to prove functionality.

Select STIG(s) in the middle pane > Click Start Scan

Run SCAP scan against server, choose your STIGs and Start Scan

Verify SCAP tool modified files after installation

Recheck Windows Explorer for OpenSSL; look at file properties for version details. Interesting, NONE of these files have versions (openssl, x509 searches show nothing file version wise)

Ask the Security Admin to re-scan!

Documentation/Links

DOD Cyber Exchange https://public.cyber.mil/stigs/scap/

VMwareTools OpenSSL vulnerabilities

Extra Extra read all about it, VMwareTools OpenSSL vulnerabilities!

Update VMwareTools to solve OpenSSL vulnerabilities CVE-2023-3446, CVE-2023-2975. The ‘VMwareTools OpenSSL vulnerabilities’ showed up two (2) weeks ago, but it took about a week for the update to post. Latest Tenable scan article shows OpenSSL update to v3.0.10 required for VMware Tools.

Update VMwareTools

Start with the Security scan and the plugin ID to mitigate ‘Tenable Scan output of OpenSSL PlugIn ID documenting problems’

Talk with your security team to identify the offending path for guidance on which application might be the culprit. The diagnostic/debug details can be a lifesaver!

Snippet of Tenable OpenSSL path from scan diagnostic of OpenSSL vulnerabilities

Newer version of VMwareTools required to fix OpenSSL vulnerabilities.

Originally, no VMwareTools update posted

VmWare tools v12.6 resolves CVE-2023-3446, CVE-2023-2975. Hopefully your virtualization team uses an Endpoint Manager to manage server configurations, and they have an application/package wrapper to install VMwareTools without this being a manual process

Either way, you’ll have to download the update download link

VmWare tools v12.6 has OpenSSL update to resolve CVE-2023-3446, CVE-2023-2975

Follow VMware’s knowledge base (KB) ‘how to’ article ‘how to’ install VMWare Tools

I typically link this with the monthly patches, where a single reboot puts all the patches into a fresh boot (applying the configuration).

Documentation/Links

Tenable article OpenSSL 3.0.0 < 3.0.10 Multiple Vulnerabilities | Tenable®

VMware KB article ‘how to’ install VMWare Tools

Security Content Automation Protocol (SCAP) download

How to install OpenSSL in windows 10? – Stack Overflow

MSSQL Addendum pack

Time to tune MSSQL alerts!

The ‘MSSQL Addendum pack’ wouldn’t be possible without Brandon Pires contributions. Brandon dealt with my many questions to better alert! If you need more background, check the ‘why addendum pack’ post.

Quick Download(s)

2012+ https://github.com/theKevinJustin/MSSQLAddendum

Capabilities

The pack is based on the SQL engineering blog and program team making multiple updates per year for SQL monitoring. The addendum creates two groups for dev/test and notification/subscription modeling. Second, the overrides, man there are a bunch! aid consumption of real issues. Lastly, most environments should be SQL 2016+, as the 2012R2 EOL/EOSL is quickly approaching in October!

MSSQL groups defined in the Addendum pack

MSSQL group discoveries require updates to be applicable to environment

Tailor addendum

First, the Addendum pack requires the MSSQL packs MUST be installed. The addendum is based on the MSSQL 2016+ version agnostic is currently supported, as the 2012,2012R2 products are near end of support.

Find/Replace the variables as needed:

Example ##TESTSERVER##|##DEVSERVER##

Save file

Overrides

Addendum pack contains discovery, monitor, and rule overrides to tune MSSQL to CSA (old PFE/CE/CSAe Microsoft Field engineer recommendations), to match the health model reducing critical ‘wake me up in the middle of the night’ alerts.

Import

Download pack, and save to your environment

Import into SCOM

Enjoy!

MSSQL Addendum references

MSSQL Engineering blog and old post here

SQL Releases TechCommunity here

Engineering team latest management pack, TechCommunity release v7.2.0.0

Import ‘gotcha’ importing new custom functionality blog