![\"Are](\"https:\/\/kevinjustin.com\/blog\/wp-content\/uploads\/2020\/04\/Stoked.jpg\")
<\/p>\n<\/p>\n
<\/p>\n
<\/p>\n
First, my thanks to Mike Johnston@Microsoft (CSS SEE SME) to help validate my steps and testing, to configure Linux FluentD on an Ubuntu server!\u00a0 Are you ready to bust a myth – OMSAgent FluentD debunked<\/p>\n
<\/p>\n
If you’re starting fresh, or just joining, start with Part 1<\/a>.\u00a0 And Part 1 configures packs and assumes SCOM agent is installed and working.\u00a0 Because it’s time to use the feature, we need to get the agent configured and tested.<\/p>\n <\/p>\n <\/p>\n This piece is missing in the doc, but the content development team has this covered in a subsequent docs article<\/a>.\u00a0 We need to load a sample log monitoring pack to the SCOM management group, so we can test functionality.<\/p>\n Grab the file here<\/a>, otherwise you can copy\/paste from the docs article pretty easily.<\/p>\n <\/p>\n <\/p>\n It’s now time to enable the OMED service on the management server, and we can start with the docs subsection<\/a><\/p>\n Navigation steps from SCOM console<\/strong> (GUI)<\/p>\n <\/p>\n Steps to set\/start service PowerShell<\/strong> (as admin)<\/p>\n # Verify service startup type is automatic<\/p>\n get-Service OMED | select -property Name,Starttype<\/span><\/p>\n # Example output<\/p>\n # Set startup type<\/p>\n <\/p>\n # Start OMED service on SCOM management server (MS)<\/p>\n start-service OMED<\/span><\/p>\n <\/p>\n Now we’re ready to test the UNIX agent!<\/p>\n <\/p>\n <\/p>\n And now it’s time to switch to the agent side.\u00a0 I’m assuming that you’ve already configured the SCOM agent on the Linux server.\u00a0 So it’s time to verify the SCOM and OMSAgent is configured and working.\u00a0 Let’s go back to the docs<\/a> subsection for our sanity check, because we need to create folders, and set ownership, etc.<\/p>\n <\/p>\n Configuring FluentD requires the SCOM management server (MS) has signed the certificate on the UNIX server.\u00a0 The docs article<\/a> tells you to generate a new certificate for FluentD, which requires the management server.<\/p>\n Sign the certs on the agent > copy to MS > sign > copy back to agent<\/p>\n \/opt\/microsoft\/scx\/bin\/tools\/scxsslconfig -c -g \/etc\/opt\/microsoft\/omsagent\/scom\/certs\/<\/span><\/p>\n 2. Rename certificates<\/p>\n cp -p omi-host-server.domain.pem to scom-cert.pem<\/span><\/p>\n cp -p omikey.pem to scom-key.pem<\/span><\/p>\n <\/p>\n 3. Copy certs to MS (sftp\/ssh via WinSCP, or your app of choice)<\/p>\n <\/p>\n 4. Sign certs on MS via scxcertconfig -sign<\/span><\/p>\n <\/p>\n Open PowerShell (as admin)<\/p>\n Go to your SCOM management server directory (hopefully d:)<\/p>\n cd ‘D:\\Program Files\\Microsoft System Center\\Operations Manager\\Server’<\/span><\/p>\n scxcertconfig -sign scom-cert.pem<\/span><\/p>\n scxcertconfig -sign scom-key.pem<\/span><\/p>\n <\/p>\n 5. Copy certs back to agent from MS (sftp\/ssh via WinSCP, or your app of choice)<\/p>\n <\/p>\n 6. Verify the SCOM certificate shows your Management Server (MS) in the DC= line in the certificate<\/p>\n openssl x509 -in scom-cert.pem -noout -text<\/span><\/p>\n <\/p>\n 7. Restart omsagent<\/p>\n As the ALLINONE server is one of my 2019 SCOM labs, I can verify that my cert is now signed by the management server (MS).\u00a0 Time to load the certificate, and then restart the agent to see if we have any errors<\/p>\n # Restart Agent<\/p>\n \/opt\/microsoft\/omsagent\/bin\/service_control restart<\/span><\/p>\n <\/p>\n <\/p>\n Verify any errors from the omsagent.log<\/p>\n Depending on where you are with your UNIX\/Linux commands, this may help provide some context or use case examples.<\/p>\n My example –<\/p>\n First error after restart was ‘permission denied’. \u00a0 FluentD runs under the omsagent ID, and needs to have access to whatever log – at least read (4).\u00a0 For the syslog example, I made omsagent the owner, and omiusers the group. \u00a0 The smarter, security hat on, choice is to leave as root and make it read capable, or add omsagent to the root group<\/p>\n <\/p>\n Search \/var\/opt\/microsoft\/omsagent\/scom\/log\/omsagent.log<\/span> for errors.\u00a0 Commands build on another, from simpler to more complex.\u00a0 Don’t worry if UNIX\/Linux is new, I’m all about examples, so hope that helps bridge the gap!<\/p>\n <\/p>\n # Tail omsagent.log for progress<\/p>\n # Option 1 Continual output updates from file<\/p>\n tail -f \/var\/opt\/microsoft\/omsagent\/scom\/log\/omsagent.log<\/span><\/p>\n # Option 2 – get last 10 lines<\/p>\n tail \/var\/opt\/microsoft\/omsagent\/scom\/log\/omsagent.log<\/span><\/p>\n <\/p>\n # Option 3 – get last 100 lines<\/p>\n tail -100 \/var\/opt\/microsoft\/omsagent\/scom\/log\/omsagent.log<\/span><\/p>\n # Option 4 – Get a little fancier – search for a string<\/p>\n grep string \/var\/opt\/microsoft\/omsagent\/scom\/log\/omsagent.log<\/span><\/p>\n # Option 5 – Specific example = error, case insensitive (-i)<\/p>\n grep -i error \/var\/opt\/microsoft\/omsagent\/scom\/log\/omsagent.log<\/span><\/p>\n <\/p>\n # Option 6 – egrep strings and -v to exclude what you don’t want to see<\/p>\n grep -i error \/var\/opt\/Microsoft\/omsagent\/scom\/log\/omsagent.log |egrep -v “Permission denied|stacktrace”<\/span><\/p>\n <\/p>\n Verify FluentD conf files and omsagent.conf has INCLUDE line<\/p>\n The INCLUDE lines allows a directory for a ‘Gold depot’ to control what log files are monitored on destination linux servers.\u00a0 The goal is a standard repository (gold depot ) to simply copy the conf file you want for logfile\/app\/daemon, restart agent, and you’re off to the races monitoring that log file.<\/p>\n <\/p>\n Verify omsagent.conf includes directory<\/strong><\/p>\n grep -i include \/etc\/opt\/Microsoft\/omsagent\/scom\/conf\/omsagent.conf<\/span><\/p>\n # If there’s output, make sure that omsagent.d path exists<\/p>\n # Verify permissions show omsagent:omiusers<\/p>\n ls -al \/etc\/opt\/Microsoft\/omsagent\/scom\/conf | grep omsagent<\/span><\/p>\n <\/p>\n 10. Back to step 8’s problem, to fix the FluentD conf files, so we can test!\u00a0 Step 9 verified that FluentD is configured via the omsagent.conf, and also for specific configuration files (.conf) in omsagent.d directory.<\/p>\n Next, we need to restart the agent to verify configuration, and any errors are seen on the FluentD side.<\/p>\n My error for ‘out_scom’ plugin was already used by some other test conf files.<\/p>\n grep -i error \/var\/opt\/Microsoft\/omsagent\/scom\/log\/omsagent.log |grep “Permission denied” |tail<\/span><\/p>\n <\/p>\n Example of omsagent.log where we have traced an event for our mylog<\/p>\n Mike explained that my error was due to having multiple FluentD conf files using the same buffer path for ‘out_scom’.\u00a0 I searched the conf files to see who had ‘out_scom’ and removed one of my old test files from months back when I was testing the feature.<\/p>\n # Example of errors in the omsagent.log<\/p>\n <\/p>\n Don’t forget to restart the omsagent for reading in the new file changes<\/p>\n # Restart Agent<\/p>\n \/opt\/microsoft\/omsagent\/bin\/service_control restart<\/span><\/p>\n <\/p>\n <\/p>\n I’ll cover building a fluentd conf file in another blog post for brevity.<\/p>\n <\/p>\n <\/p>\n Time to test our FluentD conf file and append entries into the log file!<\/p>\n Starting simple again<\/p>\n # Options<\/p>\n echo test >> \/var\/log\/mylog<\/span><\/p>\n echo 911 error >> \/var\/log\/mylog<\/span><\/p>\n # Echo entries into test logfile to mimic syslog or messages<\/span><\/p>\n echo `date +”%b %e %H:%M:%S”` MYLOG 911 test string. Call 911<\/span><\/p>\n # Verify<\/p>\n tail \/var\/log\/mylog<\/span><\/p>\n Switch over to SCOM management server, and look for alerts<\/p>\n Navigate to the Monitoring Tab > Active alerts<\/p>\n <\/p>\n <\/p>\n References for more information<\/strong><\/p>\n In case you need a refresher on all the date options\u2026 Found CyberCiti FAQ<\/a> helpful<\/p>\n All because the goal is to make the echo statement better for testing closer test\/UAT examples on string matches, etc.<\/p>\nPart one (1) quick summary<\/h2>\n
\n
\n
<\/h5>\n
Load Sample Log monitoring pack<\/h2>\n
![\"Configure](\"https:\/\/kevinjustin.com\/blog\/wp-content\/uploads\/2020\/04\/OMEDmgmtpack.png\")
<\/p>\nVerify OMED service running on Management Server<\/h2>\n
\n
\n
PS C:\\Users\\admin> Get-Service OMED | select -property name,starttype<\/div>\nName StartType
\n—- ———
\nOMED Automatic<\/div>\nConfigure SCOM\/OMSagent on Linux server<\/h2>\n
Create files and set permissions<\/h5>\n
mkdir \/etc\/opt\/microsoft\/omsagent\/scom\/conf\/omsagent.d<\/span><\/div>\nmkdir \/etc\/opt\/microsoft\/omsagent\/scom\/certs<\/span><\/div>\nmkdir \/var\/opt\/microsoft\/omsagent\/scom\/log<\/span><\/div>\nmkdir \/var\/opt\/microsoft\/omsagent\/scom\/run<\/span><\/div>\nmkdir \/var\/opt\/microsoft\/omsagent\/scom\/state<\/span><\/div>\nmkdir \/var\/opt\/microsoft\/omsagent\/scom\/tmp<\/span><\/div>\nmkdir \/home\/omsagent\/fluent-logging<\/span><\/div>\n# NOTE – This location is flexible for the path to use for log file position files<\/span><\/div>\nchown omsagent:omiusers state<\/span><\/div>\nchown omsagent:omiusers run<\/span><\/div>\n\nchown omsagent:omiusers log<\/span><\/div>\nchown omsagent:omiusers tmp<\/span><\/div>\nchown omsagent:omiusers \/home\/omsagent\/fluent-logging<\/span><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/h2>\n
Verify SCOM certificate<\/h2>\n
Overview<\/h5>\n
<\/h5>\n
Step by step instructions<\/h5>\n
\n
\n
![\"Verify](\"https:\/\/kevinjustin.com\/blog\/wp-content\/uploads\/2020\/04\/UNIX-openssl-cert.png\")
<\/div>\nVerify omsagent.log errors<\/h2>\n
![\"Configure](\"https:\/\/kevinjustin.com\/blog\/wp-content\/uploads\/2020\/04\/permissiondenied.png\")
<\/p>\n<\/h2>\n
Verify FluentD config files<\/h2>\n
![\"ls](\"https:\/\/kevinjustin.com\/blog\/wp-content\/uploads\/2020\/04\/OMSAgent-FluentD-conf-files.png\")
<\/p>\n<\/p>\n![\"Tail](\"https:\/\/kevinjustin.com\/blog\/wp-content\/uploads\/2020\/04\/OMSAgent-Log.png\")
<\/p>\nTime to test for alerts!<\/h2>\n
![\"OMSAgent](\"https:\/\/kevinjustin.com\/blog\/wp-content\/uploads\/2020\/04\/OMED-SCOM-Alerts.png\")
<\/p>\n![\"Configure](\"https:\/\/kevinjustin.com\/blog\/wp-content\/uploads\/2020\/04\/test-date.png\")
<\/p>\n
![\"OMSAgent](\"https:\/\/kevinjustin.com\/blog\/wp-content\/uploads\/2020\/04\/mylog-Date-tail.png\")
<\/p>\n