Here’s a new Knowledge Base article we published today. This one walks you through monitoring txt and CSV log files in System Center Operations Manager:
=====
Summary
Configuring System Center Operations Manager to monitor plain text and delimited text log files from various sources is straightforward but there are some scenarios of which to be aware.
More Information
See the link below for instructions on how to create a Simple Event Detection monitor for a plain text (non-delimited) log file in System Center Operations Manager 2007 and 2007 R2:
http://technet.microsoft.com/en-us/library/bb381375.aspx
NOTE This information should also apply to System Center 2012 Operations Manager
Follow the procedure at the link above to create a monitor for a delimited (CSV) log file with the following exceptions:
Step 7: In the Create Monitor Wizard, on the Select a Monitor Type page, expand Log Files, expand Text Log (CSV), expand Simple Event Detection, click Event Reset, and then click Next.
Step 10: The separator must also be entered on the Application Log Data Source page. The separator does not have to be a comma, but it must be a printable character.
Step 12: When entering a parameter name, the number in Params/Param[x] represents the field on which the operation will take place. For example, Params/Param[1] is the first field of the delimited file; Params/Param[2] is the second field of the delimited file, and so forth.
Steps for creating a rule to monitor a log file
The following steps illustrate how to create a rule that targets the Windows Computer class and monitors a log file in System Center Operations Manager 2007 and 2007 R2. For more information on selecting a target for a workflow (which includes rules and monitors) see Selecting a Target for a Workflow
1. Log on to the computer with an account that is a member of the Operations Manager Administrators user role or Operations Manager Authors user role for the Operations Manager management group.
2. In the Operations console, click the Authoring button.
3. In the Authoring pane, expand Authoring, expand Management Pack Objects, and then click Rules.
4. In the toolbar, click Scope.
5. In the Scope Management Packs Objects dialog box, in the Find text box, type Windows Computer, select the Windows Computer target check box, and then click OK.
6. In the Rules pane, right-click Windows Computer and then click Create a New Rule
7. In the Create Rule Wizard, on the Select the type of rule to create page, expand Collection Rules, expand Event Based, and select Generic Text Log or Generic CSV Text Log
NOTE If an alert will be configured, choose Alert Generating Rules, expand Event Based, and select Generic Text Log or Generic CSV Text Log
8. Select a management pack from the Select destination management pack list (in System Center Operations Manager 2012, this list picker is labeled Select Management Pack) or create a new unsealed management pack by clicking New. By default, when creating a management pack object, an override, or disabling a rule or monitor, Operations Manager saves the setting to the Default Management Pack. As a best practice, create a separate management pack for each sealed management pack to be , rather than saving customized settings to the Default Management Pack. For more information, see Default Management Pack.
9. Click Next
10. On the General page, in the Name box, type a name for the rule, and then as an option, you can type a description.
11. Click the Rule Category arrow, select the appropriate category, and then click Next.
12. On the Application Log Data Source page under Define the application log data source, in the Directory text box, type a path to where the log files are located, for example, C:\logfiles
13. In the Pattern text box, type a pattern string to select log files. For example, application??.log will find any log file that starts with ‘application’ followed by zero to two characters with an extension of ‘.log’ (e.g. application.log, application1.log, application01.log, etc.).
14. For CSV log types, enter the separator. The separator does not have to be a comma, but it must be a printable character
15. Select UTF8 if applicable, and then click Next
16. On the Build Event Expression page (for the Build First Expression),click Insert and then do the following:
a. Under Parameter Name (on the left), type Params/Param[1]. Params/Param[1] is the only field available for Generic Text log, since each line in the file is considered a single field. For Generic CSV Text Log, the number in Params/Param[x] represents the field on which the operation will take place. For example, Params/Param[1] is the first field of the delimited file; Params/Param[2] is the second field of the delimited file, and so forth.
b. Under Operator, click the pull down menu and select an operator, for example Contains
c. Under Value enter the text that this monitor should trigger on as found in the log file, for example error
17. Click Create to create the rule. If Alert Generating Rules was chosen in step 7A, click next to configure the alert, then click Create to create the rule.
After the rule is created, edit the properties of the new rule to generate an alert when the rule is triggered. See How to Edit Properties of a Rule in Operations Manager 2007for more information on editing the properties of a rule.
Steps for creating a text log alert rule on System Center 2012 Operations Manager can be found under the section To create a delimited text log alert rule at http://technet.microsoft.com/en-us/library/hh457567.aspx.
To reference the fields that were read from the log when the rule is triggered in an alert, use the Params/Param[x] syntax.
For more information on the various settings available in the Create a unit monitor and Create Rule Wizard wizards, see the section Text Log Wizard Options at http://technet.microsoft.com/en-us/library/hh457567.aspx.
Additional Information
When monitoring a log file, Operations Manager remembers the last line read within the file (a 'high water mark'). It will not re-read data before this point unless the file is deleted and recreated, or renamed and recreated, which will reset the high water mark.
An implication of this is that log files that are cleared periodically without being renamed and recreated, or deleted and recreated, will not have entries in them processed until the high water mark from before the log is cleared is exceeded.
Operations Manager cannot monitor 'circular log files' (i.e. log files that get to a certain size or line count, then start writing the newest entries at the beginning of the log) for the same reason. The log file must be deleted or renamed and then recreated, or the application configured to write to a new log once the current log is filled.
Example:
- 100 lines are written to logfile.txt
- logfile.txt is cleared of all entries
- New log entries are written to logfile.txt (position 0 of the file)
- None of the new entries will be processed until line 101 is written
Each line of a log file must end with a new line (0x0A0x0A hex sequence) before it will be read and processed by Operations Manager.
If a rule or monitor is configured to match a pattern for log file names (e.g. using the ? or * wildcard characters), it is important that only ONE log that matches the pattern is written. If multiple logs that match the pattern are being written to, the high water mark is reset to the beginning of the file with each write to a different file. The result is that all previous log entries will be reprocessed.
Example:
- The log file name pattern is generic_csv??.txt
- The current log is generic_csv01.txt and writes happen to this log.
- A new log, generic_csv02.txt, is created. Writes occur to this log.
- When the next line is written to generic_csv01.txt, the Operations Manager will read from the beginning of generic_csv.txt, not from the last point that was read from generic_csv01.txt. Lines previously processed will be processed again, possibly resulting in alerts or other actions (depending on the rule configuration).
=====
For the most current version of this article please see the following:
2691973 : Monitoring Text and CSV log files in System Center Operations Manager
J.C. Hornbeck| System Center & Security Knowledge Engineer
Get the latest System Center news onFacebookandTwitter:
App-V Team blog: http://blogs.technet.com/appv/
ConfigMgr Support Team blog: http://blogs.technet.com/configurationmgr/
DPM Team blog: http://blogs.technet.com/dpm/
MED-V Team blog: http://blogs.technet.com/medv/
Orchestrator Support Team blog: http://blogs.technet.com/b/orchestrator/
Operations Manager Team blog: http://blogs.technet.com/momteam/
SCVMM Team blog: http://blogs.technet.com/scvmm
Server App-V Team blog: http://blogs.technet.com/b/serverappv
Service Manager Team blog: http://blogs.technet.com/b/servicemanager
System Center Essentials Team blog: http://blogs.technet.com/b/systemcenteressentials
WSUS Support Team blog: http://blogs.technet.com/sus/
The Forefront Server Protection blog: http://blogs.technet.com/b/fss/
The Forefront Endpoint Security blog : http://blogs.technet.com/b/clientsecurity/
The Forefront Identity Manager blog : http://blogs.msdn.com/b/ms-identity-support/
The Forefront TMG blog: http://blogs.technet.com/b/isablog/
The Forefront UAG blog: http://blogs.technet.com/b/edgeaccessblog/