Article created by Rainer Gerhards.
Many system administrators are running Unix / Linux based monitoring and alerting for a long term. The basic idea behind a successful monitoring and alerting system is to centralize all system events at a single monitoring station. Once the information is centralized, it can be used to build an alerting system or even carry out corrective actions.
Syslog protocol
Many Unix based administration solutions are build around the syslog protocol. Syslog is a quite simple protocol. It’s whole idea is to have a server (or daemon as it is called in Unix speak) listen to incoming messages. That server gathers the received messages and writes them to log files. Minimal filtering capabilities are build into both the syslog daemon as well as the protocol itself: the so-called syslog facility and priority. The facility is meant to identify the source of a message (e. g. the print spooler subsystem or a local application). The priority really is a severity code. It runs from debug messages at the lowest to emergency priority at the top. The syslog daemon itself usually does not assign any specific meanings to the facilities and priorities. However, it can be configured to write messages depending on their facility and priority to different log files. It can also discard them totally or forward them to a chained syslog daemon – but we won’t discuss that in detail.
Once the messages have been written to a file, they can be further examined. Many system administrators nowadays use PERL scripts to parse the files and filter out important events. Older solutions are often build with SED, GREP and a lot of help from shell scripts. No matter how it is done, all solutions have in common that the syslog message file is checked periodically and automatically. With no need for manual intervention, there is also no risk of forgetting it.
The big advantage of syslog is that the message file holds all messages reported (if configured to do so). By using clever scripts, a sys admin can also detect complex error scenarios and act accordingly.
Who Reports to the Syslog Daemon?
The whole idea behind syslog is to have all sources report to the central location. As such, it is extremely important that all devices and systems report to the syslog daemon. Fortunately, syslog has been widely adopted as an industry-standard. Nowadays, all leading manufacturers support it. So it is easy to gather information from e. g. routers, switches, firewalls, SANs (storage area networks), print servers and Unix hosts into the central repository.
Unfortunately, there is one widespread used operating system that is not capable of generating syslog messages – Microsoft Windows NT. With more and more NT systems being installed, that fact is challenging the existing system monitoring and alerting infrastructure. Admins become forced to look for expensive and not integrated Windows NT management systems.
Is there any Way to make Windows Report via Syslog?
As a computer consultant, I stumbled into this problem way back in 1997. I had the task to integrate Windows NT system monitoring into an already existing syslog based system. The Windows NT box had been brought into this data center in order to run an application that was supported under Windows only (by the way: in my experience this is a common scenario for Windows to enter Unix data centers). My customer did not like the idea of running a separate Windows admin system. So I looked into Window’s capabilities to check if there would be any work around. I realized very quickly, that the essentials were build into the product. Windows NT (but not 9x) has a consistent event logging mechanism called the “event log”. This log is maintained on a per machine basis and has different section for different pieces of information (e. g. application and system events). Theoretically, it is extensible but I have never found any third-party application creating it’s own Windows event log. All well behaved applications utilize the event log. This is part of the Windows logo specification, so thing are pretty good you favorite NT application will use it, too.
But that was the end of the good news: the event logging API was very bad documented. Also, NT does not contain any functionality to push that information out of the logs automatically. And it definitely does not talk syslog.
So we decided to write a small tool, EvntSLog (Event to Syslog) to extract the information and forward it to a syslog daemon. That tool was to be implemented as an NT service, so that it could be automatically started on system startup and also be run in the background. The initial tool we crafted in early 1997 was severely limited and far from a commercial product. However, it solved it’s purpose very well and made my clients happy. In fact so happy, that they talked about this tool to their colleagues in other companies. Which yield to a number of requests for this tool and finally yield to the decision to build a real product out of it.
Now, in February 2001, the tool – under the name of EventReporter – has just been released in version 6.2 and is running at thousands of installations worldwide. And it still successfully serves the same need.
So what do I need to do to Integrate NT into my Syslog?
Fortunately, it is fairly simple. First of all, you need to check if your syslog daemon accepts messages from remote hosts. If you already use the syslogd to receive such messages, there obviously is nothing more to do. If you start with syslog, you need to check your specific product’s documentation. Most syslogd’s need a startup switch (usually “-r”) to allow reception of remote messages.
After that, you need to make sure your Windows event logs are active. Unfortunately this must be done on each system separately. To do so, start the Windows Event Viewer under Windows NT. Under Windows 10, the same tool is found in the “Computer Management” under “System Tools” and then “Windows Logs”. Right click on the section you want to configure and then select “Properties”. A window like this should open:
Windows 10 Event Log Properties
As you can see, this are the properties for the application event log. Remember that there are 3 event logs under Windows NT and up to 3 more under Windows server (depending on the server role). The properties need to be set for each event log separately.
First of all, it is important to set the properties so that logging is active. A common problem are log files that are to small. As you can see, the maximum size of the log file is configured. Once this size is reached, NT will disable event logging unless it is instructed to overwrite events as needed. As you can see, we have checked that option. You might ask if that isn’t a possible weakness because events will be overwritten without being seen. With our syslog based environment, that really is not an issue. EventReporter periodically reads all logs and forwards their content to the syslog daemon. There they are finally stored. The local Windows system just needs to have log files large enough to hold all messages that are newly logged between EventReporter iterations. I typically recommend that EventReporter is run every minute or so (it does not place a noticeable burden on the monitored system). As such, the event log buffer needs not to be very large in order to buffer all events during this period. However, with disks being that inexpensive today, I recommend leaving 1 MB for each log file – you never know. Especially security attacks might generate a large number of events in a short period of time and you most probably won’t loose them. Just in case you wonder: EventReporter tracks its last reported message by itself and as such does not report any events already reported, no matter how large the buffer. It does, however, detect truncated log files which can be a nice feature to detect security compromises.
Once you have set up the Windows event system, you need to install and configure EventReporter on each system that is to be monitored. The reason it needs to be installed on each system is performance and stability. With each instance running locally, there is no unnecessary network burden for the transmittal of unneeded events (which can be filtered out by the product) nor is there any single point of failure (like with a single system doing all the monitoring tasks). EventReporter has support for mass rollouts and its configuration settings (registry) are documented in detail. So even large installations do usually not have any issue with the local installation.
The basic EventReporter setup is easy and straightforward. Simply use the client configuration tool that comes with the product. It just needs to be told to which syslog server it should report (IP address or resolvable name). There can also a backup syslog daemon be specified. If done, EventReporter sends each message to both syslogds. Once this is done, EventReporter can be started.
On its initial start, it dumps out all events currently in the NT event logs. So expect quite a number of messages to arrive at the syslogd. After that, it will forward only new messages.
In order to fully integrate into existing environments, EventReporter can send the different Windows event logs with separate syslog facilities. The Windows severity code (Error, Warning, …) is also translated into a standard syslog priority. The message itself can easily be parsed at the receiving side and contains all information from the event log. Most importantly, it contains expanded message strings. One of the problems with the Windows event log API is that the event entries actually do not contain the event message itself. Instead, they contain an index into a so-called “message file”. This was meant to support internationalization, but has a severe drawback: if the message file is not (or no longer) installed at the system, the message text is lost. This most often happens when you try to view an event log remotely and do not have the products installed that generated events (e.g. Exchange server events when viewing from a Windows NT workstation).
EventReporter can also locally pre-filter events. This can be done much like in Windows event viewer. Filter criteria’s can be the event source, event id and other properties of the event log record. Local filtering is a powerful feature to avoid unimportant messages to be forwarded to the syslog daemon. So it helps reduce the size of the central syslogd’s log file.
A simple and effective Solution
With EventReporter, Windows NT can easily be integrated into existing syslog based management systems. The best news for Admins currently using such a system is that they do not need to add a new management system to their environment. They can also use all the powerful tools (like PERL scripts, pager activation programs and the like) they already have in place. It is no wonder that many large organizations have chosen that path.
I hope that this article on the Integration of Windows NT event logs into the Unix environment is helpful. If you have any questions or remarks, please do not hesitate to contact me at rgerhards@adiscon.com.