Enhanced Replication Monitoring

Introduction

This page describes enhancements to the basic replication system discussed in eLink Nomad Offline Configuration. What we are going to do is extend the basic replication system described there to include monitoring facilities. The enhancements described here allow a graphical presentation of the state of the offline replication on the child database server which will:

1. Present an icon on the windows notification area indicating if replication is working (or not).
2. Warn users if replication stops for any reason.
3. Give a graphical display of the entire replication system, showing the latest available status of each process.
4. Allow users to resend a data packet.

Architecture

Rather than disturb the basic replication scripts by loading them with the monitoring tasks, we will add an extra monitoring process which will run on the child database server. The only enhancements then required are to provide that process with all the additional status information. In particular, the parent database server must send its status data to the child process in order to make it available for monitoring.

The data to be monitored on each server comprises:

1. Offline Replication Service Log
2. Date/Time of last packets sent and received by the FTP scripts.

The offline replication service log can get very large, but we only need the last part of it to see the state of that process. It is important to grab the end of these logs because transferring and displaying the complete logs can be very resource consuming.

Prerequisites

1. Java (JRE) on child database server
2. tail.exe on both servers
3. JDIC libraries and executables

The tail.exe is a simple command line tool, which mimics the unix tail command, extracting the last few lines from a text file. We use this command to grab the end of the Offline Replication Service Logs. Tail.exe can be found in the Microsoft Root Tookit (rktools.exe). Note that it is not fully windows 7 compatible, and causes difficulties when installing on windows 7, but will run correctly after a reboot.

JDIC is the Java desktop interface, which allows us to add icons on the windows notification area.

Scripts

There are three scripts required:

Setup

The

Operation

Assuming the system is configured for automatic startup, a light will appear in the notification area of the windows task bar, for instance:

The colour of the light indicates the status of the offline replication system. The main colours are light green (as shown), green, yellow and red. Other colours (such as blue, grey and black) indicate fault conditions suggesting the system has not be setup correctly or that the offline replication system has not started.

A right click on the icon will bring up a menu, which allows users to quit the monitor. A left click shows a status display window, such as:

This display allows the user to see at a glance the overall status of the offline replication system. The parent and child database servers are represented, with the green arrows showing the communication between them. The large text areas at the top and bottom of the window show a portion of the log files from the offline replication service on each server. The four smaller text boxes show the status of the FTP scripts on the servers, containing the date and time at which that particular communication was successfully performed. The most recent one is highlighted in yellow.

Under normal operation, this screen will automatically update whenever there is a change in the status of the offline replication system. Thus, it animates the replication process.

In addition to the above elements, the system will also raise a dialog box if the offline replication status remains unchanged for a period of time, indicating that replication has stopped.

The resend buttons will become active if the monitoring system recognises that the system has stalled and if Child Server FTP script has a data packet which could be resent. Note that sending those packets may cause problems if not used with great care.

Interpretation

If the top left text box is highlighted (as shown above), that means that the parent server has successfully received a LOC packet from the child server. So, the system may be waiting for the eLink Offline Replication Service to run, or if that service has run, it may be about to send the next GLB packet. Inspection of the last few lines of the Offline Replication Service log (at the top of the display) will show if it has run.

Currently, due to the underlying architecture of the system, it is not possible for the top right text box to be highlighted.

If the bottom right text box is highlighted, this indicates the Child server has successfully received a data packet from the Parent server. Inspection of the Child server’s offline replication service log (at the bottom of the screen) will indicate if the service has run correctly. If it has, we are waiting for the Child server to recognise the new LOC packet and transmit it to the Parent server.

If the bottom left text box is highlighted, this indicates that the Child server has successfully sent a data packet to the Parent server, so we are waiting for the parent server to receive that packet.

Limitations

It would appear that the Offline Replication Service Logs are buffered, which means they do not necessarily show the very latest information from those services at all times. This is an eLink limitation.
The monitoring service relies on the same communication path from the parent to the child server to receive parent status data. What this means is that when the parent server sends its GLB packet to the child, it will also send an update on its status, indicating the GLB packet has been sent. At the same time, this information will only be recieved after the child process has successfully received the GLB packet. So, the top right text box can never be highlighted! More seriously, if communications are lost between the servers, no status data will be available from the parent server. In this case, the status display may be out of date with respect to the parent server.

Future Enhancements

This tool need to be more generic, with facilities to configure it for wider use. In particular, the names of the data packets are hard-coded, and the timeouts are also hard-coded.
The status display could be enhanced by showing the status of the internet connection on the child server. It may also be helpful to echo the overall notification status light on the status display.