Home

Contents

Abstract and Introduction

Theory of Operation

Terms and Concepts

Sample Configuration File

Command Reference

Return to Top

Abstract

Return to Top

This project is currently in beta for Apple OSX. It has been reverted to a pre-beta state as it needs an overhaul as a result of its port to Linux and subsequent work to insure it works on both OS's.

Once it has been re-verified for Linux, it will need stress-testing and validation under live conditions to be moved from beta to production status.

This wiki needs a lot of work (and will receive it as time permits). I'm also scrambling to clean up the documentation for Syslogd2. I've uploaded a (working copy) of a "dbd2 manual" to the files section of this project but anticipate using this wiki as the primary source of documentation for some time to come.

The DBD2 project consists of 2 binaries:

(1) A client component (dbcl)

The client has no pre-requisites or required supporting software.

(2) A server component (dbd2)

The sever component needs to have the mysqlclient library installed if native MySql support is desired.
If dbi support for multiple database types is desired, the "C"-language dbi-library API from the SourceForge project of the same name must be installed.
Other than the database library(ies), the dbd2 server has no other software requirements (besides the obvious such as a database of an appropriate type running somewhere on the network).

Both components are written completely in C.

The prototype for this project was designed to track user jobs running on multiple Linux clusters. The dbcl executable was called by the job scripts and sent internal cluster node-id, job-id, user-id, job-status and status-change-time to the server. The dbcl (client executable) is used only for non-syslog input and can be used for any purpose in any situation where the client is running the Linux OS and desires to provide a logging record in some remote (MySql) database. The IP protocol used by the client is TCP. If the connection cannot be immediately established, the client backgrounds itself and continues to try until its life-time timer expires.

The server reads the command-line input, then does a reverse-name-lookup (via cache-file or DNS), and records the time-of-receipt.

All the client data as well as the source-envelope information become available database field variables for use in user-defined SQL templates.

In the current design iteration, I have added:

Interface code for Syslogd2:

Allows syslog-priority-selector-strings to select the sql templates and databases (since syslog data is standard format). Treat syslog envelope fields (facility, priority, event-receipt-time, event-host, etc as usable database fields.

I plan for future releases to add additional features as requested or identified over time. These features include but are not limited to:

Security features to prevent malicious insertions

The system configuration file is divided into a global section and one or more output sections. (See Configuration File Format).

The system supports multiple input threadpools each of which uses 1 or more threads to support one or more input sockets (Linux, IP or a mix of both -- TCP/Streaming Linux only for now).

The system also supports one or more "worker" threadpools, each of which processes the incoming traffic from one or more "reader-threadpools".

The system supports up to 26 databases (identified as 'a' - 'z') with 1 or more threads per threadpool.

One or more databases per threadpool are supported with each database allowed to have one or more connections for parallel data insertion. Each database may also specify one or more "sql templates" defined by the administrator for insertion of one or more "sql variable sets" into an arbitrary number of database tables.

Syslog data input is optionally supported. When not declared, the syslog components are not compiled into the binary, resulting in a smaller binary image.

Only the dbd2 server needs to have the MySql client libraries --or-- the 'C' dbi drivers (found on sourceforge) -- or both installed locally.

Output is specified as type `mysql`, type `dbi`, or type `file`. `File` is simply a text file containing the output SQL lines. (The output file may also be useable as a database input-file for non-MySql database periodic updates).

If there is developer interest or any interest in assisting with documentation, I will happily accept.

Project Members:

• Ed Freesmeyer (admin)

Return to Top