0919ELSE "Search" tab

The Official X-Itools ELSE 0.9.19 User's HOWTO

Edited by Nicolas HAHN < hahnn@x-itools.com > / < hahnn@erios.org >

---

This page for version: 0.9.17 | 0.9.18
Top: Documentation for Users | Previous: "Statistics" tab | Next: "Logs" tab

---

• "Search" tab
  • "My E-mail Search Requests" window
  • Search by Message-ID
    • Import Message-IDs
  • Search by QID
  • Extended Search
    • A word about operators
    • Extended search results
  • NOQUEUE List
    • NOQUEUE search results
  • Detailed E-mail search results
    • Overview of the email details window
    • List of transactions
    • Text view: Postfix transaction details
    • Text view: Exchange transaction details
    • Graphical view

"Search" tab

"Search" tab: search a Message-ID, a QID, a rejected email or do an extended search

...Because at some point, we always need to find something.

With the "Statistics tab", the "Search tab" regroup the most used features of the ELSE.

Emails can be searched using different ways in the ELSE:

• By Message-ID: any email server normally configured should always insert a message-id line in the header. This message-id can also be generated by some MUA (Mail User Agent) instead of the MTA (Mail Transfer Agent). But too much often, no message-id is generated. Then the various SMTP servers by which has passed an email may potentially fix that behaviour and generate/insert a message-id. Message-ids identify a unique email passing through different servers.
• By QID: QID means Queue IDentifier. The QID notation comes from Postfix SMTP server. I consider we can define a "Search by QID" as being a search for a specific email transaction. Any other SMTP software can generate one or several QIDs during the processing of an email. In the case of Exchange Server logs, the ELSE generates itself QIDs for them (to give a little bit of structure to infamous Exchange Server identifiers...)
• By doing an extended Search: that allows you to set several search criteria and operators. That's the most powerful search feature of the ELSE
• By searching in the rejected emails more specifically: here also, you can set several criteria and operators.

Searching methods are arranged in the ELSE according the time needed to perform them. That means that searching by Message-ID or by QID gives an immediate answer. That's the fastest search available and by far. So each time you have a search to perform, try to do it using a message-id or a QID.

Then comes the extended search which is the most powerful one considering the number of criteria you can set with their operators, but this is the slower and by far.

And finally comes the search in rejected emails. Rejected emails are the ones that are even not processed in any way by Postfix. So, the quantity of data is extremely limited. This specific search is faster than the previous one, but not as fast as the two first ones.

About Message-IDs and QIDs, just remember that:

• a Message-ID, as written above, identify a unique email from the source to the destination
• a QID just identify a step in the processing of an email on a particular messaging server

So, we'll consider a QID is an email transaction of a specific email represented by its Message-ID. An email can be composed of an enormous number of transactions.

We'll now review each kind of search in the sections below. In the WUI, to select the right search tool, you just have to click on the bar corresponding to the title of the search you want, or to the plus/minus symbols inside the horizontal bars on the right.
But at first, it's necessary to write about another window you can see in the WUI: the "My E-mail Search Requests" window.

"My E-mail Search Requests" window

In this window will be recorded any search request you do in the ELSE, except the ones for the rejected emails (NOQUEUE List). So any search by Message-ID, QID, or extended search will go there.

"Search" tab: the ELSE remembers your search requests

If you use the ELSE very often, you will have a lot of windows opened in it showing email transactions, that are the results of your different search requests. You can have closed some windows that you'll need again some minutes or hours later, but you might not remember what research criteria you've entered previously to make your searches again.

This window is there for that purpose. You can recall any search made previously just by double-clicking a line inside. That's not the results of a previous search that are recalled. Instead, the ELSE will launch a new search with the criteria recorded in the corresponding line. That means if the results of this specific search have been updated (for example, there are more email transactions done over the time) then you will get them in the new results.

The "Clear" button at the top-right of this window will erase all its content if you click on it.

This window cannot be closed.

Search by Message-ID

Enter your message-id in the Message-id field then click the Search button and wait for the results. The results should be immediate or very fast to come.
Your message-id cannot be a partial one. It must be complete. Check that it's not written between <> symbols as well. Wildcards are not supported there.

For example, don't enter your message-id like the examples below:

9fd705e652944a4c98b32e82cd5c21e9@*{@style=color: blue;}
*@my.domain.com{@style=color: blue;}
<9fd705e652944a4c98b32e82cd5c21e9@my.domain.com>{@style=color: blue;}

But enter your message-id like this:

9fd705e652944a4c98b32e82cd5c21e9@my.domain.com{@style=color: blue;}

Results will be displayed in a new window that will be described in a below section.

Import Message-IDs

Some messaging administrators or users might have to collect a set of different message-ids via a script they made by themselves for example, or whatever.
Then, they want to get all the details regarding the emails concerned by those message-ids.

It can be boring to enter them one by one in the "Search by Message-id" feature of the ELSE. That's why the Message-Ids Import has been designed.

If you click on the "Import" button on the left, below the Message-ID field, a new window will appear to let you enter your list of message-ids in the text area, one complete message-id by line, then click the "Import" button.

Remember that your message-ids cannot be partial ones. They must be complete. Check that they are not written between <> symbols as well. Wildcards are not supported there.

You should have something like the screenshot below:

"Message-Ids Import" window: write one complete message-id by line

Once you clicked on the Import button, all your message-ids will be registered in the "My E-mail Search Requests" window, seen previously, as you can see below:

"Message-Ids Import" tab: all the message-ids have been imported

You just have to double-click on each of them to get all the details of the emails.

Search by QID

"Search" tab: search emails by QID

Enter your QID in the QID field then click the Search button and wait for the results. The results should be immediate or very fast to come.
Your QID cannot be a partial one. It must be complete. Check that it's not written between <> symbols as well. Wildcards are not supported there.

For example, don't enter your QID like the examples below:

AB022369*{@style=color: blue;}
*22369FFC{@style=color: blue;}
<AB022369FFC>{@style=color: blue;}

But enter your QID like this:

AB022369FFC{@style=color: blue;}

Results will be displayed in a new window that will be described in a below section.

Extended Search

"Search" tab: search emails using extended criteria

Extended search is to be used basically when you don't really know what you're looking for, for various reasons that can be:

• you just have an idea of the dates when an email has potentially crossed your messaging servers
• you don't really know what's the status of an email
• you don't really know the first part of the email address (mailbox name) but you know the domain name
• plenty of other reasons

The extended search feature is the most powerful one because of the number of search criteria or parameters you can set. But it's also the slowest one.

That's why you should follow a simple rule if you plan to use this search feature: always be precise as much as possible in the criteria you set in the fields, always restrict the parameters of your search as much as possible.

For example, if you have a precise enough idea of the time frame, set start and end dates/times as close as possible.

Also, the ELSE database have indexes defined on key data, but of course not on all data otherwise the database would be slower instead of faster.

That's why, always try to specify the "From:" full e-mail address, for instance.

Here are the fields you can define for an extended search:

• start and end dates and times
• User group to limit the search to the servers assigned to a specific group you are member of
• Server role to limit the search to all servers having a specific role
• full message-id (as seen above for a standard search by message-ID)
• Client IP which is the full IP address of the server that opened the SMTP connection on your servers
• Next HOP IP which is the full IP address of the server on which your servers have opened the SMTP connection
• From which is the sender e-mail address. This field has an operator, so you can enter anything else than a full e-mail address (see section about operators below)
• To which is the recipient e-mail address. This field has an operator, so you can enter anything else than a full e-mail address (see section about operators below)
• Subject which is the subject of the email. The subject have to be written here as it is written in the emails that will match the search, meaning under the encoded format. This field has an operator, so you can enter anything else than a full subject (see section about operators below)
• Status of the email. That can be any, bounced, deferred or sent
• Ext. Status which is the error message field completed by the SMTP servers upon delivery attempt

Once you are done with your search criteria, just click on the Search button then wait for the results. Depending on the processing power of the server on which is installed the ELSE, and of course also depending on your criteria, the ELSE can compute during several long minutes before to give you the results, if any.

NOTE: it's important to wait for the completion of any operation initiated by the ELSE, even if that makes you wait for 30 minutes or 1 hour. If you don't want to wait, if you think that the ELSE may have crashed, and if you close and re-open your internet browser to go on the ELSE again, you have to know that the operations started on the ELSE server side have strong chances to be still in progress. Thus, you might overload the ELSE server just by restarting operations taking a lot of time to complete, by thinking your internet browser is in a kind of blocked state.{@style=color: red;}

A word about operators

"Search" tab: some fields can have search operators defined

As you can see, some fields of the extended search works with operators as described below:

• Exact Match is the equality operator. The field defined with this operator will have to have the exact value you entered. From and To fields use this operator by default.
• Different is the not equal operator. The field defined with this operator will have to have a value that is not the one you entered.
• Domain is the equality operator on the email domain name part. Even if this operator is available for the Subject and Ext. Status fields, it will have an effect only on the From and To fields. If you set this operator let's say for the From field for example, then what you write in the field must be a full domain name: if you want to search for all emails having senders in domain yahoo.com, this field must contain yahoo.com{@style=color: blue;} and not @yahoo.com{@style=color: blue;} as value.
• Contains is the sub-string match operator. Let's say you set the To field with robert{@style=color: blue;} as value. Then the search will show you all email transactions having the word robert in their recipient e-mail address.
• Does not contains is the sub-string not match operator. Let's say you set the To field with robert{@style=color: blue;} as value. Then the search will show you all email transactions not having the word robert in their recipient e-mail address.

As a last example, let's say you want to get all the email transactions having sender email addresses in the hotmail.com domain. You can do it using two different ways:

• The first one is to set hotmail.com{@style=color: blue;} (or @hotmail.com{@style=color: blue;}) in the From field, using the Contains operator.
• The second one is to set hotmail.com{@style=color: blue;} in the From field, using the Domain operator.

Both of those methods will work and provide expected results. But, the second one will be faster than the first one because a search using the domain operator is optimized (understand have an index on the database), whereas a search on a partial email address (here the domain part) will just be an issue in term of database performance.

In conclusion, be careful to always use the right operators in the right fields for your search requests.

Extended search results

Once you've clicked on the Search button, you should see the Extended Search Results window displaying after a while, as shown below:

"Search" tab: results of an extended search

As you can see in the screenshot, 735 different email transactions are matching with your search request. Just by a fast overview of the results, you can know if the e-mail transaction has succeeded or not (status column).

NOTE: The number of output lines in the Extended search results window is limited to 5000.{@style=color: red;}

You can download those results as an excel file by clicking on the "Download" button. Your search request will then be submitted again to the server (that might take again some time to be completed), the file will be generated on server-side then downloaded to you when ready.

If you double-click on a line in this window, all the details of the email corresponding to the e-mail transaction on which you've clicked will be displayed in the E-mail Search Result window, that will be described in a below section.

NOQUEUE List

"Search" tab: use the NOQUEUE List search feature to search for all emails immediately rejected by your Postfix servers

This feature is similar to the Extended Search feature seen in the above section, except it will search only in the emails rejected by your Postfix servers.

The quantity of data provided by such rejected emails is small, that's why the fields available for this search feature is not as much important as for the extended search.

Here are the fields you can define for a NOQUEUE search:

• start and end dates and times
• User group to limit the search to the servers assigned to a specific group you are member of
• Server role to limit the search to all servers having a specific role
• Client IP which is the full IP address of the server that opened the SMTP connection on your servers
• From which is the sender e-mail address. This field has an operator, so you can enter anything else than a full e-mail address (see section about operators above)
• To which is the recipient e-mail address. This field has an operator, so you can enter anything else than a full e-mail address (see section about operators above)
• Ext. Status which is the error message field completed by the SMTP servers upon delivery attempt

Once you are done with your search criteria, just click on the Search button then wait for the results. Depending on the processing power of the server on which is installed the ELSE, and of course also depending on your criteria, the ELSE can compute during several long minutes before to give you the results, if any.

NOQUEUE search results

Once you've clicked on the Search button, you should see the NOQUEUE Search Results window displaying after a while, as shown below:

"Search" tab: results of a NOQUEUE search

As you can see in the screenshot, 10 different rejected email transactions are matching with your search request. Just by a fast overview of the results, you can know what are the reasons why those emails have been rejected.

NOTE: The number of output lines in the NOQUEUE search results window is limited to 5000.{@style=color: red;}

You can download those results as an excel file by clicking on the "Download" button. Your search request will then be submitted again to the server (that might take again some time to be completed), the file will be generated on server-side then downloaded to you when ready.

Detailed E-mail search results

In this section, we describe the most detailed view the ELSE can offer you about the emails that crossed your Postfix and/or Exchange servers.

The detailed e-mail search results window is displayed in below possible conditions:

• a search by Message-ID has been done and a result has been found
• a search by QID has been done and a result has been found
• an extended search has been done, and a specific transaction has been double-clicked in the extended search results window
• a search request previously performed has been launched again via the "My E-mail Search Requests" window.

Overview of the email details window

"Search" tab: detailed view as it is displayed by default

NOTE: please keep in mind that what we call an "email transaction" is in fact a "QID".{@style=color: red;}

The thing to know here is that whatever is the way that has driven you to this window, it will always contain all the details related to a unique Message-ID. That means that:

• if you performed a search by Message-id, all transactions related to this message-id will be available there
• if you performed a search by QID, the ELSE will retrieve the message-id of your QID, and all transactions related to this message-id will be available there
• if you performed an extended search and clicked on a specific transaction (understand a specific QID) of the extended search results window, the ELSE will retrieve the message-id of your QID, and all transactions related to this message-id will be available there

So, even if you did a search using a specific QID, at the end, you will always get all the transactions (all the QIDs) in relation with it via the message-id.

The detailed view is organized around three basic components:

• a drop-down list containing all the transactions related to the message-id (understand all the QIDs) at the very top of the window, labelled "Entry ID:"
• a first tab called "Text view" that will display all the details of the selected QID
• a second tab called "Graphical view" that will automatically generate a network map with some details about the selected QID

Then, depending of several elements like:

• is the transaction related to Postfix ?
• is it related to Microsoft Exchange server ?
• is there an NDR generated by the transaction ?
• are there any bad behaviours detected by the ELSE and generating alerts ?
• others

then the content of the text and graphical views differ.

In the text view for example, all the data are classified in various sections that can be displayed or not.

List of transactions

"Search" tab: select any transaction to get all its details

The above screenshot contains 4 different transactions.

The QIDs can have a:

• short format: they are Postfix related
• long format: they are Exchange related

Exchange QIDs are generated by the ELSE itself, which is not the case of Postfix QIDs that are very well structured and don't need to be re-worked.

The ELSE try to sort the QIDS by date/time order so in theory, each QID of an email follow the other in the right order. Usually, you've one transaction by SMTP server. But you can have several transactions for a same SMTP server, and this last behaviour is always like this for Exchange.

NOTE: it's very important that all servers that are registered in the ELSE have their clock synchronized via NTP. If it's not the case, you will get situations where QIDs are not sorted as you may expect it.{@style=color: red;}

As soon as you select a different QID in the list, the text and graphical view tabs are refreshed with corresponding data.

If you click on the "Refresh" button, the effect will be to refresh the data for each QID of the list. It will not be to refresh the list of QIDs. So, if we imagine you have requested the details of an ongoing email, in the case of a huge sending operation on a mailing list for instance, don't expect to see QIDs not listed in a previous refresh. If you want to see newly generated QIDs related to an ongoing email processing, close the window and perform your search again.

Text view: Postfix transaction details

"Search" tab: here are the various sections you can potentially see for a Postfix transaction

This screenshot shows you the various sections you will get about a Postfix transaction. Those sections are closely related to the name of the daemon forked by the postmaster (qmgr, smtp, smtpd, postsuper, ...):

• Processing server: the Postfix server on which another server has connected to for sending an email
• Header: the header of the transaction, as captured by the Postfix server (if configured to do so). Each line composing the header is limited to max. 200 characters long: this is a Postfix limitation, not an ELSE one. By default, the header section is not displayed: you've to click on it to see it.
• Attachments: details about all files attached to the e-mail: should be displayed for each attachment: the file name, file size in bytes, and date of creation. Attachments are organized by MIME type.
• Connecting client: details about the client server that connected to the Postfix server. This includes IP Address geographic location: if the IP Address is not part of a private class, and if the ELSE is able to find some data about geographic location, then the IP Address will be displayed on a Google MAP.
• Subject/Helo: details about the Subject of the email and the HELO/EHLO provided to the Postfix server. The Subject is available only if your Postfix server has been correctly configured to provide it.
• Sender / E-mail information: all information about the sender and various data about the e-mail itself like its size in bytes and the number of recipients. Also new in from version 0.9.17 of the ELSE, the reputation indicators are given for both RFC821 and RFC822 senders. Those indicators can evoluate between -100% (worst reputation) to +100% (best reputation) passing through 0 (neutral). like in real life, reputation is extremely fast to destroy, and extremely long to generate. The reputations, and that's also the same for recipient reputations, is computed according the status of the emails that are sent and received. Different weights are used according if the status is "sent" successfully, "deferred", "rejected" or "bounced". For instance, if sender email address A successfully sent an email to recipients B, C, D but recipient E generated a bounce, then the reputation of the sender email address A will be credited 3 times and decreased once, the reputation of B, C and D recipients email addresses will be increased once for each, and the reputation of email address E will be decreased. By default, their is a factor 100 between good reputation increment and bad reputation decrement.
• ALERTS!!!: This section will appear in the window only if the ELSE has detected potential abnormal behaviours. In this case, a set of alerts are displayed for your information. Those alerts can be considered absolutely normal by messaging administrators sometime, or not. This can give an idea if the email transaction is a legitimate one or not.
• Recipient's temporary delivery failure information: this is a data grid containing all the attempts on each recipient to send the email, that ended by specific delivery failures. Several details are available per line in this data grid, and even some columns that are not displayed by default and giving other details. You shouldn't see this section very often if all goes well.
• Recipient delivery tries information: this is a data grid containing all the attempts on each recipient to send the email. Several details are available per line in this data grid, and even some columns that are not displayed by default and giving other details.
• Bounce information: You can get several of those sections if some delivery ended in bounces. In addition, if your messaging administrators have configured the NDR capture feature of the ELSE, you'll also have the possibility to directly see the NDR captured here, that has been sent to the original sender. Especially interesting is the Bounce QID written in those sections, because if you copy/paste them in the "Search by QID field", you'll get all the details regarding the email generated for the bounce.
• E-mail removal from queue: details about the removal of the e-mail from the Postfix server queue, meaning the processing of the transaction by the Postfix server is terminated. If the transaction is still under processing, you'll see there a message saying "E-MAIL IS STILL IN QUEUE...".

And finally, see below an example of the Header section once displayed. You'll get the header with a color scheme:

"Search" tab: an example of a header captured by a Postfix server

Text view: Exchange transaction details

"Search" tab: here are the various sections you can potentially see for an Exchange transaction

Basically, we retrieve the same sections than for a Postfix transaction. But you have to keep in mind several differences:

• Various tasks done by the Exchange Server in the processing of the transaction have been mapped to Postfix sections when possible. So a QMGR step is simulated. A SMTPD step is simulated, and so on.
• The PID number in any section is -1, because this information is not provided in Exchange server logs
• The Header section will never show e-mail headers because header capture is not possible via simple ways
• A same Exchange Server generates several transactions for each processing step. The operation made on each transaction by an Exchange server is recorded in the ELSE self-generated Exchange QID number, and it is coded in the 4 last characters of this QID number. For example, the SMRE code means SMTP RECEIVE operation. ROHA means ROUTING HIGH AVAILABILITY. STRE means STOREDRIVER RECEIVE. And so on.

Here below are information about the additional sections you will get about an Exchange transaction:

• Exchange server SMTP session: gives you the commands exchanged during the SMTP communication. By default, the SMTP section is not displayed: you've to click on it to see it.
• Exchange server specific information: displays all information related to the specific Exchange server operation of the transaction.

Then, see below an example of the Exchange server SMTP session once displayed:

"Search" tab: an example of a SMTP session captured by an Exchange server

Graphical view

"Search" tab: transaction network map automatically generated by the ELSE

This view can be much better than the text view, because just by taking a look, you immediately know if the email has been delivered without any issue.

A color code is used on the user icons. It can be:

• Green: email has been delivered to the recipient
• Blue: email is deferred and in the server queue
• Red: email has been bounced for this recipient

If the transaction has generated NDRs, then they appear in a red color near the Processing Server, like on the screenshot above.

Be aware that the user icons shows a delivery attempt for a recipient email address. So, for example, if an email for a specific recipient has been deferred 6 times before to be delivered, you'll see 6 Blue user icons and 1 green one having the same recipient e-mail address. For each of them, you will then see that the TIME in seconds increased by some amount.

NOTE: WARNING! the number of recipient user icons displayed in the drawing is limited to 99. So, if the email transaction is more than 99 delivery attempts (you can have thousands of them for a specific transaction of a mailing list), you'll have to use the Text view instead.{@style=color: red;}

One last thing before to close this long chapter. If you get some alerts in the ALERTS!!! section of the text view for a specific transaction, you'll see an alerting symbol at the left of the user icon representing the sender in the graphical view.

Text view:

"Search" tab: Alerts that can be seen in the Text view

Graphical view:

"Search" tab: The way they are displayed in the Graphical view

---

Top: Documentation for Users | Previous: "Statistics" tab | Next: "Logs" tab
This page for version: 0.9.17 | 0.9.18