An overview of the email formatting tutorial. Formatting Emails in 4D This tutorial shows a quick and dirty way of formatting emails on screen in a similar fashion to email clients such as MS Outlook Express (when I say quick, I mean to say that it's pretty fast, and dirty, because the code is written to be fast compiled and not necessarily easy to follow). The idea is to take an email stored in a text field/variable or a BLOB field/variable, find all the hyperlinks and quoted text, then display a coloured version of the email in a 4D Write area, in a way so that the hyperlinks are activated when the user double-clicks them. The sample code has been lifted from Mail4th Express, the vapourware email client written with 4D 6.5. Although Mail4th Express has never been finished, these routines seem to work fine. The main bottleneck with this code is in the formatting the 4D Write document. It doesn't seem to matter whether this is done on or off screen, it's still slow in comparison to the parsing code. Emails that are bigger than about 64K take too long to format in our opinion. There is a faster way of creating 4D Write documents, but we will release something separately for that. Typically, on a Powerbook G3 500, parsing a 32K 4D NUG digest email and formatting in the 4D Write area takes approximately 1 second. The examples have been designed to work with 4D v6.5 and 4D Write v6.5, although they should work with v6, the 4D Write code would definitely need looking into to ensure compatibility with earlier versions. They've also been tested with 4D/4D Write 6.7. Important: Forget about using this code in an interpreted database unless you're only viewing extremely small emails. The routine used for parsing and formatting emails has been optimised for compiled running and will be extremely unpleasant for your users if you run this in interpreted mode. Example database - Uncompiled 4D 6.5 Structure Mac Email_Formatter65.sit.bin MacBinary 80k Mac Email_Formatter65.sit.hqx BinHex 108k Windows Email_Formatter65.zip ZIP 124k

What are we looking for in the email and what are we then going to do with it... What are we formatting? One of my favourite features in email clients such as Claris Emailer and later Microsoft Outlook Express was the fact that they displayed plain text emails with coloured quoted text and hyperlinks that actually did something. This was one thing that swung me away from using Matthew Whyte's Digest Reader for reading NUG Digests. I wanted to have a similar effect as in Outlook in 4D so that I could integrate it into DigestReader, then later into my own 4D based email client. The main things we need to format in the emails are quoted text, i.e. lines beginning within one or more ">". Putting these in different colours depending on the level of quotation greatly enhances the readability of emails, especially over long threads. Allowing the user to select their own colour scheme is also a big bonus. Take the following example without colour.... And the next example with colour... The coloured example is clearer in terms of the level of quotation and your eyes more directed more towards the new text. The colours are set differently depending on how many ">" signs were found on the line before the text started. Achieving this in 4D requires using 4D Write. 4D Write allows you to programmatically create and format documents and makes a pretty good text editing area even when not using it for writing letter or doing mail-merge. If you already use 4D Write in an application and want to display formatted email, this is one of the best ways of displaying it. 4D Write allows for a lot of customisation letting you provide your users with applications which look and feel like their stand alone counterparts. Hyperlinks Another thing email clients like Outlook Express do is present email address and website addresses as hyperlinks so that the user can click directly in the email to open a website or send a mail to someone. Opening a Web site in the preferred browser has been easy since 4D 6.5 as is opening a new addressed email message in your favourite email client. This presents 2 problems, one of which is completely solved by upgrading to 4D Write 6.7. First, we need to find everything in the email that could be a link. This could be made easier using regular expressions, but as of writing this article, the only available plug-ins is from Escape, and at the time of write the code, this was not available. The types of things we're looking for are email addresses enclosed in "<" and ">", e.g. <mark@mitchenall.com>, links beginning with either "http://", "mailto:", "ftp://", "gopher://", etc. These all need to be formatted as hyperlinks in a way that they can be easily clicked to open them. In 4D Write 6.7, they can be simply converted to hyperlinks as this is a new feature in 6.7. In 6.5, one of the easiest things to do is to convert the links to references, then trap click events on them. In 6.5.4 there is even a new style so that when the mouse is put over a correctly styled referenced, the mouse pointer turns into a finger. In the following example we will show the difference between the plain-text version of a part of a digest email, and the styled version using hyperlinks where appropriate. First the plain text... And the next example with colour and hyperlinks... All of the effects in the last example are possible using 4D and 4D Write version 6.5 onwards. To get the "mouse pointer turning to a finger" effect with hyperlinks, you need to be using 4D Write 6.5.4 or later.

Here we'll look at the benefits of using a BLOB to parse this sort of data as quickly as possible as well as other types of data. Parsing Email in BLOBs There are 2 reasons that the email is parsed in BLOBs in this example. Firstly, emails aren't always necessarily under 32K in size. If we used a Text field or variable, we run the risk of overflowing the variable, or creating more of a nightmare handling the email in chunks. Secondly, as you will see from a few of the examples below, parsing text in BLOBs is no more difficult than parsing text in text variables when you want to do it with the fastest possible performance. Try out the following code example using any version of 4D from v6.0.x onwards in compiled mode. See what sort of differences you get with the different loops. Note that they call complete the same number of iterations, but they all take different times to complete, some significantly different. The point of the code is to help drive home the point that text parsing doesn't have to be done using text variables and that 4D can do extremely fast parsing if it's done the correct way. C_LONGINT($startTicks;$endTicks) C_LONGINT($j;$i) C_TEXT($textVariable) C_BLOB($blobVariable) C_STRING(255;$stringVariable) C_LONGINT($longintVariable) $textVariable:=" "*256 SET BLOB SIZE($blobVariable;256) $stringVariable:=" "*256 $longintVariable:=0 `Test character matching on Text variables $startTicks:=Tickcount For($j;1;10000) For($i;1;256) If($textVariable_<$i>_=" ") `do nothing End if End for End for $endTicks:=Tickcount ALERT("Text variable character match - time to complete: "+String($endTicks-$startTicks)+" ticks") $startTicks:=Tickcount For($j;1;10000) For($i;1;256) If(Ascii($textVariable_<$i>_)=Space) `do nothing End if End for End for $endTicks:=Tickcount ALERT("Text variable ASCII match - time to complete: "+String($endTicks-$startTicks)+" ticks") `Test character matching on String variables $startTicks:=Tickcount For($j;1;10000) For($i;1;256) If($stringVariable_<$i>_=" ") `do nothing End if End for End for $endTicks:=Tickcount ALERT("String variable character match - time to complete: "+String($endTicks-$startTicks)+" ticks") $startTicks:=Tickcount For($j;1;10000) For($i;1;256) If(Ascii($stringVariable_<$i>_)=Space) `do nothing End if End for End for $endTicks:=Tickcount ALERT("String variable ASCII match - time to complete: "+String($endTicks-$startTicks)+" ticks") `Test character matching on BLOB variables $startTicks:=Tickcount For($j;1;10000) For($i;1;256) If(Char($blobVariable{i})=" ") `do nothing End if End for End for $endTicks:=Tickcount ALERT("BLOB character match - time to complete: "+String($endTicks-$startTicks)+" ticks") $startTicks:=Tickcount For($j;1;10000) For($i;1;256) If($blobVariable{$i}=Space) `do nothing End if End for End for $endTicks:=Tickcount ALERT("BLOB ASCII match - time to complete: "+String($endTicks-$startTicks)+" ticks") If you run this code compiled and check the results, you will notice that the comparisons using ASCII codes rather than character comparisons. In our email formatter, much of the time we're matching characters that have no uppercase or lowercase equivalent, or we're matching the starts of links or email addresses that don't contain accented characters in the prefix we're trying to match. The number of iterations in the example code might look high for searching through 32K emails, but when you consider the number of potential longint comparisons which need to be done on each of those 32,000 or so bytes in order to find the quotes and hyperlinks, 2.5 million iterations probably gives a reasonable test. For each character in the email we're going to check the state of the parser, then do different comparisons depending on the parser state. So reading the text via a BLOB or a text variable using character reference symbols and the Ascii function, we can parse text for particular characters the fastest. In this case, with 4D NUG Digests often being near to or greater than 32K, using BLOBs to parse the email and 4D Write to display them means we will never have any worries with large digests. Parsing text this way requires the parser to be more aware of its state at any one time. Rather than checking the input string for whether it matches a particular match string when looking for hyperlinks, we check whether the current byte in the BLOB matches the ASCII code of the current character in the search string. Once the match reaches the end of the search string with every character matching, the parser goes into a state where it knows it's found a match and now it's just waiting for something which ends the hyperlink. The parser used for gathering information about the positions of quoted text and hyperlinks within an email needs to check every character within the email at least once. Therefore, the main parser loop increments the byte offset within the email BLOB once per iteration, and state variables determine the quickest path through the various cases to determine the significance of particular characters. The parser is contained completely within a single routine, as are the formatting routines for 4D Write. This was done to optimise the performance of the parser and to keep all the code in one place relying almost completely on local variables. When comparing characters in the email blob to the various strings we need to match, the match is always carried out using the ASCII code. Even when looking for upper and lower-case versions of "http://", the ASCII code of each character is used instead of the character itself. That means we need to do two matches per character for each match string, but it's still quicker than comparing the two characters directly. The state variables keep track of the state of the parser at any one time. When the parser comes across a new line character, it needs to be looking to see if the text on the next line is quoted. Then once it knows whether or not the text is quoted, it needs to start trying to find text that looks like it could be converted to a hyperlink. On each iteration of the parser, a case statement checks the state of the parser, then does the character comparisons based on the state, so time isn't waste checking for things which couldn't possibly exist at certain locations within the email message. To view the complete code for the email parser, download the sample database which accompanies this article. It's very small and the code is well commented to show how it works. Compiler Range Checking In the sample database, the parser routine takes the 4D Write Area reference and a pointer to the BLOB containing the email as parameters. One of the first things the routine does is call COPY BLOB to copy the source BLOB into a local BLOB variable. This dramatically increases performance as the following code example will show. When the parser was first written, it was written to be as fast as possible without the need for things like Range Checking. There could never be a range-checking error with this parser, no matter what you throw at it. When it was first tested for speed, it was compiled with Range Checking switched off and when testing the speed of parsing certain emails, the parser was so quick, it registered zero ticks on the timer. After turning Range Checking on in the Compiler, the difference was so little as far as user perception goes, it's not worth worrying about (e.g. 2 ticks, instead of zero).

How to use 4D Write to display emails in their Outlook Express like glory. Colouring Text and Hyperlinks Since first writing this tutorial there has been a new release of 4D and 4D Write. In 4D Write 6.7 there is now a new hyperlink feature that allows users and developers to embed hyperlinked text into their documents much more easily than before. The sample database was written to work with 4D Write 6.5 and will still work fine in 6.7, although you may want to update it slightly so that it uses the new hyperlink features. Later I we release a 6.7 version as a component to make installation and use easier along we even faster performance. During development, a number of ways of parsing and building the formatted 4D Write areas were tested. It was through this testing that it was decided to use a BLOB to hold the email message during parsing. Another thing we discovered was the difference in speed between creating the formatted email in the 4D Write Area as it was being parsed, against, finding what needed to be formatted and where it was, then putting all the text into the 4D Write area in one (or more if there is more than 32K of text) hits and formatting by selecting the areas individually. The latter was by far the quickest. First the BLOB contain the email message is parsed to find the start and end positions of all the quoted text and what level of quotation it is. This lets us know which text to select and what colour to set it to. During the same parsing loop, we look for potential hyperlinks, and when they're found, we again store the start and end positions of them. Two sets of arrays are used to store the positions and in order to give the fastest possible performance, they are incremented in blocks of 100 elements. Once the parser has finished finding the positions of everything that need to be altered in the 4D Write Area, we can start formatting. First, the colours are set to the quoted text. This requires looping through the arrays of start and end positions of quoted text, selecting the text in the 4D Write document, then setting the appropriate colour. To cut down on the number of calls to the 4D Write area, don't necessarily colour each line separately. If two or more contiguous lines are the same colour, we select and colour them in one go to cut down on calls to 4D Write. Hyperlinks in 4D Write 6.5 Implement hyperlinks in 4D Write 6.5 is much the same as it was in much easier versions of 4D Write. There was an example database with 4D version 2 that demonstrated hypertext using 4D Write. 4D Write 6.7 makes the task a whole lot easier with the new commands and hyperlink features, but it has still been possible to implement hyperlinks for a long time. First you need to create a function that is going to return the text you see displayed as your hyperlink. e.g. `Method: hyperlinkText(linkText:text) C_TEXT($1;$0) $0:=$1 Then you need to insert an expression into your 4D Write area that replaces the text the user or you have selected, but which puts the text into the expressions as the first parameter. You're better off setting the colour and style of the link before you replace it as it saves selecting it again. e.g. `Select text and replace with a hyperlink expression (4D Write 6.5) WR SET SELECTION($writeArea;$linkStart{$j};$linkEnd{$j}) WR SET TEXT PROPERTY($writeArea;wr text color;wr blue) WR SET TEXT PROPERTY($writeArea;wr underline;wr continuous underline) WR SET TEXT PROPERTY($writeArea;wr links appearance;wr unvisited links appearance) `only available in 4D Write 6.5.4 or higher $linkText:=WR Get Selected Text($WriteArea) $expression:="hyperlinkText("+Char(Double quote+)+$linkText+Char(Double quote)+")" WR INSERT EXPRESSION($writeArea;$expression) Now you've inserted your hyperlink into the 4D Write area. The mouse pointer should turn to a finger when you put it over a hyperlink reference. All you've got to do is trap the clicked event (or double-clicked if you prefer) to act on the user's request. You can install event handlers for the various 4D Write events using WR ON EVENT. In our case, we're going to have an event handler for Clicks. When a click event occurs, we need to find out if the user clicked on one of our hyperlink references, and if they did, launch the URL using OPEN WEB URL. Here we provide a simple version for demonstration purposes. `Install 4D Write Event Handler WR ON EVENT($writeArea;wr on single click;"write_OnClicked") `Method: write_OnClicked WR GET REFERENCE($1;$table;$field;$name;$type) If (($type=2) & ($name="hyperlinkText@")) $url:=Substring($name;Position(Char(Double quote);$name)+1) $url:=Substring($url;1;Position(Char(Double quote);$url)-1) OPEN WEB URL($url) End if

Some other ways in which emails can be parsed and formatted in 4D. Alternatives When I first started writing this tutorial, when 4D version 6.5 first shipped, writing my own parser for finding quoted text and hyperlinks in an email seemed like the best way to do the job. It also helped me further understand how to write fast code in 4D and that some of this code can be extremely fast in 4D if written correctly. That said, there is now a great alternative to this method. Regular Expression Matching This is a feature that really should be built into 4D itself, but it is still possible to use Regular Expressions in 4D. Escape, the makers of such 4D plug-ins as QPix and QMedia, have released a freeware plugin QFree 2.0 which contains 4 exceptionally useful functions for doing regular expression matching on BLOBs. These searches are extremely quick, especially if you design a fast Regex pattern. Using a couple of regular expressions, you could achieve the same results as the parser presented in this tutorial with just a few lines of code. In all, it would probably be about the same speed if not slightly slower in parsing the email using the native 4D option, but the code would be very much cleaner. These four Regular Expression routines in QFree make it a must have plug-in for any 4D Developer. Developers have been using Regex patterns for years in tools like BBEdit, and in other languages such as Perl, PHP, etc. developers use regular expressions all the time. If enough 4D developers start asking for these features, hopefully 4D will eventually add them into the 4D language itself. Regex for Matching Quotations The following regular expression used with the QF_REExtract function return the positions and lengths of all quoted lines of text in an email contained in the BLOB passed. ^([\s]*[>]+[\s]*)+.*$ This matches all quoted lines, no matter what level they're at. You would still need to check how many greater than signs there were yourself to know the level. Still, it cuts the code down considerably. Regex for Hyperlinks This next Regex gets you all the things which look like they should be turned into hyperlinks. ((mailto:)?<?([-a-zA-Z0-9_%.]+@[-a-zA-Z0-9_]+([.][-a-zA-Z0-9_]+)+)>?)|([a-zA-Z]+://[-a-zA-Z0-9_]+([.][-a-zA-Z0-9_]+)+/[-a-zA-Z0-9_%/.]*) Unlike our parser, this regex matches more than things that start within "http://", etc. This also matches things which look roughly like email addresses, even if they aren't wrapped in "<" and ">". It's also not concerned with the case of the URL or start of the URL. Using QF_REExtract all the position and lengths of the links in the email are returned neatly in an array so that you can then go about the task of converting them to hyperlinks in the 4D Write document. The quickest way to try these out is to simply open a 4D Digest in BBEdit or an equivalent windows text editor, such as HomeSite, and try the regexes out there in the Find function (i.e. the "Use Grep" checkbox on BBEdit). Later we will release a version of the parser that simply uses QFree. But depending on the speed, we may still keep it all in 4D Code. That said, the 2nd regex here does a better job at finding links and email address than our parser in most cases.