A short discussion on regular expressions and their use within 4D. Using Regular Expressions in 4D Regular expressions are an extremely powerful tool for describing text. They're often included as a power user feature in many text editors and word-processors and many langauges and development environments have built-in support for them. Regular expressions are a late entrant into the 4D family of products, and even now are only available through 3rd Party Plug-ins. Hopefully, 4D will address this and have native support built-in, but for the moment, the choices available for using regular expressions in 4D are fairly good. This article will not teach the reader how to write regular expressions, but rather give some ideas of where and how regular expressions can be used with 4D. It will hopefully show readers new to regular expressions just how useful they can be and give an incentive to go and learn how to write them for themselves. I am by no means an expert on regular expressions, but I'm learning more all the time and I encourage those 4D developers who currently don't know how to use them to learn about them, especially if any of your work involves text parsing of any kind. There is an excellent book on regular expressions entitled, "Mastering Regular Expressions" by Jeffrey E. F. Friedl, which is an absolute must for anybody who wants to use and eventually master regular expressions. It's the first book I've read on the subject which does a good job of explaining how to use regular expressions to someone who has never used them before. Again, I will not try and teach regular expression in this article, but will instead provide links where you can find more information for yourself. Since discovering Regular Expressions (or Regex patterns) a year or so ago I have wondered how I managed to live so long without them. I now use them regularly when programming in PHP or 4D, but also for editing HTML and other source code. For example, using regular expressions, creating the hyperlinked 4D source code on this website was very much easier than you would think. Creating the index of all the 4D commands was done with 1 simple regex applied to each page in the index of the manual. Converting the code was also done with a series of simple regular expressions and some other find and replace operations to create the XML. Then the PHP XML process puts the two together and along with the stylesheet I get formatted 4D code with hyperlinks on the commands. Using regular expressions this took no time at all and is completely reusable. Note: This article will remain a "work in progress" for a while as we find out new ways to entice 4D developers into using regular expressions. Expect fairly regular updates to this article, if you would like to contribute some ideas, I'd love to hear from you. Planet 4D The first part of a tutorial on using Regular Expressions aimed at 4D developers is available in Planete 4D Issue 7, the French journal for 4D developers. Visit the Planete 4D website to order your copy.

This section shows how and when Regular Expressions can be used within the 4D environment. When can Regular Expression be used in 4D? Regular expresions can be used both in your applications as an effective means of What tools are available? At the time of writing there are two main options available to 4D developers wanting to make use of regular expressions. QFree 2.0 contains 4 fantastic functions for using regular expressions within your applications. QuickCodePro 4.2 (since b5) now includes "Find and Replace" in your 4D methods using regular expressions. Let's look at the two options in more detail. Using Regular Expressions in your applications There are a huge number of times when regular expressions can be used with great effect in your applications. With 4D becoming ever more internet centric, the ability to use regular expressions has become ever more important. Regular expressions are extremely useful for text processing operations in 4D. For example, validating user input, parsing and importing non-standard file formats, searching for all occurances of hyperlinks in a web-page, extracting email addresses from documents. All this can be achieved without using the power of regular expressions, but in order to make the code efficient, code-readability sacrifices need to be made and each parser tend to be fairly inflexible. This depends on the complexity of the task in hand, but with regular expressions, complex text matching can be carried out with the minimum of code and also at very reasonable speeds, even when running interpretted. 4D has no regular expression support built-in at this time, but there is a plug-in available from Escape which does the job admirably. QFree 2.0 is a free, unsupported plug-in containing a number of functions including Windows Taskbar functions, RGB Picker dialog, some URL functions, some QuickTime Atom functions, and what we're most interested in here, Regex functions. Although QFree is unsupported, I have so far had no problems with it and Escape will not leave you completely out in the code if you do have a problem. The 4 Regex functions are QF_REMatch, QF_RESubstitute, QF_RESplit and QF_REExtract. I will cover each of these commands in turn below. For the moment, I'll just say that just these 4 commands add a huge amount of power to your 4D toolbox. QFree's demo database comes with a "Regex Lab" which is simple dialog which lets you test your regular expressions out with the various options to ensure they work before setting out coding for them. It has pages with a tab control which lets you try out each of the different regex functions and see the results instantly. There are fields to let you enter your regex pattern, target text and substution text or submatch groups depending on which function you're using, plus checkboxes for each of the different additional flags which can be set for the regex engine. This simple dialog lets you get working with regular expressions in 4D quickly and examine the results. It's also an good source of example code to see how it's put together. For a free piece of software it's extremely nicely packaged. The documentation gives a fairly thorough overview of the regular expression syntax which is supported, but I think the sample database is what makes it a really fantastic piece of kit. Escape have done the 4D community a big favour putting so much effort into a piece of freeware. QFree uses Perl Compatible Regular Expressions (PCRE with certain POSIX extensions, so finding documentation on the web is very straightforward. The O'Reilly book "Mastering Regular Expressions" is really a must for anyone wanting to really get into using Regular Expressions and contains plenty of information on Perl regular expressions. QFree is cross-platform, so you can use these features on both platforms without any difficulties. Using Regular Expressions in the Method Editor A common feature in many text editors, especially those aimed at programmers, is the ability to use regular expressions for finding and replacing text within the current document. I'm sure we're all familiar with the "Use Grep" or "Use Regular Expression/Regex" checkbox in editors such as BBEdit and HomeSite and other popular HTML editors. This feature has been off limits to 4D developers until Automated Solutions Group added the support for Regex matching in QuickCodePro 4.2b5 (QCP). In the past, if a developer wanted to be able to use Regex substitutions on 4D code, he/she would have to copy the code to an editor such as BBEdit and perform the substition there, then copy the code back to 4D when complete. Now this can be done directly in the method editor via QCP's Find and Replace dialog. QCP currently allows you to apply regex matching and substitutions on the entire method, or all open method windows. Being able to apply regex substitutions across all open method windows allows you to easily perform tasks like replacing all occurances of a function call with another, but swapping the positions of the parameters at the same time. Also, ensuring the correct capitlisation of a group of variables or enforcing style guidelines on blank lines and comments. Later we will cover some specific examples. Like QFree, QCP uses PCRE so what you use in one should work in a similar fashion in the other. QCP is a commercial piece of software costing around $200 US per developer for a copy, but for the time savings you can gain by having a copy and spending a short time every month flicking through the manual, it's well worth it. It practically pays for itself the first day you own a copy even without the regular expression support. With this added benefit it makes the 4D method into a more serious piece of code editing equipment. The only problem with QCP is that it's currently only available for the Macintosh, but hopefully this will change with version 5.

Here we will look in more details at using regular expressions in your applications and the types of things they're useful for. Uses for Regular Expressions in your applications A common feature in databases which store contact details is to store a person's email address and possibly website address. Checking whether these details have been entered correctly is fairly straightforward, but still fiddly and a multi-line code affair. Using a regular expression, it's an extremely simple operation. It's even possible to find strings which look like email addresses or website addresses in a larger block of text using only a slightly modified version of the regex. Parsing and Validating Email Addresses In the case of an email address, e.g. mark@mitchenall.com, we're looking for a piece of text with the following characteristics... Starts with the username which can contain letters, numbers, full-stops, hyphens, and underscores. An '@' sign follows the mail username. The server address will normally consist of 2 or more groups of letters, numbers, hyphens or underscores (never less than 2 groups) seperated by full-stops (periods). The address is not case sensitive. This is only a very brief description and we could certainly still enter a non-conforming email address if we only checked for these characteristics. But how much 4D code is necessary to write this? Using QFree 2.0, very little 4D code is necessary. Here's an example: `Method: Validate_EmailAddress(emailAddress:text) --> Valid:boolean `Author: Mark Mitchenall C_TEXT($1;$emailAddress) C_BOOLEAN($0;$valid) $emailAddress:=$1 $valid:=False C_BLOB($regexTarget) TEXT TO BLOB($emailAddress;$regexTarget;Text without length) C_TEXT($Regex) $Regex:="^([-a-zA-Z0-9_.]+)@(([-a-zA-Z0-9_]+[.])+[a-zA-Z]+)$" ARRAY LONGINT($matchOffsets;0) ARRAY LONGINT($matchLengths;0) C_LONGINT($targetOffset;$flags) $targetOffset:=0 $flags:=0 C_LONGINT($Err) $Err:=QF_REMatch($Regex;$regexTarget;$ targetOffset;$matchOffsets;$matchLengths;$flags) If($Err=0) If($matchLengths{0}=Length($emailAddress)) `We have a successful match $mailUsername:=Substring($emailAddress;$matchOffsets{1}+1;$matchLengths{1}) $serverDomain:=Substring($emailAddress;$matchOffsets{2}+1;$matchLengths{2}) $valid:=True End if End if $0:=$valid The piece of code above checks the validity of an email address based on the rules we made up earlier. Notice that using the regex we also found the positions of the mail username and the server domain. In some applications, having these two seperate may be useful. For example, to further check the validity of the address, you could lookup the domain name to see if it exists. By extending the pattern slightly we could also parse more complex email addresses, such as those contained in emails themselves, e.g. "Mark Mitchenall" <mark@mitchenall.com> Writing a regex to parse the different parts of this address isn't really any more complicated than just testing the addresses approximate validity, but writing a fully conforming regex is pretty hard work. By the end of the book, Jeffrey Friedl shows a complete regex for matching an email address. With every possible option covered it's over 6,000 bytes long. For most purposes you can get away with something very much simpler. Parsing URLs The same applies when writing applications which need to deal with web addresses. Users who are inexperienced with the internet regularly have problems entering website addresses correctly. Using a regex you can quickly decide whether what the user entered is correct with a minimum amount of code. But also, you can extract more important information from the URL which can be used to potentially access or test the site using 4D Internet Commands. e.g. using a simple HTTP client in 4D, you generally need to provide the server address and page URL seperately. This is fairly easily to do in 4D code, but using a regex is a much more elegant way. Take the following URL as an example: http://www.mitchenall.com/resources/library/4d/regular_expressions/index.phtml If we split the URL into the different parts we could get: (scheme)://(host)/(path) We could go one further and split URLs completely, e.g. http://user:password@www.host.com:1234/sample/uri?query#fragment Escape have used this as an example in the QFree demo database, so I wont repeat the code here (see screenshot below). The code is fairly straightforward and can be easily reused in any project. In their example they use 2 regular expressions in order to get each part of the URL separately. Web developers using languages such as Perl and PHP have been using these techniques for years and if more 4D developers start complaining of the lack of support directly in 4D, perhaps 4D SA will do something about it. It would be especially good if they use QFree as an example of one way of achieving it. Another example of parsing URLs is when you want to find out what other HTML pages a particular page refers to. This is useful in many web applications, including robots and spiders but also website management tools. Using regular expressions this is now easy to achieve in 4D. Again, Escape have provided a good example of this in the QFree Demo, so I strongly advise you to download it and start playing. The example in the QFree sample is actually for parsing Bookmark files, but it works just the same with HTML pages as shown in the screenshot below. Parsing 4D Digests into individual messages Recently, the 4D NUG changed it's name. This in turn caused the long running Digest Reader 4 to break and it would no longer read digest messages. Also, there's been the addition of another English mailing list for 4D (DataBasics, see links page) which I'd also like to parse the digests for. It occured to me that there's also plenty of other mailing lists with digest options which don't get parsed by Outlook Express (also, I don't like the way Outlook Express does it), so I decided to replace the reasonably complex parsing code in Digest Reader by Matthew Whyte into a simplified version which simply used a Regex pattern. The advantage of this is that only the regex needs to be modified to be able to read different digests, so supporting different mailing lists with the same package become extremely simple. Also, if the format of the list ever changes, Matthew would only have to supply a new Regex for people to enter into their preferences. The QF_REExtract command is perfect for this task. It allows you to extract all matches and submatches of a regex pattern in a BLOB into two 2D arrays in just 1 line of code. Using these two arrays you can then easily extract the necessary text from the BLOB to do with as you please. In the case of Digest Reader, I needed to extract each individual message, plus for each one, the subject, sender's name, sender's email address, date and message contents. This was all achieved using one pattern and extracting only certain submatches. The new code to extract the messages from a digest stored in a file looks something like the following: `Method: digest_Parse `Author: Mark Mitchenall `Created: 12/24/00 at 4:01 PM C_TIME($docRef) $docRef:=Open document("";"TEXT";Get Pathname ) If (OK=1) `Load the Digest document into a BLOB DOCUMENT TO BLOB(document;$digest) `We now know what kind of Digest it is, so we just need to use the correct regex `to extract the messages, plus extract the right groups. C_TEXT($regex;$groups) C_LONGINT($flags) $regex:="!!!!!!! INSERT REGEX HERE !!!!!!"`Get the Message Regex from the mailing list record here. $groups:="1 2 4 5 7 8"`These are the submatches. Should be in the record $flags:=(qf_RECaseless+qf_REDotAll+qf_REUngreedy )`This longint should be in the record `Initialise the 2D arrays which will hold the results of the extraction ARRAY LONGINT($matchOffsets;0;0) ARRAY LONGINT($matchLengths;0;0) `Do the message extraction using QFree regex routine C_LONGINT($err) $err:=QF_REExtract($regex;$digest;$groups;$matchOffsets;$matchLengths;$flags) C_LONGINT($numMessages;$messages;$offset;$length)) `Now we just need to save those messages which the user hasn't asked to be `filtered out. $numMessages:=Size of array($matchOffsets) For ($messages;1;$numMessages) CREATE RECORD([Articles]) $offset:=$matchOffsets{$messages}{1} $length:=$matchLengths{$messages}{1} [Articles]Subject:=BLOB to text($digest;Text without length ;$offset;$length) $offset:=$matchOffsets{$messages}{2} $length:=$matchLengths{$messages}{2} [Articles]FromName:=BLOB to text($digest;Text without length ;$offset;$length) $offset:=$matchOffsets{$messages}{3} $length:=$matchLengths{$messages}{3} [Articles]FromMail:=BLOB to text($digest;Text without length ;$offset;$length) $offset:=$matchOffsets{$messages}{4} $length:=$matchLengths{$messages}{4} [Articles]ArtDate:=BLOB to text($digest;Text without length ;$offset;$length) $offset:=$matchOffsets{$messages}{5} $length:=$matchLengths{$messages}{5} COPY BLOB($digest;[Articles]Article;$offset;0;$length) SAVE RECORD([Articles]) End for End if You may noticed that I left out the regex pattern itself. This is because I didn't do a complete implementation, but left it so that Matthew could decide for himself how he wanted to store the regexes. The pattern in the demo was stored in a TEXT resource as these are fairly easy to get to and modify in a hurry. The pattern used to extract the messages was as follows (Note: here it's split over more than one line, but in reality it's all one line. It's just presented this way for readability): \r\rSubject: (.*) \rFrom: (["]?(([-a-z0-9_%.\s]+)["]?)?[\s]?<([-a-z0-9_%.]+@[-a-z0-9_]+([.][-a-z0-9_]+)+)>) \rDate: (.*) \r\r(.*) \r\r-{70} From this we extract the groups 1, 4, 5, 7, and 8 to get the Subject, Sender's name (without the quotes), Sender's email address (without the <>s), the date and the message respectively. When extracting more than 1 submatch, QFree returns its results as two 2D Longint Arrays giving the positions of each match and the lengths. If we look at the following example of a 4D NUG digest we can see below have this will be parsed by QF_REExtract with the above regular expression. Subject: 4D Tech Digest - V11 #234 - 03/11/01 Date: Sunday, March 11, 2001 8:15 PM From: 4D Tech Mailing List <4d-tech@lists.4dnug.com> To: 4D Tech Mailing List <4d-tech@lists.4dnug.com> 4D Tech Digest - V11 #234 - Sunday, March 11, 2001 QFree is a cool plugin by "Mark Mitchenall" <mark@mitchenall.com> Re: QFree is a cool plugin by "Joe Bloggs" <joe@bloggs.com> ---------------------------------------------------------------------- Subject: QFree is a cool plugin From: "Mark Mitchenall" <mark@mitchenall.com> Date: Sat, 10 Mar 2001 22:31:30 -0800 QFree is a really cool plug-in. Everyone should download themselves a copy and start getting into using the regex functions immediately. Best, Mark ---------------------------------------------------------------------- Subject: Re: QFree is a cool plugin From: "Joe Bloggs" <joe@bloggs.com> Date: Sun, 11 Mar 2001 00:20:12 -0800 "Mark Mitchenall" <mark@mitchenall.com> wrote: > QFree is a really cool plug-in. Everyone should download > themselves a copy and start getting into using the regex > functions immediately. Just downloaded it and realised how cool it was. Thanks! Joe ---------------------------------------------------------------------- End of 4D Tech Digest - V11 ********************************************************************** 4th Dimension Networked Users Group (4D NUG) FAQ: http://www.4dnug.com/4d_mailing_list_faq.html Resources: http://www.4dnug.com/resources.html Admin: Karen R Sabog <4d-admin@4dnug.com> Unsub: mailto:4d-tech-off@lists.4dnug.com ********************************************************************** The sample code below shows the values populated into the arrays by QF_REExtract. You can think of the 2D arrays as table where the first dimension is for each row, or in this case match of the regular expression, and the 2nd dimension being the column or submatch group, e.g. $matchOffset{row}{column} or $matchOffset{match}{submatch}. One thing I forgot to mention is that column zero, i.e. $matchOffset{row}{0} always contains the entire match, so in this case, the whole message, plus the headers at the top and the row of hyphens at the bottom. `First Message in digest $offset:=$matchOffset{1}{1}`position of start of 1st subject text $length:=$matchLengths{1}{1}`length of 1st subject text $subject:=BLOB to text($blob;Text without length;$offset;$length) `$subject = "QFree is a cool plugin" $offset:=$matchOffset{1}{2}`position of start of 1st sender's name $length:=$matchLengths{1}{2}`length of 1st sender's name $name:=BLOB to text($blob;Text without length;$offset;$length) `$name = "Mark Mitchenall" $offset:=$matchOffset{1}{3}`position of start of 1st sender's emailaddress $length:=$matchLengths{1}{3}`length of 1st sender's emailaddress $address:=BLOB to text($blob;Text without length;$offset;$length) `$address = "QFree is a cool plugin" $offset:=$matchOffset{1}{4}`position of start of 1st date $length:=$matchLengths{1}{4}`length of 1st date $date:=BLOB to text($blob;Text without length;$offset;$length) `$date = "Sat, 10 Mar 2001 22:31:30 -0800" $offset:=$matchOffset{1}{5}`position of start of 1st message body $length:=$matchLengths{1}{5}`length of 1st message body $body:=BLOB to text($blob;Text without length;$offset;$length) `$subject = "QFree is a really cool plug-in. Every...." `Second Message in digest $offset:=$matchOffset{2}{1}`position of start of 2nd subject text $length:=$matchLengths{2}{1}`length of 1st subject text $subject:=BLOB to text($blob;Text without length;$offset;$length) `$subject = "Re: QFree is a cool plugin" $offset:=$matchOffset{2}{2}`position of start of 2nd sender's name $length:=$matchLengths{2}{2}`length of 1st sender's name $name:=BLOB to text($blob;Text without length;$offset;$length) `$name = "Joe Bloggs" `...etc... One thing to remember with using this method of parsing the digest is that it's extremely fast, even when running interpretted. Also, it's fairly memory efficient as the BLOB is not copied in RAM and the return result is simply two 2D arrays of longints telling you the positions of all the strings you're looking for in the text and all with just one call to the plug-in. The way this has been implemented by Escape is excellent. It's a really elegant solution for many text parsing requirements.

Some more examples of regular expression usage in your applications. Another potential application for Regular Expressions QuickCodePro v4 is an exceptional product. For one thing it has made 4D Insider comments (the ones you can now edit in the Explorer window) an extremely useful tool. Firstly, you can have the parameters auto-entered if you put them in the comments. Secondly, the comments can be displayed in balloon help in the method editor. This gives developers even more reason to use them. During a project I worked on recently, there came a time when the system needed to be fully documented so that future developers would be able to figure the system out without a problem. The system had already been handled by 5 different developers at different times and documentation was sparce. I decided one of the easiest things to do would be to enter most of the information needed straight into the structure itself using the 4D Insider comments. This would give me the benefit of having on-line help for myself and future developers. But I also wanted to product other printed documentation and so I wanted to extract all these comments from 4D Insider exports allowing me to create documentation from them. Here regular expressions came to the rescue allowing me to extract the information from the exports into a database where it could then be exported to be worked on in the word processor. In the end I wrote a number of regex patterns to extract various information from the 4D Insider Exports. Because the structure of the exports wasn't as strict as I'd hoped, I had to export the various parts of the structure into separate files, i.e. one file for the comments, another for the fields and tables, another for methods, another for forms, etc. This didn't take long and once exported they take absolutely no time whatsoever to parse to be used in the documentation database. Parsing 4D Insider Exports First open your structure file in 4D Insider. Then show only the methods in the database. Choose "Export..." from the "File" menu and check only the "Documentation" checkbox and hit the Export button. This file will then just contain the comments for each project method as entered in the Explorer window. These unfortunately wont contain any of the colour or other text formatting, but you can easily extract all these comments and enter them into a database. The following regular expression, used with QF_REExtract returns the name, ID and size of the project method found in the BLOB which can then be used to extract the method contents for each one. \r+Project\sMethod\s([a-zA-Z0-9_\s]+)\s([0-9]+[/][0-9]+[/][0-9]+).\(ID.([0-9]+)...([0-9,]+).bytes\)\r+ The same format is used when you just export the project method contents using 4D Insider, so you can use the same regular expression for that if you wish to import this too. Once you have the contents of the project methods, you can easily parse them for common items. e.g. Parse object methods to extract the contents of all of the form event handlers or triggers for database actions. You can also easily search for all parameter declarations or even just local variables with great ease. You need to export the form object methods, form methods and triggers seperately and apply slightly different patterns to them, but it's still a fairly straightforward way of collecting all the documentation from your structure file. Another potential use of regular expressions is to find all comments in your source code so that they can be gathered and read separately from the source. Again, once you have this information in a database, you can easily export it to a variety of formats. e.g. creating an web-based documentation resources for developers, converting to RTF format for import into a word-processor or documentation tool, etc. The patterns used for form methods, object methods, fields and tables are: Form Methods Form\sMethod\s\[([a-zA-Z_][a-zA-Z0-9_]*)\][.]([a-zA-Z0-9_]*)\s([0-9]+[/][0-9]+[/][0-9]+).\(ID.([0-9]+)...([0-9,]+).bytes\)\r+ Groups: 1 - Tablename, 2 - Formname, 3 - Date, 5 - ID, 6 - Size Object Methods \r+Object\sMethod\s\[([a-zA-Z_][a-zA-Z0-9_]*)\][.]([a-zA-Z0-9_]*)[.]([a-zA-Z0-9_]*)\s([0-9]+[/][0-9]+[/][0-9]+).\(ID.([0-9]+)...([0-9,]+).bytes\)\r+ Groups: 1 - Tablename, 2 - Formname, 3 - ObjectMethodName, 4 - Date, 6 - ID, 7 - Size Tables \r+Table\s\[([a-zA-Z_][a-zA-Z0-9_]*)\]\s([0-9]+[/][0-9]+[/][0-9]+).\(ID.([0-9]+)...([0-9,]+).bytes\)\r+ Groups: 1 - Tablename, 2 - Date, 4 - ID, 5 - Size Fields Field\s\[([a-zA-Z_][a-zA-Z0-9_]*)\]([a-zA-Z_][a-zA-Z0-9_]*) Groups: 1 - Tablename, 2 - Formname A sample application showing these in action will be produced over the next few months. Using QFree to parse this from 4D Insider export files is extremely quick. The only current drawback is having to save each time of object seperately, and export the documentation for each type of object seperately. Still, this is only a minor drawback when you consider how fast the parsing is. In all my applications I use the same format for all my object methods when it comes to form events. I use a QuickCodePro macro each time I open a new object method which pastes my standard skeleton into the method editor, e.g. `Method: myObject `Author: Mark Mitchenall `Created: 13/3/01 at 12:48 AM C_LONGINT($formEvent) $formEvent:=Form event Case of : ($formEvent=On Load) : ($formEvent=On Clicked) : ($formEvent=On Data Change) : ($formEvent=On Unload) End case Because all my form methods and object methods use this format, I can easily parse them using a regular expression to find out which form events I've actually coded. If necessary I could then parse the individual event's code. The following regex pattern can be applied to an object or form method to retrieve which form event constants have been used. The submatch group gives you the form event constant and the whole match gives you the line with the case. Form Event Handlers \r.*[:]\s\([$]formEvent=(On\s[a-zA-Z0-9\s]*)\).*\r Groups: 1 - Form Event Constant As you will read in the next section, I have a standard way of presenting parameters in methods. This means it's very easy for me to use this information to add to the documentation or even help in the creation of some of the documentation. The following code sample is an example of the standard way I present parameters. `Method: myMethod `Author: Mark Mitchenall `Created: 13/3/01 at 12:48 AM C_TEXT($1;$textParameter)`some information about the 1st parameter C_LONGINT($2;$longintParameter)`some information about the 2nd parameter C_POINTER($3;$pointerParameter)`some information about the 3rd parameter $textParameter:=$1 $longintParameter:=$2 $pointerParameter:=$3 Once the contents of the method have been extracted from the 4D Insider Export, the following regex pattern can be applied using the QF_REExtract command to extract parameter information from the method contents. This can be very useful to help create documentation and Insider comments for use with QCP's HotHelp and AutoComplete features. Form Event Handlers C_([A-Z]+)\([$]([0-9]);[$]([a-zA-Z_0-9]+)\) Groups: 1 - Variable Type, 2 - Parameter Number, 3 - Local Variable Name In order to use these last couple of patterns, you need keep to a fairly strict style in your code. Personally, I like opening methods which take parameters in this way as I can give the local variable which takes the parameter a more meaningful name to help make the code more readable. This gives me the added benefit of having something which is easy to parse for documentation purposes later on. Using QCP Macros for creating skeletons for methods is a good way of maintain a coding style. As you will see in the next section, QCP also contains features to help save on typing these types of parameter declarations and assignments using AutoComplete and the new regular expression features.

Examples of using the preg_module for regular expression matching. Using the preg_module I wrote the preg_module after spending lots of time using PHP for web application development. PHP has two kinds of regular expression, standard POSIX compliant regular expressions, and Perl compatible regular expressions (PCRE). The Perl compatible regular expressions in PHP are based on a very similar library to those routines in QFree 2.0, but the routines themselves aren't quite as straightforward to use in QFree 2.0, although they are just as powerful. The preg_module contains 10 methods which allow the developer to use the QFree functions in a similar manner to the PHP Perl compatible regular expressions, although keeping QFree normal syntax for the regex patterns themselves. Here, I'm only going to show the routines for doing regex matching on text variables, but in the module, each of these routines also has a version for working on BLOBs. For more information on the BLOB versions of these routines, please download the module itself where there is further documentation. Function: preg_Match Earlier we looked at validating an email address using regular expressions. Although the code was fairly straightfoward, it still wasn't as easy as doing the same thing in a language such as PHP. The same code in PHP could have been written as: <?php function validate_emailaddress($emailaddress) { $pattern = '/^([-a-zA-Z0-9_.]+)@(([-a-zA-Z0-9_]+[.])+[a-zA-Z]+)$/i' ; if(preg_match($pattern, $address)) { return TRUE ; } else { return FALSE ; } } ?> The preg_module allows you to write your 4D version in a similar fashion to that in PHP. The following example shows how to write the above PHP script in 4D using the preg_module. `Method: validate_EmailAddress(emailAddress) --> TRUE if value C_TEXT($1;$emailAddress)C_BOOLEAN($0) $emailAddress:=$1 $pattern:="^([-a-zA-Z0-9_.]+)@(([-a-zA-Z0-9_]+[.])+[a-zA-Z]+)$(?i)" If (preg_Match ($pattern;$emailAddress)) $0:=True Else $0:=False End if As you can see, this routine is now very similar to the PHP example and for me, is easier to read what's going on. There is one main difference between the two, however, and that is the regex pattern itself. In PHP's Perl compatible regular expressions, the pattern syntax is very similar to Perl's itself, i.e. the pattern begins with a slash ("/") and ends with another slash plus any options, in this case, a slash then a letter "i" to denote that the match is to be case-insensitive ("/pattern/i"). QFree's regex routines, although based on PCRE, don't use this syntax for the patterns (although I may add support for this in the preg_module at a later stage). Instead, the pattern options can be inserted as "extension operators" in the form "(?....)". If you look at the 4D example code, you will see that the regex pattern is identical to the one in the PHP example, except that the leading and trailing "/"s have been removed and the "i" has been replaced with "(?i)". Using extension operators in the pattern in this way is very useful as the regex flags don't need to be passed to the QFree commands in a separate parameter (making them easier to store in TEXT resources), but also because they can be applied to a submatch within the regular expression as well as on the their pattern. The following examples show how the case-insensitivity option can be applied to a submatch. preg_Match ("ab(c)";"abc") = True preg_Match ("ab(c)";"abC") = False preg_Match ("ab((?i)c)";"abc") = True preg_Match ("ab((?i)c)";"abC") = True preg_Match ("ab((?i)c)";"ABc") = False preg_Match ("ab(c)(?i)";"ABC") = True preg_Match ("ab(c)(?i)";"aBc") = True preg_Match ("ab(c(?i))";"abC") = False`extension op not at the start of the group preg_Match ("(?i)ab(c)";"abc") = True preg_Match ("(?i)ab(c)";"ABC") = True Using preg_Match instead of QF_REMatch allows a simple regex match to be done in far less code, but at first glance, it doesn't have the same level of features as QF_REMatch because it doesn't return the matched groups. Or does it? preg_Match actually has an optional 3rd parameter which takes a pointer to a Text Array. If this parameter is passed, preg_Match returns the match and any submatches into the array passed. This works just the same as the PHP preg_match function. We can extend the previous example slightly by adding an additional text array and pointer to it. e.g. `Method: validate_EmailAddress(emailAddress) --> TRUE if value C_TEXT($1;$emailAddress)C_BOOLEAN($0) $emailAddress:=$1 $pattern:="^([-a-zA-Z0-9_.]+)@(([-a-zA-Z0-9_]+[.])+[a-zA-Z]+)$(?i)" ARRAY TEXT (aMatches;0) If (preg_Match ($pattern;$emailAddress;->aMatches)) $0:=True $mailusername:=aMatches{1} $serverdomain:=aMatches{2} Else $0:=False End if Now the example extracts the mail username and server domain from the emailaddress, but obviously, in this example nothing is actually done with them. This example is only to highlight how easy it is to extract the individual submatches using preg_Match. The text matched by the entire expression is found in aMatch{0}. Function: preg_Match_All This function is basically an easier way of calling QF_REExtract. It allows every match of a particular pattern in a piece of text to be returned in one go, into an easy to use 2D Text array. The 2D Text Array is the perfect way of returning results from this type of function and for me, makes the resultant code easier to read and understand. If we look at our earlier example for reading 4D NUG digests, we can simplify the code quite a bit by using preg_Match_All instead of QF_REExtract directly. Function: preg_Replace This function allows you to do Find and Replace using a regular expression on a piece of text. You need to pass the routine your match pattern, substitution pattern and the target text and it will return the result. This function is extremely useful in certain circumstances and is something which I use in BBEdit all the time. Being able to use regular expressions for Find and Replace is an extremely powerful feature. Unlike the QF_REReplace command in QFree, this version is non-destructive, i.e. the text you pass will not be modified in any way. This is the same with the BLOB version of this method. Function: preg_Split This function is similar to preg_Match_All, except that it returns only a single dimension array contain the text split by the regex. This method uses the QF_RESplit command to perform the split. The only difference is that it return a text array, rather than 2 longint arrays containing the offset and lengths of each segment. This has many applications, but for the moment I'll just use one simple example to try and give an idea of it's potential. The following example function splits a line of text into an array with each element containing a word from the line of text. `Method: split_Text(text; ->textArray) C_TEXT($1;$text) C_POINTER($2) $text:=$1 $pattern:="[\s,.-;:]+" If (preg_Split ($pattern;$text;$2)=0) `do something with array... Else ARRAY TEXT($2->;0) End if If we passed this current paragraph of text as the text parameter to this function, plus a pointer to a the global text array, aMatches, we would get the following elements in aMatches at the end. aMatch{1}:="If" aMatch{2}:="we" aMatch{3}:="passed" aMatch{4}:="this" aMatch{5}:="current" aMatch{6}:="paragraph" aMatch{7}:="of" aMatch{8}:="text" aMatch{9}:="as" aMatch{10}:="the" aMatch{11}:="text" aMatch{12}:="parameter" aMatch{13}:="to" aMatch{14}:="this" aMatch{15}:="function" aMatch{16}:="plus" aMatch{17}:="a" aMatch{18}:="pointer" aMatch{19}:="to" aMatch{20}:="a" aMatch{21}:="the" aMatch{22}:="global" aMatch{23}:="text" aMatch{24}:="array" aMatch{25}:="aMatches" aMatch{26}:="we" aMatch{27}:="would" aMatch{28}:="get" aMatch{29}:="the" aMatch{30}:="following" aMatch{31}:="elements" aMatch{32}:="in" aMatch{33}:="aMatches" aMatch{34}:="at" aMatch{35}:="the" aMatch{36}:="end" preg_Module Summary Developers who have spent any time using the Perl compatible regular expressions in PHP should definitely like the syntax of these routines as they're very similar to their PHP counterparts. They can also cut down on coding when only a simple match is required or when you want to work with text variables instead of the QFree regex command standard way of working with BLOBs. The module also contains BLOB equivalents of all these functions.

Some examples of useful regular expressions for use with QCP 4.2's Regex find and replace functionality. Using Regular Expressions in the Method Editor Automated Solutions Group answered many peoples' prayers when they added Regex support to QuickCodePro 4.2. The only problem with QuickCodePro now is that it's still not available for Windows, but hopefully sometime in the not too distant future, this too will be realised with version 5. So what would you use regular expression for in the method editor? There are quite a few uses for regular expression find and replace in the 4D Method Editor. The only thing which is currently missing is the ability to find and replace with a regex on the selected text only or replace all from the current cursor position, but hopefully this refinement will be made in future versions. At the time of writing it was possible to Find individual matches forwards from the current cursor position, and replace them using the regex substution one by one, so some of the following examples can still be used. Converting Parameter Declarations to Assignments In nearly all 4D method I write, I begin by declaring any parameters and assigning their values to local variables which give a better indication of the parameters contents. All my methods start the same way, and I've always wanted a quick method of converting the declarations into the assignments to save myself a bit of time. Although not the perfect solution yet, QCP comes to the rescue with this particular operation. Using the regex find and replace, I can fairly quickly copy and paste the declarations, then apply the regex to each copied line to produce my variable assignments. The following code example shows how it's done. `Method: Regex Parameter Assignment Creation C_TEXT($1;$textParameter)`some information about the 1st parameter C_LONGINT($2;$longintParameter)`some information about the 2nd parameter C_POINTER($3;$pointerParameter)`some information about the 3rd parameter $textParameter:=$1 $longintParameter:=$2 $pointerParameter:=$3 In this example I want to be able to create the 3 assignments after the 3 declarations without retyping them. Ideally I'd love it if ASG could provide some way of writing a QCP Macro for this, but in the meantime it can be achieved using the following regular expression, and a couple of steps. Find String: ^C_..*\(([$][0-9]+)[;]([$].+)\).*$ Replace String: \2:=\1 To use this regex, type the parameter declarations as shown, then make a copy of them a line below, e.g. `Method: Regex Parameter Assignment Creation C_TEXT($1;$textParameter)`some information about the 1st parameter C_LONGINT($2;$longintParameter)`some information about the 2nd parameter C_POINTER($3;$pointerParameter)`some information about the 3rd parameter C_TEXT($1;$textParameter)`some information about the 1st parameter C_LONGINT($2;$longintParameter)`some information about the 2nd parameter C_POINTER($3;$pointerParameter)`some information about the 3rd parameter Now, put the cursor in between the two groups of declarations and go to QCP's "Find and Replace" dialog. Make sure the "Regex" checkbox is selected and enter the find and replace strings as shown above. Now hit "Find" and the first declaration in the 2nd group should be highlighted. Use Cmd-R to replace the string and then Cmd-G to find the next one. Repeat until all the declarations in the 2nd group have been converted to assignments as shown above. QCP remembers the recent patterns used, so you don't necessarily have to keep retyping them. Also, if you have a regular expression you're using regularly, you can add it to the "Pattern" menu on the QCP Find and Replace dialog for easy access in future. Replacing calls to commands and changing parameters simultaneously Another example I can think of is to replace all occurances of a particular command across a group of methods with a new command, but at the same time, changing the order of the parameters. This would be a pain-staking task without regular expressions as you could never be sure that the variables or literals used in each call would be the same, so a normal find and replace would be useless. In the following example I want to replace all calls to GET LIST ITEM when 4 parameters have been passed, and convert them to 2 seperate calls from my hierarchical list component. In this case, I don't necessarily know the names of the variables I've used in the GET LIST ITEM call, so the regex match is the only sure way of getting it right. The calls to GET LIST ITEM could take the form of one of the following: GET LIST ITEM($list;Selected list item($list);$itemRef;$itemText)`a comment GET LIST ITEM($list;$position;$itemRef;$itemText) If a line using this has a comment, I want to preserve the comment in the substituted text. The replacements for these two would look like the following: $itemRef:=hlist_Item_Ref_Get($list;Selected list item($list))`a comment $itemText:=hlist_Item_Text_Get($list;Selected list item($list)) $itemRef:=hlist_Item_Ref_Get($list;$position) $itemText:=hlist_Item_Text_Get($list;$position) The following regex and substitution strings do this job for you using QCP. Note: in this example, the replace string has a line break in it. This must be entered into QCP replace string box as \n or \r don't work here. The easiest way to enter the line break is to copy and paste the string from another application. There must also be an additional line break at the end of the replace string. Find String: GET LIST ITEM\(([^;]+);([^;]+);([^;]+);([^;]+)\)(.*)$ Replace String: \3:=hlist_Item_Ref_Get(\1;\2)\5 \4:=hlist_Item_Text_Get(\1;\2) Additional Replace Options in QuickCodePro QCP has some additional replace options in addition to those offered by PCRE. These allow you to do things like adjust the capitlisation of strings matched. Options like these are useful when enforcing style guide lines, especially as 4D isn't case sensitive. It's easy to get lazy about keeping your capitlisation intact, and using these options in QCP it is now much easier to go over many routines in one go to ensure they stick to the style guide. Previously, this kind of operation would have taken many separate find and replace operations, whereas it can now be carried out in just a few. One simple example is making sure that all local variables start with a lowercase character. This may not always be practical, but it show one example of how the additional featuers can be applied.

The end of the article. Conclusion This article has shown a number of ways in which regular expressions can be applied to certain tasks which a 4D developer may have to get involved with. There are many uses of regular expressions which I haven't yet covered and hope to add to the article over the coming months. So far, I've only covered validating an email address, but there are a huge number of other form validation tasks which regular expressions made extremely easy. Another thing is that the regular expression support in QCP is a very new feature, so it still hasn't had time to sink in with me at least. I didn't expected to get this feature until version 5. The regular expression engines used in QFree 2.0 and QuickCodePro 4.2 are very similar, so you don't need to learn different types of regular expressions to get the most out of either product. Personally, I'm only sorry it took me so many years to sit down and learn about them. Disclaimer As stated earlier in the article, I am still a novice with regular expressions, but learn a bit more about them every day. As such I cannot gaurantee that any of the Regex patterns in this tutorial will work correctly in your own environment, nor can I give any warranty on them. All the patterns and code shown in this tutorial has been tested with 4th Dimension 6.7.1, QFree 2.0 and QuickCodePro 4.2b6 on a PowerMac G4/400 with MacOS 9.0.4 and worked fine here during testing. If you have a problem with any of the code displayed, please let us know so that we can get it fixed as soon as possible. As the patterns and code in this article get further optimised and improved, I will be posting it to the site. If you'd like to be kept imformed of changes to this page, please send me an email. Mark Mitchenall, March 2001