Use Regular Expression

Most people are not familiar with regular expression. You probably have never heard of the term. So what is regular expression? Regular expression is a way to describe patterns in text. When you use a word processor, you go to menu command "Edit | Find" to search for text in the document. You type in the word and do the search. But what if you need to define a more complex search than a simple word matching? For example, you may want to find all the dates written in the yyyy-mm-dd format. Regular expression is the perfect tool for these searches. Regular expression syntax is somewhat difficult to learn, but once you master the skill, it will benefit you for life because regular expression is implemented in a lot of tools like Perl, PHP, Unix, JavaScript, and even some word processors. Here are a few simple examples to show you how regular expression can be used.

•	"^The": matches any string that starts with "The"

•	"ab*": matches a string that has an "a" followed by zero or more b's ("a", "ab", "abbb", etc.)

•	"ab{3,5}": from three to five b's ("abbb", "abbbb", or "abbbbb")

•	"hi\|hello": matches a string that has either "hi" or "hello" in it

•	"a.[0-9]": matches a string that has an "a" followed by one character and a digit

•	"^[a-zA-Z]": a string that starts with a letter

Since regular expression is not well know and a little hard to learn, you may ask why it is added to Biblioscape. The number one reason is: Biblioscape needs a powerful pattern matching tool for the import filters. Biblioscape uses import filters to map tagged fields in the import file to data fields in Biblioscape database. The existing tag matching is done by simple word search. Although it is adequate in most cases, there are times when a more powerful tool is desired, especially when the data provider doesn't follow a strict rule when displaying a record. With regular expression in our arsenal, we can not only tackle those cases with ease; it opens up possibilities for import files that were considered impossible in the past. The power of regular expression will be realized gradually in this area. Besides import filter design, regular expression support is also added to the "Find and Replace" window in the References module. Although you only know the basics of regular expression, you can now make the Find and Replace feature much more powerful than before.

A regular expression "engine" is a piece of software that can process regular expressions and try to match the pattern to the given string. Usually, the engine is part of a larger application. Just like in Biblioscape, regular expression is invoked when needed. Biblioscape will make sure the right regular expression is applied to the right data. As usual in the software world, different regular expression engines are not fully compatible with each other. In the world of regular expression, Perl 5 implementation is the most popular one. Although other implementations are not identical, they are very similar. It should not be a concern for regular users. The one implemented in Biblioscape is from regexpstudio.com.

Any single character matches itself, unless it is a metacharacter with a special meaning described later. A series of characters matches that series of characters in the target string, so the pattern "bluh" would match "bluh''. When used in this way, it functions the same way as a simple word matching tool. You can cause characters that normally function as metacharacters or escape sequences to be interpreted literally by 'escaping' them by preceding them with a backslash "\", for instance: metacharacter "^" match beginning of string, but "\^" match character "^", "\\" match "\" and so on.

Characters may be specified using an escape sequences syntax much like that used in C and Perl: "\n'' matches a newline, "\t'' a tab, etc. More generally, \xnn, where nn is a string of hexadecimal digits, matches the character whose ASCII value is nn. If you need wide (Unicode) character code, You can use '\x{nnnn}', where 'nnnn' - one or more hexadecimal digits.

\x{nnnn} char with hex code nnnn (one byte for plain text and two bytes for Unicode)

You can specify a character class by enclosing a list of characters in [], which will match any one character from the list. If the first character after the "['' is "^'', the class matches any character not in the list.

foob[aeiou]r finds strings 'foobar', 'foober' etc. but not 'foobbr', 'foobcr' etc.

foob[^aeiou]r find strings 'foobbr', 'foobcr' etc. but not 'foobar', 'foober' etc.

Within a list, the "-'' character is used to specify a range, so that a-z represents all characters between "a'' and "z'', inclusive. If You want "-'' itself to be a member of a class, put it at the start or end of the list, or escape it with a backslash. If You want ']' you may place it at the start of list or escape it with a backslash.

The "^" metacharacter by default is only guaranteed to match at the beginning of the input string/text, the "$" metacharacter only at the end. Embedded line separators will not be matched by "^'' or "$''. You may, however, wish to treat a string as a multi-line buffer, such that the "^'' will match after any line separator within the string, and "$'' will match before any line separator. You can do this by switching On the modifier /m.

The \A and \Z are just like "^'' and "$'', except that they won't match multiple times when the modifier /m is used, while "^'' and "$'' will match at every internal line separator. The ".'' metacharacter by default matches any character, but if You switch Off the modifier /s, then '.' won't match embedded line separators.

Note: When used in Biblioscape import filter, the line separator doesn't apply because Biblioscape reads one line at a time and send it to regular expression engine for processing if it finds "RE(...)RE". When used in "Edit | Find" or "Edit | Replace", the line separator does apply when working agains memo fields like Notes, Abstract, Keywords, Miscellaneous.

foob\dr matches strings like 'foob1r', ''foob6r' and so on but not 'foobar', 'foobbr' and so on

foob[\w\s]r matchs strings like 'foobar', 'foob r', 'foobbr' and so on but not 'foob1r', 'foob=r' and so on

A word boundary (\b) is a spot between two characters that has a \w on one side of it and a \W on the other side of it (in either order), counting the imaginary characters off the beginning and end of the string as matching a \W.

The use of boundary meta characters works like whole word search in a word processor. For example, searching for the word "the" can be done using the regular expression \bthe\b. This specifies that we are searching for "the" with no letters on each side of it (i.e. with a word boundary on each side).

Any item of a regular expression may be followed by another type of metacharacters - iterators. Using this metacharacters you can specify the number of occurences of the previous character, metacharacter or subexpression.

So, digits in curly brackets of the form {n,m}, specify the minimum number of times to match the item n and the maximum m. The form {n} is equivalent to {n,n} and matches exactly n times. The form {n,} matches n or more times. There is no limit to the size of n or m, but large numbers will chew up more memory and slow down regular expression execution.

If a curly bracket occurs in any other context, it is treated as a regular character.

By default, pattern matching is greedy, which means that the matcher returns the longest match possible. For example, applying the pattern A.*c to AbcAbcA matches AbcAbc rather than the shorter Abc. To do nongreedy matching, a question mark must be added to the quantifier. For example, the pattern A.*?c will find the shortest match possible. You can switch all iterators into "non-greedy" mode (see the modifier /g).

You can specify a series of alternatives for a pattern using "|'' to separate them, so that fee|fie|foe will match any of "fee'', "fie'', or "foe'' in the target string (as would f(e|i|o)e). The first alternative includes everything from the last pattern delimiter ("('', "['', or the beginning of the pattern) up to the first "|'', and the last alternative contains everything from the last "|'' to the next pattern delimiter. For this reason, it's common practice to include alternatives in parentheses, to minimize confusion about where they start and end.

Alternatives are tried from left to right, so the first alternative found for which the entire expression matches, is the one that is chosen. This means that alternatives are not necessarily greedy. For example: when matching foo|foot against "barefoot'', only the "foo'' part will match, as that is the first alternative tried, and it successfully matches the target string. (This might not seem important, but it is important when you are capturing matched text using parentheses.) Also remember that "|'' is interpreted as a literal within square brackets, so if You write [fee|fie|foe] You're really only matching [feio|].

The bracketing construct ( ... ) may also be used for define regular expression subexpressions Subexpressions are numbered based on the left to right order of their opening parenthesis. First subexpression has number '1' (whole regular expression match has number '0').

(foobar){8,10} matchs strings which contain 8, 9 or 10 instances of the 'foobar'

Metacharacters \1 through \9 are interpreted as backreferences. \<n> matches previously matched subexpression #<n>. Back-references are parts of a regular expression that are in parentheses. They are counted in order in which they appear. For example, consider regular expression a((b)+)c . Backreference 0 is the whole expression. Backreference 1 is (b)+ and backreference 2 is b . If matching text is abbbbcdd, then backref 0 is abbbbc (matching part), backref 1 is bbbb, backref 2 is b. Easy, isn't it?

(['"]?)(\d+)\1 matchs '"13" (in double quotes), or '4' (in single quotes) or 77 (without quotes) etc

Modifiers are for changing the behaviour of regular expression. There are many ways to set up modifiers. Any of these modifiers may be embedded within the regular expression itself using the (?...) construct.

•	i: Do case-insensitive pattern matching (using installed in you system locale settings), see also InvertCase.

•	m: Treat string as multiple lines. That is, change "^'' and "$'' from matching at only the very start or end of the string to the start or end of any line anywhere within the string, see also Line separators.

•	s: Treat string as single line. That is, change ".'' to match any character whatsoever, even a line separators (see also Line separators), which it normally would not match.

•	g: Non standard modifier. Switching it Off You'll switch all following operators into non-greedy mode (by default this modifier is On). So, if modifier /g is Off then '+' works as '+?', '' as '?' and so on

•	x: Extend your pattern's legibility by permitting whitespace and comments (see explanation below).

The modifier /x itself needs a little more explanation. It tells the regular expression engine to ignore whitespace that is neither backslashed nor within a character class. You can use this to break up your regular expression into (slightly) more readable parts. The # character is also treated as a metacharacter introducing a comment.

This also means that if you want real whitespace or # characters in the pattern (outside a character class, where they are unaffected by /x), that you'll either have to escape them or encode them using octal or hex escapes. Taken together, these features go a long way towards making regular expressions text more readable.

You may use it into regular expression for modifying modifiers by the fly. If this construction inlined into subexpression, then it effects only into this subexpression.

A comment, the text is ignored. Note that TRegExpr closes the comment as soon as it sees a ")", so there is no way to put a literal ")" in the comment.