Python is a high level open source scripting language. Python’s built-in “re” module provides excellent support for regular expressions, with a modern and complete regex flavor. Two significant missing features, atomic grouping and possessive quantifiers, were added in Python 3.11. Though Python’s regex engine correctly handles Unicode strings, its syntax is still missing Unicode properties and shorthand character classes only match ASCII characters.

The first thing to do is to import the regexp module into your script with import re.

Regex Search and Match

Call re.search(regex, subject) to apply a regex pattern to a subject string. The function returns None if the matching attempt fails, and a Match object otherwise. Since None evaluates to False, you can easily use re.search() in an if statement. The Match object stores details about the part of the string matched by the regular expression pattern.

You can set regex matching modes by specifying a special constant as a third parameter to re.search(). re.I or re.IGNORECASE applies the pattern case insensitively. re.S or re.DOTALL makes the dot match newlines. re.M or re.MULTILINE makes the caret and dollar match after and before line breaks in the subject string. There is no difference between the single-letter and descriptive options, except for the number of characters you have to type in. To specify more than one option, “or” them together with the | operator: re.search("^a", "abc", re.I | re.M).

By default, Python’s regex engine only considers the letters A through Z, the digits 0 through 9, and the underscore as “word characters”. Specify the flag re.L or re.LOCALE to make \w match all characters that are considered letters given the current locale settings. Alternatively, you can specify re.U or re.UNICODE to treat all letters from all scripts as word characters. The setting also affects word boundaries.

Do not confuse re.search() with re.match(). Both functions do exactly the same, with the important distinction that re.search() will attempt the pattern throughout the string, until it finds a match. re.match() on the other hand, only attempts the pattern at the very start of the string. Basically, re.match("regex", subject) is the same as re.search("\Aregex", subject). Note that re.match() does not require the regex to match the entire string. re.match("a", "ab") will succeed.

Python 3.4 adds a new re.fullmatch() function. This function only returns a Match object if the regex matches the string entirely. Otherwise it returns None. re.fullmatch("regex", subject) is the same as re.search("\Aregex\Z", subject). This is useful for validating user input. If subject is an empty string then fullmatch() evaluates to True for any regex that can find a zero-length match.

To get all matches from a string, call re.findall(regex, subject). This will return an array of all non-overlapping regex matches in the string. “Non-overlapping” means that the string is searched through from left to right, and the next match attempt starts beyond the previous match. If the regex contains one or more capturing groups, re.findall() returns an array of tuples, with each tuple containing text matched by all the capturing groups. The overall regex match is not included in the tuple, unless you place the entire regex inside a capturing group.

More efficient than re.findall() is re.finditer(regex, subject). It returns an iterator that enables you to loop over the regex matches in the subject string: for m in re.finditer(regex, subject). The for-loop variable m is a Match object with the details of the current match.

Unlike re.search() and re.match(), re.findall() and re.finditer() do not support an optional third parameter with regex matching flags. Instead, you can use global mode modifiers at the start of the regex. E.g. “(?i)regex” matches regex case insensitively.

Strings, Backslashes and Regular Expressions

The backslash is a metacharacter in regular expressions. It is used to escape other metacharacters. The regex \\ matches a single backslash. \d is a single token matching a digit.

Python strings also use the backslash to escape characters. The above regexes are written as Python strings as "\\\\" and "\\d". Confusing indeed.

Fortunately, Python also has “raw strings” which do not apply special treatment to backslashes. As raw strings, the above regexes become r"\\" and r"\d". The only limitation of using raw strings is that the delimiter you’re using for the string must not appear in the regular expression, as raw strings do not offer a means to escape it.

You can use \n and \t in raw strings. Though raw strings do not support these escapes, the regular expression engine does. The end result is the same.

Unicode

Prior to Python 3.3, Python’s re module did not support any Unicode regular expression tokens. Python Unicode strings, however, have always supported the \uFFFF notation. Python’s re module can use Unicode strings. So you could pass the Unicode string u"\u00E0\\d" to the re module to match à followed by a digit. The backslash for \d was escaped, while the one for \u was not. That’s because \d is a regular expression token, and a regular expression backslash needs to be escaped. \u00E0 is a Python string token that shouldn’t be escaped. The string u"\u00E0\\d" is seen by the regular expression engine as à\d.

If you did put another backslash in front of the \u, the regex engine would see \u00E0\d. If you use this regex with Python 3.2 or earlier, it will match the literal text u00E0 followed by a digit instead.

To avoid any confusion about whether backslashes need to be escaped, just use Unicode raw strings like ur"\u00E0\d". Then backslashes don’t need to be escaped. Python does interpret Unicode escapes in raw strings.

In Python 3.0 and later, strings are Unicode by default. So the u prefix shown in the above samples is no longer necessary. Python 3.3 also adds support for the \uFFFF notation to the regular expression engine. So in Python 3.3, you can use the string "\\u00E0\\d" to pass the regex \u00E0\d which will match something like à0.

Search and Replace

re.sub(regex, replacement, subject) performs a search-and-replace across subject, replacing all matches of regex in subject with replacement. The result is returned by the sub() function. The subject string you pass is not modified.

If the regex has capturing groups, you can use the text matched by the part of the regex inside the capturing group. To substitute the text from the third group, insert \3 into the replacement string. If you want to use the text of the third group followed by a literal three as the replacement, use \g<3>3. \33 is interpreted as the 33rd group. It is an error if there are fewer than 33 groups. If you used named capturing groups, you can use them in the replacement text with \g<name>.

The re.sub() function applies the same backslash logic to the replacement text as is applied to the regular expression. Therefore, you should use raw strings for the replacement text, as I did in the examples above. The re.sub() function will also interpret \n and \t in raw strings. If you want c:\temp as a replacement, either use r"c:\\temp" or "c:\\\\temp". The 3rd backreference is r"\3" or "\\3".

Splitting Strings

re.split(regex, subject) returns an array of strings. The array contains the parts of subject between all the regex matches in the subject. Adjacent regex matches will cause empty strings to appear in the array. The regex matches themselves are not included in the array. If the regex contains capturing groups, then the text matched by the capturing groups is included in the array. The capturing groups are inserted between the substrings that appeared to the left and right of the regex match. If you don’t want the capturing groups in the array, convert them into non-capturing groups. The re.split() function does not offer an option to suppress capturing groups.

You can specify an optional third parameter to limit the number of times the subject string is split. Note that this limit controls the number of splits, not the number of strings that will end up in the array. The unsplit remainder of the subject is added as the final string to the array. If there are no capturing groups, the array will contain limit+1 items.

The behavior of re.split() has changed between Python versions when the regular expression can find zero-length matches. In Python 3.4 and prior, re.split() ignores zero-length matches. In Python 3.5 and 3.6 re.split() throws a FutureWarning when it encounters a zero-length match. This warning signals the change in Python 3.7. Now re.split() also splits on zero-length matches.

Match Details

re.search() and re.match() return a Match object, while re.finditer() generates an iterator to iterate over a Match object. This object holds lots of useful information about the regex match. I will use m to signify a Match object in the discussion below.

m.group() returns the part of the string matched by the entire regular expression. m.start() returns the offset in the string of the start of the match. m.end() returns the offset of the character beyond the match. m.span() returns a 2-tuple of m.start() and m.end(). You can use the m.start() and m.end() to slice the subject string: subject[m.start():m.end()].

If you want the results of a capturing group rather than the overall regex match, specify the name or number of the group as a parameter. m.group(3) returns the text matched by the third capturing group. m.group('groupname') returns the text matched by a named group ‘groupname’. If the group did not participate in the overall match, m.group() returns an empty string, while m.start() and m.end() return -1.

If you want to do a regular expression based search-and-replace without using re.sub(), call m.expand(replacement) to compute the replacement text. The function returns the replacement string with backreferences etc. substituted.

Regular Expression Objects

If you want to use the same regular expression more than once, you should compile it into a regular expression object. Regular expression objects are more efficient, and make your code more readable. To create one, just call re.compile(regex) or re.compile(regex, flags). The flags are the matching options described above for the re.search() and re.match() functions.

The regular expression object returned by re.compile() provides all the functions that the re module also provides directly: search(), match(), findall(), finditer(), sub() and split(). The difference is that they use the pattern stored in the regex object, and do not take the regex as the first parameter. re.compile(regex).search(subject) is equivalent to re.search(regex, subject).