Regular Expressions in Python

Regular expressions match patterns in text: pulling IPs out of a log, validating a phone number, rewriting a date format. This article covers the subset that handles almost every real task, plus the traps that make regexes unmaintainable.

Environment

re is part of the standard library; examples are verified on Python 3.13. For tricky patterns, prototype in an interactive tool such as regex101 (set the flavor to Python) before pasting it into code.

The main functions

import re

text = "Order A-1024 shipped, total 89.50 USD"

re.search(r"A-\d+", text)          # first match anywhere → Match or None
re.match(r"\d+", text)             # only at position 0 → None here
re.fullmatch(r"\d+", "1024")       # the whole string must match
re.findall(r"\d+\.?\d*", text)     # ['1024', '89.50'] — all matches as strings
re.finditer(r"\d+", text)          # iterator of Match objects
re.split(r"[,;]\s*", text)         # split on a pattern
re.sub(r"\d+", "N", text)          # replace, returns a new string
re.subn(r"\d+", "N", text)         # ('...', 2) — also reports the replacement count

Always prefix pattern strings with r (raw string). Otherwise "\d" is interpreted as a string escape and the regex engine never sees the backslash.

The Match object

m = re.search(r"total (?P<amount>[\d.]+) (\w+)", text)
m.group()             # 'total 89.50 USD' — the whole match
m.group("amount")     # '89.50' — named group
m.group(2)            # 'USD' — numbered group
m.start(), m.end()    # offsets
m.span()              # (start, end)
m.groupdict()         # {'amount': '89.50'}

Check for None before touching the result, or you get AttributeError: 'NoneType' object has no attribute 'group'.

Syntax you will actually use

Pattern	Meaning
.	Any character except newline (by default)
\d / \D	Digit / non-digit
\w / \W	Word character (letters, digits, underscore) / non-word
\s / \S	Whitespace / non-whitespace
^ / $	Start / end of string (or of a line with re.MULTILINE)
\b	Word boundary
[a-z] / [^0-9]	Character class / negated class
* + ?	0+, 1+, 0 or 1
{m,n}	Between m and n repetitions
a|b	Alternation
(...)	Capturing group
(?:...)	Non-capturing group
(?P<name>...)	Named capturing group

Practical examples:

re.findall(r"\b\d{4}-\d{2}-\d{2}\b", log)      # dates
re.findall(r"\d{1,3}(?:\.\d{1,3}){3}", log)    # IPv4-looking strings
re.findall(r"#[0-9a-fA-F]{6}", css)            # hex colors
re.fullmatch(r"\+?[0-9][0-9 ()-]{7,19}", phone)  # loose phone check

Character classes are not magic: [\d.] means “a digit or a dot”, and inside a class most metacharacters lose their meaning — only ], \, ^ (at the start) and - (between characters) need care.

Greedy versus lazy

Quantifiers are greedy by default and take as much as possible:

re.search(r"<.*>", "<a><b>").group()      # '<a><b>' — swallowed both tags
re.search(r"<.*?>", "<a><b>").group()     # '<a>' — lazy with the extra ?

Extracting a quoted value is the classic case:

re.search(r'"(.*?)"', 'name="Tom" age="18"').group(1)   # 'Tom'

Lazy quantifiers still backtrack; they only change the order in which alternatives are tried.

Compiling and flags

Compile a pattern you reuse — it is faster and keeps flags next to the pattern:

DATE = re.compile(r"(?P<year>\d{4})-(?P<month>\d{2})-(?P<day>\d{2})")

for m in DATE.finditer(log):
    print(m.group("year"))

DATE.fullmatch("2026-11-03").groupdict()   # {'year': '2026', 'month': '11', 'day': '03'}

Common flags:

re.search(r"error", line, re.IGNORECASE)   # re.I — case-insensitive
re.findall(r"^#.*", text, re.MULTILINE)    # re.M — ^ and $ match per line
re.search(r"a . b", text, re.DOTALL)       # re.S — . also matches newline
re.match(r"# comment", line, re.X)         # re.VERBOSE — whitespace and comments allowed
re.findall(r"\d+", text, re.ASCII)         # \d, \w, \b restricted to ASCII

re.VERBOSE pays for itself on anything longer than a line:

phone = re.compile(r"""
    (?P<area>\d{3})     # area code
    [- ]?               # optional separator
    (?P<num>\d{8})      # subscriber number
""", re.VERBOSE)

Note that \d matches non-ASCII digits such as ٣ by default. Add re.ASCII when the input must be 0-9 only.

Substitution tricks

Replacement strings can reference groups with \1 or \g<name>:

re.sub(r"(\d{4})-(\d{2})-(\d{2})", r"\3/\2/\1", "2026-11-03")   # '03/11/2026'

The replacement can be a function, which lets you compute each substitution:

def upper(m: re.Match[str]) -> str:
    return m.group().upper()

re.sub(r"\berror\b", upper, log, flags=re.IGNORECASE)

Because \ is also special in replacement strings, a literal backslash needs \\\\ in a regular string (equivalent to r"\\"). Returning the replacement from a lambda sidesteps the escaping rules entirely.

Common mistakes

1. Confusing match with search

re.match only anchors at position 0; re.search scans the whole string. Most real tasks want search or fullmatch.

2. Forgetting the raw string

"\b" is a backspace character and "\n" is a newline, so the pattern silently changes meaning; and since Python 3.12 an unrecognised escape such as "\d" emits a SyntaxWarning. Always write r"\b", r"\d".

3. Interpolating user input into a pattern

re.search(user_input, text)              # wrong: user can inject regex syntax
re.search(re.escape(user_input), text)   # right

4. Parsing HTML or JSON with regex

Nested tags and quoted strings are not regular languages. Use html.parser/BeautifulSoup for HTML and json for JSON.

5. Catastrophic backtracking

Nested quantifiers such as (a+)+$ backtrack exponentially on non-matching long input and can hang the process. Mitigations: flatten the pattern, bound repetitions with {1,50}, anchor with ^, or use an atomic group (?>...) (3.11+) when you know backtracking is pointless.

6. Validating with findall or search

re.search(r"\d{11}", phone) happily matches inside "xx13800138000xx". Use re.fullmatch for whole-string validation.

7. One mega-pattern instead of a pipeline

Three small, named patterns with separate tests are easier to debug than one 200-character expression. If you cannot explain a pattern in a comment, split it.

8. Ignoring re.error

Patterns assembled at runtime can raise re.error: missing ), unterminated subpattern. Compile configuration-driven patterns inside a try block and report the offending pattern.

Summary

• search finds the first match, findall returns all of them, finditer streams Match objects, sub replaces, fullmatch validates a whole string.
• Prefix patterns with r; quantifiers are greedy, add ? for lazy.
• Use re.compile for repeated patterns and re.VERBOSE for complex ones.
• Never parse HTML/JSON with regex; always re.escape user input used as a pattern.
• Bound your quantifiers — catastrophic backtracking is a real outage, not a theoretical one.

Next: put these pieces to work — pytest, a CLI project layout, and cleaning CSV/JSON data.