Jul 28, 2026

The Complete Python Guide: From Zero to Writing Real Programs

A complete Python 3.12+ guide covering syntax, data structures, functions, object-oriented programming, engineering practices, async programming, and everyday tools.

Written for Python 3.12+. Examples verified on Python 3.14. Best read front to back, but each chapter stands on its own well enough to use as a reference.

0.1 How to Use This Guide

This guide assumes you have no programming experience at all. It doesn't expect you to know math, jargon, or how computers work internally.

Three suggestions:

First, type everything out. Reading code and writing code are different skills. Understanding a snippet takes about 30 seconds; writing it yourself from a blank file might take 10 minutes — and those 10 minutes are where the learning happens. Type the examples by hand. Don't copy-paste.

Second, skip what you don't understand. Programming knowledge is a web, not a line. Chapter 4 mentions "functions," but functions aren't covered until Chapter 11. When that happens, just accept "that's how it works" and keep going. It'll click later.

Third, do every exercise before moving on. The answers are right below each question, but write your own first, run it, see the error, fix it — then look. Nodding along to an answer produces almost zero retention.

Conventions

>>> means you're typing into the interactive interpreter, with output on the next line:

python
>>> 1 + 1
2

Code blocks without >>> are complete programs saved in a .py file:

python
name = "Python"
print(name)

Comments start with #. They're for humans; Python ignores them:

python
# This line never runs
x = 10  # End-of-line comments work too

Markers used throughout:

  • ⚠️ Gotcha — a trap beginners fall into constantly
  • 💡 Tip — a way to write better code
  • 🔍 Going deeper — skip for now; come back later

0.2 Learning Roadmap

Setup ──► Syntax ──► Data structures ──► Functions ──► OOP ──► Advanced ──► Engineering
Ch 1-3     Ch 4-7        Ch 8-10          Ch 11-13    Ch 16-18   Ch 19-24    Ch 25-28
                             │
                             └──► Exceptions / files (Ch 14-15)

If you want to build something useful fast: Chapters 1–15 are enough to write automation scripts, process files, and scrape data. Everything from Chapter 16 on is about making code better, not making it work.

Time estimate (one hour a day):

  • Ch 1–10: ~2 weeks, laying the syntax foundation
  • Ch 11–18: ~2 weeks, learning to organize code
  • Ch 19–29: ~3 weeks, going from "can write it" to "writes it well"

Part 1 · Getting Started

1. What Python Is

1.1 The Short Version

Python is a programming language — a set of rules for describing to a computer what to do. You write a text file, and the Python interpreter reads and executes it.

Its core design bias: code is read by humans first, executed by machines second. The same task in Python is usually shorter than in other languages, and closer to plain English. For example, "print 1 through 5":

python
for i in range(1, 6):
    print(i)

Java would need a class, a main method, type declarations, semicolons, and braces. Python needs two lines.

1.2 What Python Is Good For

DomainTypical useCommon tools
AutomationBatch renaming, spreadsheet wrangling, scheduled jobsstdlib os / pathlib / shutil
Data analysisCleaning, statistics, plottingpandas, numpy, matplotlib
AI / MLTraining models, calling LLM APIsPyTorch, scikit-learn
Web backendsServers, REST APIsDjango, FastAPI, Flask
Web scrapingPulling data off pagesrequests, httpx, Beautiful Soup
Scientific computingSimulation, bioinformaticsscipy, sympy
DevOpsServer management, deploy scriptsAnsible, Fabric

Where it's not a good fit: mobile app frontends, browser frontends (that's JavaScript's territory), and performance-critical systems (game engines, OS kernels).

1.3 About Versions

Python had one famous schism: Python 2 and Python 3 are incompatible. Python 2 reached end of life in 2020, so today you only care about Python 3. If you see code online like print "hello" (no parentheses), that's Python 2 — ignore it.

Python 3 ships a new version every October (3.12, 3.13, 3.14…), each supported for five years. As of mid-2026, the current stable release is 3.14.

How to pick: install the latest stable release, unless a library you depend on doesn't support it yet. Everything in this guide works on 3.12 and up; version-specific features are flagged where relevant.

1.4 What "Interpreted Language" Means in Practice

Compiled languages like C translate the whole source into machine code up front, producing an executable, and then run it. Python is different: the interpreter reads and executes your code line by line.

Two direct consequences:

Upside: edit, run, done — no compile step. And you can open an interactive session to test a single line whenever you want.

Cost: pure-Python compute-bound code is typically one to two orders of magnitude slower than C or Rust (how much varies enormously by workload — when you call into numpy, which is C underneath, the gap shrinks a lot). Also, many errors only surface when that line actually executes. Misspell a variable name and Python won't warn you until control reaches it.

🔍 Going deeper: strictly speaking Python compiles source to bytecode (.pyc files) first, and a virtual machine executes that. This is an implementation detail you can ignore day to day — just know that the __pycache__ folder that appears in your project is this, and you can delete it any time.

2. Installation and Environment

2.1 Check What You Have

Open a terminal (Terminal on macOS, PowerShell on Windows) and run:

bash
python3 --version

If it prints something like Python 3.14.x, you're set. On Windows the command may be python --version.

⚠️ Gotcha: the Python bundled with macOS may be an old version, and installing packages into it pollutes the system environment. Install your own copy instead.

2.2 How to Install

Option 1: Official installer (best for beginners)

Download from python.org/downloads and double-click.

⚠️ Windows users, pay attention: on the first screen of the installer there's a checkbox at the bottom labeled "Add python.exe to PATH". Check it. If you don't, typing python in a terminal will say "command not found" — the single most common beginner stumbling block.

Option 2: Package manager (better long-term)

macOS (install Homebrew first):

bash
brew install python

Windows:

powershell
winget install Python.Python.3.14

Ubuntu/Debian:

bash
sudo apt update && sudo apt install python3 python3-pip python3-venv

Option 3: uv (modern toolchain, if you have some experience)

uv is a Python toolchain manager written in Rust. It's extremely fast and handles Python versions, virtual environments, and packages all at once:

bash
# Install uv (macOS/Linux)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install Python with uv
uv python install 3.14

If you're just starting out, use option 1 or 2. Chapter 25 covers uv in detail.

2.3 Pick an Editor

Any text editor works, but a good one gives you syntax highlighting, autocomplete, and inline errors — an enormous time saver.

EditorWho it's forNotes
VS CodeMost peopleFree, lightweight, best ecosystem. Install the official Python extension
PyCharmLarge projectsMost complete feature set; Community edition is free
Zed / CursorWant a modern feelFast, good AI integration
Jupyter NotebookData analysisCell-by-cell execution, inline charts; not suited to full programs

VS Code in three steps:

  1. Install VS Code
  2. Search the extension marketplace for "Python", install Microsoft's official one (includes Pylance)
  3. Open a .py file and hit the ▶️ button in the top right

2.4 Three Ways to Run Code

Way 1: the interactive interpreter (REPL)

Type python3 and press Enter:

Python 3.14.x (main, ...) [...]
Type "help", "copyright", "credits" or "license" for more information.
>>>

(The build info in parentheses depends on where you installed Python from and differs on every machine. Don't worry about it.)

>>> is the prompt. Type a line, press Enter, it runs immediately:

python
>>> 2 + 3
5
>>> "hello".upper()
'HELLO'

Exit with exit() or Ctrl-D (Ctrl-Z then Enter on Windows).

The REPL is perfect for checking an idea: "wait, what does this function return?" — just try it, faster than reading docs. Get in the habit of keeping one open.

💡 Tip: install IPython (pip install ipython) for a much better REPL: autocomplete, syntax highlighting, ? for help.

Way 2: run a script file

Create hello.py, put code in it, then:

bash
python3 hello.py

This is the normal way to do things.

Way 3: hit Run in your editor

F5 in VS Code, or the triangle in the corner. Same as way 2, just without typing the command.

2.5 Exercises

2.1 Start the interactive interpreter, compute 2 ** 100 (2 to the 100th power), look at the result, then exit.

2.2 Create a file first.py containing print("Python is installed") and run it from the terminal.

<details> <summary>Answers</summary>

2.1

$ python3
>>> 2 ** 100
1267650600228229401496703205376
>>> exit()

The point here: Python integers have no size limit. They never overflow the way they do in most languages.

2.2

bash
$ cat first.py
print("Python is installed")
$ python3 first.py
Python is installed

If you get python3: command not found, your PATH isn't set up — go back to 2.2 and reinstall with "Add to PATH" checked. </details>


3. Your First Program

3.1 Hello, World

python
print("Hello, World!")

Piece by piece:

  • print is a function — a packaged bit of behavior you can invoke repeatedly
  • (...) means "call" this function
  • "Hello, World!" is a string: text in quotes, passed to print as an argument

Output:

Hello, World!

3.2 More print

python
print("a", "b", "c")          # a b c   — multiple args joined by spaces
print("a", "b", sep="-")      # a-b     — custom separator
print("no newline", end="")   # suppress the trailing newline
print()                       # print a blank line

3.3 Getting Input

python
name = input("What's your name? ")
print("Hello, " + name)

Running it:

What's your name? Alice
Hello, Alice

input() pauses the program, waits for the user to press Enter, and returns what they typed as a string.

⚠️ Gotcha: input() always returns a string, even when the user types digits.

python
age = input("Age: ")   # user types 18
print(age + 1)         # TypeError: can only concatenate str (not "int") to str

Convert first:

python
age = int(input("Age: "))
print(age + 1)   # 19

3.4 Comments

python
# A single-line comment

x = 1  # An end-of-line comment

# Multi-line comments are just multiple #
# lines, like this

Python has no /* */ block comment syntax. Triple-quoted strings are often used that way, but they're actually string objects:

python
"""
Syntactically these lines are a string.
It isn't assigned to anything, so the effect is a comment.
Placed at the top of a function/class/module it has special meaning (a docstring).
"""

When to write a comment: explain why, not what.

python
# ❌ Useless — the code already says this
count = count + 1  # add one to count

# ✅ Useful — explains intent you can't read off the code
count += 1  # server counts from 1, client from 0; compensate for the offset

3.5 Indentation: Python's Most Distinctive Rule

Most languages use braces {} to mark blocks. Python uses indentation:

python
if 5 > 3:
    print("this line is inside the if")
    print("so is this one")
print("this one is outside; it always runs")

Rules:

  • Code after a colon : must be indented
  • Indentation at the same level must be consistent
  • Use 4 spaces by convention — not tabs (most editors convert tabs for you)

⚠️ Gotcha: mixing tabs and spaces raises TabError or IndentationError, and it's invisible to the eye. Turn on "render whitespace" in VS Code to diagnose it.

The upside of this design: a program's visual structure and its logical structure can never disagree. Indentation can't lie to you.

3.6 Something Slightly Real

Putting the pieces together — a BMI calculator:

python
# bmi.py — body mass index
# Categories below use the WHO standard for adults:
# underweight <18.5, normal 18.5-24.9, overweight 25.0-29.9, obese >=30
# Note some countries use different cutoffs (e.g. China's WS/T 428:
# overweight >=24, obese >=28)

print("=== BMI Calculator ===")

height = float(input("Height in meters (e.g. 1.75): "))
weight = float(input("Weight in kg: "))

bmi = weight / (height ** 2)

print(f"Your BMI is {bmi:.1f}")

if bmi < 18.5:
    print("Underweight")
elif bmi < 25:
    print("Normal")
elif bmi < 30:
    print("Overweight")
else:
    print("Obese")

Running it:

=== BMI Calculator ===
Height in meters (e.g. 1.75): 1.75
Weight in kg: 70
Your BMI is 22.9
Normal

Several things here haven't been explained yet — float(), f"...", **, if/elif/else. The next few chapters cover each. For now, just get a feel for what a useful program looks like.

3.7 Exercises

3.1 Write a program that asks for a name and age, then prints "Hi XXX, next year you'll be YY".

3.2 Write a Celsius-to-Fahrenheit converter. Formula: F = C × 9 / 5 + 32.

3.3 What's wrong with this code? Fix it.

python
print("Enter two numbers")
a = input()
b = input()
print("Sum:", a + b)

<details> <summary>Answers</summary>

3.1

python
name = input("Your name: ")
age = int(input("Your age: "))
print(f"Hi {name}, next year you'll be {age + 1}")

3.2

python
celsius = float(input("Celsius: "))
fahrenheit = celsius * 9 / 5 + 32
print(f"{celsius}°C = {fahrenheit}°F")

3.3 input() returns strings, so "3" + "5" gives "35" (concatenation) rather than 8. Fix:

python
print("Enter two numbers")
a = float(input())
b = float(input())
print("Sum:", a + b)

</details>


Part 2 · Language Basics

4. Variables and Basic Types

4.1 Variables: Naming Data

python
x = 10
message = "hello"
pi = 3.14159

= is assignment, not mathematical equality. It means "bind the value on the right to the name on the left."

Python variables don't need type declarations or forward declarations, and they can change type at any time:

python
x = 10        # x is an integer
x = "hello"   # x is a string now — perfectly legal
🔍 Going deeper: the more accurate mental model is that a variable is a label stuck onto an object, not a box holding one. a = b doesn't copy b's contents into a; it makes a and b point at the same object. This distinction becomes critical in Chapter 8 when we hit mutable objects.

4.2 Naming Rules

Hard rules (breaking them is an error):

  • Letters, digits, underscores only
  • Can't start with a digit
  • Can't be a keyword (if, for, class, etc.)

Conventions (breaking them isn't an error, just bad code):

python
user_name = "Alice"     # ✅ variables/functions: lowercase with underscores (snake_case)
MAX_RETRY = 3           # ✅ constants: ALL_CAPS
class UserProfile:      # ✅ classes: PascalCase
_internal = 1           # ✅ leading underscore: "internal, don't touch"

userName = "Alice"      # ❌ that's Java/JS style, not Python
l = 1                   # ❌ lowercase l is hard to tell from digit 1
list = [1, 2]           # ❌ shadows the built-in list

⚠️ Gotcha: don't use list, dict, str, type, id, sum, max, input, or file as variable names. Once shadowed, the original is unusable in that scope, and the resulting errors are baffling.

Viewing keywords:

python
>>> import keyword
>>> keyword.kwlist          # hard keywords: never usable as names
['False', 'None', 'True', 'and', 'as', 'assert', 'async', 'await', 'break',
 'class', 'continue', 'def', 'del', 'elif', 'else', 'except', 'finally',
 'for', 'from', 'global', 'if', 'import', 'in', 'is', 'lambda', 'nonlocal',
 'not', 'or', 'pass', 'raise', 'return', 'try', 'while', 'with', 'yield']

>>> keyword.softkwlist      # soft keywords: special only in specific syntax
['_', 'case', 'match', 'type']

Soft keywords are a mechanism Python added later: match and case (3.10's pattern matching) and type (3.12's type aliases) are only keywords inside their respective constructs. Elsewhere they're ordinary names, so match = 1 is legal — though for readability's sake, don't.

4.3 The Four Basic Types

TypeNameExamplesNotes
Integerint42, -7, 0Unbounded
Floatfloat3.14, -0.5, 2.0Decimals
Stringstr"hi", 'hi'Text
BooleanboolTrue, FalseNote the capital letters

Check a type with type():

python
>>> type(42)
<class 'int'>
>>> type(3.14)
<class 'float'>
>>> type("hi")
<class 'str'>
>>> type(True)
<class 'bool'>

4.4 Integers

python
a = 42
b = -17
c = 1_000_000     # underscores as digit separators; purely visual, value is 1000000

# Other bases
binary = 0b1010   # binary  → 10
octal = 0o17      # octal   → 15
hexa = 0xFF       # hex     → 255

Python integers have arbitrary precision — they never overflow:

python
>>> 2 ** 1000
10715086071862673209484250490600018105614048117055336074437503883703510511249361224931983788156958581275946729175531468251871452856923140435984577574698574803934567774824230985421074605062371141877954182153046474983581941267398767559165543946077062914571196477686542167660429831652624386837205668069376

4.5 Floats

python
x = 3.14
y = -0.001
z = 2.5e3      # scientific notation = 2500.0
w = 1e-4       # = 0.0001

⚠️ Gotcha: floats are imprecise

python
>>> 0.1 + 0.2
0.30000000000000004
>>> 0.1 + 0.2 == 0.3
False

This isn't a Python bug — it's how every language using IEEE 754 binary floats behaves. 0.1 is a repeating fraction in binary and can only be stored approximately.

What to do:

python
# Option 1: compare with tolerance
import math
math.isclose(0.1 + 0.2, 0.3)   # True

# Option 2: for money and anything needing exactness, use Decimal
from decimal import Decimal
Decimal("0.1") + Decimal("0.2")    # Decimal('0.3')  ✅

# Option 3: for fractions, use Fraction
from fractions import Fraction
Fraction(1, 3) + Fraction(1, 6)    # Fraction(1, 2)

💡 Tip: never use float for money. Use Decimal, or store integer cents.

4.6 Booleans

python
is_ready = True
is_done = False

bool is actually a subclass of int, with True == 1 and False == 0:

python
>>> True + True
2
>>> sum([True, False, True])   # count the Trues
2

Truthiness: any object can be used as a condition. These values are falsy:

python
False, None, 0, 0.0, 0j, "", [], (), {}, set(), range(0)

Everything else is truthy — including -1, "False", "0", and [0].

python
if []:
    print("won't run")     # empty list is falsy

if [0]:
    print("will run")      # non-empty list is truthy, even holding a 0

if "False":
    print("will run")      # a non-empty string is always truthy!

⚠️ Gotcha: if "False": is true. Don't treat strings read from config files or command lines as booleans directly.

4.7 None: the Absence of a Value

python
result = None

None is a special singleton meaning "empty," "not set yet," or "this function returned nothing."

python
def f():
    print("hi")

x = f()       # prints hi
print(x)      # None — no return statement, so None is returned

Test for None with is, not ==:

python
if x is None:        # ✅
if x is not None:    # ✅
if x == None:        # ❌ works but isn't idiomatic, and custom classes may override ==

4.8 Type Conversion

python
int("42")        # 42       string → int
int(3.99)        # 3        truncates, does NOT round!
int("0b101", 2)  # 5        specify a base
float("3.14")    # 3.14
float(7)         # 7.0
str(42)          # "42"
bool(0)          # False
bool("hi")       # True

⚠️ Gotcha: int(3.99) is 3, not 4. For rounding use round():

python
round(3.99)        # 4
round(3.14159, 2)  # 3.14

And round() uses banker's rounding (ties go to even), which also surprises people:

python
>>> round(0.5)
0
>>> round(1.5)
2
>>> round(2.5)
2

Failed conversions raise:

python
>>> int("abc")
ValueError: invalid literal for int() with base 10: 'abc'

Chapter 14 covers handling that gracefully.

4.9 Dynamically Typed vs Strongly Typed

Python is dynamically typed: a variable's type is determined at runtime and can change.

Python is also strongly typed: it won't silently perform nonsensical conversions for you.

python
>>> "3" + 5
TypeError: can only concatenate str (not "int") to str

Compare with JavaScript's "3" + 5 === "35". Python would rather raise an error than guess what you meant. That's a feature.

4.10 Exercises

4.1 Predict each value, then check in the REPL:

python
bool("")
bool(" ")
bool([])
bool([[]])
int(-3.7)
round(-3.5)
True + 1
type(1 / 1)

4.2 Which of these are legal variable names?

_x    2fast    my-var    myVar    class    Class    données    __init__

4.3 Write code that takes a number from the user and prints its square, its cube, and whether it's greater than 100.

4.4 Why is 0.1 + 0.2 != 0.3? Write one line that correctly checks whether they're "effectively equal."

<details> <summary>Answers</summary>

4.1

python
bool("")      # False — empty string
bool(" ")     # True  — one space, non-empty
bool([])      # False — empty list
bool([[]])    # True  — one element (which happens to be an empty list, but that's irrelevant)
int(-3.7)     # -3    — truncates toward zero, not floor
round(-3.5)   # -4    — banker's rounding, ties to even
True + 1      # 2     — bool subclasses int
type(1 / 1)   # <class 'float'> — / always returns a float, even when it divides evenly

4.2

  • _x ✅ legal
  • 2fast ❌ can't start with a digit
  • my-var ❌ hyphen isn't a valid character (it parses as subtraction)
  • myVar ✅ legal but not Python style
  • class ❌ keyword
  • Class ✅ legal (keywords are case-sensitive), but it looks like a class name — don't use it as a variable
  • données ✅ legal (Python 3 allows Unicode identifiers), though generally not recommended
  • __init__ ✅ legal, but dunder names have reserved meaning — don't repurpose them

4.3

python
n = float(input("Enter a number: "))
print(f"Square: {n ** 2}")
print(f"Cube: {n ** 3}")
print(f"Greater than 100: {n > 100}")

4.4 Because 0.1 and 0.2 can't be represented exactly in binary floating point, so the error compounds when they're added.

python
import math
math.isclose(0.1 + 0.2, 0.3)   # True

</details>


5. Operators

5.1 Arithmetic

python
7 + 3     # 10   add
7 - 3     # 4    subtract
7 * 3     # 21   multiply
7 / 3     # 2.3333333333333335   divide (always returns a float)
7 // 3    # 2    floor division
7 % 3     # 1    modulo
7 ** 3    # 343  exponent

⚠️ Watch negatives with `//` and `%`:

python
>>> -7 // 3
-3          # rounds toward negative infinity, not toward zero
>>> -7 % 3
2           # the sign follows the divisor

The invariant a == (a // b) * b + (a % b) always holds.

Common uses of modulo:

python
n % 2 == 0        # is it even
n % 15 == 0       # divisible by both 3 and 5
seconds % 60      # seconds → minutes and seconds
i % len(items)    # wrapping index, never out of range

5.2 Comparison

python
a == b     # equal
a != b     # not equal
a > b      # greater
a < b      # less
a >= b
a <= b

Python supports chained comparisons, which most languages don't:

python
>>> x = 5
>>> 1 < x < 10        # same as 1 < x and x < 10
True
>>> 0 <= score <= 100  # range check — very handy

5.3 Logical Operators

python
True and False    # False   — both must be true
True or False     # True    — either one
not True          # False   — negation

Short-circuit evaluation: and stops at the first falsy value, or at the first truthy one.

python
# Standard idiom: check existence before accessing
if user is not None and user.is_active:
    ...
# If user is None, and short-circuits and user.is_active is never evaluated

⚠️ Gotcha: and/or don't return booleans — they return one of the operands:

python
>>> "a" and "b"
'b'          # both truthy, returns the last
>>> "" or "default"
'default'    # left is falsy, returns the right
>>> 0 or None
None

This is often used for defaults:

python
name = user_input or "Anonymous"   # falls back when user_input is empty

But be careful when 0 or "" are legitimate values — then use an explicit check:

python
port = config_port if config_port is not None else 8080
# not config_port or 8080 — because 0 is a meaningful legal value here
# (it means "let the OS pick a port"), and or would silently replace it with 8080

5.4 Augmented Assignment

python
x = 5
x += 3     # x = x + 3 → 8
x -= 2     # 6
x *= 2     # 12
x /= 4     # 3.0  (note it became a float)
x //= 2    # 1.0
x **= 3    # 1.0
x %= 2     # 1.0

5.5 Identity: is

== compares values; is compares object identity.

python
a = [1, 2, 3]
b = [1, 2, 3]
c = a

a == b    # True  — same contents
a is b    # False — but two distinct list objects
a is c    # True  — the same object

When to use `is`: only for None, True, False, and when you genuinely mean "the same object."

python
if x is None:      # ✅
if x is True:      # usually if x: is enough
if name is "abc":  # ❌ never compare strings this way

⚠️ Gotcha: small-integer caching

python
>>> a = 256; b = 256
>>> a is b
True          # CPython caches integers from -5 to 256
>>> a = 257; b = 257
>>> a is b
False         # outside the cache

That's a CPython implementation detail — don't rely on it. Compare numbers with ==.

5.6 Membership: in

python
"a" in "abc"           # True
3 in [1, 2, 3]         # True
"key" in {"key": 1}    # True  — dicts check keys
5 not in [1, 2, 3]     # True

Performance of in varies wildly by type (covered in Chapter 9):

  • Lists/tuples: scan from the start — slower the more elements (O(n))
  • Sets/dicts: compute the location directly — essentially instant (O(1))
python
# One million elements, testing membership
big_list = list(range(1_000_000))
big_set = set(big_list)

999_999 in big_list    # slow
999_999 in big_set     # hundreds of times faster

5.7 Bitwise Operators

Rare in business code, useful for flag bits and low-level protocols.

python
5 & 3     # 1   AND     0101 & 0011 = 0001
5 | 3     # 7   OR      0101 | 0011 = 0111
5 ^ 3     # 6   XOR     0101 ^ 0011 = 0110
~5        # -6  NOT
5 << 1    # 10  left shift  (× 2)
5 >> 1    # 2   right shift (// 2)

5.8 Precedence

Highest to lowest (don't memorize this — just add parentheses when unsure):

**                          exponent
+x  -x  ~x                  unary sign, bitwise NOT
*  /  //  %                 multiplicative
+  -                        additive
<<  >>                      shifts
&                           bitwise AND
^                           bitwise XOR
|                           bitwise OR
comparisons  ==  !=  <  >  in  is
not                         logical NOT
and                         logical AND
or                          logical OR
python
2 + 3 * 4        # 14, not 20
(2 + 3) * 4      # 20
2 ** 3 ** 2      # 512 — exponent is right-associative: 2 ** (3 ** 2)
-2 ** 2          # -4 — ** binds tighter than unary minus: -(2 ** 2)

💡 Tip: parenthesize complex expressions on purpose. Saving a few characters isn't worth making the reader (including future you) stop and think about precedence.

5.9 The Walrus Operator :=

Introduced in Python 3.8, it assigns inside an expression:

python
# Traditional
data = input()
while data != "quit":
    print(data)
    data = input()

# Walrus
while (data := input()) != "quit":
    print(data)

Used well it removes duplication; overused it hurts readability. A good case:

python
# Avoid calling the function twice
if (n := len(items)) > 10:
    print(f"Too many: {n}")

5.10 Exercises

5.1 Without running it, give each value:

python
17 // 5
-17 // 5
17 % 5
-17 % 5
2 ** 3 ** 2
not 0 or 1
[] or "x"
1 == 1.0
1 is 1.0

5.2 Write a program that takes a number of seconds and prints X hours Y minutes Z seconds.

5.3 Write an expression that tests for a leap year. Rule: divisible by 4 and not by 100, or divisible by 400.

5.4 Swap two variables in one line.

<details> <summary>Answers</summary>

5.1

python
17 // 5      # 3
-17 // 5     # -4   (floor, not -3)
17 % 5       # 2
-17 % 5      # 3    (sign follows the divisor)
2 ** 3 ** 2  # 512  (right-associative)
not 0 or 1   # True (not 0 → True; True or 1 short-circuits to True)
[] or "x"    # 'x'
1 == 1.0     # True (equal values)
1 is 1.0     # False (different types, different objects)

5.2

python
total = int(input("Seconds: "))
hours = total // 3600
minutes = total % 3600 // 60
seconds = total % 60
print(f"{hours} hours {minutes} minutes {seconds} seconds")

Cleaner with divmod:

python
minutes, seconds = divmod(total, 60)
hours, minutes = divmod(minutes, 60)

5.3

python
year = int(input("Year: "))
is_leap = (year % 4 == 0 and year % 100 != 0) or year % 400 == 0
print(is_leap)

The standard library has this too: import calendar; calendar.isleap(year)

5.4

python
a, b = b, a

That's tuple unpacking (Chapter 8). Swapping in Python needs no temp variable. </details>


6. Strings

Strings are the type you'll use most. This chapter goes into detail, because processing text is one of the highest-frequency activities in programming.

6.1 Creating Strings

python
s1 = "double quotes"
s2 = 'single quotes'   # exactly equivalent
s3 = """triple quotes
span lines"""
s4 = '''single triple quotes work too'''

Single and double quotes are interchangeable — pick one and be consistent (the community leans slightly toward double). Switch when there's a conflict:

python
"He said: 'hi'"      # ✅ double outside, single inside
'She said: "hi"'     # ✅ the reverse
"He said: \"hi\""    # works, but less readable

6.2 Escape Sequences

python
"\n"     # newline
"\t"     # tab
"\\"     # a literal backslash
"\""     # double quote
"\'"     # single quote
"\u00e9" # Unicode code point → é

Raw strings r"..." turn escaping off — essential for regexes and Windows paths.

python
print("C:\new\table")       # \n and \t get interpreted; output is a mess
print(r"C:\new\table")      # C:\new\table  ✅

6.3 f-strings: Formatting Done Right

f-strings (Python 3.6+) are the first choice for everyday formatting:

python
name = "Alice"
age = 18

f"I'm {name}, {age} years old"     # I'm Alice, 18 years old
f"Next year: {age + 1}"            # any expression works
f"{name.upper()}"                  # method calls work

Format specifiers (after the colon):

python
pi = 3.14159265

f"{pi:.2f}"        # '3.14'       two decimal places
f"{pi:10.2f}"      # '      3.14' width 10, right-aligned
f"{pi:<10.2f}"     # '3.14      ' left-aligned
f"{pi:^10.2f}"     # '   3.14   ' centered
f"{pi:+.2f}"       # '+3.14'      always show the sign

n = 1234567
f"{n:,}"           # '1,234,567'  thousands separators
f"{n:_}"           # '1_234_567'
f"{n:>12}"         # '     1234567'
f"{n:012}"         # '000001234567' zero-padded

r = 0.8567
f"{r:.1%}"         # '85.7%'      percentage

f"{255:b}"         # '11111111'   binary
f"{255:o}"         # '377'        octal
f"{255:x}"         # 'ff'         hex
f"{255:X}"         # 'FF'
f"{255:08b}"       # '11111111'   zero-padded to 8 digits

The debugging `=` (Python 3.8+):

python
x = 42
print(f"{x=}")           # x=42
print(f"{x * 2 = }")     # x * 2 = 84

Far less typing than print("x =", x), and excellent for throwaway debug lines.

Other formatting styles (know them for reading older code):

python
"I'm {}, {} years old".format(name, age)   # str.format(), Python 3.0+
"I'm %s, %d years old" % (name, age)       # % formatting, oldest, from C

For everyday string building, just use f-strings. The other two aren't deprecated, and they still show up in these cases:

  • Logging: logging recommends logger.info("user %s logged in", name), so formatting only happens if the record is actually emitted — zero cost when the level filters it out. See Chapter 27.
  • Reusable templates: when the format string is defined ahead of time and filled in later (i18n strings, templates in config), only str.format() or string.Template can do it — an f-string evaluates immediately where it's written.

6.4 Strings Are Immutable

python
s = "hello"
s[0] = "H"     # TypeError: 'str' object does not support item assignment

Every method that "modifies" a string actually returns a new one:

python
s = "hello"
t = s.upper()
print(s)   # hello  — the original is unchanged
print(t)   # HELLO

6.5 Indexing and Slicing

python
s = "Python"
#    012345      forward indices
#   -654321      negative indices

s[0]      # 'P'
s[5]      # 'n'
s[-1]     # 'n'   last
s[-2]     # 'o'   second to last
s[10]     # IndexError!

Slicing is s[start:stop:step], and the rule is inclusive start, exclusive stop.

python
s = "Python"

s[0:3]     # 'Pyt'    indices 0,1,2
s[:3]      # 'Pyt'    omit start = from the beginning
s[3:]      # 'hon'    omit stop = to the end
s[:]       # 'Python' full copy
s[1:5:2]   # 'yh'     step 2
s[::-1]    # 'nohtyP' step -1 = reverse
s[::2]     # 'Pto'    every other character
s[-3:]     # 'hon'    last three
s[:-3]     # 'Pyt'    all but the last three

⚠️ Slices never raise for out-of-range — unlike indexing:

python
s[100:200]    # ''  empty string, no error
s[0:100]      # 'Python'

The same slicing rules apply to lists and tuples. It's one of the most-used operations in Python.

6.6 Common String Methods

Case

python
"hello".upper()        # 'HELLO'
"HELLO".lower()        # 'hello'
"hello world".title()  # 'Hello World'
"hello".capitalize()   # 'Hello'
"Hello".swapcase()     # 'hELLO'
"Hello".casefold()     # 'hello'  more aggressive than lower(); use for case-insensitive comparison

Trimming

python
"  hi  ".strip()       # 'hi'      both ends
"  hi  ".lstrip()      # 'hi  '    left
"  hi  ".rstrip()      # '  hi'    right
"xxhixx".strip("x")    # 'hi'      strip specific characters
"file.txt".removesuffix(".txt")   # 'file'  Python 3.9+
"pre_name".removeprefix("pre_")   # 'name'

⚠️ Gotcha: strip("abc") doesn't remove the substring "abc" — it removes any of a, b, or c from both ends. For substrings use removeprefix/removesuffix.

Searching and testing

python
"hello".find("ll")        # 2    returns -1 if not found
"hello".index("ll")       # 2    raises ValueError if not found
"hello".rfind("l")        # 3    search from the right
"hello".count("l")        # 2
"hello".startswith("he")  # True
"hello".endswith((".jpg", ".png"))   # accepts a tuple; any match counts
"ll" in "hello"           # True  ← simplest when you only need existence

Replacing

python
"a-b-c".replace("-", "+")       # 'a+b+c'
"a-b-c".replace("-", "+", 1)    # 'a+b-c'  only the first

Splitting and joining

python
"a,b,c".split(",")           # ['a', 'b', 'c']
"a b  c".split()             # ['a', 'b', 'c']  no arg = split on any whitespace, collapsing runs
"a,b,c".split(",", 1)        # ['a', 'b,c']     at most one split
"a,b,c".rsplit(",", 1)       # ['a,b', 'c']     split from the right
"line1\nline2".splitlines()  # ['line1', 'line2']
"a=1".partition("=")         # ('a', '=', '1')  three pieces

",".join(["a", "b", "c"])    # 'a,b,c'
"".join(["a", "b"])          # 'ab'
"\n".join(lines)             # join with newlines

⚠️ Gotcha: join is called on the separator with the list as the argument. Writing ["a","b"].join(",") is wrong (that's JavaScript).

💡 Tip: use join to concatenate many strings, not += in a loop:

python
# ❌ Slow: every += creates a new string
result = ""
for word in words:
    result += word

# ✅ Fast
result = "".join(words)

Content checks

python
"abc".isalpha()      # True   all letters
"123".isdigit()      # True   all digits
"a1".isalnum()       # True   letters or digits
"  ".isspace()       # True   all whitespace
"Hello".istitle()    # True
"ABC".isupper()      # True

⚠️ "".isdigit() is False (empty string), and "3.14".isdigit() is also False (the dot). The most reliable way to test "is this a number" is try/except (Chapter 14).

Alignment and padding

python
"5".zfill(3)         # '005'
"hi".ljust(10, ".")  # 'hi........'
"hi".rjust(10)       # '        hi'
"hi".center(10, "*") # '****hi****'

6.7 Strings and Encoding

In Python 3, str holds Unicode characters and bytes holds bytes. They are different types.

python
s = "héllo"
b = s.encode("utf-8")     # str → bytes: b'h\xc3\xa9llo'
s2 = b.decode("utf-8")    # bytes → str: 'héllo'

len(s)    # 5  five characters
len(b)    # 6  six bytes (é takes 2 bytes in UTF-8)

Non-Latin scripts make the gap larger — a CJK character is typically 3 bytes in UTF-8:

python
len("中文")                    # 2 characters
len("中文".encode("utf-8"))    # 6 bytes

Mnemonic: encode = human text → machine bytes; decode = bytes → text.

⚠️ Gotcha: if you don't specify an encoding when reading a file, Python uses the platform default, which on some Windows systems isn't UTF-8 — so a UTF-8 file comes back garbled or raises. Always write `encoding="utf-8"` explicitly:

python
open("file.txt", encoding="utf-8")

6.8 Comparing and Sorting Strings

python
"apple" < "banana"     # True   compares Unicode code points, character by character
"Apple" < "apple"      # True   uppercase code points are lower than lowercase
"10" < "9"             # True   ⚠️ string comparison is not numeric comparison!

Case-insensitive comparison:

python
a.casefold() == b.casefold()

6.9 Exercises

6.1 Given s = "Hello, World!", write expressions producing:

  1. "!dlroW ,olleH" (reversed)
  2. "HELLO, WORLD!"
  3. "World"
  4. "Hello"
  5. the number of l characters

6.2 Given a comma-separated string of names from user input with stray spaces (e.g. " Alice, Bob ,Carol "), produce a cleaned list.

6.3 Write a program that checks whether a string is a palindrome, ignoring case and spaces.

6.4 Given path = "/home/user/documents/report.pdf", use string methods to extract the filename report.pdf and the extension pdf.

6.5 Format 1234567.891 as "1,234,567.89".

<details> <summary>Answers</summary>

6.1

python
s = "Hello, World!"
s[::-1]           # '!dlroW ,olleH'
s.upper()         # 'HELLO, WORLD!'
s[7:12]           # 'World'
s[:5]             # 'Hello'
s.count("l")      # 3

6.2

python
raw = " Alice, Bob ,Carol "
names = [name.strip() for name in raw.split(",")]
print(names)   # ['Alice', 'Bob', 'Carol']

(List comprehensions are Chapter 10; for now just learn the idiom.)

6.3

python
text = input("Enter a phrase: ")
cleaned = text.lower().replace(" ", "")
if cleaned == cleaned[::-1]:
    print("Palindrome")
else:
    print("Not a palindrome")

6.4

python
path = "/home/user/documents/report.pdf"
filename = path.split("/")[-1]        # 'report.pdf'
ext = filename.split(".")[-1]         # 'pdf'

In real code, use pathlib (Chapter 15):

python
from pathlib import Path
p = Path(path)
p.name      # 'report.pdf'
p.suffix    # '.pdf'
p.stem      # 'report'

6.5

python
f"{1234567.891:,.2f}"    # '1,234,567.89'

</details>


7. Control Flow

7.1 if / elif / else

python
age = 18

if age >= 18:
    print("Adult")
elif age >= 13:
    print("Teenager")
else:
    print("Child")

Key points:

  • elif is short for "else if"; you can have any number
  • else is optional and there's at most one
  • Top to bottom: the first true branch runs, the rest are skipped

Nesting (avoid it when you can):

python
if is_logged_in:
    if is_admin:
        print("Admin panel")
    else:
        print("User panel")
else:
    print("Please log in")

💡 Tip: use guard clauses to flatten nesting

python
# ❌ Three levels deep
def process(user):
    if user is not None:
        if user.is_active:
            if user.has_permission:
                do_work()

# ✅ Return early, stay flat
def process(user):
    if user is None:
        return
    if not user.is_active:
        return
    if not user.has_permission:
        return
    do_work()

7.2 Conditional Expressions

python
status = "adult" if age >= 18 else "minor"

The form is A if condition else B — a different order from condition ? A : B in other languages, and it reads more like English.

Fine for simple cases. If you find yourself nesting them, switch to if/elif immediately.

7.3 for Loops

Python's for iterates over a collection; it isn't a C-style counting loop.

python
for item in [1, 2, 3]:
    print(item)

for char in "abc":
    print(char)

for key in {"a": 1, "b": 2}:
    print(key)      # iterating a dict gives you keys

range(): generating number sequences

python
range(5)         # 0, 1, 2, 3, 4        (stop is exclusive)
range(2, 5)      # 2, 3, 4
range(0, 10, 2)  # 0, 2, 4, 6, 8
range(5, 0, -1)  # 5, 4, 3, 2, 1
python
for i in range(5):
    print(i)       # 0 1 2 3 4

range doesn't build all the numbers up front — it's lazy, so range(10**9) uses almost no memory. To see the contents, convert: list(range(5)).

enumerate(): index and value together

python
fruits = ["apple", "banana", "orange"]

# ❌ C-style
for i in range(len(fruits)):
    print(i, fruits[i])

# ✅ Pythonic
for i, fruit in enumerate(fruits):
    print(i, fruit)

# Number from 1
for i, fruit in enumerate(fruits, start=1):
    print(f"{i}. {fruit}")

zip(): iterate several sequences in parallel

python
names = ["Alice", "Bob"]
ages = [25, 30]

for name, age in zip(names, ages):
    print(f"{name}: {age}")

⚠️ zip stops at the shortest input. When lengths must match, use zip(a, b, strict=True) (Python 3.10+) — mismatched lengths then raise, which is usually what you want.

reversed() / sorted()

python
for x in reversed([1, 2, 3]):     # 3 2 1
    ...
for x in sorted([3, 1, 2]):       # 1 2 3
    ...

7.4 while Loops

Repeat while a condition holds:

python
count = 0
while count < 5:
    print(count)
    count += 1

When to prefer while over for: when you don't know the number of iterations in advance.

python
# Keep asking until the input is valid
while True:
    answer = input("y/n? ").lower()
    if answer in ("y", "n"):
        break
    print("Please enter y or n")

⚠️ Gotcha: infinite loops. Forgetting to update the condition variable is the classic mistake:

python
count = 0
while count < 5:
    print(count)     # forgot count += 1 → prints 0 forever

Press Ctrl-C to interrupt.

7.5 break / continue / else

python
for i in range(10):
    if i == 3:
        continue      # skip the rest of this iteration
    if i == 6:
        break         # exit the loop entirely
    print(i)
# Output: 0 1 2 4 5

Loop `else` (a Python peculiarity — uncommon but genuinely useful): runs when the loop finishes normally, i.e. without hitting break.

python
for item in items:
    if item.is_target:
        print("Found it")
        break
else:
    print("Went through everything and found nothing")

Mnemonic: read "for ... else" as "if we never found it, then."

⚠️ break only exits the innermost loop. To leave several levels, use a flag, or extract the loops into a function and return:

python
def find_pair(matrix, target):
    for row in matrix:
        for x in row:
            if x == target:
                return True     # exits every level at once
    return False

7.6 match Statements (Python 3.10+)

Structural pattern matching — far more capable than a switch:

python
def handle(command):
    match command.split():
        case ["quit"]:
            return "Quitting"
        case ["go", direction]:
            return f"Heading {direction}"
        case ["drop", *items]:
            return f"Dropping {len(items)} items"
        case _:
            return "Didn't understand that"

It matches dicts and objects too:

python
match response:
    case {"status": 200, "data": data}:
        print(f"Success: {data}")
    case {"status": 404}:
        print("Not found")
    case {"status": code} if code >= 500:
        print(f"Server error {code}")
    case _:
        print("Unknown response")

⚠️ Gotcha: a bare name in a case is a binding, not a comparison.

python
STATUS_OK = 200
match code:
    case STATUS_OK:      # ❌ this assigns code to STATUS_OK and always matches!
        ...
    case Status.OK:      # ✅ only dotted names compare by value
        ...

For simple value dispatch, a dict is usually clearer:

python
handlers = {"start": do_start, "stop": do_stop}
handlers.get(command, do_unknown)()

7.7 pass / ... as Placeholders

python
def todo():
    pass          # syntax needs a statement; haven't decided what yet

class Empty:
    ...           # Ellipsis; same effect, common in type-stub contexts

7.8 Putting It Together: Guess the Number

python
import random

secret = random.randint(1, 100)
attempts = 0
MAX_ATTEMPTS = 7

print(f"I'm thinking of a number from 1 to 100. You get {MAX_ATTEMPTS} guesses.")

while attempts < MAX_ATTEMPTS:
    raw = input(f"Guess #{attempts + 1}: ")

    if not raw.isdigit():
        print("Please enter a number")
        continue          # invalid input doesn't cost a guess

    guess = int(raw)
    attempts += 1

    if guess == secret:
        print(f"Got it in {attempts} guesses!")
        break
    elif guess < secret:
        print("Too low")
    else:
        print("Too high")
else:
    print(f"Out of guesses. It was {secret}")

This uses while, if/elif/else, continue, break, while...else, f-strings, type conversion, and a standard library call.

7.9 Exercises

7.1 Print every number from 1 to 100 divisible by 3 but not by 5.

7.2 Classic FizzBuzz: print 1 to 30; print Fizz for multiples of 3, Buzz for multiples of 5, FizzBuzz for both, and the number otherwise.

7.3 Print a five-row triangle with loops:

*
**
***
****
*****

7.4 Read a number and determine whether it's prime.

7.5 Using while, keep reading numbers until the user enters a blank line, then print the sum and average.

7.6 Print a multiplication table.

<details> <summary>Answers</summary>

7.1

python
for n in range(1, 101):
    if n % 3 == 0 and n % 5 != 0:
        print(n)

7.2

python
for n in range(1, 31):
    if n % 15 == 0:
        print("FizzBuzz")
    elif n % 3 == 0:
        print("Fizz")
    elif n % 5 == 0:
        print("Buzz")
    else:
        print(n)

Order matters: check 15 first, or the FizzBuzz branch is unreachable.

7.3

python
for i in range(1, 6):
    print("*" * i)

7.4

python
n = int(input("Enter a number: "))

if n < 2:
    print("Not prime")
else:
    for i in range(2, int(n ** 0.5) + 1):
        if n % i == 0:
            print(f"Not prime, divisible by {i}")
            break
    else:
        print("Prime")

Checking only up to √n is the key optimization: if n = a × b with a ≤ b, then a ≤ √n.

7.5

python
numbers = []
while True:
    raw = input("Enter a number (blank to finish): ")
    if raw == "":
        break
    numbers.append(float(raw))

if numbers:
    print(f"Sum: {sum(numbers)}")
    print(f"Average: {sum(numbers) / len(numbers)}")
else:
    print("No numbers entered")

Note the if numbers: — without it, an empty list causes a division by zero.

7.6

python
for i in range(1, 10):
    for j in range(1, i + 1):
        print(f"{j}x{i}={i*j}", end="\t")
    print()

</details>


Part 3 · Data Structures

So far we've handled single values. Real programs organize data in bulk — that's what data structures are for. Python has four core containers built in: lists, tuples, dicts, and sets. Picking the right one can halve your code and make it ten times faster.

8. Lists and Tuples

8.1 Lists: Ordered and Mutable

python
fruits = ["apple", "banana", "orange"]
numbers = [1, 2, 3]
mixed = [1, "a", 3.14, True, None, [1, 2]]    # any mix of types
empty = []

Access and slicing (identical rules to strings):

python
fruits[0]      # 'apple'
fruits[-1]     # 'orange'
fruits[1:]     # ['banana', 'orange']
fruits[::-1]   # ['orange', 'banana', 'apple']

Modification — this is the big difference from strings; lists are mutable:

python
fruits[0] = "pear"
fruits[1:3] = ["grape"]      # slice assignment: replace two elements with one
python
lst = [1, 2, 3]

# Add
lst.append(4)             # [1, 2, 3, 4]        one item at the end
lst.extend([5, 6])        # [1, 2, 3, 4, 5, 6]  several items at the end
lst.insert(0, 0)          # [0, 1, 2, ...]      insert at a position
lst += [7]                # same as extend

# Remove
lst.remove(3)             # remove the first 3; ValueError if absent
x = lst.pop()             # remove and return the last item
x = lst.pop(0)            # remove and return index 0
del lst[0]                # delete by index
del lst[1:3]              # delete by slice
lst.clear()               # empty it

# Search
lst.index(2)              # index of the first 2; ValueError if absent
lst.count(2)              # how many 2s
2 in lst                  # True

# Reorder
lst.sort()                # sorts in place, returns None
lst.sort(reverse=True)    # descending
lst.reverse()             # reverses in place

⚠️ Gotcha 1: append vs extend

python
a = [1, 2]
a.append([3, 4])    # [1, 2, [3, 4]]    ← the whole list becomes one element
b = [1, 2]
b.extend([3, 4])    # [1, 2, 3, 4]      ← elements added individually

⚠️ Gotcha 2: in-place methods return None

python
lst = [3, 1, 2]
result = lst.sort()      # result is None!
print(result)            # None

sort(), reverse(), append() and friends mutate in place and return None. To get a new list, use sorted() / reversed():

python
new = sorted(lst)         # ✅ returns a new list; the original is untouched
new = list(reversed(lst)) # ✅

Mnemonic: the bare verb mutates (sort), the -ed form returns a new object (sorted).

⚠️ Gotcha 3: removing while iterating

python
lst = [1, 2, 3, 4]
for x in lst:
    if x % 2 == 0:
        lst.remove(x)     # ❌ result is [1, 3, 4] — indices shift and 3 gets skipped

The right approach — build a new list:

python
lst = [x for x in lst if x % 2 != 0]     # ✅

Or iterate backwards (if you truly must mutate in place):

python
for i in range(len(lst) - 1, -1, -1):
    if lst[i] % 2 == 0:
        del lst[i]

8.3 Sorting in Depth

python
words = ["banana", "Apple", "cherry"]

sorted(words)                          # ['Apple', 'banana', 'cherry']  uppercase first
sorted(words, key=str.lower)           # ['Apple', 'banana', 'cherry']  case-insensitive
sorted(words, key=len)                 # by length
sorted(words, reverse=True)            # descending

The key parameter takes a function applied to each element; the return value is what gets compared. This is where Python's sorting really shines:

python
people = [
    {"name": "Alice", "age": 30},
    {"name": "Bob", "age": 25},
    {"name": "Carol", "age": 30},
]

# By age
sorted(people, key=lambda p: p["age"])

# Multi-level: age first, then name
sorted(people, key=lambda p: (p["age"], p["name"]))

# Age descending, name ascending (trick: negate the number)
sorted(people, key=lambda p: (-p["age"], p["name"]))

💡 sorted is a stable sort — equal elements keep their relative order. So multi-level sorting can also be done in passes (sort by the secondary key first, then the primary).

operator.itemgetter is both faster and clearer than a lambda:

python
from operator import itemgetter
sorted(people, key=itemgetter("age", "name"))

8.4 Tuples: Ordered and Immutable

python
point = (3, 4)
single = (42,)        # ⚠️ a one-element tuple needs the comma! (42) is just a parenthesized int
empty = ()
no_paren = 3, 4       # parens are optional; the comma is what matters

Tuples can't be modified:

python
point[0] = 5     # TypeError

When to prefer a tuple over a list:

  1. The data shouldn't change — coordinates, RGB colors, a database row
  2. You need a dict key — lists aren't hashable, tuples are
python
locations = {(0, 0): "origin", (1, 1): "diagonal"}   # ✅ tuple key
locations = {[0, 0]: "origin"}                       # ❌ TypeError: unhashable type
  1. Returning multiple values from a function — that's a tuple under the hood
python
def divmod_(a, b):
    return a // b, a % b     # returns a tuple

q, r = divmod_(17, 5)        # 3, 2

8.5 Unpacking

Extremely common syntactic sugar in Python:

python
a, b = 1, 2               # a=1, b=2
a, b = b, a               # swap, no temp needed

x, y, z = [1, 2, 3]       # lists unpack too
name, age = ("Alice", 25)

# Star collects the rest
first, *rest = [1, 2, 3, 4]      # first=1, rest=[2, 3, 4]
*init, last = [1, 2, 3, 4]       # init=[1, 2, 3], last=4
a, *mid, z = [1, 2, 3, 4, 5]     # a=1, mid=[2,3,4], z=5

# Underscore by convention for values you don't need
_, important, _ = (1, 2, 3)

# Nested unpacking
(a, b), c = (1, 2), 3

Unpacking is especially nice in loops:

python
pairs = [(1, "a"), (2, "b")]
for num, letter in pairs:
    print(num, letter)

⚠️ A count mismatch raises:

python
a, b = [1, 2, 3]     # ValueError: too many values to unpack

8.6 Reference Semantics: The Big Trap

As mentioned earlier, variables are labels, not boxes. For mutable objects (lists, dicts, sets) that distinction really matters.

python
a = [1, 2, 3]
b = a            # b and a point at the same list
b.append(4)
print(a)         # [1, 2, 3, 4]  ← a changed too!

Ways to copy a list:

python
b = a.copy()        # recommended
b = a[:]            # slice copy
b = list(a)         # construct a new list

But all three are shallow — only the outer level is copied:

python
a = [[1, 2], [3, 4]]
b = a.copy()
b[0].append(99)
print(a)      # [[1, 2, 99], [3, 4]]  ← the inner lists are still shared!

Nested structures need a deep copy:

python
import copy
b = copy.deepcopy(a)     # recursively copies every level

⚠️ *Gotcha: `` with nested lists**

python
grid = [[0] * 3] * 3        # ❌ all three rows are the same list
grid[0][0] = 1
print(grid)                  # [[1, 0, 0], [1, 0, 0], [1, 0, 0]]

grid = [[0] * 3 for _ in range(3)]   # ✅ each row is a fresh list

8.7 Useful Built-ins

python
nums = [3, 1, 4, 1, 5, 9, 2, 6]

len(nums)         # 8
sum(nums)         # 31
max(nums)         # 9
min(nums)         # 1
sorted(nums)      # [1, 1, 2, 3, 4, 5, 6, 9]
any([0, 1, 0])    # True   at least one truthy
all([1, 1, 0])    # False  all must be truthy
list(reversed(nums))

# max/min also take key
max(words, key=len)                 # longest word
max(people, key=lambda p: p["age"]) # oldest person

# sum takes a start value
sum([[1], [2]], [])                 # [1, 2]  (concatenates lists, but inefficiently)

💡 On empty sequences: any([]) is False and all([]) is True ("all zero elements satisfy the condition" — vacuous truth). This bites occasionally.

8.8 Two-Dimensional Lists

python
matrix = [
    [1, 2, 3],
    [4, 5, 6],
    [7, 8, 9],
]

matrix[1][2]        # 6   row 2, column 3

# Iterating
for row in matrix:
    for value in row:
        print(value, end=" ")
    print()

# Transpose
transposed = list(zip(*matrix))    # [(1,4,7), (2,5,8), (3,6,9)]

The * in zip(*matrix) is argument unpacking: each row is passed to zip as a separate argument. It's a very idiomatic trick.

💡 For heavy numeric work, use numpy. Its 2D arrays are dozens of times faster than nested lists and much nicer to work with.

8.9 Exercises

8.1 Given nums = [3, 7, 1, 9, 4, 1, 7]:

  1. Find the max, min, and average
  2. Deduplicate and sort
  3. Find all values greater than 3
  4. Reverse the list (two ways)

8.2 What does this print, and why?

python
a = [1, 2, 3]
b = a
c = a[:]
a.append(4)
print(b, c)

8.3 Flatten [[1,2],[3,4],[5,6]] into [1,2,3,4,5,6].

8.4 Given scores = [("Alice", 85), ("Bob", 92), ("Carol", 78)], sort by score descending and print the ranking.

8.5 Write a function that removes duplicates from a list while preserving the original order.

8.6 Why is [[0]*3]*3 wrong? Write the correct way to create a 3×3 zero matrix.

<details> <summary>Answers</summary>

8.1

python
nums = [3, 7, 1, 9, 4, 1, 7]

max(nums)                      # 9
min(nums)                      # 1
sum(nums) / len(nums)          # 4.571428571428571

sorted(set(nums))              # [1, 3, 4, 7, 9]

[n for n in nums if n > 3]     # [7, 9, 4, 7]

nums[::-1]                     # slice, returns a new list
list(reversed(nums))           # same
nums.reverse()                 # in place, returns None

8.2 It prints [1, 2, 3, 4] [1, 2, 3]. b = a just adds another label to the same list, so changes to a show up in b. c = a[:] creates a new list, so c is unaffected.

8.3

python
nested = [[1, 2], [3, 4], [5, 6]]

# Way 1: comprehension (most common)
flat = [x for row in nested for x in row]

# Way 2: itertools
from itertools import chain
flat = list(chain.from_iterable(nested))

# Way 3: loop
flat = []
for row in nested:
    flat.extend(row)

8.4

python
scores = [("Alice", 85), ("Bob", 92), ("Carol", 78)]
ranked = sorted(scores, key=lambda s: s[1], reverse=True)

for rank, (name, score) in enumerate(ranked, start=1):
    print(f"#{rank}: {name}, {score} points")

8.5

python
def dedup(items):
    seen = set()
    result = []
    for item in items:
        if item not in seen:
            seen.add(item)
            result.append(item)
    return result

Since Python 3.7 dicts preserve insertion order, so there's also a one-liner:

python
def dedup(items):
    return list(dict.fromkeys(items))

8.6 [[0]*3]*3 creates three references to the same list, so changing one row changes all three.

python
grid = [[0] * 3 for _ in range(3)]   # ✅

</details>


9. Dictionaries and Sets

9.1 Dicts: Key-Value Mappings

python
person = {
    "name": "Alice",
    "age": 25,
    "city": "Berlin",
}

empty = {}
from_pairs = dict([("a", 1), ("b", 2)])
from_kwargs = dict(name="Alice", age=25)

Access:

python
person["name"]              # 'Alice'
person["email"]             # ❌ KeyError!

person.get("email")         # None, no error
person.get("email", "n/a")  # 'n/a', custom default

💡 Rule of thumb: use [] when the key must exist (and a missing key should be an error); use .get() when it's optional.

Modifying and adding:

python
person["age"] = 26              # update
person["email"] = "[email protected]"     # add if missing

person.update({"age": 27, "phone": "555-0100"})   # bulk update
person |= {"age": 28}                             # merge syntax, Python 3.9+

Deleting:

python
del person["email"]
age = person.pop("age")             # remove and return the value
age = person.pop("age", None)       # default if missing, no error
key, val = person.popitem()         # remove and return the last inserted pair
person.clear()

Iterating:

python
for key in person:                  # keys by default
    print(key)

for key in person.keys():           # explicit, equivalent
    ...
for value in person.values():
    ...
for key, value in person.items():   # most common
    print(f"{key}: {value}")

Membership:

python
"name" in person          # True   — checks keys
"Alice" in person.values() # True  — checking values requires being explicit

9.2 Properties of Dicts

Ordered: since Python 3.7, dicts are guaranteed to iterate in insertion order. (It was an implementation detail in 3.6, part of the language spec from 3.7.)

Keys must be hashable: strings, numbers, tuples, and frozensets qualify; lists, dicts, and sets don't.

python
d = {[1, 2]: "x"}     # TypeError: unhashable type: 'list'
d = {(1, 2): "x"}     # ✅

Lookup is O(1): no matter how large the dict, finding a key is essentially instantaneous. That's a dict's whole value proposition.

9.3 Dict Techniques

Counting frequencies

python
text = "hello world"
counts = {}
for char in text:
    counts[char] = counts.get(char, 0) + 1

collections.Counter is simpler:

python
from collections import Counter
counts = Counter(text)
counts.most_common(3)       # the three most frequent: [('l', 3), ('o', 2), ...]

Grouping

python
from collections import defaultdict

words = ["apple", "avocado", "banana", "blueberry"]
groups = defaultdict(list)
for word in words:
    groups[word[0]].append(word)
# {'a': ['apple', 'avocado'], 'b': ['banana', 'blueberry']}

defaultdict(list) creates an empty list automatically for missing keys, saving the if key not in d dance.

You can also use setdefault with a plain dict:

python
groups = {}
for word in words:
    groups.setdefault(word[0], []).append(word)

Merging dicts

python
a = {"x": 1, "y": 2}
b = {"y": 20, "z": 3}

merged = {**a, **b}       # {'x': 1, 'y': 20, 'z': 3}  later wins
merged = a | b            # Python 3.9+, same thing, clearer

Inverting

python
inverted = {v: k for k, v in d.items()}

Dict comprehensions

python
squares = {n: n**2 for n in range(5)}        # {0:0, 1:1, 2:4, 3:9, 4:16}
filtered = {k: v for k, v in d.items() if v is not None}

Nested dicts

python
config = {
    "database": {"host": "localhost", "port": 5432},
    "cache": {"ttl": 300},
}

config["database"]["host"]                      # 'localhost'
config.get("cache", {}).get("size", 100)        # safely reach deep values

9.4 Sets: Unordered, No Duplicates

python
s = {1, 2, 3}
s = set([1, 2, 2, 3])       # {1, 2, 3}  duplicates dropped
empty = set()               # ⚠️ not {} — that's an empty dict

Basic operations:

python
s.add(4)
s.remove(4)          # KeyError if absent
s.discard(4)         # no error if absent
s.pop()              # removes an arbitrary element (which one is unspecified — and don't
                     # count on it being random)
s.clear()
len(s)
3 in s               # O(1), very fast

Set algebra:

python
a = {1, 2, 3}
b = {2, 3, 4}

a | b        # {1, 2, 3, 4}   union            a.union(b)
a & b        # {2, 3}         intersection     a.intersection(b)
a - b        # {1}            difference       a.difference(b)
a ^ b        # {1, 4}         symmetric diff   a.symmetric_difference(b)

a <= b       # subset?         a.issubset(b)
a >= b       # superset?
a.isdisjoint(b)   # no elements in common?

The two main uses for sets:

  1. Deduplication
python
unique = list(set(items))           # but loses order
unique = list(dict.fromkeys(items)) # preserves order
  1. Fast membership tests
python
# ❌ list lookup is O(n)
banned = ["a", "b", "c", ...]      # 10,000 entries
if word in banned:  ...             # slow

# ✅ set lookup is O(1)
banned = {"a", "b", "c", ...}
if word in banned:  ...             # fast

frozenset: an immutable set, so it can be a dict key.

python
fs = frozenset([1, 2, 3])
d = {fs: "value"}     # ✅

9.5 Choosing Among the Four

NeedUse
Ordered, add/remove/modifylist
Ordered, immutable, usable as a dict keytuple
Key-value mapping, fast lookup by keydict
Deduplication, fast membership, set algebraset
Propertylisttupledictset
Ordered✅ (insertion)
Mutable
Duplicates allowedkeys unique
Index accessO(1)O(1)by key, O(1)
in lookupO(n)O(n)O(1)O(1)
Usable as dict key

9.6 The collections Module

The standard library offers a few more specialized containers:

python
from collections import Counter, defaultdict, deque, namedtuple, ChainMap

# Counter — counting
c = Counter("mississippi")
c.most_common(2)          # [('i', 4), ('s', 4)]
c["i"]                    # 4
c["z"]                    # 0 (no error)

# defaultdict — dict with automatic defaults
d = defaultdict(int)      # defaults to 0
d["x"] += 1               # no initialization needed
d = defaultdict(list)     # defaults to []

# deque — double-ended queue, O(1) at both ends
q = deque([1, 2, 3])
q.appendleft(0)           # list.insert(0, x) is O(n); deque is O(1)
q.popleft()
q = deque(maxlen=5)       # fixed size, drops the oldest when full

# namedtuple — a tuple with field names (dataclass is usually better; see Ch 18)
Point = namedtuple("Point", ["x", "y"])
p = Point(3, 4)
p.x                       # 3

9.7 Exercises

9.1 Given a block of text, count word frequencies and print the five most common.

9.2 With a = [1,2,3,4,5] and b = [4,5,6,7], find:

  1. Elements in both
  2. Elements only in a
  3. All elements from either, deduplicated

9.3 Given students = [{"name":"Alice","class":"A"}, {"name":"Bob","class":"B"}, {"name":"Carol","class":"A"}], group by class to produce {"A": ["Alice","Carol"], "B": ["Bob"]}.

9.4 Determine whether two strings are anagrams (same letters, different order — e.g. "listen" and "silent").

9.5 Why does this fail, and how do you fix it?

python
d = {}
d[[1, 2]] = "value"

9.6 Write a function that takes a nested dict and a dot-separated path string (like "database.host") and returns the value, or None if the path doesn't exist.

<details> <summary>Answers</summary>

9.1

python
from collections import Counter

text = "the quick brown fox jumps over the lazy dog the end"
words = text.lower().split()
counts = Counter(words)

for word, count in counts.most_common(5):
    print(f"{word}: {count}")

9.2

python
a = {1, 2, 3, 4, 5}
b = {4, 5, 6, 7}

a & b       # {4, 5}
a - b       # {1, 2, 3}
a | b       # {1, 2, 3, 4, 5, 6, 7}

9.3

python
from collections import defaultdict

students = [
    {"name": "Alice", "class": "A"},
    {"name": "Bob", "class": "B"},
    {"name": "Carol", "class": "A"},
]

groups = defaultdict(list)
for s in students:
    groups[s["class"]].append(s["name"])

print(dict(groups))   # {'A': ['Alice', 'Carol'], 'B': ['Bob']}

itertools.groupby also works (it's an old standard-library member, not a new feature), but it only groups adjacent equal keys, so you must sort by the same key first — less direct than the above.

9.4

python
def is_anagram(a, b):
    return sorted(a.lower()) == sorted(b.lower())

# Or with Counter, which reads more clearly
from collections import Counter
def is_anagram(a, b):
    return Counter(a.lower()) == Counter(b.lower())

Don't use set(a) == set(b) — that would call "aab" and "abb" anagrams.

9.5 Lists aren't hashable and can't be dict keys. Use a tuple:

python
d[(1, 2)] = "value"

9.6

python
def get_nested(data, path, default=None):
    current = data
    for key in path.split("."):
        if not isinstance(current, dict) or key not in current:
            return default
        current = current[key]
    return current

config = {"database": {"host": "localhost", "port": 5432}}
get_nested(config, "database.host")      # 'localhost'
get_nested(config, "database.user")      # None
get_nested(config, "cache.ttl", 300)     # 300

</details>


10. Comprehensions

Comprehensions are one of Python's most recognizable pieces of syntax. They compress "build a new collection" into a single line.

10.1 List Comprehensions

Basic form:

python
[expression for variable in iterable]
python
# Traditional
squares = []
for n in range(5):
    squares.append(n ** 2)

# Comprehension
squares = [n ** 2 for n in range(5)]      # [0, 1, 4, 9, 16]

With a filter:

python
[expression for variable in iterable if condition]
python
evens = [n for n in range(20) if n % 2 == 0]
long_words = [w for w in words if len(w) > 5]
valid = [int(x) for x in inputs if x.isdigit()]

With a conditional expression (note the different position!):

python
# if at the end = filtering (decide whether to include the element)
[n for n in nums if n > 0]

# if...else at the front = transformation (include everything, just different values)
[n if n > 0 else 0 for n in nums]

This is where beginners get confused most often. Remember: the part before `for` says what to produce; the `if` after `for` says whether to include it.

Multiple loops:

python
# Flatten a nested list
flat = [x for row in matrix for x in row]

# Equivalent to
flat = []
for row in matrix:
    for x in row:
        flat.append(x)

Note the for clauses come in the same order as the nested loops (outer first).

Cartesian product:

python
pairs = [(a, b) for a in "AB" for b in [1, 2]]
# [('A', 1), ('A', 2), ('B', 1), ('B', 2)]

10.2 Dict and Set Comprehensions

python
# Dict comprehension
{k: v for ...}
squares = {n: n**2 for n in range(5)}
inverted = {v: k for k, v in d.items()}
upper_keys = {k.upper(): v for k, v in d.items()}
filtered = {k: v for k, v in d.items() if v is not None}

# Set comprehension
{expression for ...}
unique_lengths = {len(w) for w in words}

10.3 Generator Expressions

Swap the brackets for parentheses and you get a generator — lazily evaluated, essentially free memory-wise:

python
squares_list = [n**2 for n in range(1000000)]    # builds a million elements now, tens of MB
squares_gen = (n**2 for n in range(1000000))     # almost no memory; computes one at a time

When it's the sole argument to a function, the parentheses can be omitted:

python
sum(n**2 for n in range(100))          # no need for sum((n**2 for ...))
any(x > 100 for x in nums)
max(len(w) for w in words)
"\n".join(str(x) for x in nums)

💡 When to use a generator expression: when you iterate once, when the data is large, or when you only need an aggregate. Use a list when you need repeated access, indexing, or len().

Chapter 19 covers generators properly.

10.4 When Not to Use a Comprehension

Comprehensions feel great, but overusing them produces code nobody can read.

❌ Too complex

python
result = [transform(x) if cond1(x) else other(x)
          for sublist in data if check(sublist)
          for x in sublist if x is not None and validate(x)]

Write a plain loop. Readability beats cleverness by a mile.

❌ Used purely for side effects

python
[print(x) for x in items]     # ❌ builds a list of Nones nobody wants
for x in items: print(x)      # ✅

The test: if it doesn't fit on one line, or if you have to stop and think for a second to parse it, use a loop.

10.5 Common Patterns

python
# Convert types
[int(x) for x in strings]

# Filter out None
[x for x in items if x is not None]

# Extract a field
[p["name"] for p in people]

# Clean lines of text
[line.strip() for line in lines if line.strip()]

# Index plus value
[f"{i}: {v}" for i, v in enumerate(items)]

# Conditional transformation
[x if x > 0 else 0 for x in nums]

# Pair two lists
[(a, b) for a, b in zip(list1, list2)]

# Flatten
[x for sub in nested for x in sub]

# Build a lookup dict
{item["id"]: item for item in items}

# Invert a dict
{v: k for k, v in d.items()}

# Deduplicate (unordered)
{x for x in items}

10.6 Exercises

10.1 Use comprehensions to produce:

  1. The squares of all even numbers from 1 to 20
  2. All words longer than 3 characters in a sentence, uppercased
  3. From ["1", "a", "2", "b", "3"], the convertible items as integers, dropping the rest
  4. A 3×3 multiplication table (nested list)

10.2 What's the difference between these two lines?

python
[x for x in range(10) if x % 2 == 0]
[x if x % 2 == 0 else None for x in range(10)]

10.3 Given data = {"a": 1, "b": None, "c": 3, "d": None}, use a dict comprehension to drop the None values.

10.4 In one line, sum every multiple of 7 from 1 to 100.

10.5 Transpose a 2D list using a comprehension (without zip).

<details> <summary>Answers</summary>

10.1

python
# 1
[n**2 for n in range(1, 21) if n % 2 == 0]

# 2
sentence = "the quick brown fox jumps"
[w.upper() for w in sentence.split() if len(w) > 3]

# 3
items = ["1", "a", "2", "b", "3"]
[int(x) for x in items if x.isdigit()]

# 4
[[i * j for j in range(1, 4)] for i in range(1, 4)]
# [[1, 2, 3], [2, 4, 6], [3, 6, 9]]

10.2

  • The first filters, producing 5 elements: [0, 2, 4, 6, 8]
  • The second transforms, producing 10 elements: [0, None, 2, None, 4, ...]

10.3

python
data = {"a": 1, "b": None, "c": 3, "d": None}
cleaned = {k: v for k, v in data.items() if v is not None}
# {'a': 1, 'c': 3}

10.4

python
sum(n for n in range(1, 101) if n % 7 == 0)    # 735

A generator expression, not a list comprehension — no intermediate list needed.

10.5

python
matrix = [[1, 2, 3], [4, 5, 6]]
transposed = [[row[i] for row in matrix] for i in range(len(matrix[0]))]
# [[1, 4], [2, 5], [3, 6]]

</details>


Part 4 · Organizing Code

11. Functions

Once a program grows past a few dozen lines it needs to be broken into pieces. Functions are the basic unit.

11.1 Defining and Calling

python
def greet(name):
    """Say hello."""            # docstring
    return f"Hello, {name}"

message = greet("Alice")
print(message)                  # Hello, Alice
  • def starts a definition
  • The names in parentheses are parameters
  • return sends a value back to the caller
  • Without return, a function returns None

return exits immediately:

python
def check(n):
    if n < 0:
        return "negative"
    return "non-negative"
    print("never runs")     # dead code

11.2 Parameters

Positional

python
def power(base, exponent):
    return base ** exponent

power(2, 3)      # 8   matched by position

Defaults

python
def greet(name, greeting="Hello"):
    return f"{greeting}, {name}"

greet("Alice")                   # Hello, Alice
greet("Alice", "Good morning")   # Good morning, Alice

Defaulted parameters must come after non-defaulted ones:

python
def f(a=1, b): ...       # ❌ SyntaxError

⚠️ An important trap: mutable default arguments

python
def add_item(item, items=[]):    # ❌ dangerous!
    items.append(item)
    return items

add_item("a")     # ['a']
add_item("b")     # ['a', 'b']   ← it remembered the previous call!

The reason: the default value is evaluated once, at definition time, and every call shares that same list.

The correct form:

python
def add_item(item, items=None):
    if items is None:
        items = []
    items.append(item)
    return items

Rule: default arguments must be immutable (numbers, strings, tuples, None).

Keyword arguments

Call with name=value and the order stops mattering:

python
power(exponent=3, base=2)    # 8

💡 Boolean parameters should almost always be passed by keyword — the readability difference is enormous:

python
save(data, True, False)                    # ❌ what are True and False?
save(data, overwrite=True, backup=False)   # ✅

Variadic parameters: `args and kwargs`

python
def total(*numbers):           # collects any number of positional args into a tuple
    return sum(numbers)

total(1, 2, 3)                 # 6
total(1, 2, 3, 4, 5)           # 15


def config(**options):         # collects any number of keyword args into a dict
    for key, value in options.items():
        print(f"{key} = {value}")

config(debug=True, port=8080)


def flexible(a, b=2, *args, **kwargs):
    print(a, b, args, kwargs)

flexible(1, 2, 3, 4, x=5)      # 1 2 (3, 4) {'x': 5}

The names args/kwargs are convention; the * and ** are what matter.

Unpacking at the call site (the other side of * and **):

python
nums = [1, 2, 3]
total(*nums)                   # same as total(1, 2, 3)

opts = {"debug": True, "port": 8080}
config(**opts)                 # same as config(debug=True, port=8080)

Positional-only and keyword-only parameters

python
def f(a, b, /, c, d, *, e, f):
    ...
#      ↑ everything before / must be positional
#              ↑ everything after * must be keyword
python
def create_user(name, *, admin=False, active=True):
    ...

create_user("Alice", admin=True)      # ✅
create_user("Alice", True)            # ❌ TypeError

Forcing keywords with * is good API design — it stops callers from writing create_user("Alice", True, False), which nobody can read.

11.3 Returning Multiple Values

python
def min_max(numbers):
    return min(numbers), max(numbers)     # actually returns a tuple

lo, hi = min_max([3, 1, 4])               # unpacked
result = min_max([3, 1, 4])               # (1, 4)

Beyond three return values, consider returning a dict or a dataclass (Chapter 18) for readability.

11.4 Docstrings

python
def calculate_bmi(weight, height):
    """Compute body mass index.

    Args:
        weight: mass in kilograms
        height: height in meters

    Returns:
        The BMI as a float

    Raises:
        ValueError: if height is zero or negative
    """
    if height <= 0:
        raise ValueError("Height must be positive")
    return weight / height ** 2

Viewing docs:

python
help(calculate_bmi)
calculate_bmi.__doc__

💡 A one-line function doesn't need this much ceremony — a single sentence is fine. But always document public APIs.

11.5 lambda: Anonymous Functions

python
square = lambda x: x ** 2      # same as def square(x): return x ** 2

A lambda can only contain one expression — no statements, no multiple lines.

Its real purpose is being passed to other functions:

python
sorted(people, key=lambda p: p["age"])
list(filter(lambda x: x > 0, nums))
list(map(lambda x: x * 2, nums))

⚠️ Don't assign a lambda to a variable — just use def. A def gives the function a name, which makes tracebacks far more useful:

python
square = lambda x: x ** 2      # ❌ PEP 8 explicitly discourages this
def square(x): return x ** 2   # ✅

💡 Many lambda uses have better alternatives:

python
map(lambda x: x * 2, nums)         →  [x * 2 for x in nums]
filter(lambda x: x > 0, nums)      →  [x for x in nums if x > 0]
sorted(d, key=lambda x: x[1])      →  sorted(d, key=itemgetter(1))

11.6 Functions Are First-Class

In Python, functions are objects like numbers and strings — you can pass them around:

python
def double(x):
    return x * 2

f = double              # assign to a variable (note: no parentheses)
f(5)                    # 10

funcs = [double, abs, len]        # put them in a list
for fn in funcs:
    print(fn.__name__)

def apply(fn, value):             # take one as a parameter
    return fn(value)
apply(double, 5)                  # 10

def make_multiplier(n):           # return one
    def multiplier(x):
        return x * n
    return multiplier

triple = make_multiplier(3)
triple(5)                         # 15

That last example is a closure, covered in the next chapter. This capability is the foundation of decorators (Chapter 20).

11.7 Recursion

A function calling itself:

python
def factorial(n):
    if n <= 1:          # base case — mandatory!
        return 1
    return n * factorial(n - 1)

factorial(5)            # 120

Recursion needs a base case, or it recurses until RecursionError (Python's default limit is around 1000 frames).

It suits naturally recursive structures: trees, nested directories, nested JSON.

python
def total_size(item):
    """Recursively sum all numbers in a nested list."""
    if isinstance(item, list):
        return sum(total_size(x) for x in item)
    return item

total_size([1, [2, [3, 4]], 5])    # 15

⚠️ When a loop expresses the same thing, the loop is usually faster and safer. The naive Fibonacci is the classic counterexample — fib(35) takes seconds because it recomputes subproblems an exponential number of times. functools.cache fixes it in one line:

python
from functools import cache

@cache
def fib(n):
    return n if n < 2 else fib(n-1) + fib(n-2)

fib(100)     # instant

11.8 Principles for Writing Good Functions

1. One function, one job

python
# ❌ Reads a file, computes, and prints
def process():
    data = open("f.txt").read()
    result = complicated_math(data)
    print(result)

# ✅ Split up — each is testable and reusable
def load_data(path): ...
def compute(data): ...
def report(result): ...

2. Name functions with verbs

python
get_user()  calculate_tax()  is_valid()  has_permission()  send_email()

Boolean-returning functions start with is_ / has_ / can_.

3. Keep parameters under 3–4

More than that means related parameters should be bundled into an object.

4. Avoid side effects, or make them obvious

python
# ❌ The name says "calculate" but it mutates the input
def calculate_total(items):
    items.sort()          # unexpectedly modifies the caller's list!
    return sum(items)

# ✅ Leave the input alone
def calculate_total(items):
    return sum(sorted(items))

5. Return early, nest less (the guard clauses from Chapter 7)

11.9 Exercises

11.1 Write is_prime(n).

11.2 Write a function taking any number of numeric arguments and returning their average, or 0 with no arguments.

11.3 Find and fix the bug:

python
def append_log(msg, logs=[]):
    logs.append(msg)
    return logs

11.4 Write word_count(text) returning a dict of word frequencies, ignoring case and punctuation.

11.5 Write retry(func, times=3) that calls func() and retries on exception up to times times, re-raising the last exception if all attempts fail. (Skip this until after Chapter 14 if you prefer.)

11.6 Use recursion to compute the maximum depth of a nested dict.

<details> <summary>Answers</summary>

11.1

python
def is_prime(n):
    if n < 2:
        return False
    if n == 2:
        return True
    if n % 2 == 0:
        return False
    for i in range(3, int(n ** 0.5) + 1, 2):
        if n % i == 0:
            return False
    return True

11.2

python
def average(*numbers):
    if not numbers:
        return 0
    return sum(numbers) / len(numbers)

average(1, 2, 3)     # 2.0
average()            # 0

11.3 The mutable default argument trap — every call shares one list.

python
def append_log(msg, logs=None):
    if logs is None:
        logs = []
    logs.append(msg)
    return logs

11.4

python
import re
from collections import Counter

def word_count(text):
    words = re.findall(r"\w+", text.lower())
    return dict(Counter(words))

word_count("The cat. The DOG! the cat?")
# {'the': 3, 'cat': 2, 'dog': 1}

11.5

python
def retry(func, times=3):
    last_error = None
    for attempt in range(times):
        try:
            return func()
        except Exception as e:
            last_error = e
            print(f"Attempt {attempt + 1} failed: {e}")
    raise last_error

11.6

python
def max_depth(d):
    if not isinstance(d, dict) or not d:
        return 0
    return 1 + max(max_depth(v) for v in d.values())

max_depth({"a": {"b": {"c": 1}}})    # 3

</details>


12. Scope and Closures

12.1 Four Scopes: LEGB

When Python looks up a name, it checks four places in a fixed order:

L (Local)      — inside the current function
E (Enclosing)  — an outer function (only with nested functions)
G (Global)     — module top level
B (Built-in)   — Python's built-ins (print, len, etc.)
python
x = "global"                  # G

def outer():
    x = "enclosing"           # E
    def inner():
        x = "local"           # L
        print(x)              # local
    inner()
    print(x)                  # enclosing

outer()
print(x)                      # global

The first match wins; no match means NameError.

12.2 Modifying Outer Variables

Reading an outer variable is fine:

python
counter = 0

def show():
    print(counter)    # ✅ readable

But assigning creates a new local:

python
counter = 0

def increment():
    counter = counter + 1     # ❌ UnboundLocalError

Why: Python sees counter = inside the function and classifies counter as local; then evaluating counter + 1 on the right fails because the local has no value yet.

Fixes:

python
counter = 0

def increment():
    global counter            # declare: I mean the module-level one
    counter += 1

Use nonlocal to modify an enclosing function's variable:

python
def outer():
    count = 0
    def inner():
        nonlocal count        # modifies outer's count, not a global
        count += 1
    inner()
    inner()
    return count              # 2

⚠️ `global` is a code smell. Global mutable state makes programs hard to reason about, hard to test, and hard to parallelize. Almost always the better answer is: pass it in, return it out.

python
# ❌
total = 0
def add(x):
    global total
    total += x

# ✅
def add(total, x):
    return total + x

12.3 Closures

An inner function "remembers" the outer function's variables even after the outer function has returned:

python
def make_counter():
    count = 0
    def counter():
        nonlocal count
        count += 1
        return count
    return counter

c1 = make_counter()
c1()    # 1
c1()    # 2
c1()    # 3

c2 = make_counter()
c2()    # 1  — an independent counter with its own count

make_counter has long since returned, but count is still alive because counter holds a reference to it. That's a closure.

Typical use: function factories

python
def make_validator(min_len):
    def validate(text):
        return len(text) >= min_len
    return validate

check_password = make_validator(8)
check_username = make_validator(3)

check_password("abc")      # False
check_username("abc")      # True

⚠️ The classic trap: closures created in a loop

python
funcs = []
for i in range(3):
    funcs.append(lambda: i)

[f() for f in funcs]     # [2, 2, 2]  ← all 2!

Why: all three lambdas reference the same variable `i`, and after the loop i is 2.

The fix — freeze the current value using a default argument:

python
for i in range(3):
    funcs.append(lambda i=i: i)

[f() for f in funcs]     # [0, 1, 2]  ✅

Or use a factory function:

python
def make_f(i):
    return lambda: i
funcs = [make_f(i) for i in range(3)]

12.4 Exercises

12.1 What does this print?

python
x = 10
def f():
    x = 20
    def g():
        nonlocal x
        x = 30
    g()
    print(x)
f()
print(x)

12.2 Fix this code:

python
total = 0
def add(n):
    total += n
    return total

12.3 Use a closure to write an accumulator: acc = make_accumulator(), then acc(10) returns 10, acc(5) returns 15, acc(3) returns 18.

12.4 Explain why the output below is [2, 2, 2] rather than [0, 1, 2], and give two fixes.

python
fs = [lambda: i for i in range(3)]
print([f() for f in fs])

<details> <summary>Answers</summary>

12.1 It prints 30, then 10. nonlocal x in g modifies f's local x (20 → 30). The global x is untouched.

12.2

python
# Option 1: global (not recommended)
total = 0
def add(n):
    global total
    total += n
    return total

# Option 2: make it pure (recommended)
def add(total, n):
    return total + n

# Option 3: encapsulate the state in a closure
def make_adder():
    total = 0
    def add(n):
        nonlocal total
        total += n
        return total
    return add

12.3

python
def make_accumulator():
    total = 0
    def accumulate(n):
        nonlocal total
        total += n
        return total
    return accumulate

acc = make_accumulator()
acc(10)    # 10
acc(5)     # 15
acc(3)     # 18

12.4 All three lambdas reference the same variable i, not a snapshot of its value at creation time. By the time you actually call f(), the comprehension has finished and i is sitting at its final value, 2 — so all three return 2.

The key insight: a closure captures the variable, not the value it held at that moment.

Fixes:

python
# Fix 1: capture via default argument
fs = [lambda i=i: i for i in range(3)]

# Fix 2: factory function
def make(i):
    return lambda: i
fs = [make(i) for i in range(3)]

</details>


13. Modules and Packages

Past a few hundred lines it's time to split files. Python organizes multi-file projects with modules and packages.

13.1 A Module Is a .py File

Create mathtools.py:

python
PI = 3.14159

def circle_area(r):
    return PI * r ** 2

def circle_circumference(r):
    return 2 * PI * r

Use it from another file in the same directory:

python
import mathtools

mathtools.circle_area(2)     # 12.56636
mathtools.PI                 # 3.14159

13.2 Forms of import

python
import math                       # import the whole module
math.sqrt(16)

import math as m                  # with an alias
m.sqrt(16)

from math import sqrt             # import a single name
sqrt(16)

from math import sqrt, pi, floor  # several
from math import sqrt as square_root

from math import *                # ❌ import everything — don't

⚠️ The problem with from x import *: you don't know what names came in, they may silently shadow your own variables, and readers can't tell where sqrt came from. The only acceptable use is interactive experimentation.

Which form to use:

  • Short module name, used a lot → import math; the prefix at each call site is clearest
  • Only one or two functions → from math import sqrt
  • Long name or a conflict → import numpy as np

13.3 Packages: Folders Holding Modules

myproject/
├── main.py
└── utils/
    ├── __init__.py
    ├── text.py
    └── files.py
python
# main.py
from utils.text import clean
from utils import files

import utils.text as text

A folder containing __init__.py is a regular package. That file can be completely empty — its presence marks "this is a package" and gives you a place for package-level initialization.

🔍 Going deeper: since Python 3.3 (PEP 420), a folder without __init__.py can also be imported — that's a namespace package, designed to let one package name span multiple directories (plugin systems, for example). For everyday projects, keep writing __init__.py: it's explicit, its behavior is predictable, and it avoids accidentally ending up with a namespace package because you forgot to create the file.

__init__.py can perform imports to simplify the external interface:

python
# utils/__init__.py
from .text import clean
from .files import read_json

Now outside code can write from utils import clean without caring which submodule it lives in.

Relative imports (used inside a package):

python
from .text import clean       # sibling module
from ..config import SETTINGS # one level up

⚠️ Relative imports only work inside a package. Running a file with relative imports directly gives ImportError: attempted relative import with no known parent package.

13.4 if __name__ == "__main__"

python
# tools.py

def main():
    print("Running the main program")

if __name__ == "__main__":
    main()

__name__ is a built-in variable every module has:

  • Run the file directly and __name__ is "__main__"
  • Import it and __name__ is the module name, "tools"

So the check means: run this code only when executed directly, not when imported.

Without it, import tools would accidentally trigger the main program. It's the standard idiom, present in nearly every runnable Python script.

13.5 The Module Search Path

Simplified, Python looks in this order:

  1. Built-in modules (sys, builtins — compiled into the interpreter, highest priority, can't be shadowed)
  2. The directory of the running script (the current working directory in an interactive session)
  3. Directories in the PYTHONPATH environment variable
  4. The standard library
  5. Third-party packages (site-packages)
python
import sys
print(sys.path)         # the search path actually in effect — this is authoritative

(The full rules also involve sys.meta_path, path rewriting by the site module, and more; see The import system for the complete story. Day to day, "go by sys.path, and your script's own directory comes before the standard library" is enough.)

⚠️ A very common trap: naming your own file the same as a standard library or third-party module.

# Your directory contains random.py
import random           # imports YOUR file, not the standard library!
random.randint(1, 10)   # AttributeError

Don't name files random.py, json.py, email.py, test.py, string.py, types.py, etc.

13.6 How the Standard Library Is Organized

Python comes with "batteries included" — the standard library covers most common needs with nothing to install:

python
import os, sys, math, random, json, re, time, datetime
import pathlib, collections, itertools, functools
import csv, sqlite3, urllib, http, socket
import unittest, logging, argparse, subprocess
import typing, dataclasses, enum, abc
import asyncio, threading, multiprocessing

Chapter 22 tours the most useful ones.

13.7 Example Project Layout

A typical structure for a small-to-medium project:

myproject/
├── pyproject.toml          # project config and dependency declarations
├── README.md
├── .gitignore
├── src/
│   └── myproject/
│       ├── __init__.py
│       ├── main.py         # entry point
│       ├── config.py       # configuration
│       ├── models.py       # data structures
│       ├── services/       # business logic
│       │   ├── __init__.py
│       │   └── user.py
│       └── utils/          # shared helpers
│           ├── __init__.py
│           └── text.py
└── tests/
    ├── test_models.py
    └── test_services.py

A small script doesn't need any of this — a single .py file is fine. Match the structure to the size of the project.

13.8 Exercises

13.1 Create two files: calculator.py defining add/subtract/multiply/divide, and main.py importing and using them. Add a self-test under if __name__ == "__main__" in calculator.py.

13.2 Explain the difference between these three imports and when each is appropriate:

python
import datetime
from datetime import datetime
from datetime import *

13.3 You wrote a file called json.py, and now import json inside it fails. Why?

<details> <summary>Answers</summary>

13.1

python
# calculator.py
def add(a, b):
    return a + b

def subtract(a, b):
    return a - b

def multiply(a, b):
    return a * b

def divide(a, b):
    if b == 0:
        raise ValueError("Cannot divide by zero")
    return a / b

if __name__ == "__main__":
    print("Self-test:")
    print(add(2, 3))        # 5
    print(divide(10, 2))    # 5.0
python
# main.py
from calculator import add, divide

print(add(1, 2))
print(divide(10, 4))

13.2

  • import datetime — imports the module; you write datetime.datetime.now(). Verbose, but least confusing.
  • from datetime import datetime — imports the datetime class from the module; you write datetime.now(). The most common form. Note the module and class share a name, which trips people up.
  • from datetime import * — dumps every name into your namespace, polluting it. Don't.

13.3 The module search path starts with the running script's directory, so import json finds your own json.py (importing itself) instead of the standard library. Rename the file. </details>


14. Exception Handling

Programs always hit the unexpected: a missing file, a dropped connection, a user typing nonsense. Exceptions let you respond gracefully instead of crashing.

14.1 What an Exception Looks Like

python
>>> 1 / 0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: division by zero

Reading a traceback:

Traceback (most recent call last):
  File "main.py", line 10, in <module>
    result = process(data)
  File "main.py", line 5, in process
    return int(data["value"])
KeyError: 'value'
  • Top to bottom is the call chain; the last line is where it actually failed
  • The final line, KeyError: 'value', is the exception type and message

When reading one: look at the last line first (what went wrong), then the last `File` entry that's your code (which line of yours triggered it).

14.2 try / except

python
try:
    result = 10 / 0
except ZeroDivisionError:
    print("Can't divide by zero")

Catching multiple types:

python
try:
    value = int(input("Number: "))
    result = 100 / value
except ValueError:
    print("Please enter a number")
except ZeroDivisionError:
    print("Can't be zero")
except (TypeError, KeyError):        # several at once
    print("Type or key error")

Getting the exception object:

python
try:
    int("abc")
except ValueError as e:
    print(f"Failed: {e}")            # invalid literal for int()...
    print(type(e).__name__)          # ValueError

14.3 else and finally

python
try:
    f = open("data.txt")
except FileNotFoundError:
    print("File doesn't exist")
else:
    print("Runs only if try succeeded")
    data = f.read()
finally:
    print("Runs no matter what")     # cleanup goes here

Order of execution:

  • No exception: tryelsefinally
  • Exception caught: try (to the failure) → exceptfinally
  • Exception uncaught: tryfinally → the exception propagates

finally is for releasing resources, though with (§14.7) is usually better.

💡 The value of else: it separates "code that might fail" from "code that runs on success," keeping the try block as small as possible.

14.4 Common Built-in Exceptions

ExceptionWhen it happens
ValueErrorRight type, invalid value — int("abc")
TypeErrorWrong type — "a" + 1
KeyErrorKey not in the dict
IndexErrorIndex out of range
AttributeErrorObject has no such attribute/method
FileNotFoundErrorFile doesn't exist
PermissionErrorNo permission
ZeroDivisionErrorDivision by zero
ImportError / ModuleNotFoundErrorImport failed
NameErrorUndefined variable (usually a typo)
StopIterationIterator exhausted
KeyboardInterruptUser pressed Ctrl-C

Part of the hierarchy:

BaseException
 ├── SystemExit
 ├── KeyboardInterrupt
 └── Exception          ← your own exceptions should subclass this
      ├── ValueError
      ├── TypeError
      ├── LookupError
      │    ├── KeyError
      │    └── IndexError
      ├── OSError
      │    ├── FileNotFoundError
      │    └── PermissionError
      └── ArithmeticError
           └── ZeroDivisionError

Catching a parent catches all its children:

python
except LookupError:      # catches both KeyError and IndexError
except OSError:          # catches all file/system errors

14.5 Never Catch Bare

python
# ❌ The worst possible form
try:
    do_something()
except:
    pass

Problems:

  1. A bare except: catches even KeyboardInterrupt, so the user can't Ctrl-C out
  2. pass swallows everything — when something breaks you have no idea
  3. Typos and logic bugs get hidden too

The right levels:

python
# ✅ Best: catch a specific exception
except ValueError:
    ...

# ✅ Acceptable: catch Exception and log it
except Exception as e:
    logger.exception("Processing failed")
    raise                  # log, then re-raise

# ⚠️ Only at the outermost layer (to prevent a crash)
except Exception:
    logger.exception("Unexpected error")

The principle: only catch exceptions you know how to handle. If you don't, let them propagate.

14.6 Raising Exceptions

python
def set_age(age):
    if not isinstance(age, int):
        raise TypeError(f"Age must be an integer, got {type(age).__name__}")
    if age < 0:
        raise ValueError(f"Age can't be negative: {age}")
    ...

💡 Exception messages should include the offending value. raise ValueError("Bad argument") is nearly useless; raise ValueError(f"Port must be 1-65535, got {port}") actually helps.

Re-raising:

python
try:
    risky()
except ValueError:
    logger.error("Noting this")
    raise                       # bare raise preserves the original traceback

Exception chaining (preserving the cause):

python
try:
    data = json.loads(text)
except json.JSONDecodeError as e:
    raise ConfigError("Config file is malformed") from e
    # The traceback will show "The above exception was the direct cause of..."

Custom exceptions:

python
class AppError(Exception):
    """Base class for all exceptions in this application."""

class ConfigError(AppError):
    """Configuration problem."""

class ValidationError(AppError):
    """Data validation failed."""
    def __init__(self, field, message):
        self.field = field
        super().__init__(f"{field}: {message}")


try:
    ...
except AppError as e:           # catch everything from this application at once
    ...

Defining an application-level base class is good practice — callers can choose to catch everything or just one category.

14.7 The with Statement and Context Managers

python
# ❌ Forgot to close, or an exception skipped the close
f = open("data.txt")
data = f.read()
f.close()

# ✅ with guarantees closing on exit, even if an exception occurs
with open("data.txt", encoding="utf-8") as f:
    data = f.read()

with applies to anything needing cleanup: files, database connections, network sockets, locks.

python
with open("a.txt") as fa, open("b.txt", "w") as fb:    # manage several at once
    fb.write(fa.read())

Writing your own — the easiest way is contextlib:

python
from contextlib import contextmanager
import time

@contextmanager
def timer(name):
    start = time.perf_counter()
    try:
        yield
    finally:
        elapsed = time.perf_counter() - start
        print(f"{name} took {elapsed:.3f}s")


with timer("Data processing"):
    do_heavy_work()
# Data processing took 1.234s

Everything before yield is "on entry," everything after is "on exit." Wrapping yield in try/finally is what guarantees cleanup even when an exception occurs.

You can also write it as a class (Chapter 18 makes this clearer):

python
class Timer:
    def __enter__(self):
        self.start = time.perf_counter()
        return self
    def __exit__(self, exc_type, exc_value, traceback):
        print(f"Took {time.perf_counter() - self.start:.3f}s")
        return False        # returning True swallows the exception; normally return False

14.8 EAFP vs LBYL

The Python community favors EAFP ("Easier to Ask Forgiveness than Permission" — try it and handle failure) over LBYL ("Look Before You Leap" — check first).

python
# LBYL — the common style in other languages
if os.path.exists(path):
    with open(path) as f:       # ⚠️ the file can be deleted between the check and the open
        ...

# EAFP — the Python style
try:
    with open(path) as f:
        ...
except FileNotFoundError:
    ...
python
# LBYL
if "key" in d:
    value = d["key"]
else:
    value = default

# EAFP
try:
    value = d["key"]
except KeyError:
    value = default

# Best: use what's already there
value = d.get("key", default)

EAFP avoids race conditions and skips a redundant check (exceptions cost almost nothing when they don't fire).

14.9 assert

python
def divide(a, b):
    assert b != 0, "Divisor can't be zero"
    return a / b

assert condition, message is equivalent to if not condition: raise AssertionError(message).

⚠️ Important: running with python -O removes every assert entirely. So:

  • ✅ Use assert for internal consistency checks during development ("this can never happen")
  • Never use assert for user input validation or permission checks
python
assert user.is_admin        # ❌ may be stripped in production = security hole
if not user.is_admin:       # ✅
    raise PermissionError()

14.10 Exercises

14.1 Write safe_int(s, default=0) that tries to convert a string to an int and returns the default on failure.

14.2 Rewrite this using with:

python
f = open("data.txt")
data = f.read()
f.close()

14.3 What's wrong with this?

python
try:
    result = compute()
except:
    pass

14.4 Write a function that reads a JSON config file and handles three failure modes: missing file, malformed JSON, and missing required keys — with a clear message for each.

14.5 Implement a context manager suppress_errors() that swallows exceptions in the with block and prints a warning.

14.6 Define an exception hierarchy: ShopError as the base, with OutOfStockError (carrying a product name) and PaymentError (carrying an amount).

<details> <summary>Answers</summary>

14.1

python
def safe_int(s, default=0):
    try:
        return int(s)
    except (ValueError, TypeError):
        return default

safe_int("42")     # 42
safe_int("abc")    # 0
safe_int(None)     # 0
safe_int("x", -1)  # -1

14.2

python
with open("data.txt", encoding="utf-8") as f:
    data = f.read()

14.3 Three problems:

  1. A bare except: catches KeyboardInterrupt and SystemExit, so Ctrl-C won't work
  2. pass silently swallows the error, making it undiagnosable
  3. Nothing documents which exception is expected or why ignoring it is safe

14.4

python
import json
from pathlib import Path

class ConfigError(Exception):
    pass

def load_config(path, required_keys=("host", "port")):
    p = Path(path)
    try:
        text = p.read_text(encoding="utf-8")
    except FileNotFoundError:
        raise ConfigError(f"Config file not found: {p.absolute()}") from None

    try:
        config = json.loads(text)
    except json.JSONDecodeError as e:
        raise ConfigError(f"Config isn't valid JSON (line {e.lineno}): {e.msg}") from e

    missing = [k for k in required_keys if k not in config]
    if missing:
        raise ConfigError(f"Config is missing required keys: {', '.join(missing)}")

    return config

14.5

python
from contextlib import contextmanager

@contextmanager
def suppress_errors():
    try:
        yield
    except Exception as e:
        print(f"⚠️ Ignored error: {type(e).__name__}: {e}")

with suppress_errors():
    1 / 0
print("Still running")

The standard library already has this: from contextlib import suppress; with suppress(ZeroDivisionError): ...

14.6

python
class ShopError(Exception):
    """Base class for shop-related exceptions."""

class OutOfStockError(ShopError):
    def __init__(self, product):
        self.product = product
        super().__init__(f"Out of stock: {product}")

class PaymentError(ShopError):
    def __init__(self, amount, reason="unknown reason"):
        self.amount = amount
        super().__init__(f"Payment of {amount} failed: {reason}")

</details>


15. Files and Paths

15.1 pathlib: Modern Path Handling

Older code uses os.path; new code should use pathlib — object-oriented, cross-platform, and far more readable.

python
from pathlib import Path

p = Path("data/report.txt")

# Join paths with the / operator
base = Path("/home/user")
full = base / "documents" / "file.txt"      # /home/user/documents/file.txt

# Common attributes
p.name        # 'report.txt'   filename
p.stem        # 'report'       without the extension
p.suffix      # '.txt'         the extension
p.parent      # Path('data')   parent directory
p.parts       # ('data', 'report.txt')
p.absolute()  # absolute path

# Tests
p.exists()      # does it exist
p.is_file()     # is it a file
p.is_dir()      # is it a directory

# Special locations
Path.cwd()      # current working directory
Path.home()     # user's home directory
Path(__file__).parent    # the directory of the current script (extremely useful)

Directory operations:

python
d = Path("output")
d.mkdir()                                # create; errors if it exists
d.mkdir(parents=True, exist_ok=True)     # create recursively, no error if present ← common

# Traversal
for f in d.iterdir():              # immediate children
    print(f)

for f in d.glob("*.txt"):          # match at this level
    print(f)

for f in d.rglob("*.py"):          # match recursively
    print(f)

# Files only
files = [f for f in d.rglob("*") if f.is_file()]

Renaming, deleting, copying:

python
p.rename("newname.txt")
p.unlink()                    # delete a file
p.unlink(missing_ok=True)     # no error if absent
d.rmdir()                     # delete an empty directory

import shutil
shutil.copy("a.txt", "b.txt")       # copy a file
shutil.copytree("src", "dst")       # copy a directory tree
shutil.rmtree("dir")                # delete a directory and its contents ⚠️ irreversible
shutil.move("a", "b")               # move

Shortcuts for reading and writing (one-liners for small files):

python
text = Path("a.txt").read_text(encoding="utf-8")
Path("b.txt").write_text("content", encoding="utf-8")

data = Path("img.png").read_bytes()
Path("copy.png").write_bytes(data)

15.2 open() and File Modes

python
with open("file.txt", mode="r", encoding="utf-8") as f:
    ...
ModeMeaning
"r"Read (default); error if the file doesn't exist
"w"Write, truncating existing content; creates if absent
"a"Append to the end
"x"Exclusive creation; error if the file exists
"b"Binary mode, combined with the above: "rb", "wb"
"+"Read and write: "r+"

⚠️ `"w"` erases the file. To add content, you must use "a".

⚠️ In text mode, always specify `encoding="utf-8"`. Without it Python uses the platform default, so the same code behaves differently on different machines — a classic source of cross-platform bugs.

15.3 Reading

python
with open("data.txt", encoding="utf-8") as f:
    content = f.read()          # everything at once, as one string

with open("data.txt", encoding="utf-8") as f:
    lines = f.readlines()       # a list, one element per line (newlines retained)

with open("data.txt", encoding="utf-8") as f:
    for line in f:              # ✅ line by line, memory-friendly, required for big files
        print(line.rstrip())    # rstrip removes the trailing newline

💡 For large files (hundreds of MB and up) you must iterate line by line — read() pulls the whole file into memory.

15.4 Writing

python
with open("out.txt", "w", encoding="utf-8") as f:
    f.write("first line\n")           # write does NOT add a newline
    f.write("second line\n")
    f.writelines(["a\n", "b\n"])      # nor does this

# print is more convenient (adds newlines automatically)
with open("out.txt", "w", encoding="utf-8") as f:
    print("first line", file=f)
    print("second line", file=f)

15.5 Common File Formats

JSON — the de facto standard for config and API data

python
import json

# Writing
data = {"name": "Alice", "tags": ["a", "b"], "age": 25}
with open("data.json", "w", encoding="utf-8") as f:
    json.dump(data, f, ensure_ascii=False, indent=2)
    # ensure_ascii=False → non-ASCII text stays readable instead of \uXXXX
    # indent=2 → pretty-printed

# Reading
with open("data.json", encoding="utf-8") as f:
    data = json.load(f)

# String versions
s = json.dumps(data, ensure_ascii=False)
data = json.loads(s)

⚠️ JSON supports a limited set of types (string, number, boolean, null, array, object). datetime, set, and custom objects need conversion:

python
json.dumps(obj, default=str)             # blunt: stringify anything unrecognized
json.dumps({"d": date.today().isoformat()})   # convert explicitly

CSV — tabular data

python
import csv

# Writing
rows = [{"name": "Alice", "age": 25}, {"name": "Bob", "age": 30}]
with open("out.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["name", "age"])
    writer.writeheader()
    writer.writerows(rows)

# Reading
with open("out.csv", newline="", encoding="utf-8") as f:
    for row in csv.DictReader(f):
        print(row["name"], row["age"])    # row is a dict

⚠️ Always pass newline="" when opening CSVs, or you'll get blank lines on Windows.

💡 Excel may garble UTF-8 CSVs; encoding="utf-8-sig" (with BOM) fixes it.

Other formats:

python
# Binary serialization of Python objects (only in trusted contexts! pickle can execute code)
import pickle
pickle.dump(obj, open("f.pkl", "wb"))

# TOML (reading; built in since Python 3.11)
import tomllib
with open("pyproject.toml", "rb") as f:      # note "rb"
    config = tomllib.load(f)

# YAML needs a third-party library
# pip install pyyaml
import yaml
config = yaml.safe_load(open("config.yaml", encoding="utf-8"))

15.6 Practical Recipes

Batch renaming

python
from pathlib import Path

for i, f in enumerate(sorted(Path("photos").glob("*.jpg")), start=1):
    f.rename(f.parent / f"photo_{i:03d}.jpg")

Directory size

python
total = sum(f.stat().st_size for f in Path(".").rglob("*") if f.is_file())
print(f"{total / 1024 / 1024:.2f} MB")

Find and process every Python file

python
for py in Path("src").rglob("*.py"):
    text = py.read_text(encoding="utf-8")
    if "TODO" in text:
        print(f"{py}: has todos")

Writing safely (write to a temp file first, then swap — so a crash mid-write can't corrupt the original)

python
from pathlib import Path

target = Path("important.json")
tmp = target.with_suffix(".json.tmp")
tmp.write_text(json.dumps(data), encoding="utf-8")
tmp.replace(target)      # atomic

15.7 Exercises

15.1 Write a program that counts the lines, words, and characters in a text file.

15.2 Read a file and write its lines in reverse order to another file.

15.3 Use pathlib to find every file over 1 MB in the current directory tree, sorted by size.

15.4 Export a list of contacts from a JSON file to CSV.

15.5 Why does this garble non-ASCII text on some machines, and how do you fix it?

python
with open("data.txt") as f:
    print(f.read())

15.6 Write a function that safely reads a JSON file, returning a default dict if the file is missing or malformed rather than crashing.

<details> <summary>Answers</summary>

15.1

python
from pathlib import Path

text = Path("data.txt").read_text(encoding="utf-8")
lines = text.splitlines()

print(f"Lines: {len(lines)}")
print(f"Words: {len(text.split())}")
print(f"Characters: {len(text)}")

15.2

python
from pathlib import Path

lines = Path("in.txt").read_text(encoding="utf-8").splitlines()
Path("out.txt").write_text("\n".join(reversed(lines)), encoding="utf-8")

15.3

python
from pathlib import Path

MB = 1024 * 1024
big = [f for f in Path(".").rglob("*") if f.is_file() and f.stat().st_size > MB]

for f in sorted(big, key=lambda p: p.stat().st_size, reverse=True):
    print(f"{f.stat().st_size / MB:8.2f} MB  {f}")

15.4

python
import csv, json
from pathlib import Path

contacts = json.loads(Path("contacts.json").read_text(encoding="utf-8"))

with open("contacts.csv", "w", newline="", encoding="utf-8-sig") as f:
    writer = csv.DictWriter(f, fieldnames=contacts[0].keys())
    writer.writeheader()
    writer.writerows(contacts)

15.5 No encoding was given, so Python uses the platform default (which on some Windows systems is a legacy codepage). Reading a UTF-8 file then produces mojibake or raises UnicodeDecodeError.

python
with open("data.txt", encoding="utf-8") as f:
    print(f.read())

15.6

python
import json
from pathlib import Path

def load_json(path, default=None):
    if default is None:
        default = {}
    try:
        return json.loads(Path(path).read_text(encoding="utf-8"))
    except (FileNotFoundError, json.JSONDecodeError, UnicodeDecodeError) as e:
        print(f"⚠️ Couldn't read {path} ({type(e).__name__}); using the default")
        return default

</details>


Part 5 · Object-Oriented Programming

At this point you can write useful programs. Object-oriented programming (OOP) won't let your programs do more, but it makes complex programs easier to understand and change.

16. Classes and Objects

16.1 Why Classes

Say you're managing a group of students. Dicts work:

python
student = {"name": "Alice", "scores": [85, 92, 78]}

def average(s):
    return sum(s["scores"]) / len(s["scores"])

But problems creep in:

  • Typo a key (s["score"]) and you find out at runtime
  • The data and the functions operating on it live in different places
  • Nothing guarantees every student actually has a scores field

A class bundles data with the methods that operate on it:

python
class Student:
    def __init__(self, name):
        self.name = name
        self.scores = []

    def add_score(self, score):
        self.scores.append(score)

    def average(self):
        if not self.scores:
            return 0
        return sum(self.scores) / len(self.scores)


s = Student("Alice")
s.add_score(85)
s.add_score(92)
print(s.average())      # 88.5

16.2 Anatomy of a Class

python
class Dog:
    species = "Canis familiaris"     # class attribute: shared by all instances

    def __init__(self, name, age):   # constructor
        self.name = name             # instance attributes: unique per instance
        self.age = age

    def bark(self):                  # instance method
        return f"{self.name} says woof"

    def birthday(self):
        self.age += 1
        return self.age


d = Dog("Rex", 3)       # creating an instance calls __init__ automatically
d.name                  # 'Rex'
d.bark()                # 'Rex says woof'
d.species               # 'Canis familiaris'
Dog.species             # accessible through the class too

Key ideas:

  • A class is a template; an instance (or object) is a concrete thing built from it
  • __init__ runs automatically at creation time to set up attributes
  • `self` refers to the instance itself. It's the first parameter of every instance method, and you don't pass it when calling — Python does
python
d.bark()          # what Python actually runs is Dog.bark(d)

⚠️ Forgetting self is the most common beginner mistake:

python
class Dog:
    def bark():                 # ❌ TypeError: bark() takes 0 positional arguments but 1 was given
        return "woof"

16.3 Class vs Instance Attributes

python
class Counter:
    total = 0                  # class attribute, shared

    def __init__(self):
        Counter.total += 1
        self.id = Counter.total   # instance attribute, independent

a = Counter()
b = Counter()
print(Counter.total)     # 2
print(a.id, b.id)        # 1 2

⚠️ Gotcha: mutable class attributes are shared

python
class Basket:
    items = []              # ❌ every instance shares one list!

    def add(self, x):
        self.items.append(x)

a, b = Basket(), Basket()
a.add("apple")
print(b.items)           # ['apple']  ← b changed too

The fix — put mutable data in __init__:

python
class Basket:
    def __init__(self):
        self.items = []      # ✅ a fresh list per instance

This is the same class of problem as the mutable default argument from Chapter 11.

16.4 Three Kinds of Methods

python
class Circle:
    PI = 3.14159

    def __init__(self, radius):
        self.radius = radius

    def area(self):                       # instance method: operates on an instance
        return self.PI * self.radius ** 2

    @classmethod
    def from_diameter(cls, d):            # class method: first arg is the class
        return cls(d / 2)                 # typically an alternative constructor

    @staticmethod
    def is_valid_radius(r):               # static method: unrelated to class or instance
        return r > 0                      # just logically grouped here


c1 = Circle(5)
c2 = Circle.from_diameter(10)             # same as Circle(5)
Circle.is_valid_radius(-1)                # False
KindFirst parameterWhen to use
Instance methodselfNeeds instance data (the vast majority)
Class methodclsAlternative constructors, working with class attributes
Static methodnoneRelated utility functions that need neither self nor cls

16.5 Encapsulation: Public, Protected, Private

Python has no real access control — it relies on naming conventions:

python
class Account:
    def __init__(self, balance):
        self.owner = "Alice"         # public: use freely
        self._balance = balance      # single underscore: internal, please don't touch
        self.__pin = "1234"          # double underscore: name mangling, a stronger "keep out"

    def deposit(self, amount):
        if amount <= 0:
            raise ValueError("Amount must be positive")
        self._balance += amount
  • _name — "this is an implementation detail; I may change it any time, don't depend on it." Purely a convention; Python doesn't stop you.
  • __name — triggers name mangling; it's stored as _ClassName__name. The point is avoiding accidental overrides in subclasses, not security.
python
a = Account(100)
a._balance           # accessible, but you shouldn't
a.__pin              # AttributeError
a._Account__pin      # '1234' — still reachable, so it isn't "secure"

💡 Python's philosophy here is "we're all consenting adults" — convention over enforcement.

16.6 property: Methods That Look Like Attributes

python
class Temperature:
    def __init__(self, celsius=0):
        self._celsius = celsius

    @property
    def celsius(self):
        return self._celsius

    @celsius.setter
    def celsius(self, value):
        if value < -273.15:
            raise ValueError("Below absolute zero")
        self._celsius = value

    @property
    def fahrenheit(self):            # read-only computed property
        return self._celsius * 9 / 5 + 32


t = Temperature(25)
t.celsius          # 25      accessed like an attribute, actually a method call
t.celsius = 30     # invokes the setter, which validates
t.fahrenheit       # 86.0    computed
t.fahrenheit = 100 # AttributeError: no setter
t.celsius = -300   # ValueError

The value of property: you can start with a plain attribute and later convert it into a property with validation or computation, and calling code doesn't change at all. That's why Python doesn't need getters and setters everywhere.

python
# ❌ Java style, unnecessary in Python
class Person:
    def get_name(self): return self._name
    def set_name(self, v): self._name = v

# ✅ Python style: just use a plain attribute
class Person:
    def __init__(self, name):
        self.name = name

16.7 Exercises

16.1 Define a Rectangle class with width and height, methods area() and perimeter(), and an is_square property.

16.2 What's wrong with this?

python
class Student:
    grades = []
    def add_grade(self, g):
        self.grades.append(g)

16.3 Define a BankAccount class where:

  • The balance can't be modified directly, only via deposit and withdraw
  • Withdrawing more than the balance raises
  • All transactions are recorded
  • A class method from_dict(data) builds an account from a dict

16.4 Define a Stack class supporting push, pop, peek, is_empty, and size.

16.5 Give Circle a radius property whose setter validates that the radius is positive.

<details> <summary>Answers</summary>

16.1

python
class Rectangle:
    def __init__(self, width, height):
        self.width = width
        self.height = height

    def area(self):
        return self.width * self.height

    def perimeter(self):
        return 2 * (self.width + self.height)

    @property
    def is_square(self):
        return self.width == self.height

16.2 grades is a class attribute, so all students share one list.

python
class Student:
    def __init__(self):
        self.grades = []
    def add_grade(self, g):
        self.grades.append(g)

16.3

python
class InsufficientFunds(Exception):
    pass

class BankAccount:
    def __init__(self, owner, balance=0):
        self.owner = owner
        self._balance = balance
        self._history = []

    @property
    def balance(self):
        return self._balance

    @property
    def history(self):
        return tuple(self._history)      # immutable copy, so callers can't tamper

    def deposit(self, amount):
        if amount <= 0:
            raise ValueError(f"Deposit must be positive, got {amount}")
        self._balance += amount
        self._history.append(("deposit", amount))

    def withdraw(self, amount):
        if amount <= 0:
            raise ValueError(f"Withdrawal must be positive, got {amount}")
        if amount > self._balance:
            raise InsufficientFunds(f"Balance {self._balance} can't cover {amount}")
        self._balance -= amount
        self._history.append(("withdraw", amount))

    @classmethod
    def from_dict(cls, data):
        return cls(data["owner"], data.get("balance", 0))

16.4

python
class Stack:
    def __init__(self):
        self._items = []

    def push(self, item):
        self._items.append(item)

    def pop(self):
        if self.is_empty():
            raise IndexError("pop from an empty stack")
        return self._items.pop()

    def peek(self):
        if self.is_empty():
            raise IndexError("peek at an empty stack")
        return self._items[-1]

    def is_empty(self):
        return len(self._items) == 0

    def size(self):
        return len(self._items)

    def __len__(self):              # so len(stack) works too
        return len(self._items)

    def __repr__(self):
        return f"Stack({self._items})"

16.5

python
class Circle:
    def __init__(self, radius):
        self.radius = radius        # this already triggers the setter's validation

    @property
    def radius(self):
        return self._radius

    @radius.setter
    def radius(self, value):
        if value <= 0:
            raise ValueError(f"Radius must be positive, got {value}")
        self._radius = value

</details>


17. Inheritance and Polymorphism

17.1 Inheritance: Reuse and Specialize

python
class Animal:
    def __init__(self, name):
        self.name = name

    def speak(self):
        return "..."

    def describe(self):
        return f"{self.name} says: {self.speak()}"


class Dog(Animal):              # Dog inherits from Animal
    def speak(self):            # override the parent's method
        return "Woof"


class Cat(Animal):
    def speak(self):
        return "Meow"


Dog("Rex").describe()       # 'Rex says: Woof'
Cat("Whiskers").describe()  # 'Whiskers says: Meow'

A subclass automatically gets all the parent's attributes and methods, and can:

  • Use them as-is (describe isn't overridden, it's inherited)
  • Override them (speak differs per subclass)
  • Extend (add methods the parent doesn't have)

17.2 super(): Calling the Parent

python
class Animal:
    def __init__(self, name, age):
        self.name = name
        self.age = age


class Dog(Animal):
    def __init__(self, name, age, breed):
        super().__init__(name, age)     # let the parent initialize first
        self.breed = breed              # then add your own

super() also works in regular methods, letting you "add to" the parent's behavior:

python
class LoggedList(list):
    def append(self, item):
        print(f"Adding: {item}")
        super().append(item)            # call list's original append

💡 When you override `__init__`, you almost always need to call `super().__init__()` — otherwise the parent's setup never runs.

17.3 Polymorphism: Same Call, Different Behavior

python
animals = [Dog("Rex"), Cat("Whiskers"), Animal("Generic")]

for a in animals:
    print(a.speak())        # each object responds according to its own type

The caller doesn't need to know the concrete class — only that it has a speak method. That's polymorphism, and it's where OOP earns its keep: adding a new animal requires changing no existing code.

17.4 Duck Typing

Python's polymorphism doesn't require an inheritance relationship. "If it walks like a duck and quacks like a duck, it's a duck":

python
class Duck:
    def speak(self): return "Quack"

class Robot:                       # completely unrelated to Animal
    def speak(self): return "Beep"

for x in [Dog("Rex"), Robot()]:
    print(x.speak())               # both work

This buys enormous flexibility. The cost is that errors only surface at runtime — which is why type hints (Chapter 21) earn their keep on large projects.

17.5 isinstance and issubclass

python
isinstance(d, Dog)          # True
isinstance(d, Animal)       # True   — a subclass instance is also a parent instance
isinstance(d, (Dog, Cat))   # True   — a tuple works

issubclass(Dog, Animal)     # True
type(d) is Dog              # True   — exact type, ignores inheritance

💡 Prefer isinstance over type(x) == Dog — the former respects inheritance.

⚠️ But a screenful of isinstance checks usually signals a design problem:

python
# ❌ Every new animal means editing this function
def make_sound(animal):
    if isinstance(animal, Dog):
        return "Woof"
    elif isinstance(animal, Cat):
        return "Meow"

# ✅ Polymorphism: new animals need no change here
def make_sound(animal):
    return animal.speak()

17.6 Abstract Base Classes

To require subclasses to implement certain methods:

python
from abc import ABC, abstractmethod

class Shape(ABC):
    @abstractmethod
    def area(self):
        ...

    @abstractmethod
    def perimeter(self):
        ...

    def describe(self):                  # concrete methods are allowed
        return f"Area {self.area()}, perimeter {self.perimeter()}"


class Rectangle(Shape):
    def __init__(self, w, h):
        self.w, self.h = w, h
    def area(self):
        return self.w * self.h
    def perimeter(self):
        return 2 * (self.w + self.h)


Shape()          # TypeError: Can't instantiate abstract class Shape
Rectangle(3, 4).describe()   # 'Area 12, perimeter 14'

The point of an ABC is making the contract explicit. It tells others "to subclass me, implement these methods," and forgetting one fails immediately at instantiation rather than at some later call.

17.7 Multiple Inheritance and the MRO

Python lets a class inherit from several parents:

python
class Serializable:
    def to_json(self):
        import json
        return json.dumps(self.__dict__)

class Comparable:
    def __lt__(self, other):
        return self.value < other.value

class Item(Serializable, Comparable):
    def __init__(self, value):
        self.value = value

The MRO (Method Resolution Order) determines lookup order:

python
Item.__mro__
# (Item, Serializable, Comparable, object)

Python computes it with C3 linearization; roughly "left to right, depth first, but a subclass always precedes its parents."

⚠️ Multiple inheritance easily produces tangled problems (diamond inheritance, initialization order). Practical guidance:

  • Prefer composition over inheritance
  • Reserve multiple inheritance for mixins (small classes providing one added capability, holding no state)
python
# ❌ Inheritance: is a Car an Engine? No.
class Car(Engine): ...

# ✅ Composition: a Car has an Engine
class Car:
    def __init__(self):
        self.engine = Engine()

17.8 When Inheritance Is the Right Tool

Signs it fits:

  • There's a genuine "is-a" relationship (Dog is an Animal)
  • A subclass can substitute for the parent anywhere (Liskov substitution)
  • There's substantial shared behavior

Signs it doesn't:

  • You just want to reuse a few lines (use a function or composition)
  • The hierarchy is more than three levels deep (nearly always a design problem)
  • Subclasses override most of the parent's methods (the abstraction is wrong)

A Python-specific note: don't subclass list, dict, and other built-ins to "add features" — many built-in methods won't route through your overrides. Use composition, or subclass collections.UserList / UserDict.

17.9 Exercises

17.1 Define an Employee base class (with name, base_salary, and calculate_pay()), plus Manager (pay = base × 1.5 + bonus) and Intern (pay = base × 0.6).

17.2 Use an abstract base class to define a Storage interface requiring save(key, value) and load(key). Write two implementations: MemoryStorage (a dict) and FileStorage (a JSON file).

17.3 Why does dog.name fail here?

python
class Animal:
    def __init__(self, name):
        self.name = name

class Dog(Animal):
    def __init__(self, breed):
        self.breed = breed

dog = Dog("Shiba")
print(dog.name)

17.4 Explain the difference between composition and inheritance, and give an example where composition is the right choice.

<details> <summary>Answers</summary>

17.1

python
class Employee:
    def __init__(self, name, base_salary):
        self.name = name
        self.base_salary = base_salary

    def calculate_pay(self):
        return self.base_salary

    def __repr__(self):
        return f"{type(self).__name__}({self.name}, {self.calculate_pay():.0f})"


class Manager(Employee):
    def __init__(self, name, base_salary, bonus=0):
        super().__init__(name, base_salary)
        self.bonus = bonus

    def calculate_pay(self):
        return self.base_salary * 1.5 + self.bonus


class Intern(Employee):
    def calculate_pay(self):
        return self.base_salary * 0.6


staff = [Manager("Alice", 20000, 5000), Intern("Bob", 10000), Employee("Carol", 15000)]
for e in staff:
    print(e)

17.2

python
import json
from abc import ABC, abstractmethod
from pathlib import Path

class Storage(ABC):
    @abstractmethod
    def save(self, key, value): ...

    @abstractmethod
    def load(self, key): ...


class MemoryStorage(Storage):
    def __init__(self):
        self._data = {}
    def save(self, key, value):
        self._data[key] = value
    def load(self, key):
        return self._data.get(key)


class FileStorage(Storage):
    def __init__(self, path):
        self.path = Path(path)
        if not self.path.exists():
            self.path.write_text("{}", encoding="utf-8")

    def _read(self):
        return json.loads(self.path.read_text(encoding="utf-8"))

    def save(self, key, value):
        data = self._read()
        data[key] = value
        self.path.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8")

    def load(self, key):
        return self._read().get(key)

17.3 Dog.__init__ overrides the parent's but never calls super().__init__(), so self.name is never set.

python
class Dog(Animal):
    def __init__(self, name, breed):
        super().__init__(name)
        self.breed = breed

17.4

  • Inheritance expresses "is-a": a Dog is an Animal
  • Composition expresses "has-a": a Car has an Engine

Composition is more flexible: you can swap the part at runtime, you're not bound to a hierarchy, and a change to the parent can't silently break you.

Example: logging shouldn't be added by making every class inherit from LoggerBase. Instead:

python
class Service:
    def __init__(self, logger):
        self.logger = logger        # composition — inject any logger you like

</details>


18. Special Methods and Data Classes

18.1 Special (Dunder) Methods

Methods with double underscores on both sides are special methods (dunder = double underscore). They let your classes plug into Python's built-in syntax.

python
class Vector:
    def __init__(self, x, y):
        self.x, self.y = x, y

    def __repr__(self):                      # the developer-facing representation
        return f"Vector({self.x}, {self.y})"

    def __str__(self):                       # the user-facing representation
        return f"({self.x}, {self.y})"

    def __eq__(self, other):                 # ==
        if not isinstance(other, Vector):
            return NotImplemented
        return (self.x, self.y) == (other.x, other.y)

    def __hash__(self):                      # define __hash__ whenever you define __eq__
        return hash((self.x, self.y))

    def __add__(self, other):                # +
        return Vector(self.x + other.x, self.y + other.y)

    def __mul__(self, scalar):               # *
        return Vector(self.x * scalar, self.y * scalar)

    def __abs__(self):                       # abs()
        return (self.x ** 2 + self.y ** 2) ** 0.5

    def __len__(self):                       # len()
        return 2

    def __getitem__(self, i):                # v[0]
        return (self.x, self.y)[i]

    def __bool__(self):                      # if v:
        return bool(self.x or self.y)


v1 = Vector(1, 2)
v2 = Vector(3, 4)

v1 + v2          # Vector(4, 6)
v1 * 3           # Vector(3, 6)
abs(v2)          # 5.0
v1 == Vector(1, 2)   # True
len(v1)          # 2
v1[0]            # 1
print(v1)        # (1, 2)       — uses __str__
v1               # Vector(1, 2) — the REPL uses __repr__

18.2 Special Method Reference

Representation

python
__repr__(self)      # repr(x) — for debugging; ideally reconstructs the object
__str__(self)       # str(x), print(x) — for humans
__format__(self, spec)   # f"{x:spec}"

💡 If you only write one, write __repr____str__ falls back to it when undefined. A good __repr__ looks like code that would rebuild the object.

Comparison

python
__eq__  __ne__  __lt__  __le__  __gt__  __ge__

💡 With the functools.total_ordering decorator you only implement __eq__ and __lt__; the rest are generated.

Arithmetic

python
__add__  __sub__  __mul__  __truediv__  __floordiv__  __mod__  __pow__
__radd__ ...     # reflected operations (3 * vector calls vector.__rmul__)
__iadd__ ...     # in-place operations (+=)
__neg__  __abs__

Containers

python
__len__(self)               # len(x)
__getitem__(self, key)      # x[key]
__setitem__(self, key, v)   # x[key] = v
__delitem__(self, key)      # del x[key]
__contains__(self, item)    # item in x
__iter__(self)              # for i in x
__next__(self)              # the iterator protocol

Others

python
__call__(self, ...)         # make instances callable like functions: x()
__enter__ / __exit__        # the with statement
__hash__(self)              # usable as a dict key / set member
__bool__(self)              # if x:
__getattr__(self, name)     # fallback for missing attributes
__slots__ = ("x", "y")      # restrict attributes, save memory

18.3 dataclass: Less Boilerplate

A plain data class repeats a lot:

python
class Point:
    def __init__(self, x, y):
        self.x = x
        self.y = y
    def __repr__(self):
        return f"Point(x={self.x}, y={self.y})"
    def __eq__(self, other):
        return (self.x, self.y) == (other.x, other.y)

dataclass does it in one line:

python
from dataclasses import dataclass

@dataclass
class Point:
    x: float
    y: float

p = Point(1, 2)
p                    # Point(x=1, y=2)      __repr__ generated
p == Point(1, 2)     # True                 __eq__ generated

Common options:

python
from dataclasses import dataclass, field

@dataclass(frozen=True)      # immutable, and __hash__ is generated
class Config:
    host: str = "localhost"          # with defaults
    port: int = 8080
    tags: list[str] = field(default_factory=list)   # ⚠️ mutable defaults need field
    _secret: str = field(default="", repr=False)     # excluded from repr


@dataclass(order=True)       # generates < > <= >=, comparing fields in order
class Version:
    major: int
    minor: int

⚠️ Mutable defaults must use field(default_factory=list); writing tags: list = [] raises an error (dataclass shields you from the Chapter 11 trap).

Post-initialization:

python
@dataclass
class Rectangle:
    width: float
    height: float
    area: float = field(init=False)      # not a constructor parameter

    def __post_init__(self):
        self.area = self.width * self.height

Conversion helpers:

python
from dataclasses import asdict, astuple, replace

asdict(p)                  # {'x': 1, 'y': 2}
astuple(p)                 # (1, 2)
replace(p, x=10)           # Point(x=10, y=2)  returns a new instance

💡 When to use a dataclass: any class whose main job is holding data. It's safer than a dict (a misspelled field name errors immediately) and less work than a hand-written class. Make it your default.

18.4 Comparing Data Container Options

OptionMutableValidates typesBest for
dictLoose structure, data straight from JSON
namedtupleLightweight immutable records
dataclassOptional❌ (hints only)The default
NamedTuple (typing)Need tuple behavior plus type hints
pydantic.BaseModel✅ at runtimeHandling external input (APIs, config)
python
# typing.NamedTuple — an annotated immutable record
from typing import NamedTuple
class Point(NamedTuple):
    x: float
    y: float

# pydantic (third party: pip install pydantic) — actually validates and coerces
from pydantic import BaseModel
class User(BaseModel):
    name: str
    age: int
User(name="Alice", age="25")     # age is coerced to int 25
User(name="Alice", age="abc")    # ValidationError

For data arriving from outside (HTTP requests, config files, user input), use pydantic; for internal structures, use dataclass.

18.5 Enum

Named constants instead of magic strings:

python
from enum import Enum, auto

class Status(Enum):
    PENDING = "pending"
    ACTIVE = "active"
    CLOSED = "closed"

Status.ACTIVE            # <Status.ACTIVE: 'active'>
Status.ACTIVE.value      # 'active'
Status.ACTIVE.name       # 'ACTIVE'
Status("active")         # <Status.ACTIVE: 'active'>  lookup by value

for s in Status:         # iterable
    print(s)
python
class Color(Enum):
    RED = auto()         # auto-assigns 1, 2, 3
    GREEN = auto()
    BLUE = auto()

from enum import StrEnum       # Python 3.11+, members usable directly as strings
class Env(StrEnum):
    DEV = "dev"
    PROD = "prod"

Env.DEV == "dev"         # True

Benefits: typos fail immediately, your IDE autocompletes, and you can enumerate every valid value.

python
# ❌ Magic string
if order.status == "actve":     # typo, silently never matches

# ✅ Enum
if order.status is Status.ACTIVE:   # a typo raises AttributeError

18.6 Exercises

18.1 Add __sub__ (subtraction) and __rmul__ (so 3 * v works) to Vector.

18.2 Use a dataclass for a Book: title, author, year, tags (list, empty by default), and isbn (excluded from repr).

18.3 Define a Money class as a frozen dataclass supporting addition and comparison, storing the amount as a Decimal and raising when currencies differ.

18.4 Define a Playlist class supporting len(), playlist[0], for song in playlist, and "title" in playlist.

18.5 Refactor this with an Enum:

python
def get_discount(user_type):
    if user_type == "vip":
        return 0.8
    elif user_type == "svip":
        return 0.6
    return 1.0

<details> <summary>Answers</summary>

18.1

python
class Vector:
    def __init__(self, x, y):
        self.x, self.y = x, y

    def __repr__(self):
        return f"Vector({self.x}, {self.y})"

    def __sub__(self, other):
        return Vector(self.x - other.x, self.y - other.y)

    def __mul__(self, k):
        return Vector(self.x * k, self.y * k)

    __rmul__ = __mul__          # scalar multiplication commutes, so reuse it

18.2

python
from dataclasses import dataclass, field

@dataclass
class Book:
    title: str
    author: str
    year: int
    tags: list[str] = field(default_factory=list)
    isbn: str = field(default="", repr=False)

18.3

python
from dataclasses import dataclass
from decimal import Decimal

@dataclass(frozen=True, order=True)
class Money:
    amount: Decimal
    currency: str = "USD"

    def __add__(self, other):
        if self.currency != other.currency:
            raise ValueError(f"Currency mismatch: {self.currency} vs {other.currency}")
        return Money(self.amount + other.amount, self.currency)

    def __str__(self):
        return f"{self.amount:.2f} {self.currency}"


a = Money(Decimal("10.50"))
b = Money(Decimal("5.25"))
print(a + b)      # 15.75 USD

18.4

python
class Playlist:
    def __init__(self, songs=None):
        self._songs = list(songs or [])

    def add(self, song):
        self._songs.append(song)

    def __len__(self):
        return len(self._songs)

    def __getitem__(self, i):
        return self._songs[i]

    def __iter__(self):
        return iter(self._songs)

    def __contains__(self, song):
        return song in self._songs

    def __repr__(self):
        return f"Playlist({len(self)} songs)"

Note: implementing __getitem__ alone is enough for for and in to work, but implementing __iter__ and __contains__ explicitly is both faster and clearer.

18.5

python
from enum import Enum

class UserType(Enum):
    NORMAL = "normal"
    VIP = "vip"
    SVIP = "svip"

DISCOUNTS = {
    UserType.NORMAL: 1.0,
    UserType.VIP: 0.8,
    UserType.SVIP: 0.6,
}

def get_discount(user_type: UserType) -> float:
    return DISCOUNTS[user_type]

</details>


Part 6 · Advanced Features

19. Iterators and Generators

19.1 Iterables and Iterators

An iterable is anything you can loop over with for — lists, strings, dicts, files, ranges.

An iterator is the object that actually performs "give me the next one."

python
lst = [1, 2, 3]
it = iter(lst)         # get an iterator from an iterable

next(it)               # 1
next(it)               # 2
next(it)               # 3
next(it)               # raises StopIteration

What a for loop does under the hood:

python
# for x in lst: print(x)
# is equivalent to:
it = iter(lst)
while True:
    try:
        x = next(it)
    except StopIteration:
        break
    print(x)

Iterators are single-use:

python
it = iter([1, 2, 3])
list(it)     # [1, 2, 3]
list(it)     # []  ← already exhausted

⚠️ This bites people:

python
z = zip([1,2], "ab")
list(z)      # [(1, 'a'), (2, 'b')]
list(z)      # []  ← zip returns an iterator!

map, filter, zip, enumerate, and reversed all return iterators in Python 3. If you need to use one more than once, wrap it in list() first.

19.2 Writing an Iterator by Hand

python
class Countdown:
    def __init__(self, start):
        self.start = start

    def __iter__(self):
        return CountdownIterator(self.start)


class CountdownIterator:
    def __init__(self, current):
        self.current = current

    def __iter__(self):
        return self

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        self.current -= 1
        return self.current + 1


for n in Countdown(3):
    print(n)      # 3 2 1

That's a lot of ceremony. Generators compress it to three lines.

19.3 Generator Functions

A function containing yield becomes a generator function:

python
def countdown(n):
    while n > 0:
        yield n
        n -= 1

for x in countdown(3):
    print(x)      # 3 2 1

How it runs:

  1. Calling countdown(3) executes no code; it returns a generator object
  2. Each next() resumes the function from where it last yielded
  3. It pauses at the next yield, handing out the value
  4. When the function ends, StopIteration is raised automatically
python
def demo():
    print("start")
    yield 1
    print("middle")
    yield 2
    print("end")

g = demo()          # prints nothing
next(g)             # prints "start", returns 1
next(g)             # prints "middle", returns 2
next(g)             # prints "end", raises StopIteration

19.4 Why Generators Matter: Laziness

Memory:

python
# ❌ Builds ten million numbers at once, hundreds of MB
def squares_list(n):
    return [i ** 2 for i in range(n)]

# ✅ One at a time, constant memory
def squares_gen(n):
    for i in range(n):
        yield i ** 2

sum(squares_gen(10_000_000))    # essentially zero memory

Infinite sequences:

python
def naturals():
    n = 0
    while True:
        yield n
        n += 1

from itertools import islice
list(islice(naturals(), 5))     # [0, 1, 2, 3, 4]

Streaming large files:

python
def read_large_file(path):
    with open(path, encoding="utf-8") as f:
        for line in f:
            yield line.rstrip()

def filter_errors(lines):
    for line in lines:
        if "ERROR" in line:
            yield line

def parse(lines):
    for line in lines:
        yield line.split("|")

# A pipeline that never holds more than one line in memory
for parts in parse(filter_errors(read_large_file("huge.log"))):
    print(parts)

This is generators at their most elegant — split the logic into small generators and compose them like Unix pipes.

19.5 yield from

Delegate iteration to another iterable:

python
def chain(*iterables):
    for it in iterables:
        yield from it              # same as: for x in it: yield x

list(chain([1, 2], "ab"))          # [1, 2, 'a', 'b']

Especially useful for recursion:

python
def flatten(nested):
    for item in nested:
        if isinstance(item, list):
            yield from flatten(item)
        else:
            yield item

list(flatten([1, [2, [3, [4]]], 5]))    # [1, 2, 3, 4, 5]

19.6 itertools: The Iterator Toolbox

A set of efficient, fully lazy iterator utilities in the standard library:

python
from itertools import (count, cycle, repeat, chain, islice,
                       groupby, product, permutations, combinations,
                       accumulate, takewhile, dropwhile, tee, zip_longest)

# Infinite iterators
count(10, 2)                 # 10, 12, 14, ...
cycle("ABC")                 # A B C A B C ...
repeat("x", 3)               # x x x

# Combining
chain([1,2], [3,4])          # 1 2 3 4
islice(range(100), 5, 10)    # 5 6 7 8 9   (slicing for arbitrary iterators)
zip_longest([1,2], "abc", fillvalue=None)   # (1,'a') (2,'b') (None,'c')

# Combinatorics
list(product("AB", repeat=2))       # AA AB BA BB
list(permutations("ABC", 2))        # AB AC BA BC CA CB
list(combinations("ABC", 2))        # AB AC BC

# Accumulation
list(accumulate([1, 2, 3, 4]))            # [1, 3, 6, 10]  running sum
list(accumulate([1, 2, 3], func=max))     # running maximum

# Conditional slicing
list(takewhile(lambda x: x < 3, [1,2,3,1]))   # [1, 2]  stops at the first false
list(dropwhile(lambda x: x < 3, [1,2,3,1]))   # [3, 1]  skips the leading run

# Grouping (⚠️ you must sort by the same key first!)
data = sorted(people, key=lambda p: p["city"])
for city, group in groupby(data, key=lambda p: p["city"]):
    print(city, list(group))

⚠️ groupby only groups adjacent equal elements, so you must sort first or the result won't be what you expect. For true grouping, use defaultdict.

19.7 Advanced Generator Use

send(): pushing values into a generator

python
def accumulator():
    total = 0
    while True:
        value = yield total
        if value is None:
            break
        total += value

acc = accumulator()
next(acc)          # prime it, returns 0
acc.send(10)       # 10
acc.send(5)        # 15

Rarely needed — just be aware it exists. It's related to the machinery underlying async programming (Chapter 24).

19.8 Exercises

19.1 Write a generator fibonacci() that yields the Fibonacci sequence indefinitely. Use islice to take the first 10.

19.2 Write a generator chunks(iterable, size) that splits a sequence into fixed-size chunks: chunks([1,2,3,4,5], 2)[1,2] [3,4] [5].

19.3 Why is the second print empty?

python
data = zip([1,2,3], "abc")
print(list(data))
print(list(data))

19.4 Use generators to read a log file and return only lines containing "ERROR", capped at the first 100.

19.5 Compare the memory usage of these two lines and explain the difference:

python
sum([x**2 for x in range(10**7)])
sum(x**2 for x in range(10**7))

<details> <summary>Answers</summary>

19.1

python
from itertools import islice

def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

list(islice(fibonacci(), 10))    # [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

19.2

python
from itertools import islice

def chunks(iterable, size):
    it = iter(iterable)
    while chunk := list(islice(it, size)):
        yield chunk

list(chunks([1,2,3,4,5], 2))     # [[1, 2], [3, 4], [5]]

Python 3.12+ has this built in as itertools.batched:

python
from itertools import batched
list(batched([1,2,3,4,5], 2))    # [(1, 2), (3, 4), (5,)]

19.3 zip returns an iterator, which is single-use. The first list() consumed it. If you need it more than once, materialize it:

python
data = list(zip([1,2,3], "abc"))

19.4

python
from itertools import islice

def error_lines(path):
    with open(path, encoding="utf-8") as f:
        for line in f:
            if "ERROR" in line:
                yield line.rstrip()

for line in islice(error_lines("app.log"), 100):
    print(line)

The key point: the file is never fully loaded into memory, and reading stops once 100 matches are found.

19.5

  • The first builds a list of ten million integers (~400 MB), then sums it
  • The second uses a generator, holding one number at a time (tens of bytes)

Same result, seven orders of magnitude difference in memory. Since parentheses can be omitted when the generator expression is a function's sole argument, the second form costs nothing extra to write — make it your default habit. </details>


20. Decorators

20.1 What a Decorator Is

A decorator is a function that takes a function and returns a new one. Its purpose is adding behavior without modifying the original function's code.

python
def my_decorator(func):
    def wrapper():
        print("before")
        func()
        print("after")
    return wrapper


@my_decorator
def say_hello():
    print("Hello")

say_hello()
# before
# Hello
# after

@my_decorator is just syntactic sugar for:

python
say_hello = my_decorator(say_hello)

20.2 Handling Arguments and Return Values

The wrapper above takes no arguments, so decorating a function with parameters would fail. The general form:

python
import functools

def my_decorator(func):
    @functools.wraps(func)                    # preserve the original's metadata
    def wrapper(*args, **kwargs):             # accept anything
        print(f"calling {func.__name__}")
        result = func(*args, **kwargs)        # forward it all
        print(f"returned {result}")
        return result                         # don't forget this!
    return wrapper


@my_decorator
def add(a, b):
    return a + b

add(1, 2)

⚠️ Don't omit `@functools.wraps(func)`. Without it, the decorated function's __name__, __doc__, and type hints all become the wrapper's, breaking debugging and documentation tools:

python
add.__name__     # with wraps → 'add'; without → 'wrapper'

⚠️ Don't forget `return result`. If you do, every decorated function silently returns None — a nasty bug to track down.

20.3 Practical Decorators

Timing

python
import time, functools

def timer(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        try:
            return func(*args, **kwargs)
        finally:
            print(f"{func.__name__} took {time.perf_counter() - start:.4f}s")
    return wrapper

Retrying

python
def retry(times=3, delay=1, exceptions=(Exception,)):
    def decorator(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            last = None
            for attempt in range(1, times + 1):
                try:
                    return func(*args, **kwargs)
                except exceptions as e:
                    last = e
                    print(f"Attempt {attempt}/{times} failed: {e}")
                    if attempt < times:
                        time.sleep(delay)
            raise last
        return wrapper
    return decorator


@retry(times=3, delay=2, exceptions=(ConnectionError,))
def fetch(url):
    ...

Note that a decorator with parameters needs three levels: the outermost takes the decorator's arguments, the middle takes the function, and the innermost is the actual wrapper.

Logging

python
import logging

def logged(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        logging.info("calling %s(%r, %r)", func.__name__, args, kwargs)
        try:
            result = func(*args, **kwargs)
            logging.info("%s returned %r", func.__name__, result)
            return result
        except Exception:
            logging.exception("%s raised", func.__name__)
            raise
    return wrapper

20.4 Decorators in the Standard Library

`functools.cache` / `lru_cache` — memoization

python
from functools import cache, lru_cache

@cache                        # Python 3.9+, unbounded cache
def fib(n):
    return n if n < 2 else fib(n-1) + fib(n-2)

fib(100)          # instant; without caching it would run forever

@lru_cache(maxsize=128)       # bounded; evicts the least recently used
def expensive_query(user_id):
    ...

expensive_query.cache_info()      # hit rate
expensive_query.cache_clear()     # reset

⚠️ Arguments must be hashable (no lists or dicts). The cached function must be pure (same input always produces the same output), or you'll serve stale data.

`functools.cached_property` — a cached attribute

python
from functools import cached_property

class Dataset:
    @cached_property
    def stats(self):
        print("computing...")        # prints only once
        return heavy_computation(self.data)

d = Dataset()
d.stats     # computing... then returns
d.stats     # returns the cache

`@property` / `@staticmethod` / `@classmethod` — from Chapter 16; they're decorators too.

`@dataclass` — from Chapter 18.

`functools.singledispatch` — dispatch on type

python
from functools import singledispatch

@singledispatch
def describe(obj):
    return f"Unknown type: {obj}"

@describe.register
def _(obj: int):
    return f"Integer {obj}"

@describe.register
def _(obj: list):
    return f"List with {len(obj)} items"

describe(42)        # 'Integer 42'
describe([1,2])     # 'List with 2 items'

20.5 Class-Based Decorators and Decorating Classes

Implementing a decorator as a class (clearer when you need state):

python
class CountCalls:
    def __init__(self, func):
        functools.update_wrapper(self, func)
        self.func = func
        self.count = 0

    def __call__(self, *args, **kwargs):
        self.count += 1
        return self.func(*args, **kwargs)


@CountCalls
def hello():
    print("hi")

hello(); hello()
hello.count        # 2

Decorating a class (decorators can target classes too):

python
def add_repr(cls):
    def __repr__(self):
        attrs = ", ".join(f"{k}={v!r}" for k, v in vars(self).items())
        return f"{cls.__name__}({attrs})"
    cls.__repr__ = __repr__
    return cls


@add_repr
class Point:
    def __init__(self, x, y):
        self.x, self.y = x, y

Point(1, 2)        # Point(x=1, y=2)

20.6 Stacking Decorators

python
@a
@b
@c
def f(): ...

# equivalent to f = a(b(c(f)))

Applied bottom-up, executed top-down. Order matters:

python
@app.route("/users")       # routing must be outermost
@login_required            # check auth first
@timer                     # then time it
def get_users(): ...

20.7 Exercises

20.1 Write a @debug decorator that prints the function name, arguments, and return value.

20.2 Write a @validate_positive decorator that checks all positional arguments are positive numbers, raising ValueError otherwise.

20.3 Find the two bugs in this decorator:

python
def logger(func):
    def wrapper(*args):
        print(f"calling {func.__name__}")
        func(*args)
    return wrapper

20.4 Write a parameterized decorator @rate_limit(calls_per_second=2) that throttles how often a function can be called.

20.5 Optimize this with @cache and explain why it gets faster:

python
def count_paths(m, n):
    if m == 1 or n == 1:
        return 1
    return count_paths(m-1, n) + count_paths(m, n-1)

<details> <summary>Answers</summary>

20.1

python
import functools

def debug(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        sig = ", ".join([*map(repr, args), *(f"{k}={v!r}" for k, v in kwargs.items())])
        print(f"→ {func.__name__}({sig})")
        result = func(*args, **kwargs)
        print(f"← {func.__name__} returned {result!r}")
        return result
    return wrapper

20.2

python
def validate_positive(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        for i, a in enumerate(args):
            if isinstance(a, (int, float)) and a <= 0:
                raise ValueError(f"{func.__name__} argument {i+1} must be positive, got {a}")
        return func(*args, **kwargs)
    return wrapper

20.3 Two bugs:

  1. No return func(*args) — every decorated function returns None
  2. No @functools.wraps(func) — the original's metadata is lost

Also wrapper(*args) rejects keyword arguments, which is a third problem. Fixed:

python
def logger(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        print(f"calling {func.__name__}")
        return func(*args, **kwargs)
    return wrapper

20.4

python
import time, functools

def rate_limit(calls_per_second=2):
    min_interval = 1.0 / calls_per_second
    def decorator(func):
        last_called = 0.0
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            nonlocal last_called
            elapsed = time.perf_counter() - last_called
            if elapsed < min_interval:
                time.sleep(min_interval - elapsed)
            last_called = time.perf_counter()
            return func(*args, **kwargs)
        return wrapper
    return decorator

20.5

python
from functools import cache

@cache
def count_paths(m, n):
    if m == 1 or n == 1:
        return 1
    return count_paths(m-1, n) + count_paths(m, n-1)

count_paths(18, 18)    # instant

Why: the naive recursion recomputes the same (m, n) pair an exponential number of times (count_paths(3,3) evaluates count_paths(2,2) repeatedly). With caching each pair is computed once, dropping complexity from O(2^(m+n)) to O(m×n). </details>


21. Type Hints

Python is dynamically typed, and type hints don't affect execution — they're for humans and tools. But on any project past a few hundred lines they pay for themselves.

21.1 Basic Syntax

python
name: str = "Alice"
age: int = 25
scores: list[int] = [90, 85]

def greet(name: str, times: int = 1) -> str:
    return f"Hello, {name}! " * times

⚠️ Annotations are not enforced:

python
def add(a: int, b: int) -> int:
    return a + b

add("x", "y")     # 'xy' — runs fine; Python doesn't care

Actual checking requires a type checker (§21.6).

21.2 Common Types

python
from typing import Any, Optional, Union, Callable, Iterator, TypeVar

# Basic types are written directly
x: int
y: float
s: str
b: bool

# Containers (Python 3.9+ uses the built-in types; no more from typing import List)
nums: list[int]
pairs: dict[str, int]
point: tuple[float, float]           # fixed length, different types per position
row: tuple[int, ...]                 # arbitrary length, uniform type
tags: set[str]

# Possibly None
name: str | None = None              # Python 3.10+, preferred
name: Optional[str] = None           # older form, equivalent

# One of several types
value: int | str
value: Union[int, str]               # older form

# Function types
handler: Callable[[int, str], bool]  # takes int and str, returns bool
callback: Callable[..., None]        # any arguments, returns None

# Anything (effectively turns checking off — use sparingly)
data: Any

21.3 Slightly More Advanced

Generic functions

python
from typing import TypeVar

T = TypeVar("T")

def first(items: list[T]) -> T | None:
    return items[0] if items else None

first([1, 2, 3])       # the checker knows this returns int | None
first(["a", "b"])      # and this returns str | None

Python 3.12+ has cleaner syntax:

python
def first[T](items: list[T]) -> T | None:
    return items[0] if items else None

Annotations in classes

python
from dataclasses import dataclass

@dataclass
class User:
    name: str
    age: int
    email: str | None = None

    def is_adult(self) -> bool:
        return self.age >= 18

Protocols (structural typing) — describe "what methods it has" rather than "what it inherits from," which is the type-system counterpart to duck typing:

python
from typing import Protocol

class Drawable(Protocol):
    def draw(self) -> None: ...

def render(item: Drawable) -> None:      # any object with a draw method qualifies
    item.draw()

Type aliases

python
type UserId = int                        # Python 3.12+
type JsonValue = dict[str, "JsonValue"] | list["JsonValue"] | str | int | float | bool | None

UserId = int                             # older form

Special return types

python
from typing import NoReturn

def fail(msg: str) -> NoReturn:          # never returns normally
    raise RuntimeError(msg)

def gen() -> Iterator[int]:              # a generator
    yield 1

21.4 What Annotations Buy You

1. IDE autocompletion

python
def process(user):          # the IDE has no idea what user has
    user.          # ← no suggestions

def process(user: User):    # now it knows
    user.          # ← name, age, is_adult...

2. Documentation

python
# Can you tell what this does without reading the body?
def calc(a, b, c): ...

# Obvious
def calculate_discount(price: Decimal, rate: float, is_vip: bool) -> Decimal: ...

3. Catching bugs early

A checker will find unhandled None, swapped arguments, and wrong return types as you write.

21.5 When to Annotate

Always:

  • Public function and method signatures
  • Complex data structures
  • Anything in a team project or long-lived codebase

Optional:

  • Throwaway scripts
  • Obvious local variables (count = 0 doesn't need count: int = 0)
  • Small private helpers

💡 Adopt gradually. You don't have to annotate a whole codebase at once. Start with new code and the core modules, then expand.

21.6 Type Checkers

mypy — the most mature option

bash
pip install mypy
mypy your_file.py

pyright / basedpyright — from Microsoft, fast; VS Code's Pylance is built on it

ty / pyrefly — a new generation released in 2025, written in Rust, one to two orders of magnitude faster than mypy. As of mid-2026 they're still in beta: inference coverage and error-message quality lag behind mypy and pyright. Worth trying, but don't make one your project's only gate yet.

Configuration (in pyproject.toml):

toml
[tool.mypy]
python_version = "3.12"
strict = true                    # recommended for new projects
warn_return_any = true
warn_unused_ignores = true

[[tool.mypy.overrides]]
module = "some_untyped_lib.*"
ignore_missing_imports = true

Silencing one line:

python
result = weird_call()  # type: ignore[attr-defined]

21.7 When You Want Runtime Validation

Type hints don't check anything at runtime. For that, use pydantic:

python
from pydantic import BaseModel, Field, field_validator

class User(BaseModel):
    name: str = Field(min_length=1, max_length=50)
    age: int = Field(ge=0, le=150)
    email: str

    @field_validator("email")
    @classmethod
    def check_email(cls, v: str) -> str:
        if "@" not in v:
            raise ValueError("Invalid email format")
        return v


User(name="Alice", age=25, email="[email protected]")     # ✅
User(name="", age=200, email="bad")             # ValidationError listing every problem

The division of labor:

  • Internal code: dataclass + type hints + mypy (zero runtime cost)
  • External input (HTTP, config files, CLI args): pydantic (real validation)

21.8 Exercises

21.1 Add complete type hints to this function:

python
def find_users(names, min_age=0, active_only=True):
    return [u for u in database if u.name in names]

21.2 Write annotations for:

  • A dict mapping strings to lists of integers
  • A value that may be a string or None
  • A function taking a string and returning a bool
  • A list of (name, age, is_active) triples

21.3 What's wrong with this annotation?

python
def get_config() -> dict:
    return {"host": "localhost", "port": 8080}

21.4 Use a Protocol to define a "serializable" interface requiring a to_dict() method.

<details> <summary>Answers</summary>

21.1

python
from collections.abc import Sequence

def find_users(
    names: Sequence[str],
    min_age: int = 0,
    active_only: bool = True,
) -> list[User]:
    return [u for u in database if u.name in names]

Using Sequence[str] instead of list[str] is more permissive — callers can pass a tuple. That's the "accept broadly, return specifically" principle.

21.2

python
d: dict[str, list[int]]
s: str | None
f: Callable[[str], bool]
records: list[tuple[str, int, bool]]

21.3 Bare dict is too vague — it says nothing about key or value types.

python
def get_config() -> dict[str, str | int]: ...

# Better: make the structure explicit with TypedDict or a dataclass
from typing import TypedDict

class Config(TypedDict):
    host: str
    port: int

def get_config() -> Config: ...

21.4

python
from typing import Protocol, Any

class Serializable(Protocol):
    def to_dict(self) -> dict[str, Any]: ...


def save_all(items: list[Serializable]) -> None:
    for item in items:
        write(item.to_dict())

Any class with a to_dict() method satisfies this protocol — no explicit inheritance needed. </details>


22. Standard Library Tour

Python ships with a remarkably broad standard library. This chapter surveys the most useful parts — don't memorize it, just know these exist and look up the docs when you need them.

22.1 Dates and Times

python
from datetime import datetime, date, time, timedelta, timezone

now = datetime.now()                    # local time
utc = datetime.now(timezone.utc)        # UTC (use this for storage)
today = date.today()

# Constructing
d = date(2026, 7, 26)
dt = datetime(2026, 7, 26, 14, 30, 0)

# Formatting
now.strftime("%Y-%m-%d %H:%M:%S")       # '2026-07-26 14:30:00'
now.isoformat()                         # '2026-07-26T14:30:00.123456'

# Parsing
datetime.strptime("2026-07-26", "%Y-%m-%d")
datetime.fromisoformat("2026-07-26T14:30:00")

# Arithmetic
tomorrow = today + timedelta(days=1)
delta = date(2026, 12, 31) - today
delta.days                              # days remaining

# Timestamps
now.timestamp()                         # 1785…
datetime.fromtimestamp(1785000000)

Common format codes: %Y year, %m month, %d day, %H hour, %M minute, %S second, %A weekday, %B month name

⚠️ Time zones are the biggest trap here. The rules:

  • Always store and transmit UTC
  • Convert to local time only for display
  • For real time-zone work, use zoneinfo (built in since Python 3.9)
python
from zoneinfo import ZoneInfo

berlin = datetime.now(ZoneInfo("Europe/Berlin"))
utc_time = berlin.astimezone(timezone.utc)

22.2 Measuring Time

python
import time

time.time()               # Unix timestamp (affected by clock adjustments)
time.perf_counter()       # high-resolution; use this to measure durations
time.monotonic()          # monotonic clock, never goes backwards
time.sleep(1.5)           # pause

# Timing a block
start = time.perf_counter()
do_work()
print(f"{time.perf_counter() - start:.4f}s")

For microbenchmarks, use timeit:

python
import timeit
timeit.timeit("'-'.join(str(n) for n in range(100))", number=10000)

22.3 Randomness

python
import random

random.random()                     # a float in [0.0, 1.0)
random.randint(1, 6)                # an int from 1 to 6 (both inclusive)
random.randrange(0, 10, 2)          # one of 0,2,4,6,8
random.uniform(1.5, 3.5)            # a float in a range
random.choice(["a", "b", "c"])      # pick one
random.choices(items, k=3)          # pick 3 with replacement
random.choices(items, weights=[1,5,2], k=3)   # weighted
random.sample(items, 3)             # pick 3 without replacement
random.shuffle(lst)                 # shuffle in place

random.seed(42)                     # fixed seed for reproducibility (handy in tests)

⚠️ `random` is not cryptographically secure. For passwords, tokens, and keys you must use secrets:

python
import secrets

secrets.token_hex(16)               # '3d8f2a...'  random hex string
secrets.token_urlsafe(32)           # URL-safe random string
secrets.choice(alphabet)            # secure random choice
secrets.compare_digest(a, b)        # constant-time comparison, resists timing attacks

22.4 Math

python
import math

math.pi, math.e, math.inf, math.nan
math.sqrt(16)          # 4.0
math.floor(3.7)        # 3    round down
math.ceil(3.2)         # 4    round up
math.trunc(-3.7)       # -3   truncate toward zero
math.factorial(5)      # 120
math.gcd(12, 18)       # 6    greatest common divisor
math.lcm(4, 6)         # 12   least common multiple (3.9+)
math.log(100, 10)      # 2.0
math.hypot(3, 4)       # 5.0  Euclidean distance
math.isclose(a, b)     # approximate float comparison
math.isnan(x), math.isinf(x)
math.comb(5, 2)        # 10   combinations
math.dist((0,0), (3,4))  # 5.0

For statistics, use statistics:

python
import statistics as st

st.mean([1,2,3,4])       # 2.5
st.median([1,2,3,4])     # 2.5
st.mode([1,1,2])         # 1
st.stdev([1,2,3,4])      # sample standard deviation
st.pstdev([1,2,3,4])     # population standard deviation
st.quantiles(data, n=4)  # quartiles

22.5 System and Processes

python
import os, sys, platform

os.environ.get("HOME")           # read an environment variable
os.environ["MY_VAR"] = "x"       # set one (affects this process only)
os.cpu_count()                   # number of CPUs
os.getpid()                      # process ID

sys.argv                         # command-line arguments; argv[0] is the script name
sys.exit(1)                      # exit; non-zero signals failure
sys.platform                     # 'darwin' / 'win32' / 'linux'
sys.version_info                 # (3, 14, 6, 'final', 0)
sys.stdout, sys.stderr, sys.stdin

platform.system()                # 'Darwin' / 'Windows' / 'Linux'

Running external commands:

python
import subprocess

# Recommended: pass a list (escaping handled), check the exit code, capture output
result = subprocess.run(
    ["git", "status", "--short"],
    capture_output=True,
    text=True,                  # treat output as str rather than bytes
    check=True,                 # raise if the exit code is non-zero
)
print(result.stdout)

⚠️ Never use `shell=True` with interpolated user input — that's a command injection hole:

python
subprocess.run(f"rm {filename}", shell=True)     # ❌ filename="; rm -rf /" ends your day
subprocess.run(["rm", filename])                 # ✅

22.6 Command-Line Arguments

python
import argparse

parser = argparse.ArgumentParser(description="Batch-process files")
parser.add_argument("input", help="input file path")
parser.add_argument("-o", "--output", default="out.txt", help="output path")
parser.add_argument("-v", "--verbose", action="store_true", help="verbose output")
parser.add_argument("-n", "--count", type=int, default=10)
parser.add_argument("--mode", choices=["fast", "safe"], default="safe")

args = parser.parse_args()
print(args.input, args.output, args.verbose)

You get --help and type validation for free.

💡 More modern options: Typer (built on type hints, half the code) or Click. Both need pip.

22.7 Text Processing

python
import textwrap, string, difflib, unicodedata

textwrap.fill(long_text, width=70)       # wrap to a width
textwrap.dedent(indented_text)           # strip common indentation (great for multiline strings)
textwrap.shorten(text, width=50)         # truncate with an ellipsis

string.ascii_lowercase                   # 'abcdefghijklmnopqrstuvwxyz'
string.digits, string.punctuation

difflib.get_close_matches("aple", ["apple", "banana"])   # ['apple'] fuzzy matching
difflib.unified_diff(lines1, lines2)                     # generate a diff

22.8 Data Handling

python
import json, csv, sqlite3, pickle, base64, hashlib, uuid

# Hashing
hashlib.sha256(b"hello").hexdigest()
hashlib.md5(data).hexdigest()            # ⚠️ md5 is broken; never use it for passwords

# Password storage requires a purpose-built algorithm (third-party)
# pip install argon2-cffi  or  bcrypt

# UUIDs
uuid.uuid4()                             # random unique ID
str(uuid.uuid4())                        # 'f47ac10b-58cc-...'

# Base64
base64.b64encode(b"data").decode()
base64.b64decode("ZGF0YQ==")

# SQLite (a zero-config local database, included in the stdlib)
conn = sqlite3.connect("app.db")
conn.execute("CREATE TABLE IF NOT EXISTS users (id INTEGER PRIMARY KEY, name TEXT)")
conn.execute("INSERT INTO users (name) VALUES (?)", ("Alice",))   # ⚠️ always parameterize
conn.commit()
for row in conn.execute("SELECT * FROM users"):
    print(row)
conn.close()

⚠️ SQL must be parameterized (the ? placeholders). Never build SQL with f-strings — that's SQL injection.

22.9 Networking

python
import urllib.request, urllib.parse

# The stdlib works, but it's verbose
with urllib.request.urlopen("https://api.example.com/data") as r:
    data = json.loads(r.read())

urllib.parse.urlencode({"q": "python", "page": 1})    # 'q=python&page=1'
urllib.parse.quote("some text")                       # URL encoding

💡 For real projects use a third-party library; the difference is night and day:

python
# pip install httpx
import httpx

r = httpx.get("https://api.example.com/data", params={"q": "python"}, timeout=10)
r.raise_for_status()
data = r.json()

# POST JSON
r = httpx.post(url, json={"name": "Alice"}, headers={"Authorization": "Bearer ..."})

requests is the established choice; httpx is the modern alternative — nearly the same API, plus async support. It handles HTTP/2 as well, but that needs an optional dependency: pip install "httpx[http2]", and you must enable it explicitly with httpx.Client(http2=True).

22.10 Functional Tools

python
import functools, operator

functools.reduce(operator.add, [1,2,3,4])     # 10  (just use sum here)
functools.partial(int, base=2)("1010")        # 10  freeze some arguments

operator.itemgetter(1)                        # get index 1, for key=
operator.attrgetter("name")                   # get an attribute
operator.methodcaller("upper")                # call a method

A practical use for partial:

python
from functools import partial

# Create a pre-configured function
save_json = partial(json.dump, ensure_ascii=False, indent=2)
save_json(data, f)      # no need to repeat those two arguments every time

22.11 Others Worth Knowing

python
import copy            # deepcopy
import glob            # file globbing (pathlib.glob is nicer)
import tempfile        # temp files and directories
import zipfile, tarfile, gzip     # compression
import shutil          # high-level file operations
import warnings        # emit warnings
import inspect         # runtime object introspection (for writing frameworks)
import weakref         # weak references
import decimal, fractions         # exact numerics
import heapq           # heaps / priority queues
import bisect          # binary search on sorted lists
import array           # compact numeric arrays
import struct          # binary packing
import html, xml       # markup languages
import email, smtplib  # email
import calendar        # calendars
import pprint          # pretty-print nested structures

A practical use for heapq — top N:

python
import heapq
heapq.nlargest(3, data, key=lambda x: x["score"])
heapq.nsmallest(3, data)

tempfile — don't invent your own names in /tmp:

python
import tempfile
with tempfile.TemporaryDirectory() as tmpdir:
    path = Path(tmpdir) / "work.txt"
    ...    # the whole directory is deleted when the with block exits

22.12 Exercises

22.1 Write a program that computes how many days remain until the end of the current year.

22.2 Generate a secure 12-character random password with uppercase, lowercase, and digits.

22.3 Write a command-line tool that takes a directory path and an --ext option, and counts files with that extension.

22.4 Use sqlite3 to build a to-do database with add, list, and mark-complete operations.

22.5 Identify and fix the security problem here:

python
import subprocess
filename = input("Filename: ")
subprocess.run(f"cat {filename}", shell=True)

<details> <summary>Answers</summary>

22.1

python
from datetime import date

today = date.today()
year_end = date(today.year, 12, 31)
print(f"{(year_end - today).days} days left in {today.year}")

22.2

python
import secrets, string

alphabet = string.ascii_letters + string.digits
password = "".join(secrets.choice(alphabet) for _ in range(12))
print(password)

Use secrets, not random — the latter's pseudo-random output is predictable.

22.3

python
import argparse
from pathlib import Path

parser = argparse.ArgumentParser(description="Count files")
parser.add_argument("directory", type=Path)
parser.add_argument("--ext", default="py", help="extension without the dot")
args = parser.parse_args()

files = list(args.directory.rglob(f"*.{args.ext}"))
print(f"{args.directory} contains {len(files)} .{args.ext} files")

22.4

python
import sqlite3
from contextlib import closing

def init(conn):
    conn.execute("""
        CREATE TABLE IF NOT EXISTS todos (
            id INTEGER PRIMARY KEY,
            title TEXT NOT NULL,
            done INTEGER DEFAULT 0
        )
    """)
    conn.commit()

def add(conn, title):
    conn.execute("INSERT INTO todos (title) VALUES (?)", (title,))
    conn.commit()

def list_all(conn):
    for row in conn.execute("SELECT id, title, done FROM todos ORDER BY id"):
        mark = "✓" if row[2] else " "
        print(f"[{mark}] {row[0]}. {row[1]}")

def complete(conn, todo_id):
    conn.execute("UPDATE todos SET done = 1 WHERE id = ?", (todo_id,))
    conn.commit()

with closing(sqlite3.connect("todo.db")) as conn:
    init(conn)
    add(conn, "Learn Python")
    add(conn, "Do the exercises")
    complete(conn, 1)
    list_all(conn)

22.5 shell=True plus unsanitized user input equals command injection. Typing a.txt; rm -rf ~ would execute the deletion.

python
import subprocess
filename = input("Filename: ")
subprocess.run(["cat", filename], check=True)     # ✅ no shell involved

Better still, don't shell out at all:

python
from pathlib import Path
print(Path(filename).read_text(encoding="utf-8"))

</details>


23. Regular Expressions

A regular expression is a small language for describing text patterns. It's powerful, and it's easy to write something nobody can read.

23.1 Basic Usage

python
import re

text = "My number is 555-1234-5678, backup 555-8765-4321"

re.search(r"\d{3}-\d{4}-\d{4}", text)      # first match; returns a Match or None
re.findall(r"\d{3}-\d{4}-\d{4}", text)     # all matches, as a list
re.finditer(r"\d+", text)                  # all matches, as an iterator (with positions)
re.sub(r"\d", "*", text)                   # substitute
re.split(r"[,;]", text)                    # split on a pattern
re.fullmatch(r"\d+", "123")                # the whole string must match
re.match(r"My", text)                      # match only at the start

⚠️ Always write regex patterns as raw strings `r"..."`, or Python will interpret \d as an escape sequence first.

Match objects:

python
m = re.search(r"(\d{3})-(\d{4})", text)
if m:
    m.group()      # '555-1234'   the whole match
    m.group(1)     # '555'        the first capture group
    m.group(2)     # '1234'
    m.groups()     # ('555', '1234')
    m.start()      # start position
    m.span()       # (13, 21)

23.2 Syntax Reference

Character classes

.        any character (except newline)
\d       digit          \D  non-digit
\w       word char      \W  non-word
\s       whitespace     \S  non-whitespace
[abc]    a, b, or c
[^abc]   anything but a, b, c
[a-z]    a through z
[0-9a-fA-F]  hex characters

Quantifiers

*        zero or more
+        one or more
?        zero or one
{3}      exactly 3
{2,5}    2 to 5
{2,}     at least 2
*?  +?  ??  {n,m}?    non-greedy versions (match as few as possible)

Anchors

^        start of string (start of line in multiline mode)
$        end of string
\b       word boundary
\B       non-boundary

Groups

(...)         capture group
(?:...)       non-capturing group (groups without capturing; more efficient)
(?P<name>...) named group
(?=...)       positive lookahead (must be followed by)
(?!...)       negative lookahead (must not be followed by)
(?<=...)      positive lookbehind (must be preceded by)
(?<!...)      negative lookbehind
|             alternation

23.3 Greedy vs Non-Greedy

The most common source of confusion:

python
text = "<b>bold</b><i>italic</i>"

re.findall(r"<.*>", text)      # ['<b>bold</b><i>italic</i>']  greedy: runs to the end
re.findall(r"<.*?>", text)     # ['<b>', '</b>', '<i>', '</i>']  non-greedy ✅

By default * and + are greedy — they match as much as possible. Adding ? makes them non-greedy.

23.4 Useful Patterns

python
# Email (simplified; a fully correct email regex is famously monstrous — validate with a library)
r"[\w.+-]+@[\w-]+\.[\w.]+"

# US phone number
r"\d{3}-\d{3}-\d{4}"

# URL
r"https?://[^\s]+"

# IPv4
r"\b(?:\d{1,3}\.){3}\d{1,3}\b"

# Date YYYY-MM-DD
r"\d{4}-\d{2}-\d{2}"

# CJK characters
r"[一-鿿]+"

# HTML tags (simple cases only; use Beautiful Soup to parse HTML)
r"<[^>]+>"

# Quoted content
r'"([^"]*)"'

23.5 Named Groups and Substitution

python
# Named groups make this far more readable
pattern = r"(?P<year>\d{4})-(?P<month>\d{2})-(?P<day>\d{2})"
m = re.search(pattern, "Date: 2026-07-26")
m.group("year")      # '2026'
m.groupdict()        # {'year': '2026', 'month': '07', 'day': '26'}

# Referencing groups in a replacement
re.sub(r"(\d{4})-(\d{2})-(\d{2})", r"\3/\2/\1", "2026-07-26")   # '26/07/2026'
re.sub(pattern, r"\g<day>/\g<month>/\g<year>", text)            # by name

# Replacing with a function (for complex logic)
def upper_match(m):
    return m.group().upper()

re.sub(r"\b\w{5,}\b", upper_match, "hello world hi")   # 'HELLO WORLD hi'

23.6 Flags

python
re.IGNORECASE  / re.I    # case-insensitive
re.MULTILINE   / re.M    # ^ and $ match at each line
re.DOTALL      / re.S    # . also matches newlines
re.VERBOSE     / re.X    # allow whitespace and comments

re.search(r"hello", text, re.IGNORECASE)

re.VERBOSE makes complex patterns readable:

python
pattern = re.compile(r"""
    (\d{3})     # area code
    [-\s]?      # optional separator
    (\d{3})     # exchange
    [-\s]?
    (\d{4})     # line number
""", re.VERBOSE)

23.7 Precompiling

When you'll use a pattern many times, compiling it first is faster:

python
pattern = re.compile(r"\d+")

pattern.search(text)
pattern.findall(text)
pattern.sub("X", text)

23.8 When Not to Use Regex

Don't parse structured formats with regex:

python
# ❌ Regex on HTML/XML/JSON — these are nested structures regex can't express
re.findall(r"<div>(.*?)</div>", html)

# ✅ Use a real parser
from bs4 import BeautifulSoup      # HTML
import json                         # JSON
import csv                          # CSV

Don't use regex for simple cases:

python
re.search(r"^abc", s)          → s.startswith("abc")
re.search(r"abc", s)           → "abc" in s
re.sub(r"abc", "x", s)         → s.replace("abc", "x")
re.split(r",", s)              → s.split(",")

String methods are faster and clearer.

Watch out for catastrophic backtracking: certain patterns slow down exponentially on specific inputs, which can be weaponized into a denial-of-service attack.

python
r"(a+)+b"     # hangs on "aaaaaaaaaaaaaaaaaaaaac"

Avoid nesting unbounded quantifiers. Be especially careful with untrusted input.

23.9 Exercises

23.1 Write a regex to extract all email addresses from a block of text.

23.2 Batch-convert dates in "2026-07-26" format to "26 July 2026" style — or, more simply, to "2026/07/26".

23.3 Write a function that validates password strength: at least 8 characters, with at least one uppercase, one lowercase, and one digit.

23.4 Why does this regex for HTML tag contents give the wrong result, and how do you fix it?

python
re.findall(r"<b>(.*)</b>", "<b>a</b> and <b>b</b>")

23.5 Extract all IP addresses and timestamps from a log, producing a list of dicts. Log format: 192.168.1.1 - [2026-07-26 14:30:00] "GET /api"

<details> <summary>Answers</summary>

23.1

python
import re
emails = re.findall(r"[\w.+-]+@[\w-]+\.[\w.-]+", text)

This is a practical approximation only. For strict validation use the email-validator library — or just send a confirmation email, which is the only truly reliable check.

23.2

python
re.sub(r"(\d{4})-(\d{2})-(\d{2})", r"\1/\2/\3", text)

23.3

python
import re

def is_strong(password: str) -> bool:
    if len(password) < 8:
        return False
    checks = [r"[a-z]", r"[A-Z]", r"\d"]
    return all(re.search(c, password) for c in checks)

You could do it in one regex with lookaheads, but it's much harder to read:

python
r"^(?=.*[a-z])(?=.*[A-Z])(?=.*\d).{8,}$"

Separate checks have another advantage: you can tell the user exactly what's missing.

23.4 .* is greedy, so it matches from the first <b> all the way to the last </b>, giving ['a</b> and <b>b'].

python
re.findall(r"<b>(.*?)</b>", text)     # ['a', 'b']  ✅ non-greedy

23.5

python
import re

log = '192.168.1.1 - [2026-07-26 14:30:00] "GET /api"\n10.0.0.5 - [2026-07-26 14:31:02] "POST /login"'

pattern = re.compile(
    r"(?P<ip>\d{1,3}(?:\.\d{1,3}){3}).*?"
    r"\[(?P<timestamp>[\d-]+ [\d:]+)\]"
)

records = [m.groupdict() for m in pattern.finditer(log)]
# [{'ip': '192.168.1.1', 'timestamp': '2026-07-26 14:30:00'}, ...]

</details>


24. Concurrency and Async

This chapter answers one question: can the program do something else while it's waiting?

24.1 Three Ways to Do Several Things at Once

ApproachModuleGood forHow it works
ThreadsthreadingI/O-bound work (network, files)Multiple threads in one process, shared memory
ProcessesmultiprocessingCPU-bound work (computation)Separate processes, true parallelism
AsyncasyncioMassive concurrent I/OCooperative switching within one thread

Which kind is your workload?

  • I/O-bound: most of the time is spent waiting (for the network, disk, database). CPU usage is low.
  • CPU-bound: most of the time is spent computing (image processing, numerics, encryption). CPU is pinned.

24.2 The GIL: Why Python Threads Don't Speed Up Computation

CPython has a Global Interpreter Lock: only one thread can execute Python bytecode at a time.

Consequences:

  • ❌ Threads cannot speed up pure computation (four threads run no faster than one — often slower)
  • ✅ Threads can speed up I/O (a thread waiting on I/O releases the GIL so others can run)
🔍 Going deeper: the free-threaded build (i.e. no GIL) was introduced as an experiment in Python 3.13, and with the acceptance of PEP 779 it became an officially supported build in 3.14. It's still an optional, non-default build — you install python3.14t separately (the t means threaded). Single-threaded performance still trails the standard build by somewhere in the single digits to ~10%, and third-party C extensions are still catching up. So the advice in this chapter stands for now; revisit once your critical dependencies advertise free-threading support.

24.3 Threads

python
from concurrent.futures import ThreadPoolExecutor
import httpx

urls = ["https://example.com/1", "https://example.com/2", ...]

def fetch(url):
    return httpx.get(url, timeout=10).text

# Use the high-level API rather than raw Thread objects
with ThreadPoolExecutor(max_workers=10) as pool:
    results = list(pool.map(fetch, urls))

When you need per-task results and error handling:

python
from concurrent.futures import ThreadPoolExecutor, as_completed

with ThreadPoolExecutor(max_workers=10) as pool:
    futures = {pool.submit(fetch, url): url for url in urls}
    for future in as_completed(futures):        # handle each as it finishes
        url = futures[future]
        try:
            print(url, len(future.result()))
        except Exception as e:
            print(f"{url} failed: {e}")

Thread safety: multiple threads mutating the same variable causes problems.

python
import threading

counter = 0
lock = threading.Lock()

def increment():
    global counter
    for _ in range(100000):
        with lock:              # only one thread at a time in here
            counter += 1

💡 A better approach is avoiding shared mutable state: let each thread return a result and aggregate in the main thread. queue.Queue is itself thread-safe and works well for inter-thread communication.

24.4 Processes

python
from concurrent.futures import ProcessPoolExecutor

def heavy_compute(n):
    return sum(i * i for i in range(n))

if __name__ == "__main__":              # ⚠️ required on Windows/macOS
    with ProcessPoolExecutor() as pool:
        results = list(pool.map(heavy_compute, [10**7] * 4))

Processes bypass the GIL and genuinely use multiple cores. The costs:

  • Process startup is expensive
  • Data must be serialized between processes (via pickle); large objects are slow to move
  • No shared memory objects (you need multiprocessing.Manager or shared memory)

⚠️ `if __name__ == "__main__"` is not optional here — without it, child processes re-import the main module and spawn processes infinitely.

24.5 asyncio

python
import asyncio
import httpx

async def fetch(client, url):            # async def defines a coroutine
    r = await client.get(url)            # await yields control here
    return r.text

async def main():
    async with httpx.AsyncClient() as client:
        tasks = [fetch(client, url) for url in urls]
        results = await asyncio.gather(*tasks)      # run them all concurrently
    return results

asyncio.run(main())                      # start the event loop

Core concepts:

  • A function defined with async def is a coroutine; calling it doesn't execute it, it returns a coroutine object
  • await means "this will take a while — let something else run meanwhile"
  • The event loop schedules among all the coroutines
  • asyncio.run() is the async entry point to your program

Common APIs:

python
await asyncio.sleep(1)                      # async sleep (doesn't block other tasks)
await asyncio.gather(*coros)                # run concurrently, return a list of results
await asyncio.wait_for(coro, timeout=5)     # timeout control
task = asyncio.create_task(coro)            # schedule a background task
await task

# Task groups (Python 3.11+) — safer, better error handling
async with asyncio.TaskGroup() as tg:
    tg.create_task(work1())
    tg.create_task(work2())
# Waits for all tasks on exit; if one fails, the rest are cancelled

async is contagious:

python
async def a(): ...
async def b():
    await a()          # await only works inside an async function
def c():
    await a()          # ❌ SyntaxError

Once you use async, everything up the call chain must be async too. That's asyncio's biggest cost — you can't adopt it in just one small place.

⚠️ Never call blocking functions from async code:

python
async def bad():
    time.sleep(1)                    # ❌ blocks the whole event loop; every task stalls
    requests.get(url)                # ❌ same

async def good():
    await asyncio.sleep(1)           # ✅
    await client.get(url)            # ✅ use an async HTTP client

    # When you must call something blocking, push it to a thread
    result = await asyncio.to_thread(blocking_function, arg)

24.6 Choosing

Is the work CPU-bound?
├─ Yes → ProcessPoolExecutor (or use numpy / a Rust extension)
└─ No (I/O-bound)
    ├─ Concurrency in the dozens → ThreadPoolExecutor (simple, no style change)
    └─ Concurrency in the hundreds or thousands → asyncio (far better memory profile)

Practical advice:

  1. Don't reach for concurrency first. Measure, confirm it's actually the bottleneck, then decide. Premature concurrency doubles complexity.
  2. Prefer `concurrent.futures` — the API is uniform, and switching a thread pool for a process pool is a one-word change.
  3. asyncio suits projects designed for it from the start, not retrofits onto synchronous code.
  4. Use a batch API when one exists — one request for 100 IDs beats 100 concurrent requests every time.

24.7 Complete Example: Concurrent Downloads

python
import asyncio
import httpx
from pathlib import Path

async def download(client: httpx.AsyncClient, url: str, dest: Path,
                   sem: asyncio.Semaphore) -> str:
    async with sem:                       # cap concurrency; don't flatten the server
        try:
            r = await client.get(url, timeout=30, follow_redirects=True)
            r.raise_for_status()
            dest.write_bytes(r.content)
            return f"✅ {url} → {dest.name}"
        except httpx.HTTPError as e:
            return f"❌ {url}: {e}"


async def main(urls: list[str], outdir: Path) -> None:
    outdir.mkdir(parents=True, exist_ok=True)
    sem = asyncio.Semaphore(5)            # at most 5 at a time

    async with httpx.AsyncClient() as client:
        tasks = [
            download(client, url, outdir / f"{i:03d}.dat", sem)
            for i, url in enumerate(urls)
        ]
        for coro in asyncio.as_completed(tasks):
            print(await coro)


if __name__ == "__main__":
    asyncio.run(main(["https://example.com"] * 10, Path("downloads")))

Note the Semaphore — firing hundreds of unthrottled requests gets you rate-limited at best and IP-banned at worst.

24.8 Exercises

24.1 Explain why threads won't speed this up:

python
def compute():
    return sum(i*i for i in range(10**7))

24.2 Use ThreadPoolExecutor to read ten files concurrently.

24.3 Convert this synchronous code to asyncio:

python
import time
def task(n):
    time.sleep(1)
    return n * 2

results = [task(i) for i in range(5)]   # takes 5 seconds

24.4 What's wrong with this async code?

python
async def fetch_all(urls):
    results = []
    for url in urls:
        results.append(await fetch(url))
    return results

<details> <summary>Answers</summary>

24.1 This is CPU-bound. Because of the GIL, only one thread executes Python bytecode at a time, so threads just switch back and forth without computing in parallel — and pay the switching overhead. Use ProcessPoolExecutor.

24.2

python
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path

paths = list(Path("data").glob("*.txt"))

def read(p):
    return p.name, p.read_text(encoding="utf-8")

with ThreadPoolExecutor(max_workers=10) as pool:
    for name, content in pool.map(read, paths):
        print(name, len(content))

24.3

python
import asyncio

async def task(n):
    await asyncio.sleep(1)
    return n * 2

async def main():
    return await asyncio.gather(*(task(i) for i in range(5)))

results = asyncio.run(main())    # takes about 1 second

24.4 This runs serially — each await fetch(url) waits for that request to finish before starting the next. There's no concurrency at all; it's exactly as slow as synchronous code.

python
async def fetch_all(urls):
    return await asyncio.gather(*(fetch(url) for url in urls))

This is the single most common async mistake: `await` inside a loop means serial execution. To get concurrency you must create all the coroutines/tasks first and await them together. </details>


Part 7 · Engineering Practices

There's still a gap between writing code and shipping software. Nothing in this part adds a feature to your program, but it determines whether anyone — including you in three months — can maintain it.

25. Virtual Environments and Dependencies

25.1 Why Virtual Environments

Suppose you have two projects: A needs django==3.2, B needs django==5.0. If every package is installed into the system Python, they collide.

A virtual environment gives each project its own Python and package directory. This isn't an optional best practice — it's mandatory.

25.2 venv: The Standard Library Approach

bash
# Create (generates a .venv directory)
python3 -m venv .venv

# Activate
source .venv/bin/activate          # macOS / Linux
.venv\Scripts\activate             # Windows

# Once active the prompt shows (.venv)
# Now python and pip point at the environment's copies

pip install requests
pip list

deactivate                         # exit

Add .venv/ to .gitignore — virtual environments don't belong in version control; others recreate theirs.

25.3 Everyday pip Commands

bash
pip install requests                 # install
pip install "django>=4.0,<5.0"       # version range
pip install -r requirements.txt      # install from a file
pip install -e .                     # install this project in editable mode
pip install --upgrade requests
pip uninstall requests
pip list                             # what's installed
pip list --outdated                  # what can be upgraded
pip show requests                    # details and dependencies
pip freeze > requirements.txt        # export exact versions of everything

⚠️ pip freeze dumps every package including transitive dependencies. It's better to maintain "what I directly need" by hand — see the next section.

25.4 pyproject.toml: Modern Dependency Declaration

The current standard is to put project metadata and dependencies in pyproject.toml:

toml
[project]
name = "myproject"
version = "0.1.0"
description = "One line about it"
requires-python = ">=3.12"
dependencies = [
    "httpx>=0.27",
    "pydantic>=2.0",
]

[project.optional-dependencies]
dev = [
    "pytest>=8.0",
    "ruff>=0.6",
    "mypy>=1.11",
]

[project.scripts]
mycli = "myproject.cli:main"        # after install, `mycli` runs it

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.ruff]
line-length = 100

[tool.pytest.ini_options]
testpaths = ["tests"]

Install with:

bash
pip install -e ".[dev]"      # the project itself plus dev dependencies

uv unifies Python version management, virtual environments, dependency resolution, and package installation into one tool, 10–100× faster than pip.

bash
# Start a project (creates pyproject.toml)
uv init myproject
cd myproject

# Add dependencies (creates the venv, updates pyproject.toml and uv.lock)
uv add httpx pydantic
uv add --dev pytest ruff mypy

# Run (ensures the environment is current first)
uv run python main.py
uv run pytest

# Sync the environment to the lockfile
uv sync

# Manage Python versions
uv python install 3.14
uv python pin 3.14

# Run a tool without installing it
uv tool run ruff check .
uvx ruff check .              # shorthand

The `uv.lock` file records exact versions and hashes for every dependency, guaranteeing an identical environment on any machine. Commit it to git.

Comparable tools: Poetry (mature, large community), PDM, Hatch. Pick one and stick with it.

25.6 Principles

1. Separate direct dependencies from the lockfile

  • pyproject.toml lists what you directly need, with loose ranges (httpx>=0.27)
  • uv.lock / requirements.txt pins exact versions for reproducibility

2. Constraints shouldn't be too loose or too tight

toml
"httpx"              # ❌ too loose; a future major version can break you outright
"httpx==0.27.0"      # ❌ too tight; you miss security fixes
"httpx>=0.27,<1.0"   # ✅ allows minor updates, blocks breaking changes

3. Update and audit regularly

bash
pip list --outdated
pip-audit                    # check for known vulnerabilities

4. Add dependencies sparingly

Every dependency is a liability: it may have vulnerabilities, may be abandoned, may conflict. Ask first: can the standard library do this? Is one function worth pulling in a whole library?

25.7 Exercises

25.1 Create a new directory, set up a virtual environment in it, install httpx, write a script fetching data from an API, then export requirements.txt.

25.2 Explain why .venv/ belongs in .gitignore while uv.lock should be committed.

25.3 What's wrong with this dependency declaration?

toml
dependencies = ["django", "requests==2.25.0", "numpy>=1.0"]

<details> <summary>Answers</summary>

25.1

bash
mkdir demo && cd demo
python3 -m venv .venv
source .venv/bin/activate
pip install httpx

cat > main.py << 'EOF'
import httpx

r = httpx.get("https://api.github.com/repos/python/cpython", timeout=10)
r.raise_for_status()
data = r.json()
print(f"{data['full_name']}: {data['stargazers_count']} stars")
EOF

python main.py
pip freeze > requirements.txt

25.2

  • .venv/ is a locally generated artifact: large (tens to hundreds of MB) and tied to a specific OS and CPU architecture, so it's useless on another machine. Others rebuild it from the manifest.
  • uv.lock records exact dependency versions and hashes. Committing it is what makes your team's, CI's, and production's environments identical — the foundation of reproducible builds.

25.3 Three problems:

  • django has no constraint at all — a future major release will break the project without warning
  • requests==2.25.0 is pinned to one exact version — no security patches, and it easily conflicts with other packages' requirements
  • numpy>=1.0 has an unrealistically low floor and no ceiling — nothing here actually supports numpy 1.0

Better:

toml
dependencies = [
    "django>=5.0,<6.0",
    "requests>=2.31,<3.0",
    "numpy>=1.26,<3.0",
]

</details>


26. Testing

26.1 Why Write Tests

Not to "prove the code is correct," but to:

  • Change code without fear — this is the biggest payoff by far
  • Force you to write code that's testable (which usually means better designed)
  • Serve as the most accurate documentation there is

Past a certain size, a project without tests becomes one nobody dares touch.

26.2 Getting Started with pytest

bash
pip install pytest

calculator.py:

python
def add(a, b):
    return a + b

def divide(a, b):
    if b == 0:
        raise ValueError("Cannot divide by zero")
    return a / b

test_calculator.py:

python
import pytest
from calculator import add, divide

def test_add():
    assert add(2, 3) == 5

def test_add_negative():
    assert add(-1, -1) == -2

def test_divide():
    assert divide(10, 2) == 5

def test_divide_by_zero():
    with pytest.raises(ValueError, match="Cannot divide by zero"):
        divide(10, 0)

Running:

bash
pytest                    # everything
pytest -v                 # show each test name
pytest test_calc.py       # one file
pytest -k "divide"        # only tests whose names contain divide
pytest -x                 # stop at the first failure
pytest --lf               # rerun only last session's failures
pytest -q                 # quiet output

Conventions:

  • Test files are named test_*.py or *_test.py
  • Test functions are named test_*
  • Use plain assert; pytest generates detailed failure output automatically

26.3 Parameterized Tests

Test one behavior against several inputs without copy-pasting:

python
@pytest.mark.parametrize("a, b, expected", [
    (2, 3, 5),
    (0, 0, 0),
    (-1, 1, 0),
    (0.1, 0.2, pytest.approx(0.3)),      # use approx for floats
])
def test_add(a, b, expected):
    assert add(a, b) == expected

One function, four test cases, and a failure pinpoints exactly which one.

26.4 Fixtures: Preparing Test Data

python
import pytest

@pytest.fixture
def sample_users():
    return [
        {"name": "Alice", "age": 25},
        {"name": "Bob", "age": 17},
    ]

def test_adults(sample_users):            # parameter name matches the fixture; injected automatically
    adults = [u for u in sample_users if u["age"] >= 18]
    assert len(adults) == 1

Fixtures with cleanup:

python
@pytest.fixture
def temp_db(tmp_path):                    # tmp_path is a built-in fixture
    db_path = tmp_path / "test.db"
    conn = sqlite3.connect(db_path)
    conn.execute("CREATE TABLE users (id INTEGER, name TEXT)")
    yield conn                            # the test runs here
    conn.close()                          # cleanup afterwards

Scopes:

python
@pytest.fixture(scope="function")   # default: fresh for every test function
@pytest.fixture(scope="module")     # once per file
@pytest.fixture(scope="session")    # once for the whole run (e.g. starting a database)

Handy built-in fixtures:

python
def test_file(tmp_path):              # a temp directory (a Path object)
    (tmp_path / "a.txt").write_text("hi")

def test_output(capsys):              # capture printed output
    print("hello")
    assert capsys.readouterr().out == "hello\n"

def test_env(monkeypatch):            # temporarily patch env vars / attributes
    monkeypatch.setenv("API_KEY", "test")
    monkeypatch.setattr("mymodule.CONFIG", {"debug": True})

Fixtures defined in conftest.py are visible to every test in that directory without importing.

26.5 What Good Tests Look Like

1. Arrange–Act–Assert

python
def test_withdraw():
    account = BankAccount("Alice", 100)    # Arrange
    account.withdraw(30)                   # Act
    assert account.balance == 70           # Assert

2. One test, one behavior

python
# ❌ When it fails you don't know which step broke
def test_account():
    a = BankAccount("Alice", 100)
    a.deposit(50)
    assert a.balance == 150
    a.withdraw(30)
    assert a.balance == 120
    with pytest.raises(...): a.withdraw(1000)

# ✅ Three separate tests
def test_deposit_increases_balance(): ...
def test_withdraw_decreases_balance(): ...
def test_withdraw_too_much_raises(): ...

3. Names should say what's being tested

python
def test_1():                                       # ❌
def test_withdraw_more_than_balance_raises():       # ✅

That name is what you see when it fails — it should tell you the problem directly.

4. Tests must be independent

No dependence on execution order, no shared mutable state. Every test runs standalone.

5. Focus on boundaries and errors

python
# Empty input, single element, maximum value, None, negatives, very long strings
def test_empty_list(): ...
def test_single_element(): ...
def test_none_input_raises(): ...

Normal cases rarely break. Bugs live at the edges.

26.6 Mocking: Isolating External Dependencies

Tests shouldn't depend on the network, a database, or the clock:

python
from unittest.mock import patch, Mock

def get_user_name(user_id):
    r = httpx.get(f"https://api.example.com/users/{user_id}")
    return r.json()["name"]


def test_get_user_name():
    fake = Mock()
    fake.json.return_value = {"name": "Alice"}

    with patch("mymodule.httpx.get", return_value=fake):
        assert get_user_name(1) == "Alice"

⚠️ Heavy mocking is a design smell. If a function needs five mocks to test, it's doing too much. The better fix is dependency injection:

python
# ❌ Hardcoded dependency, only testable via mocks
def get_user_name(user_id):
    r = httpx.get(...)

# ✅ Dependency passed in; tests supply a fake
def get_user_name(user_id, client=httpx):
    r = client.get(...)

26.7 Coverage

bash
pip install pytest-cov
pytest --cov=myproject --cov-report=term-missing
Name                 Stmts   Miss  Cover   Missing
--------------------------------------------------
myproject/core.py       45      3    93%   67-69

⚠️ Coverage is a useful diagnostic, not a goal. 100% coverage doesn't mean bug-free (executing a line isn't the same as asserting it's correct). But low-coverage areas genuinely aren't tested, which is worth a look.

The practical move: read the Missing column and ask "was leaving these untested deliberate?"

26.8 Other Kinds of Tests

doctest — run the examples in your docs as tests:

python
def add(a, b):
    """Return the sum of two numbers.

    >>> add(2, 3)
    5
    >>> add(-1, 1)
    0
    """
    return a + b
bash
pytest --doctest-modules

The benefit: your documentation can never go stale.

Property-based testing (hypothesis) — generate many inputs looking for counterexamples:

python
from hypothesis import given, strategies as st

@given(st.lists(st.integers()))
def test_sort_is_idempotent(lst):
    assert sorted(sorted(lst)) == sorted(lst)

hypothesis explores edge cases automatically (empty lists, extreme values, duplicates) and, on finding a failure, "shrinks" it to the smallest reproducing case. Excellent for pure functions.

26.9 Exercises

26.1 Write tests for this function covering the normal case and every boundary:

python
def parse_age(text):
    age = int(text)
    if not 0 <= age <= 150:
        raise ValueError(f"Age out of range: {age}")
    return age

26.2 Write parameterized tests for a FizzBuzz function.

26.3 Write a fixture that provides a temp directory containing three test files.

26.4 What's wrong with this test?

python
def test_everything():
    assert add(1, 2) == 3
    assert divide(10, 2) == 5
    assert multiply(3, 4) == 12

<details> <summary>Answers</summary>

26.1

python
import pytest
from mymodule import parse_age

@pytest.mark.parametrize("text, expected", [
    ("0", 0),
    ("25", 25),
    ("150", 150),
    (" 42 ", 42),          # int() ignores surrounding whitespace
])
def test_parse_age_valid(text, expected):
    assert parse_age(text) == expected


@pytest.mark.parametrize("text", ["-1", "151", "1000"])
def test_parse_age_out_of_range(text):
    with pytest.raises(ValueError, match="out of range"):
        parse_age(text)


@pytest.mark.parametrize("text", ["abc", "", "3.14", None])
def test_parse_age_invalid_input(text):
    with pytest.raises((ValueError, TypeError)):
        parse_age(text)

26.2

python
@pytest.mark.parametrize("n, expected", [
    (1, "1"),
    (3, "Fizz"),
    (5, "Buzz"),
    (15, "FizzBuzz"),
    (30, "FizzBuzz"),
    (7, "7"),
])
def test_fizzbuzz(n, expected):
    assert fizzbuzz(n) == expected

26.3

python
@pytest.fixture
def sample_dir(tmp_path):
    (tmp_path / "a.txt").write_text("content A", encoding="utf-8")
    (tmp_path / "b.txt").write_text("content B", encoding="utf-8")
    (tmp_path / "c.log").write_text("log", encoding="utf-8")
    return tmp_path

def test_count_txt(sample_dir):
    assert len(list(sample_dir.glob("*.txt"))) == 2

tmp_path is built into pytest and is cleaned up automatically.

26.4 Three problems:

  1. It tests three things; if the first assert fails, the other two never run
  2. The name test_everything conveys nothing — a failure tells you nothing about what broke
  3. It should be split into test_add, test_divide, and test_multiply

</details>


27. Debugging and Logging

27.1 print Debugging

The most primitive and most used technique. Don't be embarrassed to use it — just use it well:

python
print(f"{user=}")                  # user=User(name='Alice')  — the f-string = syntax
print(f"{len(items)=}, {items[:3]=}")

Remember to delete them when you're done. Or — just use logging instead; see §27.3.

27.2 Breakpoint Debugging

The built-in debugger, pdb:

python
def process(data):
    result = transform(data)
    breakpoint()                   # Python 3.7+; execution stops here
    return result

Common commands:

n (next)      execute the next line
s (step)      step into a function
c (continue)  run to the next breakpoint
l (list)      show surrounding code
p name        print a variable
pp name       pretty-print
w (where)     show the call stack
u / d         move up and down the stack
q (quit)      exit

Inside pdb you can evaluate arbitrary Python expressions, which makes it very flexible.

IDE debuggers (VS Code / PyCharm) are more intuitive: click the gutter to set a breakpoint, press F5, and you get all variables, stepping, and conditional breakpoints. Prefer this.

Dropping into the debugger on failure:

bash
python -m pdb -c continue script.py     # stop where it crashed
pytest --pdb                            # enter pdb on test failure

27.3 logging: Why It Beats print

python
import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
    datefmt="%H:%M:%S",
)

logger = logging.getLogger(__name__)

logger.debug("Detailed diagnostic information")
logger.info("Normal operational information")
logger.warning("Something's off but we can continue")
logger.error("Something failed")
logger.critical("Fatal problem")

Advantages over print:

  • Filter by level — see DEBUG in development, WARNING and up in production, without editing code
  • Automatic timestamps, module names, line numbers
  • Route to files, rotate by size, ship to remote services
  • Third-party library logs are managed the same way

Choosing a level:

LevelUse for
DEBUGDetailed diagnostics, enabled only while investigating
INFONormal milestones (service started, job finished)
WARNINGDoesn't break anything but deserves attention (deprecated API, a retry)
ERRORA feature failed, but the program continues
CRITICALThe program can't continue

Logging exceptions:

python
try:
    risky()
except Exception:
    logger.exception("Processing failed")      # includes the full traceback automatically

logger.exception() only works inside an except block; it's equivalent to logger.error(..., exc_info=True).

Use %s, not f-strings:

python
logger.info("User %s logged in from %s", username, ip)     # ✅ formats only if emitted
logger.info(f"User {username} logged in")                  # ⚠️ always formats, even when DEBUG is off

Writing to files with rotation:

python
from logging.handlers import RotatingFileHandler

handler = RotatingFileHandler(
    "app.log", maxBytes=10_000_000, backupCount=5, encoding="utf-8"
)
handler.setFormatter(logging.Formatter(
    "%(asctime)s [%(levelname)s] %(name)s:%(lineno)d - %(message)s"
))
logging.getLogger().addHandler(handler)

The recommended pattern — configure once at the entry point; every other module just calls getLogger(__name__):

python
# main.py
import logging

def setup_logging(verbose: bool = False):
    logging.basicConfig(
        level=logging.DEBUG if verbose else logging.INFO,
        format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
    )
    logging.getLogger("httpx").setLevel(logging.WARNING)   # quiet a noisy library

if __name__ == "__main__":
    setup_logging()
    main()

⚠️ Library code should never call `basicConfig` — that's the application's job. A library only calls getLogger(__name__) and logs.

💡 For prettier output, try Rich: from rich.logging import RichHandler.

27.4 A Debugging Approach

1. Read the error message

Python 3.11+ error output is genuinely helpful:

File "main.py", line 5
    print(items[i][j])
          ^^^^^^^^^^^
IndexError: list index out of range

The ^^^ markers point at exactly which subexpression failed.

2. Minimize the reproduction

Strip the failing code down to the smallest case. Often you find the cause partway through stripping.

3. Bisect

Add a print or breakpoint in the middle to determine whether the problem is in the first half or the second, then repeat.

4. Check your assumptions

Most bugs come from "I thought it was X, it's actually Y." Print your assumptions explicitly:

python
print(f"{type(data)=}, {len(data)=}, {data[:2]=}")

5. The usual suspects

SymptomLikely cause
Result is off by oneOff-by-one error (range excludes the stop, indices start at 0)
Changed A, B changed tooShared references (§8.6)
Function accumulates across callsMutable default argument (§11.2)
Closures in a loop are all identicalVariable capture (§12.3)
Floats compare unequalPrecision (§4.5)
Text comes out garbledMissing encoding (§6.7)
Second iteration is emptyExhausted iterator (§19.1)
import can't find a moduleFilename collides with the stdlib (§13.5)

27.5 Exercises

27.1 Configure logging for a script: INFO and above to the console, DEBUG and above to a file.

27.2 What's wrong with this logging call?

python
logger.debug(f"Processing data: {expensive_serialize(data)}")

27.3 Use breakpoint() to debug this function and find out why the result is wrong:

python
def average(nums):
    total = 0
    for n in nums:
        total += n
    return total / len(nums) - 1

<details> <summary>Answers</summary>

27.1

python
import logging

def setup_logging(log_file="app.log"):
    root = logging.getLogger()
    root.setLevel(logging.DEBUG)

    fmt = logging.Formatter("%(asctime)s [%(levelname)s] %(name)s: %(message)s")

    console = logging.StreamHandler()
    console.setLevel(logging.INFO)
    console.setFormatter(fmt)

    file = logging.FileHandler(log_file, encoding="utf-8")
    file.setLevel(logging.DEBUG)
    file.setFormatter(fmt)

    root.addHandler(console)
    root.addHandler(file)

The key: set the logger itself to DEBUG (the lowest), and let each handler apply its own filter.

27.2 The f-string calls expensive_serialize(data) unconditionally, even when the level is INFO and this DEBUG record is discarded.

python
logger.debug("Processing data: %s", data)      # formats only if actually emitted
# or
if logger.isEnabledFor(logging.DEBUG):
    logger.debug("Processing data: %s", expensive_serialize(data))

27.3 The bug is the trailing - 1; operator precedence makes it (total / len(nums)) - 1.

python
def average(nums):
    if not nums:
        return 0
    return sum(nums) / len(nums)

How you'd find it: put breakpoint() before the return and inspect p total, p len(nums), and p total / len(nums) — the intermediate values are right, so the final step must be wrong. </details>


28. Code Style and Tooling

28.1 PEP 8: The Official Style Guide

You don't need to memorize it. The essentials:

python
# Indentation: 4 spaces
# Line length: 79 (PEP 8) or 88-100 (modern practice; ruff defaults to 88)

# Naming
variable_name = 1          # variables, functions: snake_case
CONSTANT_VALUE = 1         # constants: UPPER_CASE
class ClassName: ...       # classes: PascalCase
def function_name(): ...
_internal = 1              # internal use: leading underscore

# Blank lines
class A:              # two blank lines between classes and top-level functions

    def method(self): # one blank line between methods
        pass

# Spacing
x = 1                 # ✅ spaces around operators
x=1                   # ❌
f(a, b)               # ✅ space after commas
f(a,b)                # ❌
f(x=1)                # ✅ no spaces around = for keyword arguments
d["key"]              # ✅ no spaces just inside brackets
d[ "key" ]            # ❌

# Import order: standard library → third party → local, blank line between groups
import os
import sys

import httpx
import pydantic

from myproject.utils import helper

Other important conventions:

python
if x is None:              # ✅ compare to None with is
if not items:              # ✅ testing for an empty sequence
if len(items) == 0:        # ❌ verbose

if isinstance(x, int):     # ✅
if type(x) == int:         # ❌

try:                       # ✅ catch specific exceptions
    ...
except ValueError:
    ...
except:                    # ❌ bare except
    ...

28.2 ruff: One Tool for Formatting and Linting

ruff is a linter and formatter written in Rust. It's extremely fast and has largely replaced the black + flake8 + isort combination.

bash
pip install ruff

ruff check .                # find problems
ruff check --fix .          # auto-fix what it can
ruff format .               # format the code

Configure it in pyproject.toml:

toml
[tool.ruff]
line-length = 100
target-version = "py312"

[tool.ruff.lint]
select = [
    "E", "W",    # pycodestyle
    "F",         # pyflakes (unused variables, imports, etc.)
    "I",         # isort (import ordering)
    "N",         # pep8-naming
    "UP",        # pyupgrade (modernize syntax automatically)
    "B",         # bugbear (common bug patterns)
    "SIM",       # simplify
    "RUF",       # ruff's own rules
]
ignore = ["E501"]        # let the formatter handle line length

💡 Let your editor format on save — install the Ruff extension in VS Code, enable format-on-save, and you never think about style again.

28.3 pre-commit: Automatic Checks Before Every Commit

bash
pip install pre-commit

.pre-commit-config.yaml:

yaml
repos:
  - repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.6.9              # ← example version; use the command below to get the latest
    hooks:
      - id: ruff
        args: [--fix]
      - id: ruff-format

  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.6.0              # ← same
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: check-yaml
      - id: check-added-large-files
bash
pre-commit autoupdate    # bump every rev to each repo's latest tag
pre-commit install       # install the git hook

💡 rev must be a concrete tag (not a branch name) so everyone runs the same version. After writing the config, run pre-commit autoupdate once — don't copy version numbers out of tutorials.

From then on, git commit runs the checks automatically and blocks the commit if they fail. That catches the overwhelming majority of trivial problems.

28.4 General Advice for Better Code

1. Naming matters most

python
d = get()                        # ❌ says nothing
active_users = fetch_users()     # ✅

Good names eliminate the need for most comments. Spending 30 seconds on a name is worth it.

2. Keep functions short

Past 30–40 lines, consider splitting. A function that doesn't fit on a screen forces the reader to scroll constantly, and comprehension cost rises sharply.

3. Avoid deep nesting

More than three levels of indentation means it's time to refactor: return early, extract a function, use a comprehension.

4. Eliminate duplication — but don't abstract too early

Wait until the same logic appears a third time. The second occurrence might be a coincidence. Abstractions built too early are usually wrong, and fixing them hurts more than the duplication did.

5. Explicit beats implicit

python
from mymodule import *          # ❌
from mymodule import parse      # ✅

def f(*args, **kwargs): ...     # ❌ unless you genuinely need it
def f(name, age): ...           # ✅

6. Fail loudly and early

python
# ❌ Silent failure; the problem resurfaces far away in a strange form
def get_config(key):
    return config.get(key)

# ✅ Missing config errors right here
def get_config(key):
    if key not in config:
        raise KeyError(f"Missing config key: {key}")
    return config[key]

7. Write a README

Even if you're the only user. You in three months is also "someone else." At minimum it should cover: what this is, how to install it, how to run it.

28.5 The Zen of Python

Type import this in the interpreter:

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

This isn't mysticism; it's a practical decision procedure. When torn between two approaches, ask which is more explicit, simpler, and more readable.

28.6 Exercises

28.1 Rewrite this per PEP 8:

python
def calc( x,y ):
  Result=x*2+y
  if Result>10 :
    return  True
  else :
    return False

28.2 What's wrong with this function? Refactor it.

python
def p(d):
    r = []
    for i in d:
        if i['s'] == 1:
            if i['a'] > 18:
                if i['n'] != '':
                    r.append(i['n'].upper())
    return r

28.3 Set up ruff and pre-commit for a new project.

<details> <summary>Answers</summary>

28.1

python
def calc(x, y):
    return x * 2 + y > 10

Beyond formatting, several improvements:

  • Result shouldn't be capitalized
  • if cond: return True else: return False is just return cond
  • More importantly: calc, x, and y are meaningless names; a real project would use descriptive ones

28.2 Problems:

  1. p, r, d, and i carry no information
  2. The keys s, a, and n are opaque magic strings
  3. Three levels of nested ifs
  4. No type hints or documentation

Refactored:

python
def get_active_adult_names(users: list[dict]) -> list[str]:
    """Return uppercased names of all active adult users."""
    return [
        user["name"].upper()
        for user in users
        if user["status"] == ACTIVE and user["age"] > 18 and user["name"]
    ]

Better still, use a dataclass instead of dicts:

python
@dataclass
class User:
    name: str
    age: int
    status: Status

def get_active_adult_names(users: list[User]) -> list[str]:
    return [u.name.upper() for u in users
            if u.status is Status.ACTIVE and u.age > 18 and u.name]

28.3

bash
uv add --dev ruff pre-commit

pyproject.toml:

toml
[tool.ruff]
line-length = 100
target-version = "py312"

[tool.ruff.lint]
select = ["E", "W", "F", "I", "N", "UP", "B", "SIM"]

.pre-commit-config.yaml:

yaml
repos:
  - repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.6.9
    hooks:
      - id: ruff
        args: [--fix]
      - id: ruff-format
bash
pre-commit autoupdate
pre-commit install

</details>


29. What to Learn Next

Congratulations — you now have Python's core. Where you go next depends on what you want to build.

29.1 First, Build a Few Complete Projects

There's a gap between learning syntax and building things, and the only way across is building things. In rough order of difficulty:

Beginner

  • A command-line to-do manager (file I/O + JSON + argparse)
  • A bulk file organizer (pathlib + regex)
  • A password generator and strength checker (secrets + regex)
  • A currency or unit converter (API calls + caching)

Intermediate

  • Scrape and analyze web data (httpx + Beautiful Soup + pandas)
  • A personal expense tracker (SQLite + dataclass + charts)
  • A static blog generator converting Markdown to HTML
  • An automated daily report (scheduled job + email/chat delivery)

Advanced

  • A REST API service (FastAPI + database + auth)
  • A small interpreter or compiler
  • Implement classic algorithms and data structures from scratch
  • A tool you personally use every day

"Complete" is the key word: include error handling, tests, a README, and enough polish that someone else could use it. A finished small tool teaches far more than a half-built exercise.

29.2 Choose a Direction

Web backends

  • FastAPI — modern, fast, built on type hints; the default for new projects
  • Django — batteries included; great for content sites and admin backends
  • Flask — light and flexible
  • Around them: SQLAlchemy (ORM), Alembic (migrations), PostgreSQL, Redis, Docker

Data analysis

  • pandas / Polars (Polars is faster and gaining ground quickly)
  • numpy, matplotlib / plotly / seaborn
  • Jupyter Notebook
  • SQL is a required companion skill

Machine learning / AI

  • scikit-learn (classical ML)
  • PyTorch (the deep learning mainstream)
  • Hugging Face transformers (pretrained models)
  • LLM API SDKs (anthropic, openai)
  • Around them: linear algebra, probability and statistics

Automation / DevOps

  • Playwright (browser automation; nicer than Selenium)
  • paramiko / Fabric (SSH)
  • Ansible
  • schedule / APScheduler (scheduled jobs)

Desktop and GUI

  • Textual (terminal UIs; surprisingly beautiful)
  • PySide6 / PyQt (desktop GUIs)
  • Streamlit / Gradio (quick UIs for data apps)

Games

  • pygame (2D)
  • Arcade

29.3 Going Deeper in Python

Recommended reading

  • The official tutorial and library reference — docs.python.org
  • Fluent Python — essential for truly understanding Python; the 2nd edition covers through 3.10
  • Effective Python — 90 concrete, highly practical items
  • Real Python — consistently good tutorials

Topics worth studying

  • The descriptor protocol and metaclasses (needed only when writing frameworks)
  • CPython internals and performance tuning
  • C extensions / PyO3 (writing Python extensions in Rust)
  • The memory model and garbage collection

Read source code: the standard library's pathlib.py and dataclasses.py are excellent study material; among third-party packages, httpx and attrs are worth reading.

29.4 Learning Programming Itself

The language is just a tool. These transfer to any language:

  • Git and version control — learn it today; it's foundational
  • The Linux command line — ls/cd/grep/find/pipes deliver an enormous efficiency gain
  • SQL and database design — nearly every project needs it
  • HTTP and networking basics — request/response, status codes, REST
  • Algorithms and data structures — you don't need to grind puzzles, but you do need complexity intuition
  • System design — how to split modules, handle concurrency, and deal with failure

29.5 Habits That Keep You Improving

  1. Write a little every day, even 20 minutes. Consistency beats intensity.
  2. Read other people's code. Pick a small library you use and read it end to end.
  3. Refactor your old code. Something you wrote three months ago will make you wince — that means you improved. Fix it.
  4. Write it down. Blog, document, explain it to someone. If you can explain it, you understand it.
  5. Contribute to open source. Start with documentation fixes and small bugs.
  6. Use AI well, but don't stop at copy-paste. Ask it why, not just what.

29.6 A Few Honest Words

You don't need to memorize the syntax. Professionals look things up constantly. What matters is knowing that a thing exists, not recalling its exact usage.

Getting stuck is normal. Everyone spends two hours hunting a typo. It doesn't mean you're not cut out for this — it is this.

Ugly code that runs is a win. Make it work, make it right, make it fast — in that order, not the reverse.

Building something beats finishing tutorials. One rough tool you actually use is worth more than ten completed courses.

Have fun with it.


Appendix A. Common Errors Cheat Sheet

ErrorTypical causeHow to investigate
SyntaxError: invalid syntaxMissing colon or paren, wrong punctuation, keyword used as a nameCheck the reported line and the one above it
IndentationErrorInconsistent indentation, or tabs mixed with spacesEnable "render whitespace" in your editor
NameError: name 'x' is not definedTypo, not assigned yet, wrong scopeCheck spelling and where it's defined
TypeError: unsupported operand type(s)Type mismatch, e.g. "3" + 5Check both operands with type()
TypeError: 'NoneType' object is not subscriptableIndexing None — usually a function returned NoneCheck whether the upstream function returns anything
TypeError: ... takes 0 positional arguments but 1 was givenA method is missing selfAdd self
ValueError: invalid literal for int()A failed conversion like int("abc")Validate before converting, or use try/except
ValueError: too many values to unpackCount mismatch when unpackingCheck both sides' lengths
IndexError: list index out of rangeOut-of-range index, often an off-by-one in range(len(x))Check bounds; consider enumerate
KeyError: 'xxx'The dict has no such keyUse .get() or check with in first
AttributeError: 'X' object has no attribute 'y'Misspelled method, or the object isn't the type you assumedprint(type(obj), dir(obj))
ModuleNotFoundError: No module named 'x'Not installed, venv not activated, or a filename collisionCheck pip list; confirm you're in the venv
ImportError: cannot import name 'x'The name doesn't exist, or a circular importCheck spelling; circular imports need refactoring
FileNotFoundErrorWrong path, or the working directory isn't what you thinkprint(Path.cwd()); use absolute paths
UnicodeDecodeErrorEncoding mismatchAdd encoding="utf-8"
ZeroDivisionErrorDivision by zero, often averaging an empty listCheck before dividing
RecursionErrorRecursion too deep or missing a base caseCheck the termination condition
UnboundLocalErrorAssigning to a global inside a functionUse global/nonlocal, or pass it as a parameter
StopIterationIterator exhaustedIterators are single-use; list() it if you need it twice
RuntimeError: dictionary changed size during iterationAdding or removing keys while iteratingIterate over a copy: list(d.keys())

A general procedure:

  1. Read the last line — the exception type and message
  2. Find the lowest frame in the traceback that's your code
  3. Add print(f"{variable=}") just before that line
  4. Check whether the variable's type and value match your expectation
  5. If you're still stuck, paste the full error into a search engine

Appendix B. Built-in Functions Reference

python
# Types and conversion
type(x)  isinstance(x, T)  issubclass(A, B)
int()  float()  str()  bool()  bytes()  complex()
list()  tuple()  dict()  set()  frozenset()

# Math
abs()  round()  pow(a, b, mod)  divmod(a, b)
min()  max()  sum()
bin()  oct()  hex()      # to base-prefixed strings
ord("A")  chr(65)         # character ↔ code point

# Sequences
len()  sorted()  reversed()  enumerate()  zip()
range()  slice()
all()  any()
filter(f, it)  map(f, it)
iter()  next()

# Objects
dir(x)              # list all attributes
vars(x)             # return __dict__
getattr(x, "name", default)
setattr(x, "name", value)
hasattr(x, "name")
delattr(x, "name")
id(x)  hash(x)
callable(x)
repr(x)  format(x, spec)

# I/O
print()  input()  open()

# Execution
eval("1+1")         # ⚠️ dangerous; never use on untrusted input
exec(code)          # ⚠️ same
compile()

# Misc
help(x)
globals()  locals()
super()
property()  staticmethod()  classmethod()
breakpoint()

Appendix C. Syntax Cheat Sheet

python
# ============ Variables and types ============
x = 10                      # int
y = 3.14                    # float
s = "text"                  # str
b = True                    # bool
n = None                    # NoneType

# ============ Containers ============
lst = [1, 2, 3]             # list   ordered, mutable
tup = (1, 2, 3)             # tuple  ordered, immutable
dct = {"k": "v"}            # dict   key-value mapping
st  = {1, 2, 3}             # set    unordered, unique

# ============ Strings ============
f"{name} is {age} years old"
f"{pi:.2f}"  f"{n:,}"  f"{r:.1%}"  f"{x=}"
s.strip().lower().split(",")
",".join(items)
s.replace(old, new)
s.startswith(p)  s.endswith(p)
s[::-1]                     # reverse
r"raw\string"

# ============ Slicing ============
s[start:stop:step]          # stop is exclusive
s[:3]  s[3:]  s[::-1]  s[-3:]

# ============ Control flow ============
if cond:
    ...
elif other:
    ...
else:
    ...

x = a if cond else b        # conditional expression

for item in iterable:
    if skip: continue
    if stop: break
else:                       # runs only if no break
    ...

while cond:
    ...

match value:
    case pattern: ...
    case _: ...

# ============ Loop helpers ============
range(start, stop, step)
enumerate(items, start=1)
zip(a, b, strict=True)
reversed(items)  sorted(items, key=f, reverse=True)

# ============ Comprehensions ============
[f(x) for x in it if cond]          # list
{k: v for k, v in it}               # dict
{f(x) for x in it}                  # set
(f(x) for x in it)                  # generator

# ============ Functions ============
def f(a, b=1, *args, key=None, **kwargs) -> int:
    """Docstring."""
    return a + b

lambda x: x * 2

f(*list_args, **dict_kwargs)        # unpacking at the call site

# ============ Classes ============
class Child(Parent):
    class_attr = 0

    def __init__(self, x):
        super().__init__()
        self.x = x

    def method(self): ...

    @property
    def value(self): return self._v

    @classmethod
    def create(cls): return cls()

    @staticmethod
    def helper(): ...

@dataclass
class Point:
    x: float
    y: float = 0.0

# ============ Exceptions ============
try:
    ...
except (ValueError, TypeError) as e:
    ...
except Exception:
    raise
else:
    ...
finally:
    ...

raise ValueError(f"Specific detail: {value}")
raise NewError("...") from original

with open(path, encoding="utf-8") as f:
    ...

# ============ Generators ============
def gen():
    yield 1
    yield from other_iterable

# ============ Decorators ============
import functools

def deco(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        return func(*args, **kwargs)
    return wrapper

@deco
def f(): ...

# ============ Type hints ============
def f(a: int, b: str = "") -> list[int]: ...
x: str | None = None
d: dict[str, list[int]] = {}

# ============ Files ============
from pathlib import Path

p = Path("dir") / "file.txt"
p.read_text(encoding="utf-8")
p.write_text(text, encoding="utf-8")
p.exists()  p.is_file()  p.name  p.stem  p.suffix  p.parent
list(p.parent.glob("*.txt"))
list(Path(".").rglob("*.py"))

# ============ Frequent imports ============
import json, re, os, sys, math, random, time
from pathlib import Path
from datetime import datetime, date, timedelta
from collections import Counter, defaultdict, deque
from dataclasses import dataclass, field
from functools import cache, wraps, partial
from itertools import chain, islice, groupby, product
from enum import Enum, auto
from typing import Any, Protocol, TypeVar

# ============ Script entry point ============
def main() -> None:
    ...

if __name__ == "__main__":
    main()

That's the end of the guide. Have fun out there.