Chapter 15: Structuring and Packaging Python Applications

Throughout this book, we have focused on the craft of writing elegant, efficient, and robust Python code. The final skill that distinguishes a senior developer is the ability to assemble that code into a well-structured, maintainable, and distributable application. A collection of scripts is not a project. A professional project is organized, documented, and ready for collaboration and deployment.

This chapter covers the modern conventions for structuring a Python project, managing its dependencies, and packaging it for distribution. We will focus on the current best practices centered around the pyproject.toml standard, which has unified the Python packaging ecosystem.

This chapter covers:

  1. A standard, scalable project directory structure.

  2. The central role of pyproject.toml for packaging and metadata.

  3. Modern dependency management with virtual environments and lock files.

  4. Best practices for imports in a large codebase.

The Anatomy of a Modern Python Project

A clean, predictable structure is the foundation of a maintainable project. While minor variations exist, the following layout is a robust and widely adopted standard.

my_project/
├── .venv/                  # Virtual environment (excluded from Git)
├── src/
│   └── my_package/
│       ├── __init__.py
│       ├── module_one.py
│       └── subpackage/
│           ├── __init__.py
│           └── module_two.py
├── tests/
│   ├── __init__.py
│   ├── test_module_one.py
│   └── test_subpackage.py
├── .gitignore
├── pyproject.toml          # The single source of truth for project config
└── README.md

  • src/ Layout: Placing your actual package (my_package) inside a src directory is a crucial best practice. It prevents a common and frustrating class of import problems. Without src/, Python might accidentally import the local package in your project root instead of the installed version, leading to "works on my machine" issues. This layout forces you to install your project (e.g., with pip install -e .) to run your tests, ensuring your test environment mirrors a real installation.

  • tests/: Tests are kept separate from the application code. This is clean and ensures your testing framework and test files are not bundled into your final distributable package.

  • .venv/: Always work inside a virtual environment to isolate project dependencies. This directory should always be in your .gitignore.

  • pyproject.toml: This is the most important file. It replaces the old collection of setup.py, setup.cfg, requirements.txt, and MANIFEST.in with a single, standardized configuration file.

The Modern Standard: pyproject.toml

Defined in PEP 518 and expanded in PEP 621, pyproject.toml is the unified configuration file for a Python project. It tells tools like pip and build how to build your project and what its dependencies are.

Here's a simple but complete example:

  • [build-system]: Specifies the build tool. setuptools is the classic and most common, but others like hatchling, flit, or poetry-core can be used.

  • [project]: Contains all the metadata about your package: its name, version, dependencies, supported Python versions, etc. This is the information that will appear on the Python Package Index (PyPI).

  • dependencies: This key lists the abstract dependencies your project needs to run. These are typically broad version ranges.

Modern Dependency Management

While pyproject.toml lists your direct dependencies, it doesn't solve the problem of ensuring reproducible builds. Your dependency httpx >= 0.20 might install version 0.21 today and 0.25 a year from now, which could break your application.

The solution is a lock file, which records the exact version of every single package (including sub-dependencies) used in a working environment.

Two popular modern tools that manage this automatically are Poetry and PDM. They both use the pyproject.toml file and maintain a separate lock file (poetry.lock or pdm.lock).

  • They provide a single command to install all dependencies from the lock file, guaranteeing identical environments.

  • They manage virtual environments for you.

  • They provide commands to build and publish your package.

If you prefer a lighter-weight approach, the pip-tools package is an excellent choice. You list your abstract dependencies in a file like requirements.in and run pip-compile to generate a fully pinned requirements.txt file, which acts as your lock file.

Import Strategy: Absolute vs. Relative

In a large project, consistent imports are key to readability.

  • Absolute Imports: from my_package.subpackage import module_two

    • These are explicit and clear. You always know exactly where the imported object is coming from in the project structure. They are the highly recommended default.

  • Relative Imports: from . import module_two (from within subpackage/__init__.py)

    • These can be useful for refactoring within a subpackage, as you can rename the parent package without breaking imports. However, they can make it harder to understand a module's dependencies at a glance.

    • Rule of Thumb: Prefer absolute imports. Use relative imports only for closely-related modules within the same subpackage.

Summary

Building a professional Python application goes beyond writing code. It requires a disciplined approach to project structure, dependency management, and packaging. By adopting the modern standards—a src layout, a comprehensive pyproject.toml file, and a dependency manager that uses lock files—you create a project that is not only scalable and maintainable but also easy for other developers to contribute to and for you to deploy with confidence. This disciplined structure is the final hallmark of a senior Python developer.

PreviousChapter 14: Design Patterns in the Pythonic WorldNextConclusion: The Journey Continues

Last updated