Chillymosh's Blog

Back
13 min read

Generics in Python: Simplifying Complexity

A walk through of the basics regarding generics in Python with plenty of examples.

While hanging out in a Discord server a few days ago, someone brought up the topic of generics in Python. This sparked curiosity among several members, leading to questions about what generics are and how or when to use them. Generics aren’t unique to Python, many programming languages use them as fundamental tools for creating robust and efficient software.

This conversation got me thinking it would be a great topic to explore together on this blog. So, let’s dive into the basics of using generics in Python, discuss some variants, and discover how they can streamline our code and enhance its reliability.

What are generics?#

Generics are designed to enhance the flexibility and safety of your code. Here’s what they aim to do:

  • Enable Flexibility: They allow functions, methods, and classes to work with different data types while preserving information about the relationships between these data types, such as the consistency between input arguments and return values.
  • Improve Type Safety: Generics help specify how different types can interact more explicitly, reducing errors by clarifying the data types used in operations.

We achieve these benefits by using generic types—placeholders for data types—instead of specific or parent types when defining components. This allows your code to be both flexible and safe, accommodating various data types while ensuring operations are performed correctly.

If that sounds a bit abstract, don’t worry! The best way to understand generics is to see them in action. Let’s dive into some simple examples to see how you can use generics in your Python projects to make your code more robust and flexible.

Using generics in Python#

I’ll begin by using the traditional approach familiar to most Python developers. This involves utilising TypeVar from the typing module. As we progress through the post, I’ll also cover the Type Parameter Syntax introduced by PEP-695 ↗ in Python 3.12. This update brings several enhancements that simplify the declaration and usage of generics, and I’ll highlight some of its key nuances. It is also important to note that Python will not do anything special with the generics, it is dependent on the person running a type checker e.g. Pyright / Pylance, MyPy or Pyre.

This function accepts a list containing elements of any type and returns the first element. It uses the “ type to indicate that it can handle elements of any type without restriction.

from typing import Any

def get_first_item(items: list[Any]) -> Any:
    return items[0]

a: list[int] = [1, 2, 3]
b: list[str] = ["hello", "world"]
c: list[list[int]] = [[1, 2], [3, 4, 5]]

print(get_first_item(a))
print(get_first_item(b))
print(get_first_item(c))
python

So why choose a generic over using Any? The key advantage of using a generic lies in its ability to ensure consistency and type safety. With a generic, you can guarantee that the type returned by the function matches the type of the elements in the provided list. This consistency is not assured with Any.

from typing import TypeVar

T = TypeVar("T")

def get_first_item(items: list[T]) -> T:
    return items[0]

a: list[int] = [1, 2, 3]
b: list[str] = ["hello", "world"]
c: list[list[int]] = [[1, 2], [3, 4, 5]]

print(get_first_item(a))
print(get_first_item(b))
print(get_first_item(c))
python

Since the specific type of elements list contains is not of concern, rather only that the function returns the same type as provided in the list, we define a generic type named T with T = TypeVar("T"). This approach replaces the use of Any, which lacks this level of type specificity.

By using T, we introduce type safety to the get_first_item() function, safeguarding against issues like returning a transformed type or an incorrect literal. This setup ensures that our function’s behaviour remains predictable and consistent, protecting against unintended coding errors.

def get_first_item(items: list[T]) -> T:
    return str(items[0]) # Expression of type "str" is incompatible with return type "T@get_first_item"
python

Let’s look at one more simple example before we move on.

The below will warn that get_first_key() is expecting to return a key and not a value from the dictionary. But, as mentioned near the start of this post, this will not prevent the code from running and it will still print the first value, which is 1.

from typing import TypeVar

K = TypeVar("K")
V = TypeVar("V")

def get_first_key(container: dict[K, V]) -> K:
    return list(container.values())[0] # Expression of type "V@get_first" is incompatible with return type "K@get_first"

test: dict[str, int] = {"k": 1}
print(get_first_key(test))
python

Limiting generics to specific types#

In the previous examples, we utilised generic types that could represent any possible type. But what if we want to narrow down and specifically limit the types that a generic can represent?

from typing import TypeVar

T = TypeVar("T", str, int)

def combine_elements(items: list[T]) -> str:
    return "".join(str(item) for item in items)

print(combine_elements(["hello", "world"]))  # Output: helloworld
print(combine_elements([1, 2, 3]))  # Output: 123

print(combine_elements([1.0, 2.0, 3.0])) # Argument of type "list[float]" cannot be assigned to parameter "items" of type "List[T@combine_elements]" in function "combine_elements"
python

Here we constrain T to only allow str and int types. When we try passing a list of floats we see the type checker warn that combine_elements() is not expecting this type.

We can also set an upper bound constraint, limiting the generic type to that specific type and any of its subclasses.

from typing import TypeVar

T = TypeVar("T", bound=int)

class X(int):
    pass

class Y(int):
    pass

def combine_coords(hour: T, minute: T) -> tuple[T,T]:
    return hour, minute

combine_coords(X(14), Y(50))
combine_coords(X(20), 50)

combine_coords("10", "50") # Argument of type "Literal['10']" cannot be assigned to parameter "hour" of type "T@combine_coords" in function "combine_coords" - Type "Literal['10']" is incompatible with type "int"
python

Generics with classes#

By using generics in classes, developers can design components that are not tied to specific data types, thus increasing the utility and scalability of their code. A prime example is the implementation of data structures, such as lists, queues, and trees, which can hold any type of data but enforce type consistency within an instance. This approach eliminates common bugs that arise from type mismatches and enhances code safety.

from typing import Generic, TypeVar

T = TypeVar("T")

class Box(Generic[T]):
    def __init__(self) -> None:
        self._contents: list[T] = []

    def add(self, item: T) -> None:
        self._contents.append(item)

    def get_contents(self) -> list[T]:
        return self._contents

toy_box = Box[str]()
toy_box.add("Teddy Bear")
toy_box.add("Toy Car")

bolt_box = Box[int]() 
bolt_box.add(10) 
bolt_box.add(20)

bolt_box.add("Toy Hammer")

print(toy_box.get_contents())
python

We have created a Box class—a versatile container that can hold various specialised items. In our scenario, we have two types of boxes: a toy box and a bolt box. The toy box is designated for toys, represented as strings, like “Teddy Bear” or “Toy Car”, and must be put away before bedtime. The bolt box is used for storing different sizes of bolts, represented as integers.

However, we’ve made a mistake by trying to add a “Toy Hammer” to our bolt box. Our type checker, as usual, will alert us to this mistake.

Argument of type "Literal['Toy Hammer']" cannot be assigned to parameter "item" of type "int" in function "add" "Literal['Toy Hammer']" is incompatible with "int"

But remember it will not prevent us from adding “Toy Hammer” to the list in Box.

Variants of generics#

There are three variants of generics, which simply refer to different behaviours or specifications that can influence how type checking behaves.

Invariance#

By default, all type variables in Python are invariant. Invariance means that if you have a generic type G[T], it cannot accept G[U], even if U is a subclass or superclass of T. Most mutable containers are invariant. e.g. list and dict.

For example, list[Animal] can not be used where list[Cat] is expected, where Animal is a superclass of Cat.

Covariance#

A type variable is covariant if it allows a generic class to be substituted with a subclass. This is typically used for return types, where a method of a generic class promises to return a type T, and it’s safe to use a type U such that U is a subclass of T. You declare this in the TypeVar.

It’s also important to note that when a generic class is covariant, it is recommended to be immutable / read-only. Some examples of this are typing.Sequence, typing.FrozenSet and typing.Union. Remember this for later in the post.

from typing import Generic, TypeVar

T_co = TypeVar("T_co", covariant=True, bound="Animal")

class Animal: ...

class Cat(Animal): ...

class Dog(Animal): ...

class Snake: ...

class AnimalShelter(Generic[T_co]):
    def __init__(self, animal: T_co) -> None:
        self.animal = animal
        
def adopt_animal(shelter: AnimalShelter[Animal]) -> Animal:
    return shelter.animal

cat = Cat()
dog = Dog()
snake = Snake()

cat_shelter = AnimalShelter(cat)
dog_shelter = AnimalShelter(dog)
reptile_shelter = AnimalShelter(snake)

adopt_cat = adopt_animal(cat_shelter)
adopt_dog = adopt_animal(dog_shelter)
adopt_snake = adopt_animal(reptile_shelter)
python
  • We define a TypeVar T_co that is marked as a covariant by setting covariant=True and bounding it to the Animal class. This means T_co can accept any subclass of Animal.
  • Using T_co, we create an AnimalShelter generic class that is initialised with an instance of Animal or its subclasses.
  • We define a function adopt_animal() that takes an AnimalShelter housing any Animal and returns the animal.
  • We create shelters for different animals, cats, dogs, and hypothetically, snakes. Since snakes are not a subclass of Animal they will trigger a type error:

Argument of type "Snake" cannot be assigned to parameter "animal" of type "T_co@AnimalShelter" in function "__init__" Type "Snake" is incompatible with type "Animal"

Contravariance#

Allows substitution of a type with its supertype. For example, if a function expects a type of Cat, then it will also accept its supertype, Animal.

We define a type variable T_contra that is contravariant and bounded by the class Animal. This setup implies that wherever a specific type like Cat is expected, any of its supertypes (up to Animal) can be used instead.

from typing import Generic, TypeVar

T_contra = TypeVar("T_contra", contravariant=True, bound="Animal")

class Animal:
    def speak(self) -> str:
        return "Some noise"

class Cat(Animal):
    def speak(self) -> str:
        return "Meow"

class Handler(Generic[T_contra]):
    def handle(self, t: T_contra) -> None:
        print(t.speak())

def handle_animal(a: Handler[Cat]) -> None:
    a.handle(Cat())

handler: Handler[Animal] = Handler()
handle_animal(handler) 

cat_handler: Handler[Cat] = Handler()
dog_handler: Handler[Dog] = Handler()
handle_animal(cat_handler) 
handle_animal(dog_handler) # Type parameter "T_contra@Handler" is contravariant, but "Dog" is not a supertype of "Cat"
python

Generics with protocols#

In practical applications, it’s often beneficial to use a Protocol to define an interface. This approach ensures that any subclass implementing the interface will include the necessary methods, as enforced by type checking warnings.

from typing import Generic, TypeVar, Protocol

T_co = TypeVar("T_co", covariant=True, bound="Animal")
T_contra = TypeVar("T_contra", contravariant=True, bound="Animal")

class Animal(Protocol):
    def speak(self) -> str: ...

class Cat:
    def speak(self) -> str:
        return "Meow"

class Dog:
    def speak(self) -> str:
        return "Woof"

class Fish:
    def swim(self) -> None: ...
        
class Handler(Generic[T_contra]):
    def handle(self, t: T_contra) -> None:
        print(t.speak())

class AnimalShelter(Generic[T_co]):
    def __init__(self, animal: T_co) -> None:
        self.animal = animal

def adopt_animal(shelter: AnimalShelter[Animal]) -> Animal:
    return shelter.animal


def handle_animal(a: Handler[Cat]) -> None:
    a.handle(Cat())

handler: Handler[Animal] = Handler()
handle_animal(handler) 

cat_handler: Handler[Cat] = Handler()
fish_handler: Handler[Fish] = Handler() # "Fish" is incompatible with protocol "Animal"
handle_animal(cat_handler) 
handle_animal(fish_handler) # Type parameter "T_contra@Handler" is contravariant, but "Fish" is not a supertype of "Cat"

cat = Cat()
dog = Dog()
fish = Fish()
cat_shelter = AnimalShelter(cat)
dog_shelter = AnimalShelter(dog)
fish_shelter = AnimalShelter(fish) # Type "Fish" is incompatible with type "Animal"
python

In the example above, the Fish class does not implement the speak() method, therefore it does not meet the requirements of the Animal protocol. Because of that, you can’t use Fish in situations where you need something that fits the Animal protocol, whether it’s covariant or contravariant.

Python 3.12 Generics (PEP 695)#

With PEP 695, we can streamline how generics are defined. Let’s take our very first example.

from typing import TypeVar

T = TypeVar("T")

def get_first_item(items: list[T]) -> T:
    return items[0]
This would simply become

def get_first_item[T](items: list[T]) -> T:
    return items[0]
python

Note how we did not need to import and define the TypeVar for each generic. How about with a Protocol?

from typing import Protocol, TypeVar


C1 = TypeVar("C1")
C2 = TypeVar("C2") 

class CurrencyExchangeMapper(Protocol[C1, C2]):
    def to(self, amount: C1) -> C2: ...
    def from_(self, amount: C2) -> C1: ...

class USDGBPExchangeMapper(CurrencyExchangeMapper[float, float]):
    def __init__(self, exchange_rate: float, reverse_rate: float) -> None:
        self.exchange_rate = exchange_rate
        self.reverse_rate = reverse_rate

    def to(self, amount: float) -> float:
        return amount * self.exchange_rate

    def from_(self, amount: float) -> float:
        return amount * self.reverse_rate

exchange_mapper = USDGBPExchangeMapper(exchange_rate=0.777255, reverse_rate=1.28658)
print(f"100 USD is {exchange_mapper.to(100):.2f} GBP") 
print(f"100 GBP is {exchange_mapper.from_(100):.2f} USD")
python

The only part that changes is the CurrencyExchangeMapper class.

class CurrencyExchangeMapper[C1, C2](Protocol):
    def to(self, amount: C1) -> C2: ...
    def from_(self, amount: C2) -> C1: ...
python

Setting bounds and limiting types in PEP 695 are similar, but there is an important difference. With the enhancements introduced in PEP 695, type variance does not always need to be explicitly defined because it can often be inferred from the context.

from typing import Sequence

class DataBundle[T, B: Sequence[bytes], S: (int, str)]:
    def __init__(self, general: T, binary_data: B, identifier: S) -> None:
        self.general = general
        self.binary_data = binary_data
        self.identifier = identifier
        
# Pre 3.12

from typing import TypeVar, Generic, Sequence

T = TypeVar("T", contravariant=True)
B = TypeVar("B", bound=Sequence[bytes], covariant=True)
S = TypeVar("S", int, str)

class OldDataBundle(Generic[T, B, S]):
    def __init__(self, general: T, binary_data: B, identifier: S) -> None:
        self.general = general
        self.binary_data = binary_data
        self.identifier = identifier
python

Recall when I said to remember that covariants are recommended to be immutable / read-only? Well here is an example showing how PEP 695 is enforcing this with class generics. I personally find this stricter approach to be more correct, but it is a differing behaviour to class Thing(Generic[T]).

Note how this does not warn of any errors.

from typing import Generic, TypeVar

T_co = TypeVar("T_co", bound="Animal", covariant=True)

class Animal:
    pass

class Cat(Animal):
    pass

class AnimalShelter(Generic[T_co]):
    def __init__(self, animal: T_co) -> None:
        self.animal = animal

def adopt_animal(box: AnimalShelter[Animal]) -> Animal:
    return box.animal
    
cat = Cat()
rescue_cat = AnimalShelter(cat) 
my_cat = adopt_animal(rescue_cat)
python

Now if we convert this to the PEP 695 way

class Animal:
    pass

class Cat(Animal):
    pass

class AnimalShelter[T_co: Animal]:
    def __init__(self, animal: T_co) -> None:
        self.animal = animal

def adopt_animal(box: AnimalShelter[Animal]) -> Animal:
    return box.animal

cat = Cat()
rescue_cat = AnimalShelter(cat) 
my_cat = adopt_animal(rescue_cat) # "AnimalShelter[Cat]" is incompatible with "AnimalShelter[Animal]"
 Type parameter "T_co@AnimalShelter" is invariant, but "Cat" is not the same as "Animal"
python

Because of inference, it deems AnimalShelter[Animal] to be invariant as self.animal is mutable. There are two ways to resolve this.

The first is by using typing.Final. This informs the type checker that the name cannot be re-assigned or overridden in a subclass.

from typing import Final

class Animal:
    pass

class Cat(Animal):
    pass

class AnimalShelter[T_co: Animal]:
    def __init__(self, animal: T_co) -> None:
        self.animal: Final = animal
python

The second, and the way I believe is more correct, is by making self._animal private and using a @property, essentially making it “read-only”.

class Animal:
    pass

class Cat(Animal):
    pass

class AnimalShelter[T_co: Animal]:
    def __init__(self, animal: T_co) -> None:
        self._animal = animal
        
    @property
    def animal(self) -> Animal:
        return self._animal

def adopt_animal(box: AnimalShelter[Animal]) -> Animal:
    return box.animal

cat = Cat()
rescue_cat = AnimalShelter(cat) 
my_cat = adopt_animal(rescue_cat)
python

Final thoughts#

Both approaches have their place and choosing which way you want to handle your generics in Python will essentially come down some key points.

  • Project and team scale: Larger, more complex projects might still benefit from the explicit nature of traditional generics, particularly for maintaining strict type control across large teams.
  • Codebase Maturity: Introducing new syntax in a mature codebase requires careful consideration, training, and potential refactoring. New projects have the freedom to adopt the latest features from the start.
  • Development Philosophy: Some teams prioritise using the latest Python features to keep their codebase modern and concise, while others value the stability and clarity brought by the established patterns.
  • Personal project: Use whatever you want.

Thanks for reading and if you wish to dig further into generics in Python check out the following resources:

https://peps.python.org/pep-0484/#generics ↗

https://peps.python.org/pep-0695/ ↗

https://mypy.readthedocs.io/en/stable/generics.html ↗

Generics in Python: Simplifying Complexity
https://chillymosh.com/blog/generics-in-python-simplifying-complexity
Author Chillymosh
Published at 28 July 2024