Python Poetry Updates

When updating poetry, make sure to update poetry’s dependencies in addition to poetry itself:

poetry self update  # updates poetry
poetry self lock  # resolves poetry's dependencies
poetry self install  # installs any dependency changes

# maybe necessary
poetry cache clear <cachename> --all

This is not immediately evident to me. I had previously assumed that poetry self update would make sure poetry is as up to date as it can be.

However, if you poetry self show, you can see a list of the dependencies that poetry has. Those get updated independent of poetry itself, it seems.

Debugging 🔗

Background: I’m using a “legacy” repository/source (i.e. any source which is not PyPI) apparently resolves package metadata different than when using PyPI, because this was not an issue if I just used PyPI and not my org’s Artifactory pull-through cache.

I spent FAR TOO LONG trying to understand why this resolved transitive dependencies:

poetry add pydantic@2.6.4

and this did not:

poetry add pydantic@2.7.0

Some interesting discovery:

# using most recent version of poetry
$ poetry --version
Poetry (version 1.8.2)

# PyPI lists dependencies correctly for 2.6.4 (for the sake of comparing what was released by the vendor)
$ pip install 'pydantic==2.6.4' --dry-run --ignore-installed --quiet --report - | jq '.install[0].metadata.requires_dist'
[
  "annotated-types>=0.4.0",
  "pydantic-core==2.16.3",
  "typing-extensions>=4.6.1",
  "email-validator>=2.0.0; extra == 'email'"
]

# PyPI lists dependencies correctly for 2.7.0
$ pip install 'pydantic==2.7.0' --dry-run --ignore-installed --quiet --report - | jq '.install[0].metadata.requires_dist'
[
  "annotated-types>=0.4.0",
  "pydantic-core==2.18.1",
  "typing-extensions>=4.6.1",
  "email-validator>=2.0.0; extra == 'email'"
]

This doesn’t appear to be a Pydantic error - both the version I had working previously (2.6.4) and the version which doesn’t work anymore (2.7.0) list their dependencies correctly.

# Artifactory lists the same dependencies as PyPI
$ pip install 'pydantic==2.6.4' --dry-run --ignore-installed --quiet -i <artifactory endpoint> --report - | jq '.install[0].metadata.requires_dist'
[
  "annotated-types>=0.4.0",
  "pydantic-core==2.16.3",
  "typing-extensions>=4.6.1",
  "email-validator>=2.0.0; extra == 'email'"
]

# Artifactory lists the same dependencies as PyPI
$ pip install 'pydantic==2.7.0' --dry-run --ignore-installed --quiet -i <artifactory endpoint> --report - | jq '.install[0].metadata.requires_dist'
[
  "annotated-types>=0.4.0",
  "pydantic-core==2.18.1",
  "typing-extensions>=4.6.1",
  "email-validator>=2.0.0; extra == 'email'"
]

This doesn’t appear to be an Artifactory issue - I get the same output as when querying PyPI.

Therefore, this must be an issue with poetry… I tried reading the source code, but there was more abstraction than I wanted to try to deal with trying to keep in my head, so I chose to use pdb to step-through code.

# linux
$ head -n 1 $(which poetry)
#!/home/andrew/.local/share/pypoetry/venv/bin/python

# packages are in ~/.local/share/pypoetry/venv/lib/site-packages

# darwin
$ head -n 3 $(which poetry)
#!/bin/sh
'''exec' '/Users/andrew/Library/Application Support/pypoetry/venv/bin/python' '"$0" "$@"  
' '''
# packages are in ~/Library/Application Support/pypoetry/venv/lib/site-packages

Find a place that looks like it could be of interest in the poetry source code, and add a breakpoint(), run poetry lock, and then happy debugging.

I ended up getting to poetry/src/poetry/inspection/info.py::PackageInfo::from_wheel(), which is where I could find that, with pydantic 2.6.4, wheel.requires_dist returned the dependencies, but with pydantic 2.7.0, the list was empty.

In another session, I downloaded the wheel file, installed pkginfo with pip, and inspected the wheelfile for 2.7.0, and saw that I got the dependencies. The difference was that pkginfo I installed with pip was version 1.10.0, and poetry show showed that poetry’s pkginfo was…something else, I don’t remember anymore. It’s not important.

The point being that I spent FAR TOO LONG doing this step through debugging (and jeez, even figuring out how to figure out where the poetry lib was installed) just to find out that I needed to update poetry’s packages, and that isn’t done by default when updating poetry.

Finally, if the cache was incorrect, you need to clear the cache and reinstall. Caches are listed by the source name. For example, my pyproject.toml had this:

[[tool.poetry.source]]
name = "artifactory"
url = "<url>"
priority = "primary"

I needed to poetry cache clear artifactory --all before the resolution was corrected.