Add packaging guide entry on dependency locking

[alexarmstrongvi]

Addresses Issue #491. First PR to the project. Open to feedback if I am missing general style and teaching philosophy or if it's preferred that this be shorter/longer.

[sneakers-the-rat]

maybe before the examples, we give a sense of the kinds of operations that need to be done in prose:

• create the lockfile
• update the lockfile
• update a single package in the lockfile
• creating different forms of the lockfile for different python versions, OSes, etc.

[alexarmstrongvi]

Changed: cd2c12f
Doesn't seem like PDM supports updating a single package in the lockfile like uv and poetry apart from some workaround using the --constraint arg so I left that out of the examples. I figure people will go to the linked docs if they really are going to use lock files and want to see tool specific workflows.

[sneakers-the-rat]

It does!

https://pdm-project.org/en/latest/reference/cli/#update_2

Positional Arguments:

packages: If packages are given, only update them

It looks like the links in the docs are a little wonky, where the narrative docs link to the wrong place in the CLI docs.

I'll double check this later when at keyboard

[alexarmstrongvi]

Ah. The docs said "Update package(s) in pyproject.toml" instead of in lock file so I skipped over that but I see now what it means. Added: bd49ceb

[sneakers-the-rat]

The docs are def a little messy on pdm, trying to help out a bit with that atm

[sneakers-the-rat]

with dep cooldowns, i think only the 'too rare' case is a problem. I would probably frame this as "one risk when developing using lockfiles is that your lockfile will fall too far behind the most recent versions of your dependencies. When people install your package, they typically will not be using your lockfile, and will install the latest versions of the packages supported by their environment. If your dependencies update and break something you rely on, you might not notice it until someone reports it to you."

[sneakers-the-rat]

to be more specific, i basically think we should just say "you should use dependency cooldowns" as an unambiguous recommendation, but then on the other end give a bit of discussion to build intuition about why not updating frequently would be bad, which is the more common problem to have.

[alexarmstrongvi]

I agree "too rare" is the more likely situation. As far as falling behind on package updates, is there anything beyond bugfixes, security patches, and performance improvements that should be mentioned as reasons to update?

Regarding people installing your package and ignoring lock files (presumably for the case of a library), I intended the tip at the end about CI testing different environments to address this.

A discussion on dependency cooldowns seems best added into sections on building packages and CI as it's something that should be added to tool configs instead of invoked when generating a lock file

[sneakers-the-rat]

reasons to update

well the main one w.r.t. keeping lockfiles updated is detecting backwards incompatible API changes in the deps. Different reasons for updating for package consumers at install time vs. updating locked deps at Dev time, and different considerations for libs vs. Apps. But for the sake of this section I think we can give simple guidance like "use dep cooldowns and update & test lockfile frequently"

I intended the tip at the end about CI testing different environments to address this.

I'll take another full read later to see how reading flow goes, on phone rn

it's something that should be added to tool configs instead of invoked when generating a lock file

I'm not sure what you mean, dep cooldowns are something that can both be done by the consuming installer (ignoring lockfile) and configured for the lockfile resolver, and both should be done. Since goal here is to give intuition and "best practices" guidance, we should just say "you should configure your package to use dep cooldowns when locking" - I think all the major package managers support this now, I just pulled this into pdm a week or two ago

[alexarmstrongvi]

But for the sake of this section I think we can give simple guidance like "use dep cooldowns and update & test lockfile frequently"

Gotcha. Added a new recommendation box: 7fb2aa9

I'm not sure what you mean

I just meant I don't normally see people call sync with cooldown CLI args uv sync --exclude-newer "3 days" and instead put it in a user or project config

[tool.uv]
exclude-newer = "3 days"

so that its not something you have to remember every time you make/update lock files. In either case the recommendation to use cooldowns was added above.

[sneakers-the-rat]

Ok ya totally, on the same page. Asking ppl to remember a special set of params on every invocation is no good.

[sneakers-the-rat]

I personally don't read the lockfile when it updates, and so we might not want to give the impression to newbies that it's normal to read lockfile changes since it can be daunting. My pattern is basically "update the lockfile, run the tests." maybe i'll keep a loose eye for transitive dependencies if e.g. something suddenly starts pulling in a huge dep like scipy or something. maybe the thing to communicate is like "what to do if you update and a new version breaks your stuff" and how to handle adding e.g. temporary version caps

[alexarmstrongvi]

Clarified its not necessary and that testing is more important: a1664bc

[sneakers-the-rat]

I like the changed language, when others get around to reviewing this I imagine there will be more said on version capping, which is a hot topic for us lol, but this is one case where it makes sense - the version range should reflect the range of versions that work for a given state of the package. Ideally one would just adapt to the changes in the upstream dep (and maybe scooch up the dep version floor if the changes aren't backwards compatible), but cutting a release with a version cap temporarily until you can make the fix so that it actually runs when people install it is a reasonable case for a cap.

[sneakers-the-rat]

I have done this, where i have one branch of the tests run with a pip install and the rest run with the lockfile as a sentinel, we might want to add a bit more scaffolding in the form of an example CI action for this. Usually I just to that on linux with latest python.

[alexarmstrongvi]

Seems good to reference the testing section here and then add an example there. To avoid the combinatoric explosion of maintaining guides on local vs CI testing, for various tools (e.g. nox, hatch, GitHub Actions), with and without lock files, what are your thoughts on making a high level point that testing can setup environments from a lock files instead of resolving at runtime, showing one example replacing run: python -m pip install ... with run: uv sync --frozen, and the linking to documentation for various tools?

[sneakers-the-rat]

Yeah, good point, agreed. Don't need a full CI example for this, having just one CI example for updating the lockfile is plenty for one page. Just having like a two line thing like

# install most recent dependencies compatible with [project.dependencies]
python -m pip install .

# install exact versions in lockfile
uv sync --frozen

As you say is probably plenty. I can imagine that helping generally with "wait when would I do one command vs the other" and "what does pip do/what does uv do" confusion as well.

[alexarmstrongvi]

Added a dropdown box giving more detail on cooldowns: 69d49dd

I am not familiar with hatch and nox workflows but after a quick search I wasn't seeing how to enable cooldowns or just calling uv instead of pip. The most recent pip releases have a PIP_UPLOADED_PRIOR_TO env var that can be used so perhaps that is the solution to recommend for the Tests section.

[sneakers-the-rat]

I think right above this we need a section like "why are lockfiles good" or "why do they exist" to orient people to what they are doing and why before getting into how to work with them.

something like "lockfiles create a predictable development environment that helps reduce problems where code runs on one person's machine but crashes on another. together with CI result logs, they are a versioned record of the exact set of code that passed or failed the tests. for applications (as opposed to libraries) that are intended to be used as-is rather than depended on by other packages, they allow someone to install and run it and be confident that it will work, without accidentally installing a more recent version of some dependency that is incompatible"

[alexarmstrongvi]

Added more motivation for lock files in line with your points in the intro paragraph: 88d6b78

[sneakers-the-rat]

platform and python version deps are the main things that my lockfiles vary by: maybe generalized like "lockfiles can be customized to include all the different versions that are applicable across different operating systems, python versions, and other platform markers, while that information can only be stored in requirements.txt files through file naming conventions."

idk that text isn't very clear either, but something like that

[alexarmstrongvi]

Clarified: 88f7445

[sneakers-the-rat]

nice, thank you for drafting this! we have needed this for sure. i think the main thing that's TODO here is providing a recommended workflow, getting a lockfile updating setup can have some weird pitfalls that aren't obvious. i am pretty bad about doing this across all my packages because usually i am very stingy with deps in the first place, and otherwise will be touching them frequently enough that i relock, but the thing that i end up doing is doing a weekly automated PR that just updates the lockfiles and runs the tests. check for any added dependencies, if no new deps, merge it if tests pass. if there are new deps, list them in the PR.

I am not sure the current state of the tooling but the final straw for me that pushed me to switch away from using poetry was when i requested and drafted a command that would just print the deps that would change from a lock update, but they didn't want to do that for some reason, so i just ended up writing some heinous bash scripts for it, but if it's more reasonable now we should include an example CI workflow for that. Not saying mine is correct or the best, so that can look like anything, but i think that's sort of the major thing that is a pain in the ass about lockfiles and so would be one of the main things we want to provide in the guide.

[sneakers-the-rat]

Lookin gewd to me. I'll wait on others to review before I do a final read so I don't just load down the PR with my opinions and take up all the space.

[alexarmstrongvi]

Sounds good. I appreciate the quick feedback

[ucodery]

This guide is high quality. It has great information, and every dimension I wanted to see covered, was. And with a very modern take on dependencies. I think this will become a very important part of our guide.

Some notes for your consideration

[ucodery]

I really like your motivation section, a few notes on the small details, take these as suggestions

• pyproject.toml: "defines the widest possible set of environments". Ideally yes, all supported, but I don't want readers to feel they must either validate every possible combination (impossible) or list only a few known-to-work configurations.
• pylock.toml: "used for deployment" I think development is the wrong word choice here, as it is only during active development that the lock files should actually be changing, or not even used but instead installing from requirements to see what changes will be required. Yes, basing CI off of lock files makes for more reproducible checks, but that can be seen as a necessary step in the release process that is validating the lock file tied to the next release as much as the code in that release. I would like to focus the lock file motivation more on deploying and releasing (even if it is a pre-release) than on development workflow.

The way I see it, pyproject should strive to eliminate false-negatives, denying users the ability to install software because is doesn't fit their environment; this will create some false-positives, where it was installed but won't work. pylock, on the other hand, should reduce false-positives, if it installs the environment is known-good, but this disallows installing into unknown-good environments and restricts deployment possibilities.

[sneakers-the-rat]

pyproject should strive to eliminate false-negatives. pylock, on the other hand, should reduce false-positives

+1 here, this is a good framing.

i am not sure i am getting your point in the second bullet, but maybe one distinction (that i don't think is being made yet) is between application-like things and library-like things, where when i am using an application-like thing i do indeed run from the lockfile in deployment, but not in library-like things (mostly as a byproduct of how python's import machinery/historical reasons/etc. that's mostly a digression). not sure if that would be a useful distinction or TMI as a motivating example of how lockfiles are used/how their use depends on how the package is used.

[ucodery]

I'm not convinced we should be adding a mini-tutorial for every tool. This guide is meant to provide a single happy path for packaging and this seems distracting. We also don't use poetry or pdm anywhere else in the guide (we use hatch, which can use pip or uv underneath, so the uv [pip] section would seem to cover all those cases).

[ucodery]

Maybe adding some data to the comparison table would be a better place

python-package-guide/package-structure-code/python-package-build-tools.md

Line 210 in 072e979

### Package tool features table

[sneakers-the-rat]

if we take these out, maybe links to the relevant docs? i think it's useful to see that the same concepts apply across different packaging frontends, but yeah maybe we don't need to write and maintain tutorials for each tool.

[ucodery]

How can an install command be entirely replaced by a lock command?

[ucodery]

I think we need to address best practices of which lock file to keep in VCS. You mentioned the native and universal formats supported by tools, but at most one should be committed. The question is then which one is our best practice for application developers?

[sneakers-the-rat]

imo we should recommend pylock bc it's the standard (and using it ubiquitously would definitely be a better situation for ecosystem at large than continuing to use frontend-specific lockfiles)

[ucodery]

Ambitiously, yes. More realistically, a lock file can be presumed to be a valid resolution to the set of dependencies that will work with this code. Whether or not testing was done in that environment is not captured by a lock file.

[sneakers-the-rat]

(this might be my fault, i was thinking of this as motivation for lockfile: if you have CI logs by commit, then what you have is a complete versioned record of state of code + dependencies + tests, but that might be too subtle/unnecessary)