// HACKER NEWS — CYBERSECURITY
uv: Deduplicate all files in the wheel cache
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
On main, we support content-addressed caching, but only at the wheel-level. That is, if you download the same wheel twice from different sources, they share a cache entry. But files within or across wheels are not deduplicated at all.
This PR adds deduplication at the file level: every file is now stored under its BLAKE3 hash in a files-v0 bucket. We hardlink these objects into their original locations in archive-v0, so the installation step doesn't change at all -- we're just deduping within the cache (and cache cleanup removes file objects when their hardlink count drops to one).
In the prior proposal (#19694), we included the following table:
So we're saving 545.2 MiB on my local machine, or about 10% of the cache.
In return, the net effect seems to be something like a <4% slowdown for cold installs (and no effect on warm installs), which I think is probably worthwhile here.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
We benchmarked the optimization in 85c3f7485 against the binary-only cache at a3f977ece, with --preview-features content-addressed-cache enabled on both. These are medians for the complete uv pip install process; positive changes mean slower.