restore: download package sources concurrently before installing by marcosnav · Pull Request #760 · rstudio/packrat

marcosnav · 2026-10-06T18:56:50Z

Intent

Cold restores that install many packages are about 3 times faster.
A 115-package Shiny app restores in 33.5 s instead of 104.7 s on Linux with Posit Package Manager binaries.

Here is a report with results for conducted experimentation:
https://connect.staging.pct.posit.it/content/4b3c4ebb-b863-47be-9709-cd9b1170fead/

Approach

Before installing anything, restore() downloads the sources of every uncached package from CRAN-like repositories with one curl --parallel.
The install loop is unchanged: it finds the files already on disk, and downloads anything the prefetch missed as before.

It only runs when curl 7.75.0 or later is the download method and packrat installs from source repositories, and it uses the same download.file.extra as other downloads.
On Windows, packages keep downloading one at a time, following the vendored renv, which runs its parallel work sequentially there.
packrat.prefetch.sources turns it off, and packrat.prefetch.concurrency sets the number of simultaneous downloads (default 8, the experiments showed that more than 8 at a time do not show a substantial difference).

Validation

On Linux, with an empty packrat cache:

Point a project's lockfile at a Package Manager binary repository, for example https://p3m.dev/cran/__linux__/noble/latest, and use options(download.file.method = "curl").
Run packrat::restore().
The log shows Prefetching sources for N packages (8 at a time) ... fetched N of N, and the install loop makes no downloads.
Run it again with options(packrat.prefetch.sources = FALSE) and compare the restore time.

🤖 Generated with Claude Code

restore() downloaded each package just before installing it, one curl process at a time. It now fetches the sources of every package it will install from a CRAN-like repository with a single `curl --parallel` before the install loop starts. Anything the prefetch can't fetch is downloaded by the install loop as before, and a prefetch error is a warning, never a failed restore. The prefetch runs only when curl 7.75.0 or later is the download method and packrat installs from source repositories. It reuses the download's curl arguments, so download.file.extra (user agent, .netrc) carries over. Options: packrat.prefetch.sources and packrat.prefetch.concurrency.

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

marcosnav · 2026-10-06T19:21:07Z

+prefetchEnabled <- function() {
+  isTRUE(getOption("packrat.prefetch.sources", TRUE))


Should the parallel download be on by default?

marcosnav · 2026-10-06T19:22:18Z

+prefetchConcurrency <- function() {
+  concurrency <- as.integer(getOption("packrat.prefetch.concurrency", 8L))
+  if (length(concurrency) != 1L || is.na(concurrency) || concurrency < 1L) {
+    stop("'packrat.prefetch.concurrency' must be a positive integer")


Not sure we need to have the packrat.prefetch.concurrency option. Experiments showed that a concurrency of 8 is great improvement and a higher number does not make much difference.

I don't feel strongly either way

On Windows, restore() keeps downloading one package at a time, following the vendored renv, which runs its parallel work sequentially there. Document this, and mock is.windows() in the tests that exercise the prefetch past that check so they run on Windows too.

karawoo

Could we add a test that confirms that if a package is prefetched then installPkg() doesn't download it again?

karawoo · 2026-10-07T21:51:05Z

+  extra <- ""
+  if (identical(getOption("download.file.method"), "curl")) {
+    extra <- paste(getOption("download.file.extra", ""), collapse = " ")
+  }


This omits a user agent header when download.file.method isn't set. Package manager will then return source packages instead of binaries. Without the prefetch, current package versions get downloaded with libcurl which sends R's user agent so package manager would return binaries.

We could fix this by adding -A with getOption("HTTPUserAgent") when extra doesn't already set a user agent.

Good catch, prefetch now passes -A with getOption("HTTPUserAgent") unless download.file.extra already sets a user agent (-A, --user-agent, or a User-Agent: header).

karawoo · 2026-10-07T21:53:35Z

+  # On Windows, packages keep downloading one at a time, following the
+  # vendored renv, which runs its parallel work sequentially there (see
+  # renv_parallel_cores()).
+  if (!prefetchEnabled() || is.windows()) {
+    return(invisible())
+  }


renv runs sequentially on windows because it uses mclapply which uses forking and doesn't work on windows. I don't think this applies to curl --parallel

Ah you're right! Thank you! I removed the Windows check and the notes in NEWS and the docs, now Windows gets the prefetch too.

karawoo · 2026-10-07T22:04:31Z

+prefetchEnabled <- function() {
+  isTRUE(getOption("packrat.prefetch.sources", TRUE))


karawoo · 2026-10-07T22:05:04Z

+prefetchConcurrency <- function() {
+  concurrency <- as.integer(getOption("packrat.prefetch.concurrency", 8L))
+  if (length(concurrency) != 1L || is.na(concurrency) || concurrency < 1L) {
+    stop("'packrat.prefetch.concurrency' must be a positive integer")


I don't feel strongly either way

karawoo · 2026-10-07T22:09:38Z

+pkgRecord <- function(name, version, source = "CRAN", ...) {
+  list(name = name, version = version, source = source, hash = "hash", ...)
+}
+
+availableMatrix <- function(versions, repository) {
+  matrix(
+    c(versions, rep(repository, length(versions))),
+    ncol = 2,
+    dimnames = list(names(versions), c("Version", "Repository"))
+  )
+}
+
+skip_if_no_parallel_curl <- function() {
+  version <- curlVersion()
+  if (is.null(version) || version < prefetchMinimumCurlVersion) {
+    skip("curl with --parallel and %{exitcode} is not available")
+  }
+}


could these helpers go into a helper.R file under tests/testthat/? that is the conventional location for R test helpers.

Moved to tests/testthat/helper.R 👍

karawoo · 2026-10-07T22:27:26Z

+  # When binary repositories are in use, installPkg() installs binaries with
+  # install.packages() and never needs these sources.
+  if (hasBinaryRepositories() && binaryRepositoriesEnabled()) {
+    return(invisible())
+  }


This comment isn't entirely accurate. Even when binary repositories are in use, installPkg() only installs a binary when that version is in the binary index.

packrat/R/restore.R

Lines 630 to 637 in dc9618c

# Try downloading a binary (when appropriate).

if (

hasBinaryRepositories() &&

binaryRepositoriesEnabled() &&

isFromCranlikeRepo(pkgRecord, repos) &&

pkgRecord$name %in% availablePackagesBinary(repos = repos)[, "Package"] &&

versionMatchesDb(pkgRecord, availablePackagesBinary(repos = repos))

) {

Packages whose version isn't there are downloaded as sources one at a time, and could benefit from the prefetch. Could we either update the comment or use the same checks as installPkg() here?

Oh! yes, I went with the same checks as installPkg(). That condition is now installsFromBinaryRepository() in restore.R and both places use it. The prefetch only skips packages that will be installed as binaries, so versions missing from the binary index still get prefetched.

- Send R's user agent (HTTPUserAgent) with the prefetch unless download.file.extra already sets one, so Package Manager serves binaries as it does for single-package downloads. - Prefetch on Windows too: curl --parallel doesn't fork, so renv's reason for running sequentially there doesn't apply. - With binary repositories in use, skip only the packages installPkg() installs as binaries, using the same checks (now shared as installsFromBinaryRepository()), instead of skipping the prefetch. - Move test helpers to tests/testthat/helper.R, and test that installPkg() uses a prefetched source without downloading it again.

marcosnav · 2026-10-08T20:31:34Z

Could we add a test that confirms that if a package is prefetched then installPkg() doesn't download it again?

Yes! Added new tests for that. It runs a real prefetch from a file:// repo, calls installPkg() with getSourceForPkgRecord() mocked to fail, then checks the prefetched file is what gets installed.

marcosnav · 2026-10-08T20:32:37Z

@karawoo Thank you for the review! Ready for another look

karawoo

thanks for the updates! I found just a couple of other things, I think this is close though.

karawoo · 2026-10-09T04:57:08Z

+      paste(available[pkgRecord$name, "Repository"], pkgSrcFile, sep = "/")
+    } else {
+      paste(
+        sub("/+$", "", repos[[1]]),


this tries to prefetch packages from the first configured repository and assumes a CRAN layout. but if a package is not present in the first repo, or it's not a CRAN layout, we aren't able to prefetch it. could the prefetch retry failed downloads similar to downloadArchiveWithRetries?

Yes, added in 618daf2. Ready for another look.

Co-authored-by: Kara Woo <karawoo@users.noreply.github.com>

The prefetch tried archived packages only at the CRAN archive layout of the first repository, so a package in another repository or a non-CRAN layout was left to the install loop. The archive URLs downloadArchiveWithRetries() tries are now built by archiveUrls(), and the prefetch tries them in the same order: each round is one `curl --parallel` that tries the next URL of every package not yet downloaded. archiveUrls() strips trailing slashes from repository URLs, so the install loop no longer requests paths like `cran//src/contrib`.

marcosnav and others added 2 commits October 6, 2026 12:54

Apply batched suggestions from code review

caa50df

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

marcosnav commented Oct 6, 2026

View reviewed changes

marcosnav marked this pull request as ready for review October 6, 2026 20:45

marcosnav requested review from aronatkins and karawoo October 6, 2026 20:45

karawoo reviewed Oct 7, 2026

View reviewed changes

marcosnav requested a review from karawoo October 8, 2026 20:32

marcosnav force-pushed the prefetch-package-sources branch from 152016e to 3ba2841 Compare October 9, 2026 17:17

karawoo reviewed Oct 9, 2026

View reviewed changes

marcosnav and others added 3 commits October 9, 2026 16:00

Update R/prefetch.R

f329163

Co-authored-by: Kara Woo <karawoo@users.noreply.github.com>

Update R/packrat.R

91dccf9

Co-authored-by: Kara Woo <karawoo@users.noreply.github.com>

marcosnav requested a review from karawoo October 13, 2026 13:45

karawoo approved these changes Oct 13, 2026

View reviewed changes

marcosnav merged commit 8f45f08 into main Oct 14, 2026
10 checks passed

marcosnav deleted the prefetch-package-sources branch October 14, 2026 01:40

		prefetchEnabled <- function() {
		isTRUE(getOption("packrat.prefetch.sources", TRUE))

	# Try downloading a binary (when appropriate).
	if (
	hasBinaryRepositories() &&
	binaryRepositoriesEnabled() &&
	isFromCranlikeRepo(pkgRecord, repos) &&
	pkgRecord$name %in% availablePackagesBinary(repos = repos)[, "Package"] &&
	versionMatchesDb(pkgRecord, availablePackagesBinary(repos = repos))
	) {

Conversation

marcosnav commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Intent

Approach

Validation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

karawoo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marcosnav commented Oct 8, 2026

Uh oh!

marcosnav commented Oct 8, 2026

Uh oh!

karawoo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

marcosnav commented Oct 6, 2026 •

edited

Loading