Incremental static builds

[matthewp]

Summary

Incremental static builds, allowing Astro to skip regenerating prerendered pages whose template dependencies and per-path data are unchanged since the previous build.

Links

• Rendered RFC
• Stage 2 Proposal
• Stage 1 Proposal

[spaceemotion]

A couple points coming to my mind:

1. Cyclic Dependencies:

   I can kind of imagine there being quite a few cases where there's a build that makes the module dependency chains kind of cyclic? would that mess up the plans for the cache key generation?

2. Conditional Modules:

   How will the system handle conditional async module loading? I am guessing those are handled by the vite module graph already? (e.g. client scripts that import different scripts depending on backend flags, fetch calls, etc.)

3. Randomness in Assert handling / Code Transformation output:

   I already noticed that the current image pipeline generates different images just by sheer randomness of the optimization process.

   As in; i rebuild and get different images each time (same visual output, just not byte-identical)

[matthewp]

For your first 2 points, the graph doesn't change based on these scenarios. Cyclical dependencies might creates unwanted invalidations; you wind up with more pages being rebuilt than you would like, but from Astro's perspective this is the correct thing to do, as any of those modules changing could affect the output so we must treat it as an invalidation. In reality I think this only happens if you have a module that is used by a lot of pages and it changes often. Like if you somehow added an abstraction on top of content collections itself.

For dynamic imports, it doesn't matter if you conditionally import based on data. The invalidation is based on the graph, so if a dynamic import changes, that page is invalidated. It doesn't wait to see what happens at runtime. This is an extra reminder to make sure the code includes dynamic imports in the graph though.

[matthewp]

About image generation, I wouldn't expect there to be randomness in that process. That sounds like a possible bug, please report if it's something you can recreate.

[matthewp]

In review withastro/astro#17084. it was brought up that middleware can effect output. That's true, middleware can literally do anything, it can mutate every response. So from the framework perspective we can't trust any project with middleware. That would be overkill. I think this is likely a documentation thing, but maybe there's another solution.

[ematipico]

That's odd, though. i18n is purely middleware; does that mean that projects that use i18n routing can't use this feature?

[adamchal]

• [medium][design] packages/astro/src/core/build/incremental-content-collector.ts:13-43 - Content tracking uses one global Set. Concurrent renders overwrite each other's collection, while out-of-process prerenderers such as Cloudflare cannot populate it at all. Because content-data modules are excluded from route hashes, affected pages can remain cached after imported content components change. Tracking must be request-scoped and transported across custom prerenderers.

This finding and f0236de workaround is a bit of a bummer and I’m confused by the finding, because I think it is mixing two things:

1. “Concurrent renders overwrite each other's collection” seems more practical to solve; but
2. “Out-of-process prerenderers such as Cloudflare cannot populate it at all” sounds more like a completely separate issue and is a known-limitation throughout code comments.

Item 2 (out-of-process prerenderers) is annoying. They would need to have a way to inject-back their content collection entries and that inter-process communication is a bit rabbit hole. I agree that these out-of-process adapters are just at-risk at this stage.

However, I am not super familiar with how generate.ts gets multi-threaded when build.concurrency is greater than 1. But, is it too big of an effort to handle multiple writers to the Set()?

Maybe something like beginContentEntryCollection() continues to create the Set() if undefined, but also pushes a deferred Promise.withResolvers() to another globalThis array and returns the resolve/reject methods that generate.ts would use instead of endContentEntryCollection(). I’m not exactly sure where the Promise.all(…) would be awaited safely and performantly, but it might be a safer pattern even in the case of build.concurrency = 1.

[matthewp]

@ematipico No this doesn't affect i18n. What I mean is that middleware could do:

export const onRequest = defineMiddleware((ctx, next) => {
  const response = await next();
  
  if(ctx.url.pathname.startsWith('/foo')) {
    const html = await response.text();
    const newHtml = mutateResponseHtml(html);
    return new Response(html, ....)
  }
})

It can change the response of any page. We only invalidate on cacheKey changes, we don't know about this sort of thing middleware might be doing.

[matthewp]

@adamchal That's a workaround for the experimental release, I want to resolve that before stable. We should stop using global state and concurrency will work just fine, but that's a bigger thing to resolve than I had time for for this release.

[sgalcheung]

If the cacheKey has been set to a JSON value, this will not be regenerated when the JSON has been changed.
1、Set cacheKey to an object type

export const MagazineIssueSchema = z.object({
  id: z.string(),
  // 年份
  year: z.number(),

......

  // 杂志目录
  sections: z.array(MagazineSectionSchema),
});

export const MagazineSectionSchema = z.object({
  // 栏目标题
  title: z.string(),

  // 栏目副标题（文章总标题）
  subTitle: z.string().optional(),

  // 栏目文章
  items: z.array(MagazineItemSchema),
});

export async function getStaticPaths() {
  const zgjjjcEntries = await getCollection('zgjjjcs');

  return [
    ...zgjjjcEntries.map((entry) => ({
      params: {
        magazine: 'zgjjjc',
        slug: entry.id,
      },
      **cacheKey: String(entry.data.sections),**
    })),
  ];
}

2 、 Then we'll get a "[object Object],[object Object]".

// node_modules/.astro/incremental-build.json
{
	"version": 1,
	"configHash": "15d467264622b576c47f0313c659d2b525b10f6c7d68a81a9122172722f40ef8",
	"lockfileHash": "e1755bbd6e33d1e5ec993e2e9e9b5dc14545aa50cfca521e8c3dda2c21a3f4eb",
	"keyDigest": "c04b31b1a24a95483721fe0bbbceb9438fc80f2b0d68abe84ae3f35d9df7e042",
	"routes": {
		"src/pages/[magazine]/[slug].astro": {
			"dependencyHash": "9e2a96445ea81e652d120549322c93ffe35b24e0ee282ac9f7909d36e593d89e",
			"paths": {
				"/zgjjjc/202501": {
					"cacheKey": "[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]",
					"outputFile": "zgjjjc/202501/index.html"
				},
				"/zgjjjc/202607": {
					"cacheKey": "[object Object],[object Object]",
					"outputFile": "zgjjjc/202607/index.html"
				},
				"/zgjjjc/202614": {
					"cacheKey": "[object Object],[object Object]",
					"outputFile": "zgjjjc/202614/index.html"
				},
					"outputFile": "cpaj/202607/index.html"
				}
			}
		}
	}
}

I find that this source code parses the cacheKey to String.

https://github.com/withastro/astro/blob/7c4bf1b0c3d609281082ab196947c7f67e1c1088/packages/astro/src/runtime/prerender/static-paths.ts#L123

I'm not sure if this is a design issue, but the cacheKey can only accept string types?

But in the RFC, if data changed will not caught by cacheKey, Wouldn't that defeat the purpose of the design?

In this case, I want to check if the data(sections) values ​​of my Collection have changed. If they have, rebuild the collection; otherwise, use the cache. But As matthewp sad "The cacheKey is set by the user and is expected to be unique", I'm not sure if I've misunderstood something.

I saw the origin discussion #1096 ,and withastro/astro#16240 (comment) also mention this,

cacheKey: string on GetStaticPathsItem while CollectionEntry.digest is string | number — their own fixture uses String(entry.digest), which hints at the friction.
cacheKey is a flat user-controlled string, not a hash of inputs. If users hand back e.g. post.updatedAt, reordering or schema migrations that don't bump that timestamp will silently reuse stale HTML. The same correctness concern the dependency-key model in this PR tries to take off the user's plate.

For more, I created an issue withastro/astro#17635 and a PR withastro/astro#17638.

[matthewp]

@sgalcheung You want to use JSON.stringify() not String() here.

[sgalcheung]

Can you? JSON.stringify(), no cache will be generated.

{
	"version": 1,
	"configHash": "15d467264622b576c47f0313c659d2b525b10f6c7d68a81a9122172722f40ef8",
	"lockfileHash": "e1755bbd6e33d1e5ec993e2e9e9b5dc14545aa50cfca521e8c3dda2c21a3f4eb",
	"keyDigest": "d25f9c402703f9c98ceaa055fe85609fa28fe4fa785e289bc2d024c60757baef",
	"routes": {}
}

Hello everyone, please take this issue seriously. We are currently unable to communicate. I believe the main problem is the unclear definition of requirements, which has led to greater confusion in its use.

[delucis]

Summarising some feedback from some initial testing of the API. There’s also a bit more on Discord in this feedback thread: https://discord.com/channels/830184174198718474/1536673147364057209

TL;DR — the design being top-level and depending on getStaticPaths() feels restrictive. It blocks some use cases I was investigating and seems (anecdotally) to be making it harder for users to understand.

Feedback

1. The new API is only available in routes that use getStaticPaths(), but caching in a regular single route/endpoint is also desirable. For example, converting an example.astro to [example].astro and adding a getStaticPaths() just to get access to caching support feels clunky.

2. The new API is available only at the very top route level. However, users do not expect to have to know all data inputs to a route at the top level; it is very common for data and other non-import graph inputs to be accessed at the component level. This means the new API requires quite a mental shift in how people architect their component trees, pulling anything that isn’t an import all the way up to the top level, which is not always practical and sometimes impossible. This also weakens a current design advantage of Astro in how it isolates component logic from global concerns.

   Anecdotally, we’re seeing this aspect of the API design lead to people forgetting to include deeply nested inputs in the cacheKey, resulting in incorrect cache hits and out-of-date build output.

   For projects like Starlight, this aspect of the design also make it very hard/impossible to use the current API because there’s no easy way to know if user code deeper in the tree should invalidate the cache.

3. Pulling all data up to the top-level is not ideal for dev-time performance. Doing everything in getStaticPaths() means all data for all routes must be calculated to serve a single route during dev. This can be slow for large sites (e.g. imagine calculating full page data for all 6,000 Astro docs pages to render 1).

4. DX-wise, being left with no utilities for creating hashes/digests from your cache inputs is not ideal. I think it’s OK to be out-of-scope for now potentially, but eventually it would be nice to not leave this entirely to users because correctly hashing data is not always obvious. As an example, in astro-emit-asset users can pass data objects as cache keys and the library hashes them. That may be promising too much for Astro to do, but a utility that does something similar could be helpful.

Opportunities

Instead of a route-level entry in getStaticPaths(), support a component-level API that allows components to declare their data inputs. This would address points one and two above:

1. The API would be available in any component and route, not only those that support getStaticPaths().
2. Users could co-locate caching logic with data input logic where it is used
   • this simplifies remembering all cache inputs
   • this allows “distributed” responsibility for cacheability, which means library and user code can both contribute to a route’s cache key (e.g. Starlight template + user component)

Here’s an example API for how this might work (naming could be bikeshedded of course). It is aligned with the getStaticPaths() style of things but can exist at any level of the component tree (or in static endpoints):

---
import { getEntry } from 'astro:content';
import fs from 'node:fs/promises';

// Inputs not tracked by the import graph are loaded inside a special function
export const getStaticData = async (ctx) => {
  const entry = await getEntry('pages', ctx.props.id);
  const file = await fs.readFile('./some-data.csv', 'utf-8');
  return {
    cacheKey: ctx.digest([entry.digest, file]),
    data: { entry, file }
  };
}

const { entry, file } = Astro.staticData;
---

Render time behaviour:

1. getStaticPaths() for a route (if present) is used to discover all paths.
2. getStaticData() is called for each component in the tree that exports one, collecting the data required to render.
3. If any getStaticData() call returns a cacheKey that has changed, the route is rendered, otherwise the cached output is reused.

Ideally components should know about their props when collecting data to support use like

<BlogCard id="foo" />

where <BlogCard> then does something like getEntry('blog', props.id). I don’t think this would be feasible though — knowing props requires running render code (e.g. imagine {ids.map(id => <BlogCard {id} />)}). That’s definitely an annoying detail here.

Some user scenarios

• As a user, I want to load a blog post with getEntry() without having to remember to also update getStaticPaths().
• As a user, I want to fetch() an API response in a component and still benefit from correct caching.
• As a user, I expect Astro to know about its own APIs and am surprised I have to babysit my usage of them to make caching work.
• As a library author, I want to inject routes but also let users contribute to whether they should be cached or not.

Final notes

I hope this doesn’t come off as too critical — obviously this is a big, exciting, and challenging feature! My current evaluation from a user POV is somewhat negative though: I feel there are a lot of somewhat painful tradeoffs required to adopt the API such that you would only use it in extreme cases where it is absolutely essential to reduce build time. If the intention is for it to be an advanced API only used in rare cases, that may be acceptable, but I wish the migration path from not using caching to using caching could be smoother rather than requiring users to re-architect their site data usage.

[markws62]

Allowing build.concurrency > 1 would be a great help as sometimes even changes typically affect around a lot of pages so generating those pages with build.concurrency = 1 takes a considerable amount of time.

[ematipico]

@delucis

There's the other side of the coin, which shouldn't be overlooked. Updating the cache of a component could potentially have a drastic side effect on all its consumers (importers). At a point where many pages, even more than you might think, could get updated.

So I would like if there was a comparison against this mental model, because I doubt there's a winner, and we should make sure we find a compromise.

[delucis]

Updating the cache of a component could potentially have a drastic side effect on all its consumers (importers). At a point where many pages, even more than you might think, could get updated.

I’m not sure I understand the concern — that seems desirable behaviour to me? If a component changed, the page should be updated. Not re-rendering a page that contains components that changed is a bug surely?

[adamchal]

There are two other potential performance gotchas I've come across with the Cloudflare adapter.

1. Base64 / JSON stringifying of the prerendered body

cloudflare/src/utils/prerender.ts#L152-L161

const envelope: PrerenderEnvelope = {
  status: response.status,
  statusText: response.statusText,
  headers: [...response.headers.entries()],
  body: arrayBufferToBase64(bufferedBody),
  metadata: { contentEntryKeys, staticImages },
};
return new Response(JSON.stringify(envelope), {
  headers: { 'Content-Type': 'application/json' },
});

This can be punishing, especially if your pages are large.

2. incrementalBuild: true with default concurrency

Even if I just set incrementalBuild: true and leave the default concurrency (which usually gets set >1 based on OS thread reporting), I noticed a big slowdown, even though I knew incremental caching would be disabled because concurrency was above 1.

The problem is that what disqualifies incremental caching on the Astro side, in astro/src/core/build/generate.ts#L110-L120:

if (options.settings.config.experimental.incrementalBuild) {
  // Per-path content-entry and image tracking is collected through a single
  // process-global side channel, which cannot attribute records to the right
  // path once renders interleave. Rather than risk skipping a stale page, the
  // cache is disabled when the build renders paths concurrently.
  if (options.settings.config.build.concurrency > 1) {
    logger.warn(
      'build',
      'The incremental build cache is disabled because `build.concurrency` is greater than 1.',
    );
  } else {

is not picked up on the adapter side, in cloudflare/src/index.ts#L526-L533:

hasBuildImageService,
hasBindingImageService: isBindingBuild,
userImageServiceEntrypoint: hasUserBuildImageService
  ? resolveImageServiceEntrypoint(_config.image.service.entrypoint, _config.root)
  : undefined,
incremental: _config.experimental?.incrementalBuild ?? false,
logger,

So the prerenderer (workerd) still does all of its base64 encoding only for the Node generator side to throw it out. A possible simple fix:

--- a/packages/integrations/cloudflare/src/index.ts
+++ b/packages/integrations/cloudflare/src/index.ts
@@ -528,7 +528,9 @@
 							userImageServiceEntrypoint: hasUserBuildImageService
 								? resolveImageServiceEntrypoint(_config.image.service.entrypoint, _config.root)
 								: undefined,
-							incremental: _config.experimental?.incrementalBuild ?? false,
+							incremental:
+								(_config.experimental?.incrementalBuild ?? false) &&
+								_config.build.concurrency === 1,
 							logger,
 						}),
 					);

The bigger issue

The first item reminds me of the base64 encoding of the images in collectStaticImages(). My preference would be to avoid base64 encoding when communicating cross-process, in the case of adapters (Cloudflare) running their own separate process for the prerenderer, and instead rely on streaming the Response bytes.

But that also brings up the bigger question of how to address concurrency. Fundamentally, I think the cross-channel communication between the Node-side build generator and the adapter-side prerenderer needs a better way to communicate so it can handle race conditions, efficient byte streaming, and so on.

[sgalcheung]

1. Base64 / JSON stringifying of the prerendered body

Using fs.stat to retrieve metadata might be better.
Reading a 50MB file and calculating its hash can take hundreds of milliseconds or even longer; however, using fs.stat to get the file's modification time (mtime) and size (size) only takes a few microseconds.

Regarding the handling of cacheKeys, I agree with delucis's opinion; the PR: Fix cachekey in incremental could have done even better.

4. DX-wise, being left with no utilities for creating hashes/digests from your cache inputs is not ideal. I think it’s OK to be out-of-scope for now potentially, but eventually it would be nice to not leave this entirely to users because correctly hashing data is not always obvious. As an example, in astro-emit-asset users can pass data objects as cache keys and the library hashes them. That may be promising too much for Astro to do, but a utility that does something similar could be helpful.

[ematipico]

Updating the cache of a component could potentially have a drastic side effect on all its consumers (importers). At a point where many pages, even more than you might think, could get updated.

I’m not sure I understand the concern — that seems desirable behaviour to me? If a component changed, the page should be updated. Not re-rendering a page that contains components that changed is a bug surely?

Components that come from libraries. If they change due to lock file change, there might be changes that users don't want. I am not saying that it's not desirable, and not even that your proposal is sound, but I see possible drawbacks too.

[matthewp]

@adamchal yeah I don't love that either, have been thinking about it. HTTP is internal to how Cloudflare communicates, the Astro prerenderer API doesn't prescribe that. So perhaps internally the Cloudflare adapter can use multiple requests for a build instead of a single one, to bring back the other parts.

[adamchal]

@matthewp yeah, my thinking was:
Better trade-off: Paying for the many (local) HTTP IPC calls is usually cheaper than userland encoding + decoding.
Too Far: Cap'n Proto or a separate binary comm channel—in theory could be possible with workerd’s socket support (unqualified claim).

Are you trying to bite off the concurrency issue too? I haven’t dug in how deep that would cut, but I’m sure this IPC would be a side-effect or directly affected.

@sgalcheung great call out to use a the cheaper fs.stat calls. For cache invalidation though, we have to be more confident than a mtime and size consistency. The file hash cost is quite low compared to the IPC channel between Astro node generator + separate process adapters like Cloudflare.

[matthewp]

@delucis Outside of whether we should do it or not, from an implementation standpoint, getStaticData() would need to run in dev and the build like getStaticPaths does. That probably means crawling the graph on each request in dev to find modules with getStaticData. That sounds like a bad idea for performance.

[matthewp]

I have a branch with support for concurrency. I'll take the perf issues into consideration there.

[delucis]

getStaticData() would need to run in dev and the build like getStaticPaths does. That probably means crawling the graph on each request in dev to find modules with getStaticData. That sounds like a bad idea for performance.

Very likely that I’m missing some technical details of the render flow, but my thinking was that you’d use it the same way we use the render export currently (pseudo-code):

import Component, { getStaticData } from 'compiled-user-component';

const data = getStaticData();
Component({ result, props, slots, data });

with internal compiler stuff looking like:

const $$Index = $$createComponent(($$result, $$props, $$slots, $$data) => {
	const Astro = $$result.createAstro($$props, $$slots, $$data);
	Astro.self = $$Index;
	const { myData } = Astro.data;
	return $$render`${$$maybeRenderHead($$result)}<div>${myData}</div>`;
}, "example.astro", undefined);

I’d think everything is already in the graph so no change in performance? And certainly not worse than running data manipulation for all pages for each render in dev.

Dev also doesn’t need any cache handling I’d assume? So getStaticData() could just be a very thin wrapper as part of the render — its outputs would only be special cased for builds.

[matthewp]

@adamchal I've addressed the encoding issue you raised in this PR, along with enabling concurrency: withastro/astro#17795

[adamchal]

@matthewp ran a test of this against two real live repos—the performance and concurrency seem really good, but I haven’t studied the implementation.

[seansmyth]

I'm trying this out on a project and I'm coming across an issue that it's processing each page every build even if nothing's actually changed.

Claude seems to think it might be the astro-icon package - generating unique things every build but I'm not sure.

I've commented out any instances of and run two builds back to back and Claude seems to think that that's the issue.

`So to directly answer "why are they being listed again" — it's not that your pages are getting rebuilt unnecessarily because of something wrong in what we built. It's that Astro's dependencyHash check (the other half of canSkip(), checked before cacheKey even gets compared) is coming out different on every single build, even when the source is byte-identical. Per Astro's own source, that hash is built from the actual bundled/transformed code of every module in the page's render graph — Layout, every component it pulls in, their CSS — not from your files specifically. Something in that pipeline is producing non-deterministic output build-to-build. The most likely suspect given the log is astro-icon's Loaded icons from src/icons step, which re-scans and rebuilds the icon registry fresh every run — if that doesn't produce byte-identical output each time (e.g. non-deterministic ordering), it alone would poison the hash for every page that uses an icon.

This is a real limitation, but it sits one layer below anything we've touched — incrementalBuild is still explicitly an experimental Astro feature (that's literally in the comment we already have in astro.config.mjs), and we've now proven the CMS-timestamp cacheKey side of this is solid and stable. The dependencyHash instability is Astro/Vite/astro-icon's own build determinism, not something staticPaths.js or cacheKey.js can influence.

There is a way to actually pin down which specific module is unstable — I'd temporarily patch Astro's own hashModules() (in node_modules/astro/dist/core/build/plugins/plugin-incremental.js) to log a hash per module, run two builds back to back, and diff which module id changed. That's real node_modules surgery for a one-off diagnosis (reverts on your next npm install), and it's a genuine rabbit hole with no guarantee the culprit is fixable from userland even once found.`

`Confirmed — this nails it down completely. The cached output copies in node_modules/.astro/dist/case-studies/.html are still timestamped from Build 1 (15:12:5x), untouched by Build 2. The live dist/case-studies/.html files did update at Build 2's time (15:14:26), but that's just Astro's restoreOutputFile() copying the cached HTML across — not a fresh render. No CMS fetch happened for these 5 pages in Build 2. The skip actually worked.

So: astro-icon was indeed the source of the dependencyHash instability. With disabled, two back-to-back builds now produce identical dependency hashes and Astro correctly reuses the previous build's output.

Obviously we can't ship the site with every icon commented out — that was purely diagnostic. The real fix is figuring out what about astro-icon's output isn't deterministic. Let me check its config/version to see if there's a known cause:

Used 14 tools

Found the actual root cause, and it's outside our codebase entirely. Tracing further into @iconify/tools (a dependency of astro-icon), its scanDirectory() function (node_modules/@iconify/tools/lib/misc/scan.mjs) reads your icons folder with plain fs.promises.readdir() and pushes results into an array with no sort applied:`

[matthewp]

We should have something to let the user know why a page is a cache miss. Will have to think of a way to show this without it becoming too much information.

[florian-lefebvre]

We have some kind of internal framework based on Astro at Seibert and we would need this because it's used for dozens of sites with thousands of mdx pages. But we face the same limitation as Starlight where stuff could happen anywhere in the tree, so it's unfeasible to just set a cacheKey of some kind at the top