Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add baking script for archival bakes #4502

Merged
merged 27 commits into from
Feb 11, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
d97fecd
wip: add (very basic) bake script for archival
marcelgerber Jan 23, 2025
c498e9f
feat(bake): allow passing in an assetMap
marcelgerber Jan 28, 2025
a536689
enhance: pass assetMap all the way down
marcelgerber Jan 28, 2025
e403c34
enhance: actually hash & copy over assets
marcelgerber Jan 28, 2025
c3162c9
enhance: bake dods & variable files
marcelgerber Jan 28, 2025
9eaebd9
enhance: resolve runtime assets, too, and change to absolute paths
marcelgerber Jan 29, 2025
67697df
fix: use correct assets in preload directives
marcelgerber Jan 29, 2025
f24dd90
enhance: better asset path construction
marcelgerber Jan 29, 2025
00aca45
enhance: use base36 hashes
marcelgerber Jan 29, 2025
ac540ce
enhance: create a manifest alongside archived grapher page
marcelgerber Jan 29, 2025
f105b7e
enhance: put grapher page into date-stamped dir
marcelgerber Jan 29, 2025
f4e5a9d
refactor: rename `viteAssetMap` -> `staticAssetMap`
marcelgerber Feb 4, 2025
346bc58
enhance: copy over static files from `public/` folder
marcelgerber Feb 4, 2025
f2444b3
refactor: rename `AssetMapEntry` -> `AssetMap`
marcelgerber Feb 4, 2025
b677f39
refactor: pull out hashing
marcelgerber Feb 4, 2025
7cc3235
enhance: build some safety around assetMap reads
marcelgerber Feb 4, 2025
d79497c
enhance: allow copying to `latest` dir
marcelgerber Feb 5, 2025
0568a34
enhance: make source maps work
marcelgerber Feb 5, 2025
1a031a7
feat: directly hash a stream
marcelgerber Feb 10, 2025
d163d7d
enhance: properly content-hash `owid.mjs` file
marcelgerber Feb 10, 2025
54e4561
enhance: report failures to Sentry
marcelgerber Feb 10, 2025
8adfb4b
enhance: better error handling
marcelgerber Feb 10, 2025
90daaf7
refactor: make `runtimeAssetMap` available under `window._OWID_RUNTIM…
marcelgerber Feb 11, 2025
141d073
enhance: also handle old-style grapher pages
marcelgerber Feb 11, 2025
2a7d42c
refactor: only set `_OWID_RUNTIME_ASSET_MAP` if present
marcelgerber Feb 11, 2025
9ed4b8e
style: remove unnecessary import
marcelgerber Feb 11, 2025
0a69e23
fix: import Sentry instrumentation
marcelgerber Feb 11, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ dist/
**/tsup.config.bundled*.mjs
cfstorage/
vite.*.mjs
archive/*

# Sentry Config File
.env.sentry-build-plugin
87 changes: 77 additions & 10 deletions baker/GrapherBaker.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ import {
DbPlainChart,
DbRawChartConfig,
DbEnrichedImage,
AssetMap,
} from "@ourworldindata/types"
import ProgressBar from "progress"
import {
Expand All @@ -58,7 +59,15 @@ const renderDatapageIfApplicable = async (
grapher: GrapherInterface,
isPreviewing: boolean,
knex: db.KnexReadonlyTransaction,
imageMetadataDictionary?: Record<string, DbEnrichedImage>
{
imageMetadataDictionary,
staticAssetMap,
runtimeAssetMap,
}: {
imageMetadataDictionary?: Record<string, DbEnrichedImage>
staticAssetMap?: AssetMap
runtimeAssetMap?: AssetMap
} = {}
) => {
const variable = await getVariableOfDatapageIfApplicable(grapher)

Expand All @@ -81,6 +90,8 @@ const renderDatapageIfApplicable = async (
useIndicatorGrapherConfigs: false,
pageGrapher: grapher,
imageMetadataDictionary,
staticAssetMap,
runtimeAssetMap,
},
knex
)
Expand All @@ -94,16 +105,23 @@ const renderDatapageIfApplicable = async (
export const renderDataPageOrGrapherPage = async (
grapher: GrapherInterface,
knex: db.KnexReadonlyTransaction,
imageMetadataDictionary?: Record<string, DbEnrichedImage>
{
imageMetadataDictionary,
staticAssetMap,
runtimeAssetMap,
}: {
imageMetadataDictionary?: Record<string, DbEnrichedImage>
staticAssetMap?: AssetMap
runtimeAssetMap?: AssetMap
} = {}
): Promise<string> => {
const datapage = await renderDatapageIfApplicable(
grapher,
false,
knex,
imageMetadataDictionary
)
const datapage = await renderDatapageIfApplicable(grapher, false, knex, {
imageMetadataDictionary,
staticAssetMap,
runtimeAssetMap,
})
if (datapage) return datapage
return renderGrapherPage(grapher, knex)
return renderGrapherPage(grapher, knex, { staticAssetMap, runtimeAssetMap })
}

export async function renderDataPageV2(
Expand All @@ -114,13 +132,17 @@ export async function renderDataPageV2(
useIndicatorGrapherConfigs,
pageGrapher,
imageMetadataDictionary = {},
staticAssetMap,
runtimeAssetMap,
}: {
variableId: number
variableMetadata: OwidVariableWithSource
isPreviewing: boolean
useIndicatorGrapherConfigs: boolean
pageGrapher?: GrapherInterface
imageMetadataDictionary?: Record<string, ImageMetadata>
staticAssetMap?: AssetMap
runtimeAssetMap?: AssetMap
},
knex: db.KnexReadonlyTransaction
) {
Expand Down Expand Up @@ -217,6 +239,8 @@ export async function renderDataPageV2(
imageMetadata={imageMetadata}
faqEntries={faqEntries}
tagToSlugMap={tagToSlugMap}
staticAssetMap={staticAssetMap}
runtimeAssetMap={runtimeAssetMap}
/>
)
}
Expand All @@ -237,7 +261,14 @@ export const renderPreviewDataPageOrGrapherPage = async (

const renderGrapherPage = async (
grapher: GrapherInterface,
knex: db.KnexReadonlyTransaction
knex: db.KnexReadonlyTransaction,
{
staticAssetMap,
runtimeAssetMap,
}: {
staticAssetMap?: AssetMap
runtimeAssetMap?: AssetMap
} = {}
) => {
const postSlug = urlToSlug(grapher.originUrl || "") as string | undefined
// TODO: update this to use gdocs posts
Expand All @@ -258,10 +289,46 @@ const renderGrapherPage = async (
relatedArticles={relatedArticles}
baseUrl={BAKED_BASE_URL}
baseGrapherUrl={BAKED_GRAPHER_URL}
staticAssetMap={staticAssetMap}
runtimeAssetMap={runtimeAssetMap}
/>
)
}

export const bakeSingleGrapherPageForArchival = async (
bakedSiteDir: string,
grapher: GrapherInterface,
knex: db.KnexReadonlyTransaction,
{
imageMetadataDictionary,
staticAssetMap,
runtimeAssetMap,
}: {
imageMetadataDictionary?: Record<string, DbEnrichedImage>
staticAssetMap?: AssetMap
runtimeAssetMap?: AssetMap
} = {}
) => {
const outPathHtml = `${bakedSiteDir}/grapher/${grapher.slug}.html`
await fs.writeFile(
outPathHtml,
await renderDataPageOrGrapherPage(grapher, knex, {
imageMetadataDictionary,
staticAssetMap,
runtimeAssetMap,
})
)
const outPathManifest = `${bakedSiteDir}/grapher/${grapher.slug}.manifest.json`

// TODO: right now, this only contains the asset maps. it may in the future also contain input
// hashes of the config and data files.
await fs.writeFile(
outPathManifest,
JSON.stringify({ staticAssetMap, runtimeAssetMap }, undefined, 2)
)
console.log(outPathHtml, outPathManifest)
}

const bakeGrapherPage = async (
bakedSiteDir: string,
imageMetadataDictionary: Record<string, DbEnrichedImage>,
Expand Down
Loading
Loading