Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

7x docs and stability improvements #365

Merged
merged 138 commits into from
Dec 6, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
138 commits
Select commit Hold shift + click to select a range
6738025
doc updates
twelch Oct 18, 2024
f12a02f
more updates
twelch Oct 18, 2024
45f0a7e
add prepublish step, document use of workers, how to match tests, pub…
twelch Oct 18, 2024
684b1e4
break out accuracy limitations doc
twelch Oct 18, 2024
707c4d9
deprecate fgbFetchAll for loadFgb, add toolbox doc, drop deprecated o…
twelch Oct 22, 2024
edfc1b5
Break out tutorials and more
twelch Oct 22, 2024
eae4654
update preprocessing and geoprocessing docs
twelch Oct 23, 2024
74dacf0
genPreprocessor: add minSize and enforceMinSize options, more doc add…
twelch Oct 23, 2024
2c24b78
update reportclient doc
twelch Oct 24, 2024
ff93e95
add projectclient doc, fix links, other updates
twelch Oct 29, 2024
7767a26
add vite babel plugins for storybook, migrate away from toNullSketch …
twelch Oct 29, 2024
1e26f20
remove null sketch from template smoke test output
twelch Oct 29, 2024
9d52999
add toChildProperties helper, major cleanup 6.1 tutorial
twelch Oct 30, 2024
068cd18
update 7x tutorials, couple backports to 6.1, add devcontainer guide
twelch Oct 31, 2024
6b7ef4b
drop global datasources from basic-project template. add codespaces …
twelch Nov 1, 2024
9d0d299
update smoke test output
twelch Nov 1, 2024
4d9f10f
refactor genRandomFeature script to support sketches too, drop genRan…
twelch Nov 1, 2024
6c7f5e5
update genRandomFeature, add third part data doc, break out link data…
twelch Nov 2, 2024
2e267e4
further newproject, add precalc, third party and data guides
twelch Nov 2, 2024
b080983
rework system setup sections, fix broken links
twelch Nov 5, 2024
8edee9c
update windows section
twelch Nov 5, 2024
88b9118
front page and tutorial updates
twelch Nov 5, 2024
aed0322
windows clarity
twelch Nov 5, 2024
905c920
newproject start refine
twelch Nov 6, 2024
b0653a2
drop global datasources other than boundaries, including from simpleF…
twelch Nov 6, 2024
b08d879
Deprecate polygonPreprocessorSmokeTest, create polygonSmokeTest funct…
twelch Nov 6, 2024
e1d98d3
newproject updates
twelch Nov 6, 2024
ca29b6b
genRandomFeature: fix outdir to vary whether sketch or feature type o…
twelch Nov 6, 2024
5c45236
drop bbox and planning init questions and basic properties remove tem…
twelch Nov 6, 2024
f41562d
drop template question from init, update tutorial method for getting …
twelch Nov 6, 2024
339d4e9
rename genRandomFeature script to genRandomPolygon
twelch Nov 6, 2024
0626c22
remove old genRandom scripts on upgrade
twelch Nov 6, 2024
2a80ad8
strip down simpleFunction and blankFunction, add to newproject
twelch Nov 7, 2024
995ef81
tune simpleFunction and blankFunction, requireProperties not required…
twelch Nov 7, 2024
37bf1b6
Fix genRandomPolygon name bug, storybook no open browser, remove defa…
twelch Nov 7, 2024
7ca7a36
migrate to loadFgb internally
twelch Nov 11, 2024
67ab4d8
drop deprecated loadCogWindow and bboxToPixelEdge
twelch Nov 11, 2024
6a750b8
include dataproviders as top-level export
twelch Nov 11, 2024
72a2b70
switch newProject to sampleProject, and add start of sampleProject, d…
twelch Nov 11, 2024
48d4c61
update newproject, sampleproject, regen api docs with dataproviders c…
twelch Nov 11, 2024
f1bddf9
fix broken links, refine sampleproject layout
twelch Nov 11, 2024
3d4ba98
dataImport: conditional ask additional properties question
twelch Nov 11, 2024
2116349
tune dataImport
twelch Nov 11, 2024
32262ae
tutorial bump gp version, sort init language alphabetical
twelch Nov 11, 2024
302af6e
unregister blankFunction in blank project default config, add SimpleR…
twelch Nov 11, 2024
ce7e614
breaking: deprecate genClipLoader, remove genPreprocessor and clipToP…
twelch Nov 12, 2024
fbf82af
code comments
twelch Nov 12, 2024
3d92d79
nit
twelch Nov 12, 2024
f2cbab1
update example data link
twelch Nov 12, 2024
79a93f6
add validatePolygon preprocessor, with ensureValidPolygon helper, typ…
twelch Nov 12, 2024
ab91f1d
clean up polygonSmokeTest doc
twelch Nov 12, 2024
b6869c5
sampleproject updates
twelch Nov 12, 2024
414a957
move preprocessor test features to base, remove test output, so that …
twelch Nov 12, 2024
c18221b
add clipToOceanEez to basic project, add unit tests for preprocessors…
twelch Nov 15, 2024
420f4d1
preprocessing docs. other doc updates on rebuild
twelch Nov 15, 2024
18344bb
drop use of dataproviders module, switch remaining fgbFetchAll to loa…
twelch Nov 16, 2024
9a6e020
update newproject
twelch Nov 16, 2024
6966f10
switch back to separate dataproviders module to isolate node-fetch us…
twelch Nov 16, 2024
ff15cf6
rollback flatgeobuf to 3.33.0 for now
twelch Nov 16, 2024
f18945d
migrate last dataprovider use
twelch Nov 16, 2024
74b09b5
upgrade flatgeobuf, remove unused webstreams polyfill, drop node-fetc…
twelch Nov 16, 2024
5585f11
add vitest-fetch-mock dep to base project
twelch Nov 16, 2024
6d78258
add back dataproviders module to top-level export and migrate use
twelch Nov 16, 2024
4bcc35a
Fix SimpleCard error and exports, add roundDecimalFormat helper funct…
twelch Nov 17, 2024
823e634
Fix i18nAsync bug using base english strings instead of letting compo…
twelch Nov 17, 2024
e7a67cb
update base english terms
twelch Nov 17, 2024
181b950
update sampleproject, rebuild api docs
twelch Nov 17, 2024
a9eef6c
simplify sketchOrCollection variable in SimpleCard and add translatio…
twelch Nov 18, 2024
d928b4d
update newproject
twelch Nov 18, 2024
61881c6
bump next version
twelch Nov 18, 2024
5ccc733
newproject nit
twelch Nov 18, 2024
e74d5d7
simplify SimpleCard, cleanup BlankCard, update translation
twelch Nov 18, 2024
2e46054
bump gp
twelch Nov 18, 2024
e1155b0
move upgrade doc to basic tutorials, add gp-workspace upgrade instruc…
twelch Nov 19, 2024
4849773
remove .story-cache folders from dist build, update template-ocean-ee…
twelch Nov 19, 2024
828c7ca
move upgrade doc to basic tutorials, add gp-workspace upgrade instruc…
twelch Nov 19, 2024
7109136
remove .story-cache folders from dist build, update template-ocean-ee…
twelch Nov 19, 2024
881c7fe
move blank functions over to templace-ocean-eez, out of temlate-blank…
twelch Nov 19, 2024
6185307
add back simplefunction smoke test output
twelch Nov 19, 2024
7556da8
update story-cache to copy from eez template
twelch Nov 19, 2024
0202cad
Merge branch '7x-docs' of https://github.com/seasketch/geoprocessing …
twelch Nov 19, 2024
cbc16fa
template functions cleanup
twelch Nov 19, 2024
89cb4e3
switch vectorFunction to use higher level getFeatures instead of load…
twelch Nov 19, 2024
c8c122b
genReportStories - replace spaces in sketch name with underscore
twelch Nov 19, 2024
110297d
better link concepts
twelch Nov 19, 2024
5811610
improve transform of sketch name to valid variable name
twelch Nov 20, 2024
149414e
add sketch prefix to report story variable name
twelch Nov 20, 2024
d79aa33
fix sketch variable name in genReportStories
twelch Nov 20, 2024
d1a5e43
Add Card to createReport component name plus other output polish, dro…
twelch Nov 20, 2024
c892c6d
Add user attributes to generated sample sketches, nit createReport ou…
twelch Nov 20, 2024
6f25e54
add user attributes to genRandomPolygons
twelch Nov 20, 2024
0df3131
update genRandomPolygon notes, add simple-card-view image
twelch Nov 21, 2024
cf8289a
genRandomPolygon: start ids at 1
twelch Nov 22, 2024
5281141
Add simple report modifications
twelch Nov 22, 2024
0ca77f4
add getFgbPath method to projectClientBase, update createFunction out…
twelch Nov 22, 2024
e3422fb
Fix getFgbPath
twelch Nov 22, 2024
a7e99bc
Add getFeaturesForSketchBboxes helper, clarify MetricGroup type prope…
twelch Nov 22, 2024
f101820
fix export
twelch Nov 22, 2024
8a3ebaa
Make MetricGroup type property optional, rebuild docs
twelch Nov 22, 2024
04477e0
doc updates
twelch Nov 23, 2024
7dec397
update getFeaturesForBboxes with split feature, add splitBBoxAntimeri…
twelch Nov 26, 2024
3b9403a
antimeridian code example
twelch Nov 26, 2024
b3610d7
doc updates
twelch Nov 26, 2024
7710599
writeResultOutput, switch to ensureDirSync, flatgeobuf test cleanup
twelch Nov 26, 2024
6650253
createFunction - default to async, createClient - look in ocean eez t…
twelch Nov 26, 2024
a2a4dd9
undeprecate clip function, createClient - fix story paths, sampleProj…
twelch Nov 26, 2024
60d721e
export intersectInChunks, intersectInChunksArea, intersectSum. Move …
twelch Nov 27, 2024
be6333b
bump gp in docs
twelch Nov 27, 2024
5516e93
update toolbox
twelch Nov 27, 2024
4e26825
shape up toolbox doc, shift subdividing into data import doc
twelch Nov 28, 2024
b5fbef3
shuffle couple functions, toolbox doc
twelch Nov 28, 2024
7812e47
projectClient - add getClassDatasource helper function, drop requirem…
twelch Nov 28, 2024
8a1dbf1
getFeaturesForSketchBBoxes - create options argument
twelch Nov 28, 2024
14a0340
benthic report
twelch Nov 28, 2024
d7e62ea
createReport: switch vector function to use getFeaturesForSketchBBoxe…
twelch Nov 28, 2024
f31afe7
projectClient: add getClassKey helper
twelch Nov 29, 2024
550bf57
bump doc gp version
twelch Nov 29, 2024
7c74015
bug - don't use datasource classKey in vectorFunction
twelch Nov 29, 2024
f744778
vectorFunction fix bug in last update, rasterFunction migrate to getC…
twelch Nov 29, 2024
7c0e21f
MetricGroup schema - ensure at least one class item present
twelch Nov 29, 2024
00bdf3f
create:report clarify raster stat options, start octocoral report
twelch Nov 29, 2024
a1f34a8
metricGroupSchema: require datasourceId and classKey at metric group …
twelch Nov 29, 2024
0b1f594
rename getClassDatasouce to getMetricGroupDatasource, rename getClass…
twelch Dec 2, 2024
a18a34f
remove default objective
twelch Dec 2, 2024
34877d7
sampleproject update
twelch Dec 3, 2024
d44ac4e
improve precalc logging, use loadCog directly,
twelch Dec 3, 2024
7781b02
Consolidate report pages into TabReport, drop ViabilityPage
twelch Dec 3, 2024
1286ae8
update sampleProject
twelch Dec 3, 2024
560d76f
doc rebuild
twelch Dec 3, 2024
65d6dd1
projectClientBase - fix getMetricGroupDatasource and getMetricGroupCl…
twelch Dec 3, 2024
5c7cb4e
merge advanced concepts into concepts page, update CLI doc, update ar…
twelch Dec 3, 2024
d71ed53
update sampleProject, simplify getting started docs
twelch Dec 4, 2024
b87c710
update sampleproject, add preprocessing images, bump to 7x-docs.128
twelch Dec 5, 2024
631641e
broken link fixes
twelch Dec 5, 2024
05706f8
polish sample tutorial
twelch Dec 6, 2024
08ce389
sampleproject polish
twelch Dec 6, 2024
5ff22bd
refactor newproject, remove reportclient doc
twelch Dec 6, 2024
a1a4ea3
rebuild docs
twelch Dec 6, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
16 changes: 15 additions & 1 deletion .vscode/launch.json
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,21 @@
"cwd": "${workspaceFolder}/packages/template-ocean-eez",
"runtimeArgs": [
"--inspect-brk",
"${workspaceRoot}/packages/template-ocean-eez/node_modules/.bin/geoprocessing",
"${workspaceRoot}/node_modules/.bin/geoprocessing",
"test"
],
"console": "integratedTerminal",
"internalConsoleOptions": "neverOpen",
"port": 9229
},
{
"name": "template-blank-project - debug tests",
"type": "node",
"request": "launch",
"cwd": "${workspaceFolder}/packages/template-blank-project",
"runtimeArgs": [
"--inspect-brk",
"${workspaceRoot}/node_modules/.bin/geoprocessing",
"test"
],
"console": "integratedTerminal",
Expand Down
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
# Geoprocessing

All-in-one solution for developing low-cost and low-maintenance geoprocessing functions and reports for the web, with Typescript. Simplified publish to cloud with auto-scaling to meet high demand.
All-in-one solution for developing low-cost and low-maintenance geoprocessing functions and reports for SeaSketch. Simplified publish to cloud with auto-scaling to meet high demand.

This framework is part of the [SeaSketch](https://seasketch.org) ecosystem.

- [Documentation](https://github.com/seasketch/geoprocessing/wiki)
- [Changelog](https://github.com/seasketch/geoprocessing/blob/dev/CHANGELOG.md)
- [Documentation](https://seasketch.github.io/geoprocessing)
- [Releases](https://github.com/seasketch/geoprocessing/releases)

## Who is this framework for?

This framework is primarily designed for people that want to create and host their own geoprocessing functions and reports and plug them into their SeaSketch project, though it is not dependent on SeaSketch. It is used for all SeaSketch [projects](https://github.com/seasketch/geoprocessing/network/dependents?package_id=UGFja2FnZS0xMTc3OTQ1NDg5).
This framework is designed for people with coding experience that want to create and host their own geoprocessing functions and reports and plug them into their SeaSketch project. It is used for all SeaSketch [projects](https://github.com/seasketch/geoprocessing/network/dependents?package_id=UGFja2FnZS0xMTc3OTQ1NDg5).

## Goals

Expand Down
1 change: 1 addition & 0 deletions eslint.config.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ export default [
rules: {
"@typescript-eslint/no-explicit-any": "off",
"@typescript-eslint/no-unused-vars": "warn",
"@typescript-eslint/ban-ts-comment": "off",
"react/no-unescaped-entities": "off",
trailingComma: "off",
"unicorn/prevent-abbreviations": "off",
Expand Down
38 changes: 20 additions & 18 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 6 additions & 3 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,9 @@
"name": "root",
"private": true,
"workspaces": [
"packages/*"
"packages/geoprocessing",
"packages/base-project",
"packages/template-blank-project"
],
"license": "BSD-3-Clause",
"devDependencies": {
Expand Down Expand Up @@ -51,9 +53,10 @@
"publish:experimental:canary": "BRANCH_NAME=`git rev-parse --symbolic-full-name --abbrev-ref HEAD`; if [ $BRANCH_NAME != \"dev\" -a $BRANCH_NAME != \"main\" ] ; then CI=1 lerna publish --force-publish --canary --preid experimental-${BRANCH_NAME} --dist-tag=experimental-${BRANCH_NAME} ; fi",
"publish:alpha": "CI=1 lerna publish --force-publish --preid alpha --dist-tag=alpha",
"publish:beta": "CI=1 lerna publish --force-publish --preid beta --dist-tag=beta",
"publish:stable": "npm run test && CI=1 lerna publish",
"publish:backport": "npm run test && CI=1 lerna publish --preid backport --dist-tag=backport",
"publish:stable": "CI=1 lerna publish",
"publish:backport": "CI=1 lerna publish --preid backport --dist-tag=backport",
"prepare": "husky",
"prepublish": "npm test && lerna run --scope=@seasketch/geoprocessing prepare",
"publish:translation": "npx tsx packages/geoprocessing/src/i18n/bin/publishTerms.ts",
"import:translation": "npx tsx packages/geoprocessing/src/i18n/bin/importTerms.ts",
"sync:translation": "npm run extract:translationt && npm run publish:translation && npm run import:translation",
Expand Down
2 changes: 1 addition & 1 deletion packages/base-project/.prettierignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@
**/.svn
**/.hg
node_modules
examples/output
examples
9 changes: 6 additions & 3 deletions packages/base-project/.storybook/genReportStories.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,6 @@ import {
import { GpStoryConfig } from "./types.js";
import { v4 as uuid } from "uuid";

// Not currently used, project space responsible for its own storybook+vite setup

if (!process.env.PROJECT_PATH) {
throw new Error("PROJECT_PATH environment variable not set");
}
Expand Down Expand Up @@ -162,6 +160,11 @@ for (const storyConfig of storyConfigs) {
...(childProperties ? { childProperties } : {}),
};

// Convert sketch name to valid variable name, which is displayed as the story name
const sketchVariableName = sketch.properties.name
.replace(/^[^a-zA-Z$_\p{L}]/u, "_") // Replace invalid starting characters
.replaceAll(/[^a-zA-Z0-9$_\p{L}]/gu, "_"); // Replace invalid subsequent characters

const story = `
import React from "react";
import { ${storyConfig.componentName} } from '${importFromCacheStr}';
Expand All @@ -180,7 +183,7 @@ for (const storyConfig of storyConfigs) {
language: "en"
});

export const ${sketch.properties.name.replaceAll("-", "_")} = () => (
export const ${sketchVariableName} = () => (
<Translator>
<${storyConfig.componentName} />
</Translator>
Expand Down
86 changes: 58 additions & 28 deletions packages/base-project/.vscode/schemas.json
Original file line number Diff line number Diff line change
Expand Up @@ -53,37 +53,67 @@
],
"additionalProperties": false
},
"metricsSchema": {
"metricGroupSchema": {
"type": "array",
"items": {
"type": "object",
"properties": {
"metricId": { "type": "string" },
"type": { "type": "string" },
"datasourceId": { "type": "string" },
"classKey": { "type": "string" },
"classes": {
"type": "array",
"items": {
"type": "object",
"properties": {
"classId": { "type": "string" },
"classKey": { "type": "string" },
"datasourceId": { "type": "string" },
"display": { "type": "string" },
"numericClassId": { "type": "number" },
"layerId": { "type": "string" },
"objectiveId": { "type": "string" }
"oneOf": [
{
"type": "object",
"properties": {
"metricId": { "type": "string" },
"type": { "type": "string" },
"datasourceId": { "type": "string" },
"classKey": { "type": "string" },
"classes": {
"type": "array",
"items": {
"type": "object",
"properties": {
"classId": { "type": "string" },
"display": { "type": "string" },
"numericClassId": { "type": "number" },
"layerId": { "type": "string" },
"objectiveId": { "type": "string" }
},
"required": ["classId", "display"],
"additionalProperties": false
}
},
"required": ["classId", "display"],
"additionalProperties": false
}
"layerId": { "type": "string" },
"objectiveId": { "type": "string" }
},
"required": ["metricId", "classes", "datasourceId"],
"additionalProperties": false
},
"layerId": { "type": "string" },
"objectiveId": { "type": "string" }
},
"required": ["metricId", "type", "classes"],
"additionalProperties": false
{
"type": "object",
"properties": {
"metricId": { "type": "string" },
"type": { "type": "string" },
"classes": {
"type": "array",
"items": {
"type": "object",
"properties": {
"classId": { "type": "string" },
"classKey": { "type": "string" },
"datasourceId": { "type": "string" },
"display": { "type": "string" },
"numericClassId": { "type": "number" },
"layerId": { "type": "string" },
"objectiveId": { "type": "string" }
},
"required": ["classId", "display", "datasourceId"],
"additionalProperties": false
}
},
"layerId": { "type": "string" },
"objectiveId": { "type": "string" }
},
"required": ["metricId", "classes"],
"additionalProperties": false
}
]
}
},
"objectivesSchema": {
Expand Down Expand Up @@ -431,7 +461,7 @@
"oneOf": [
{ "$ref": "#/definitions/basicSchema" },
{ "$ref": "#/definitions/objectivesSchema" },
{ "$ref": "#/definitions/metricsSchema" },
{ "$ref": "#/definitions/metricGroupSchema" },
{ "$ref": "#/definitions/geographiesSchema" },
{ "$ref": "#/definitions/datasourcesSchema" }
]
Expand Down
4 changes: 1 addition & 3 deletions packages/base-project/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
# Getting Started

This is a geoprocessing project. It was created using the `geoprocessing init` command, as part of the tutorial to [create a new project](https://github.com/seasketch/geoprocessing/wiki/Tutorials#create-a-new-geoprocessing-project).

Once established, you can follow the tutorial to [setup an existing project on a new system](https://github.com/seasketch/geoprocessing/wiki/Tutorials#setup-an-exising-project-on-your-local-system)
This is a geoprocessing project created using the SeaSketch geoprocessing framework.
3 changes: 2 additions & 1 deletion packages/base-project/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
"reimport:data": "geoprocessing reimport:data",
"start:data": "http-server data/dist -c-1",
"start:client": "geoprocessing start:client",
"storybook": "PROJECT_PATH=$(pwd) npx tsx .storybook/genReportStories.ts && storybook dev -p 6006",
"storybook": "PROJECT_PATH=$(pwd) npx tsx .storybook/genReportStories.ts && storybook dev -p 6006 --no-open",
"sync:translation": "npm run extract:translation && npm run publish:translation && npm run import:translation",
"synth": "geoprocessing synth",
"test": "start-server-and-test start:data 8080 'npm run __test'",
Expand Down Expand Up @@ -72,6 +72,7 @@
"storybook": "^8.2.9",
"vite": "^5.4.2",
"vitest": "^2.0.5",
"vitest-fetch-mock": "^0.4.2",
"zx": "^8.1.5"
}
}
Loading
Loading