Simplify fiboa publish for catalog-driven publication

.gitignore

@@ -16,6 +16,7 @@ __pycache__/
 # Coverage
 .coverage
 .coverage.*
+converage.json
 coverage.xml
 htmlcov/
 
@@ -27,3 +28,5 @@ htmlcov/
 /tests/data-files/convert/*/extracted.*/
 /tests/data-files/convert/**/*.gfs
 *.tmp
+
+build/

CHANGELOG.md

@@ -6,16 +6,57 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
-
+- Put the easting in x whatever the source's CRS says: a source that honours EPSG's axis order ships its coordinates northing first (Jordbruksverket's shapefile declares it), and GeoParquet stores x, y — so the tiles, the STAC bbox and the bbox covering all read such a file as (lat, lon)
+- EE: mint a crop code — PRIA publishes the crop as free text and none of its own, and the crop extension requires `crop:code` and `crop:code_list` — and publish taotletud_maakasutus as `land_use` (arable, permanent grassland, restored grassland, permanent crops, black fallow) — the only classification the source has, since it publishes the crop as free text and no crop code at all; and keep pollu_id as `parcel_id`, because it repeats in a few rows of some editions (16 of the 165,244 in 2016); and a first test with a fixture
+- SE: eleven editions, 2015-2025 — the campaign is a filter on one WFS layer, so every year the service holds is a variant (it answers 2015 through 2025), over https because the http URL redirects
+- BG: the ministry's GeoServer publishes Agricultural_Land_<year> for 2021-2025, not the Arable_Land_2024 the converter asked for; the 2021 and 2022 layers are a different release again (block and usage in one ELGIDENT field, with an area column the later ones lack), PHBIDENT identifies the block rather than the polygon so it is published as block_id, and the Bulgarian names need UTF-8 forced because GeoServer writes the charset into a .cst file GDAL does not read
+- 
+- SK: nine editions, 2018-2026, from the per-campaign datasets on data.slovensko.sk (the single edition was labelled by its download year and held the 2024 file); the crop names are matched with their no-break spaces normalised, which alone maps 102,835 grassland fields
+- CZ: support the GPZ_DP schema of the 2019-2022 editions (renamed crop/area columns, no application date, ENTITA_ID is the block), recover the crop codes the 2020 edition leaves empty, map the 120 crop codes EuroCrops does not carry, and publish a declaration that straddles two land blocks once
+- Add DuckDB BaseConverter for efficiently transforming large datasets
+- Fix `use_variant_as_determination`: the determination:datetime column was dropped again because it was not listed in `columns` (affected DK, HR)
+- Declare the beautifulsoup4 dependency that the ES-PV and ES-VC converters import
+- EE: name the cached WFS responses (ee_gsaa_<year>.gml)
+- ES-MD: the archive no longer nests RECINTO.shp in a folder
+- SK: KODKD is the (non-unique, sometimes empty) LPIS block code, keep it as block_id and use the row index as id
+- ES-CM: always read the year-named SIGPAC service (the unnamed one moved on to 2025 with a different id field)
+- REST converters: do not keep an error response as a cached page
+- ES-CB: determination date from the variant year (was an empty string, which broke the STAC temporal extent)
+- ES-CAT: the 2024 download is a shapefile package, not a GeoPackage; 34 crop names new in 2024 added to the mapping
+- CZ: find the shapefile in nested archive folders (2026)
+- DE-BB: read the shapefile as cp1252 (its .cpg wrongly says UTF-8)
+- NL: new PDOK download location (rvo/gewaspercelen/atom), add the 2026 concept edition
+- DE-TH: note the INSPIRE download service
+- Drop rows without a crop:code (required by the crop extension) with a warning instead of failing the conversion (BE-VLG 2023, ES-CN had one such row each); more than 1% missing is an error
+- Europe-LAND converters: use crop_name as crop:code when the file's crop_code column is empty (LT 2024)
+- BE-VLG: derive determination:datetime from the variant year instead of a constant date
+- `fiboa publish` no longer uploads to S3 or generates README/LICENSE files. It creates GeoParquet, PMTiles and a STAC Collection with relative links, `file:size`/`file:checksum` and a web-map-links v1.3.0 `pmtiles` link. Publishing is done by catalogs such as the [harmonized field data catalog](https://github.com/fieldsoftheworld/harmonized-field-data-catalog).
+- IE: the feature id is the id, because localId is the LPIS parcel reference and repeats where a parcel is declared twice (214 of the 1,027,438 rows of 2022); it is published as `parcel_id`, and the parcels get a `metrics:area` computed from their geometry, which the GML does not carry
+- The nine converters that publish field blocks say so in their title: at_block, de_bb_block, de_mv, de_nds_block, de_nrw, de_sh, de_th, lu and nl_block read "Field blocks for ...", where five of them read "Field boundaries" like the crop-field collections
+
+- Declare the beautifulsoup4 dependency the ES-PV and ES-VC converters import
+- ES-CL: the ITACyL server is https-only, the 2025 shapefiles sit in province subfolders, and C_REFREC is the identifier
+- A test refuses a fixture above 5 MB, committed or merely lying in the fixture folder, because a failing convert test downloads the real source there
+- ES-MD: find RECINTO.shp wherever the archive puts it
+- REST converters: a variant may name its own service (es_ib keeps its yearly editions in a second one), every request is retried, pages included, and a joined layer's table prefixes are stripped for every converter
+- REST converters: page by id windows instead of server-side sorting, key cached pages by service as well as layer, discover the qualified key field on joined layers, and never keep an error response or a broken download as a cached page
+- ES-IB: editions 2022-2026 (the yearly layers moved to the GOIB_SIGPAC_HISTORIC_IB service), the provider is the Balearic government, not Navarra's, and the joined field names are handled by the REST mixin
+- Update vecorel-cli to v0.2.17:
+  - GeoJSON is read as UTF-8 as the format mandates, instead of the platform locale (cp1252 on Windows mangled umlauts)
+  - GeoJSON files with a byte order mark no longer fail to read
+  - Drop the per-converter UTF-8 workarounds in de_bw and de_he, now redundant
+- Converter for Spain (whole), based on the FEGA 2025+ data
 - Add Italy Tuscany (IT-1) basd on EuroCrops v2
 - Suuport multiple years for CZ
 - Multiple years for DE_sh
 - Multiple year support for HR
 - Introduce FiboaBaseConverter.use_variant_as_determination for setting proper determination_date
 - Update years for DK (2025, 2026)
+- Converter for Spain (whole), based on the FEGA 2025+ data
 - Update fr-converter to support 2021/2022 files
 - Converter for Bavaria, Germany LPIS field blocks (de_by_block)
 - Converter for Hesse, Germany LPIS reference parcels
+- PerFileBaseConverter: per-file migration and merging the result, decreasing memory requirements for large data sets
 - Update vecorel-cli to v0.2.16:
   - Converter output is sorted by Hilbert distance
   - Commands exit with a non-zero exit code when they report a failure
@@ -32,7 +73,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 - Fix the column additions of the determination fields in the AI4SF converter
 - Add HCAT to datasets where possible
 - Updated years & variants for at_crop, be_vlg, es_an, es_cl, es_pv, ie, pt, se
-- Extend create_stac, include include fiboa data
+- Extend create_stac, include fiboa data
 - Publish command; skip hidden files, generate better texts
 - Fix to vecorel: converter.license and provider should be string
 - Added a Dockerfile to simplify working with fiboa

README.md

@@ -62,7 +62,7 @@ fiboa CLI supports various commands to work with the files:
     - [Improve a fiboa Parquet file](#improve-a-fiboa-parquet-file)
     - [Update an extension template with new names](#update-an-extension-template-with-new-names)
     - [Converter for existing datasets](#converter-for-existing-datasets)
-    - [Publish datasets to source coop or your own s3 repository](#publish-datasets-to-source-coop-or-your-own-s3-repository)
+    - [Publish datasets](#publish-datasets)
   - [Development](#development)
     - [Implement a converter](#implement-a-converter)
   - [Run in Docker](#run-in-docker)
@@ -193,46 +193,43 @@ Use any of the IDs from the list to convert an existing dataset to fiboa:
 
 See [Implement a converter](#implement-a-converter) for details about how to
 
-### Publish datasets to source coop or your own s3 repository
+### Publish datasets
 
 `fiboa publish <dataset> -o <target>`
 
-The publish converts and publishes a fiboa dataset to source coop or your own s3 repository. The target directory 
-will be filled with the following files:
+Converts and validates a fiboa dataset and prepares everything that is needed to publish it
+in a (STAC-based) catalog. The target directory will be filled with the following files:
 
 ```
 <target>/
-  <dataset>.parquet
-  <dataset>.pmtiles      # requires working ogr2ogr and tippecanoe
-  stac/collection.json
-  README.md              # generated if --generate-meta/-gm flag is present
-  LICENSE.txt            # generated if --generate-meta/-gm flag is present
+  <dataset>[-<variant>].parquet
+  <dataset>[-<variant>].pmtiles   # requires working ogr2ogr and tippecanoe
+  collection.json                 # STAC Collection with relative links to the files above
 ```
 
-This directory is synchronized to the s3 repository (default source.coop/fiboa/data).
+The STAC Collection carries `file:size` and `file:checksum` for the files and a
+`pmtiles` link (web-map-links extension). Existing files in the target directory are reused,
+delete them to regenerate. Uploading to a bucket and catalog-specific metadata
+(README, styles, thumbnails, ...) are the job of the catalog that publishes the data, e.g. the
+[harmonized field data catalog](https://github.com/fieldsoftheworld/harmonized-field-data-catalog).
 
-**Requirements**: Requires the [aws CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) to be installed, 
-and `AWS_ACCESS_KEY_ID` with `AWS_SECRET_ACCESS_KEY` environment variables. Also, for generating the pmtiles file, 
-it requires [ogr2ogr](https://gdal.org/programs/ogr2ogr.html) and [tippecanoe](https://github.com/mapbox/tippecanoe).
+The command runs:
 
-The command executes the following steps:
-
-- `fiboa convert` to generate a fiboa parquet dataset. All convert parameters are passed to the converter.
-- `fiboa validate` to validate the fiboa dataset
-- creates a <dataset>.pmtiles from the parquet file. Uses ogr2ogr and tippecanoe
-- `fiboa create-stac-collection` to create a STAC collection
-- `fiboa publish` to publish the fiboa dataset to a source coop or your own s3 repository
+- `fiboa convert` to create a `<dataset>[-<variant>].parquet` file
+- `fiboa validate` to validate the GeoParquet file
+- `ogr2ogr | tippecanoe` to create the PMTiles file
+- `fiboa create-stac-collection` to create the STAC Collection
 
 Examples:
 
-- `fiboa publish at_crop -o data/at_crop`
-- `fiboa publish -c /tmp/cache -gm br_conab -o data/br_conab`
+- `fiboa publish at -o data/at`
+- `fiboa publish -c /tmp/cache nl --variant 2025 -o data/nl/2025`
 
 Relevant parameters:
 
-- `--generate-meta/-gm` Generatse the README.md and LICENSE.txt files if absent, based on data-survey and converter properties.
-- `--data-url` The URL to the data repository, used when generating the README
-- `--s3-upload-path` The `aws s3 sync` target. Defaults to `s3://source.coop/fiboa/data` . Uploading requires the `aws` CLI, and `AWS_ACCESS_KEY_ID` with `AWS_SECRET_ACCESS_KEY` environment variables.
+- `--variant` Choose the variant (e.g. year) of a dataset, defaults to the first variant.
+- `--no-pmtiles` Skip PMTiles generation.
+- `--tippecanoe-opts` Options passed to tippecanoe, defaults to `-zg --drop-densest-as-needed --extend-zooms-if-still-dropping`.
 
 Check `fiboa publish --help` for more details.