development.md

Local development, testing, and known gaps

Setup

python3 -m venv .venv
.venv/bin/pip install -e ../dgp -e .          # local core library + this package
.venv/bin/pip install aiopg 'sqlalchemy<2'    # aiopg isn't declared; see publishing.md for why <2

The server needs a Postgres, because init_pg opens an aiopg engine at startup. A throwaway cluster is enough:

initdb -D /tmp/pg -U postgres -A trust
pg_ctl -D /tmp/pg -o "-p 5499 -k '' -h 127.0.0.1" -l /tmp/pg.log start
# (-k '' avoids macOS's 103-byte limit on Unix socket paths when the data dir path is long)

Running the standalone server

The server loads taxonomies/index.yaml relative to the working directory, and writes the .cache/ preview cache there too. So run it from a directory that has a taxonomies/ folder, such as dgp-app or datacity-importer, or a test fixture of your own:

cd /path/with/taxonomies
BASE_PATH=/tmp/dgp DATABASE_URL=postgresql://postgres@127.0.0.1:5499/postgres \
  /path/to/dgp-server/.venv/bin/python -m dgp_server.server       # → http://127.0.0.1:8000/api

uid=$(curl -s -XPOST localhost:8000/api/config -H 'content-type: application/json' \
        -d '{"source": {"path": "data.csv"}, "taxonomy": {"id": "my-taxonomy"}}' | jq -r .uid)
curl -N localhost:8000/api/events/$uid      # the SSE stream; ends with "data: close"

• BASE_PATH/<uid>/config.yaml shows the analyzed config after the run.
• Server logs are quiet unless you raise the level of the dgp-server and dgp loggers. dgp-app does this in server_dgp.py.

To run the real dgp-app subclass instead, run python server_dgp.py from dgp-app, which serves on port 5001. It needs DATASETS_DATABASE_URL, plus an ETLS_DATABASE_URL that has dgp-app's etl_pipeline and etl_taxonomies tables.

Exercising the pipeline without the server

Most behaviour lives in the core library and can be driven in-process. This is the same loop as DgpServer.events:

from dgp.core import Config, Context
from dgp.genera import SimpleDGP, LoaderDGP, PostLoaderDGP, TransformDGP, EnricherDGP
from dgp.taxonomies import TaxonomyRegistry
from dataflows import Flow

cfg = Config(None, data=my_dgp_config)                 # filename=None → nothing written to disk
ctx = Context(cfg, TaxonomyRegistry('taxonomies/index.yaml'))
for _ in range(5):
    dgp = SimpleDGP(cfg, ctx, steps=[LoaderDGP, PostLoaderDGP, TransformDGP, EnricherDGP])
    ok = dgp.analyze()
    if not cfg.dirty:
        break
print(ok, [str(e) for e in dgp.errors], cfg._unflatten()['model'])
rows, dp, _ = Flow(dgp.flow()).results()

To test publishing, set cfg['publish.allowed'] = True, re-run the analysis, then run Flow(dgp.flow(), publish_flow(cfg, create_engine(url), 'append')).process(). If a taxonomy module creates its DB engine at import time (datacity does), set its env vars before you construct the TaxonomyRegistry.

Tests and lint

• There are no tests. pytest.ini and tox.ini point at tests/, which doesn't exist, so make test (tox, py37) has nothing to run. The core dgp repo has only one sample file, dgp/genera/test_sample.py.
• make lint runs pylama dgp_server (pyflakes, mccabe and pep8; max line length 120). The develop extra installs pylama and tox.
• To verify a change, use the recipes above: the curl/SSE run for protocol changes, the in-process loop for config and analysis changes, and SQLite or a scratch Postgres for publishing.

Known issues and gaps

I found all of these in the code; the ones marked (verified) I reproduced.

Protocol and server:

1. The flow runs synchronously inside the aiohttp handler and blocks that worker's event loop. Production uses 4 gunicorn workers to cope.
2. Config starts out dirty, so there are always at least two analysis phases and a t: c message. The dirty flag reflects only the most recent set(). (verified)
3. Requesting /events/<uid> for an unknown uid returns only close, with no error message.
4. ConfigHeaderMappings and the core map guesser mutate the shared taxonomy registry (header_mapping) on every request, and never remove entries.
5. The preview cache (.cache/<hash>, relative to the working directory) is never invalidated when the data behind an unchanged URL changes.
6. /configs and configurations.py's fallback create_all read os.environ['DATABASE_URL'], which isn't set in dgp-app. config_storer.ConfigStorerDGP is unused.
7. log.py attaches a handler but sets no level, so INFO logs are dropped unless the host app sets one.

Configuration and mapping (core dgp):

8. The normalize form of a header mapping is never applied by the guesser; you have to spell it out in model.mapping. (verified)
9. constants has to be a list of pairs. A dict is flattened away silently. (verified)
10. structure.skip_cols is detected but never applied.
11. source.headers conflicts with the loader's own headers argument.
12. A taxonomy's own loading_module / preloading_module is ignored; only _common_'s is used.
13. header_mapping accepts only a single file.
14. taxonomy.columnTypes is a per-pipeline snapshot, refreshed only in preview. A config that is published without a preview has no column types. (verified)

Publishing:

15. On SQLAlchemy 2.x the clear_by_source DELETE is rolled back. (verified)
16. _source isn't in the primary key, and there's one table per taxonomy, so sources whose keys overlap collide. (verified for muni_budgets)
17. source_field_name doesn't rename the column that gets added.
18. The index creation is retried, and logged as an error, on every run.
19. dgp-app's FileLoaderDGP rewrites source.path to a cache path that includes a timestamp, which makes _source unstable for re-uploaded files.

Packaging:

20. aiopg isn't declared as a dependency.
21. The Dockerfile is stale (dgp_server==0.0.61, Python 3.7).
22. The classifiers and tox target Python 3.7, while production runs 3.12.
23. Git tags stop at 0.0.61.