2GIS Almaty — coffee shop GIS analysis

Offline 2GIS place data for Almaty: ETL into PostgreSQL, coffee-shop filtering, maps, and competitor analysis.

Status

Area	Status	Notes
MSI download & update check	Working	Fetches latest Almaty archive from 2GIS download page
.dgdat parse → Parquet → PostgreSQL	Working	Native decoder, schema DDL, upsert load
twogis.places view	Working	Joins orgs, addresses, streets, cities
Coffee shop name filter	Working	Heuristic classifier in twogis/analytics/coffee_filter.py — no rubric/category in offline data
Lat/lon from 2GIS coords	Partial	Only branches with large-integer map_x/map_y decode to WGS-84 (~fraction of all places)
Address geocoding (Nominatim)	Partial	Fills gaps for map/EDA; rate-limited, imprecise for some streets, not all addresses resolve
Coffee location EDA notebook	Partial	Maps and rival-radius analysis work for geocoded subset; coverage limited by geodata gap above
Full DB refresh on update	TBD	New MSI versions detected; reload strategy not finalized

Geodata limitation

2GIS offline dumps store internal map coordinates (map_x, map_y) — not always present, and not WGS-84 lat/lon. When values exist in the large-integer format, the places view converts them:

• lon = map_x / 51_000_000
• lat = map_y / 100_000_000

Many branches have no usable coordinates. The notebook falls back to Nominatim (OpenStreetMap) street-level geocoding. That is slow (1 req/sec), caches results in data/geocode_cache.json, and is imperfect — building numbers and Kazakh street names are not always matched correctly. Maps and distance-based rival queries only cover places with coords (from either source).

---

Quick start

git clone https://github.com/DocMorg/caffe_gis_project.git
cd caffe_gis_project
pip install -r requirements.txt
cp .env.example .env   # fill in DATABASE_URL and NOMINATIM_USER_AGENT

Download the Almaty MSI manually from the 2GIS site into data/msi/, or:

python main.py check-update --download
python main.py dgdat-etl

Coffee shop analysis:

jupyter notebook coffee_location_eda.ipynb

Sample SQL queries: EDA.sql.

---

How it works

2GIS MSI (Almaty)  →  extract .dgdat  →  parse binary  →  Parquet  →  PostgreSQL (twogis schema)
                                                                              ↓
                                                         places view (lat/lon when map_x/y decode)
                                                                              ↓
                                              coffee filter (name heuristics) + optional Nominatim
                                                                              ↓
                                              Folium maps, rival-radius analysis (notebook + EDA.sql)

1. Update check — twogis/updates/checker.py scrapes the 2GIS download page for a newer 2GISData_Almaty-*.msi.
2. Parse — twogis/dgdat/binary.py decodes the proprietary .dgdat format; parquet.py exports relational tables.
3. Load — twogis/db/loader.py applies DDL from ddl/ and upserts into schema twogis.
4. Places view — ddl/places_view.sql joins branches to org names and addresses; exposes lat/lon when internal coords decode.
5. Analytics — twogis/analytics/coffee_filter.py classifies coffee competitors from org names; the notebook geocodes missing addresses and builds maps.

After a successful parse, the MSI in data/msi/ is deleted. After DB load, data/gisdata/ and data/buffer_parquet/ are cleaned up.

---

Environment (.env)

Copy .env.example to .env (gitignored). Required and optional variables:

Variable	Required	Description
DATABASE_URL	Yes	PostgreSQL connection string (e.g. Neon)
NOMINATIM_USER_AGENT	Yes (for notebook)	App name + contact email per OSM usage policy
DB_SCHEMA	No	Postgres schema name (default: twogis)
GEOCODE_MAX	No	Max non-priority addresses to geocode per notebook run (default: 120)
PGHOST, PGPORT, PGDATABASE, PGUSER, PGPASSWORD, PGSSLMODE	No	Decomposed DB vars for GUI clients / psql

---

Project layout

main.py                 # CLI entry point
twogis/
  config.py             # paths, schema, .env loading
  dgdat/
    binary.py           # native .dgdat decoder
    msi.py              # MSI extract, path resolution
    parquet.py          # Parquet export
    pipeline.py         # offline ETL orchestration
  db/
    loader.py           # PostgreSQL DDL + upsert
  analytics/
    coffee_filter.py    # coffee-shop name classifier
  updates/
    checker.py          # check/download new Almaty DB from 2GIS site
ddl/                    # SQL table definitions + places view
EDA.sql                 # sample analytics queries
coffee_location_eda.ipynb
data/
  msi/                  # downloaded MSI installers
  gisdata/              # extracted .dgdat files (gitignored)
  buffer_parquet/       # ETL intermediate (gitignored)
  geocode_cache.json    # Nominatim cache (gitignored)

---

CLI commands

# Check for a newer 2GIS Almaty database
python main.py check-update
python main.py check-update --download   # download + extract if newer

# Full offline ETL (MSI → Parquet → PostgreSQL)
python main.py dgdat-etl
python main.py dgdat-etl --skip-db       # Parquet only

# Parse .dgdat to Parquet only
python main.py parse

---

Database

Schema: twogis (configurable via DB_SCHEMA)

Tables / views	Description
organizations, cities, streets, address_elements, fil_org_links, fil_address_links	Offline archive data
places (view)	Joined places with lat/lon when internal coords decode