Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,6 @@ build

# Ignore DevSpace cache and log folder
.devspace/

# MkDocs build output
site/
13 changes: 13 additions & 0 deletions .readthedocs.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
version: 2

build:
os: ubuntu-24.04
tools:
python: "3.14"
jobs:
install:
# Read the Docs wraps each job in /bin/sh -c '...', so single quotes here break out of it.
- pip install "mkdocs>=1.6,<2" "mkdocs-material>=9.5" "pygments>=2.18" "pymdown-extensions>=10.7"

mkdocs:
configuration: mkdocs.yml
4 changes: 2 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ Please note we have a code of conduct, please follow it in all your interactions
## Pull Request Process

1. Ensure any install or build dependencies are removed before the end of the layer when doing a build.
2. Update the README.md with details of changes to the interface, this includes new environment variables, exposed ports, useful file locations and container parameters.
3. Increase the version numbers is managed by the GitHub Actions CI workflow `tagging.yml`
2. Update the pages under `docs/` with details of changes to the interface, this includes new command-line flags, exposed ports, useful file locations and container parameters.
3. CI tags each image built from `main` as `latest` and as `vYYYY.MM` for the year and month. A chart change needs a higher `version` in `charts/reverse-geocoding/Chart.yaml`, which the Charts workflow releases from `main`.
4. You may merge the Pull Request in once you have the sign-off of from the project team, or if you do not have permission to do that, you may request a reviewer to merge it for you.

## Benchmark
Expand Down
67 changes: 27 additions & 40 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,70 +1,57 @@
# Golang Reverse GeoCoding
# Reverse Geocoding

[![CI](https://github.com/saidsef/faas-reverse-geocoding/actions/workflows/ci.yml/badge.svg)](https://github.com/saidsef/faas-reverse-geocoding/actions/workflows/ci.yml)
[![Go Report Card](https://goreportcard.com/badge/github.com/saidsef/faas-reverse-geocoding)](https://goreportcard.com/report/github.com/saidsef/faas-reverse-geocoding)
![GitHub go.mod Go version](https://img.shields.io/github/go-mod/go-version/saidsef/faas-reverse-geocoding)
[![GoDoc](https://godoc.org/github.com/saidsef/faas-reverse-geocoding?status.svg)](https://pkg.go.dev/github.com/saidsef/faas-reverse-geocoding?tab=doc)
![GitHub release(latest by date)](https://img.shields.io/github/v/release/saidsef/faas-reverse-geocoding)
![Commits](https://img.shields.io/github/commits-since/saidsef/faas-reverse-geocoding/latest.svg)
![GitHub](https://img.shields.io/github/license/saidsef/faas-reverse-geocoding)

Reverse geocoding is used to find places or addresses near a latitude, longitude pair. Picture a map showing building outlines but no labels, then clicking on a building and being shown the name of the business. That is reverse geocoding.
Reverse geocoding turns a latitude and longitude into an address. This service takes the coordinates as JSON, asks OpenStreetMap's Nominatim or BigDataCloud for the address at that point, and returns the provider's answer. It keeps each answer in memory, for 60 minutes by default, so repeat lookups of the same place skip the provider.

This repository holds a small web service that performs reverse geocoding to determine whether specified geo coordinates has an address. If it is then the response will contain attributes associated with the matched address, municipality, county, country, etc.
- **Lookup** - takes `lat` and `lon` in a JSON body and returns the address record for that point
- **Cache** - answers repeat lookups that round to the same 3 decimal places from memory
- **Health** - answers `GET` on any path with `{"status": "healthy"}`
- **Metrics** - counts upstream requests per provider on a Prometheus endpoint

> By default, this will cache requests for 60 minutes to prevent repeated request(s) being sent again and again. The percision is limited to 3 decimal places.
## Quick start

## Geographic Coordinates

The number of decimal places in geographic coordinates determines the accuracy of the location. Each additional decimal place increases the precision of the coordinates:

| Decimal Places | Precision |
|:--------------:|:----------------:|
| 1 | ±10 kilometres |
| 2 | ±1 kilometre |
| 3 | ±100 metres |
| 4 | ±10 metres |
| 5 | ±1 metre |
| 6 | ±10 centimetres |

## Deployment

### Helm Chart

```shell
```sh
helm repo add geocode https://saidsef.github.io/faas-reverse-geocoding
helm repo update
helm upgrade --install geocode geocode/reverse-geocoding --namespace geocode --create-namespace
kubectl port-forward -n geocode svc/geocode 8080:8080
```

> *NOTE:* API can be accessed via port-forward `Service` or via Enabling `Ingress`

### Kustomization
From a second terminal:

```shell
kubectl apply -k deployment/
```sh
curl -d '{"lat":"41.40338","lon":"2.17403"}' http://localhost:8080/
```

> *NOTE:* API can be accessed via port-forward `Service` or via updating `Ingress`
## Documentation

Take it for a test drive:
[Read the Docs](https://faas-reverse-geocoding.readthedocs.io/en/latest/) hosts the same pages.

```shell
curl -d '{"lat":"41.40338","lon":"2.17403"}' http://localhost:8080/
```
| Page | Contents |
|------|----------|
| [Installation](https://github.com/saidsef/faas-reverse-geocoding/blob/main/docs/installation.md) | Requirements, the Helm chart, the Kustomize overlays, the container image, running locally, building from source, log output |
| [Configuration](https://github.com/saidsef/faas-reverse-geocoding/blob/main/docs/configuration.md) | Command-line flags, setting flags in Kubernetes, Helm values |
| [API](https://github.com/saidsef/faas-reverse-geocoding/blob/main/docs/api.md) | Endpoints, the request body, responses, errors, coordinate precision |
| [Architecture](https://github.com/saidsef/faas-reverse-geocoding/blob/main/docs/architecture.md) | What one request does, the cache, the upstream providers, timeouts, logging |
| [Metrics](https://github.com/saidsef/faas-reverse-geocoding/blob/main/docs/metrics.md) | Prometheus endpoint, metric names and labels, scrape setup |

```python
from requests import post
data='{"lat":"41.40338","lon":"2.17403"}'
r = post('http://localhost:8080/', data=data)
print(r.text)
```
## Requirements

The Helm chart needs Kubernetes 1.27 or later, and the image runs on `linux/amd64` nodes only. Building from source needs the Go version that `go.mod` sets.

## Source

Our latest and greatest source of *Reverse Geocoding* can be found on [GitHub]. [Fork us](https://github.com/saidsef/faas-reverse-geocoding/fork)!
Our latest and greatest source of *Reverse Geocoding* can be found on [GitHub](https://github.com/saidsef/faas-reverse-geocoding). [Fork us](https://github.com/saidsef/faas-reverse-geocoding/fork)!

## Contributing

We would :heart: you to contribute by making a [pull request](https://github.com/saidsef/faas-reverse-geocoding/pulls).

Please read the official [Contribution Guide](./CONTRIBUTING.md) for more information on how you can contribute.
Please read the official [Contribution Guide](https://github.com/saidsef/faas-reverse-geocoding/blob/main/CONTRIBUTING.md) for more information on how you can contribute.
11 changes: 11 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Documentation

Start with [Installation](./installation.md), and send a first request with the examples in [API](./api.md).

| Page | Contents |
|------|----------|
| [Installation](./installation.md) | Requirements, the Helm chart, the Kustomize overlays, the container image, running locally, building from source, log output |
| [Configuration](./configuration.md) | Command-line flags, setting flags in Kubernetes, Helm values |
| [API](./api.md) | Endpoints, the request body, responses, errors, coordinate precision |
| [Architecture](./architecture.md) | What one request does, the cache, the upstream providers, timeouts, logging |
| [Metrics](./metrics.md) | Prometheus endpoint, metric names and labels, scrape setup |
84 changes: 84 additions & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# API

The service listens on the port [`-port`](./configuration.md#command-line-flags) sets, 8080 by default. `/metrics` serves [Prometheus metrics](./metrics.md) to any method. Every other path goes to the geocoding handler, where the method alone decides the response:

| Method | Response |
|--------|----------|
| `POST` | The address record for the coordinates in the body, described in [Reverse geocoding](#reverse-geocoding) |
| `GET` | `{"status": "healthy"}` with status 200 |
| Any other, `HEAD` included | `{"status": "method not allowed"}` with status 405 |

## Reverse geocoding

```sh
curl -d '{"lat":"41.40338","lon":"2.17403"}' http://localhost:8080/
```

```python
from requests import post

r = post("http://localhost:8080/", data='{"lat":"41.40338","lon":"2.17403"}')
print(r.status_code, r.headers.get("X-Cache-Status"), r.text)
```

### Request body

The body is a JSON object, and the service reads it whatever the `Content-Type` header says:

| Field | Type | Accepted values |
|-------|------|-----------------|
| `lat` | String | A decimal latitude from `-90` to `90` |
| `lon` | String | A decimal longitude from `-180` to `180` |

Both fields are required and both must be JSON strings. A JSON number in either field returns 400. The service parses each string as a 32-bit float, rejects surrounding spaces, and ignores any other field in the object.

### Response

A successful lookup returns status 200 with these headers:

| Header | Value |
|--------|-------|
| `Content-Type` | `application/json` |
| `X-Cache-Status` | `MISS` when the service asked an upstream provider, `HIT` when it answered from the [cache](./architecture.md#cache) |

The body is the JSON that one of two upstream providers returned. On each cache miss the service picks a provider at random, and the two return different fields:

| Provider | Request | Response reference |
|----------|---------|--------------------|
| Nominatim | `https://nominatim.openstreetmap.org/reverse?format=json&zoom=18&addressdetails=1&lat=…&lon=…` | [Nominatim reverse geocoding](https://nominatim.org/release-docs/latest/api/Reverse/) |
| BigDataCloud | `https://api.bigdatacloud.net/data/reverse-geocode-client?latitude=…&longitude=…&localityLanguage=en` | [Reverse geocode to city API](https://www.bigdatacloud.com/free-api/free-reverse-geocode-to-city-api) |

A client cannot choose the provider. A cache hit returns whichever provider answered the first request for that location.

The service decodes the provider's JSON and encodes it again. Object keys come back in alphabetical order, and `<`, `>` and `&` inside strings come back as `<`, `>` and `&`.

### Errors

Every error response has `Content-Type: text/plain; charset=utf-8` and a body that ends in a newline. An error after a cache miss keeps `X-Cache-Status: MISS`. The service caches no error.

| Status | Body | Cause |
|--------|------|-------|
| 400 | `EOF` | The body is empty |
| 400 | `invalid character 'h' looking for beginning of value`, or another JSON decoder error | The body is not JSON |
| 400 | `json: cannot unmarshal number into Go struct field .lat of type string` | `lat` or `lon` is a JSON number |
| 400 | `strconv.ParseFloat: parsing "…": invalid syntax`, quoting the value | `lat` or `lon` is missing, which quotes an empty value, or holds something other than a decimal number |
| 400 | `Latitude and/or longitude values are out of range.` | `lat` or `lon` lies outside its range |
| 405 | `{"status": "method not allowed"}` | The method is neither `GET` nor `POST` |
| 500 | `Failed to fetch data` | The upstream request failed, or took longer than 10 seconds |
| 500 | `Failed to decode response` | The upstream body is not JSON, whatever its status |
| The upstream status | `External API error: ` followed by the upstream JSON in Go syntax, such as `map[error:rate limited]` | The upstream returned JSON with a status other than 200 |

## Coordinate precision

The cache key rounds both values to 3 decimal places. Within one pod, every request that rounds to the same pair shares one cache entry, so `41.4034,2.1736` gets the answer cached for `41.40338,2.17403`. The request to the provider carries 6 decimal places of the 32-bit value, so `41.40338` goes out as `41.403381`.

One degree of latitude spans about 111 km. A degree of longitude spans the same at the equator and less towards the poles. Each decimal place divides the step by ten:

| Decimal places | Step in latitude |
|:--------------:|:----------------:|
| 1 | 11.1 km |
| 2 | 1.11 km |
| 3 | 111 m |
| 4 | 11.1 m |
| 5 | 1.11 m |
| 6 | 11.1 cm |
55 changes: 55 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Architecture

```mermaid
flowchart TD
req([POST on any path]) --> decode{lat and lon parse<br/>and lie in range?}
decode -->|no| e400[400]
decode -->|yes| key[Round both to<br/>3 decimal places]
key --> hit{Cached and<br/>not expired?}
hit -->|yes| cached[200 from the cache<br/>X-Cache-Status: HIT]
hit -->|no| pick[Pick Nominatim or<br/>BigDataCloud at random]
pick --> upstream[GET with a<br/>10 second timeout]
upstream --> count[Count the call in<br/>host_request_total]
count -->|error or timeout| e500a[500]
count -->|body not JSON| e500b[500]
count -->|JSON, status other than 200| passthru[The upstream status]
count -->|JSON, status 200| store[Cache for -cache minutes]
store --> fresh[200<br/>X-Cache-Status: MISS]
```

## Request handling

The HTTP server in [`geocode.go`](https://github.com/saidsef/faas-reverse-geocoding/blob/main/geocode.go) has two routes. `/metrics` goes to the Prometheus handler. `/` matches every other path and goes to the geocoding handler in [`latitude_longitude.go`](https://github.com/saidsef/faas-reverse-geocoding/blob/main/internal/handlers/latitude_longitude.go), behind a middleware that logs each request. [API](./api.md) lists what each method returns.

## Cache

The cache in [`cache.go`](https://github.com/saidsef/faas-reverse-geocoding/blob/main/internal/cache/cache.go) is a map held in the memory of each process. Its key is the latitude and longitude rounded to 3 decimal places, and its value is the decoded JSON from the provider. Only a lookup that returns status 200 goes into the cache, and it stays for the number of minutes `-cache` sets.

The cache has no size limit and no background cleanup. An expired entry leaves memory only when a later request reads its key, so memory grows with the number of distinct locations requested. A restart empties the cache.

Replicas do not share a cache. The Service in the Kustomize base sets `ClientIP` session affinity for 60 seconds, so repeat requests from one client within that time reach the same pod. The Service in the Helm chart sets no affinity.

## Upstream providers

Each cache miss picks one of the two providers with `crypto/rand` and sends it one request. The service does not retry, and does not fall back to the other provider when the first one fails. The HTTP client gives up after 10 seconds and sends Go's default `User-Agent`, `Go-http-client/1.1`.

Both providers publish terms of use:

- The [Nominatim usage policy](https://operations.osmfoundation.org/policies/nominatim/) allows at most one request a second and requires a `User-Agent` that identifies the application. It states that stock User-Agents as set by HTTP libraries do not qualify.
- BigDataCloud documents the [reverse geocode to city API](https://www.bigdatacloud.com/free-api/free-reverse-geocode-to-city-api) as client-side only. Its fair use policy treats server-side calls as a breach, and a breach can lead to a temporary IP ban, with the endpoint returning HTTP 402.

A 402, like any status other than 200, reaches the client as that status when its body is JSON, and as 500 when it is not. [Errors](./api.md#errors) lists the bodies.

## Timeouts

| Timeout | Value | Applies to |
|---------|-------|------------|
| `ReadHeaderTimeout` | 15 seconds | Reading the request headers |
| `ReadTimeout` | 15 seconds | Reading the whole request |
| `WriteTimeout` | 10 seconds | Writing the response, counted from the end of the request headers |
| `IdleTimeout` | 30 seconds | Keeping an idle keep-alive connection open |
| HTTP client `Timeout` | 10 seconds | The whole upstream request, body included |

## Logging

The service logs through logrus as JSON to standard output, at level `info`, or `debug` with `-verbose`. [Log output](./installation.md#log-output) shows the lines a request writes.
45 changes: 45 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Configuration

The binary takes three command-line flags and reads no environment variables. The Helm chart has values of its own, which change what the chart deploys but not how the binary behaves.

## Command-line flags

| Flag | Values | Default | Description |
|------|--------|---------|-------------|
| `-port` | A TCP port number | `8080` | The port for every endpoint, `/metrics` included |
| `-cache` | A whole number of minutes | `60` | How long a successful lookup stays in the cache. With `0` or less, every request goes to the upstream provider |
| `-verbose` | `true` or `false`, and `-verbose` on its own means `true` | `false` | Sets the log level to `debug`, which adds a line for each cache lookup and the error behind each rejected request |

Each pod keeps its own cache. [Cache](./architecture.md#cache) describes how entries expire.

## Setting flags in Kubernetes

Neither the Kustomize base nor the Helm chart passes any arguments, so the binary runs with the defaults above. The image sets `CMD` and no `ENTRYPOINT`, so a container that sets `args` alone fails to start with `exec: "-cache": executable file not found in $PATH`. Set `command` as well:

```yaml
containers:
- name: geocode
command: ["/usr/bin/faas-reverse-geocoding"]
args: ["-cache", "30", "-verbose"]
```

The chart has no value for `command` or `args`, so a chart install always runs with the defaults.

The manifests, the chart and the image `HEALTHCHECK` all expect port 8080. A different `-port` needs the matching change to `containerPort`, the probes and the Service, and the chart template fixes `containerPort` at 8080.

## Helm values

[`values.yaml`](https://github.com/saidsef/faas-reverse-geocoding/blob/main/charts/reverse-geocoding/values.yaml) holds every value and its default. These values behave in ways the file does not show:

| Value | Behaviour |
|-------|-----------|
| `image.tag` | Defaults to `latest` with `image.pullPolicy` set to `IfNotPresent`, so a node keeps the first `latest` it pulled. A `vYYYY.MM` tag from [Container image](./installation.md#container-image) pins the version. An empty tag falls back to the chart's `appVersion`, which CI never publishes as an image tag |
| `replicaCount` | Has no effect while `autoscaling.enabled` is `true` |
| `service.port` | Sets the Service port only. The Service targets the container port by its name, `http`, and the container port stays 8080 |
| `ingress.hosts` | The template reads `host` from each entry and ignores `paths`. Every host routes `/` with `pathType: Prefix` |
| `ingress.className` | Sets `spec.ingressClassName` on the Ingress |
| `autoscaling.enabled` | Creates an `autoscaling/v2` HorizontalPodAutoscaler between `minReplicas` and `maxReplicas`, with targets for average CPU and memory use as a percentage of the pod's requests |
| `serviceAccount.create` | Defaults to `false`, and the pods run as the namespace's `default` service account. With `true`, the chart creates a ServiceAccount named `serviceAccount.name` and runs the pods as it. Either way the pods get no service account token |
| `podAnnotations` | Defaults to `prometheus.io/scrape: "true"` and `filter.by.port.name: "true"`, described in [Metrics](./metrics.md#scraping) |
| `livenessProbe`, `readinessProbe` | Replace the probes whole |
| `nodeSelector` | Empty by default. The image runs on `linux/amd64` only, so a cluster with `arm64` nodes needs `kubernetes.io/arch: amd64` here |
Loading