Skip to content

feat: fort map-data consumer — gyms, stations & pokestops - with DNF filtering - #1228

Open
jfberry wants to merge 62 commits into
WatWowMap:mainfrom
jfberry:feat/fort-consumer
Open

feat: fort map-data consumer — gyms, stations & pokestops - with DNF filtering#1228
jfberry wants to merge 62 commits into
WatWowMap:mainfrom
jfberry:feat/fort-consumer

Conversation

@jfberry

@jfberry jfberry commented Jul 16, 2026

Copy link
Copy Markdown

Deploying this (end users)

Two small config changes — one on Golbat, one on ReactMap. Once both are set, fort map-data (gyms, pokestops, stations) is served from Golbat's in-memory fort API instead of SQL, with automatic SQL fallback if Golbat is ever unreachable — so it's safe to enable on a dual (endpoint + DB) source.

1 · Golbat — run Golbat with UnownHash/Golbat#385 (it already includes the reward type-20 quest decode from #382) and turn on the in-memory fort index. fort_in_memory is a top-level option and defaults to false:

# golbat config.toml  (top level, above the [sections])
fort_in_memory = true

Make sure your Golbat API secret is set. Sanity-check the endpoint is live:

curl -H "X-Golbat-Secret: <secret>" http://<golbat-host>:<port>/api/fort/available
# 200 + { "pokestops": {...}, "gyms": {...}, "stations": {...} }

2 · ReactMap — add endpoint + secret to the scanner DB source whose useFor includes your fort types (gym/pokestop/station). This is the same Golbat endpoint most instances already point their pokémon source at; the DB source simply becomes dual (endpoint + DB):

// config/local.json → database.schemas[]   (your existing scanner DB source)
{
  "host": "127.0.0.1", "port": 3306,
  "username": "...", "password": "...", "database": "golbat_db",
  "endpoint": "http://<golbat-host>:<port>",   // ← add
  "secret": "<golbat api secret>",             // ← add (must match Golbat's)
  // "httpAuth": { "username": "...", "password": "..." },  // ← optional, if Golbat is behind Basic auth
  "useFor": ["gym", "pokestop", "station", "spawnpoint", "weather", "..."]
}

Restart ReactMap — this is a server-only change (no client rebuild). Gyms/pokestops/stations now fetch from Golbat; markers, popups, filters and deep-links keep working; and if Golbat is down or fort_in_memory is off, each fort type transparently falls back to SQL on the same DB.

Prefer to scope it tighter? Split a useFor: ["gym","pokestop","station"]-only copy of the DB source with the endpoint added and drop those types from the original — but it isn't necessary.

Confirm it's live (logs): each ~15-min availability refresh logs a line like [GYM] loaded available from http://<golbat>:<port>/api/fort/available — …, and map pans stop issuing fort SQL on endpoint-backed sources. A Golbat outage instead logs a fallback warn and reverts to SQL.


What this does

Routes fort map data — gyms, pokestops, and stations (getAll, getOne, getAvailable) — through Golbat's in-memory fort API (UnownHash/Golbat#385) when a scanner source has an endpoint, with transparent SQL fallback. Map pans stop issuing fort SQL entirely on endpoint-backed sources, and Golbat's DNF filtering returns only the forts that will render.

Architecture

  • Endpoint branch per model (mem set on the source): POST /api/<type>/scan → per-record pure mapper → the existing, unchanged secondaryFilter. On any failure (503 / network / bad shape) the model logs and falls through to SQL — a dual source degrades gracefully.
  • DNF is a superset narrow; secondaryFilter guarantees exactness. Pure backends (server/src/filters/fort/*) translate the map filters into Golbat DNF clauses (OR across clauses, AND within). The one hard invariant is never under-return; anything inexpressible (quest title/target conditions, gender, gym badges, time-window cutoffs) poisons to match-all or stays a client-side residual.
  • Exact key semantics in the translation (form-exact pokemon pairs, per-reward-type clauses, amount-exact mega/stardust/xp, reward type-20 treated as mega, grunt-class exclusion subtraction, station liveness) so the DNF result matches what renders.
  • Access-control parity: the endpoint paths reproduce the SQL path's gates — strict area-restriction denial, freshness, secondaryFilter's per-perm field gating — and getOne returns only {lat, lon} like SQL.
  • Observability: each fetch logs the narrowing (DNF(n): X matched -> Y after secondaryFilter (-residual) | <clause shapes>), so any translation gap is visible in production logs.
  • Deep-link parity: an off-viewport onlyManualId fort is fetched via the by-id endpoint and joins the candidate set, mirroring SQL's (bbox) OR id = ?.

Source shapes

A scanner source can be pure-DB (unchanged), dual (endpoint + DB — fort reads use Golbat, everything else and the fallback use the bound DB), or pure-endpoint (endpoint only, no DB — degrades gracefully where a query would need a DB). Enabling the endpoint on an existing DB source (the common case above) makes it dual.

Verified live (dual source, endpoint vs SQL)

Path Result
Pokestop quests 2373 forts scanned → 8 returned → 8 rendered (−0 residual)
Pokestop invasions (+ exclude grunts) 2806 scanned → 107 → 107 (−0)
Pokestop lures / showcases validated
Stations (incl. station_active, gmax) validated; inactive-viewing unaffected (match-all preserved)
Gyms (raids/teams/badges) validated, residual 0

Requirements

  • Golbat with UnownHash/Golbat#385 (fort_in_memory = true), deployed before this — Golbat's scan decoder ignores unknown request fields, so an out-of-date Golbat degrades to match-all/SQL rather than erroring.
  • Per scanner source: endpoint + secret (and optional httpAuth) as for pokémon; DB config unchanged (used for fallback + un-migrated queries).

No test framework exists in this repo; verification was throwaway node goldens per pure module (mappers, DNF backends), eslint/prettier, a full correctness/security/concurrency review of the branch, and the live parity runs above.

🤖 Generated with Claude Code

turtlesocks-bot and others added 15 commits June 5, 2026 13:43
…wMap#1225)

* fix(scanArea): prevent crash when area feature has no name/key

Guard the scan area search filter against features missing a properties.key (which happens when a scan area polygon has no name set), instead of throwing TypeError: Cannot read properties of undefined (reading 'toLowerCase').

Also fixes a longstanding typo (geoJsonFilName / geoJsonFilname -> geoJsonFileName) in the multi-domain example config and docs.

* fix: copilot comments

---------

Co-authored-by: Mygod <contact-git@mygod.be>
@jfberry

jfberry commented Jul 16, 2026

Copy link
Copy Markdown
Author

Validating in a test environment

Scope: this branch routes only gyms through Golbat (getAll/getOne/getAvailable). Pokestop markers/popups and stations still use SQL. Live validation needs Golbat #385 deployed with fort_in_memory = true.

1. Golbat prereqs — deploy #385, fort_in_memory = true; sanity-check (S=secret, G=golbat url):

curl -H "X-Golbat-Secret: $S" "$G/api/gym/available"
curl -H "X-Golbat-Secret: $S" -H 'Content-Type: application/json' -XPOST "$G/api/gym/scan" \
  -d '{"min":{"latitude":LAT1,"longitude":LON1},"max":{"latitude":LAT2,"longitude":LON2},"limit":1000,"filters":[]}'
curl -H "X-Golbat-Secret: $S" "$G/api/gym/id/<gym-id>"

2. ReactMapgit checkout feat/fort-consumer && yarn install, restart (server-only changes — no client rebuild).

3. Config — add endpoint + secret to the scanner DB source whose useFor includes gym (makes it a dual source). If you already did this for the Phase-1 pokestop test, gyms are already routed.

4. Confirm the endpoint path (logs):

  • ReactMap, each ~15-min refresh: [GYMS] [GYM] loaded available from Golbat endpoint …/api/gym/available — …
  • getAll/getOne are silent on success; a [GYM] /api/gym/scan … falling back to SQL warn = the endpoint failed → SQL.
  • Golbat: available-gyms built in … each time getAvailable is hit.

5. Functional — gym + raid markers render; a popup shows team/slots/defenders and raid boss + moves + CP; the filter drawer populates teams, tiers (r), eggs (e), raid bosses.

6. Golden parity (endpoint vs SQL): curl "$R/api/v1/available/gyms?current=1" with fort_in_memory on (endpoint) vs off (SQL fallback); diff the two → expect match (bar time-sensitive active-raid keys).

7. Fallback — stop Golbat or set fort_in_memory=false → gyms should still render via SQL + the fallback warn appears (no hard endpoint dependency; a timeout falls back too).

8. Area restrictions — as an area-restricted user, confirm gyms appear only within permitted areas (exercises the new client-side filterRTree) — the one behavior that isn't a straight SQL mirror, so the most important to verify.

jfberry and others added 7 commits July 17, 2026 15:37
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…st-layer availability fix

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jfberry
jfberry force-pushed the feat/fort-consumer branch from dadba73 to af62c30 Compare July 17, 2026 14:38
@jfberry jfberry changed the title feat: fort map-data consumer — gyms (stations/pokestops/DNF to follow) feat: fort map-data consumer — gyms, stations & pokestops - with DNF filtering Jul 17, 2026
jfberry and others added 2 commits July 17, 2026 16:06
… badge poison

Review findings:
- tier-override mode matches raid_level alone (curated boss/egg keys
  under-returned tier raids on endpoint sources)
- b<display_type> keys move to the onlyEventStops gate (secondaryFilter's
  events branch consumes them, not the invasions branch)
- endpoint rows resolve the quest layer as dual-capable (pure-endpoint ctx
  flags made effectiveQuestLayer resolve to 'both')
- poison on onlyGymBadges only: onlyBadge defaults to 'all' for every user
  with the gymBadges perm and was silently disabling gym DNF entirely

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the all-gyms/ex/in-battle poison with real narrowing: every gym-layer
display requires the team/slot match (hasGym = enabler && (team || slot)), so
team_id/available_slots clauses mirroring finalTeams/finalSlots are a tight
superset for all four enablers — ex/ar/in-battle narrowing stays residual.
The standalone is_ar_scan_eligible clause is subsumed (ar-shown gyms also
need the team match). Badge viewing still poisons. Power-up narrowing is gone
for good — power-ups are no longer in the game.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jfberry
jfberry marked this pull request as ready for review July 17, 2026 15:42
@jfberry

jfberry commented Jul 17, 2026

Copy link
Copy Markdown
Author

This is now in my production and I have verified the DNFs through manual testing

@jfberry

jfberry commented Jul 26, 2026

Copy link
Copy Markdown
Author

Six fixed, one is a repeat that's already handled.

Fixed (f8c29173, e95142be)

  • Serialize endpoint quest rewards as Stringquest_rewards and quest_conditions are stringified in the scan mapper so the GraphQL String fields coerce (they're raw JSON-string columns on the SQL path); parseRdmRewards already handles a string internally.
  • Manual Pokéstop before capped candidates — the off-viewport by-id fort is now prepended to the scan result, so secondaryFilter's resultLimit cap examines it first and the deep link survives a dense viewport (gym/pokestop/station).
  • Invalidate availability TTL on manager changeloadLocalContexts (startup/reload) now forces setAvailable, so a hot config reload with a fresh manager isn't short-circuited by the category-keyed TTL. The per-session stampede path stays non-forced/TTL-protected.
  • Preserve manager-owned metadata on empty refreshesgetAvailable no longer overwrites quest conditions / rarity with a total-failure empty result (runScannerSources returns only fulfilled sources, so empty ⇒ every source failed → retain last-good).
  • Don't swallow DB-backed source failuresgetOne / search / submissionCells now route through runScannerSources, which isolates the (expected) unbound pure-endpoint rejection while logging genuine SQL-source failures, instead of the raw allSettled that hid them.
  • Keep rocket-reward DNF a true superset — the a<pokemon> filter now narrows by incident display type only (incident_display_type: [1,2,3,4], a safe rocket-incident superset), never by the reward pokemon. Display type is reliable (it comes from the GMO); the reward is a slot-1-only value from an optional invasion check, so a reward-derived incident_character could drop a stop secondaryFilter accepts (e.g. across a rotation). secondaryFilter confirms the actual reward from its slot/metadata data. This removes the earlier metadata-grunt expansion and its empty-map poison.

Not changed

  • Scan limit before residual filters — this is the same point from the prior two rounds. It's addressed by a dedicated, operator-tunable Golbat cap (max_fort_results, default 9000, independent of max_pokemon_results); the residual gates (area polygons, freshness, adv conditions) can't be pushed server-side, so the deliberate design is a high tunable ceiling rather than over-fetching/pagination, which would broaden every scan.

All lint/node --check clean; the DNF, mapper serialization, and availability paths are unit-checked. Assumes Golbat ≥ #385 + #382.

@Mygod

Mygod commented Jul 26, 2026

Copy link
Copy Markdown
Collaborator

The endpoint station path bypasses the SQL field projection, while successful empty availability snapshots leave stale filters indefinitely. The new refresh TTL is also unavailable through the repository's environment-variable interface.

Full review comments:

  • [P1] Strip station totals from non-dynamax results — server/src/models/Station.js:795-795
    When an endpoint-backed request uses the all/inactive station layer for a caller without dynamax, spreading the raw Golbat object retains total_stationed_pokemon and total_stationed_gmax; later processing clears the battle fields but not these totals, while the SQL path selects them only when includeBattleData is true. Because GraphQL callers choose the Station selection set, they can request these fields despite the permission projection. Construct the base projection explicitly and add the totals only under includeBattleData.

  • [P2] Accept successful empty availability snapshots — server/src/services/EventManager.js:139-139
    When the last active option in a category disappears, such as when stations.battles becomes empty, Db.getAvailable legitimately returns [], but this branch treats that exactly like an all-source failure. The previous persisted list remains visible and availableUpdatedAt is never set, so every session retries while clients keep receiving stale filters. Carry source-success status separately and clear the category on a successful empty snapshot.

  • [P2] Expose the refresh TTL through environment config — config/default.json:62-62
    Container deployments using custom-environment-variables.json cannot override this new throttle because no mapping was added for api.availableRefreshSeconds, even though every other non-array scalar default has one. Consequently an API_AVAILABLE_REFRESH_SECONDS setting is ignored and the endpoint-scan throttle remains fixed at 60 seconds; add a numeric mapping alongside queryUpdateHours.

…env TTL)

Three ReactMap-side fixes.

- [P1] Station endpoint no longer leaks total_stationed_pokemon /
  total_stationed_gmax to non-dynamax callers. The endpoint spreads the raw
  Golbat record, so those totals rode along regardless of permission; the SQL
  path selects them only under includeBattleData. finalizeEndpointStation now
  strips them when battle data is not included, so a GraphQL selection cannot
  read stationed/gmax counts without the dynamax perm.
- [P2] Distinguish a total-source-failure availability refresh from a genuine
  empty snapshot. getAvailable returns null when every source failed (caller
  retains last-good); a successful empty result returns [] and clears the
  category, so a drawer whose last option disappears no longer shows stale
  filters forever. EventManager retains only on null; the /api/v1/available
  route guards its sorts with || [].
- [P2] Map api.availableRefreshSeconds to API_AVAILABLE_REFRESH_SECONDS in
  custom-environment-variables.json so container deployments can override the
  refresh throttle, matching every other scalar api default.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jfberry

jfberry commented Jul 26, 2026

Copy link
Copy Markdown
Author

All three fixed in 2cd14701.

  • Strip station totals from non-dynamax results — the station endpoint spread the raw Golbat record, so total_stationed_pokemon / total_stationed_gmax rode along regardless of perms while the SQL path selects them only under includeBattleData. A new finalizeEndpointStation strips both when battle data isn't included, so a GraphQL selection can't read the stationed/gmax counts without the dynamax perm.
  • Accept successful empty availability snapshots — good catch on the flip side of the earlier empty-refresh guard. getAvailable now distinguishes the two empty cases: it returns null on total source failure (every source rejected → caller keeps its last-good drawer) and [] on a genuine empty snapshot (sources succeeded, no active options → commit and clear). So a category whose last option disappears (e.g. stations.battles emptying) now clears instead of showing stale filters, while a transient outage still retains. EventManager retains only on null and the /api/v1/available route guards its sorts with || [].
  • Expose the refresh TTL through env config — added the availableRefreshSecondsAPI_AVAILABLE_REFRESH_SECONDS (number) mapping in custom-environment-variables.json, alongside the other scalar api defaults, so API_AVAILABLE_REFRESH_SECONDS is honored.

Lint / node --check / JSON all clean.

@Mygod

Mygod commented Jul 26, 2026

Copy link
Copy Markdown
Collaborator

Endpoint-backed scans can omit valid markers because they cap Golbat results before ReactMap's local gates, and explicit availability refreshes can reuse stale snapshots. The exported availability contract also diverges from the consumed Golbat payload.

Full review comments:

  • [P2] Apply the scan limit after local gates — server/src/models/Pokestop.js:886-886
    When the viewport contains more than queryLimits.pokestops DNF matches—especially for an area-restricted user—Golbat stops its R-tree walk as soon as this request-side limit is reached. ReactMap applies filterRTree, freshness, and secondary filters afterward, so rejected stops consume the cap and valid stops later in the scan never arrive; the same pre-filter limit is sent in Gym.js and Station.js. Apply the final marker limit after these gates or move the gates into the endpoint.

  • [P2] Bypass the snapshot cache on forced refreshes — server/src/utils/fortAvailable.js:50-50
    When PUT /api/v1/available or a configuration reload runs within 30 seconds of a previous fort refresh, force=true bypasses only EventManager's TTL; this cache hit still returns the old resolved snapshot or cached failure without querying Golbat. The route can report success with stale options, and API_AVAILABLE_REFRESH_SECONDS values below 30 seconds cannot take effect for fort categories. Preserve batch coalescing while allowing forced refreshes to start a fresh cache generation.

  • [P3] Align the availability types with Golbat's response — packages/types/lib/server.d.ts:84-90
    When typed code consumes the current fort-scan-map-data payload, this interface rejects it: invasion tuples include slot 2/3 fields and have no count, while lure and showcase tuples also omit count. The mapper already reads slot 2/3, so these exported declarations and the analogous mapper JSDoc contracts are inconsistent with the endpoint and prevent fixtures or consumers from being type-checked accurately.

jfberry and others added 2 commits July 27, 2026 08:33
Two ReactMap-side fixes (the third review item, scan-limit-before-residual, is
the recurring point already handled by Golbat max_fort_results).

- [P2] Forced availability refreshes now start a fresh combined-snapshot
  generation. force=true previously bypassed only EventManager's TTL, not the
  fortAvailable combined cache, so a config reload or PUT /api/v1/available
  within the window reused a stale snapshot (or cached failure). loadLocalContexts
  and the PUT route now call bustCombinedFortCache() before the forced batch; the
  concurrent fort refreshes still coalesce via the repopulated window. The
  combined-cache window now tracks api.availableRefreshSeconds instead of a fixed
  30s, so sub-30s API_AVAILABLE_REFRESH_SECONDS values take effect for forts.

- [P3] Align the exported availability types (server.d.ts) and the mapper JSDoc
  with the Golbat payload: AvailablePokestopInvasion gains slot2/slot3 fields and
  drops count; AvailablePokestopLure / AvailablePokestopShowcase drop count
  (quest keeps count, which Golbat does return).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The fort scan requests sent to Golbat passed limit: queryLimits.<type>,
which Golbat applies as a hard traversal cap BEFORE ReactMap's local gates
(filterRTree, freshness, secondaryFilter) run. A viewport with more forts
than the display limit could therefore drop valid forts: rejected forts
earlier in the scan consumed the cap, leaving valid ones past it unseen.

Send limit: 0 instead, so Golbat traverses up to its own server-side
max_fort_results backstop, and apply queryLimits.<type> as a display cap
AFTER the local gates:
- pokestop: already capped via secondaryFilter's resultLimit
- gym: final.slice(0, queryLimits.gyms)
- station: stations.slice(0, queryLimits.stations)

This mirrors the SQL path, where .limit(queryLimits.<type>) runs after
the WHERE clause, and relies on Golbat's max_fort_results as the true
traversal bound.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jfberry

jfberry commented Jul 27, 2026

Copy link
Copy Markdown
Author

All three fixed.

[P2] Apply the scan limit after local gates (a8564fbd) — the fort scans now send limit: 0, so Golbat traverses up to its own server-side max_fort_results backstop rather than stopping at the display limit before ReactMap's local gates run. queryLimits.<type> is now applied purely as a display cap after filterRTree/freshness/secondaryFilter:

  • pokestop — already capped via secondaryFilter's resultLimit
  • gym — final.slice(0, queryLimits.gyms)
  • station — stations.slice(0, queryLimits.stations)

This mirrors the SQL path, where .limit(queryLimits.<type>) runs after the WHERE clause, and relies on Golbat's max_fort_results as the true traversal bound.

[P2] Bypass the snapshot cache on forced refreshes (be254651) — PUT /api/v1/available and config reloads now call bustCombinedFortCache(), clearing the combined-snapshot map so the forced setAvailable batch starts a fresh generation instead of reusing a cached window. Batch coalescing is preserved (the concurrent fort batch still shares one Golbat request via the repopulated entry). The window now tracks api.availableRefreshSeconds, so sub-30s throttles take effect for fort categories.

[P3] Align the availability types with Golbat's response (be254651) — AvailablePokestopInvasion gains slot2_*/slot3_* and drops count; the lure and showcase tuples drop count too (quest keeps it). The mapper JSDoc typedefs were brought in line with the same shape.

@Mygod

Mygod commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

Forced availability refreshes can retain data from a replaced source, and endpoint-backed station data loses or invents user-visible battle metadata. These are functional regressions despite the successful lint and build checks.

Full review comments:

  • [P2] Do not reuse stale work for forced availability refreshes — server/src/services/EventManager.js:108-108
    When a database/endpoint hot reload or explicit PUT /api/v1/available overlaps a category refresh, this check runs before force is considered and returns the promise created with the old Db and cache generation. The forced call therefore never queries the newly installed manager, and the old result can be committed and TTL-stamped after reload; make pending work manager/generation-aware or let forced refreshes supersede it.

  • [P2] Supply updated timestamps for endpoint station battles — server/src/models/Station.js:812-813
    For endpoint-backed upcoming or hidden battle cards, Golbat's ApiStationBattleResult has no updated field, so forwarding each battle unchanged through enrichStationBattle leaves battles[].updated undefined. The SQL path selects station_battle.updated, the GraphQL query requests it, and StationBattleTimer uses it for last_seen; populate the per-battle timestamp or extend the endpoint payload before mapping.

  • [P2] Skip boss keys for battles without a known Pokémon — server/src/models/stationAvailableMapper.js:16-16
    When Golbat reports an active or upcoming battle before its boss is known, it encodes pokemon_id and form as 0, so this publishes 0-0 alongside the valid tier key. EventManager.addAvailable then treats that numeric key as a real Pokémon and injects id 0 into the masterfile/filter catalog, while endpoint battle records carry a null boss and matchesStationBattleFilter rejects them, making the advertised filter nonfunctional; only add the Pokémon key when pokemon_id > 0.

…attles)

Three ReactMap-side fixes from the latest review; none touch Golbat or
broaden its queries.

- Forced availability refreshes no longer reuse superseded in-flight work.
  setAvailable's single-flight guard now applies only to non-forced calls; a
  forced refresh (hot reload / PUT /api/v1/available / scheduled interval)
  always starts a fresh refresh against the current Db, and a monotonic
  per-category generation token gates the commit so an older in-flight refresh
  can't commit or TTL-stamp its (old-manager) result after a reload. Non-forced
  session-init stampedes still coalesce on the shared pending promise.

- Endpoint-backed station battles now carry an `updated` timestamp. Golbat's
  scan battle has no per-battle `updated`, so it is populated from the station's
  own `updated` (b.updated ?? apiStation.updated ?? null), mirroring the SQL
  getFallbackStationBattle path so StationBattleTimer has a last_seen to render.

- Station availability no longer advertises a phantom boss. A battle whose boss
  isn't known yet arrives with a null (Golbat) or 0 pokemon_id; mapStationAvailable
  now guards the boss key with Number(pokemon_id) > 0 so it doesn't inject a
  null-null/0-0 Pokémon into the filter catalog that no marker can match. The
  tier key (j{level}) is still published while the boss is unknown.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jfberry

jfberry commented Jul 27, 2026

Copy link
Copy Markdown
Author

All three fixed in e0e2dabf.

[P2] Forced availability refreshes no longer reuse superseded work (EventManager.js) — the single-flight guard now applies only to non-forced calls; a forced refresh (hot reload / PUT /api/v1/available / scheduled interval) always starts a fresh refresh against the current Db, and a monotonic per-category generation token gates the commit so an older in-flight refresh can't commit or TTL-stamp its (old-manager) result after a reload. Non-forced session-init stampedes still coalesce on the shared pending promise.

[P2] Endpoint station battles carry updated (Station.js) — Golbat's scan battle has no per-battle updated, so it's now populated from the station's own updated (b.updated ?? apiStation.updated ?? null), mirroring the SQL getFallbackStationBattle path so StationBattleTimer has a last_seen.

[P2] Skip boss keys for unknown bosses (stationAvailableMapper.js) — Number(b.pokemon_id) > 0 now guards the boss key, so an active/upcoming battle with an unknown boss (Golbat sends null, coerced to NaN here) no longer advertises a null-null/0-0 filter option that no marker can match; the tier key (j{level}) is still published while the boss is unknown.

@Mygod

Mygod commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

Thanks. The unknown-boss guard is functionally correct, but I don’t think the other two findings are fully resolved.

For battles[].updated, the fix belongs in Golbat. These are multi-battle records, whose SQL equivalent uses station_battle.updated; station.updated is only the legacy single-battle fallback. Copying one station-wide timestamp onto every battle fabricates last_seen values. Golbat already stores StationBattleData.Updated, so it should expose that in ApiStationBattleResult and ReactMap should pass it through unchanged.

The generation token fixes the immediate stale Event.available commit, but forced refreshes can now overlap on the same Db. Db.getAvailable() mutates questConditions and rarity before the generation check, allowing discarded work to overwrite current metadata. Scheduled intervals also capture the original Db and are not necessarily restarted on a database-only reload, so a later timer can still commit data from the replaced manager. The refresh lifecycle needs to be manager-aware, with same-manager work serialized or committed as one atomic snapshot.

The pokemon_id > 0 guard can remain. Please correct its explanation/JSDoc: Golbat availability uses 0 as the unknown sentinel, Number(null) is 0 rather than NaN, and the payload has non-null pokemon_id/form fields with no count.

@jfberry

jfberry commented Jul 27, 2026

Copy link
Copy Markdown
Author

Thanks for your observation about leaking sentinals, that's a poor api - I'll fix that in golbat and push the reactmap changes later; other comments are bang on, they'll be fixed at the same time

@Mygod

Mygod commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

Hello fellow human

jfberry and others added 2 commits July 27, 2026 22:44
…aware refresh

Companion to Golbat's availability null contract and per-battle `updated`,
plus the availability-refresh lifecycle fixes from the review.

A2 (null availability): the availability mappers already withhold a boss key
via `> 0` guards, so a null pokemon_id flows through identically to the old 0.
JSDoc/types updated to number|null (station battle, raid, showcase focus +
type_id, invasion slots), and the incorrect station comment corrected — Golbat
now sends null, and Number(null) is 0 (not NaN), so `> 0` still withholds the
key while shipping the tier key. Quests unchanged.

B1 (per-battle updated): Golbat now exposes the real per-battle `updated` on
ApiStationBattleResult, so the station endpoint passes `b.updated` through
unchanged instead of borrowing the station-wide timestamp (which fabricated
last_seen for multi-battle cards).

C1 (intervals): a database-only reload now restarts the availability intervals,
rebinding their captured Db to the freshly installed manager (previously only
an events reload restarted them, leaving timers querying the replaced manager).

C2 (atomic gated refresh): DbManager.getAvailable is now pure, returning
{ available, conditions?, rarity? }; EventManager commits the drawer AND the
manager metadata (questConditions/rarity, via applyAvailableMetadata) together
under the generation gate, so a superseded refresh can overwrite neither.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
getAvailable now resolves { available, conditions?, rarity? } | null, not the
old Promise<T[]>; correct the @ts-check annotation (drop the stale @template T)
and add a JSDoc for applyAvailableMetadata so callers' `.available` access
type-checks.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jfberry

jfberry commented Jul 27, 2026

Copy link
Copy Markdown
Author

Thanks — all addressed, and the updated/sentinel ones went into Golbat as you noted they should (companion PR UnownHash/Golbat#385, 959cb17).

battles[].updated — fixed in Golbat: ApiStationBattleResult now exposes the stored per-battle StationBattleData.Updated, and ReactMap passes it through unchanged, so multi-battle cards get the real station_battle.updated rather than a station-wide fallback.

Availability 0 sentinel — rather than only correcting the guard's comment, Golbat's availability output now emits null where the DB column is NULL (unknown boss / egg / type-based showcase / unconfirmed slot), matching what a consumer would read from the row. The pokémon/form pair moves together (null id ⇒ null form; a present id keeps form 0). ReactMap's existing > 0 guards already handle it; the JSDoc/types are now number | null, and yes — Number(null) is 0, not NaN, and the battle availability struct has no count.

Generation token — extended to close both gaps you flagged:

  • getAvailable is now pure and returns { available, conditions?, rarity? }; EventManager commits the drawer and the manager metadata (questConditions/rarity) together under the generation gate, so a superseded refresh can no longer mutate metadata ahead of the check.
  • A database-only reload now restarts the scheduled intervals, so a timer's captured Db is rebound to the freshly installed manager instead of committing the replaced one's data.

@jfberry

jfberry commented Jul 27, 2026

Copy link
Copy Markdown
Author

These versions deployed on my prod - no raids atm, but everything working as expected otherwise

@Mygod

Mygod commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

The endpoint-backed paths introduce request amplification on failed searches, persist expected by-ID misses as errors, and disable healthy endpoints when a dual source's DB probe fails. These are operational and fallback correctness regressions.

Full review comments:

  • [P1] Stop retrying after all search sources reject — server/src/services/DbManager.js:615-618
    When every source rejects—normally for a pure-endpoint Gym/Pokestop/Station source whose search method has no endpoint branch, or during a DB outage—runScannerSources collapses the batch to [], so this loop treats the failure as a successful empty radius and keeps widening. With default limits, one Gym search can issue about 557 failing queries and warning logs before returning empty; preserve an all-failed signal or skip unsupported sources and stop expanding.

  • [P2] Exclude fort-by-id 404s from failed-request logging — server/src/utils/evalScannerQuery.js:100-107
    When a manual or deep-linked fort ID is absent from an endpoint—especially with multiple scanner sources—Golbat returns 404 and this helper routes it through fetchJson, whose normal-miss exemption only covers /api/pokemon/id. The expected miss therefore logs an error and writes a timestamped request file under logs/ before becoming null; repeated misses create unnecessary log noise and files, so fort-by-id 404s should be handled as normal misses.

  • [P2] Initialize endpoint context before probing the fallback DB — server/src/services/DbManager.js:329-332
    When a dual endpoint+DB source's schema check rejects, such as during a transient scanner DB outage, control jumps to the outer catch before this overlay runs. The source is left without mem, secret, or httpAuth, causing endpoint-capable fort queries to bypass the healthy endpoint and execute against the failed DB; initialize endpoint context independently of the DB capability probe.

@Mygod

Mygod commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

These versions deployed on my prod - no raids atm, but everything working as expected otherwise

Unfortunately, the systems you touched have a fuckton of features + edge cases etc.

@jfberry

jfberry commented Jul 28, 2026

Copy link
Copy Markdown
Author

These versions deployed on my prod - no raids atm, but everything working as expected otherwise

Unfortunately, the systems you touched have a fuckton of features + edge cases etc.

Yes, and you didn't want me to build a test suite lol

@jfberry

jfberry commented Jul 28, 2026

Copy link
Copy Markdown
Author

Raids availability working correctly also. So lmk if any other review angles pick up anything

- Stop the radius-widening search when every source rejects. runScannerSources
  returns [] only when all sources rejected (a genuine-empty source is fulfilled
  with [], giving [[]]); an all-pure-endpoint fort search (no endpoint branch on
  search) or a DB outage otherwise spun ~557 failing queries + warnings until the
  2s cap. Break on data.length === 0.

- Treat fort-by-id 404s as normal misses. fetchJson's miss exemption only covered
  /api/pokemon/id, so a getOne / manual / deep-link miss on gym/pokestop/station
  logged an error and dumped the request under logs/. Broaden to the by-id
  endpoints via /\/api\/(pokemon|gym|pokestop|station)\/id\//.

- Keep endpoint context when the DB probe fails. getDbContext ran schemaCheck
  inside the try and overlaid mem/secret/httpAuth after it, so a schemaCheck
  rejection (transient scanner-DB outage) skipped the overlay and left a dual
  source without mem — its fort queries then bypassed the healthy endpoint and
  hit the down DB. schemaCheck now has its own try/catch (degrading SQL flags to
  {}), and the endpoint overlay + assignment always run.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jfberry

jfberry commented Jul 28, 2026

Copy link
Copy Markdown
Author

All three fixed in b197123a (ReactMap-only, no Golbat changes).

[P1] Stop retrying after all search sources reject (DbManager.search) — runScannerSources returns [] only when every source rejected (a genuine-empty source is fulfilled with [], giving [[]]), so it's a clean all-failed signal: the loop now breaks on data.length === 0 instead of treating it as an empty radius and widening. That kills the ~557 failing queries + warnings an all-pure-endpoint fort search (or a DB outage) used to spin before the 2s cap.

[P2] Exclude fort-by-id 404s from failed-request logging (fetchJson) — the normal-miss exemption was /api/pokemon/id only; it's now /\/api\/(pokemon|gym|pokestop|station)\/id\//, so a getOne / manual / deep-link miss on a fort id returns quietly (mirroring an empty SQL lookup) instead of logging an error and writing a timestamped request file under logs/.

[P2] Initialize endpoint context before probing the fallback DB (getDbContext) — schemaCheck now runs in its own try/catch (degrading the SQL capability flags to {} on failure), and the endpoint overlay (mem/secret/httpAuth) + source assignment run regardless. So a transient scanner-DB outage no longer strips a dual source's endpoint context — its fort queries keep using the healthy endpoint instead of falling back to the down DB.

@Mygod

Mygod commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Endpoint-backed fort scans can silently truncate valid markers, and the endpoint gym path breaks the no-badge filter. These are functional regressions in the newly added endpoint flows.

Full review comments:

  • [P1] Do not treat limit zero as an unbounded scan — server/src/models/Pokestop.js:892-892
    When a viewport's DNF matches more than Golbat's configured max_fort_results (9000 by default), limit: 0 uses that finite server cap rather than disabling the cap. Golbat stops its R-tree traversal before the area, freshness, and secondaryFilter gates here, so rejected candidates can consume the cap and silently omit valid markers even when fewer than ReactMap's display limit survive; Gym and Station send the same value. Use pagination, an actual unbounded producer mode, upstream filtering, or truncation detection with fallback.

  • [P2] Exclude badged gyms from the endpoint's none view — server/src/models/Gym.js:580-580
    When an endpoint-backed user with existing badges enables only the badge layer with badge = none, this candidate path does not reproduce the SQL query's whereNotIn(...userBadges). The shared secondaryFilter accepts every candidate through (actualBadge === 'none' && onlyGymBadges), so previously badged gyms are returned too. Exclude the userBadges IDs here or make the final predicate verify badge absence.

@jfberry

jfberry commented Jul 28, 2026

Copy link
Copy Markdown
Author

For the first point, the maximum limit is aligned with the model for pokemon settings as well - and is a very high limit for forts. The AI suggestion is stupid. If individual users actually have a real problem with this (ha!) they can increase the limit.
I'll look at the second point.

The endpoint candidate set isn't pre-filtered by badge (per-user ReactMap data
Golbat can't know), so the shared secondaryFilter's badge clause —
`(actualBadge === 'none' && onlyGymBadges)` — matched every candidate, letting
previously-badged gyms leak into the `none` view that the SQL path removes via
`whereNotIn(userBadges)`.

Replace the always-true clause with one that mirrors the SQL whereIn/whereNotIn:
`onlyGymBadges && (actualBadge === 'none' ? !newGym.badge : !!newGym.badge)` —
the `none` view keeps only gyms with no user badge, a specific/all badge view
keeps only those with one. It's a no-op for the SQL path (results are already
badge-filtered) and closes the endpoint leak. Verified equivalent across all
badge views (none/tier/all/off) for both paths.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jfberry

jfberry commented Jul 28, 2026

Copy link
Copy Markdown
Author

Fixed in e7ea3d8e.

[P2] Exclude badged gyms from the endpoint's none view (Gym.js) — the endpoint candidate set isn't pre-filtered by badge (per-user ReactMap data Golbat can't know), so the shared secondaryFilter's badge clause (actualBadge === 'none' && onlyGymBadges) was always true and let previously-badged gyms through, unlike the SQL path's whereNotIn(userBadges).

Replaced the always-true clause with one that verifies badge membership per view, mirroring the SQL whereIn/whereNotIn:

onlyGymBadges && (actualBadge === 'none' ? !newGym.badge : !!newGym.badge)

So the none view keeps only gyms the user has no badge for, and a specific/all badge view keeps only those they do (newGym.badge is only ever set when onlyGymBadges). It's a no-op for the SQL path — those results are already whereIn/whereNotIn'd — and closes the endpoint leak. Verified equivalent across every badge view (none / tier / all / off) for both the SQL and endpoint paths.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants