Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
3fc1fe3
docs: add design spec for Python API and API tokens
krokicki Aug 24, 2026
bad95ca
docs: add implementation plan for Python API and API tokens
krokicki Aug 24, 2026
cb439b3
feat: add api_tokens table and CRUD functions
krokicki Aug 24, 2026
45c147f
fix: use token_hex for token_id to avoid delimiter collision
krokicki Aug 24, 2026
44f9c3b
feat: add API token scope table and matching logic
krokicki Aug 24, 2026
4985c3a
feat: authenticate API requests with scoped bearer tokens
krokicki Aug 24, 2026
49082d2
feat: add /api/tokens management endpoints
krokicki Aug 24, 2026
00848bc
fix: reject whitespace-only token names, dedupe scopes on create
krokicki Aug 24, 2026
e916bca
feat: add Python client with absolute path resolution
krokicki Aug 24, 2026
635edd7
feat: add file operations to the Python client
krokicki Aug 24, 2026
b517467
fix: use TestClient instead of a custom async-transport adapter
krokicki Aug 24, 2026
cffd970
fix: ls() raises on a file instead of returning an empty list
krokicki Aug 24, 2026
04a9f15
feat: add data link and Neuroglancer link methods to the client
krokicki Aug 25, 2026
c978cc7
feat: add job methods to the Python client
krokicki Aug 25, 2026
17b5869
fix: require parameters explicitly in submit_job, add payload-shaping…
krokicki Aug 25, 2026
5282521
feat: add frontend TanStack Query hooks for API tokens
krokicki Aug 25, 2026
9896672
fix: revert unnecessary resolvePathToFsp tests from API tokens task
krokicki Aug 25, 2026
ef14879
feat: add API tokens page with create, list, and revoke
krokicki Aug 25, 2026
66637af
fix: guard create-dialog reopen/cancel and require revoke confirmation
krokicki Aug 25, 2026
b17175c
test: add end-to-end spec for API token creation and revocation
krokicki Aug 25, 2026
4b06a68
test: clear api_tokens table in E2E database cleanup
krokicki Aug 25, 2026
152ff03
style: run prettier on api-tokens spec and db-cleanup util
krokicki Aug 25, 2026
7959b41
test: assert token survives the first Revoke click
krokicki Aug 25, 2026
4001d5b
docs: use a neutral path in the client module docstring
krokicki Aug 25, 2026
a20594d
fix: final review pass on API tokens branch
krokicki Aug 25, 2026
ab48572
feat: describe each token scope and warn on the dangerous ones
krokicki Aug 25, 2026
5840e55
feat: lay out token scopes in two columns and enforce write implies read
krokicki Aug 25, 2026
a1f9981
feat: add an interactive walkthrough script for the Python client
krokicki Aug 25, 2026
3fa970f
fix: say so when the server has no file shares configured
krokicki Aug 25, 2026
04e46dd
fix: stop enumerating every file share in the no-match error
krokicki Aug 25, 2026
beeec48
docs: amend the Python API spec with implementation divergences
krokicki Aug 25, 2026
47e5ac5
feat: make the supported API token scope set server-configurable
krokicki Aug 25, 2026
8b01ca6
fix: surface why a token revoke failed
krokicki Aug 25, 2026
4dc6efa
fix: normalize separators in abspath, and compare paths portably in t…
krokicki Aug 25, 2026
26c2b3f
feat: log the username and token id for token-authenticated requests
krokicki Aug 25, 2026
cb9a9e7
added python package link
krokicki Aug 25, 2026
669f6cc
fix: stop rendering API token page text in the brand accent purple
krokicki Aug 25, 2026
2d3d8b7
docs: name env vars plainly instead of using $VAR shell syntax
krokicki Aug 26, 2026
082ddb4
feat: serve data links as an S3 bucket so they browse in Neuroglancer
krokicki Aug 28, 2026
d0da456
fix: keep names that need encoding readable through a listing
krokicki Aug 28, 2026
a3727f7
feat: add a JSON logging mode for Kibana
krokicki Aug 28, 2026
65d48a1
chore: drop Windows support
krokicki Aug 28, 2026
1738646
fix: drop a stray comment that clobbered the cli.py shebang
krokicki Aug 28, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 2 additions & 14 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,8 @@ concurrency:

jobs:
build:
name: Build (${{ matrix.os }})
runs-on: ${{ matrix.os }}

strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest]
name: Build
runs-on: ubuntu-latest

env:
VITE_ENABLE_TASKS: true
Expand All @@ -37,43 +32,36 @@ jobs:
manifest-path: pyproject.toml

- name: Install dependencies
shell: bash
run: |
set -eux
pixi run dev-install

- name: Lint
if: matrix.os == 'ubuntu-latest'
run: |
set -eux
pixi run node-prettier-check && pixi run node-eslint-check

- name: Test the frontend
shell: bash
run: |
set -eux
pixi run test-frontend

- name: Test the backend
shell: bash
run: |
set -eux
pixi run test-backend

- name: Build the frontend
shell: bash
run: |
set -eux
pixi run node-build

- name: Package the app
if: matrix.os == 'ubuntu-latest'
run: |
set -eux
pixi run pypi-build

- name: Upload package to GH
if: matrix.os == 'ubuntu-latest'
uses: actions/upload-artifact@v4
with:
name: extension-artifacts
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ Fileglancer has a React front-end and a FastAPI backend. Uvicorn is used to mana

- [User guide](https://janeliascicomp.github.io/fileglancer-docs/)
- [Developer guide](docs/Development.md)
- [Structured logging](docs/StructuredLogging.md)

## Related repositories

Expand Down
52 changes: 52 additions & 0 deletions docs/StructuredLogging.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Structured Logging

Setting `log_format: json` (or `FGC_LOG_FORMAT=json`) makes Fileglancer write one JSON object per line to stdout instead of human-readable text. The field names follow the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/index.html) (ECS), so Kibana's built-in visualizations understand them without any mapping work.

The default remains `text`; nothing changes unless you turn this on. Turn it on in production, leave it off for development.

## What gets logged

Every log record carries `@timestamp`, `log.level`, `log.logger`, `message`, `service.name`, `service.version` and `process.pid`. Records emitted while serving a request also carry `trace.id`, which is returned to the client in the `x-request-id` response header — so a user reporting a problem can quote an id that pulls up every line for that request, including the ones logged inside the handler.

One access line is logged per request, tagged `event.dataset: fileglancer.access`, with:

| Field | Notes |
|---|---|
| `event.duration` | Request duration in **nanoseconds** (the ECS unit). |
| `http.request.method`, `http.version` | |
| `http.response.status_code` | |
| `http.response.body.bytes` | Present when the response declared a Content-Length. |
| `url.path`, `url.query`, `url.domain` | Path is logged as the client encoded it. |
| `client.ip`, `client.port` | Corrected for the reverse proxy when running behind one. |
| `user_agent.original` | |
| `user.name` | Session or API-token user, absent when unauthenticated. |
| `labels.token_id` | Present on API-token requests; the id to revoke. |
| `labels.endpoint` | Handler function name. Group latency by this — raw paths have unbounded cardinality and are useless to aggregate over. |
| `error.type`, `error.message`, `error.stack_trace` | On records logged with an exception. |

Per-user worker subprocesses log in the same format. In text mode their output is forwarded through the parent tagged `[worker:username]`; in JSON mode they write their own records straight to the shared stdout, identifiable by `process.pid`.

## Shipping to Elasticsearch

Fileglancer logs to stdout, so a shipper reading the container's log files needs only to parse the JSON. For Filebeat:

```yaml
filebeat.inputs:
- type: container
paths:
- /var/lib/docker/containers/*/*.log
json.keys_under_root: true
json.add_error_key: true
```

Since ECS field names are written as dotted keys, Elasticsearch expands them into the usual nested ECS document on ingest.

## Kibana

`event.duration` is a plain number, so a percentile aggregation over it gives p95/p99 directly. Set the field's format to *Duration* (input: nanoseconds) in the data view to have Kibana render it readably. Break it down by `labels.endpoint` to find a slow API route, by `user.name` to find a heavy user, or by `http.response.status_code` for error rates.

## Known limitation

`event.duration` is measured until the response starts, not until the last byte is sent, so file downloads are recorded as time-to-first-byte rather than transfer time. (x2s3, which serves the bulk data, does measure full transfer time.) Charting download latency here would mean rewriting the access log middleware as pure ASGI, the way x2s3's is.

Uvicorn logs two lines ("Started server process", "Waiting for application startup") before it imports the app, so those stay plain text on every start. Everything after that is JSON.
25 changes: 25 additions & 0 deletions docs/config.yaml.template
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,13 @@
#
log_level: INFO

#
# Log output format: 'text' for human-readable logs, or 'json' for one
# ECS-shaped JSON object per line on stdout, ready for a log shipper to
# forward to Elasticsearch/Kibana. See docs/StructuredLogging.md.
#
log_format: text

#
# Database URL (runtime - unprivileged user)
# For SQLite: sqlite:///fileglancer.db
Expand Down Expand Up @@ -139,6 +146,24 @@ session_cookie_secure: true
# - https://ai-cryoet.int.janelia.org
# - https://nextflow.int.janelia.org:8444

# Which API token scopes this server supports. A scope left out cannot be
# granted to a new token, and is ignored on tokens that already hold it -- so
# removing one here revokes the capability from existing tokens rather than
# only affecting new ones.
#
# files:write and jobs:write are absent from the default shown below because
# both amount to full access to the user's files: jobs:write runs arbitrary
# code as the user, so it reaches every file regardless of the files:* scopes
# granted. Enable them per server, deliberately.
#
# api_token_scopes:
# - files:read
# - files:write
# - links:read
# - links:write
# - jobs:read
# - jobs:write

#
# Maximum size in bytes accepted by a PUT /api/content upload. Guards the
# streaming write path against an unbounded upload filling the disk. Set to 0
Expand Down
Loading
Loading