Note

This repository was branched out of storiny/web which was originally a monolith. If you're looking for the original git history before the split, you can find it over on the original repository.

Discovery

Discovery microservice powers embedding of third-party content on Storiny.

---

Tech stack

• Framework: Actix Web serves our HTTP endpoints and middlewares
• State & caching: Redis handles rate limiting
• Metadata & DOM parsing: Visdom, html5ever, and markup5ever_rcdom for parsing, querying and safely manipulating HTML DOM nodes
• HTTP client: reqwest for making HTTP requests to oEmbed endpoints and web pages
• Serialization: serde and serde_json for mapping oEmbed specifications into typed Rust structures
• Telemetry: Sentry for tracing, monitoring, and capturing errors in production

---

URL discovery & oEmbed

The heart of this repository is its embed resolution logic (src/routes/embed.rs and src/utils/). Whenever a user pastes a link into a story/document, this service determines exactly how to render it:

• as a rich interactive iframe
• a native video
• a photo
• or a standard link card

Under the hood

1. Request & URL decompression: When the client requests an embed from the frontend, they pass an LZ-compressed URL to our /embed/{compressed_url} endpoint. Inside the route handler, we use lz-str to decompress the string back into a valid URL. This simplifies caching behaviour as all of our /embed/ endpoints are heavily cached to prevent third-party rate limits.

2. Provider resolution (providers.rs): We maintain a curated registry of supported oEmbed providers in src/providers/providers.json. The decompressed URL is matched against pre compiled Regex schemas to see if it belongs to a known provider (like YouTube, Twitter, Spotify...).

   • If a match is found, we know exactly which endpoint to hit and whether the provider supports light/dark binary themes.
   • If no match is found, we immediately fallback to our custom metadata scraper.

3. oEmbed fetching & parsing (spec.rs): For supported providers, we make a request to their oEmbed endpoint. The JSON response is parsed into our EmbedResponse and EmbedType enums (Photo, Video, Link, Rich).

   • Based on the EmbedType, we parse the returned HTML using visdom, inject our own iframe styling, apply responsive aspect ratio paddings, and safely sanitize the output.
   • The heavily customized HTML (or raw JSON for scripted embeds) is then piped through the sailfish templating engine and returned to the client.

4. Metadata fallback: If the link isn't supported by an oEmbed provider or if the provider returns a generic Link type, we fall back to get_metadata. This utility fetches the raw HTML of the webpage and extracts opengraph (og:) tags, Twitter card tags, titles and descriptions; returning a JSON metadata payload for rendering link preview cards on the frontend.

---

Structure

• src/: The microservice
  • providers/: Provider definitions and the providers.json registry. Compiles Regex matchers on boot
  • routes/: HTTP handlers and controllers
  • utils/: Utilities
  • middlewares/: Actix middlewares (rate limiting)
  • spec.rs: Strictly typed struct definitions for the standard oEmbed specification

---

API endpoints

• GET /embed/{compressed_url}: Takes an LZ-compressed URL, resolves its provider, fetches the oEmbed data or falls back to raw metadata scraping and returns the sanitized HTML (or JSON for scripted embeds) for embedding on the frontend. This endpoint is heavily cached (using Cloudflare) to prevent third-party rate limits.
• GET /provider_check/{compressed_url}: Checks whether an embed provide is supported. Decompresses the URL and checks it against our providers.json registry. Returns an OK ({provider_name}) response if supported, or an error otherwise.
• GET /health: Health check endpoint. Returns OK.

---

Launching locally

1. Prerequisites

• Rust
• Redis
• Just: a handy command runner (install via cargo install just)

2. Environment setup

Copy the mapping file and fill in your own environment variables:

cp .env.mapping .env

3. Running the service

Fire up the development server with host reload (requires cargo-watch and optionally bunyan for pretty logs):

just dev

4. Testing

We use cargo-nextest for running our test suites:

just test