Skip to content

About

Indexing Indexer Selection Algorithm - Standalone containerized service for The Graph DIPs indexer selection

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Subgraph DIPs Indexer Selection

Choosing which indexers earn paid indexing agreements on The Graph

CI Latest release

At a glance | Background | What it produces | How it works | Measuring | Choosing | Repository layout


The Indexing Indexer Selection Algorithm (IISA) for The Graph's Direct Indexing Payments service. It decides which indexers should be selected to index and serve a given subgraph deployment, ranking every candidate by a score that multiplies its reliability, quality and value, drawn from real query traffic and on-chain stake.


At a glance

Question Answer
What this service decides Which indexers should hold the paid agreements for each subgraph deployment
Who asks it The DIPs gateway service, dipper, over HTTP
When scores refresh Once a day, when the measurement job replays recent gateway query traffic
What is measured Economic security, price, latency, uptime, freshness, success rate
How an indexer scores well Serve queries reliably and quickly, price competitively, keep graph-node current

Background

The Graph is a decentralised network where independent operators, called indexers, organise blockchain data into subgraphs and answer queries against them. Most of an indexer's income comes from protocol indexing rewards, which are spread across the whole network. Direct Indexing Payments (DIPs) sit alongside that: indexers are paid a recurring fee — quoted in GRT per 30 days — to commit to indexing and serving a specific subgraph deployment. At first these agreements are funded by the protocol itself.

That arrangement only works if someone decides which indexers should receive those paid agreements, keeps the best-performing ones in place, and replaces the ones that fall behind. Making that decision well is the job of this project.

What it produces

Given a subgraph deployment and the number of indexers it needs, the service returns the set of indexers that should serve it, each with its price for the requested chain. The answer is a recommendation. The DIPs infrastructure that calls it — a service known as dipper — compares the recommendation against the agreements that already exist and works out which to add and which to cancel. This project never signs or cancels an agreement itself; it only advises, so the answer can also shift between calls as fresh scores arrive.

How it works

The system has two cooperating parts that share nothing on disk. A daily score-computation job measures how every indexer has actually been performing and pushes the results to the service. A long-running HTTP service keeps the latest scores in memory and answers selection requests. The job sends its results to the service over the network, and the service is the single owner of the data it serves. Both run on Kubernetes — the service as a long-lived deployment, the measurement work as a scheduled job that runs once a day and then exits.

flowchart LR
    A["Gateway query traffic<br/>(Redpanda)"]
    B["Graph Network subgraph<br/>(stake)"]
    C["Per-indexer price<br/>and graph-node version"]
    D["Score-computation job<br/>(runs daily, then exits)"]
    E["IISA service<br/>(holds latest scores)"]
    F["DIPs infrastructure<br/>(dipper)"]

    A --> D
    B --> D
    C --> D
    D -->|pushes scores| E
    E -->|serves selections| F
Loading

Measuring indexers

Once a day the job replays roughly the last few weeks of real gateway query traffic from a Redpanda (Kafka) topic — a window the operator can widen or narrow — and combines it with live stake data from the Graph Network subgraph and a fresh price quote fetched from each indexer. From that it derives, for every indexer:

  • Economic security — slashable stake measured against the query fees the indexer has earned. More stake standing behind fewer fees means more to lose by misbehaving.
  • Price — what the indexer charges to serve the deployment, quoted both per 30 days and per billion entities served.
  • Latency — how quickly it answers, estimated with a regression that accounts for the geographic distance between the indexer and where queries originate. The regression gives each indexer's gap to the median indexer, with a margin for how little data backs it; adding the median indexer's own latency to that gap gives the indexer's latency in milliseconds (latency_ms, never below 0). The median indexer's latency is the median response time once each query's own indexer gap is taken out, since the median query alone leans towards the fast indexers the gateway sends most queries to.
  • Uptime — how much of the observed window the indexer was available.
  • Success rate — the share of its queries that returned a usable result.

Uptime and success rate are also worked out on each chain the indexer serves, from the chain each replayed query names. The indexer's overall uptime and success rate are the query-weighted average over all its chains, and a chain with fewer than 10,000 of its queries in the window leans on that overall value: with n queries there, the chain's own value counts with weight n / (n + 250) and the overall value makes up the rest, so 2 small chains keep their order and an indexer on 1 chain keeps that chain's value. Selection scores an indexer on the request chain's values when it has them, and on its overall values otherwise. The job also sends each indexer's queries on each chain, queries_by_chain, the counts that weigh its chains in that blend, so a reader can weigh the indexer's chains the same way.

Because the traffic is enormous, the job samples it instead of reading every record, and balances the sample so each indexer is represented fairly rather than being drowned out by the busiest ones. The sampling is seeded, so a given seed and the same input reproduce the same scores.

The measurement is built to degrade rather than disappear. If geographic lookups are unavailable it falls back to neutral latency scores; if stake cannot be read, it reuses the stake and fees the service holds from a run that read them, for up to 60 hours, since they barely move in a day, and leaves out any indexer it has no reading for beside those it has; if the wider pipeline fails it publishes today's prices with the quality, stake and fees the service holds from a run that measured them, again for up to 60 hours, and only with nothing recent to carry over does it publish real prices with equal quality scores rather than going dark. It can also be told to exclude any indexer running below a minimum graph-node version, so stale software does not win paid work.

Checking freshness (off by default)

The sync-status fetcher can also check, once a minute, that each indexer serves fresh data on every chain it offers DIPs on and has priced, among the networks in The Graph's networks registry. For each indexer and chain it picks at random 1 subgraph the indexer first allocated to at least FRESHNESS_MIN_ALLOCATION_AGE_DAYS days ago (7 by default) and still has open, asks the indexer for that subgraph's latest block through the gateway, as a customer's query would reach it, and counts the answer as fresh when the block is at most FRESHNESS_BAR_SECONDS old (36 by default). Each minute's outcomes go to the service about 30 seconds after the minute ends. A failure on the gateway's side or the fetcher's own is reported as void rather than held against the indexer, and so is a minute in which most indexers didn't answer at all. The block time comes from the indexer's own signed answer, so the check catches an indexer that is behind, but not one that changes its software to report a false block time.

2 more rules keep an indexer from being marked down for something that isn't its fault. When an indexer refuses the gateway's payment, the check is void, with a warning in the log naming the indexer at most once every 10 minutes: the indexer may be owed more than the gateway's escrow holds, which IISA can't see, so a refusal can't be shown to be the indexer's fault. When an indexer is behind on a subgraph, up to 3 other indexers allocated to that subgraph are asked the same question. If at least 2 indexers are behind on it while fresh on another subgraph of the same chain, and no indexer has answered fresh on it in the last 10 minutes, the subgraph is broken for everyone, so it is left out of every indexer's checks for an hour and the stale answers on it are void. Being behind on every subgraph doesn't count, since that is what an outage at a shared RPC provider looks like. A subgraph found not broken isn't asked about again for 10 minutes, and nor is one that any indexer answered fresh on in the last 10 minutes. Each of those extra questions waits at most 10 seconds for an answer, and a verdict that arrives after its minute has been pushed leaves that minute's fail standing, though the subgraph is still left out from then on.

It runs only when FRESHNESS_PROBE_ENABLED is true. It then needs FRESHNESS_GATEWAY_URL, the gateway's base address such as https://gateway.thegraph.com/api, and FRESHNESS_GATEWAY_API_KEY, and the fetcher refuses to start without them. Every check is a paid query, so switching it on spends query fees on that API key, about 1,440 queries a day for each indexer and chain, plus up to 7 more for each stale answer that asks other indexers, with at most 20 of those a minute. FRESHNESS_REFRESH_INTERVAL (900 seconds) sets how often it re-reads which indexers and subgraphs to check, FRESHNESS_REQUEST_TIMEOUT (25 seconds) how long a check waits for an answer, and FRESHNESS_MAX_CONCURRENCY (50) how many checks run at once. The fetcher's /health body shows when the last minute was pushed.

If the prober stops unexpectedly, the fetcher's /health answers 503 so the pod is restarted. A prober that runs but can't push, because the service or the gateway is down, leaves the answer alone, so someone else's outage doesn't restart the pod over and over.

Only chains in The Graph's networks registry are checked, since those are the only chain names a selection can ask about, and a made-up name in an indexer's list is left out with a warning. The fetcher reads the registry from FRESHNESS_NETWORKS_REGISTRY_URL, by default The Graph's own copy with the copy on GitHub as a fallback, before its first check and then every 6 hours. When a read fails it keeps the last copy that loaded, and until one has loaded it checks nothing. FRESHNESS_ADDITIONAL_NETWORKS adds comma-separated chain names the registry lacks, such as a test chain, and /health shows when the registry last loaded and how many networks it holds.

Choosing indexers

When asked to fill a deployment, the service gives each indexer a total score between 0 and 1, the product of 4 factors: uptime × freshness × quality × value. Uptime and freshness are factors on their own. Quality and value are each a weighted average of their signals:

Factor Signal Better Weight
Uptime Uptime higher 1
Freshness Freshness higher 1
Quality Success rate higher 0.2
Quality Latency lower 0.8
Value Economic security (stake-to-fees) higher 0.45
Value Price (per 30 days) lower 0.45
Value Price (per billion entities) lower 0.10

Better is the direction that earns a higher score: latency and price are inverted during scoring, so a lower raw value scores higher. Because the factors multiply, a 0 on uptime or freshness sinks the whole score, while a weak signal inside quality or value only pulls that factor down. These are the weights selection always uses: the source of truth is QUALITY_WEIGHTS and VALUE_WEIGHTS in src/iisa/indexer_selection.py, and a selection request has no way to change them. docs/scoring.md sets the formula out in full, with every constant and a worked example.

Uptime and success rate are each scored on the indexer's own share, raised to the 6th power, so 90% scores about 0.53 and 80% about 0.26, and an indexer's score on them doesn't move when a better indexer becomes a candidate.

A per-entity price is scored against the payer's own ceiling, max_grt_per_billion_entities_per_30_days in the selection request: 1 at a price of 0, falling in a straight line to 0 at the ceiling. When the request sets no ceiling it is scored against the highest per-entity price among the candidates instead.

Latency is scored on a fixed curve of the indexer's latency in milliseconds, latency_ms from the score job: 1 at 0 ms, about 0.96 at 50 ms, 0.52 at 250 ms and 0.04 at 500 ms, never below 0.001, so an indexer's score doesn't depend on the other candidates. An indexer without a latency scores 0. Scores without latency_ms, as from a score job run without GeoIP, a degraded run or a job older than the field, keep the job's own latency score, lat_normalized_score, scaled across every indexer the job measured, and the service logs a warning about it at most once every 10 minutes.

From those scores it selects the number of indexers the caller asks for, shaped by a handful of constraints. It stays within the price ceiling the caller specifies and skips indexers that do not serve the requested chain. When more than one indexer is needed it tries to spread them across distinct operators and locations, falling back to the best available when no combination satisfies that. When it knows which indexers are already synced to the deployment, it favours them, so queries can be served the moment an agreement is accepted. An indexer counts as synced on a deployment only when its own server says so and it also holds an active allocation on that deployment on-chain: the hourly sync-status fetcher checks each claim against the Graph Network subgraph, since a claim alone would let any indexer list every deployment. Given a gateway address and API key (IISA_SYNC_CHECK_GATEWAY_URL and IISA_SYNC_CHECK_GATEWAY_API_KEY), the service also checks a candidate's claim at the moment it could win the candidate a place: when a selection looks for an indexer to add or swap in, it asks every candidate that could be picked ahead of the rest on its claim, all at once and through the gateway as a paying customer would, for the deployment's latest block, and confirms a claim only if the indexer signed the answer and the block is at most IISA_SYNC_CHECK_BAR_SECONDS old (36 by default). A candidate is asked only when being synced could decide the pick: it must leave the group spread over 2 operators and 2 locations, beat a member it would replace by the replacement margin, and total at least 60% of the best total among the candidates it could pick once the group, counting indexers picked earlier in the same selection, has a synced indexer. Indexers already in the group are never asked: their claims count as given. If any claim asked is not confirmed within IISA_SYNC_CHECK_TIMEOUT_SECONDS (5 by default), for any reason including an unreachable gateway, selection decides again counting only the members' claims and the confirmed ones, so every other candidate competes on its score. At most 50 candidates are asked for 1 selection, best scores first; with more, the rest are left unasked and selection also decides again. It leaves a working incumbent in place, stepping in only once the group is already at its target size and an incumbent totals under 15% of the best candidate's total while a challenger beats it by more than half of that best total, so the set does not churn on small fluctuations. The cut-offs are fractions of the best total because products run lower than sums: a good indexer totals about 0.5. And it never picks an indexer the caller has blocked. The exact thresholds behind these rules live alongside the weights in src/iisa/indexer_selection.py.

Economic security is scored by placing each indexer's stake-to-fees ratio on a straight line between the lowest and the highest ratio among the indexers the request can keep or pick, with no cap on how far the highest can stretch that line. The fees are the query fees the indexer has earned plus, when dipper sends them with the request, the DIPs fees dipper expects it to earn from its active agreements. This is a deliberate choice, and it has a consequence worth knowing: 1 indexer with an extremely high ratio pushes everyone else's score on this signal towards 0. With 4 indexers at ratios of 500, 1,000, 2,000 and 5,000, they score 0, 0.11, 0.33 and 1; add a fifth at 1,000,000,000,000 and the first 4 all score effectively 0 while it scores 1. Economic security is 45% of value, and value multiplies the whole score, so in a selection the outlier's value is up to 0.45 ahead of everyone else's on equal prices, and the others are told apart on value only by the 2 prices. The choice is kept because the outlier really does put more stake behind each unit of fees than any other indexer can, so its lead reflects a real difference rather than a fault in the measurement. An indexer with neither query fees nor expected DIPs fees has no ratio, and scores the maximum of 1; it is placed just above the highest real ratio, so it does not push the others down the way an extreme ratio does.

Each indexer is scored per chain, since its prices and freshness only exist on a chain. GET /scores/weighted and POST /get-score report, for each chain an indexer has priced, its total, the 4 factors and the parts behind them, scored against the other indexers that priced that chain, along with the indexer's queries there in the daily job's window (queries, null when the job sent none). IISA gives no overall score: a reader who wants 1 number per indexer can take the average of its chain totals weighted by those queries, counting a null as no queries when the indexer has counts on other chains. When all its counts are null, the plain average of its chain totals is the fallback. That happens for scores from a degraded run or from a job that doesn't send counts yet, including rows carried over from one, and for an indexer that got no gateway queries on any chain it has priced. GET /scores/weighted also reports formula, the constants the scores are worked out with: the power shares are raised to, the weight of each part of quality and value under the name the chains' components give it, the 2 numbers in the daily job's blend of a chain with all the indexer's chains, and the fewest checks a freshness share is taken over, so a reader can show the working without copying them.

Freshness

An indexer can keep answering queries while it has stopped following a chain, so none of the other scores above notice it serving stale data. To catch that, a prober checks once a minute whether each indexer is serving up-to-date data on each chain it offers DIPs on, and sends that minute's outcomes to POST /freshness. Each outcome is checked on its own: one that breaks the rules for an outcome, or repeats an indexer and chain already in the push, is left out, counted in the answer's refused and noted in a warning, and the rest of the minute still counts. The service keeps them in freshness.json on its volume, and GET /freshness reports, for each indexer on each chain, its latest outcome, how it did over the last 10 minutes and its passed and counted checks over the last 28 days.

From the last 10 minutes the service works out a stale mark. An indexer is marked on a chain when it passed fewer than half of the minutes that counted. A minute the service never received, or a check voided by a fault on IISA's or the gateway's side, doesn't count; a minute the service received without an outcome for the indexer counts as not passed if it came before the indexer's first outcome, and doesn't count after that, so a push that leaves indexers out doesn't mark them, unless it goes on leaving them out for long enough, as the next paragraph explains. When fewer than 5 minutes counted, that paragraph's rule decides the mark instead. An indexer seen for the first time starts marked, so it has to pass checks before the mark clears.

Every refused payment is void, since IISA can't tell whether a refusal is the indexer's fault: what the indexer is owed is kept only in its own records. Refusals therefore never touch its score, but they can't keep it pickable either. When fewer than 5 of an indexer's last 10 minutes on a chain counted, it is judged instead on its last 5 counted checks there (passed, failed or found no evidence, the last counting as not passed). It stays pickable only when all 5 fall within the chain's last 60 answered minutes and at least 3 of them passed, and is marked otherwise, however its other checks came to be void or missing. An answered minute is one in which some check on the chain passed or failed, and so went through the gateway. A minute that isn't answered on a chain, as during a gateway outage or while the prober's API key is out of funds, changes no mark there, even while indexers with no allocation old enough to check go on finding no evidence, which the fetcher works out without asking the gateway. An indexer none of whose checks has ever counted keeps the mark it started with. The fetcher's warning about refused payments names each indexer refusing them at most once every 10 minutes, and GET /freshness shows, for each indexer on each chain, its last counted check (last_counted_minute) and how many of its last 5 counted checks are recent enough (recent_counted) and passed (recent_passed).

The mark changes selection on every chain except those listed, comma-separated, in IISA_FRESHNESS_EXEMPT_CHAINS, which is empty by default. On a chain that isn't exempt, an indexer that is marked, or that has no freshness record yet, isn't picked for new agreements and doesn't set the scale the other candidates are scored on, the way an indexer that declined the subgraph is treated, unless no fresh candidate is left to fill the group. Dipper needs the number of indexers it asked for, so the places no fresh candidate can take go to the stale candidates with the best scores, scored again with them setting each metric's scale, and the service logs a warning naming the chain and the indexers. A stale candidate never replaces a member. Otherwise, since a stale candidate drops out of the rows the other indexers are scaled against, exactly as a declined one does, the score of an indexer already in the group can shift a little when that happens. The mark never takes an agreement away from an indexer that already holds one. On an exempt chain the mark is only recorded and reported. On a chain where no indexer's allocation is old enough to be checked yet (7 days by default), every check finds no evidence, so none passes or fails and the chain is picked from as if it were exempt, as the next paragraph explains. Once some indexer's checks there pass or fail, an indexer still finding no evidence counts as not passed, so it stays marked until its allocation is old enough.

Marks only stay trustworthy while checks keep going through the gateway, so a chain is enforced only while some check on it passed or failed within the last IISA_FRESHNESS_MAX_AGE_MINUTES minutes by the service's clock, 60 by default. A void check doesn't keep it enforced, and nor does one that found no evidence, which the fetcher works out without asking the gateway. When the prober stops, or keeps pushing but nothing goes through the gateway because it is down or the prober's key is out of funds, the marks stop changing, so once that limit passes the chain is picked from as if it were exempt, and selections there log a warning saying how long it has been since a check there passed or failed, at most once every 10 minutes per chain, with how many were held back since the last. The first check to pass or fail again brings enforcement back. A chain where no check passed or failed in the 28 days kept, such as one the prober doesn't check, every chain while the prober is off, or one where every check finds no evidence, is picked from as if it were exempt too, with only a debug line in the log: enforcement waits for checks to arrive rather than leaving nobody to pick. GET /freshness lists every chain with freshness records, and every exempt chain, and shows for each whether it isn't exempt (enforced), whether its marks are deciding selection right now (enforcing), when its earliest freshness record began (first_minute), whether freshness counts in its scores (weighed), the last minute a check on it passed or failed (last_answered_minute) and the last minute one counted, no evidence included (last_counted_minute).

Freshness is also 1 of the factors, on the request's chain. An indexer's score there is its share of checks passed over the last 28 days, taken over at least 10,080 checks (7 days of minutes) so a newcomer can't score 1 on a good hour, then raised to the 6th power, so 90% scores about 0.53 and 80% about 0.26. An indexer with no record on the chain scores 0. On a chain whose earliest freshness record is less than 7 days old, or that has none, such as every chain while the prober is off, freshness is left out and counts as 1, rather than every indexer scoring 0. The same goes for a chain where no check has passed or failed within IISA_FRESHNESS_MAX_AGE_MINUTES by the service's clock, the rule that also stops its marks deciding selection, since the scores go stale with the checks: once the prober stops, or nothing goes through the gateway, freshness drops out there until a check passes or fails again. An exempt chain ignores freshness altogether, the factor as well as the mark, since a chain is exempted when its checks can't be trusted or passed.

The record of each selection

A selection's reply names only the picked indexers and their prices. To show why those indexers were picked, the service writes 1 record for every selection it answers. The record holds the request, and every candidate with the total it was ranked on, the 4 factors and the parts behind them, its operator and location, whether it says it has synced the deployment and what selection made of that claim, the pool it stood in, whether it was picked and, if it wasn't, why. It also lists the indexers the price checks left out before scoring with the check each one failed, each change made to the group in the order it happened, and the final group with whether it is spread over 2 operators and 2 locations. Every number is read from the table selection ranked the candidates on, not worked out again afterwards, and a picked indexer's numbers are the ones it was picked on, so the record can't disagree with the pick. Writing it changes nothing in the reply.

Each record is logged as 1 JSON line starting selection_record. The most recent records are also kept in memory and served by GET /selections?deployment=<deployment ID>, newest first, behind the same token check as the other routes that only read: at most 20 for a deployment and 128 MB in all, the oldest dropped first. On generated scores a record measured about 24 KB for a selection over 40 candidates and 59 KB over 100, so 128 MB holds about 5,100 of the first or 2,200 of the second. They are lost when the service restarts, which leaves the log line as the only copy. They are also only complete while 1 process answers both the selections and the reads, which is how the service runs, as 1 replica with 1 worker; with more, each process would hold only the selections it answered itself.

How long the log line lasts, and whether it arrives whole, depends on the cluster, and neither has been checked on the cluster the service runs on. kubectl logs shows the pod that is running now and reads only its current log file, so it reaches back no further than the last rotation, which the kubelet does at 10 MiB unless the cluster sets another size. Anything older has to come from wherever the cluster ships its logs. A long line may be split or cut on the way there: containerd by default splits a line over 16 KB, which a record of about 30 candidates already is, into pieces that the log collector has to join again, and a collector can limit the size of 1 entry, as Cloud Logging does at about 256 KiB. The service holds the whole line, from its timestamp to its end, to 256 KB.

A record whose line would be over that, which takes several hundred candidates or a request naming thousands of indexers, is logged and kept with truncated set and without its long lists: the candidates, the indexers the price checks left out and the request's own lists. It still names the final group and each change made to it, unless those alone are over the limit, which takes a request naming thousands of members. A warning is logged either way. A request that is refused, or a selection that fails inside the service, leaves no record. The record is written before the reply is sent, so there can be one for a reply the caller never received, for example when the caller had stopped waiting. The service's /openapi.json describes every field of a record and the values each can take.

To read the records in the running pod's current log file:

kubectl logs deployment/iisa | grep -F 'selection_record {' | sed 's/^[^{]*//' | jq .

Repository layout

Path What lives there
src/iisa/ The HTTP service: the score cache, the scoring, and the selection logic.
cronjobs/compute_scores/ The daily measurement job: gateway-traffic replay, metric computation, and the client that pushes scores.
k8s/ The Kubernetes manifests for the service and the daily job.
tests/ The service's tests, run in CI against the exact versions pinned in uv.lock, which the service image installs, under the Python in .python-version that the image runs.
tests/cronjob/ The daily job's tests, run in CI against the exact versions in its requirements.txt, which its image installs. That file is compiled from the ranges in requirements.in with uv pip compile; its first lines give the command.

Running the tests

uv sync --extra dev then pytest runs the service's tests. The daily job's tests in tests/cronjob/ also need its own packages, from uv pip install -r cronjobs/compute_scores/requirements.txt; without them pytest skips that folder and says so at the top of its output.

About

Indexing Indexer Selection Algorithm - Standalone containerized service for The Graph DIPs indexer selection

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages