Add ORDS v0.3 repair data export endpoint

[ardelato]

Description

Adds GET /api/v2/repairs, which exports repair records in Open Repair Data Standard v0.3. The Open Repair Alliance asked for a codified export rather than a periodic database extract, and there's no endpoint for it in the codebase today.

This started in our fork at iFixit, where it's merged and running: iFixit/restarters#58. It isn't a straight cherry-pick. That version sits on an API-client auth layer, a feature flag and Helm config that are specific to our deployment, none of which apply here, so this is rebuilt on develop against your conventions with a handful of fixes on top. Happy to adjust anything that doesn't fit.

Changes

• Endpoint: GET /api/v2/repairs, in the existing v2 group behind auth:api, the same way the Zapier and Repair Together endpoints are already served. JSON or CSV, with updated_since, event_start, event_end, powered, page and per_page. Per-page tops out at 1000 rather than 100 because it's a bulk export. Only approved events on approved groups are visible, and soft-deleted events are excluded. Files: API/RepairController.php, routes/api.php.
• Column mapping: the fourteen standard columns in spec order. year_of_manufacture isn't stored, so it's derived from the event year minus devices.age. country converts groups.country_code from alpha-2 to alpha-3, and product_category_id is a name lookup because idcategories doesn't line up with the standard's ids. Files: Services/Ords/OrdsRecordMapper.php, Helpers/Iso3166.php.
• Vocabulary: config/ords.php follows ORA's published data rather than their tableschema.json, which is stale in places. The standard collapses the screen-size and laptop-size category splits, carries Unknown as a real repair_status value, and drops the "the" from "No way to open the product" as seeded in 2018_11_12_135805. partner_product_category uses the <category> ~ <item_type> shape from your own published rows.
• Redaction: devices.problem, devices.item_type and devices.brand are all volunteer-written, and item_type is validated as a bare string in DeviceController with no vocabulary behind it. All three go through ProblemTextScrubber, which strips HTML and redacts emails, phone numbers, digit runs of 8 or more, and URL query strings.
• Config guard: ORDS_ID_PREFIX and ORDS_DATA_PROVIDER have no defaults, and the endpoint returns 503 while either is blank. The id is the stable key ORA upserts on across releases, so publishing under an unassigned namespace would overwrite another provider's rows. Nothing is exported until you set them.
• Audit log: every export logs the caller, the filters, the row count and the redaction tallies. EnsureAPIToken mints a token for every logged-in user, so any account can reach this, and that log is the only trace afterwards if someone asks who pulled what.

Output was validated against ORA's published tableschema.json. Column names and order match exactly and every declared constraint passes, except their id regex, which uses a hyphen while all of their own published rows use an underscore.

On the free-text columns

The scrubber is best-effort. It takes out contact details and identifiers, but personal names aren't pattern-detectable and stay in. Two bypasses turned up during review, both fixed with regression tests: whitespace or inline markup inside an address (owner@\nexample.com, owner@ex<b>ample</b>.com) defeated the email pattern, and a > inside a quoted HTML attribute ended the tag early and left the address whole.

As written, problem ships on every export with no toggle. That's the call I'd flag hardest for you, since it's your community's data under your terms, not ours. If you'd rather it were opt-in, or dropped entirely for a structured-fields-only export, that's a small change and I'm happy to make it.

Deferred

Three gaps, all of which I'm happy to split out or fold in, whichever you prefer.

updated_since keys off devices.updated_at, so approving an old event leaves its devices invisible to an incremental consumer. Widening it to events.updated_at and groups.updated_at would fix it.

Pagination is offset-based, so a row leaving the set mid-crawl shifts later pages, and a deep crawl is quadratic. Keyset pagination on devices.iddevices would fix both.

The route inherits the app-wide 300/min bucket from RouteServiceProvider. That's generous for a request that runs three full passes over the devices/events/groups/categories join, but I didn't want to guess a limit that might throttle ORA's own crawl.

QA Notes

Run php artisan l5-swagger:generate after pulling. TestCase::get() validates every /api/v2 response against the generated schema, so the suite depends on it.

To exercise it, set ORDS_ID_PREFIX and ORDS_DATA_PROVIDER and pull a full export against the spec. The prefix carries its own separator: ORDS_ID_PREFIX=restarters_ produces restarters_12345. Worth confirming with ORA which prefix is assigned to you before publishing under it, since it's the upsert key.

Tests: 69 covering the config guard, column mapping, both vocabularies, visibility, filters, pagination, CSV output and redaction.

[edwh]

Thanks for this. I've reviewed it with AI assistance, so treat the findings below as a starting point rather than gospel — push back on anything that looks wrong.

Fixes pushed to pr900-fixes (your branch plus one commit). I couldn't push to yours directly — GitHub doesn't allow maintainer edits when the fork belongs to an organisation.

Authorization. I think the right model here is Administrator-only. As written the endpoint returns every device from every approved group to any authenticated caller, and as you noted, EnsureAPIToken gives every logged-in user a permanent token — so in practice that's any account on the site. The Zapier exports require Administrator; the Repair Together ones limit results to the caller's own networks. Network-scoping doesn't fit here since ORA wants the whole dataset, so I've gone with Administrator. Middleware stays auth:api, which was right.

Redaction. Three gaps, all with tests:

• The phone pattern only accepted spaces, brackets, dots and hyphens, so 020/7946/0958, 020,7946,0958 and en-dash forms went out untouched — each group is too short for the 8-digit run to catch. Added / and ,, and folded typographic dashes and full-width digits to ASCII first, alongside your full-width @ handling. Widened the date guard to match, so 15/06/2024 is still left alone. Side effect: a slash-grouped serial now reads [phone removed] rather than [number removed].
• The strict email pass has no digit exclusion, so cost 10@2.50 each matched as an address (the spaced form is fine, as your comment says). Changed the guard to "contains a letter" rather than "numeric TLD" — a numeric-TLD rule would also have stopped user@192.168.1.10 redacting.
• Query-string stripping required an explicit scheme, so www.example.com/parts?gclid=… kept its tracking codes. Added www. and bare host.tld/path arms; the bare one needs a path so prose isn't cut at a question mark.

event_end. The date-only check read H:i:s off the parsed value, which is 00:00:00 for a real timestamp landing on midnight in its own offset — 2024-06-15T00:00:00-05:00 became a bound of 2024-06-16 04:59:59 UTC, about a day too wide. Now decided from the raw input. Your existing cases are unchanged.

Config. id_prefix and data_provider normalised in one place so the guard and the emitted value can't drift, and non-strings count as unset — ORDS_ID_PREFIX=true reaches config() as a boolean and (string) true is "1", which passed the guard. false, null and empty already failed closed.

updated_since. I've done this one rather than leaving it deferred, since ORA will be pulling incrementally and it loses data quietly. Keyed on devices.updated_at alone, approving an old event or its group made devices exportable without touching the device row, so they never appeared in an incremental crawl again. Both the filter and the watermark now use GREATEST(devices.updated_at, events.updated_at, groups.updated_at) — all three are ON UPDATE CURRENT_TIMESTAMP, so they maintain themselves. Your existing updated_since test now backdates all three rows, since "unchanged" means something wider than it did.

The other two deferred items look right to me as they stand.

One thing I want to think about rather than change: problem going out on every export. Group name plus event date plus a hand-typed item_type can still point at a person even with contact details stripped. I may come back to you about making it opt-in.

[sonarqubecloud]

Quality Gate passed

Issues
13 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[ardelato]

I have added both commits into the branch.

The Administrator gate is the right call and I've got no pushback. I originally thought about adding it but didn't want to overtighten things before it was confirmed. Given we'd want to mirror the Zapier path, ORA pulling with an admin token makes sense.

I checked and confirmed the redaction gaps. The / and , would have been a problem, since 020/7946/0958 is an ordinary way to write a number and it was going out whole. Slash-grouped serials now reading [phone removed] instead of [number removed] seems like the right trade. The label's wrong but the digits are gone.

event_end was a genuine bug and I missed it. Reading H:i:s off the parsed value meant any real timestamp landing on midnight in its own offset got widened by about a day.

Same with ORDS_ID_PREFIX=true. I didn't think about env() casting the literal, and (string) true being "1" slipping past a guard meant to stop an unassigned namespace is the exact failure it was there to catch.

Thanks for doing updated_since instead of leaving it deferred. One note for later: none of those three columns are indexed, so the filter was already a full scan and this doesn't make it worse.

There's also one small commit of mine on top, docs(ords), which just moves a docblock. The one explaining why the shared test fixture is an Administrator got stranded on backdate() when the two commits interleaved. Comments only.

On problem, I think you're right to be uneasy. Scrubbing gets contact details and identifiers out but it can't do anything about context. Group name plus event date plus a hand-typed item_type still narrows it a long way.

We hit the same question in our fork and went with opt-in: ORDS_INCLUDE_PROBLEM off by default, and ORDS_SCRUB_PROBLEM as a separate toggle, so turning the endpoint on and publishing the free-text column stay two decisions. values.yaml. That's Helm config so it doesn't port over directly, but the shape would. We could do something like that here.