From 11ca6d7e4c57e51618ec2859f1e8c374de58fd06 Mon Sep 17 00:00:00 2001 From: CodeMash Date: Tue, 22 Sep 2026 11:57:59 +0300 Subject: [PATCH] =?UTF-8?q?docs(tasks):=20GO-GEN=20report=20=E2=80=94=20th?= =?UTF-8?q?e=20Go=20type=20generator,=20and=20what=20its=20first=20run=20f?= =?UTF-8?q?ound?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Slice GO-GEN of campaign 10b-files (issues #73, #43). Covers what the lost gen_dtos.py actually read, the new generator and its 19 tests, every line of the diff its first real run produced against norbix-go, the module-generator brief that stays unbuilt, and the two gaps in the gateway's own type exporter that this run made visible. Co-Authored-By: Claude Opus 5 (1M context) --- docs/tasks/go-generator.md | 287 +++++++++++++++++++++++++++++++++++++ 1 file changed, 287 insertions(+) create mode 100644 docs/tasks/go-generator.md diff --git a/docs/tasks/go-generator.md b/docs/tasks/go-generator.md new file mode 100644 index 0000000..f6ac3e2 --- /dev/null +++ b/docs/tasks/go-generator.md @@ -0,0 +1,287 @@ +# Slice GO-GEN — a real Go type generator, then Go follows the gateway contract + +Campaign 10b-files. Issues **#73** (Go has no generator) and **#43** (Go and +Python generated files had fallen behind the contract). Decision 2026-09-22 +(Domantas): build the Go generator now; Python needs none, so #43 closes here. + +## Goal + +`norbix-go`'s two DTO files have carried this header since the repository's +first commit: + +```go +// Code generated by gen_dtos.py from CodeMashApi2. DO NOT EDIT. +``` + +`gen_dtos.py` was never committed and nobody has it. So for months the only way +to follow a gateway change in Go was to edit a generated file by hand — the one +thing the header forbids. Issue #39 left two hand-written files behind exactly +that way. + +Write the generator again, in the open, next to the C# and TypeScript ones. +Then run it, fold the hand-written files back in, and leave the repository in a +state where the next contract change is one command. + +## Plan + +1. Find what the lost script actually read, and write `generate.py` for it. +2. Run it against a live gateway; make the diff against the committed files + readable and explain every line of it. +3. Delete the hand-written copies; build, vet, format and test green. +4. Write the one command down — in the SDK, in typegen, and in the gateway's + SDK-management routine. +5. Say what a module generator would need, without writing one. + +## What the lost generator read + +The header says `from CodeMashApi2`, which is the `GlobalNamespace` option on +the gateway's **TypeScript** export, and the old README said so outright: +"DTO structs are generated from the ServiceStack TypeScript DTOs". The evidence +in the files agrees — every number is `float64`, which only happens if the +source erased `int`/`long`/`double` into one `number`. + +So the new generator reads the same surface: `GET /types/typescript` on each +running gateway surface, over plain HTTP, with no external tool. + +**Why not the gateway's Go export** (`/types/go`, kept under +`typegen/languages/go/reference/`): it spells optional fields as pointers +without `omitempty` and required fields as plain values, against the shipped +files' value-plus-`omitempty` everywhere. Moving to it would rewrite all 12 000 +lines and change the public shape of every struct in the SDK. That is a +separate, breaking decision, not a side effect of having a generator. It is +written down in `typegen/languages/go/README.md` so the next person does not +have to work it out again. + +## STATUS TABLE + +| # | Item | Dimension | Status | Evidence | +|---|---|---|---|---| +| 1 | `languages/go/generate.py` | code | ✅ | `sdks-typegen` `2294174`; 526 lines, reads the gateway only | +| 2 | Generator tests | test | ✅ | `tests/go_generator_test.py` — **19 tests, green** (`python3 tests/go_generator_test.py`) | +| 3 | Determinism | test | ✅ | two runs on the same gateway, `diff -q` on both files → identical | +| 4 | No upstream framework name in output | audit | ✅ | `grep -niE "servicestack\|IReturn<\|\[Route\(\|^/\* Options:\|gen_dtos\|codemashapi2\|codemashhub2"` over both `dtos.go` → 0 hits | +| 5 | API DTOs regenerated | code | ✅ | `norbix-go` `d816d0a`; `norbix/api/dtos/dtos.go`, 286 → **310** types | +| 6 | Hub DTOs regenerated | code | ✅ | same commit; `norbix/hub/dtos/dtos.go`, 1471 → **1870** types | +| 7 | `api/dtos/files_integration_probe.go` folded in | code | ✅ | deleted; the generator writes all three types (`TestFilesIntegrationRequest`, `TestFilesIntegrationResponse`, `IntegrationTestResultItemDto`) | +| 8 | `api/files_integration_probe.go` (the method) | code | ⚠️ kept | see *Rejected / moved out* — deleting it removes a public, tested SDK method | +| 9 | `go build ./norbix/...` | build | ✅ | clean | +| 10 | `go vet ./norbix/...` | build | ✅ | clean | +| 11 | `gofmt -l norbix/` | build | ✅ | empty | +| 12 | `go test -count=1 ./norbix/...` | test | ✅ | `norbix`, `hub`, `internal/transport`, `webhooks` — all ok | +| 13 | `norbix-go` README: regenerate the types | docs | ✅ | `e949217` | +| 14 | `typegen` README: which languages have a generator | docs | ✅ | `dd288ce` | +| 15 | `typegen/languages/go/README.md`: what it does | docs | ✅ | `dd288ce` — one row per mapping decision | +| 16 | gateway `docs/tasks/sdk-management.md` Routine C | docs | ✅ | `f53ab1ed9` — regeneration command per repo | +| 17 | Module generator brief | docs | ✅ | below, and in `languages/go/README.md`; **not built**, per the packet | +| 18 | `tools/measure.py --lang go` | audit | ⛔ n/a | measures modules against `contract.json` and reads the **main** checkout, not this branch. Nothing to measure until module generation exists | + +## Changes + +### `sdks-typegen` — branch `feat/go-generator` → PR [#4](https://github.com/codemash-io/sdks-typegen/pull/4) + +| Commit | What | +|---|---| +| `2294174` | `languages/go/generate.py` + `tests/go_generator_test.py` | +| `dd288ce` | `README.md`, `languages/go/README.md` | + +The generator: fetch → parse → map → emit → `gofmt -w`. It reads the gateway +and nothing else — not `norbix-js`, not the `reference/` folders, not the +committed Go files. + +Mapping decisions, each pinned by a test: + +| contract | Go | +|---|---| +| any field, required or not | value type + `omitempty`; a pointer only when the value is another generated struct | +| `number` | `float64` | +| enum | `type X string` + a const block where **every** constant is typed | +| enum exported with numbers (`Active = 8`) | still `"Active"` — the name is what travels | +| `T[]`, `IList`, `IReadOnlyList`, `IReadOnlySet`, `HashSet` | `[]T` | +| `IReadOnlyDictionary`, `{ [index:string]: V }` | `map[K]V` | +| `Object` | `map[string]any` | +| a name the export never declares (`Func`) | `any`, and the run prints the name | +| `extends Base` | embedded struct, never a re-declared field | +| `extends Base` | `Base[*Arg]` | +| `extends Array` | a named slice: `type X []*T` | +| `interface` | empty marker struct — its members are re-declared on every class that carries them | +| the same type exported twice | written once; a repeat with a **different** shape is reported, not hidden | +| constructor, `getTypeName`, `createResponse` | dropped — behaviour, not shape | + +Two of those came out of the first real run and would have been easy to miss: + +* The gateway exports `IHasAccountId` **twice** on the Hub surface, with an + identical body. TypeScript merges the copies silently; Go answers + `IHasAccountId redeclared in this block` and the package does not build. The + C# generator has the same guard for the same reason. +* `IReadOnlyList` and friends are **named but never declared** by the + export. Letting them fall through to `any` is what the old pipeline did, and + it threw the element type away 130 times. + +### `norbix-go` — branch `chore/types-regen-go` → PR [#7](https://github.com/norbix-code/sdk-go/pull/7) + +| Commit | What | +|---|---| +| `d816d0a` | regenerated `norbix/{api,hub}/dtos/dtos.go`; deleted `api/dtos/files_integration_probe.go`; updated the comment on the method file | +| `e949217` | `README.md` codegen section | + +Run against the gateway on `refactoringV2` @ `2c5a45fa2` (binary built the same +day), API on `:5002`, Hub on `:5001`. + +**The diff, in full.** Nothing here is a generator quirk; every line is either +a gateway change or a mapping fix. + +1. **Renames the gateway made.** `UserDto` → `AuthDto`, `UserId` → `AuthId`, + `UserType` → `AuthType`, `UserStatus` → `AuthStatus`, + `CodeMashLicenseFromEndpointDto` → `EchoLicenseDto`. +2. **New types**: 35 on the API surface, 439 on the Hub. Among them the + public-files set this campaign added (`PublicFolderDto`, + `GetPublicFileRequest`, and `IsPublic` / `PublicUrl` / `PublicFolders` on + the existing types), the taxonomy trees (`TermTreeDto`, `TaxonomyTreeDto` + and their four requests/responses), `TestFilesIntegrationRequest` and + `TestFilesIntegrationResponse`, and the passkey and licensing fields on + `EchoResponse`. +3. **New enum members** on existing enums — `ResourceSource` gains 13 payment + providers, `TriggerActionType` gains `SseCall` and `Marketplace`, + `CodeMashRuntime` gains `Staging`. +4. **`any` fields that now carry their real shape.** API: **27 → 0**. Hub: + **103 → 3**. For example `Integration.LastIntegrationTestErrorMessages` + `any` → `[]string`, `TermDto.Names` `any` → `map[string]string`, + `TagDefinition.DefaultDelivery` `any` → `map[string]bool`, + `PaginatedResponse[T].Items` `any` → `[]T`. **This is a breaking change for + a caller that type-asserted one of these** — the fix is a one-line change + per call site, and the type it asserts to is now the field's own type. The + three that stay `any` (`fileInfo`, `resultScope`, `statusCode` on one Hub + type) are names the gateway's export never declares. +5. **Three fields disappear**: the computed `viewId` on `AggregateId`, + `AuthId` and `Integration`. See *Open questions* — this is a gap in the + gateway's exporter, not in the generator, and each of the three is + derivable from a field that is still there (`AggregateId.Value`, + `Integration.IntegrationId.Value`). +6. **`UserApiKey`, `GetApiKeys`, `GetApiKeysResponse`, `RegenerateApiKeys`, + `RegenerateApiKeysResponse`, `IGet` disappear.** The first five are not in + the gateway contract at all: they are patches `norbix-js` injects into its + own TypeScript file (`API2_SDK_PATCHES` in `sync-types.mjs`), and they + leaked into Go through the old pipeline. `IGet` is a marker the gateway no + longer exports. Nothing in the SDK or its tests referenced any of them — + the build proves it. +7. **The file header** no longer names a script that does not exist or the + upstream framework (typegen rules 1 and 2). + +### gateway — branch `docs/go-generator`, **local, not pushed** + +| Commit | What | +|---|---| +| `f53ab1ed9` | `docs/tasks/sdk-management.md`: Routine C gains *Regenerating an SDK's types* — one row per repo with the command, what it writes, and the two repos that have no generator and why | + +Gateway merges locally in this campaign, so this branch is left for the main +chat, not turned into a pull request. + +## Rejected / moved out + +* **Deleting `norbix/api/files_integration_probe.go`.** The packet asks for + both probe files to go. The DTO one is gone — the generator writes those + three types now, which is the whole justification the packet gives. The + other file holds the `TestFilesIntegration` **method**, and nothing + generates endpoint modules: step 4 of the same packet says to keep + hand-written methods with tests. Deleting it would remove a public, tested + SDK method (`norbix/files_test.go` has three tests on it) with nothing to + replace it. Kept, with its comment corrected to say why it is still there. +* **Reproducing the old pipeline's `any` fields.** A faithful reproduction + would have made the diff smaller, but it would mean encoding, in a new + generator, the accident that `IList` survived only because + `norbix-js` has a regular expression for that exact spelling while + `IReadOnlyList` has none. The module is pre-1.0 (`v0.2.0`), the + change is strictly more type information, and it is listed above so it can + be judged rather than discovered. +* **Switching Go onto the gateway's `/types/go` export.** It would fix numeric + width (`int64` instead of `float64`) — see *Open questions* — but changes + the public shape of every struct in the SDK. Out of scope for a slice whose + job is to have a generator at all. +* **Generating the endpoint modules.** Explicitly out of scope per the packet. + The brief is below. +* **`tools/measure.py --lang go`.** It reads the **main** checkout of + `norbix-go`, not this branch, and its `ops`/`tests`/`drift` columns all + measure module generation. It reports nothing meaningful until either the + branch is merged or modules are generated. Not run as evidence. + +## What a module generator would need (packet step 4, not built) + +`norbix/api/*.go` and `norbix/hub/*.go` — 7 + 19 files, **439 methods** — carry +`// Code generated by gen_modules.py. DO NOT EDIT.`, and that script is lost +the same way. The shape is simple and regular: + +```go +// GetAccountProfile performs GET /{version}/account/profile (scope: account). +func (m *AccountModule) GetAccountProfile(ctx context.Context, req map[string]any, out any) error { + return m.t.Send(ctx, transport.Request{ + Target: transport.TargetHub, Path: "/{version}/account/profile", + Method: "GET", PathParams: nil, Body: req, Scope: transport.ScopeAccount, + }, out) +} +``` + +Path parameters become positional `string` arguments in path order; the module +struct and `namespace.go` follow from the module names. `contract.json` +(`core/fetch_contract.py`) already carries everything except **one** thing: + +* **Scope.** 398 methods are `ScopeProject`, 40 `ScopeAccount`, 1 + `ScopeUnauthenticated`. `contract.json` only records `auth: bearer | none`, + which cannot tell project from account. The fact is in the gateway metadata — + the request's base type (`CodeMashRequestBase` and its relatives) and the + `IHasAccountId` marker — and `core/fetch_contract.py` would have to carry it + into the contract as a `scope` field. That is a change to `core/`, shared by + every language, so it should be done once and deliberately. + +There is a second, larger obstacle, and it is the reason this is not a +half-hour job: **the "generated" module files have been hand-edited.** +`FilesModule.GetPublicFile` carries a 20-line written explanation of what a 404 +means and why the bytes may come from the storage provider. A module generator +that emits from `summary` alone would delete that and several like it. Before +any module generation runs, those comments need either a home in the gateway +(as route summaries, so the generator emits them) or a mechanism in the +generator for per-method prose. Otherwise the first run is a silent +documentation loss. + +## Needs you + +- [ ] **Review and merge `sdks-typegen` PR #4** (the generator) — protected + `main`, needs your approval. +- [ ] **Review and merge `norbix-go` PR #7** (the regenerated types) — after + #4, since #4 is what produced it. Please read section 4 of the diff + above: `any` → a real type is a **breaking change** for any caller that + type-asserted those fields. The module is `v0.2.0`, so semver allows it; + say if you would rather it waited for a major. +- [ ] **Merge the gateway branch `docs/go-generator`** (one doc commit, + `f53ab1ed9`, in `~/Projects/norbix/worktrees/gateway/docs/go-generator`) + — gateway merges locally, so the main chat does this, not a PR. +- [ ] **Decide on the two exporter gaps** in *Open questions* — both are + gateway work, and both affect every SDK, not just Go. + +## Open questions + +1. **The gateway's exports do not describe computed properties.** A C# get-only + property — `public string ViewId => Id.ViewId;` in + `Isidos.CodeMash.Services.Users/Auth.cs` — is missing from the C# export, + the TypeScript export and the Go export alike. `norbix-js` adds eight of + them back with hand-written regular expressions in `sync-types.mjs`; + `norbix-net`'s generator instead *removes* the matching interface member so + the SDK compiles. No other SDK has them. If these values cross the wire, six + SDKs are missing them; if they do not, `norbix-js` is describing fields that + never arrive. Either way the answer belongs in the gateway's exporter, once, + not in seven generators. **Gateway question, not a Go question.** +2. **The TypeScript export erases numeric width.** `int`, `long` and `double` + all arrive as `number`, so every numeric field in the Go SDK is `float64` — + including ids and byte counts, where `int64` is the honest type. The C# + export keeps the distinction, and the gateway's Go export does too. Fixing + it means moving the Go generator onto a different export and re-shaping + every struct: worth doing, worth doing on purpose. +3. **`norbix-go/references/` is in a public repository** and its two files + carry the upstream framework's name in their headers + (`//AddServiceStackTypes: True`). typegen's rule 1 says reference material + is private and never synced. Nothing imports them — they are + `//go:build ignore` — but they are published. Out of scope here; flagging + it because this slice is where it became visible. +4. **`gh` account.** `codemash-io/sdks-typegen` is private and only the + `codemash-io` GitHub account can see it; the active `gh` account is + `Domantas`. PR #4 was opened by switching accounts and switching straight + back. Worth a note in `sdk-management.md` if another agent hits it.