Skip to content

docs(tasks): GO-GEN report — the Go type generator #18

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Domantas merged 1 commit into from
Sep 22, 2026
Merged

docs(tasks): GO-GEN report — the Go type generator #18

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
287 changes: 287 additions & 0 deletions docs/tasks/go-generator.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,287 @@
# Slice GO-GEN — a real Go type generator, then Go follows the gateway contract

Campaign 10b-files. Issues **#73** (Go has no generator) and **#43** (Go and
Python generated files had fallen behind the contract). Decision 2026-09-22
(Domantas): build the Go generator now; Python needs none, so #43 closes here.

## Goal

`norbix-go`'s two DTO files have carried this header since the repository's
first commit:

```go
// Code generated by gen_dtos.py from CodeMashApi2. DO NOT EDIT.
```

`gen_dtos.py` was never committed and nobody has it. So for months the only way
to follow a gateway change in Go was to edit a generated file by hand — the one
thing the header forbids. Issue #39 left two hand-written files behind exactly
that way.

Write the generator again, in the open, next to the C# and TypeScript ones.
Then run it, fold the hand-written files back in, and leave the repository in a
state where the next contract change is one command.

## Plan

1. Find what the lost script actually read, and write `generate.py` for it.
2. Run it against a live gateway; make the diff against the committed files
readable and explain every line of it.
3. Delete the hand-written copies; build, vet, format and test green.
4. Write the one command down — in the SDK, in typegen, and in the gateway's
SDK-management routine.
5. Say what a module generator would need, without writing one.

## What the lost generator read

The header says `from CodeMashApi2`, which is the `GlobalNamespace` option on
the gateway's **TypeScript** export, and the old README said so outright:
"DTO structs are generated from the ServiceStack TypeScript DTOs". The evidence
in the files agrees — every number is `float64`, which only happens if the
source erased `int`/`long`/`double` into one `number`.

So the new generator reads the same surface: `GET /types/typescript` on each
running gateway surface, over plain HTTP, with no external tool.

**Why not the gateway's Go export** (`/types/go`, kept under
`typegen/languages/go/reference/`): it spells optional fields as pointers
without `omitempty` and required fields as plain values, against the shipped
files' value-plus-`omitempty` everywhere. Moving to it would rewrite all 12 000
lines and change the public shape of every struct in the SDK. That is a
separate, breaking decision, not a side effect of having a generator. It is
written down in `typegen/languages/go/README.md` so the next person does not
have to work it out again.

## STATUS TABLE

| # | Item | Dimension | Status | Evidence |
|---|---|---|---|---|
| 1 | `languages/go/generate.py` | code | ✅ | `sdks-typegen` `2294174`; 526 lines, reads the gateway only |
| 2 | Generator tests | test | ✅ | `tests/go_generator_test.py` — **19 tests, green** (`python3 tests/go_generator_test.py`) |
| 3 | Determinism | test | ✅ | two runs on the same gateway, `diff -q` on both files → identical |
| 4 | No upstream framework name in output | audit | ✅ | `grep -niE "servicestack\|IReturn<\|\[Route\(\|^/\* Options:\|gen_dtos\|codemashapi2\|codemashhub2"` over both `dtos.go` → 0 hits |
| 5 | API DTOs regenerated | code | ✅ | `norbix-go` `d816d0a`; `norbix/api/dtos/dtos.go`, 286 → **310** types |
| 6 | Hub DTOs regenerated | code | ✅ | same commit; `norbix/hub/dtos/dtos.go`, 1471 → **1870** types |
| 7 | `api/dtos/files_integration_probe.go` folded in | code | ✅ | deleted; the generator writes all three types (`TestFilesIntegrationRequest`, `TestFilesIntegrationResponse`, `IntegrationTestResultItemDto`) |
| 8 | `api/files_integration_probe.go` (the method) | code | ⚠️ kept | see *Rejected / moved out* — deleting it removes a public, tested SDK method |
| 9 | `go build ./norbix/...` | build | ✅ | clean |
| 10 | `go vet ./norbix/...` | build | ✅ | clean |
| 11 | `gofmt -l norbix/` | build | ✅ | empty |
| 12 | `go test -count=1 ./norbix/...` | test | ✅ | `norbix`, `hub`, `internal/transport`, `webhooks` — all ok |
| 13 | `norbix-go` README: regenerate the types | docs | ✅ | `e949217` |
| 14 | `typegen` README: which languages have a generator | docs | ✅ | `dd288ce` |
| 15 | `typegen/languages/go/README.md`: what it does | docs | ✅ | `dd288ce` — one row per mapping decision |
| 16 | gateway `docs/tasks/sdk-management.md` Routine C | docs | ✅ | `f53ab1ed9` — regeneration command per repo |
| 17 | Module generator brief | docs | ✅ | below, and in `languages/go/README.md`; **not built**, per the packet |
| 18 | `tools/measure.py --lang go` | audit | ⛔ n/a | measures modules against `contract.json` and reads the **main** checkout, not this branch. Nothing to measure until module generation exists |

## Changes

### `sdks-typegen` — branch `feat/go-generator` → PR [#4](https://github.com/codemash-io/sdks-typegen/pull/4)

| Commit | What |
|---|---|
| `2294174` | `languages/go/generate.py` + `tests/go_generator_test.py` |
| `dd288ce` | `README.md`, `languages/go/README.md` |

The generator: fetch → parse → map → emit → `gofmt -w`. It reads the gateway
and nothing else — not `norbix-js`, not the `reference/` folders, not the
committed Go files.

Mapping decisions, each pinned by a test:

| contract | Go |
|---|---|
| any field, required or not | value type + `omitempty`; a pointer only when the value is another generated struct |
| `number` | `float64` |
| enum | `type X string` + a const block where **every** constant is typed |
| enum exported with numbers (`Active = 8`) | still `"Active"` — the name is what travels |
| `T[]`, `IList<T>`, `IReadOnlyList<T>`, `IReadOnlySet<T>`, `HashSet<T>` | `[]T` |
| `IReadOnlyDictionary<K,V>`, `{ [index:string]: V }` | `map[K]V` |
| `Object` | `map[string]any` |
| a name the export never declares (`Func<IDisposable>`) | `any`, and the run prints the name |
| `extends Base` | embedded struct, never a re-declared field |
| `extends Base<Arg>` | `Base[*Arg]` |
| `extends Array<T>` | a named slice: `type X []*T` |
| `interface` | empty marker struct — its members are re-declared on every class that carries them |
| the same type exported twice | written once; a repeat with a **different** shape is reported, not hidden |
| constructor, `getTypeName`, `createResponse` | dropped — behaviour, not shape |

Two of those came out of the first real run and would have been easy to miss:

* The gateway exports `IHasAccountId` **twice** on the Hub surface, with an
identical body. TypeScript merges the copies silently; Go answers
`IHasAccountId redeclared in this block` and the package does not build. The
C# generator has the same guard for the same reason.
* `IReadOnlyList<T>` and friends are **named but never declared** by the
export. Letting them fall through to `any` is what the old pipeline did, and
it threw the element type away 130 times.

### `norbix-go` — branch `chore/types-regen-go` → PR [#7](https://github.com/norbix-code/sdk-go/pull/7)

| Commit | What |
|---|---|
| `d816d0a` | regenerated `norbix/{api,hub}/dtos/dtos.go`; deleted `api/dtos/files_integration_probe.go`; updated the comment on the method file |
| `e949217` | `README.md` codegen section |

Run against the gateway on `refactoringV2` @ `2c5a45fa2` (binary built the same
day), API on `:5002`, Hub on `:5001`.

**The diff, in full.** Nothing here is a generator quirk; every line is either
a gateway change or a mapping fix.

1. **Renames the gateway made.** `UserDto` → `AuthDto`, `UserId` → `AuthId`,
`UserType` → `AuthType`, `UserStatus` → `AuthStatus`,
`CodeMashLicenseFromEndpointDto` → `EchoLicenseDto`.
2. **New types**: 35 on the API surface, 439 on the Hub. Among them the
public-files set this campaign added (`PublicFolderDto`,
`GetPublicFileRequest`, and `IsPublic` / `PublicUrl` / `PublicFolders` on
the existing types), the taxonomy trees (`TermTreeDto`, `TaxonomyTreeDto`
and their four requests/responses), `TestFilesIntegrationRequest` and
`TestFilesIntegrationResponse`, and the passkey and licensing fields on
`EchoResponse`.
3. **New enum members** on existing enums — `ResourceSource` gains 13 payment
providers, `TriggerActionType` gains `SseCall` and `Marketplace`,
`CodeMashRuntime` gains `Staging`.
4. **`any` fields that now carry their real shape.** API: **27 → 0**. Hub:
**103 → 3**. For example `Integration.LastIntegrationTestErrorMessages`
`any` → `[]string`, `TermDto.Names` `any` → `map[string]string`,
`TagDefinition.DefaultDelivery` `any` → `map[string]bool`,
`PaginatedResponse[T].Items` `any` → `[]T`. **This is a breaking change for
a caller that type-asserted one of these** — the fix is a one-line change
per call site, and the type it asserts to is now the field's own type. The
three that stay `any` (`fileInfo`, `resultScope`, `statusCode` on one Hub
type) are names the gateway's export never declares.
5. **Three fields disappear**: the computed `viewId` on `AggregateId`,
`AuthId` and `Integration`. See *Open questions* — this is a gap in the
gateway's exporter, not in the generator, and each of the three is
derivable from a field that is still there (`AggregateId.Value`,
`Integration.IntegrationId.Value`).
6. **`UserApiKey`, `GetApiKeys`, `GetApiKeysResponse`, `RegenerateApiKeys`,
`RegenerateApiKeysResponse`, `IGet` disappear.** The first five are not in
the gateway contract at all: they are patches `norbix-js` injects into its
own TypeScript file (`API2_SDK_PATCHES` in `sync-types.mjs`), and they
leaked into Go through the old pipeline. `IGet` is a marker the gateway no
longer exports. Nothing in the SDK or its tests referenced any of them —
the build proves it.
7. **The file header** no longer names a script that does not exist or the
upstream framework (typegen rules 1 and 2).

### gateway — branch `docs/go-generator`, **local, not pushed**

| Commit | What |
|---|---|
| `f53ab1ed9` | `docs/tasks/sdk-management.md`: Routine C gains *Regenerating an SDK's types* — one row per repo with the command, what it writes, and the two repos that have no generator and why |

Gateway merges locally in this campaign, so this branch is left for the main
chat, not turned into a pull request.

## Rejected / moved out

* **Deleting `norbix/api/files_integration_probe.go`.** The packet asks for
both probe files to go. The DTO one is gone — the generator writes those
three types now, which is the whole justification the packet gives. The
other file holds the `TestFilesIntegration` **method**, and nothing
generates endpoint modules: step 4 of the same packet says to keep
hand-written methods with tests. Deleting it would remove a public, tested
SDK method (`norbix/files_test.go` has three tests on it) with nothing to
replace it. Kept, with its comment corrected to say why it is still there.
* **Reproducing the old pipeline's `any` fields.** A faithful reproduction
would have made the diff smaller, but it would mean encoding, in a new
generator, the accident that `IList<string>` survived only because
`norbix-js` has a regular expression for that exact spelling while
`IReadOnlyList<string>` has none. The module is pre-1.0 (`v0.2.0`), the
change is strictly more type information, and it is listed above so it can
be judged rather than discovered.
* **Switching Go onto the gateway's `/types/go` export.** It would fix numeric
width (`int64` instead of `float64`) — see *Open questions* — but changes
the public shape of every struct in the SDK. Out of scope for a slice whose
job is to have a generator at all.
* **Generating the endpoint modules.** Explicitly out of scope per the packet.
The brief is below.
* **`tools/measure.py --lang go`.** It reads the **main** checkout of
`norbix-go`, not this branch, and its `ops`/`tests`/`drift` columns all
measure module generation. It reports nothing meaningful until either the
branch is merged or modules are generated. Not run as evidence.

## What a module generator would need (packet step 4, not built)

`norbix/api/*.go` and `norbix/hub/*.go` — 7 + 19 files, **439 methods** — carry
`// Code generated by gen_modules.py. DO NOT EDIT.`, and that script is lost
the same way. The shape is simple and regular:

```go
// GetAccountProfile performs GET /{version}/account/profile (scope: account).
func (m *AccountModule) GetAccountProfile(ctx context.Context, req map[string]any, out any) error {
return m.t.Send(ctx, transport.Request{
Target: transport.TargetHub, Path: "/{version}/account/profile",
Method: "GET", PathParams: nil, Body: req, Scope: transport.ScopeAccount,
}, out)
}
```

Path parameters become positional `string` arguments in path order; the module
struct and `namespace.go` follow from the module names. `contract.json`
(`core/fetch_contract.py`) already carries everything except **one** thing:

* **Scope.** 398 methods are `ScopeProject`, 40 `ScopeAccount`, 1
`ScopeUnauthenticated`. `contract.json` only records `auth: bearer | none`,
which cannot tell project from account. The fact is in the gateway metadata —
the request's base type (`CodeMashRequestBase` and its relatives) and the
`IHasAccountId` marker — and `core/fetch_contract.py` would have to carry it
into the contract as a `scope` field. That is a change to `core/`, shared by
every language, so it should be done once and deliberately.

There is a second, larger obstacle, and it is the reason this is not a
half-hour job: **the "generated" module files have been hand-edited.**
`FilesModule.GetPublicFile` carries a 20-line written explanation of what a 404
means and why the bytes may come from the storage provider. A module generator
that emits from `summary` alone would delete that and several like it. Before
any module generation runs, those comments need either a home in the gateway
(as route summaries, so the generator emits them) or a mechanism in the
generator for per-method prose. Otherwise the first run is a silent
documentation loss.

## Needs you

- [ ] **Review and merge `sdks-typegen` PR #4** (the generator) — protected
`main`, needs your approval.
- [ ] **Review and merge `norbix-go` PR #7** (the regenerated types) — after
#4, since #4 is what produced it. Please read section 4 of the diff
above: `any` → a real type is a **breaking change** for any caller that
type-asserted those fields. The module is `v0.2.0`, so semver allows it;
say if you would rather it waited for a major.
- [ ] **Merge the gateway branch `docs/go-generator`** (one doc commit,
`f53ab1ed9`, in `~/Projects/norbix/worktrees/gateway/docs/go-generator`)
— gateway merges locally, so the main chat does this, not a PR.
- [ ] **Decide on the two exporter gaps** in *Open questions* — both are
gateway work, and both affect every SDK, not just Go.

## Open questions

1. **The gateway's exports do not describe computed properties.** A C# get-only
property — `public string ViewId => Id.ViewId;` in
`Isidos.CodeMash.Services.Users/Auth.cs` — is missing from the C# export,
the TypeScript export and the Go export alike. `norbix-js` adds eight of
them back with hand-written regular expressions in `sync-types.mjs`;
`norbix-net`'s generator instead *removes* the matching interface member so
the SDK compiles. No other SDK has them. If these values cross the wire, six
SDKs are missing them; if they do not, `norbix-js` is describing fields that
never arrive. Either way the answer belongs in the gateway's exporter, once,
not in seven generators. **Gateway question, not a Go question.**
2. **The TypeScript export erases numeric width.** `int`, `long` and `double`
all arrive as `number`, so every numeric field in the Go SDK is `float64` —
including ids and byte counts, where `int64` is the honest type. The C#
export keeps the distinction, and the gateway's Go export does too. Fixing
it means moving the Go generator onto a different export and re-shaping
every struct: worth doing, worth doing on purpose.
3. **`norbix-go/references/` is in a public repository** and its two files
carry the upstream framework's name in their headers
(`//AddServiceStackTypes: True`). typegen's rule 1 says reference material
is private and never synced. Nothing imports them — they are
`//go:build ignore` — but they are published. Out of scope here; flagging
it because this slice is where it became visible.
4. **`gh` account.** `codemash-io/sdks-typegen` is private and only the
`codemash-io` GitHub account can see it; the active `gh` account is
`Domantas`. PR #4 was opened by switching accounts and switching straight
back. Worth a note in `sdk-management.md` if another agent hits it.
Loading