[mattdjenkinson]

Organizations can now verify a domain on their own, as an OrganizationDomain, without adding it to one of their projects. Email domain joining accepts a domain that either an OrganizationDomain or a project Domain verifies.

Until now, the only way to prove an organization owned acme.com was to add it as a Domain in one of its projects. That made a company list its email domain in a project, as if it served something there, and tied joining to that project: deleting the project paused joining. Raised in datum-cloud/cloud-portal#1808.

The resource

apiVersion: resourcemanager.miloapis.com/v1alpha1
kind: OrganizationDomain
metadata:
  name: acme.com
  namespace: organization-acme-corp
spec:
  domainName: acme.com
status:
  verification:
    dnsRecord:
      name: _milo-verification.acme.com
      type: TXT
      content: 3f9c0e…           # 32 random hex characters
    lastAttemptTime: …
    nextAttemptTime: …
  conditions:
  - type: Verified
    status: "False"
    reason: RecordNotFound
    message: Add a TXT record named _milo-verification.acme.com with the value 3f9c0e… at your DNS provider. …

• Naming: an OrganizationDomain lives in its organization's namespace and is named after its domain, so an organization has one per domain. spec.domainName is immutable.
• Validation: the webhook rejects:
  • public suffixes such as co.uk;
  • public email providers such as gmail.com;
  • upper case;
  • a name that isn't the domain;
  • a namespace that isn't the request organization's.
• Verification: the controller publishes a random TXT value, then looks the record up. It checks every minute for the first hour, every 5 minutes for the first day, every 30 minutes for the first week, and every 6 hours after that.
  • Once it finds the record it sets Verified=True and clears status.verification. A verified domain stays verified, as project Domains do, so the record can be removed afterwards.
  • When the record isn't there yet, the RecordNotFound, RecordMismatch and LookupFailed reasons say why.
• Record name: the prefix is _milo-verification by default. The new --organization-domain-record-prefix flag lets a deployment brand it, for example _datum-verification.

Email domain joining

• emaildomain.Verifier.VerifiedDomains now returns the organization's verified OrganizationDomains as well as the verified Domains in its projects. Both the Organization webhook and the joining controller use it, so either kind of verification works.
• The joining controller watches OrganizationDomains. When a domain becomes verified, or is removed, it reconciles the organization straight away rather than waiting for the 10-minute recheck.
• The "not verified" instructions now point owners at the organization's domains first.

Permissions

• Write access is owner-only. Owners can add and remove domains because resourcemanager-admin inherits the new organizationdomain-admin. Datum's owner role inherits resourcemanager-admin.
• Read access: resourcemanager-reader and resourcemanager.miloapis.com-organization-viewer inherit organizationdomain-reader.
• ProtectedResource: resourcemanager.miloapis.com-organizationdomain, with Organization as its parent.

Rollout

• Feature gate: everything is behind the existing EmailDomainJoining gate, which is on in staging. With the gate off, the webhook refuses new OrganizationDomains and the controller leaves them alone.
• Activity log: a new policy shows "added/removed the domain acme.com" in the activity log.

Testing

• Controller: publishing the record, verifying once it appears (next to other values at the same name), waiting for the next attempt, mismatch and lookup failure reasons, the configurable prefix, and doing nothing with the gate off.

• Webhook: each rejection above, subdomains, private public suffixes such as acme.github.io, and the organization context on create and update.

• Verifier: OrganizationDomains are included, other organizations' domains are not, and an unreadable project doesn't hide a verified OrganizationDomain.

• Joining controller: the OrganizationDomain watch's mapping and predicate.

• End to end: test/resource-management/organization-domain is a new Chainsaw test, and it passes against task dev:setup. It covers:

  • the webhook's rejections;
  • the published TXT record;
  • domainName immutability;
  • joining rejecting the domain while it's unverified and accepting it once verified;
  • an offer for a user at the domain and none for a user elsewhere;
  • the offer being withdrawn when the domain is removed.

  CI can't publish a DNS record, so the test marks the domain verified through the status subresource. The test environment authorizes with Kubernetes RBAC, so it doesn't cover which IAM roles grant access.

• task generate is committed and go test ./... passes.

Also fixed

project-suspension-lifecycle and the projectsuspension admission test both created a cluster-scoped test-user-admin ClusterRoleBinding, and each deleted it during cleanup. When the two ran at the same time, the first to finish removed the other's access. Adding a test changed the timing enough to expose this race, so each test now has its own binding.