lfx-v2-query-service

HTTP service for LFX API consumers to perform access-controlled queries for LFX resources, including typeahead and full-text search.

Agents working in this repo should start with CLAUDE.md. API contract truth lives under docs/.

Architecture Overview

The implementation follows the clean architecture principles where:

Domain Layer: Contains business logic and interfaces
Service Layer: Orchestrates business operations
Infrastructure Layer: Implements external dependencies
Presentation Layer: Handles HTTP/API concerns (generated by Goa)

Directory Structure

├── .github/                        # Github files
│   └── workflows/                  # Github Action workflow files
├── charts/                         # Helm charts
├── design/                         # GOA design specification files
├── gen/                            # GOA generated code (HTTP server, client, OpenAPI)
├── cmd/                            # Services (main packages)
│   └── service/                    # Service implementation
├── internal/                       # Internal service packages
│   ├── domain/                     # Domain logic layer
│   │   ├── model/                  # Domain models and entities
│   │   └── port/                   # Domain interfaces/ports
│   ├── service/                    # Business logic/use cases layer
│   ├── infrastructure/             # Infrastructure layer
│   └── middleware/                 # HTTP middleware components
└── pkg/                            # Shared packages for internal and external services

Key Components

Domain Layer (`internal/domain/`)

ResourceService: Contains business logic and validation

Model (`internal/domain/model/`)

Domain Models: Core business entities and data structures
Value Objects: Immutable objects that represent domain concepts

Ports (`internal/domain/port/`)

ResourceSearcher Interface: Defines the contract for resource search operations
OrganizationSearcher Interface: Defines the contract for organization search operations
AccessControlChecker Interface: Defines the contract for access control operations
Authenticator Interface: Defines the contract for authentication operations

Service Layer (`internal/service/`)

Business Logic: Application-specific business rules and operations
Service Orchestration: Coordinates between domain models and infrastructure

Infrastructure Layer (`internal/infrastructure/`)

Authentication Implementation

The authentication system provides JWT-based authentication with support for Heimdall tokens:

JWT Authentication (internal/infrastructure/auth/jwt.go):
- Validates JWT tokens using JWKS (JSON Web Key Set)
- Extracts custom claims including principal and email
- Supports PS256 signature algorithm (default for Heimdall)
- Configurable JWKS URL and audience
Mock Authentication (internal/infrastructure/mock/auth.go):
- Uses environment variable for mock principal
- Bypasses JWT validation for local development

Authentication Configuration:

AUTH_SOURCE: Choose between "mock" or "jwt" (default: "jwt")
JWKS_URL: JSON Web Key Set endpoint URL
JWT_AUDIENCE: Intended audience for JWT tokens
JWT_AUTH_DISABLED_MOCK_LOCAL_PRINCIPAL: Mock principal for development

OpenSearch Implementation

The OpenSearch implementation includes query templates, a searcher, and a client for interacting with the OpenSearch cluster.

NATS Implementation

The NATS implementation consists of a client, access control logic, and request/response models for messaging and access control.

Clearbit Implementation

The Clearbit implementation provides organization search capabilities using the Clearbit Company API. It includes a client for API communication, searcher for organization queries, and configuration management for API credentials and settings.

Dependency Injection

Dependency injection starts in cmd/main.go, with concrete implementations selected in cmd/service/providers.go based on environment configuration and then injected into the service constructor.

Benefits of This Architecture

Testability: Easy to swap implementations for testing
Flexibility: Can easily switch between different search backends and access control systems
Maintainability: Clear separation of concerns
Scalability: Easy to add new search and access control implementations
Independence: Layers don't depend on external frameworks

Docker

Building the Docker Image

Example: Build the Docker image with Make

make docker-build

Running with Docker

Basic Docker Run

make docker-run

Usage

Queryable Resource Types

All resource types searchable via this service are listed in docs/resource-catalog.md, organized by the service that indexes them. Each entry links to that service's indexer contract, the authoritative reference for its data schemas, tags, access control config, and parent references.

For agent-oriented contract docs:

docs/query-service-contract.md: query API, access filtering, OpenSearch fields, tags, filters, CEL, page size, date ranges, and clause limits.
docs/indexed-data-types.md: active queryable resource types and where their service-owned contracts live.

Running Locally

With Mock Implementation (Default for Development)

# Using mock implementations
SEARCH_SOURCE=mock ACCESS_CONTROL_SOURCE=mock ORG_SEARCH_SOURCE=mock AUTH_SOURCE=mock \
JWT_AUTH_DISABLED_MOCK_LOCAL_PRINCIPAL=dev_user \
PAGE_TOKEN_SECRET=12345678901234567890123456789012 go run ./cmd

# With custom port
SEARCH_SOURCE=mock ACCESS_CONTROL_SOURCE=mock ORG_SEARCH_SOURCE=mock AUTH_SOURCE=mock \
JWT_AUTH_DISABLED_MOCK_LOCAL_PRINCIPAL=dev_user \
PAGE_TOKEN_SECRET=12345678901234567890123456789012 go run ./cmd -p 3000

With Production Services

# production-like setup
SEARCH_SOURCE=opensearch \
ORG_SEARCH_SOURCE=clearbit \
ACCESS_CONTROL_SOURCE=nats \
OPENSEARCH_URL={{placeholder}} \
OPENSEARCH_INDEX=resources \
NATS_URL={{placeholder}} \
PAGE_TOKEN_SECRET=12345678901234567890123456789012 \
CLEARBIT_CREDENTIAL=your_clearbit_api_key \
CLEARBIT_BASE_URL=https://company.clearbit.com \
CLEARBIT_TIMEOUT=30s \
CLEARBIT_MAX_RETRIES=5 \
CLEARBIT_RETRY_DELAY=2s \
go run ./cmd

Available Environment Variables

Search Implementation:

SEARCH_SOURCE: Choose between "mock" or "opensearch" (default: "opensearch")

Organization Search Implementation:

ORG_SEARCH_SOURCE: Choose between "mock" or "clearbit" (default: "clearbit")

OpenSearch Configuration:

OPENSEARCH_URL: OpenSearch URL (default: http://localhost:9200)
OPENSEARCH_INDEX: OpenSearch index name (default: "resources")
PAGE_TOKEN_SECRET: 32-character secret used to encode/decode opaque page tokens. Required for any deployment: the service calls global.PageTokenSecret(ctx) (which fatals if the var is unset) whenever it generates a page_token for a full page, so even an ordinary first-page query can crash the process if this is missing.

Access Control Implementation:

ACCESS_CONTROL_SOURCE: Choose between "mock" or "nats" (default: "nats")

NATS Configuration:

NATS_URL: NATS server URL (default: nats://localhost:4222)
NATS_TIMEOUT: Request timeout duration (default: "10s")
NATS_MAX_RECONNECT: Maximum reconnection attempts (default: "3")
NATS_RECONNECT_WAIT: Time between reconnection attempts (default: "2s")

Clearbit Configuration:

CLEARBIT_CREDENTIAL: Clearbit API key (required for organization search)
CLEARBIT_BASE_URL: Clearbit Company API base URL (default: https://company.clearbit.com)
CLEARBIT_AUTOCOMPLETE_BASE_URL: Clearbit Autocomplete API base URL (default: https://autocomplete.clearbit.com)
CLEARBIT_TIMEOUT: HTTP client timeout for API requests (default: "10s")
CLEARBIT_MAX_RETRIES: Maximum number of retry attempts for failed requests (default: "3")
CLEARBIT_RETRY_DELAY: Delay between retry attempts (default: "1s")

Authentication Configuration:

AUTH_SOURCE: Choose between "mock" or "jwt"
JWKS_URL: JSON Web Key Set endpoint URL
JWT_AUDIENCE: Intended audience for JWT tokens
JWT_AUTH_DISABLED_MOCK_LOCAL_PRINCIPAL: Mock principal for development (required when AUTH_SOURCE=mock)

Server Configuration:

-p: HTTP port (default: "8080")
-bind: Interface to bind on (default: "*")
-d: Enable debug logging

API Usage

The service exposes a RESTful API through the Goa framework with JWT authentication:

Resource Search API

GET /query/resources?name=committee&type=committee&v=1
Authorization: Bearer <jwt_token>

Parameters:

name: Resource name or alias (supports typeahead search)
type: Resource type to filter by
parent: Parent resource for hierarchical queries
tags: Array of tags to filter by (OR logic - matches resources with any of these tags)
tags_all: Array of tags to filter by (AND logic - matches resources that have all of these tags)
filters, filters_all, filters_or: field:value filters against data.*
cel_filter: CEL expression for advanced post-query filtering
filter_grants: direct to pre-filter to resources with direct FGA grants (requires type)
date_field: Date field to filter on (within data object) - used with date_from and/or date_to
date_from: Start date (inclusive). Format: ISO 8601 datetime or date-only (YYYY-MM-DD). Date-only uses start of day UTC
date_to: End date (inclusive). Format: ISO 8601 datetime or date-only (YYYY-MM-DD). Date-only uses end of day UTC
sort: Sort order (name_asc, name_desc, updated_asc, updated_desc, best_match)
page_size: Number of results per page (1-1000, default 50)
page_token: Pagination token
v: API version (required)

Response:

{
  "resources": [
    {
      "type": "committee",
      "id": "123",
      "data": {
        "name": "Technical Advisory Committee",
        "description": "Main technical governance body",
        "status": "active"
      }
    }
  ],
  "page_token": "opaque-token-or-omitted",
  "cache_control": "public, max-age=300"
}

At least one of name, parent, type, tags, or filter_grants must be provided; a request with only cel_filter, date_*, filters*, sort, or pagination parameters is rejected with 400 Bad Request.

Resource Count API

GET /query/resources/count?type=committee&v=1
Authorization: Bearer <jwt_token>

Accepts the same filter parameters as GET /query/resources except cel_filter, filter_grants, sort, page_size, and page_token, and returns a count rather than the resources themselves:

{ "count": 42, "has_more": false }

has_more is true when the count is not guaranteed to be exhaustive and the client should request a narrower query.

For API contract details (page size, date ranges, CEL filter, clause limits, anonymous vs authenticated semantics, access-control flow), see docs/query-service-contract.md.

Organization Search API

Query Organizations:

GET /query/orgs?name=Linux Foundation&domain=linuxfoundation.org&v=1
Authorization: Bearer <jwt_token>

Parameters:

name: Organization name (optional)
domain: Organization domain or website URL (optional)
v: API version (required)

Response:

{
  "name": "Linux Foundation",
  "domain": "linuxfoundation.org",
  "industry": "Non-Profit",
  "sector": "Technology",
  "employees": "100-499"
}

Organization Suggestions API:

GET /query/orgs/suggest?query=linux&v=1
Authorization: Bearer <jwt_token>

Parameters:

query: Search query for organization suggestions (required, minimum 1 character)
v: API version (required)

Response:

{
  "suggestions": [
    {
      "name": "Linux Foundation",
      "domain": "linuxfoundation.org",
      "logo": "https://example.com/logo.png"
    }
  ]
}

Clearbit API Integration

The service integrates with Clearbit's Company API to provide enriched organization data for search operations. This integration allows the service to fetch detailed company information including industry classification, employee count, and domain information.

Clearbit API Setup

1. Obtain API Credentials

To use Clearbit integration, you need to obtain an API key from Clearbit:

Sign up for a Clearbit account at https://clearbit.com
Navigate to your API settings to generate an API key
Copy the API key for use in your environment configuration

2. Configure Environment Variables

Set the required environment variables for Clearbit integration:

# Required: Clearbit API key
export CLEARBIT_CREDENTIAL=your_clearbit_api_key_here

# Optional: Custom configuration (defaults shown)
export CLEARBIT_BASE_URL=https://company.clearbit.com
export CLEARBIT_TIMEOUT=10s
export CLEARBIT_MAX_RETRIES=3
export CLEARBIT_RETRY_DELAY=1s

# Set organization search source to use Clearbit
export ORG_SEARCH_SOURCE=clearbit

3. API Usage and Features

The Clearbit integration supports the following search operations:

Search by Company Name:

Searches for companies using their registered business name
Falls back to domain-based search for additional data enrichment

Search by Domain:

More accurate search method using company domain names
Provides comprehensive company information

4. Error Handling

The Clearbit integration includes robust error handling:

404 Not Found: Returns appropriate "organization not found" errors
Rate Limiting: Automatic retry with exponential backoff
Network Issues: Configurable retry attempts with delays
API Errors: Proper error propagation with context

Testing

The clean architecture makes testing straightforward:

// Use mock implementations for unit tests
searcher := mock.NewMockResourceSearcher()
searcher.AddResource(testResource)

accessChecker := mock.NewMockAccessControlChecker()
// CheckAccess returns a map keyed by the per-resource access-check line
accessChecker.SetCheckAccessResponse(map[string]string{
    "project:abc#viewer@user:123": "true",
})

resourceFilter := mock.NewMockResourceFilter()

service := service.NewResourceSearch(searcher, accessChecker, resourceFilter)
result, err := service.QueryResources(ctx, criteria)

Extending the Architecture

To add a new search implementation:

Create a new package in internal/infrastructure/
Implement the domain.ResourceSearcher interface
Add configuration options to main.go
Update the dependency injection switch statement

To add a new access control implementation:

Create a new package in internal/infrastructure/
Implement the domain.AccessControlChecker interface
Add configuration options to main.go
Update the dependency injection switch statement

Development

Prerequisites

This project uses the GOA Framework for API generation. You'll need to install GOA before building the project.

Installing GOA Framework

Follow the GOA installation guide to install GOA:

go install goa.design/goa/v3/cmd/goa@latest

Verify the installation:

goa version

Building and Development

1. Generate Code

The project uses GOA to generate API code from the design specification. Run the following command to generate all necessary code:

goa gen github.com/linuxfoundation/lfx-v2-query-service/design

This command generates:

HTTP server and client code
OpenAPI specification
Service interfaces and types
Transport layer implementations

2. Initial Project Structure

Note: The initial cmd structure was generated using GOA's example generator:

goa example github.com/linuxfoundation/lfx-v2-query-service/design

This command generated the basic server structure, which was then customized and adjusted to fit our project's clean architecture principles.

3. Development Workflow

Make design changes: Edit files in the design/ directory
Regenerate code: Run goa gen github.com/linuxfoundation/lfx-v2-query-service/design after design changes
Build the project:
```
go build ./cmd
```

Run with mock data (for development):

SEARCH_SOURCE=mock ACCESS_CONTROL_SOURCE=mock ORG_SEARCH_SOURCE=mock AUTH_SOURCE=mock \
JWT_AUTH_DISABLED_MOCK_LOCAL_PRINCIPAL=dev_user \
PAGE_TOKEN_SECRET=12345678901234567890123456789012 go run ./cmd

Run tests:

make test

# or run go test to set custom flags
go test ./... -v

Lint the code

# From the root of the directory, run megalinter (https://megalinter.io/latest/mega-linter-runner/) to ensure the code passes the linter checks. The CI/CD has a check that uses megalinter.
npx mega-linter-runner .

Docker build + K8

# Build the dockerfile (from the root of the repo)
docker build -t lfx-v2-query-service:<release_number> .

# Install the helm chart for the service into the lfx namespace (from the root of the repo)
helm install lfx-v2-query-service ./charts/lfx-v2-query-service/ -n lfx

# Once you have already installed the helm chart and need to just update it, use the following command (from the root of the repo):
helm upgrade lfx-v2-query-service ./charts/lfx-v2-query-service/ -n lfx

Contributing

To contribute to this repository:

Fork the repository
Make your changes
Submit a pull request

License

Copyright The Linux Foundation and each contributor to LFX.

This project’s source code is licensed under the MIT License. A copy of the license is available in LICENSE.

This project’s documentation is licensed under the Creative Commons Attribution 4.0 International License (CC-BY-4.0). A copy of the license is available in LICENSE-docs.

Name		Name	Last commit message	Last commit date
Latest commit History 265 Commits
.claude/skills		.claude/skills
.github/workflows		.github/workflows
charts/lfx-v2-query-service		charts/lfx-v2-query-service
cmd		cmd
design		design
docs		docs
gen		gen
internal		internal
pkg		pkg
.gitignore		.gitignore
.ko.yaml		.ko.yaml
.mega-linter.yml		.mega-linter.yml
.yamllint		.yamllint
CLAUDE.md		CLAUDE.md
CODEOWNERS		CODEOWNERS
Dockerfile		Dockerfile
LICENSE		LICENSE
LICENSE-docs		LICENSE-docs
Makefile		Makefile
README.md		README.md
SECURITY.md		SECURITY.md
go.mod		go.mod
go.sum		go.sum
revive.toml		revive.toml

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

lfx-v2-query-service

Architecture Overview

Directory Structure

Key Components

Domain Layer (internal/domain/)

Model (internal/domain/model/)

Ports (internal/domain/port/)

Service Layer (internal/service/)

Infrastructure Layer (internal/infrastructure/)

Authentication Implementation

OpenSearch Implementation

NATS Implementation

Clearbit Implementation

Dependency Injection

Benefits of This Architecture

Docker

Building the Docker Image

Example: Build the Docker image with Make

Running with Docker

Basic Docker Run

Usage

Queryable Resource Types

Running Locally

With Mock Implementation (Default for Development)

With Production Services

Available Environment Variables

API Usage

Resource Search API

Resource Count API

Organization Search API

Clearbit API Integration

Clearbit API Setup

1. Obtain API Credentials

2. Configure Environment Variables

3. API Usage and Features

4. Error Handling

Testing

Extending the Architecture

Development

Prerequisites

Installing GOA Framework

Building and Development

1. Generate Code

2. Initial Project Structure

3. Development Workflow

Contributing

License

About

Resources

License

Licenses found

Security policy

Uh oh!

Stars

Watchers

Forks

Releases 28

Packages 0

Uh oh!

Uh oh!

Contributors

Uh oh!

Languages

Domain Layer (`internal/domain/`)

Model (`internal/domain/model/`)

Ports (`internal/domain/port/`)

Service Layer (`internal/service/`)

Infrastructure Layer (`internal/infrastructure/`)

Packages