Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/guide/en/advanced-map.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Use this index when you need to customize internals: custom middleware, adapters
- [Middleware pipelines deep dive](middleware-pipelines.md) — dispatcher lifecycle, request mutations, and per-pipeline contracts.
- [Callable definitions and middleware factories](callable-definitions-extended.md) — container-aware definitions for middleware factories.
- [Error handling internals](error-handling-advanced.md) — failure pipeline flow, built-in components, and custom middleware.
- [Envelope metadata and stack reconstruction](envelopes-metadata-internals.md) — stack resolution and metadata merging.
- [Envelope metadata and serialization](envelopes-metadata-internals.md) — metadata merging and what survives a round-trip.
- [Handler resolver pipeline](message-handler-advanced.md) — alternative handler lookup strategies.

## Queue adapters and interoperability
Expand Down
26 changes: 13 additions & 13 deletions docs/guide/en/best-practices.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ Messages should contain only the data needed for processing. Avoid embedding lar
#### Bad

```php
new Message(ProcessImageHandler::class, [
new GenericMessage(ProcessImageHandler::class, [
'imageData' => base64_encode($imageContent), // Large binary data
'operations' => ['resize', 'watermark'],
]);
Expand All @@ -154,7 +154,7 @@ new Message(ProcessImageHandler::class, [
#### Good

```php
new Message(ProcessImageHandler::class, [
new GenericMessage(ProcessImageHandler::class, [
'imageId' => 12345, // You can get the image itself from DB or disk by id in a message handler
'operations' => ['resize', 'watermark'],
]);
Expand All @@ -171,7 +171,7 @@ new Message(ProcessImageHandler::class, [
#### Bad

```php
new Message(SendEmailHandler::class, [
new GenericMessage(SendEmailHandler::class, [
'to' => 'user@example.com',
'mailer' => $this->mailer, // Object instance
'callback' => fn() => $this->log(), // Closure
Expand All @@ -181,7 +181,7 @@ new Message(SendEmailHandler::class, [
#### Good

```php
new Message(SendEmailHandler::class, [
new GenericMessage(SendEmailHandler::class, [
'to' => 'user@example.com',
'subject' => 'Welcome',
'templateId' => 'welcome-email',
Expand All @@ -203,7 +203,7 @@ new Message(SendEmailHandler::class, [

```php
// External system pushes messages with handler class name
new Message('\App\Queue\EmailHandler', ['to' => 'user@example.com']);
new GenericMessage('\App\Queue\EmailHandler', ['to' => 'user@example.com']);
```

#### Good
Expand All @@ -219,7 +219,7 @@ return [
];

// External system uses stable type
new Message('send-email', ['to' => 'user@example.com']);
new GenericMessage('send-email', ['to' => 'user@example.com']);
```

#### Why
Expand All @@ -242,14 +242,14 @@ return [
],
];

$queue->push(new Message('generate-report', ['reportId' => 123]));
$queue->push(new GenericMessage('generate-report', ['reportId' => 123]));
```

#### Good

```php
// No configuration needed
$queue->push(new Message(
$queue->push(new GenericMessage(
GenerateReportHandler::class,
['reportId' => 123]
));
Expand Down Expand Up @@ -328,7 +328,7 @@ $this->logger->info('Queued task');

```php
$pushedMessage = $queue->push($message);
$id = $pushedMessage->getMetadata()[IdEnvelope::META_ID] ?? null;
$id = $pushedMessage->getMeta()[IdEnvelope::META_ID] ?? null;

$this->logger->info('Queued task', [
'messageId' => $id,
Expand Down Expand Up @@ -478,7 +478,7 @@ See [Loops](loops.md) and [Performance tuning](performance-tuning.md) for more d
// Testing through the queue (integration test)
public function testProcessesPayment(): void
{
$queue->push(new Message(ProcessPaymentHandler::class, ['paymentId' => 123]));
$queue->push(new GenericMessage(ProcessPaymentHandler::class, ['paymentId' => 123]));
// Hard to verify behavior, slow, requires queue setup
}
```
Expand All @@ -495,7 +495,7 @@ final class ProcessPaymentHandlerTest extends TestCase
$this->createMock(PaymentRepository::class),
);

$message = new Message(ProcessPaymentHandler::class, [
$message = new GenericMessage(ProcessPaymentHandler::class, [
'paymentId' => 123,
]);

Expand Down Expand Up @@ -599,7 +599,7 @@ See [Consuming messages from external systems](consuming-messages-from-external-
#### Bad

```php
new Message(SendEmailHandler::class, [
new GenericMessage(SendEmailHandler::class, [
'userEmail' => 'user@example.com',
'apiKey' => 'secret-key-here', // Secret in message
'password' => 'user-password', // Sensitive data
Expand All @@ -609,7 +609,7 @@ new Message(SendEmailHandler::class, [
#### Good

```php
new Message(SendEmailHandler::class, [
new GenericMessage(SendEmailHandler::class, [
'userId' => 123,
'templateId' => 'welcome',
]);
Expand Down
2 changes: 1 addition & 1 deletion docs/guide/en/configuration-manual.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ $logger = new NullLogger(); // replace with your PSR-3 logger in production

// Define message handlers
$handlers = [
DownloadFileMessage::TYPE => [FileDownloader::class, 'handle'],
DownloadFileMessage::TYPE => [RemoteFileHandler::class, 'handle'],
];

// Create middleware dispatchers
Expand Down
54 changes: 36 additions & 18 deletions docs/guide/en/envelopes-metadata-internals.md
Original file line number Diff line number Diff line change
@@ -1,35 +1,53 @@
# Envelope metadata and stack reconstruction
# Envelope metadata and serialization

This document explains the internal mechanics of envelope metadata handling and stack reconstruction during serialization/deserialization.
This document explains how envelope metadata is built and what survives serialization.

For basic envelope usage and concepts, see [Envelopes](envelopes.md).

## Metadata and envelope stacking
## Metadata merging

Every envelope contributes its own metadata to the resulting metadata array.
Every envelope contributes its own metadata to the resulting metadata array. `Yiisoft\Queue\Message\Envelope` merges
the wrapped message metadata with the envelope's own metadata, the envelope's own values taking precedence:

`Yiisoft\Queue\Message\Envelope` (base class) also maintains an envelope stack in message metadata under `EnvelopeInterface::ENVELOPE_STACK_KEY` (`"envelopes"`).
```php
$message = new GenericMessage('my-type', null);
$envelope = new DelayEnvelope(new IdEnvelope($message, 7), 3.5);

When `getMetadata()` is called on an envelope, it returns:
$envelope->getMeta(); // ['yii-id' => 7, 'yii-delay' => 3.5]
```

- the wrapped message metadata,
- plus an updated `"envelopes"` stack (previous stack + current envelope's FQCN),
- plus envelope-specific metadata.
`getType()` and `getPayload()` are delegated to the wrapped message, so an envelope is transparent for both.

Because envelopes wrap other messages, multiple envelopes form a stack.
## Envelopes are flattened, not stacked

## Restoring envelopes from metadata
`Envelope::__construct()` unwraps nested envelopes, so `getMessage()` always returns the innermost message:

If metadata contains the `"envelopes"` key with an array of envelope class names, the serializer will try to rebuild the stack by wrapping the message with each envelope in reverse order.
```php
$envelope->getMessage(); // GenericMessage, not IdEnvelope
```

During this process:
The envelope classes themselves are not recorded anywhere. What an envelope contributes is its metadata, and that is
all that is carried further.

- The `"envelopes"` key is removed from the base message metadata (it is set to an empty array before creating the base message).
- Each envelope from the list is applied to the message using `EnvelopeInterface::fromMessage(...)` in reverse order.
- A value is applied only if it is a string, the class exists, and it implements `EnvelopeInterface`. Otherwise it is ignored.
## Serialization

`Yiisoft\Queue\Message\Serializer\MessageSerializer` writes three keys — `type`, `payload` and `meta` — and restores a
message by resolving the class from `type` and applying `withMeta()`. Envelope wrappers are therefore not rebuilt on
the way back; the values they carried are still in the metadata and can be read through the envelope that owns them:

```php
IdEnvelope::fromMessage($message)->getId();
DelayEnvelope::fromMessage($message)->getDelaySeconds();
```

> **Note:** do not register an `Envelope` subclass as the class for a message type in the class resolver. Unserializing
> such a type calls `Envelope::fromPayload()`, which always throws a `LogicException`. Register the wrapped message
> class instead.

## FailureEnvelope metadata merging

`Yiisoft\Queue\Middleware\FailureHandling\FailureEnvelope` stores failure-handling metadata under the `FailureEnvelope::META_FAILURE` key (`"yii-failure"`).
`Yiisoft\Queue\Middleware\FailureHandling\FailureEnvelope` stores failure-handling metadata under the
`FailureEnvelope::META_FAILURE` key (`"yii-failure"`).

The envelope performs special deep-merge logic when building `getMetadata()` to combine failure metadata from nested envelopes.
Its constructor deep-merges the failure metadata already present on the message with the metadata being added, so
values accumulated by earlier failure-handling middlewares are preserved rather than overwritten.
17 changes: 9 additions & 8 deletions docs/guide/en/envelopes.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@

An *envelope* is a message container that wraps another message and adds metadata.

An envelope implements `Yiisoft\Queue\Message\EnvelopeInterface`, which itself extends `Yiisoft\Queue\Message\MessageInterface`.
An envelope extends the abstract `Yiisoft\Queue\Message\Envelope` class, which implements
`Yiisoft\Queue\Message\MessageInterface`.

## How an envelope behaves

Expand All @@ -11,17 +12,17 @@ An envelope is transparent to callers: it delegates the wrapped message type and
- `getType()` is delegated to the wrapped message.
- `getPayload()` is delegated to the wrapped message.

Envelopes modify the metadata returned by `getMetadata()` and may provide convenience methods for accessing specific metadata entries (for example, `getId()` in an ID envelope).
Envelopes modify the metadata returned by `getMeta()` and may provide convenience methods for accessing specific metadata entries (for example, `getId()` in an ID envelope).

## Creating envelopes

To wrap a message into an envelope, envelope classes provide:
To wrap a message into an envelope, use either the envelope constructor or:

- `EnvelopeInterface::fromMessage(MessageInterface $message): static`
- `Envelope::fromMessage(MessageInterface $message): static`, which restores the envelope's own parameters from the
message metadata.

and, via `MessageInterface` inheritance, also support:

- `Envelope::fromPayload(string $type, mixed $payload): static`
`Envelope::fromPayload()` is inherited from `MessageInterface`, but it always throws a `LogicException`: an envelope
cannot be created from a raw payload, it can only wrap an existing message.

## Built-in envelopes

Expand All @@ -35,4 +36,4 @@ This envelope is used to carry the adapter-provided message ID through the messa

`Yiisoft\Queue\Middleware\FailureHandling\FailureEnvelope` stores failure-handling metadata.

See [Errors and retryable messages](error-handling.md) for error handling concepts and [Envelope metadata and stack reconstruction](envelopes-metadata-internals.md) for details on how failure metadata is merged.
See [Errors and retryable messages](error-handling.md) for error handling concepts and [Envelope metadata and serialization](envelopes-metadata-internals.md) for details on how failure metadata is merged.
50 changes: 19 additions & 31 deletions docs/guide/en/middleware-pipelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,11 @@

Yii Queue uses middlewares to run custom logic around message pushing and message processing.

A middleware is a piece of code that receives a request object and can either:
A middleware is a piece of code that receives the message being pushed, or a request object wrapping it, and can
either:

- change the request (for example, change the message, adapter, or error handling behavior) and continue the pipeline, or
- change what it received (for example, wrap the message into an envelope, or change the error handling behavior)
and continue the pipeline, or
- stop the pipeline by returning without calling the next handler.

The pipeline mechanism is similar to HTTP middleware, but applied to queue messages.
Expand All @@ -21,8 +23,9 @@ Common reasons to add middlewares:
You can log message ids, queue names, attempts, and failures in a consistent way.
- **Modify the message payload**
You can obfuscate sensitive data, normalize payload, add extra fields required by consumers, or wrap a message into envelopes.
- **Route and schedule**
You can switch queue, choose a different adapter, or add delay when the adapter supports it.
- **Schedule**
You can add delay when the adapter supports it. Routing to another queue is not done in a middleware: resolve the
producer for that queue through `QueueProducerProviderInterface` instead, see [Queue names](queue-names.md).

## Pipelines overview

Expand Down Expand Up @@ -74,38 +77,23 @@ The required interface depends on the pipeline:

The push pipeline is executed when calling `QueueProducerInterface::push()`.

Push middlewares can:
Push middlewares receive the message and return a message, so they can wrap it into envelopes, add metadata,
obfuscate data, and so on.

- Modify the message (wrap it into envelopes, add metadata, obfuscate data, etc.).
- Modify the adapter (add delay, route to a different backend, etc.).

In particular, push middlewares may define or replace the adapter that will be used to push the message. This can be useful when:

- You choose a backend dynamically (for example, based on message type or payload).
- You route messages to different queues/backends (for example, `critical` vs `low`).
- You apply scheduling/delay logic in a middleware.

The adapter is set by returning a modified request:

```php
return $pushRequest->withAdapter($adapter);
```

### Adapter must be configured by the end of the pipeline

The push pipeline ends with a final handler that actually pushes the message using the adapter.

If the adapter is not configured by the time the pipeline reaches the final handler,
`Yiisoft\Queue\Exception\AdapterNotConfiguredException` is thrown.
The adapter is not part of the pipeline: it is fixed when the producer is built, and the push pipeline ends with a
final handler that pushes through it (`AdapterPushHandler` for `AsyncQueueProducer`, `SynchronousPushHandler` for
`SyncQueueProducer`). To route messages to different backends, resolve a different producer by queue name through
`QueueProducerProviderInterface` instead — see [Queue names](queue-names.md).

### Custom push middleware

Implement `PushMiddlewareInterface` and return a modified `PushRequest` from `processPush()`:
Implement `PushMiddlewareInterface` and return the message to pass down the pipeline:

```php
return $pushRequest
->withMessage($newMessage)
->withAdapter($newAdapter);
public function processPush(MessageInterface $message, PushHandlerInterface $handler): MessageInterface
{
return $handler->handlePush(new MyEnvelope($message));
}
```

## Consume pipeline
Expand All @@ -128,7 +116,7 @@ The pipeline receives a `FailureHandlingRequest` that contains:

- the message
- the caught exception
- the queue instance
- the queue name

The pipeline is selected by queue name; if there is no queue-specific pipeline configured,
`FailureMiddlewareDispatcher::DEFAULT_PIPELINE` is used.
Expand Down
28 changes: 11 additions & 17 deletions docs/guide/en/performance-tuning.md
Original file line number Diff line number Diff line change
Expand Up @@ -232,7 +232,7 @@ public function processConsume(ConsumeRequest $request, ConsumeHandlerInterface
{
// Lightweight logging
$this->logger->debug('Processing message', [
'id' => $request->getMessage()->getMetadata()[IdEnvelope::META_ID] ?? null,
'id' => $request->getMessage()->getMeta()[IdEnvelope::META_ID] ?? null,
]);

return $handler->handleConsume($request);
Expand All @@ -249,7 +249,7 @@ Instead of sending many small messages, batch them when possible:

```php
foreach ($userIds as $userId) {
$queue->push(new Message(SendEmailHandler::class, [
$queue->push(new GenericMessage(SendEmailHandler::class, [
'userId' => $userId,
]));
}
Expand All @@ -258,7 +258,7 @@ foreach ($userIds as $userId) {
**Good** (1 message):

```php
$queue->push(new Message(SendBulkEmailHandler::class, [
$queue->push(new GenericMessage(SendBulkEmailHandler::class, [
'userIds' => $userIds, // Process in batches
]));
```
Expand All @@ -277,21 +277,15 @@ public function handle(MessageInterface $message): void
}
```

### Avoid deep envelope stacking
### Envelope depth costs nothing at read time

While [envelope](envelopes.md) stacking is optimized, deep nesting still has overhead:
[Envelopes](envelopes.md) do not nest at runtime: each one merges its metadata into the wrapped message's metadata
and keeps the innermost message, so `getMeta()`, `getType()` and `getPayload()` cost the same whatever the wrapping
looked like.

```php
// Avoid excessive wrapping
$message = new Message(...);
$message = new Envelope1($message);
$message = new Envelope2($message);
$message = new Envelope3($message);
$message = new Envelope4($message);
$message = new Envelope5($message); // Too many layers
```

Keep envelope depth reasonable (typically 2-3 layers).
What each wrap does cost is one metadata merge when it is constructed, which is negligible next to any handler doing
real work. Wrap as many times as the code reads best — see
[Envelope metadata and serialization](envelopes-metadata-internals.md) for the mechanics.

## Database optimization

Expand Down Expand Up @@ -410,7 +404,7 @@ $start = microtime(true);
$count = 10000;

for ($i = 0; $i < $count; $i++) {
$queue->push(new Message(TestHandler::class, [
$queue->push(new GenericMessage(TestHandler::class, [
'id' => $i,
'data' => $this->generateRealisticData(),
]));
Expand Down