jaiminho

A broker agnostic implementation of the outbox and other message resilience patterns for Django apps.

Migrating from 1.x.x

SIGN_EVENTS and VERIFY_EVENTS_SIGNATURE are enabled by default once you install the latest version. However, if you're migrating from a previous version to the latest one, you'll have to do it in two steps:

• Install the latest version of django-jaiminho and set VERIFY_EVENTS_SIGNATURE to False;
• Ensure there are no unsent events without a signature and set VERIFY_EVENTS_SIGNATURE to True.

If there are unset events without a signature and VERIFY_EVENTS_SIGNATURE is enabled, your events will be considered tampered and won't be relayed.

Getting Started

To use jaiminho with your project, you just need to do 6 steps:

1 - Install it

python -m pip install jaiminho

2 - Add jaiminho to the INSTALLED_APPS

INSTALLED_APPS = [
    ...
    "jaiminho"
]

3 - Run migrations

python manage.py migrate

4 - Configure jaiminho options in Django settings.py:

JAIMINHO_CONFIG = {
    "PERSIST_ALL_EVENTS": False,
    "DELETE_AFTER_SEND": True,
    "DEFAULT_ENCODER": DjangoJSONEncoder,
    "PUBLISH_STRATEGY": "publish-on-commit",
}

5 - Decorate your functions with @save_to_outbox

from jaiminho.send import save_to_outbox

@save_to_outbox
def any_external_call(**kwargs):
    # do something
    return

If your project writes to more than one database, pass the alias your decorated function's transaction runs on through the using argument, so the outbox event is persisted and the on-commit hook is bound to that same alias:

from jaiminho.send import save_to_outbox

@save_to_outbox(using="replica")
def any_external_call(**kwargs):
    # do something
    return

@save_to_outbox_stream accepts the same using argument. When omitted, Jaiminho keeps its previous behavior of relying on Django's default database alias.

6 - Run the relay events command

python manage.py events_relay --run-in-loop --loop-interval 1

If you don't use --run-in-loop option, the relay command will run only 1 time. This is useful in case you want to configure it as a cronjob.

Details

Jaiminho @save_to_outbox decorator will intercept decorated function and persist it in a database table in the same transaction that is active in the decorated function context. The event relay command, is a separated process that fetches the rows from this table and execute the functions. When an outage happens, the event relay command will keep retrying until it succeeds. This way, eventual consistency is ensured by design.

Some errors, however, are not transient: if the decorated function raises one of the exceptions configured through NON_RETRYABLE_EXCEPTIONS (BadSignature, ModuleNotFoundError and AttributeError by default), retrying is guaranteed to fail again in the exact same way. Under the Publish on Commit strategy, Jaiminho does not persist an event for these failures (or deletes it if it was already persisted), so a single permanently-failing event can no longer poison the outbox table. Under the Keep Order strategy this behavior does not apply: dropping an event would break delivery order for whatever is queued behind it, so a non-retryable failure still gets the relayer stuck, same as any other failure.

Configuration options

• PUBLISH_STRATEGY - Strategy used to publish events (publish-on-commit, keep-order)
• PERSIST_ALL_EVENTS - Saves all events and not only the ones that fail, default is False. Only applicable for { "PUBLISH_STRATEGY": "publish-on-commit" } since all events needs to be stored on keep-order strategy.
• DELETE_AFTER_SEND - Delete the event from the outbox table immediately, after a successful send
• DEFAULT_ENCODER - Default Encoder for the payload (overwritable in the function call)
• SIGN_EVENTS - Signs events to support verification later
• VERIFY_EVENTS_SIGNATURE - Verifies previously generated signatures
• NON_RETRYABLE_EXCEPTIONS - Tuple of exception classes that are never retried: an event that fails with one of these is dropped instead of being persisted for retry. Only applies to the publish-on-commit strategy; under keep-order these failures still get the relayer stuck, to preserve delivery order. Defaults to (BadSignature, ModuleNotFoundError, AttributeError). Setting this replaces the default tuple rather than extending it, so include those three as well if you still want them covered.

Non-retryable exceptions in detail

• BadSignature - stored signature doesn't match the payload (VERIFY_EVENTS_SIGNATURE). Usually tampering, or signing config (SECRET_KEY, SIGN_EVENTS) changed after the event was persisted.
• ModuleNotFoundError / AttributeError - the outbox stores a dill reference to the decorated function's module path, not its source. The relay re-imports that path when it runs. If the function's module is later moved/renamed (ModuleNotFoundError) or the function itself is renamed/removed (AttributeError), any event already persisted under the old code will fail to deserialize.

Risk: deploying such a move/rename while events are still waiting to be relayed can lose them - they're dropped as non-retryable instead of retried. Only affects publish-on-commit (in-flight events); keep-order never drops non-retryable failures, it just gets stuck. Low-probability but real; mitigate by draining the outbox before the deploy, or excluding these two from NON_RETRYABLE_EXCEPTIONS for it.

Configuring it

from jaiminho.errors import DEFAULT_NON_RETRYABLE_EXCEPTIONS

JAIMINHO_CONFIG = {
    # replaces the defaults, so re-include them to keep them covered
    "NON_RETRYABLE_EXCEPTIONS": DEFAULT_NON_RETRYABLE_EXCEPTIONS + (MyPermanentError,),
}

Strategies

Keep Order

This strategy is similar to transactional outbox described by Chris Richardson. The decorated function intercepts the function call and saves it on the local DB to be executed later. A separate command relayer will keep polling local DB and executing those functions in the same order it was stored. Be carefully with this approach, if any execution fails, the relayer will get stuck as it would not be possible to guarantee delivery order. This includes failures configured as NON_RETRYABLE_EXCEPTIONS - they are not dropped under this strategy, precisely because doing so would break the order guarantee.

Publish on commit

This strategy will always execute the decorated function after current transaction commit. With this approach, we don't depend on a relayer (separate process / cronjob) to execute the decorated function and deliver the message. Failed items will only be retried through relayer. Although this solution has a better performance as only failed items is delivered by the relay command, we cannot guarantee delivery order.

Relay Command

We already provide a command to relay items from DB, EventRelayCommand. The way you should configure depends on the strategy you choose. For example, on Publish on Commit Strategy you can configure a cronjob to run every a couple of minutes since only failed items are published by the command relay. If you are using Keep Order Strategy, you should run relay command in loop mode as all items will be published by the command, e.g call_command(events_relay.Command(), run_in_loop=True, loop_interval=0.1).

How to clean older events

You can use Jaiminho's EventCleanerCommand in order to do that. It will query for all events that were sent before a given time interval (e.g. last 7 days) and will delete them from the outbox table.

The default time interval is 7 days. You can use the TIME_TO_DELETE setting to change it. It should be added to JAIMINHO_CONFIG and must be a valid timedelta.

Running as cron jobs

You can run those commands in a cron job. Here are some config examples:

  - name: relay-failed-outbox-events
    schedule: "*/15 * * * *"
    suspend: false
    args:
      - ddtrace-run
      - python
      - manage.py
      - events_relay
    resources:
      requests:
        cpu: 1
      limits:
        memory: 384Mi

  - name: delete-old-outbox-events
    schedule: "0 5 * * *"
    suspend: false
    args:
      - ddtrace-run
      - python
      - manage.py
      - event_cleaner
    resources:
      requests:
        cpu: 1
      limits:
        memory: 384Mi

Relay per stream and Overwrite publish strategy

Different streams can have different requirements. You can save separate events per streams by using the @save_to_outbox_stream decorator:

@save_to_outbox_stream("my-stream")
def any_external_call(payload, **kwargs):
    # do something
    pass

you can also overwrite publish strategy configure on settings:

@save_to_outbox_stream("my-stream", PublishStrategyType.KEEP_ORDER)
def any_external_call(payload, **kwargs):
    # do something
    pass

And then, run relay command with stream filter option

python manage.py relay_event True 0.1 my-stream

In the example above, True is the option for run_in_loop; 0.1 for loop_interval; and my_stream is the name of the stream.

Signals

Jaiminho triggers the following Django signals:

Signal	Description
event_published	Triggered when an event is sent successfully
event_failed_to_publish	Triggered when an event is not sent, being added to the Outbox table queue
event_permanently_failed	Triggered by @save_to_outbox's on-commit hook when the decorated function raises one of NON_RETRYABLE_EXCEPTIONS; the event is dropped (or deleted, if already persisted) instead of being retried
event_permanently_failed_by_events_relay	Triggered by the events_relay command when a relayed event fails with one of NON_RETRYABLE_EXCEPTIONS; the event is given up on instead of being retried. Only fires for events using the publish-on-commit strategy - under keep-order the relayer gets stuck instead

All of the signals above are sent with sender (the original decorated function, or None if it could not be resolved) and event_payload (the first positional argument passed to the decorated function, or {} if there wasn't one).

How to collect metrics from Jaiminho?

You could use the Django signals triggered by Jaiminho to collect metrics. Consider the following code as example:

from django.dispatch import receiver

@receiver(event_published)
def on_event_sent(sender, event_payload, **kwargs):
    metrics.count(f"event_sent_successfully: {event_payload}")

@receiver(event_failed_to_publish)
def on_event_send_error(sender, event_payload, **kwargs):
    metrics.count(f"event_failed: {event_payload}")

Jaiminho with Celery

Jaiminho can be very useful for adding reliability to Celery workflows. Writing to the database and enqueuing Celery tasks in the same workflow is very common in many applications, and this pattern can benefit greatly from the outbox pattern to ensure message delivery reliability.

Instead of configuring the @save_to_outbox decorator for every individual Celery task, you can integrate it at the Celery class level by overriding the send_task method, which is used by Celery to enqueue new tasks. This way, all tasks automatically benefit from the outbox pattern without requiring individual configuration.

Here's how to implement this:

from celery import Celery
from jaiminho.send import save_to_outbox


class CeleryWithJaiminho(Celery):
    """
    Custom Celery class that inherits from Celery base class
    and adds Jaiminho functionality
    """
    
    @save_to_outbox
    def send_task(self, *args, **kwargs):
        """Send task with outbox pattern for reliability"""
        return super().send_task(*args, **kwargs)


app = CeleryWithJaiminho("tasks")

With this approach, all tasks sent through your Celery app will automatically use the outbox pattern, ensuring that task enqueuing is resilient to transient failures and network issues.

Development

Create a virtualenv

virtualenv venv
pip install -r requirements-dev.txt
tox -e py39

Collaboration

If you want to improve or suggest improvements, check our CONTRIBUTING.md file.

License

This project is licensed under MIT License.

Security

If you have any security concern or report feel free to reach out to security@loadsmart.com;