Error-handling and Logging

[PACHAKUTlQ]

Refactor backend to add logging and error handling following modern best practice, for better code quality, observability and security.
The new error handling should take care of all known and unknown issues, log and return error for known issues, and log with stack trace for unknown issues.
The log injection vulnerabilities reported by CodeQL in #27 is NOT completely fixed, there are other points where user-controlled data flow into log. These must be fixed with the refactor using something like:

logging.error("action: %s", action)  # New line characters auto escaped

instead of:

logging.error(f"action: {action}")

and potentially sanitize it to prevent XSS when viewing log (Defense-in-depth, but may break things. Pending discussion).

[A-lexisL]

#30 has changed f-string to %. And this commit add Filter object to logger, sanitizing common injections.
Still, need to check whether args passed to logger will not leak important information

[A-lexisL]

still some issue for logging: due to improper error handling, some data passed to logger is dict instead of string, which is imcompatible with %s. So I have create another branch refactor/error for error-handling and logging. #30 is only for web cq now.

[A-lexisL]

Django REST Framework Exception Handling and Logging Strategy

files to be covered: apps/auth/views.py, apps/auth/utils.py, apps/web/views.py(only part 3 logging).

This prompt outlines a three-part strategy for handling errors and logging within a Django REST Framework (DRF) application. The goal is to create a clear, maintainable, and robust system by centralizing response generation and differentiating error types.

Part 1: Leaf Function Behavior

Core Principle: Functions below the view layer (e.g., utility functions, services, business logic, hereafter referred to as "leaf functions") must be decoupled from HTTP response generation. Their responsibility is to execute a task and report its outcome.

Requirements:

Successful Execution: On success, a leaf function must return a Python native data type (e.g., dict, list, a specific object). It must not return a DRF Response object. Only view api functions could return a DRF Response object.

Failed Execution: When an error occurs, a leaf function must raise an appropriate exception. The exception should contain all necessary information about the failure. As codes after raising exception will not be executed, returning values for failed execution is insignificant. It's accepted to return None/empty value in this situation.

More information about exception is as follows and Part 1 will not address error handling.

Part 2: Differentiating Predictable vs. Unpredictable Errors

Core Principle: Errors should be categorized to distinguish between known, anticipated failures and unexpected, system-level problems. This dictates how they are handled and logged.

Requirements:

Predictable Errors:

Definition: These are foreseeable issues, often related to invalid business logic or user input. They are raised intentionally by user functions.

Each distinct predictable error must be represented by its own custom exception class (e.g., class InvalidActionError(Exception): pass). The layer structure of exceptions are corresponding to the layer structure of codes.

Each predictable error should carry an error message, e.g. throw InvalidActionError("Invalid Action"). Note that this message is user-oriented so it should not contain secret(calling stack, business, etc) information of our app and should not contain user-source information(e.g. "Invalid f{action}" where action is submitted by the user). Normally, if a Child Exception is caught and Parent Exception is reraised, Parent Exception will use its own error message so the error message of child Exception will be dumped.

Requirements: Predictable Exception Structure
The exception hierarchy should be designed with multiple levels of specificity, following this pattern:

• Level 1: Base Application Exception
  Definition: A generic, top-level exception class should be defined for each Django application (e.g., for an auth app, create AuthError).
  Inheritance: This class should inherit from Python's base Exception.

• Level 2: Module or Feature Exception
  Definition: Within an application, create more specific exception classes for distinct modules, services, or features (e.g., for a "Verify Turnstile" utility within the auth app, create VerifyTurnstileError).
  Inheritance: This class must inherit from the Base Application Exception (e.g., class VerifyTurnstileError(AuthError):).

• Level 3: Specific Error Condition Exception
  Definition: For each predictable failure within a feature, create a highly specific exception class that describes the exact error (e.g., VerifyTurnstileTimeoutError, VerifyTurnstileInvalidTokenError).
  Inheritance: These classes must inherit from their parent Module/Feature Exception (e.g., class VerifyTurnstileTimeoutError(VerifyTurnstileError):).

Implementation:

1. Every layer will have a try-except clause to catch all predictable errors raised by the neighboring lower layers. e.g. if a neighboring lower layer will throw A1Exception, A2Exception which inherit from AException, then the parent layer should use try xxx except AExcection according to LSP.

2. The view layer will catch all possible errors with Level 1 Exception, and will not reraise errors.

Unpredictable Errors:

Definition: raised by non-user functions(lib or Django).

Implementation: They will only be handled by the most top level in the view layer by a catch-all except Exception.

Part 3: Logging Strategy

Core Principle: Logging should be context-aware. Log entries for predictable errors should be concise and informative, while logs for unpredictable errors must include full diagnostic details.

Logging Content and Format

1. Content: Please invesitgate https://docs.python.org/3/library/logging.html and real python projects on github and see what content should be contained in logging message. Note the core principle for logging is developer-oriented(that is to provide enough information for developers to quickly locate bugs) while also prevent user-injection.

2. Format: use %s/%d/.. style msg instead of f-string, and make sure the type of arg matches the specification in msg.
   e.g. logger.warning("Review serializer errors: %s", serializer.errors) may report error because serializer.errors is of ReturnList | ReturnDict | Any and may be incompatible of %s

Requirements:

Logging Predictable Errors:

Location:

1. Logging should occur at the point of detection of the most lower layer of exceptions, immediately before the custom exception is raised.

Method: Use logging.warning(msg,arg) or logging.error(msg,arg).

Logging Unpredictable Errors:

Location: Logging occur only in the top-level exception handler.

Method: Use logging.exception() to log stack trace.

logging configuration

django logging config: https://docs.djangoproject.com/en/3.1/topics/logging/

1. configure LOGGING in settings.py.

2. only top-level logger has handler, other loggers will only create LogRecord and then propate to higher logger.

3. add SanitizationFilter to all the loggers.