Implement batch processing for serializers with BatchHandlerInterface…

README.md

@@ -193,6 +193,103 @@ By default, all handlers have priority 0. Except:
 `BasicHandler` has highest priority (10) - this handles simple scalar values, so generally you want it to be first.
 `ObjectHandler` has lowest priority (-1) - this handles nested iterables/objects that no other handler supports.
 
+### Batch handlers
+
+A handler that resolves its value through I/O (a database lookup, an API call) would do it once per item of
+a serialized collection, and once per property of a deserialized object. Two interfaces let it do the I/O once
+for the whole batch instead: `BatchSerializeHandlerInterface` for the way out, `BatchDeserializeHandlerInterface`
+for the way in.
+
+#### Serialization
+
+Implement `BatchSerializeHandlerInterface` and the serializer will hand you every value of the collection as a
+`BatchItem` before it asks you to serialize the first one, so you can resolve them all at once:
+
+```php
+use AnzuSystems\SerializerBundle\Context\SerializationContext;
+use AnzuSystems\SerializerBundle\Handler\BatchItem;
+use AnzuSystems\SerializerBundle\Handler\Handlers\AbstractHandler;
+use AnzuSystems\SerializerBundle\Handler\Handlers\BatchSerializeHandlerInterface;
+use AnzuSystems\SerializerBundle\Metadata\Metadata;
+
+final class AuthorHandler extends AbstractHandler implements BatchSerializeHandlerInterface
+{
+    /** @var array<int, Author> */
+    private array $authors = [];
+
+    public function prepareSerializeBatch(SerializationContext $context, BatchItem ...$items): void
+    {
+        $ids = array_filter(array_map(static fn (BatchItem $item): mixed => $item->value, $items), 'is_int');
+        foreach ($this->authorRepository->findByIds(array_diff($ids, array_keys($this->authors))) as $author) {
+            $this->authors[$author->getId()] = $author;
+        }
+    }
+
+    /**
+     * @param int|null $value
+     */
+    public function serialize(mixed $value, Metadata $metadata, SerializationContext $context): ?array
+    {
+        // one query for the whole collection instead of one per item
+    }
+}
+```
+
+The handler is still forced on the property the usual way, `#[Serialize(handler: AuthorHandler::class)]`.
+Serialization output is not affected in any way - preparing a batch only changes how many times the handler
+has to go and fetch something.
+
+Worth knowing before you rely on it:
+
+- `prepareSerializeBatch()` is called **once per serialized collection**, with every value the handler owns across
+  every item, each carrying the `Metadata` of the property it came from - so a handler parametrized by metadata
+  (`customType`, `strategy`, `orderBy`) can group the items itself instead of being handed them pre-split.
+- The values it reads are reused when serializing, so a getter behind a batched property is called once per item.
+- A collection nested inside every item of another collection is prepared once per parent item.
+- It may be called **several times per request** (a response can contain more than one collection), so it has
+  to be idempotent - keep what you already resolved. Keep it in a store that is reset between runs, though:
+  handlers are container singletons, so an unbounded map on the handler itself outlives the response in a
+  worker.
+- Only `array` and `Doctrine\Common\Collections\Collection` are prepared. A generator or a plain iterator is
+  skipped, because traversing it twice would consume the data that is about to be serialized; such handlers
+  fall back to resolving value by value.
+- Only handlers forced via `#[Serialize(handler: ...)]` are prepared, not the automatically resolved ones.
+- Values arrive in the order of the collection, duplicates and nulls included. Filtering is up to the handler.
+
+#### Deserialization
+
+Implement `BatchDeserializeHandlerInterface` and you get every property of the object about to be deserialized
+that this handler is responsible for, each as a `BatchItem` carrying the raw value and its `Metadata`, before the
+first `deserialize()` call:
+
+```php
+use AnzuSystems\SerializerBundle\Handler\BatchItem;
+use AnzuSystems\SerializerBundle\Handler\Handlers\BatchDeserializeHandlerInterface;
+
+final class AuthorHandler extends AbstractHandler implements BatchDeserializeHandlerInterface
+{
+    public function prepareDeserializeBatch(BatchItem ...$items): void
+    {
+        // group by whatever the handler keys its lookups on, then one fetch per group
+    }
+}
+```
+
+The point is the grouping the handler itself controls: three properties pointing at the same target type share
+one lookup, and a property holding a single value joins the same batch instead of costing a lookup of its own.
+
+Worth knowing:
+
+- The pass covers **one object at a time**, the properties present in the incoming data that this handler owns.
+  A list payload (`deserialize($json, Foo::class, [])`) is prepared for the whole list first, so N items cost one
+  batch, not N. Nested objects are prepared when their own turn comes, not together with their parent.
+- Properties without a setter are skipped unless they are constructor arguments, matching what deserialization
+  itself does.
+- It is guaranteed to run **before the first `deserialize()` call** of that object, constructor arguments
+  included, so a handler may treat it as the one place where it fetches - `EntityIdHandler` does exactly that
+  and afterwards only reads Doctrine's identity map, which is what keeps an id that does not exist from costing
+  a query of its own.
+
 ### Automatically generated API documentation via NelmioApiDocBundle 
 
 Model describer will be automatically registered if [NelmioApiDocBundle](https://github.com/nelmio/NelmioApiDocBundle) is present.

docker-compose.yml

@@ -17,7 +17,6 @@ services:
   mysql:
     image: mysql:8
     command:
-      - --default-authentication-plugin=mysql_native_password
       - --disable-log-bin
     env_file:
       - .env.docker.dist

src/Handler/BatchItem.php

@@ -0,0 +1,16 @@
+<?php
+
+declare(strict_types=1);
+
+namespace AnzuSystems\SerializerBundle\Handler;
+
+use AnzuSystems\SerializerBundle\Metadata\Metadata;
+
+final readonly class BatchItem
+{
+    public function __construct(
+        public mixed $value,
+        public Metadata $metadata,
+    ) {
+    }
+}

src/Handler/HandlerResolver.php

@@ -19,6 +19,18 @@ public function __construct(
     ) {
     }
 
+    /**
+     * @throws SerializerException
+     */
+    public function getHandler(string $handlerClass): HandlerInterface
+    {
+        try {
+            return $this->handlerLocator->get($handlerClass);
+        } catch (NotFoundExceptionInterface|ContainerExceptionInterface $exception) {
+            throw new SerializerException('Unable to get handler.', 0, $exception);
+        }
+    }
+
     /**
      * @throws SerializerException
      */

src/Handler/Handlers/BatchDeserializeHandlerInterface.php

@@ -0,0 +1,16 @@
+<?php
+
+declare(strict_types=1);
+
+namespace AnzuSystems\SerializerBundle\Handler\Handlers;
+
+use AnzuSystems\SerializerBundle\Exception\SerializerException;
+use AnzuSystems\SerializerBundle\Handler\BatchItem;
+
+interface BatchDeserializeHandlerInterface extends HandlerInterface
+{
+    /**
+     * @throws SerializerException
+     */
+    public function prepareDeserializeBatch(BatchItem ...$items): void;
+}

src/Handler/Handlers/BatchSerializeHandlerInterface.php

@@ -0,0 +1,17 @@
+<?php
+
+declare(strict_types=1);
+
+namespace AnzuSystems\SerializerBundle\Handler\Handlers;
+
+use AnzuSystems\SerializerBundle\Context\SerializationContext;
+use AnzuSystems\SerializerBundle\Exception\SerializerException;
+use AnzuSystems\SerializerBundle\Handler\BatchItem;
+
+interface BatchSerializeHandlerInterface extends HandlerInterface
+{
+    /**
+     * @throws SerializerException
+     */
+    public function prepareSerializeBatch(SerializationContext $context, BatchItem ...$items): void;
+}

src/Handler/Handlers/EntityIdHandler.php

@@ -7,24 +7,45 @@
 use AnzuSystems\SerializerBundle\Attributes\Serialize;
 use AnzuSystems\SerializerBundle\Context\SerializationContext;
 use AnzuSystems\SerializerBundle\Exception\SerializerException;
+use AnzuSystems\SerializerBundle\Handler\BatchItem;
 use AnzuSystems\SerializerBundle\Helper\SerializerHelper;
 use AnzuSystems\SerializerBundle\Metadata\Metadata;
 use Doctrine\Common\Collections\ArrayCollection;
 use Doctrine\Common\Collections\Collection;
+use Doctrine\DBAL\Exception as DbalException;
+use Doctrine\DBAL\Types\Type;
 use Doctrine\ORM\EntityManagerInterface;
+use Doctrine\ORM\Mapping\ClassMetadata as DoctrineClassMetadata;
+use Doctrine\Persistence\Mapping\MappingException;
 use ReflectionException;
 use ReflectionMethod;
 use ReflectionNamedType;
 use Symfony\Component\TypeInfo\TypeIdentifier;
 use Symfony\Component\Uid\Uuid;
 
-final class EntityIdHandler extends AbstractHandler
+final class EntityIdHandler extends AbstractHandler implements BatchDeserializeHandlerInterface
 {
     public function __construct(
         private readonly EntityManagerInterface $entityManager,
     ) {
     }
 
+    public function prepareDeserializeBatch(BatchItem ...$items): void
+    {
+        $idsByEntityClass = [];
+        foreach ($items as $item) {
+            $entityClass = (string) ($item->metadata->customType ?? $item->metadata->type);
+            $value = $item->value;
+            foreach (is_iterable($value) ? $value : [$value] as $id) {
+                $idsByEntityClass[$entityClass][] = $id;
+            }
+        }
+
+        foreach ($idsByEntityClass as $entityClass => $ids) {
+            $this->preloadEntities($ids, $entityClass);
+        }
+    }
+
     public function serialize(mixed $value, Metadata $metadata, SerializationContext $context): array|object|int|null|string
     {
         if (null === $value) {
@@ -71,14 +92,8 @@ public function deserialize(mixed $value, Metadata $metadata): mixed
             return null;
         }
         if (is_iterable($value)) {
-            $entities = [];
-            foreach ($value as $id) {
-                /** @psalm-suppress ArgumentTypeCoercion */
-                $entity = $this->entityManager->find((string) $metadata->customType, $id);
-                if ($entity) {
-                    $entities[] = $entity;
-                }
-            }
+            $entityClass = (string) $metadata->customType;
+            $entities = $this->loadEntities(iterator_to_array($value, preserve_keys: false), $entityClass);
             if (is_a($metadata->type, Collection::class, true)) {
                 return new ArrayCollection($entities);
             }
@@ -112,6 +127,101 @@ public function describe(string $property, Metadata $metadata): array
         return $description;
     }
 
+    /**
+     * The deserializer runs prepareDeserializeBatch() before it asks for any property, so an id the identity map
+     * does not know is one the database does not have - a find() fallback would only repeat the query that missed.
+     *
+     * @param list<mixed> $ids
+     *
+     * @return list<object>
+     */
+    private function loadEntities(array $ids, string $entityClass): array
+    {
+        /** @psalm-suppress ArgumentTypeCoercion */
+        $classMetadata = $this->entityManager->getClassMetadata($entityClass);
+
+        $entities = [];
+        foreach ($ids as $id) {
+            /** @psalm-suppress ArgumentTypeCoercion */
+            $entity = is_int($id) || is_string($id)
+                ? $this->tryGetManaged($id, $classMetadata)
+                : $this->entityManager->find($entityClass, $id);
+            if (null === $entity || false === $entity instanceof $entityClass) {
+                continue;
+            }
+
+            $entities[] = $entity;
+        }
+
+        return $entities;
+    }
+
+    /**
+     * @param list<mixed> $ids
+     */
+    private function preloadEntities(array $ids, string $entityClass): void
+    {
+        try {
+            /** @psalm-suppress ArgumentTypeCoercion */
+            $classMetadata = $this->entityManager->getClassMetadata($entityClass);
+        } catch (MappingException) {
+            // The handler also serves value objects that merely carry a getId(), and warming up is never worth
+            // breaking a payload over.
+            return;
+        }
+
+        $unmanagedIds = [];
+        foreach ($ids as $id) {
+            if (false === is_int($id) && false === is_string($id)) {
+                continue;
+            }
+            if (null === $this->tryGetManaged($id, $classMetadata)) {
+                $unmanagedIds[$id] = $id;
+            }
+        }
+        if ([] === $unmanagedIds) {
+            return;
+        }
+
+        /** @psalm-suppress ArgumentTypeCoercion */
+        $this->entityManager->getRepository($entityClass)
+            ->findBy([$classMetadata->getSingleIdentifierFieldName() => $unmanagedIds]);
+    }
+
+    private function tryGetManaged(int|string $id, DoctrineClassMetadata $classMetadata): ?object
+    {
+        $identifier = $classMetadata->getSingleIdentifierFieldName();
+
+        /** @psalm-suppress ArgumentTypeCoercion */
+        $entity = $this->entityManager->getUnitOfWork()
+            ->tryGetById(
+                [$identifier => $this->toIdentifierValue($id, $identifier, $classMetadata)],
+                $classMetadata->rootEntityName,
+            );
+
+        return is_object($entity) ? $entity : null;
+    }
+
+    /**
+     * Doctrine registers an entity under the identifier its type produced, so a raw json id has to go through the
+     * same conversion - an uppercase uuid would otherwise look like an id that does not exist, and this path has
+     * no find() to fall back on.
+     */
+    private function toIdentifierValue(int|string $id, string $identifier, DoctrineClassMetadata $classMetadata): mixed
+    {
+        $typeName = $classMetadata->getTypeOfField($identifier);
+        if (null === $typeName) {
+            return $id;
+        }
+
+        try {
+            return Type::getType($typeName)
+                ->convertToPHPValue($id, $this->entityManager->getConnection()->getDatabasePlatform());
+        } catch (DbalException) {
+            return $id;
+        }
+    }
+
     private function getOrderedIDs(array $ids, Metadata $metadata): Collection
     {
         $uuids = false;
@@ -125,7 +235,8 @@ private function getOrderedIDs(array $ids, Metadata $metadata): Collection
             return $id;
         }, $ids);
         /** @psalm-suppress ArgumentTypeCoercion */
-        $dqb = $this->entityManager->getRepository((string) $metadata->customType)->createQueryBuilder('entity');
+        $dqb = $this->entityManager->getRepository((string) $metadata->customType)
+            ->createQueryBuilder('entity');
         $dqb
             ->select('entity.id')
             ->where('entity.id IN (:ids)')
@@ -143,7 +254,8 @@ private function getOrderedIDs(array $ids, Metadata $metadata): Collection
             }
 
             return $id;
-        }, $dqb->getQuery()->getSingleColumnResult());
+        }, $dqb->getQuery()
+            ->getSingleColumnResult());
 
         return new ArrayCollection($resultIds);
     }