Skip to content

Commit eccaba6

Browse files
abnegateclaude
abnegate
and
claude
committed
Merge branch 'main' into feat-query-lib
Twenty-eight commits. The interesting part is that this branch moved most of Database.php into traits while main added a feature to the monolithic file, so git produced an eight-thousand-line conflict with nothing aligned. Every resolution below is main's change re-applied onto this branch's structure rather than a side taken. main's drop-unknown-attributes (#946): the property, the two accessors and removeUnknownAttributes() land on Database.php, and the two change-detection call sites on Traits\Documents where updateDocument and the batch path now live. Mirror gains the delegating setter. The helper reads attribute keys through the value objects this branch introduces -- Attribute, Document or array -- rather than $attribute['$id'], and asks supports(Capability::DefinedAttributes) where main asked getSupportForAttributes(), which this branch replaced. main's fulltext fix (#826): the unicode-aware sanitiser replaces the reserved-character list in SQL and Postgres, and the empty-term guards go in beside it. main patched MariaDB and Postgres separately; this branch had already consolidated MySQL's search into SQL, so the guard is written once there and MariaDB inherits it. Both of main's new tests come across, ported to the value-object API: createCollection takes a Collection, createAttribute an Attribute, createIndex an Index, and the capability checks go through supports(). Separately, addressing review: Event\DomainEvent is now Event\Domain. The namespace already says event, and no consumer has a competing Domain symbol, so no import needed aliasing. The local variables and createDomainEvent() keep their names -- they describe the action, and Event\Domain still reads as "domain event" through the namespace. phpstan at level max and pint are clean, and the unit suite is 1636 tests / 6259 assertions green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2 parents 771a8e2 + d5b2eca commit eccaba6

14 files changed

Lines changed: 321 additions & 29 deletions

File tree

‎src/Database/Adapter/Postgres.php‎

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1576,13 +1576,21 @@ protected function getSQLCondition(Query $query, array &$binds, ?string $forColl
15761576

15771577
case Method::Search:
15781578
$searchVal = $query->getValue();
1579-
$binds[":{$placeholder}_0"] = $this->getFulltextValue(\is_string($searchVal) ? $searchVal : '');
1579+
$fulltextValue = $this->getFulltextValue(\is_string($searchVal) ? $searchVal : '');
1580+
if ($fulltextValue === '') {
1581+
return '0 = 1';
1582+
}
1583+
$binds[":{$placeholder}_0"] = $fulltextValue;
15801584

15811585
return "to_tsvector(regexp_replace({$attribute}, '[^\w]+',' ','g')) @@ websearch_to_tsquery(:{$placeholder}_0)";
15821586

15831587
case Method::NotSearch:
15841588
$notSearchVal = $query->getValue();
1585-
$binds[":{$placeholder}_0"] = $this->getFulltextValue(\is_string($notSearchVal) ? $notSearchVal : '');
1589+
$fulltextValue = $this->getFulltextValue(\is_string($notSearchVal) ? $notSearchVal : '');
1590+
if ($fulltextValue === '') {
1591+
return '1 = 1';
1592+
}
1593+
$binds[":{$placeholder}_0"] = $fulltextValue;
15861594

15871595
return "NOT (to_tsvector(regexp_replace({$attribute}, '[^\w]+',' ','g')) @@ websearch_to_tsquery(:{$placeholder}_0))";
15881596

@@ -2280,9 +2288,11 @@ protected function bindOperatorParams(PDOStatement|DatabasePDOStatement|PDOState
22802288
protected function getFulltextValue(string $value): string
22812289
{
22822290
$exact = str_ends_with($value, '"') && str_starts_with($value, '"');
2283-
$value = str_replace(['@', '+', '-', '*', '.', "'", '"'], ' ', $value);
2284-
$value = preg_replace('/\s+/', ' ', $value); // Remove multiple whitespaces
2285-
$value = trim($value ?? '');
2291+
2292+
/** Keep only unicode letters, numbers, underscores, and whitespace. */
2293+
$value = preg_replace('/[^\p{L}\p{N}_\s]/u', ' ', $value) ?? '';
2294+
$value = preg_replace('/\s+/', ' ', $value) ?? '';
2295+
$value = trim($value);
22862296

22872297
if (! $exact) {
22882298
$value = str_replace(' ', ' or ', $value);

‎src/Database/Adapter/SQL.php‎

Lines changed: 13 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -5382,13 +5382,21 @@ protected function getSQLCondition(Query $query, array &$binds, ?string $forColl
53825382

53835383
case Method::Search:
53845384
$searchVal = $query->getValue();
5385-
$binds[":{$placeholder}_0"] = $this->getFulltextValue(\is_string($searchVal) ? $searchVal : '');
5385+
$fulltextValue = $this->getFulltextValue(\is_string($searchVal) ? $searchVal : '');
5386+
if ($fulltextValue === '') {
5387+
return '0 = 1';
5388+
}
5389+
$binds[":{$placeholder}_0"] = $fulltextValue;
53865390

53875391
return "MATCH({$alias}.{$attribute}) AGAINST (:{$placeholder}_0 IN BOOLEAN MODE)";
53885392

53895393
case Method::NotSearch:
53905394
$notSearchVal = $query->getValue();
5391-
$binds[":{$placeholder}_0"] = $this->getFulltextValue(\is_string($notSearchVal) ? $notSearchVal : '');
5395+
$fulltextValue = $this->getFulltextValue(\is_string($notSearchVal) ? $notSearchVal : '');
5396+
if ($fulltextValue === '') {
5397+
return '1 = 1';
5398+
}
5399+
$binds[":{$placeholder}_0"] = $fulltextValue;
53925400

53935401
return "NOT (MATCH({$alias}.{$attribute}) AGAINST (:{$placeholder}_0 IN BOOLEAN MODE))";
53945402

@@ -5498,10 +5506,9 @@ protected function getFulltextValue(string $value): string
54985506
{
54995507
$exact = str_ends_with($value, '"') && str_starts_with($value, '"');
55005508

5501-
/** Replace reserved chars with space. */
5502-
$specialChars = '@,+,-,*,),(,<,>,~,"';
5503-
$value = str_replace(explode(',', $specialChars), ' ', $value);
5504-
$value = (string) preg_replace('/\s+/', ' ', $value); // Remove multiple whitespaces
5509+
/** Keep only unicode letters, numbers, underscores, and whitespace. */
5510+
$value = preg_replace('/[^\p{L}\p{N}_\s]/u', ' ', $value) ?? '';
5511+
$value = preg_replace('/\s+/', ' ', $value) ?? '';
55055512
$value = trim($value);
55065513

55075514
if (empty($value)) {

‎src/Database/Database.php‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -271,6 +271,8 @@ class Database
271271

272272
protected bool $validate = true;
273273

274+
protected bool $dropUnknownAttributes = false;
275+
274276
protected bool $preserveDates = false;
275277

276278
protected bool $preserveSequence = false;
@@ -996,6 +998,25 @@ public function setPreserveDates(bool $preserve): static
996998
return $this;
997999
}
9981000

1001+
public function getDropUnknownAttributes(): bool
1002+
{
1003+
return $this->dropUnknownAttributes;
1004+
}
1005+
1006+
/**
1007+
* Drop attributes missing from the collection schema instead of rejecting the write.
1008+
*
1009+
* Enable this where the schema is owned by the application rather than the caller, so a
1010+
* deploy that writes an attribute before its migration has run degrades to a warning
1011+
* instead of failing every write.
1012+
*/
1013+
public function setDropUnknownAttributes(bool $drop): static
1014+
{
1015+
$this->dropUnknownAttributes = $drop;
1016+
1017+
return $this;
1018+
}
1019+
9991020
/**
10001021
* Get whether date preservation is enabled.
10011022
*
@@ -1542,6 +1563,8 @@ public function encode(Document $collection, Document $document, bool $applyDefa
15421563
$attributes[] = $attribute;
15431564
}
15441565

1566+
$known = [];
1567+
15451568
foreach ($attributes as $attribute) {
15461569
if ($attribute instanceof Attribute) {
15471570
$key = $attribute->key;
@@ -1561,6 +1584,7 @@ public function encode(Document $collection, Document $document, bool $applyDefa
15611584
} else {
15621585
continue;
15631586
}
1587+
$known[$key] = true;
15641588
/** @var array<string> $filters */
15651589
$exists = $document->offsetExists($key);
15661590
$value = $exists ? $document[$key] : null;
@@ -1612,6 +1636,55 @@ public function encode(Document $collection, Document $document, bool $applyDefa
16121636
$document->setAttribute($key, $value);
16131637
}
16141638

1639+
return $this->removeUnknownAttributes($collection, $document, $known);
1640+
}
1641+
1642+
/**
1643+
* Remove attributes the collection schema does not declare.
1644+
*
1645+
* Used ahead of change detection on update and upsert so a dropped key is not
1646+
* counted as a write. encode() also calls this once it has collected the known
1647+
* set while iterating the schema.
1648+
*
1649+
* @param array<string, true>|null $known Attribute ids already collected
1650+
*/
1651+
protected function removeUnknownAttributes(Document $collection, Document $document, ?array $known = null): Document
1652+
{
1653+
if (! $this->dropUnknownAttributes || ! $this->adapter->supports(Capability::DefinedAttributes)) {
1654+
return $document;
1655+
}
1656+
1657+
if ($known === null) {
1658+
$known = [];
1659+
$declared = $collection->getAttribute('attributes', []);
1660+
foreach (\is_array($declared) ? $declared : [] as $attribute) {
1661+
$key = match (true) {
1662+
$attribute instanceof Attribute => $attribute->key,
1663+
$attribute instanceof Document => $attribute->getId(),
1664+
\is_array($attribute) => \is_string($attribute[Document::ID] ?? null) ? $attribute[Document::ID] : '',
1665+
default => '',
1666+
};
1667+
$known[$key] = true;
1668+
}
1669+
}
1670+
1671+
$dropped = [];
1672+
foreach (\array_keys($document->getArrayCopy()) as $key) {
1673+
if (\str_starts_with((string) $key, '$') || isset($known[$key])) {
1674+
continue;
1675+
}
1676+
1677+
$dropped[] = $key;
1678+
$document->removeAttribute((string) $key);
1679+
}
1680+
1681+
if (! empty($dropped)) {
1682+
Console::warning(
1683+
'Dropped unknown attributes "'.\implode('", "', $dropped).'" from collection "'.$collection->getId().'"'
1684+
.($this->adapter->getTenant() === null ? '' : ' on tenant '.$this->adapter->getTenant())
1685+
);
1686+
}
1687+
16151688
return $document;
16161689
}
16171690

‎src/Database/Event/Collection/Created.php‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,9 @@
44

55
use Utopia\Database\Document;
66
use Utopia\Database\Event;
7-
use Utopia\Database\Event\DomainEvent;
7+
use Utopia\Database\Event\Domain;
88

9-
class Created extends DomainEvent
9+
class Created extends Domain
1010
{
1111
public function __construct(
1212
string $collection,

‎src/Database/Event/Collection/Deleted.php‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,9 @@
33
namespace Utopia\Database\Event\Collection;
44

55
use Utopia\Database\Event;
6-
use Utopia\Database\Event\DomainEvent;
6+
use Utopia\Database\Event\Domain;
77

8-
class Deleted extends DomainEvent
8+
class Deleted extends Domain
99
{
1010
public function __construct(
1111
string $collection,

‎src/Database/Event/DispatcherHook.php‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ public function handle(Event $event, mixed $data): void
5252
}
5353
}
5454

55-
private function createDomainEvent(Event $event, mixed $data): ?DomainEvent
55+
private function createDomainEvent(Event $event, mixed $data): ?Domain
5656
{
5757
return match ($event) {
5858
Event::DocumentCreate, Event::DocumentsCreate => $data instanceof Document

‎src/Database/Event/Document/Created.php‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,9 @@
44

55
use Utopia\Database\Document;
66
use Utopia\Database\Event;
7-
use Utopia\Database\Event\DomainEvent;
7+
use Utopia\Database\Event\Domain;
88

9-
class Created extends DomainEvent
9+
class Created extends Domain
1010
{
1111
public function __construct(
1212
string $collection,

‎src/Database/Event/Document/Deleted.php‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,9 @@
33
namespace Utopia\Database\Event\Document;
44

55
use Utopia\Database\Event;
6-
use Utopia\Database\Event\DomainEvent;
6+
use Utopia\Database\Event\Domain;
77

8-
class Deleted extends DomainEvent
8+
class Deleted extends Domain
99
{
1010
public function __construct(
1111
string $collection,

‎src/Database/Event/Document/Updated.php‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,9 @@
44

55
use Utopia\Database\Document;
66
use Utopia\Database\Event;
7-
use Utopia\Database\Event\DomainEvent;
7+
use Utopia\Database\Event\Domain;
88

9-
class Updated extends DomainEvent
9+
class Updated extends Domain
1010
{
1111
public function __construct(
1212
string $collection,

src/Database/Event/DomainEvent.php renamed to src/Database/Event/Domain.php

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
use Utopia\Database\Event;
66

7-
class DomainEvent
7+
class Domain
88
{
99
public readonly \DateTimeImmutable $occurredAt;
1010

0 commit comments

Comments
 (0)