Skip to content

perf: bulk materialization publication writes - #411

Open
suguanYang wants to merge 1 commit into
mainfrom
fix/wangbinqi/remove-duplicate-migration-test
Open

perf: bulk materialization publication writes#411
suguanYang wants to merge 1 commit into
mainfrom
fix/wangbinqi/remove-duplicate-migration-test

Conversation

@suguanYang

Copy link
Copy Markdown
Contributor

Summary

  • replace ORM row-by-row map-unit/token persistence with SQLAlchemy Core bulk batches
  • cap demo materialization database publication with a cross-instance Redis leased semaphore (default concurrency 2)
  • add Logfire stage timing for uploads, base rows, publication, snapshot, commit, and semaphore wait

Evidence

  • local production dump: 50k token rows, ORM flush 12.54s vs Core bulk 5.74s (~2.2x faster) with the existing five indexes
  • Core bulk wall time: concurrency 1 = 4.63s, 2 = 4.99s, 4 = 6.85s
  • no production schema, index, or data changes

Validation

  • demo contract: 10 passed
  • retrieval contract: 16 passed
  • worker contract: 330 passed
  • Ruff and Pyright passed

__all__ = [
"RedisService",
"RedisServiceFactory",
"RedisPublicationSemaphore",
"RedisService",
"RedisServiceFactory",
"RedisPublicationSemaphore",
"SyncRedisPublicationSemaphore",
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants