diff --git a/docs/content/docs/configuration/android.mdx b/docs/content/docs/configuration/android.mdx index 9f2b1055..d0a5a2cc 100644 --- a/docs/content/docs/configuration/android.mdx +++ b/docs/content/docs/configuration/android.mdx @@ -39,9 +39,30 @@ To omit R\*Tree from the bundled build, set this boolean in your app's `package. `enableRTree` defaults to `true` independently of `performanceMode`. A non-boolean value fails Gradle configuration. Disabling it omits the default `SQLITE_ENABLE_RTREE` definition. Do not add `-DSQLITE_ENABLE_RTREE=0` to `nitroSqliteFlags` to disable it: SQLite checks whether the macro is defined, so that value still enables the module. Custom compile definitions can enable the module even when the package setting is `false`. +## Commit I/O + +SQLite makes a committed transaction durable by writing it to storage and asking the operating system to flush the written data. In the default rollback journal mode, a commit first copies the original pages into a journal file and flushes it, then writes the new pages to the database file and flushes again, and finally removes the journal. For small writes, these flushes take most of a commit's time. + +Android builds compile SQLite with two I/O options that Android's own platform SQLite also uses. Neither changes `PRAGMA synchronous`, and a commit is as durable as before: + +- `HAVE_FDATASYNC=1` flushes with `fdatasync()` instead of `fsync()`. On Linux, `fdatasync()` still flushes the metadata needed to read the data back, such as the file size, and skips metadata such as modification times. +- `SQLITE_ENABLE_BATCH_ATOMIC_WRITE=1` commits a transaction through the F2FS filesystem's atomic write when the database file supports it. The filesystem then stores either all of the transaction's pages or none of them, so SQLite does not need a journal file. When the filesystem lacks the feature or the atomic write fails, SQLite commits with the rollback journal as usual. + +To build without batch atomic writes, set this boolean in your app's `package.json` and rebuild the native app: + +```json +{ + "nitroSQLite": { + "enableBatchAtomicWrite": false + } +} +``` + +`enableBatchAtomicWrite` defaults to `true`, and a non-boolean value fails Gradle configuration. As with R\*Tree, SQLite only checks whether the macro is defined, so `-DSQLITE_ENABLE_BATCH_ATOMIC_WRITE=0` in `nitroSqliteFlags` does not disable it. To flush with `fsync()` instead, add `-DHAVE_FDATASYNC=0` to `nitroSqliteFlags`. + ## Threading -The native library opens each database with `SQLITE_OPEN_FULLMUTEX` and serializes calls on each handle. The JavaScript connection helper coordinates work per managed connection and rejects conflicting synchronous work. `nitroSqliteFlags` can change compile-time SQLite behavior, including `SQLITE_THREADSAFE`. Nitro SQLite rejects [independent connections](/docs/guides/multiple-connections) when SQLite was built with `SQLITE_THREADSAFE=0`; other database handles can still run concurrently, so an app disabling mutexes must serialize SQLite calls across the process. +The native library serializes all work on each database handle with its own lock, so it opens handles with `SQLITE_OPEN_NOMUTEX` instead of also taking SQLite's per-call connection mutex. Each connection runs its async queries, batches, and file imports one at a time on a native thread of its own. The thread starts with the connection's first such call and stops once the connection is closed and its pending work has finished. The JavaScript connection helper coordinates work per managed connection and rejects conflicting synchronous work. `nitroSqliteFlags` can change compile-time SQLite behavior, including `SQLITE_THREADSAFE`. Nitro SQLite rejects [independent connections](/docs/guides/multiple-connections) when SQLite was built with `SQLITE_THREADSAFE=0`; other database handles can still run concurrently, so an app disabling mutexes must serialize SQLite calls across the process. The app's `nitroSQLite.threadSafe` and `nitroSQLite.performanceMode` settings also configure Android's SQLite defaults. Custom `nitroSqliteFlags` override individual defaults. The native build additionally applies its CMake compiler options, including `-O2`. diff --git a/docs/content/docs/guides/parameters-and-results.mdx b/docs/content/docs/guides/parameters-and-results.mdx index 1e1d763f..7bb0d575 100644 --- a/docs/content/docs/guides/parameters-and-results.mdx +++ b/docs/content/docs/guides/parameters-and-results.mdx @@ -68,4 +68,4 @@ try { When a native `NitroSQLiteException` reaches a managed helper, `error.type` holds a [`NitroSQLiteExceptionType`](/api/react-native-nitro-sqlite/type-aliases/NitroSQLiteExceptionType) such as `SqlExecutionError`. It is `undefined` for JavaScript errors, including connection queue errors, and unrecognized native categories. These categories are not SQLite's numeric error codes. The native categories are `UnknownError`, `DatabaseCannotBeOpened`, `EncryptionNotEnabled`, `DatabaseCannotBeDecrypted`, `DatabaseNotOpen`, `UnableToAttachToDatabase`, `SqlExecutionError`, `CouldNotLoadFile`, and `NoBatchCommandsProvided`. See [encrypt a database](/docs/guides/encryption#handle-open-failures) for the encryption errors. The current native `DatabaseNotOpen` helper emits `UnableToAttachToDatabase`, and a missing database file emits `SqlExecutionError`. Calls through `NitroSQLite.native` throw raw native errors. -For exact types, see [`QueryResult`](/api/react-native-nitro-sqlite/type-aliases/QueryResult), [`NitroSQLiteQueryResult`](/api/react-native-nitro-sqlite/hybrid-objects/NitroSQLiteQueryResult), and [`NitroSQLiteQueryResultRows`](/api/react-native-nitro-sqlite/type-aliases/NitroSQLiteQueryResultRows). The row generic narrows `rows._array` and `rows.item()`, but not the raw `results` array. Query results also inherit Nitro hybrid object members; see [native access](/docs/guides/sync-and-async#global-helpers-and-native-access). +For exact types, see [`QueryResult`](/api/react-native-nitro-sqlite/type-aliases/QueryResult), [`NitroSQLiteQueryResult`](/api/react-native-nitro-sqlite/interfaces/NitroSQLiteQueryResult), and [`NitroSQLiteQueryResultRows`](/api/react-native-nitro-sqlite/type-aliases/NitroSQLiteQueryResultRows). The row generic narrows `rows._array` and `rows.item()`, but not the raw `results` array. Query results are plain JavaScript objects; see [native access](/docs/guides/sync-and-async#global-helpers-and-native-access) for results returned without the connection helper. diff --git a/docs/content/docs/guides/performance.mdx b/docs/content/docs/guides/performance.mdx index 5b121771..ca579f8f 100644 --- a/docs/content/docs/guides/performance.mdx +++ b/docs/content/docs/guides/performance.mdx @@ -21,6 +21,21 @@ console.log(rows._array) An async call still consumes device CPU and database I/O. Paginate large result sets instead of bringing every row into JavaScript at once. +## Reuse SQL text + +Before SQLite runs a statement, it compiles the SQL text into a program for its virtual machine. Compiling one statement is quick, but the cost adds up when the same statement runs many times, such as an `INSERT` in a loop. + +Each connection keeps the compiled programs of its 32 most recently used queries and data changes: statements that start with `SELECT`, `INSERT`, `UPDATE`, `DELETE`, `REPLACE`, `WITH`, or `VALUES`. When `execute()` or `executeAsync()` receives the same SQL text again, it reuses the program with the new parameters. Pass values as parameters rather than formatting them into the SQL so the text stays identical between calls. Other statements, such as `PRAGMA`, `ATTACH`, and `BEGIN`, compile on every call because SQLite can apply their effect while compiling them. Use [prepared statements](/docs/guides/prepared-statements) when you want to manage a compiled statement's lifetime yourself. + +```ts +for (const metric of metrics) { + db.execute('INSERT INTO metrics (key, value) VALUES (?, ?)', [ + metric.key, + metric.value, + ]) +} +``` + ## Group related writes One `executeBatchAsync()` call executes a fixed set of statements inside a native transaction. Use nested parameter arrays for repeated SQL. Use `db.transaction()` when application logic must inspect a result between writes. Both keep a transaction open while they work; keep the scope short and await each operation inside a transaction callback. diff --git a/docs/content/docs/guides/sync-and-async.mdx b/docs/content/docs/guides/sync-and-async.mdx index 8561d654..42066435 100644 --- a/docs/content/docs/guides/sync-and-async.mdx +++ b/docs/content/docs/guides/sync-and-async.mdx @@ -20,6 +20,8 @@ console.log(smallResult.results, largerResult.results) Use synchronous calls when immediate results are useful and the work is small. Prefer async calls for queries that may scan many rows, batches, or file imports. [Performance guidance](/docs/guides/performance) covers other ways to keep work bounded. +An async query still delivers its rows to JavaScript, and only the JavaScript thread can turn them into objects. For a large result, `executeAsync()` hands rows over in batches while SQLite reads the rest, so building the objects overlaps with the query instead of starting after it. The promise still resolves once, with every row in order. + An `open()` connection submits ordinary async statements in call order. Native operations on that connection run one at a time in submission order, but await a write before starting a read that depends on it so you can handle write errors. Batches and transactions wait for preceding statements and reserve the connection until complete. A synchronous operation, `close()`, or `delete()` throws a busy error if work is pending or active. Await async work before calling a synchronous method on the same connection: ```ts @@ -38,7 +40,7 @@ Prepared statement `executeAsync()` reserves the connection queue until it finis The [`NitroSQLite` export](/api/react-native-nitro-sqlite/variables/NitroSQLite) also has `execute`, `executeAsync`, `prepare`, `executeBatch`, `executeBatchAsync`, and `transaction` helpers that take a database name. `NitroSQLite.open(options)` is the same helper as the named [`open()` export](/api/react-native-nitro-sqlite/functions/open). Global queries join the default connection's JavaScript queue when the name was opened through `open()`. They do not address independent connections; use the object returned by `open()` for those. Without a default managed connection, global `execute` and `executeAsync` call the native methods directly and still need an open native database handle. Global `prepare`, batches, and transactions need a default connection opened through `open()`. -`NitroSQLite.native` exposes the underlying [`NitroSQLiteNative` hybrid object](/api/react-native-nitro-sqlite/hybrid-objects/NitroSQLiteNative). Its methods take a database name and return raw results without the connection helper's `rows` container. They also throw native errors without converting them to `NitroSQLiteError`. +`NitroSQLite.native` exposes the underlying [`NitroSQLiteNative` hybrid object](/api/react-native-nitro-sqlite/hybrid-objects/NitroSQLiteNative). Its methods take a database name and return raw results without the connection helper's `rows` container. They also throw native errors without converting them to `NitroSQLiteError`. Its `executeAsync()` accepts an optional `onRows` callback that receives leading rows in batches, in order; the resolved `results` then holds only the rows after the last batch. ```ts import { NitroSQLite } from 'react-native-nitro-sqlite' @@ -52,6 +54,6 @@ try { } ``` -Native calls bypass the JavaScript queue. If you use them alongside an `open()` connection, coordinate access yourself. A native statement can run inside an active connection transaction without joining its callback's sequence. The raw result is a Nitro hybrid object with `name`, `toString()`, `equals(other)`, and `dispose()` members. Disposing it makes that result unusable. `dispose()` is not a database `close()` call, and ordinary garbage collection handles these objects. Do not dispose the shared `NitroSQLite.native` instance during normal database cleanup. +Native calls bypass the JavaScript queue. If you use them alongside an `open()` connection, coordinate access yourself. A native statement can run inside an active connection transaction without joining its callback's sequence. The raw result is a plain JavaScript object with `rowsAffected`, `insertId`, `results`, and `metadata`; ordinary garbage collection reclaims it. `NitroSQLite.native` itself is a Nitro hybrid object with a `dispose()` member. Do not dispose that shared instance during normal database cleanup; `dispose()` is not a database `close()` call. The `NitroSQLite` export spreads the hybrid instance, but inherited native methods do not appear on that top-level object. Call them through `.native`. If you compile SQLite with `SQLITE_THREADSAFE=0`, also serialize access across separate database handles and native threads. See the [iOS](/docs/configuration/ios#thread-safety-and-performance-mode) and [Android](/docs/configuration/android#threading) configuration pages. diff --git a/docs/next.config.mjs b/docs/next.config.mjs index e8f9ba36..bb2b45c2 100644 --- a/docs/next.config.mjs +++ b/docs/next.config.mjs @@ -47,9 +47,9 @@ const config = { }, { source: - '/api/react-native-nitro-sqlite/interfaces/NitroSQLiteQueryResult', - destination: '/api/react-native-nitro-sqlite/hybrid-objects/NitroSQLiteQueryResult', + destination: + '/api/react-native-nitro-sqlite/interfaces/NitroSQLiteQueryResult', permanent: true, }, ] diff --git a/docs/scripts/check-api.mjs b/docs/scripts/check-api.mjs index 1cce93a1..96fc4c3e 100644 --- a/docs/scripts/check-api.mjs +++ b/docs/scripts/check-api.mjs @@ -239,7 +239,7 @@ assert.ok( corePackageHome.indexOf('## Classes'), 'Hybrid Objects must appear before other API groups', ) -for (const name of ['NitroSQLiteNative', 'NitroSQLiteQueryResult']) { +for (const name of ['NitroSQLiteNative']) { assert.ok( pagePaths.has(`react-native-nitro-sqlite/hybrid-objects/${name}.mdx`), `Missing Hybrid Object page: ${name}`, diff --git a/example/tests/unit/specs/operations/execute.spec.ts b/example/tests/unit/specs/operations/execute.spec.ts index c7f56039..e21a7b69 100644 --- a/example/tests/unit/specs/operations/execute.spec.ts +++ b/example/tests/unit/specs/operations/execute.spec.ts @@ -7,6 +7,8 @@ import type { ColumnType } from 'react-native-nitro-sqlite' import type { NitroSQLiteQueryResult } from '@nitro-sqlite/specs/NitroSQLiteQueryResult.nitro' const QUERY_RESULT_SIZES = [60, 1_000, 10_000] +// Asynchronous reads hand rows to JavaScript in batches of 256; cover the batch edges. +const ASYNC_BATCH_EDGE_SIZES = [255, 256, 257, 512, 513] const metadataQuery = ` SELECT boolean_value, @@ -150,14 +152,29 @@ export default function registerExecuteUnitTests() { const db = createQueryResultTestDb('query_result_rows_async') try { - for (const size of QUERY_RESULT_SIZES) { + for (const size of [...QUERY_RESULT_SIZES, ...ASYNC_BATCH_EDGE_SIZES]) { const result = await db.executeAsync( 'SELECT * FROM QueryResultRows ORDER BY id LIMIT ?', [size], ) expectQueryResultRows(result, size) + expect(result.results).toBe(result.rows._array) + expect( + result.rows._array.every((row, index) => row.id === index + 1), + ).toBe(true) } + + await db.transaction(async (tx) => { + const result = await tx.executeAsync( + 'SELECT * FROM QueryResultRows ORDER BY id LIMIT ?', + [1_000], + ) + expectQueryResultRows(result, 1_000) + expect( + result.rows._array.every((row, index) => row.id === index + 1), + ).toBe(true) + }) } finally { db.close() db.delete() @@ -248,7 +265,7 @@ export default function registerExecuteUnitTests() { }) describe('Select', () => { - it('keeps positional columns and repeated result reads independent', () => { + it('keeps positional columns in one plain result array', () => { const result = testDb.execute( 'SELECT 1 AS duplicate, 2 AS duplicate, 3.5 AS "café", NULL AS nullable, zeroblob(2) AS payload', ) @@ -263,14 +280,9 @@ export default function registerExecuteUnitTests() { ).toEqual([0, 0]) expect(result.metadata?.duplicate?.index).toBe(0) - const firstRead = result.results - const secondRead = result.results - expect(secondRead).not.toBe(firstRead) - expect(secondRead[0]).not.toBe(firstRead[0]) - firstRead[0]!.duplicate = 9 - expect(secondRead[0]?.duplicate).toBe(2) - expect(result.results[0]?.duplicate).toBe(2) - expect(result.rows.item(0)?.duplicate).toBe(2) + // The native result is a plain object: `results` is one array, shared with `rows`. + expect(result.results).toBe(result.results) + expect(result.rows._array).toBe(result.results) }) it('preserves column metadata for empty results', () => { diff --git a/packages/react-native-nitro-sqlite/android/sqlite-flags.gradle b/packages/react-native-nitro-sqlite/android/sqlite-flags.gradle index 40e34142..3f63391d 100644 --- a/packages/react-native-nitro-sqlite/android/sqlite-flags.gradle +++ b/packages/react-native-nitro-sqlite/android/sqlite-flags.gradle @@ -19,6 +19,7 @@ project.ext.resolveNitroSqliteDefaultFlags = { File appPackageFile, String custo def threadSafe = readBooleanFlag("threadSafe") def performanceMode = readBooleanFlag("performanceMode") def enableRTree = readBooleanFlag("enableRTree") + def enableBatchAtomicWrite = readBooleanFlag("enableBatchAtomicWrite") def customFlagNames = (customFlags =~ /-D([A-Za-z_][A-Za-z0-9_]*)/).collect { it[1] }.toSet() def defaultFlags = [threadSafe ? '-DSQLITE_THREADSAFE=1' : '-DSQLITE_THREADSAFE=0'] @@ -26,6 +27,16 @@ project.ext.resolveNitroSqliteDefaultFlags = { File appPackageFile, String custo if (enableRTree) { defaultFlags += '-DSQLITE_ENABLE_RTREE=1' } + // Without this SQLite substitutes fsync() for fdatasync(). Linux's fdatasync() also flushes the + // metadata needed to read the data back, such as file size, so commits stay durable. Android's + // platform SQLite makes the same choice. + defaultFlags += '-DHAVE_FDATASYNC=1' + // On F2FS, commit transactions with the filesystem's atomic batch write instead of a rollback + // journal file. SQLite falls back to the journal when the filesystem lacks support. + // Like RTree, SQLite only checks whether the macro is defined. + if (enableBatchAtomicWrite) { + defaultFlags += '-DSQLITE_ENABLE_BATCH_ATOMIC_WRITE=1' + } if (performanceMode) { defaultFlags += [ "-DSQLITE_DQS=0", diff --git a/packages/react-native-nitro-sqlite/cpp/NitroSQLiteDatabaseConnections.cpp b/packages/react-native-nitro-sqlite/cpp/NitroSQLiteDatabaseConnections.cpp index eb6e4f14..21b01cbe 100644 --- a/packages/react-native-nitro-sqlite/cpp/NitroSQLiteDatabaseConnections.cpp +++ b/packages/react-native-nitro-sqlite/cpp/NitroSQLiteDatabaseConnections.cpp @@ -1,15 +1,22 @@ #include "NitroSQLiteDatabaseConnections.hpp" #include "NitroSQLiteDatabaseMigration.hpp" #include "NitroSQLiteException.hpp" +#include "NitroSQLiteLogs.hpp" #include +#include #include #include #include #include +#include #include #include #include +#ifdef ANDROID +#include +#endif + namespace margelo::nitro::rnnitrosqlite { namespace fs = std::filesystem; @@ -149,6 +156,69 @@ namespace { } // namespace +// The thread owns this state, so it outlives a worker destroyed by its own last operation. +struct SerialWorker::State { + explicit State(std::string workerName) : name(std::move(workerName)) {} + + const std::string name; + std::mutex mutex; + std::condition_variable condition; + std::queue> operations; + bool stopping = false; +}; + +SerialWorker::SerialWorker(std::string name) : _state(std::make_shared(std::move(name))) { + _thread = std::thread([state = _state] { +#ifdef ANDROID + // Settling a promise posts to the JS thread through JNI, which attaches and detaches an + // unattached thread on every call. Stay attached for the thread's lifetime instead. + facebook::jni::ThreadScope jniScope; +#endif + while (true) { + std::function operation; + { + std::unique_lock lock(state->mutex); + state->condition.wait(lock, [&] { return state->stopping || !state->operations.empty(); }); + if (state->operations.empty()) { + return; + } + operation = std::move(state->operations.front()); + state->operations.pop(); + } + try { + operation(); + } catch (const std::exception& error) { + LOGE("Async operation on database %s failed while settling its promise: %s", state->name.c_str(), error.what()); + } catch (...) { + LOGE("Async operation on database %s failed while settling its promise", state->name.c_str()); + } + // Releasing the operation may drop the last reference to the connection that owns this worker. + operation = nullptr; + } + }); +} + +SerialWorker::~SerialWorker() { + { + std::lock_guard lock(_state->mutex); + _state->stopping = true; + } + _state->condition.notify_one(); + if (_thread.get_id() == std::this_thread::get_id()) { + _thread.detach(); + } else { + _thread.join(); + } +} + +void SerialWorker::enqueue(std::function operation) { + { + std::lock_guard lock(_state->mutex); + _state->operations.push(std::move(operation)); + } + _state->condition.notify_one(); +} + SQLiteConnection::SQLiteConnection(std::string connectionName, fs::path path, bool isReadOnly, sqlite3* handle) : name(std::move(connectionName)), physicalPath(canonicalDatabasePath(path)), readOnly(isReadOnly), database(handle) {} @@ -156,11 +226,20 @@ SQLiteConnection::~SQLiteConnection() { close(); } +void SQLiteConnection::enqueueAsync(std::function operation) { + std::lock_guard lock(asyncWorkerMutex); + if (!asyncWorker) { + asyncWorker = std::make_unique(name); + } + asyncWorker->enqueue(std::move(operation)); +} + void SQLiteConnection::close() noexcept { std::lock_guard lock(mutex); if (database == nullptr) { return; } + statementCache.clear(); sqlite3_close_v2(database); database = nullptr; } @@ -206,7 +285,9 @@ void DatabaseConnections::openKey(const std::string& key, const fs::path& path, validateEncryptionKey(encryptionKey); const auto physicalPath = canonicalDatabasePath(path); rejectPlaintextDatabase(physicalPath, encryptionKey); - const int flags = (readOnly ? SQLITE_OPEN_READONLY : SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE) | SQLITE_OPEN_FULLMUTEX; + // Every use of the handle and its statements holds SQLiteConnection::mutex, so SQLite's own + // per-call connection mutex would only add a lock and unlock to each API call. + const int flags = (readOnly ? SQLITE_OPEN_READONLY : SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE) | SQLITE_OPEN_NOMUTEX; sqlite3* rawDatabase = nullptr; const int result = sqlite3_open_v2(physicalPath.string().c_str(), &rawDatabase, flags, nullptr); std::unique_ptr database(rawDatabase, sqlite3_close_v2); diff --git a/packages/react-native-nitro-sqlite/cpp/NitroSQLiteDatabaseConnections.hpp b/packages/react-native-nitro-sqlite/cpp/NitroSQLiteDatabaseConnections.hpp index 614b376e..31b07c80 100644 --- a/packages/react-native-nitro-sqlite/cpp/NitroSQLiteDatabaseConnections.hpp +++ b/packages/react-native-nitro-sqlite/cpp/NitroSQLiteDatabaseConnections.hpp @@ -3,6 +3,7 @@ #ifdef SQLITE_ENABLE_SEE #define SQLITE_HAS_CODEC 1 #endif +#include "NitroSQLiteStatementCache.hpp" #include "sqlite/sqlite3.h" #include #include @@ -10,13 +11,34 @@ #include #include #include -#include #include +#include namespace margelo::nitro::rnnitrosqlite { +/** Runs operations in FIFO order on one dedicated thread. */ +class SerialWorker final { +public: + /** Start the thread. @p name identifies the worker in error logs. */ + explicit SerialWorker(std::string name); + /** Run the remaining operations, then stop the thread. */ + ~SerialWorker(); + + SerialWorker(const SerialWorker&) = delete; + SerialWorker& operator=(const SerialWorker&) = delete; + + /** Queue an operation. Exceptions it throws are logged and dropped. */ + void enqueue(std::function operation); + +private: + struct State; + + std::shared_ptr _state; + std::thread _thread; +}; + /** One native SQLite handle, its file identity, and its operation locks. */ -struct SQLiteConnection final : std::enable_shared_from_this { +struct SQLiteConnection final { SQLiteConnection(std::string name, std::filesystem::path physicalPath, bool readOnly, sqlite3* database); ~SQLiteConnection(); @@ -33,13 +55,14 @@ struct SQLiteConnection final : std::enable_shared_from_this { const bool readOnly; sqlite3* database; std::recursive_mutex mutex; + /** Reusable statements for repeated SQL. Guarded by mutex and cleared on close. */ + SQLiteStatementCache statementCache; private: - void drainAsync(); - - std::mutex asyncQueueMutex; - std::queue> asyncQueue; - bool asyncWorkerRunning = false; + std::mutex asyncWorkerMutex; + // Started by the first async operation. One dedicated thread, rather than a shared pool that + // spreads operations across its threads, lets the OS scheduler see the connection's sustained load. + std::unique_ptr asyncWorker; }; /** Shared ownership of a native connection across pending operations. */ diff --git a/packages/react-native-nitro-sqlite/cpp/NitroSQLiteOperations.cpp b/packages/react-native-nitro-sqlite/cpp/NitroSQLiteOperations.cpp index f012640d..8cc739b4 100644 --- a/packages/react-native-nitro-sqlite/cpp/NitroSQLiteOperations.cpp +++ b/packages/react-native-nitro-sqlite/cpp/NitroSQLiteOperations.cpp @@ -1,12 +1,9 @@ #include "NitroSQLiteOperations.hpp" #include "NitroSQLiteException.hpp" -#include "NitroSQLiteLogs.hpp" #include "NitroSQLiteStatementGroup.hpp" #include "NitroSQLiteUtils.hpp" -#include "hybridObjects/HybridNitroSQLiteQueryResult.hpp" #include "sqlite/sqlite3.h" #include -#include #include #include #include @@ -30,37 +27,6 @@ namespace margelo::nitro::rnnitrosqlite { static constexpr double kInt64MinAsDouble = static_cast(std::numeric_limits::min()); static constexpr double kInt64UpperBoundAsDouble = -kInt64MinAsDouble; -void SQLiteConnection::enqueueAsync(std::function operation) { - std::lock_guard lock(asyncQueueMutex); - if (!asyncWorkerRunning) { - Promise::async([connection = shared_from_this()] { connection->drainAsync(); }); - asyncWorkerRunning = true; - } - asyncQueue.push(std::move(operation)); -} - -void SQLiteConnection::drainAsync() { - while (true) { - std::function operation; - { - std::lock_guard lock(asyncQueueMutex); - if (asyncQueue.empty()) { - asyncWorkerRunning = false; - return; - } - operation = std::move(asyncQueue.front()); - asyncQueue.pop(); - } - try { - operation(); - } catch (const std::exception& error) { - LOGE("Async operation on database %s failed while settling its promise: %s", name.c_str(), error.what()); - } catch (...) { - LOGE("Async operation on database %s failed while settling its promise", name.c_str()); - } - } -} - void sqliteOpenDb(DatabaseConnections& connections, const std::string& dbName, const std::string& docPath, bool readOnly, const std::optional& encryptionKey) { #ifdef NITRO_SQLITE_VEC @@ -184,6 +150,44 @@ namespace { return statement; } + // Borrows the connection's cached statement for a query, or prepares one, and returns it + // reset to the cache when the scope ends. Requires the connection's mutex. + class CachedStatement final { + public: + CachedStatement(SQLiteConnection& connection, const std::string& query) + : _connection(connection), _query(query), _cacheable(SQLiteStatementCache::isCacheable(query)) { + if (_cacheable) { + _statement = connection.statementCache.take(query); + } + if (_statement == nullptr) { + _statement = prepareStatement(connection.database, query, std::nullopt).release(); + } + } + + ~CachedStatement() { + if (!_cacheable) { + sqlite3_finalize(_statement); + return; + } + sqlite3_reset(_statement); + sqlite3_clear_bindings(_statement); + _connection.statementCache.put(_query, _statement); + } + + CachedStatement(const CachedStatement&) = delete; + CachedStatement& operator=(const CachedStatement&) = delete; + + sqlite3_stmt* get() const { + return _statement; + } + + private: + SQLiteConnection& _connection; + const std::string& _query; + const bool _cacheable; + sqlite3_stmt* _statement = nullptr; + }; + template void consumeStatement(sqlite3* db, sqlite3_stmt* statement, OnRow&& onRow) { while (true) { @@ -202,7 +206,11 @@ namespace { } } - std::shared_ptr executeStatement(sqlite3* db, sqlite3_stmt* statement) { + // Rows per batch handed to a row batch handler. Each batch costs a hand-off to the JS thread, + // and a smaller final batch shortens the conversion left after the statement completes. + constexpr size_t kRowBatchSize = 256; + + NitroSQLiteQueryResult executeStatement(sqlite3* db, sqlite3_stmt* statement, const SQLiteRowBatchHandler& onRows = nullptr) { int columnCount = 0; std::vector columnNames; std::vector rows; @@ -266,6 +274,10 @@ namespace { } rows.push_back(std::move(row)); + if (onRows && rows.size() == kRowBatchSize) { + onRows(SQLiteQueryResults(columnNames, std::move(rows))); + rows = {}; + } }); if (!columnsCaptured) { @@ -285,24 +297,29 @@ namespace { metadata->insert({columnName, std::move(columnMeta)}); } - int rowsAffected = sqlite3_changes(db); - long long latestInsertRowId = sqlite3_last_insert_rowid(db); - return std::make_shared(SQLiteQueryResults(std::move(columnNames), std::move(rows)), - static_cast(latestInsertRowId), rowsAffected, std::move(metadata)); + NitroSQLiteQueryResult result; + result.rowsAffected = sqlite3_changes(db); + result.insertId = static_cast(sqlite3_last_insert_rowid(db)); + result.results = SQLiteQueryResults(std::move(columnNames), std::move(rows)); + result.metadata = std::move(metadata); + return result; } } // namespace -std::shared_ptr sqliteExecute(const SQLiteConnectionPtr& connection, const std::string& query, - const std::optional& params) { +NitroSQLiteQueryResult sqliteExecute(const SQLiteConnectionPtr& connection, const std::string& query, + const std::optional& params, const SQLiteRowBatchHandler& onRows) { std::lock_guard lock(connection->mutex); sqlite3* db = connection->database; if (db == nullptr) { throw NitroSQLiteException::DatabaseNotOpen(connection->name); } - auto statement = prepareStatement(db, query, params); - return executeStatement(db, statement.get()); + CachedStatement statement(*connection, query); + if (params) { + bindStatement(statement.get(), *params); + } + return executeStatement(db, statement.get(), onRows); } SQLiteOperationResult sqliteExecuteCommand(const SQLiteConnectionPtr& connection, const std::string& query, @@ -355,7 +372,7 @@ SQLitePreparedStatement::~SQLitePreparedStatement() { finalize(); } -std::shared_ptr SQLitePreparedStatement::execute(const std::optional& params) { +NitroSQLiteQueryResult SQLitePreparedStatement::execute(const std::optional& params) { std::lock_guard lock(_state->mutex); std::lock_guard connectionLock(_state->connection->mutex); diff --git a/packages/react-native-nitro-sqlite/cpp/NitroSQLiteOperations.hpp b/packages/react-native-nitro-sqlite/cpp/NitroSQLiteOperations.hpp index 279324fd..1bbd578a 100644 --- a/packages/react-native-nitro-sqlite/cpp/NitroSQLiteOperations.hpp +++ b/packages/react-native-nitro-sqlite/cpp/NitroSQLiteOperations.hpp @@ -1,7 +1,7 @@ #pragma once #include "NitroSQLiteDatabaseConnections.hpp" -#include "hybridObjects/HybridNitroSQLiteQueryResult.hpp" +#include "NitroSQLiteQueryResult.hpp" #include "sqlite/sqlite3.h" #include #include @@ -25,7 +25,7 @@ class SQLitePreparedStatement { ~SQLitePreparedStatement(); /** Reset and execute with new bindings. Throws after finalization or connection closure. */ - std::shared_ptr execute(const std::optional& params); + NitroSQLiteQueryResult execute(const std::optional& params); /** Release the native statement. Repeated calls are safe. */ void finalize(); /** Check whether the native statement has been released. */ @@ -53,8 +53,11 @@ void sqliteAttachDb(const SQLiteConnectionPtr& connection, const std::string& do void sqliteDetachDb(const SQLiteConnectionPtr& connection, const std::string& alias); -std::shared_ptr sqliteExecute(const SQLiteConnectionPtr& connection, const std::string& query, - const std::optional& params); +/** Execute one statement. With @p onRows, full batches of rows go to it as they are read and the + * returned result holds only the rows after the last batch. + */ +NitroSQLiteQueryResult sqliteExecute(const SQLiteConnectionPtr& connection, const std::string& query, + const std::optional& params, const SQLiteRowBatchHandler& onRows = nullptr); SQLiteOperationResult sqliteExecuteCommand(const SQLiteConnectionPtr& connection, const std::string& query, const std::optional& params = std::nullopt); diff --git a/packages/react-native-nitro-sqlite/cpp/NitroSQLiteQueryResults.hpp b/packages/react-native-nitro-sqlite/cpp/NitroSQLiteQueryResults.hpp index 621801c8..e12712c9 100644 --- a/packages/react-native-nitro-sqlite/cpp/NitroSQLiteQueryResults.hpp +++ b/packages/react-native-nitro-sqlite/cpp/NitroSQLiteQueryResults.hpp @@ -2,6 +2,7 @@ #include "NitroSQLiteTypes.hpp" #include +#include #include #include #include @@ -17,8 +18,8 @@ struct SQLiteQueryResultData { std::vector rows; }; -// The getter returns this small value. Its shared storage remains owned by the -// HybridObject, so repeated reads produce independent JavaScript arrays. +// Rows read by native code and converted to JavaScript objects on the JS thread. +// The shared storage keeps copies of a query result cheap. struct SQLiteQueryResults { SQLiteQueryResults() : data(std::make_shared()) {} SQLiteQueryResults(std::vector columnNames, std::vector rows) @@ -27,6 +28,9 @@ struct SQLiteQueryResults { std::shared_ptr data; }; +/** Receives leading result rows in batches while the statement still runs. */ +using SQLiteRowBatchHandler = std::function; + } // namespace margelo::nitro::rnnitrosqlite namespace margelo::nitro { @@ -54,7 +58,7 @@ struct JSIConverter final { return array; } - // `results` is a native, read-only getter. Nitro never passes it from JS. + // Query results only travel from native code to JavaScript. static rnnitrosqlite::SQLiteQueryResults fromJSI(jsi::Runtime&, const jsi::Value&) { throw std::logic_error("SQLite query results cannot be passed from JavaScript"); } diff --git a/packages/react-native-nitro-sqlite/cpp/NitroSQLiteStatementCache.hpp b/packages/react-native-nitro-sqlite/cpp/NitroSQLiteStatementCache.hpp new file mode 100644 index 00000000..45cdad6b --- /dev/null +++ b/packages/react-native-nitro-sqlite/cpp/NitroSQLiteStatementCache.hpp @@ -0,0 +1,120 @@ +#pragma once + +#include "sqlite/sqlite3.h" +#include +#include +#include +#include +#include +#include + +namespace margelo::nitro::rnnitrosqlite { + +/** Least recently used prepared statements of one connection, keyed by SQL text. + * Callers hold the connection's mutex. Cached statements are always reset with cleared bindings. + */ +class SQLiteStatementCache final { +public: + static constexpr size_t kCapacity = 32; + + SQLiteStatementCache() = default; + ~SQLiteStatementCache() { + clear(); + } + + SQLiteStatementCache(const SQLiteStatementCache&) = delete; + SQLiteStatementCache& operator=(const SQLiteStatementCache&) = delete; + + /** Whether reusing a statement for @p sql is equivalent to preparing it again. + * Only queries and data changes qualify. Statements such as PRAGMA, ATTACH, or BEGIN can apply + * their effect while SQLite prepares them, so they are always prepared anew. + */ + static bool isCacheable(std::string_view sql) { + const std::string_view keyword = firstKeyword(sql); + for (const std::string_view cacheable : {"SELECT", "INSERT", "UPDATE", "DELETE", "REPLACE", "WITH", "VALUES"}) { + if (equalsIgnoringCase(keyword, cacheable)) { + return true; + } + } + return false; + } + + /** Remove and return the statement cached for @p sql, or nullptr if there is none. */ + sqlite3_stmt* take(const std::string& sql) { + const auto found = _index.find(sql); + if (found == _index.end()) { + return nullptr; + } + sqlite3_stmt* statement = found->second->second; + _entries.erase(found->second); + _index.erase(found); + return statement; + } + + /** Cache a reset statement for @p sql, finalizing the least recently used one when full. */ + void put(const std::string& sql, sqlite3_stmt* statement) { + if (const auto found = _index.find(sql); found != _index.end()) { + sqlite3_finalize(statement); + return; + } + if (_entries.size() == kCapacity) { + sqlite3_finalize(_entries.back().second); + _index.erase(_entries.back().first); + _entries.pop_back(); + } + _entries.emplace_front(sql, statement); + _index.emplace(_entries.front().first, _entries.begin()); + } + + /** Finalize every cached statement. Call before closing the connection. */ + void clear() noexcept { + for (const auto& entry : _entries) { + sqlite3_finalize(entry.second); + } + _index.clear(); + _entries.clear(); + } + +private: + using Entries = std::list>; + + // Skips whitespace and SQL comments, then returns the leading run of letters. + static std::string_view firstKeyword(std::string_view sql) { + size_t position = 0; + while (position < sql.size()) { + if (std::isspace(static_cast(sql[position]))) { + position++; + } else if (sql.substr(position, 2) == "--") { + const size_t lineEnd = sql.find('\n', position); + position = lineEnd == std::string_view::npos ? sql.size() : lineEnd + 1; + } else if (sql.substr(position, 2) == "/*") { + const size_t commentEnd = sql.find("*/", position + 2); + position = commentEnd == std::string_view::npos ? sql.size() : commentEnd + 2; + } else { + break; + } + } + size_t end = position; + while (end < sql.size() && std::isalpha(static_cast(sql[end]))) { + end++; + } + return sql.substr(position, end - position); + } + + static bool equalsIgnoringCase(std::string_view value, std::string_view upperCase) { + if (value.size() != upperCase.size()) { + return false; + } + for (size_t i = 0; i < value.size(); i++) { + if (std::toupper(static_cast(value[i])) != upperCase[i]) { + return false; + } + } + return true; + } + + Entries _entries; + std::unordered_map _index; +}; + +} // namespace margelo::nitro::rnnitrosqlite diff --git a/packages/react-native-nitro-sqlite/cpp/hybridObjects/HybridNitroSQLite.cpp b/packages/react-native-nitro-sqlite/cpp/hybridObjects/HybridNitroSQLite.cpp index e8ba0048..df6f21d9 100644 --- a/packages/react-native-nitro-sqlite/cpp/hybridObjects/HybridNitroSQLite.cpp +++ b/packages/react-native-nitro-sqlite/cpp/hybridObjects/HybridNitroSQLite.cpp @@ -7,7 +7,6 @@ #include "../NitroSQLiteMacros.hpp" #include "../NitroSQLiteOperations.hpp" #include "HybridNitroSQLitePreparedStatement.hpp" -#include "HybridNitroSQLiteQueryResult.hpp" #include #include #include @@ -208,25 +207,25 @@ void HybridNitroSQLite::detach(const std::string& mainDbName, const std::string& sqliteDetachDb(_connections.get(mainDbName), alias); }; -std::shared_ptr HybridNitroSQLite::execute(const std::string& dbName, const std::string& query, - const std::optional& params) { +NitroSQLiteQueryResult HybridNitroSQLite::execute(const std::string& dbName, const std::string& query, + const std::optional& params) { return sqliteExecute(_connections.get(dbName), query, params); }; -std::shared_ptr>> -HybridNitroSQLite::executeAsync(const std::string& dbName, const std::string& query, const std::optional& params) { +std::shared_ptr> HybridNitroSQLite::executeAsync(const std::string& dbName, const std::string& query, + const std::optional& params, + const std::optional& onRows) { const auto copiedParams = copyArrayBufferParamsForBackground(params); SQLiteConnectionPtr connection; try { connection = _connections.get(dbName); } catch (...) { - return Promise>::rejected(std::current_exception()); + return Promise::rejected(std::current_exception()); } - return enqueueConnectionOperation>( - connection, [connection, query, copiedParams]() -> std::shared_ptr { - auto result = sqliteExecute(connection, query, copiedParams); - return result; + return enqueueConnectionOperation( + connection, [connection, query, copiedParams, onRows = onRows.value_or(nullptr)]() -> NitroSQLiteQueryResult { + return sqliteExecute(connection, query, copiedParams, onRows); }); }; diff --git a/packages/react-native-nitro-sqlite/cpp/hybridObjects/HybridNitroSQLite.hpp b/packages/react-native-nitro-sqlite/cpp/hybridObjects/HybridNitroSQLite.hpp index 03fd9c01..aca8e47d 100644 --- a/packages/react-native-nitro-sqlite/cpp/hybridObjects/HybridNitroSQLite.hpp +++ b/packages/react-native-nitro-sqlite/cpp/hybridObjects/HybridNitroSQLite.hpp @@ -3,7 +3,6 @@ #include "../NitroSQLiteDatabaseConnections.hpp" #include "../NitroSQLiteTypes.hpp" #include "HybridNitroSQLitePreparedStatementSpec.hpp" -#include "HybridNitroSQLiteQueryResultSpec.hpp" #include "HybridNitroSQLiteSpec.hpp" namespace margelo::nitro::rnnitrosqlite { @@ -64,14 +63,15 @@ class HybridNitroSQLite : public HybridNitroSQLiteSpec { * @param params Optional values bound to the placeholders. * @return Native rows, affected row count, insert ID, and metadata. */ - std::shared_ptr execute(const std::string& dbName, const std::string& query, - const std::optional& params) override; + NitroSQLiteQueryResult execute(const std::string& dbName, const std::string& query, + const std::optional& params) override; /** Execute one SQL statement on a background thread. * @return A promise of the native query result. */ - std::shared_ptr>> - executeAsync(const std::string& dbName, const std::string& query, const std::optional& params) override; + std::shared_ptr> executeAsync(const std::string& dbName, const std::string& query, + const std::optional& params, + const std::optional& onRows) override; /** Prepare one SQL statement for repeated execution on an open connection. */ std::shared_ptr prepare(const std::string& dbName, const std::string& query) override; diff --git a/packages/react-native-nitro-sqlite/cpp/hybridObjects/HybridNitroSQLitePreparedStatement.cpp b/packages/react-native-nitro-sqlite/cpp/hybridObjects/HybridNitroSQLitePreparedStatement.cpp index a9c47038..11caae01 100644 --- a/packages/react-native-nitro-sqlite/cpp/hybridObjects/HybridNitroSQLitePreparedStatement.cpp +++ b/packages/react-native-nitro-sqlite/cpp/hybridObjects/HybridNitroSQLitePreparedStatement.cpp @@ -1,5 +1,4 @@ #include "HybridNitroSQLitePreparedStatement.hpp" -#include "HybridNitroSQLiteQueryResult.hpp" #include namespace margelo::nitro::rnnitrosqlite { @@ -30,18 +29,17 @@ namespace { HybridNitroSQLitePreparedStatement::HybridNitroSQLitePreparedStatement(std::shared_ptr statement) : HybridObject(TAG), _statement(std::move(statement)) {} -std::shared_ptr -HybridNitroSQLitePreparedStatement::execute(const std::optional& params) { +NitroSQLiteQueryResult HybridNitroSQLitePreparedStatement::execute(const std::optional& params) { return _statement->execute(params); } -std::shared_ptr>> +std::shared_ptr> HybridNitroSQLitePreparedStatement::executeAsync(const std::optional& params) { const auto copiedParams = copyArrayBufferParamsForBackground(params); const auto statement = _statement; - return Promise>::async( - [statement, copiedParams]() -> std::shared_ptr { return statement->execute(copiedParams); }); + return Promise::async( + [statement, copiedParams]() -> NitroSQLiteQueryResult { return statement->execute(copiedParams); }); } void HybridNitroSQLitePreparedStatement::finalize() { diff --git a/packages/react-native-nitro-sqlite/cpp/hybridObjects/HybridNitroSQLitePreparedStatement.hpp b/packages/react-native-nitro-sqlite/cpp/hybridObjects/HybridNitroSQLitePreparedStatement.hpp index 2fd1181f..966acfab 100644 --- a/packages/react-native-nitro-sqlite/cpp/hybridObjects/HybridNitroSQLitePreparedStatement.hpp +++ b/packages/react-native-nitro-sqlite/cpp/hybridObjects/HybridNitroSQLitePreparedStatement.hpp @@ -14,10 +14,9 @@ class HybridNitroSQLitePreparedStatement : public HybridNitroSQLitePreparedState explicit HybridNitroSQLitePreparedStatement(std::shared_ptr statement); /** Execute with new bindings on the calling thread. */ - std::shared_ptr execute(const std::optional& params) override; + NitroSQLiteQueryResult execute(const std::optional& params) override; /** Execute with new bindings on a background thread. */ - std::shared_ptr>> - executeAsync(const std::optional& params) override; + std::shared_ptr> executeAsync(const std::optional& params) override; /** Release the native statement. */ void finalize() override; /** Check whether the native statement has been released. */ diff --git a/packages/react-native-nitro-sqlite/cpp/hybridObjects/HybridNitroSQLiteQueryResult.cpp b/packages/react-native-nitro-sqlite/cpp/hybridObjects/HybridNitroSQLiteQueryResult.cpp deleted file mode 100644 index 72036d90..00000000 --- a/packages/react-native-nitro-sqlite/cpp/hybridObjects/HybridNitroSQLiteQueryResult.cpp +++ /dev/null @@ -1,80 +0,0 @@ -#include "HybridNitroSQLiteQueryResult.hpp" - -namespace margelo::nitro::rnnitrosqlite { - -namespace { - - constexpr size_t nodePadding = 24; - - size_t getResultsExternalMemorySize(const SQLiteQueryResults& results) { - const auto& data = *results.data; - size_t size = sizeof(SQLiteQueryResultData) + sizeof(void*) * 2; - size += data.columnNames.capacity() * sizeof(std::string); - for (const auto& name : data.columnNames) { - size += name.capacity(); - } - size += data.rows.capacity() * sizeof(SQLiteQueryResultRow); - for (const auto& row : data.rows) { - size += row.capacity() * sizeof(SQLiteValue); - for (const auto& value : row) { - if (!value) { - continue; - } - if (const auto* text = std::get_if(&*value)) { - size += text->capacity(); - } - } - } - - return size; - } - - /** - * Compute the approximate external memory size of the table metadata. - * We include: - * - Column name string capacities (map keys), - * - Metadata contents, especially the `name` string on each metadata entry. - */ - size_t getMetadataExternalMemorySize(const SQLiteQueryTableMetadata& metadata) { - size_t size = metadata.bucket_count() * sizeof(void*); - size += metadata.size() * (sizeof(SQLiteQueryTableMetadata::value_type) + nodePadding); - - for (const auto& [columnName, columnMeta] : metadata) { - size += columnName.capacity(); - size += columnMeta.name.capacity(); - } - - return size; - } - -} // namespace - -std::optional HybridNitroSQLiteQueryResult::getInsertId() { - return _insertId; -} - -double HybridNitroSQLiteQueryResult::getRowsAffected() { - return _rowsAffected; -} - -SQLiteQueryResults HybridNitroSQLiteQueryResult::getResults() { - return _results; -}; - -std::optional HybridNitroSQLiteQueryResult::getMetadata() { - return _metadata; -} - -size_t HybridNitroSQLiteQueryResult::getExternalMemorySize() noexcept { - size_t size = sizeof(*this); - - size += getResultsExternalMemorySize(_results); - - if (_metadata) { - size += getMetadataExternalMemorySize(*_metadata); - } - - return size; -} - -} // namespace margelo::nitro::rnnitrosqlite diff --git a/packages/react-native-nitro-sqlite/cpp/hybridObjects/HybridNitroSQLiteQueryResult.hpp b/packages/react-native-nitro-sqlite/cpp/hybridObjects/HybridNitroSQLiteQueryResult.hpp deleted file mode 100644 index 1bf003d8..00000000 --- a/packages/react-native-nitro-sqlite/cpp/hybridObjects/HybridNitroSQLiteQueryResult.hpp +++ /dev/null @@ -1,50 +0,0 @@ -#pragma once - -#include "../NitroSQLiteQueryResults.hpp" -#include "HybridNitroSQLiteQueryResultSpec.hpp" -#include - -namespace margelo::nitro::rnnitrosqlite { - -/** Native result object returned by a single SQL execution. */ -class HybridNitroSQLiteQueryResult : public HybridNitroSQLiteQueryResultSpec { -public: - HybridNitroSQLiteQueryResult() : HybridObject(TAG) {} - HybridNitroSQLiteQueryResult(SQLiteQueryResults results, std::optional insertId, double rowsAffected, - std::optional metadata) - : HybridObject(TAG), _insertId(insertId), _rowsAffected(rowsAffected), _results(std::move(results)), _metadata(std::move(metadata)) {} - -private: - std::optional _insertId; - double _rowsAffected; - SQLiteQueryResults _results; - std::optional _metadata; - -public: - /** Return the connection's last insert row ID, which may be from an earlier statement. */ - std::optional getInsertId() override; - /** Return SQLite's latest row change count, which may be stale for a read-only query. */ - double getRowsAffected() override; - /** Return result rows keyed by column name. */ - SQLiteQueryResults getResults() override; - /** Return column metadata when available. */ - std::optional getMetadata() override; - - /** - * Approximate the native memory used by this query result. - * - * We account for: - * - The size of this C++ object (`sizeof(*this)`), - * - Positional rows and one result-level collection of column names, - * - String values stored in the result set, - * - BLOB shared pointers in the positional cells (Nitro accounts for the - * backing bytes when it exposes an ArrayBuffer to JavaScript), - * - Column metadata strings. - * - * This is a best-effort estimate and intentionally focuses on external - * heap allocations that can put pressure on the JS GC. - */ - size_t getExternalMemorySize() noexcept override; -}; - -} // namespace margelo::nitro::rnnitrosqlite diff --git a/packages/react-native-nitro-sqlite/nitrogen/generated/android/RNNitroSQLite+autolinking.cmake b/packages/react-native-nitro-sqlite/nitrogen/generated/android/RNNitroSQLite+autolinking.cmake index cad7718c..3b5d4635 100644 --- a/packages/react-native-nitro-sqlite/nitrogen/generated/android/RNNitroSQLite+autolinking.cmake +++ b/packages/react-native-nitro-sqlite/nitrogen/generated/android/RNNitroSQLite+autolinking.cmake @@ -36,7 +36,6 @@ target_sources( ../nitrogen/generated/shared/c++/HybridNitroSQLiteSpec.cpp ../nitrogen/generated/shared/c++/HybridNitroSQLiteOnLoadSpec.cpp ../nitrogen/generated/shared/c++/HybridNitroSQLitePreparedStatementSpec.cpp - ../nitrogen/generated/shared/c++/HybridNitroSQLiteQueryResultSpec.cpp # Android-specific Nitrogen C++ sources ../nitrogen/generated/android/c++/JHybridNitroSQLiteOnLoadSpec.cpp ) diff --git a/packages/react-native-nitro-sqlite/nitrogen/generated/shared/c++/HybridNitroSQLitePreparedStatementSpec.hpp b/packages/react-native-nitro-sqlite/nitrogen/generated/shared/c++/HybridNitroSQLitePreparedStatementSpec.hpp index 42a64e96..5ede22ad 100644 --- a/packages/react-native-nitro-sqlite/nitrogen/generated/shared/c++/HybridNitroSQLitePreparedStatementSpec.hpp +++ b/packages/react-native-nitro-sqlite/nitrogen/generated/shared/c++/HybridNitroSQLitePreparedStatementSpec.hpp @@ -13,11 +13,10 @@ #error NitroModules cannot be found! Are you sure you installed NitroModules properly? #endif -// Forward declaration of `HybridNitroSQLiteQueryResultSpec` to properly resolve imports. -namespace margelo::nitro::rnnitrosqlite { class HybridNitroSQLiteQueryResultSpec; } +// Forward declaration of `NitroSQLiteQueryResult` to properly resolve imports. +namespace margelo::nitro::rnnitrosqlite { struct NitroSQLiteQueryResult; } -#include -#include "HybridNitroSQLiteQueryResultSpec.hpp" +#include "NitroSQLiteQueryResult.hpp" #include #include #include @@ -57,8 +56,8 @@ namespace margelo::nitro::rnnitrosqlite { public: // Methods - virtual std::shared_ptr execute(const std::optional, std::string, double>>>>& params) = 0; - virtual std::shared_ptr>> executeAsync(const std::optional, std::string, double>>>>& params) = 0; + virtual NitroSQLiteQueryResult execute(const std::optional, std::string, double>>>>& params) = 0; + virtual std::shared_ptr> executeAsync(const std::optional, std::string, double>>>>& params) = 0; virtual void finalize() = 0; protected: diff --git a/packages/react-native-nitro-sqlite/nitrogen/generated/shared/c++/HybridNitroSQLiteQueryResultSpec.cpp b/packages/react-native-nitro-sqlite/nitrogen/generated/shared/c++/HybridNitroSQLiteQueryResultSpec.cpp deleted file mode 100644 index 72d05f44..00000000 --- a/packages/react-native-nitro-sqlite/nitrogen/generated/shared/c++/HybridNitroSQLiteQueryResultSpec.cpp +++ /dev/null @@ -1,24 +0,0 @@ -/// -/// HybridNitroSQLiteQueryResultSpec.cpp -/// This file was generated by nitrogen. DO NOT MODIFY THIS FILE. -/// https://github.com/mrousavy/nitro -/// Copyright © Marc Rousavy @ Margelo -/// - -#include "HybridNitroSQLiteQueryResultSpec.hpp" - -namespace margelo::nitro::rnnitrosqlite { - - void HybridNitroSQLiteQueryResultSpec::loadHybridMethods() { - // load base methods/properties - HybridObject::loadHybridMethods(); - // load custom methods/properties - registerHybrids(this, [](Prototype& prototype) { - prototype.registerHybridGetter("rowsAffected", &HybridNitroSQLiteQueryResultSpec::getRowsAffected); - prototype.registerHybridGetter("insertId", &HybridNitroSQLiteQueryResultSpec::getInsertId); - prototype.registerHybridGetter("results", &HybridNitroSQLiteQueryResultSpec::getResults); - prototype.registerHybridGetter("metadata", &HybridNitroSQLiteQueryResultSpec::getMetadata); - }); - } - -} // namespace margelo::nitro::rnnitrosqlite diff --git a/packages/react-native-nitro-sqlite/nitrogen/generated/shared/c++/HybridNitroSQLiteQueryResultSpec.hpp b/packages/react-native-nitro-sqlite/nitrogen/generated/shared/c++/HybridNitroSQLiteQueryResultSpec.hpp deleted file mode 100644 index f484e749..00000000 --- a/packages/react-native-nitro-sqlite/nitrogen/generated/shared/c++/HybridNitroSQLiteQueryResultSpec.hpp +++ /dev/null @@ -1,70 +0,0 @@ -/// -/// HybridNitroSQLiteQueryResultSpec.hpp -/// This file was generated by nitrogen. DO NOT MODIFY THIS FILE. -/// https://github.com/mrousavy/nitro -/// Copyright © Marc Rousavy @ Margelo -/// - -#pragma once - -#if __has_include() -#include -#else -#error NitroModules cannot be found! Are you sure you installed NitroModules properly? -#endif - -// Forward declaration of `NitroSQLiteQueryColumnMetadata` to properly resolve imports. -namespace margelo::nitro::rnnitrosqlite { struct NitroSQLiteQueryColumnMetadata; } - -#include -#include "NitroSQLiteQueryResults.hpp" -#include -#include "NitroSQLiteQueryColumnMetadata.hpp" -#include - -namespace margelo::nitro::rnnitrosqlite { - - using namespace margelo::nitro; - - /** - * An abstract base class for `NitroSQLiteQueryResult` - * Inherit this class to create instances of `HybridNitroSQLiteQueryResultSpec` in C++. - * You must explicitly call `HybridObject`'s constructor yourself, because it is virtual. - * @example - * ```cpp - * class HybridNitroSQLiteQueryResult: public HybridNitroSQLiteQueryResultSpec { - * public: - * HybridNitroSQLiteQueryResult(...): HybridObject(TAG) { ... } - * // ... - * }; - * ``` - */ - class HybridNitroSQLiteQueryResultSpec: public virtual HybridObject { - public: - // Constructor - explicit HybridNitroSQLiteQueryResultSpec(): HybridObject(TAG) { } - - // Destructor - ~HybridNitroSQLiteQueryResultSpec() override = default; - - public: - // Properties - virtual double getRowsAffected() = 0; - virtual std::optional getInsertId() = 0; - virtual SQLiteQueryResults getResults() = 0; - virtual std::optional> getMetadata() = 0; - - public: - // Methods - - - protected: - // Hybrid Setup - void loadHybridMethods() override; - - protected: - // Tag for logging - static constexpr auto TAG = "NitroSQLiteQueryResult"; - }; - -} // namespace margelo::nitro::rnnitrosqlite diff --git a/packages/react-native-nitro-sqlite/nitrogen/generated/shared/c++/HybridNitroSQLiteSpec.hpp b/packages/react-native-nitro-sqlite/nitrogen/generated/shared/c++/HybridNitroSQLiteSpec.hpp index bd2f3bfa..87856cee 100644 --- a/packages/react-native-nitro-sqlite/nitrogen/generated/shared/c++/HybridNitroSQLiteSpec.hpp +++ b/packages/react-native-nitro-sqlite/nitrogen/generated/shared/c++/HybridNitroSQLiteSpec.hpp @@ -13,8 +13,8 @@ #error NitroModules cannot be found! Are you sure you installed NitroModules properly? #endif -// Forward declaration of `HybridNitroSQLiteQueryResultSpec` to properly resolve imports. -namespace margelo::nitro::rnnitrosqlite { class HybridNitroSQLiteQueryResultSpec; } +// Forward declaration of `NitroSQLiteQueryResult` to properly resolve imports. +namespace margelo::nitro::rnnitrosqlite { struct NitroSQLiteQueryResult; } // Forward declaration of `HybridNitroSQLitePreparedStatementSpec` to properly resolve imports. namespace margelo::nitro::rnnitrosqlite { class HybridNitroSQLitePreparedStatementSpec; } // Forward declaration of `BatchQueryResult` to properly resolve imports. @@ -26,13 +26,15 @@ namespace margelo::nitro::rnnitrosqlite { struct FileLoadResult; } #include #include -#include -#include "HybridNitroSQLiteQueryResultSpec.hpp" +#include "NitroSQLiteQueryResult.hpp" #include #include #include #include #include +#include "NitroSQLiteQueryResults.hpp" +#include +#include #include "HybridNitroSQLitePreparedStatementSpec.hpp" #include "BatchQueryResult.hpp" #include "BatchQueryCommand.hpp" @@ -76,8 +78,8 @@ namespace margelo::nitro::rnnitrosqlite { virtual void drop(const std::string& dbName, const std::optional& location, const std::optional& connectionId) = 0; virtual void attach(const std::string& mainDbName, const std::string& dbNameToAttach, const std::string& alias, const std::optional& location) = 0; virtual void detach(const std::string& mainDbName, const std::string& alias) = 0; - virtual std::shared_ptr execute(const std::string& dbName, const std::string& query, const std::optional, std::string, double>>>>& params) = 0; - virtual std::shared_ptr>> executeAsync(const std::string& dbName, const std::string& query, const std::optional, std::string, double>>>>& params) = 0; + virtual NitroSQLiteQueryResult execute(const std::string& dbName, const std::string& query, const std::optional, std::string, double>>>>& params) = 0; + virtual std::shared_ptr> executeAsync(const std::string& dbName, const std::string& query, const std::optional, std::string, double>>>>& params, const std::optional>& onRows) = 0; virtual std::shared_ptr prepare(const std::string& dbName, const std::string& query) = 0; virtual BatchQueryResult executeBatch(const std::string& dbName, const std::vector& commands) = 0; virtual std::shared_ptr> executeBatchAsync(const std::string& dbName, const std::vector& commands) = 0; diff --git a/packages/react-native-nitro-sqlite/nitrogen/generated/shared/c++/NitroSQLiteQueryResult.hpp b/packages/react-native-nitro-sqlite/nitrogen/generated/shared/c++/NitroSQLiteQueryResult.hpp new file mode 100644 index 00000000..693a4c1a --- /dev/null +++ b/packages/react-native-nitro-sqlite/nitrogen/generated/shared/c++/NitroSQLiteQueryResult.hpp @@ -0,0 +1,100 @@ +/// +/// NitroSQLiteQueryResult.hpp +/// This file was generated by nitrogen. DO NOT MODIFY THIS FILE. +/// https://github.com/mrousavy/nitro +/// Copyright © Marc Rousavy @ Margelo +/// + +#pragma once + +#if __has_include() +#include +#else +#error NitroModules cannot be found! Are you sure you installed NitroModules properly? +#endif +#if __has_include() +#include +#else +#error NitroModules cannot be found! Are you sure you installed NitroModules properly? +#endif +#if __has_include() +#include +#else +#error NitroModules cannot be found! Are you sure you installed NitroModules properly? +#endif +#if __has_include() +#include +#else +#error NitroModules cannot be found! Are you sure you installed NitroModules properly? +#endif + +// Forward declaration of `NitroSQLiteQueryColumnMetadata` to properly resolve imports. +namespace margelo::nitro::rnnitrosqlite { struct NitroSQLiteQueryColumnMetadata; } + +#include +#include "NitroSQLiteQueryResults.hpp" +#include +#include "NitroSQLiteQueryColumnMetadata.hpp" +#include + +namespace margelo::nitro::rnnitrosqlite { + + /** + * A struct which can be represented as a JavaScript object (NitroSQLiteQueryResult). + */ + struct NitroSQLiteQueryResult final { + public: + double rowsAffected SWIFT_PRIVATE; + std::optional insertId SWIFT_PRIVATE; + margelo::nitro::rnnitrosqlite::SQLiteQueryResults results SWIFT_PRIVATE; + std::optional> metadata SWIFT_PRIVATE; + + public: + NitroSQLiteQueryResult() = default; + explicit NitroSQLiteQueryResult(double rowsAffected, std::optional insertId, margelo::nitro::rnnitrosqlite::SQLiteQueryResults results, std::optional> metadata): rowsAffected(rowsAffected), insertId(insertId), results(results), metadata(metadata) {} + + public: + // NitroSQLiteQueryResult is not equatable because these properties are not equatable: results + }; + +} // namespace margelo::nitro::rnnitrosqlite + +namespace margelo::nitro { + + // C++ NitroSQLiteQueryResult <> JS NitroSQLiteQueryResult (object) + template <> + struct JSIConverter final { + static inline margelo::nitro::rnnitrosqlite::NitroSQLiteQueryResult fromJSI(jsi::Runtime& runtime, const jsi::Value& arg) { + jsi::Object obj = arg.asObject(runtime); + return margelo::nitro::rnnitrosqlite::NitroSQLiteQueryResult( + JSIConverter::fromJSI(runtime, obj.getProperty(runtime, PropNameIDCache::get(runtime, "rowsAffected"))), + JSIConverter>::fromJSI(runtime, obj.getProperty(runtime, PropNameIDCache::get(runtime, "insertId"))), + JSIConverter::fromJSI(runtime, obj.getProperty(runtime, PropNameIDCache::get(runtime, "results"))), + JSIConverter>>::fromJSI(runtime, obj.getProperty(runtime, PropNameIDCache::get(runtime, "metadata"))) + ); + } + static inline jsi::Value toJSI(jsi::Runtime& runtime, const margelo::nitro::rnnitrosqlite::NitroSQLiteQueryResult& arg) { + jsi::Object obj(runtime); + obj.setProperty(runtime, PropNameIDCache::get(runtime, "rowsAffected"), JSIConverter::toJSI(runtime, arg.rowsAffected)); + obj.setProperty(runtime, PropNameIDCache::get(runtime, "insertId"), JSIConverter>::toJSI(runtime, arg.insertId)); + obj.setProperty(runtime, PropNameIDCache::get(runtime, "results"), JSIConverter::toJSI(runtime, arg.results)); + obj.setProperty(runtime, PropNameIDCache::get(runtime, "metadata"), JSIConverter>>::toJSI(runtime, arg.metadata)); + return obj; + } + static inline bool canConvert(jsi::Runtime& runtime, const jsi::Value& value) { + if (!value.isObject()) { + return false; + } + jsi::Object obj = value.getObject(runtime); + if (!nitro::isPlainObject(runtime, obj)) { + return false; + } + if (!JSIConverter::canConvert(runtime, obj.getProperty(runtime, PropNameIDCache::get(runtime, "rowsAffected")))) return false; + if (!JSIConverter>::canConvert(runtime, obj.getProperty(runtime, PropNameIDCache::get(runtime, "insertId")))) return false; + if (!JSIConverter::canConvert(runtime, obj.getProperty(runtime, PropNameIDCache::get(runtime, "results")))) return false; + if (!JSIConverter>>::canConvert(runtime, obj.getProperty(runtime, PropNameIDCache::get(runtime, "metadata")))) return false; + return true; + } + }; + +} // namespace margelo::nitro diff --git a/packages/react-native-nitro-sqlite/src/__tests__/execute.test.ts b/packages/react-native-nitro-sqlite/src/__tests__/execute.test.ts index 6830858c..e56dd39b 100644 --- a/packages/react-native-nitro-sqlite/src/__tests__/execute.test.ts +++ b/packages/react-native-nitro-sqlite/src/__tests__/execute.test.ts @@ -101,10 +101,41 @@ describe('execute', () => { dbName, query, params, + expect.any(Function), ) expect(HybridNitroSQLite.executeAsync).toHaveBeenCalledTimes(3) }) + it('joins rows delivered in batches before the remaining result rows', async () => { + jest + .mocked(HybridNitroSQLite.executeAsync) + .mockImplementationOnce(async (_dbName, _query, _params, onRows) => { + onRows?.([{ value: 1 }, { value: 2 }]) + onRows?.([{ value: 3 }]) + return nativeResult([{ value: 4 }]) + }) + + const result = await executeAsync(dbName, query, params) + expect(result.results).toEqual([ + { value: 1 }, + { value: 2 }, + { value: 3 }, + { value: 4 }, + ]) + expect(result.rows._array).toBe(result.results) + expect(result.rows.length).toBe(4) + expect(result.rows.item(3)).toEqual({ value: 4 }) + }) + + it('keeps the native rows when no batch was delivered', async () => { + const native = nativeResult([{ value: 7 }]) + jest.mocked(HybridNitroSQLite.executeAsync).mockResolvedValueOnce(native) + + const result = await executeAsync(dbName, query, params) + expect(result.results).toBe(native.results) + expect(result.rows._array).toBe(native.results) + }) + it('converts native asynchronous errors and releases a managed queue', async () => { openDatabaseQueue(dbName) jest diff --git a/packages/react-native-nitro-sqlite/src/__tests__/session.test.ts b/packages/react-native-nitro-sqlite/src/__tests__/session.test.ts index b3c6409b..ab114c76 100644 --- a/packages/react-native-nitro-sqlite/src/__tests__/session.test.ts +++ b/packages/react-native-nitro-sqlite/src/__tests__/session.test.ts @@ -81,6 +81,7 @@ describe('open', () => { dbName, 'SELECT 2', undefined, + expect.any(Function), ) db.close() @@ -97,6 +98,7 @@ describe('open', () => { dbName, 'BEGIN TRANSACTION', undefined, + expect.any(Function), ) expect(HybridNitroSQLite.execute).toHaveBeenCalledWith( dbName, @@ -329,11 +331,13 @@ describe('open', () => { 'transaction-2', 'SELECT 3', undefined, + expect.any(Function), ) expect(HybridNitroSQLite.executeAsync).not.toHaveBeenCalledWith( 'transaction-1', 'SELECT 2', undefined, + expect.any(Function), ) gate.resolve() diff --git a/packages/react-native-nitro-sqlite/src/__tests__/transaction.test.ts b/packages/react-native-nitro-sqlite/src/__tests__/transaction.test.ts index df791e7c..52d1cc6b 100644 --- a/packages/react-native-nitro-sqlite/src/__tests__/transaction.test.ts +++ b/packages/react-native-nitro-sqlite/src/__tests__/transaction.test.ts @@ -39,6 +39,7 @@ describe('transaction', () => { dbName, 'BEGIN TRANSACTION', undefined, + expect.any(Function), ) expect(HybridNitroSQLite.execute).toHaveBeenNthCalledWith( 1, @@ -51,6 +52,7 @@ describe('transaction', () => { dbName, 'SELECT ?', [2], + expect.any(Function), ) expect(HybridNitroSQLite.execute).toHaveBeenLastCalledWith( dbName, @@ -159,6 +161,7 @@ describe('transaction', () => { dbName, 'BEGIN EXCLUSIVE TRANSACTION', undefined, + expect.any(Function), ) expect(HybridNitroSQLite.execute).toHaveBeenCalledTimes(1) expect(HybridNitroSQLite.execute).toHaveBeenCalledWith( diff --git a/packages/react-native-nitro-sqlite/src/index.ts b/packages/react-native-nitro-sqlite/src/index.ts index 1985c928..39b01799 100644 --- a/packages/react-native-nitro-sqlite/src/index.ts +++ b/packages/react-native-nitro-sqlite/src/index.ts @@ -33,6 +33,7 @@ export type { NitroSQLite as NitroSQLiteNative } from './specs/NitroSQLite.nitro export type { NitroSQLitePreparedStatement } from './specs/NitroSQLitePreparedStatement.nitro' export type { NitroSQLiteQueryResult, + NitroSQLiteQueryRows, NitroSQLiteQueryColumnMetadata, } from './specs/NitroSQLiteQueryResult.nitro' export type { TypeOrmNitroSQLiteConnection } from './typeORM' diff --git a/packages/react-native-nitro-sqlite/src/operations/execute.ts b/packages/react-native-nitro-sqlite/src/operations/execute.ts index 43724e61..47e0f86c 100644 --- a/packages/react-native-nitro-sqlite/src/operations/execute.ts +++ b/packages/react-native-nitro-sqlite/src/operations/execute.ts @@ -88,12 +88,21 @@ export async function executeAsyncNative( params?: SQLiteQueryParams, ): Promise> { try { + // Large results arrive in batches, so rows are converted while SQLite reads the rest. + let leadingRows: Row[] | undefined const nativeResult = await HybridNitroSQLite.executeAsync( dbName, query, params, + (rows) => { + if (leadingRows === undefined) { + leadingRows = rows as Row[] + } else { + Array.prototype.push.apply(leadingRows, rows as Row[]) + } + }, ) - return buildJSQueryResult(nativeResult) + return buildJSQueryResult(nativeResult, leadingRows) } catch (error) { throw NitroSQLiteError.fromError(error) } @@ -101,9 +110,14 @@ export async function executeAsyncNative( export function buildJSQueryResult( result: NitroSQLiteQueryResult, + leadingRows?: Row[], ): QueryResult { - const resultWithRows = result as QueryResult - const results = result.results as Row[] + const resultWithRows = result as QueryResult & { results: Row[] } + if (leadingRows !== undefined) { + Array.prototype.push.apply(leadingRows, result.results as Row[]) + resultWithRows.results = leadingRows + } + const results = resultWithRows.results resultWithRows.rows = { _array: results, diff --git a/packages/react-native-nitro-sqlite/src/specs/NitroSQLite.nitro.ts b/packages/react-native-nitro-sqlite/src/specs/NitroSQLite.nitro.ts index dd30bc97..38f4986d 100644 --- a/packages/react-native-nitro-sqlite/src/specs/NitroSQLite.nitro.ts +++ b/packages/react-native-nitro-sqlite/src/specs/NitroSQLite.nitro.ts @@ -5,7 +5,10 @@ import type { FileLoadResult, SQLiteQueryParams, } from '../types' -import type { NitroSQLiteQueryResult } from './NitroSQLiteQueryResult.nitro' +import type { + NitroSQLiteQueryResult, + NitroSQLiteQueryRows, +} from './NitroSQLiteQueryResult.nitro' import type { NitroSQLitePreparedStatement } from './NitroSQLitePreparedStatement.nitro' /** Native database operations exposed through `NitroSQLite.native`. @@ -94,12 +97,16 @@ export interface NitroSQLite * @param dbName Name of an open database. * @param query SQL statement with optional positional placeholders. * @param params Positional values bound to SQL placeholders. + * @param onRows Receives leading rows in batches, in order, while SQLite still reads the rest, so + * JavaScript can convert a batch while the next one is read. Batched rows are not repeated in the + * result's `results`, which then holds only the rows after the last batch. * @returns A promise of the native query result. */ executeAsync( dbName: string, query: string, params?: SQLiteQueryParams, + onRows?: (rows: NitroSQLiteQueryRows) => void, ): Promise /** Prepare one SQL statement on an open native connection for repeated execution. * Finalize the returned statement before closing its connection. diff --git a/packages/react-native-nitro-sqlite/src/specs/NitroSQLiteQueryResult.nitro.ts b/packages/react-native-nitro-sqlite/src/specs/NitroSQLiteQueryResult.nitro.ts index dcb8e0c8..efab64c1 100644 --- a/packages/react-native-nitro-sqlite/src/specs/NitroSQLiteQueryResult.nitro.ts +++ b/packages/react-native-nitro-sqlite/src/specs/NitroSQLiteQueryResult.nitro.ts @@ -1,23 +1,24 @@ -import type { CustomType, HybridObject } from 'react-native-nitro-modules' +import type { CustomType } from 'react-native-nitro-modules' import type { ColumnType, SQLiteValue } from '../types' -/** Native result of one SQL statement. The managed API also adds a `rows` adapter. */ -export interface NitroSQLiteQueryResult - extends HybridObject<{ - ios: 'c++' - android: 'c++' - }> { +/** Result rows keyed by column name. Native rows become JavaScript objects on the JavaScript thread. */ +export type NitroSQLiteQueryRows = CustomType< + Record[], + 'margelo::nitro::rnnitrosqlite::SQLiteQueryResults', + { include: 'NitroSQLiteQueryResults.hpp' } +> + +/** Native result of one SQL statement, delivered as a plain JavaScript object. + * The managed API also adds a `rows` adapter. + */ +export interface NitroSQLiteQueryResult { /** SQLite's latest row change count. For a read-only query it may reflect an earlier write. */ readonly rowsAffected: number /** Last insert row ID for this connection. It may refer to an earlier statement. */ readonly insertId?: number /** Rows keyed by result column names. */ - readonly results: CustomType< - Record[], - 'SQLiteQueryResults', - { include: 'NitroSQLiteQueryResults.hpp' } - > + readonly results: NitroSQLiteQueryRows /** Column metadata keyed by result column name, when available. */ readonly metadata?: Record diff --git a/packages/react-native-nitro-sqlite/tests/cpp/serialWorker.test.cpp b/packages/react-native-nitro-sqlite/tests/cpp/serialWorker.test.cpp new file mode 100644 index 00000000..1c86f156 --- /dev/null +++ b/packages/react-native-nitro-sqlite/tests/cpp/serialWorker.test.cpp @@ -0,0 +1,108 @@ +#include "NitroSQLiteDatabaseConnections.hpp" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +using margelo::nitro::rnnitrosqlite::SerialWorker; + +namespace { + +void expect(bool condition, const std::string& message) { + if (!condition) { + throw std::runtime_error(message); + } +} + +// Holds the worker from inside its own operation and signals once that reference is gone. +struct LastReference { + std::shared_ptr worker; + std::promise released; + + ~LastReference() { + worker.reset(); + released.set_value(); + } +}; + +} // namespace + +int main() { + try { + { + SerialWorker worker("fifo"); + std::mutex mutex; + std::vector order; + std::thread::id workerThread; + std::promise done; + for (int i = 0; i < 100; i++) { + worker.enqueue([&, i] { + std::lock_guard lock(mutex); + order.push_back(i); + workerThread = std::this_thread::get_id(); + }); + } + worker.enqueue([] { throw std::runtime_error("expected failure"); }); + worker.enqueue([&] { done.set_value(); }); + expect(done.get_future().wait_for(std::chrono::seconds(5)) == std::future_status::ready, + "operations after a throwing operation must still run"); + std::lock_guard lock(mutex); + expect(order.size() == 100, "every operation must run"); + for (int i = 0; i < 100; i++) { + expect(order[i] == i, "operations must run in submission order"); + } + expect(workerThread != std::this_thread::get_id(), "operations must run off the submitting thread"); + } + + { + // Destroying the worker must run the operations it still holds before the thread stops. + std::atomic completed = 0; + { + SerialWorker worker("drain"); + for (int i = 0; i < 50; i++) { + worker.enqueue([&] { + std::this_thread::sleep_for(std::chrono::microseconds(100)); + completed++; + }); + } + } + expect(completed == 50, "destruction must drain queued operations"); + } + + { + // The last reference to a worker may be dropped by one of its own operations. + std::mutex gateMutex; + std::condition_variable gate; + bool open = false; + auto worker = std::make_shared("self-release"); + auto reference = std::make_shared(); + reference->worker = worker; + auto releasedFuture = reference->released.get_future(); + worker->enqueue([reference = std::move(reference), &gateMutex, &gate, &open] { + std::unique_lock lock(gateMutex); + gate.wait(lock, [&] { return open; }); + }); + worker.reset(); + { + std::lock_guard lock(gateMutex); + open = true; + } + gate.notify_one(); + expect(releasedFuture.wait_for(std::chrono::seconds(5)) == std::future_status::ready, + "a worker released by its own operation must not deadlock"); + } + + std::cout << "[PASS] serial worker order, failures, draining and self-release\n"; + return 0; + } catch (const std::exception& error) { + std::cerr << "[FAIL] " << error.what() << '\n'; + return 1; + } +} diff --git a/packages/react-native-nitro-sqlite/tests/cpp/statementCache.test.cpp b/packages/react-native-nitro-sqlite/tests/cpp/statementCache.test.cpp new file mode 100644 index 00000000..84d508a7 --- /dev/null +++ b/packages/react-native-nitro-sqlite/tests/cpp/statementCache.test.cpp @@ -0,0 +1,104 @@ +#include "NitroSQLiteStatementCache.hpp" +#include +#include +#include + +using margelo::nitro::rnnitrosqlite::SQLiteStatementCache; + +namespace { + +void expect(bool condition, const std::string& message) { + if (!condition) { + throw std::runtime_error(message); + } +} + +sqlite3_stmt* prepare(sqlite3* db, const std::string& sql) { + sqlite3_stmt* statement = nullptr; + if (sqlite3_prepare_v2(db, sql.c_str(), -1, &statement, nullptr) != SQLITE_OK) { + throw std::runtime_error(sqlite3_errmsg(db)); + } + return statement; +} + +int liveStatements(sqlite3* db) { + int count = 0; + for (sqlite3_stmt* statement = sqlite3_next_stmt(db, nullptr); statement != nullptr; statement = sqlite3_next_stmt(db, statement)) { + count++; + } + return count; +} + +} // namespace + +int main() { + sqlite3* db = nullptr; + try { + for (const char* sql : {"SELECT 1", " select * FROM t", "insert into t VALUES (1)", "UPDATE t SET a = 1", "DELETE FROM t", + "REPLACE INTO t VALUES (1)", "WITH x AS (SELECT 1) SELECT * FROM x", "VALUES (1)", + "-- leading comment\nSELECT 1", "/* block */ SELECT 1", "\n\t/* a */ -- b\n insert INTO t VALUES (2)"}) { + expect(SQLiteStatementCache::isCacheable(sql), std::string("expected cacheable: ") + sql); + } + for (const char* sql : + {"PRAGMA cache_size = 100", "BEGIN", "COMMIT", "ROLLBACK", "ATTACH DATABASE 'a' AS a", "DETACH a", "CREATE TABLE t (a)", + "DROP TABLE t", "VACUUM", "SAVEPOINT s", "", " ", "-- only a comment", "/* unterminated", "SELECTED", "1"}) { + expect(!SQLiteStatementCache::isCacheable(sql), std::string("expected not cacheable: ") + sql); + } + + if (sqlite3_open(":memory:", &db) != SQLITE_OK) { + throw std::runtime_error("cannot open an in-memory database"); + } + { + SQLiteStatementCache cache; + expect(cache.take("SELECT 1") == nullptr, "an empty cache has no statements"); + + sqlite3_stmt* first = prepare(db, "SELECT 1"); + cache.put("SELECT 1", first); + expect(cache.take("SELECT 1") == first, "a cached statement is returned for the same SQL"); + expect(cache.take("SELECT 1") == nullptr, "a taken statement leaves the cache"); + + // A returned statement runs again with new bindings. + sqlite3_stmt* bound = prepare(db, "SELECT ?"); + sqlite3_bind_int(bound, 1, 7); + expect(sqlite3_step(bound) == SQLITE_ROW && sqlite3_column_int(bound, 0) == 7, "first execution"); + sqlite3_reset(bound); + sqlite3_clear_bindings(bound); + cache.put("SELECT ?", bound); + sqlite3_stmt* reused = cache.take("SELECT ?"); + sqlite3_bind_int(reused, 1, 9); + expect(sqlite3_step(reused) == SQLITE_ROW && sqlite3_column_int(reused, 0) == 9, "reused execution sees new bindings"); + sqlite3_reset(reused); + cache.put("SELECT ?", reused); + + // A second statement for SQL that is already cached is finalized instead of replacing it. + cache.put("SELECT 1", first); + cache.put("SELECT 1", prepare(db, "SELECT 1")); + expect(cache.take("SELECT 1") == first, "the cached statement is kept when a duplicate is returned"); + cache.put("SELECT 1", first); + + cache.clear(); + expect(liveStatements(db) == 0, "clear finalizes every cached statement"); + + // The least recently used statement is finalized once the cache is full. + for (size_t i = 0; i <= SQLiteStatementCache::kCapacity; i++) { + const std::string sql = "SELECT " + std::to_string(i); + cache.put(sql, prepare(db, sql)); + } + expect(liveStatements(db) == static_cast(SQLiteStatementCache::kCapacity), "the cache never exceeds its capacity"); + expect(cache.take("SELECT 0") == nullptr, "the least recently used statement is evicted"); + sqlite3_stmt* newest = cache.take("SELECT " + std::to_string(SQLiteStatementCache::kCapacity)); + expect(newest != nullptr, "the most recently used statement stays cached"); + sqlite3_finalize(newest); + } + expect(liveStatements(db) == 0, "destroying the cache finalizes its statements"); + sqlite3_close(db); + std::cout << "[PASS] statement cache classification, reuse, eviction and cleanup\n"; + return 0; + } catch (const std::exception& error) { + std::cerr << "[FAIL] " << error.what() << '\n'; + if (db != nullptr) { + sqlite3_close_v2(db); + } + return 1; + } +} diff --git a/scripts/android-sqlite-flags/build.gradle b/scripts/android-sqlite-flags/build.gradle index 21013d04..080f5f42 100644 --- a/scripts/android-sqlite-flags/build.gradle +++ b/scripts/android-sqlite-flags/build.gradle @@ -9,24 +9,28 @@ tasks.register('testSqliteFlags') { resolveNitroSqliteDefaultFlags(packageFile, customFlags) } + def ioFlags = ['-DHAVE_FDATASYNC=1', '-DSQLITE_ENABLE_BATCH_ATOMIC_WRITE=1'] def defaultFlags = flagsFor([:], '') - assert defaultFlags.size() == 11 + assert defaultFlags.size() == 13 assert defaultFlags.contains('-DSQLITE_THREADSAFE=1') assert defaultFlags.contains('-DSQLITE_DQS=0') assert defaultFlags.contains('-DSQLITE_DEFAULT_WAL_SYNCHRONOUS=1') assert defaultFlags.contains('-DSQLITE_ENABLE_RTREE=1') + assert defaultFlags.containsAll(ioFlags) assert resolveNitroSqliteDefaultFlags(new File(temporaryDir, 'missing.json'), '') == defaultFlags - assert flagsFor([nitroSQLite: [performanceMode: false]], '') == ['-DSQLITE_THREADSAFE=1', '-DSQLITE_ENABLE_RTREE=1'] + assert flagsFor([nitroSQLite: [performanceMode: false]], '') == ['-DSQLITE_THREADSAFE=1', '-DSQLITE_ENABLE_RTREE=1'] + ioFlags assert flagsFor([nitroSQLite: [threadSafe: false]], '').contains('-DSQLITE_THREADSAFE=0') - assert flagsFor([nitroSQLite: [threadSafe: false, performanceMode: false]], '') == ['-DSQLITE_THREADSAFE=0', '-DSQLITE_ENABLE_RTREE=1'] + assert flagsFor([nitroSQLite: [threadSafe: false, performanceMode: false]], '') == ['-DSQLITE_THREADSAFE=0', '-DSQLITE_ENABLE_RTREE=1'] + ioFlags assert !flagsFor([nitroSQLite: [enableRTree: false]], '').any { it.startsWith('-DSQLITE_ENABLE_RTREE') } - assert flagsFor([nitroSQLite: [threadSafe: false, performanceMode: false, enableRTree: false]], '') == ['-DSQLITE_THREADSAFE=0'] + assert !flagsFor([nitroSQLite: [enableBatchAtomicWrite: false]], '').any { it.startsWith('-DSQLITE_ENABLE_BATCH_ATOMIC_WRITE') } + assert flagsFor([nitroSQLite: [threadSafe: false, performanceMode: false, enableRTree: false, enableBatchAtomicWrite: false]], '') == ['-DSQLITE_THREADSAFE=0', '-DHAVE_FDATASYNC=1'] def customFlags = flagsFor([:], '-DSQLITE_THREADSAFE=0 -DSQLITE_DQS=3 -DSQLITE_ENABLE_FTS5=1') assert !customFlags.any { it.startsWith('-DSQLITE_THREADSAFE=') || it.startsWith('-DSQLITE_DQS=') } assert !flagsFor([:], '-DTHREADSAFE=0').any { it.startsWith('-DSQLITE_THREADSAFE=') } assert !flagsFor([:], '-DSQLITE_ENABLE_RTREE=1').any { it.startsWith('-DSQLITE_ENABLE_RTREE') } + assert !flagsFor([:], '-DHAVE_FDATASYNC=0').any { it.startsWith('-DHAVE_FDATASYNC') } def expectInvalid = { Map appPackage, String message -> try { @@ -41,5 +45,6 @@ tasks.register('testSqliteFlags') { expectInvalid([nitroSQLite: [threadSafe: 1]], 'nitroSQLite.threadSafe in package.json must be true or false') expectInvalid([nitroSQLite: [performanceMode: 'false']], 'nitroSQLite.performanceMode in package.json must be true or false') expectInvalid([nitroSQLite: [enableRTree: 'false']], 'nitroSQLite.enableRTree in package.json must be true or false') + expectInvalid([nitroSQLite: [enableBatchAtomicWrite: 0]], 'nitroSQLite.enableBatchAtomicWrite in package.json must be true or false') } } diff --git a/scripts/test-cpp.sh b/scripts/test-cpp.sh index 58afcb12..e86ea6c2 100644 --- a/scripts/test-cpp.sh +++ b/scripts/test-cpp.sh @@ -60,6 +60,38 @@ clang++ \ -o /tmp/statementGroupTests /tmp/statementGroupTests +clang++ \ + -std=c++20 \ + -Wall \ + -Wextra \ + -Werror \ + -Ipackages/react-native-nitro-sqlite/cpp \ + -Ipackages/react-native-nitro-sqlite/cpp/sqlite \ + packages/react-native-nitro-sqlite/tests/cpp/statementCache.test.cpp \ + /tmp/sqlite3.o \ + -ldl \ + -lm \ + -pthread \ + -o /tmp/statementCacheTests +/tmp/statementCacheTests + +clang++ \ + -std=c++20 \ + -Wall \ + -Wextra \ + -Werror \ + -Ipackages/react-native-nitro-sqlite/cpp \ + -Ipackages/react-native-nitro-sqlite/cpp/sqlite \ + packages/react-native-nitro-sqlite/cpp/NitroSQLiteDatabaseConnections.cpp \ + packages/react-native-nitro-sqlite/cpp/NitroSQLiteDatabaseMigration.cpp \ + packages/react-native-nitro-sqlite/tests/cpp/serialWorker.test.cpp \ + /tmp/sqlite3.o \ + -ldl \ + -lm \ + -pthread \ + -o /tmp/serialWorkerTests +/tmp/serialWorkerTests + clang \ -std=c11 \ -DSQLITE_THREADSAFE=0 \