[geary/wip/726281-text-attachment-crlf: 2/13] Make database classes more amenable to asynchronous use.
- From: Michael Gratton <mjog src gnome org>
- To: commits-list gnome org
- Cc:
- Subject: [geary/wip/726281-text-attachment-crlf: 2/13] Make database classes more amenable to asynchronous use.
- Date: Fri, 18 May 2018 23:27:12 +0000 (UTC)
commit 743b24dd5daa6c2082298c33016242904235e7d6
Author: Michael James Gratton <mike vee net>
Date: Tue May 8 09:18:06 2018 +0930
Make database classes more amenable to asynchronous use.
This makes both the open() and open_connection() methods on
Geary.DB.Database asynchronous, which allows the VersionedDatabase
open_background() and its hackery to be removed, and upgrades to be
performed asynchronously as well. It also adds a
exec_transaction_async() method to Connection, allowing an existing
object to also be used to establish an async transaction.
* src/engine/db/db-connection.vala (Connection): Add
exec_transaction_async method, update doc comments.
* src/engine/db/db-database.vala (Database): Make open and
open_connection async by executing SQLite code in a background thread,
update call sites. Move job management code out of
exec_transaction_async into a new internal add_async_job() method so it
can be used by Connection. Add unit tests.
* src/engine/db/db-transaction-async-job.vala (TransactionAsyncJob): Add
an optional internal connection method and make the job's cancellable
an internal property so a Connection instance can specify itself for
the transaction.
* src/engine/db/db-versioned-database.vala (VersionedDatabase): Remove
open_background() hack since open() is now async. Make version upgrade
hooks for derived classes async and update call sites. Use a
Nonblocking.Mutex rather than GLib mutex so upgrade exclusion works
asynchronously. Add unit tests.
* src/engine/imap-db/imap-db-database.vala (Database): Make database
upgrade methods async and execute SQL in async instructions now that
the bases classes support it. Add unit tests.
src/engine/db/db-connection.vala | 65 +++-
src/engine/db/db-database.vala | 154 ++++++----
src/engine/db/db-transaction-async-job.vala | 36 ++-
src/engine/db/db-versioned-database.vala | 283 +++++++++-------
src/engine/imap-db/imap-db-account.vala | 2 +-
src/engine/imap-db/imap-db-database.vala | 416 +++++++++++-------------
src/engine/imap-db/imap-db-gc.vala | 7 +-
test/CMakeLists.txt | 3 +
test/engine/db/db-database-test.vala | 125 +++++++
test/engine/db/db-versioned-database-test.vala | 54 +++
test/engine/imap-db/imap-db-database-test.vala | 48 +++
test/meson.build | 3 +
test/test-engine.vala | 3 +
13 files changed, 762 insertions(+), 437 deletions(-)
---
diff --git a/src/engine/db/db-connection.vala b/src/engine/db/db-connection.vala
index c9390cb..ff44d4c 100644
--- a/src/engine/db/db-connection.vala
+++ b/src/engine/db/db-connection.vala
@@ -1,16 +1,19 @@
-/* Copyright 2016 Software Freedom Conservancy Inc.
+/*
+ * Copyright 2016 Software Freedom Conservancy Inc.
*
* This software is licensed under the GNU Lesser General Public License
* (version 2.1 or later). See the COPYING file in this distribution.
*/
/**
- * A Connection represents a connection to an open database. Because SQLite uses a
- * synchronous interface, all calls are blocking. Db.Database offers asynchronous queries by
- * pooling connections and invoking queries from background threads.
+ * A Connection represents a connection to an open database.
*
- * Connections are associated with a Database. Use Database.open_connection() to create
- * one.
+ * Because SQLite uses a synchronous interface, all calls are
+ * blocking. Db.Database offers asynchronous queries by pooling
+ * connections and invoking queries from background threads.
+ *
+ * Connections are associated with a Database. Use
+ * Database.open_connection() to create one.
*
* A Connection will close when its last reference is dropped.
*/
@@ -97,8 +100,8 @@ public class Geary.Db.Connection : Geary.Db.Context {
*
* There is no way to retrieve a result iterator from this call.
*
- * This may be called from a TransactionMethod called within exec_transaction() or
- * Db.Database.exec_transaction_async().
+ * This may be called from a TransactionMethod called within
+ * {@link exec_transaction} or {@link exec_transaction_async}.
*
* See [[http://www.sqlite.org/c3ref/exec.html]]
*/
@@ -116,8 +119,8 @@ public class Geary.Db.Connection : Geary.Db.Context {
*
* There is no way to retrieve a result iterator from this call.
*
- * This can be called from a TransactionMethod called within exec_transaction() or
- * Db.Database.exec_transaction_async().
+ * This may be called from a TransactionMethod called within
+ * {@link exec_transaction} or {@link exec_transaction_async}.
*/
public void exec_file(File file, Cancellable? cancellable = null) throws Error {
check_cancelled("Connection.exec_file", cancellable);
@@ -150,14 +153,18 @@ public class Geary.Db.Connection : Geary.Db.Context {
public int get_busy_timeout_msec() {
return busy_timeout_msec;
}
-
+
/**
- * Sets busy timeout time in milliseconds. Zero or a negative value indicates that all
- * operations that SQLite returns BUSY will be retried until they complete with success or error.
- * Otherwise, after said amount of time has transpired, DatabaseError.BUSY will be thrown.
+ * Sets busy timeout time in milliseconds.
*
- * This is imperative for exec_transaction() and Db.Database.exec_transaction_async(), because
- * those calls will throw a DatabaseError.BUSY call immediately if another transaction has
+ * Zero or a negative value indicates that all operations that
+ * SQLite returns BUSY will be retried until they complete with
+ * success or error. Otherwise, after said amount of time has
+ * transpired, DatabaseError.BUSY will be thrown.
+ *
+ * This is imperative for {@link exec_transaction} {@link
+ * exec_transaction_async}, because those calls will throw a
+ * DatabaseError.BUSY call immediately if another transaction has
* acquired the reserved or exclusive locks.
*/
public void set_busy_timeout_msec(int busy_timeout_msec) throws Error {
@@ -415,7 +422,31 @@ public class Geary.Db.Connection : Geary.Db.Context {
return outcome;
}
-
+
+ /**
+ * Starts a new asynchronous transaction for this connection.
+ *
+ * Asynchronous transactions are handled via background
+ * threads. The background thread calls {@link exec_transaction};
+ * see that method for more information about coding a
+ * transaction. The only caveat is that the {@link
+ * TransactionMethod} passed to it must be thread-safe.
+ *
+ * Throws {@link DatabaseError.OPEN_REQUIRED} if not open.
+ */
+ public async TransactionOutcome exec_transaction_async(TransactionType type,
+ TransactionMethod cb,
+ Cancellable? cancellable)
+ throws Error {
+ // create job to execute in background thread
+ TransactionAsyncJob job = new TransactionAsyncJob(
+ this, type, cb, cancellable
+ );
+
+ this.database.add_async_job(job);
+ return yield job.wait_for_completion_async();
+ }
+
public override Connection? get_connection() {
return this;
}
diff --git a/src/engine/db/db-database.vala b/src/engine/db/db-database.vala
index 7312032..a558ef1 100644
--- a/src/engine/db/db-database.vala
+++ b/src/engine/db/db-database.vala
@@ -1,19 +1,23 @@
-/* Copyright 2016 Software Freedom Conservancy Inc.
+/*
+ * Copyright 2016 Software Freedom Conservancy Inc.
+ * Copyright 2018 Michael Gratton <mike vee net>
*
* This software is licensed under the GNU Lesser General Public License
* (version 2.1 or later). See the COPYING file in this distribution.
*/
/**
- * Database represents an SQLite file. Multiple Connections may be opened to against the
- * Database.
+ * Represents a single SQLite database.
*
- * Since it's often just more bookkeeping to maintain a single global connection, Database also
- * offers a master connection which may be used to perform queries and transactions.
+ * Each database supports multiple {@link Connection}s that allow SQL
+ * queries to be executed, however if a single connection is required
+ * by an app, this class also provides convenience methods to execute
+ * queries against a common ''master'' connection.
*
- * Database also offers asynchronous transactions which work via connection and thread pools.
- *
- * NOTE: In-memory databases are currently unsupported.
+ * This class offers a number of asynchronous methods, however since
+ * SQLite only supports a synchronous API, these are implemented using
+ * a pool of background threads. Asynchronous transactions are
+ * available via {@link exec_transaction_async}.
*/
public class Geary.Db.Database : Geary.Db.Context {
@@ -85,27 +89,42 @@ public class Geary.Db.Database : Geary.Db.Context {
assert(outstanding_async_jobs == 0);
}
}
-
+
/**
- * Opens the Database, creating any files and directories it may need in the process depending
- * on the DatabaseFlags.
+ * Prepares the database for use.
*
- * NOTE: A Database may be closed, but the Connections it creates will always be valid as
- * they hold a reference to their source Database. To release a Database's resources, drop all
- * references to it and its associated Connections, Statements, and Results.
+ * This will create any needed files and directories, check the
+ * database's integrity, and so on, depending on the flags passed
+ * to this method.
+ *
+ * NOTE: A Database may be closed, but the Connections it creates
+ * will always be valid as they hold a reference to their source
+ * Database. To release a Database's resources, drop all
+ * references to it and its associated Connections, Statements,
+ * and Results.
*/
- public virtual void open(DatabaseFlags flags, PrepareConnection? prepare_cb,
- Cancellable? cancellable = null) throws Error {
+ public virtual async void open(DatabaseFlags flags,
+ PrepareConnection? prepare_cb,
+ Cancellable? cancellable = null)
+ throws Error {
if (is_open)
return;
-
+
this.flags = flags;
this.prepare_cb = prepare_cb;
if (this.file != null && (flags & DatabaseFlags.CREATE_DIRECTORY) != 0) {
- File db_dir = this.file.get_parent();
- if (!db_dir.query_exists(cancellable))
+ GLib.File db_dir = this.file.get_parent();
+ try {
+ yield db_dir.query_info_async(
+ GLib.FileAttribute.STANDARD_TYPE,
+ GLib.FileQueryInfoFlags.NONE,
+ GLib.Priority.DEFAULT,
+ cancellable
+ );
+ } catch (GLib.IOError.NOT_FOUND err) {
db_dir.make_directory_with_parents(cancellable);
+ }
}
if (threadsafe()) {
@@ -116,13 +135,16 @@ public class Geary.Db.Database : Geary.Db.Context {
} else {
warning("SQLite not thread-safe: asynchronous queries will not be available");
}
-
- if ((flags & DatabaseFlags.CHECK_CORRUPTION) != 0)
- check_for_corruption(flags, cancellable);
-
+
+ if ((flags & DatabaseFlags.CHECK_CORRUPTION) != 0) {
+ yield Nonblocking.Concurrent.global.schedule_async(() => {
+ check_for_corruption(flags, cancellable);
+ }, cancellable);
+ }
+
is_open = true;
}
-
+
private void check_for_corruption(DatabaseFlags flags, Cancellable? cancellable) throws Error {
// Open a connection and test for corruption by creating a dummy table,
// adding a row, selecting the row, then dropping the table ... can only do this for
@@ -193,10 +215,15 @@ public class Geary.Db.Database : Geary.Db.Context {
/**
* Throws DatabaseError.OPEN_REQUIRED if not open.
*/
- public Connection open_connection(Cancellable? cancellable = null) throws Error {
- return internal_open_connection(false, cancellable);
+ public async Connection open_connection(Cancellable? cancellable = null)
+ throws Error {
+ Connection? cx = null;
+ yield Nonblocking.Concurrent.global.schedule_async(() => {
+ cx = internal_open_connection(false, cancellable);
+ }, cancellable);
+ return cx;
}
-
+
private Connection internal_open_connection(bool master, Cancellable? cancellable) throws Error {
check_open();
@@ -277,45 +304,60 @@ public class Geary.Db.Database : Geary.Db.Context {
Cancellable? cancellable = null) throws Error {
return get_master_connection().exec_transaction(type, cb, cancellable);
}
-
+
/**
- * Asynchronous transactions are handled via background threads using a pool of Connections.
- * The background thread calls Connection.exec_transaction(); see that method for more
- * information about coding a transaction. The only caveat is that the TransactionMethod
- * must be thread-safe.
+ * Starts a new asynchronous transaction using a new connection.
*
- * Throws DatabaseError.OPEN_REQUIRED if not open.
+ * Asynchronous transactions are handled via background
+ * threads. The background thread opens a new connection, and
+ * calls {@link Connection.exec_transaction}; see that method for
+ * more information about coding a transaction. The only caveat is
+ * that the {@link TransactionMethod} passed to it must be
+ * thread-safe.
+ *
+ * Throws {@link DatabaseError.OPEN_REQUIRED} if not open.
*/
- public async TransactionOutcome exec_transaction_async(TransactionType type, TransactionMethod cb,
- Cancellable? cancellable) throws Error {
+ public async TransactionOutcome exec_transaction_async(TransactionType type,
+ TransactionMethod cb,
+ Cancellable? cancellable)
+ throws Error {
+ TransactionAsyncJob job = new TransactionAsyncJob(
+ null, type, cb, cancellable
+ );
+ add_async_job(job);
+ return yield job.wait_for_completion_async();
+ }
+
+ /** Adds the given job to the thread pool. */
+ internal void add_async_job(TransactionAsyncJob new_job) throws Error {
check_open();
-
- if (thread_pool == null)
- throw new DatabaseError.GENERAL("SQLite thread safety disabled, async operations unallowed");
-
- // create job to execute in background thread
- TransactionAsyncJob job = new TransactionAsyncJob(type, cb, cancellable);
-
- lock (outstanding_async_jobs) {
- outstanding_async_jobs++;
+
+ if (this.thread_pool == null) {
+ throw new DatabaseError.GENERAL(
+ "SQLite thread safety disabled, async operations unallowed"
+ );
}
-
- thread_pool.add(job);
-
- return yield job.wait_for_completion_async();
+
+ lock (this.outstanding_async_jobs) {
+ this.outstanding_async_jobs++;
+ }
+
+ this.thread_pool.add(new_job);
}
-
+
// This method must be thread-safe.
private void on_async_job(owned TransactionAsyncJob job) {
// *never* use master connection for threaded operations
- Connection? cx = null;
+ Connection? cx = job.cx;
Error? open_err = null;
- try {
- cx = open_connection();
- } catch (Error err) {
- open_err = err;
- debug("Warning: unable to open database connection to %s, cancelling AsyncJob: %s",
- this.path, err.message);
+ if (cx == null) {
+ try {
+ cx = internal_open_connection(false, job.cancellable);
+ } catch (Error err) {
+ open_err = err;
+ debug("Warning: unable to open database connection to %s, cancelling AsyncJob: %s",
+ this.path, err.message);
+ }
}
if (cx != null)
diff --git a/src/engine/db/db-transaction-async-job.vala b/src/engine/db/db-transaction-async-job.vala
index 3f278f3..9e41968 100644
--- a/src/engine/db/db-transaction-async-job.vala
+++ b/src/engine/db/db-transaction-async-job.vala
@@ -1,25 +1,34 @@
-/* Copyright 2016 Software Freedom Conservancy Inc.
+/*
+ * Copyright 2016 Software Freedom Conservancy Inc.
*
* This software is licensed under the GNU Lesser General Public License
* (version 2.1 or later). See the COPYING file in this distribution.
*/
private class Geary.Db.TransactionAsyncJob : BaseObject {
+
+ internal Connection? cx { get; private set; default = null; }
+ internal Cancellable cancellable { get; private set; }
+
private TransactionType type;
private unowned TransactionMethod cb;
- private Cancellable cancellable;
private Nonblocking.Event completed;
private TransactionOutcome outcome = TransactionOutcome.ROLLBACK;
private Error? caught_err = null;
-
- public TransactionAsyncJob(TransactionType type, TransactionMethod cb, Cancellable? cancellable) {
+
+
+ public TransactionAsyncJob(Connection? cx,
+ TransactionType type,
+ TransactionMethod cb,
+ Cancellable? cancellable) {
+ this.cx = cx;
this.type = type;
this.cb = cb;
this.cancellable = cancellable ?? new Cancellable();
-
- completed = new Nonblocking.Event();
+
+ this.completed = new Nonblocking.Event();
}
-
+
public void cancel() {
cancellable.cancel();
}
@@ -82,17 +91,16 @@ private class Geary.Db.TransactionAsyncJob : BaseObject {
return false;
}
-
+
// No way to cancel this because the callback thread *must* finish before
// we move on here. Any I/O the thread is doing can still be cancelled
// using our cancel() above.
public async TransactionOutcome wait_for_completion_async()
throws Error {
- yield completed.wait_async();
- if (caught_err != null)
- throw caught_err;
-
- return outcome;
+ yield this.completed.wait_async();
+ if (this.caught_err != null)
+ throw this.caught_err;
+
+ return this.outcome;
}
}
-
diff --git a/src/engine/db/db-versioned-database.vala b/src/engine/db/db-versioned-database.vala
index e4a8b57..8992e79 100644
--- a/src/engine/db/db-versioned-database.vala
+++ b/src/engine/db/db-versioned-database.vala
@@ -1,13 +1,32 @@
-/* Copyright 2016 Software Freedom Conservancy Inc.
+/*
+ * Copyright 2016 Software Freedom Conservancy Inc.
+ * Copyright 2018 Michael Gratton <mike vee net>
*
* This software is licensed under the GNU Lesser General Public License
* (version 2.1 or later). See the COPYING file in this distribution.
*/
+/**
+ * A SQLite database with a versioned, upgradeable schema.
+ *
+ * This class uses the SQLite user version pragma to track the current
+ * version of a database, and a set of SQL scripts (one per version)
+ * to manage updating from one version to another. When the database
+ * is first opened by a call to {@link open}, its current version is
+ * checked against the set of available scripts, and each available
+ * version script above the current version is applied in
+ * order. Derived classes may override the {@link pre_upgrade} and
+ * {@link post_upgrade} methods to perform additional work before and
+ * after an upgrade script is executed, and {@link starting_upgrade}
+ * and {@link completed_upgrade} to be notified of the upgrade process
+ * starting and finishing.
+ */
public class Geary.Db.VersionedDatabase : Geary.Db.Database {
- public delegate void WorkCallback();
- private static Mutex upgrade_mutex = Mutex();
+
+ private static Geary.Nonblocking.Mutex upgrade_mutex =
+ new Geary.Nonblocking.Mutex();
+
public File schema_dir { get; private set; }
@@ -26,58 +45,48 @@ public class Geary.Db.VersionedDatabase : Geary.Db.Database {
/**
* Called by {@link open} if a schema upgrade is required and beginning.
*
- * If called by {@link open_background}, this will be called in the context of a background
- * thread.
- *
* If new_db is set to true, the database is being created from scratch.
*/
protected virtual void starting_upgrade(int current_version, bool new_db) {
}
-
+
/**
* Called by {@link open} just before performing a schema upgrade step.
- *
- * If called by {@link open_background}, this will be called in the context of a background
- * thread.
*/
- protected virtual void pre_upgrade(int version) {
+ protected virtual async void pre_upgrade(int version, Cancellable? cancellable)
+ throws Error {
}
-
+
/**
* Called by {@link open} just after performing a schema upgrade step.
- *
- * If called by {@link open_background}, this will be called in the context of a background
- * thread.
*/
- protected virtual void post_upgrade(int version) {
+ protected virtual async void post_upgrade(int version, Cancellable? cancellable)
+ throws Error {
}
-
+
/**
* Called by {@link open} if a schema upgrade was required and has now completed.
- *
- * If called by {@link open_background}, this will be called in the context of a background
- * thread.
*/
protected virtual void completed_upgrade(int final_version) {
}
-
- private File get_schema_file(int db_version) {
- return schema_dir.get_child("version-%03d.sql".printf(db_version));
- }
-
+
/**
- * Creates or opens the database, initializing and upgrading the schema.
+ * Prepares the database for use, initializing and upgrading the schema.
*
- * If it's detected that the database has a schema version that's unavailable in the schema
- * directory, throws {@link DatabaseError.SCHEMA_VERSION}. Generally this indicates the
- * user attempted to load the database with an older version of the application.
+ * If it's detected that the database has a schema version that's
+ * unavailable in the schema directory, throws {@link
+ * DatabaseError.SCHEMA_VERSION}. Generally this indicates the
+ * user attempted to load the database with an older version of
+ * the application.
*/
- public override void open(DatabaseFlags flags, PrepareConnection? prepare_cb,
- Cancellable? cancellable = null) throws Error {
- base.open(flags, prepare_cb, cancellable);
-
+ public override async void open(DatabaseFlags flags,
+ PrepareConnection? prepare_cb,
+ Cancellable? cancellable = null)
+ throws Error {
+ yield base.open(flags, prepare_cb, cancellable);
+
// get Connection for upgrade activity
- Connection cx = open_connection(cancellable);
+ Connection cx = yield open_connection(cancellable);
int db_version = cx.get_user_version_number();
debug("VersionedDatabase.upgrade: current database schema for %s: %d",
@@ -90,117 +99,137 @@ public class Geary.Db.VersionedDatabase : Geary.Db.Database {
// Initialize new database to version 1 (note the preincrement in the loop below)
if (db_version < 0)
db_version = 0;
-
- // Check for database schemas newer than what's available in the schema directory; this
- // happens some times in development or if a user attempts to roll back their version
- // of the app without restoring a backup of the database ... since schema is so important
- // to database coherency, need to protect against both
- //
- // Note that this is checking for a schema file for the current version of the database
- // (assuming it's version 1 or better); the next check autoincrements to look for the
- // *next* version of the database
- if (db_version > 0 && !get_schema_file(db_version).query_exists(cancellable)) {
- throw new DatabaseError.SCHEMA_VERSION("%s schema %d unknown to current schema plan",
- this.path, db_version);
+
+ if (db_version > 0) {
+ // Check for database schemas newer than what's available
+ // in the schema directory; this happens some times in
+ // development or if a user attempts to roll back their
+ // version of the app without restoring a backup of the
+ // database ... since schema is so important to database
+ // coherency, need to protect against both
+ //
+ // Note that this is checking for a schema file for the
+ // current version of the database (assuming it's version
+ // 1 or better); the next check autoincrements to look for
+ // the *next* version of the database
+ if (!yield exists(get_schema_file(db_version), cancellable)) {
+ throw new DatabaseError.SCHEMA_VERSION(
+ "%s schema %d unknown to current schema plan",
+ this.path, db_version
+ );
+ }
}
// Go through all the version scripts in the schema directory and apply each of them.
bool started = false;
for (;;) {
File upgrade_script = get_schema_file(++db_version);
- if (!upgrade_script.query_exists(cancellable))
+ if (!yield exists(upgrade_script, cancellable)) {
break;
-
+ }
+
if (!started) {
starting_upgrade(db_version, new_db);
started = true;
}
-
- // Since these upgrades run in a background thread, there's a possibility they
- // can run in parallel. That leads to Geary absolutely taking over the machine,
- // with potentially several threads all doing heavy database manipulation at
- // once. So, we wrap this bit in a mutex lock so that only one database is
- // updating at once. It means overall it might take a bit longer, but it keeps
- // things usable in the meantime. See <https://bugzilla.gnome.org/show_bug.cgi?id=724475>.
- upgrade_mutex.@lock();
-
- pre_upgrade(db_version);
-
- check_cancelled("VersionedDatabase.open", cancellable);
-
+
+ // Since these upgrades run in a background thread,
+ // there's a possibility they can run in parallel. That
+ // leads to Geary absolutely taking over the machine, with
+ // potentially several threads all doing heavy database
+ // manipulation at once. So, we wrap this bit in a mutex
+ // lock so that only one database is updating at once. It
+ // means overall it might take a bit longer, but it keeps
+ // things usable in the meantime. See
+ // <https://bugzilla.gnome.org/show_bug.cgi?id=724475>.
+ int token = yield VersionedDatabase.upgrade_mutex.claim_async(
+ cancellable
+ );
+
+ Error? locked_err = null;
try {
- debug("Upgrading database to version %d with %s", db_version, upgrade_script.get_path());
- cx.exec_transaction(TransactionType.EXCLUSIVE, (cx) => {
- cx.exec_file(upgrade_script, cancellable);
- cx.set_user_version_number(db_version);
-
- return TransactionOutcome.COMMIT;
- }, cancellable);
+ yield execute_upgrade(
+ cx, db_version, upgrade_script, cancellable
+ );
} catch (Error err) {
- warning("Error upgrading database to version %d: %s", db_version, err.message);
- upgrade_mutex.unlock();
-
- throw err;
+ locked_err = err;
+ }
+
+ VersionedDatabase.upgrade_mutex.release(ref token);
+
+ if (locked_err != null) {
+ throw locked_err;
}
-
- post_upgrade(db_version);
-
- upgrade_mutex.unlock();
}
-
+
if (started)
completed_upgrade(db_version);
}
-
- /**
- * Opens the database in a background thread so foreground work can be performed while updating.
- *
- * Since {@link open} may take a considerable amount of time for a {@link VersionedDatabase},
- * background_open() can be used to perform that work in a thread while the calling thread
- * "pumps" a {@link WorkCallback} every work_cb_msec milliseconds. In general, this is
- * designed for allowing an event queue to execute tasks or update a progress monitor of some
- * kind.
- *
- * Note that the database is not opened while the callback is executing and so it should not
- * call into the database (unless it's a call safe to use prior to open).
- *
- * If work_cb_sec is zero or less, WorkCallback is called continuously, which may or may not be
- * desired.
- *
- * @see open
- */
- public void open_background(DatabaseFlags flags, PrepareConnection? prepare_cb,
- WorkCallback work_cb, int work_cb_msec, Cancellable? cancellable = null) throws Error {
- // use a SpinWaiter to safely wait for the thread to exit while occassionally calling the
- // WorkCallback (which can not abort in current impl.) to do foreground work.
- Synchronization.SpinWaiter waiter = new Synchronization.SpinWaiter(work_cb_msec, () => {
- work_cb();
-
- // continue (never abort)
- return true;
- });
-
- // do the open in a background thread
- Error? thread_err = null;
- Thread<bool> thread = new Thread<bool>.try("Geary.Db.VersionedDatabase.open()", () => {
- try {
- open(flags, prepare_cb, cancellable);
- } catch (Error err) {
- thread_err = err;
+
+ private async void execute_upgrade(Connection cx,
+ int db_version,
+ GLib.File upgrade_script,
+ Cancellable? cancellable)
+ throws Error {
+ debug("Upgrading database to version %d with %s",
+ db_version, upgrade_script.get_path());
+
+ check_cancelled("VersionedDatabase.open", cancellable);
+ try {
+ yield pre_upgrade(db_version, cancellable);
+ } catch (Error err) {
+ if (!(err is IOError.CANCELLED)) {
+ warning("Error executing pre-upgrade for version %d: %s",
+ db_version, err.message);
}
-
- // notify the foreground waiter we're done
- waiter.notify();
-
- return true;
- });
-
- // wait until thread is completed and then dispose of it
- waiter.wait();
- thread = null;
-
- if (thread_err != null)
- throw thread_err;
+ throw err;
+ }
+
+ check_cancelled("VersionedDatabase.open", cancellable);
+ try {
+ yield cx.exec_transaction_async(TransactionType.EXCLUSIVE, (cx) => {
+ cx.exec_file(upgrade_script, cancellable);
+ cx.set_user_version_number(db_version);
+
+ return TransactionOutcome.COMMIT;
+ }, cancellable);
+ } catch (Error err) {
+ if (!(err is IOError.CANCELLED)) {
+ warning("Error upgrading database to version %d: %s",
+ db_version, err.message);
+ }
+ throw err;
+ }
+
+ check_cancelled("VersionedDatabase.open", cancellable);
+ try {
+ yield post_upgrade(db_version, cancellable);
+ } catch (Error err) {
+ if (!(err is IOError.CANCELLED)) {
+ warning("Error executing post-upgrade for version %d: %s",
+ db_version, err.message);
+ }
+ throw err;
+ }
}
-}
+ private File get_schema_file(int db_version) {
+ return schema_dir.get_child("version-%03d.sql".printf(db_version));
+ }
+
+ private async bool exists(GLib.File target, Cancellable? cancellable) {
+ bool ret = true;
+ try {
+ yield target.query_info_async(
+ GLib.FileAttribute.STANDARD_TYPE,
+ GLib.FileQueryInfoFlags.NONE,
+ GLib.Priority.DEFAULT,
+ cancellable
+ );
+ } catch (Error err) {
+ ret = false;
+ }
+ return ret;
+ }
+
+}
diff --git a/src/engine/imap-db/imap-db-account.vala b/src/engine/imap-db/imap-db-account.vala
index cfa9f8e..a91acd0 100644
--- a/src/engine/imap-db/imap-db-account.vala
+++ b/src/engine/imap-db/imap-db-account.vala
@@ -264,7 +264,7 @@ private class Geary.ImapDB.Account : BaseObject {
account_information.primary_mailbox.address);
try {
- yield db.open_async(
+ yield db.open(
Db.DatabaseFlags.CREATE_DIRECTORY | Db.DatabaseFlags.CREATE_FILE |
Db.DatabaseFlags.CHECK_CORRUPTION,
cancellable);
} catch (Error err) {
diff --git a/src/engine/imap-db/imap-db-database.vala b/src/engine/imap-db/imap-db-database.vala
index 61c7510..d7686d2 100644
--- a/src/engine/imap-db/imap-db-database.vala
+++ b/src/engine/imap-db/imap-db-database.vala
@@ -1,4 +1,5 @@
-/* Copyright 2016 Software Freedom Conservancy Inc.
+/*
+ * Copyright 2016 Software Freedom Conservancy Inc.
*
* This software is licensed under the GNU Lesser General Public License
* (version 2.1 or later). See the COPYING file in this distribution.
@@ -34,15 +35,12 @@ private class Geary.ImapDB.Database : Geary.Db.VersionedDatabase {
}
/**
- * Opens the ImapDB database.
- *
- * This should only be done from the main thread, as it is designed to pump the event loop
- * while the database is being opened and updated.
+ * Prepares the ImapDB database for use.
*/
- public async void open_async(Db.DatabaseFlags flags, Cancellable? cancellable) throws Error {
- open_background(flags, on_prepare_database_connection, pump_event_loop,
- OPEN_PUMP_EVENT_LOOP_MSEC, cancellable);
-
+ public new async void open(Db.DatabaseFlags flags, Cancellable? cancellable)
+ throws Error {
+ yield base.open(flags, on_prepare_database_connection, cancellable);
+
// Tie user-supplied Cancellable to internal Cancellable, which is used when close() is
// called
if (cancellable != null)
@@ -103,140 +101,133 @@ private class Geary.ImapDB.Database : Geary.Db.VersionedDatabase {
base.close(cancellable);
}
-
- private void pump_event_loop() {
- while (MainContext.default().pending())
- MainContext.default().iteration(true);
- }
-
+
protected override void starting_upgrade(int current_version, bool new_db) {
this.new_db = new_db;
- // can't call the ProgressMonitor directly, as it's hooked up to signals that expect to be
- // called in the foreground thread, so use the Idle loop for this
- Idle.add(() => {
- // don't use upgrade_monitor for new databases, as the upgrade should be near-
- // instantaneous. Also, there's some issue with GTK when starting the progress
- // monitor while GtkDialog's are in play:
- // https://bugzilla.gnome.org/show_bug.cgi?id=726269
- if (!new_db && !upgrade_monitor.is_in_progress)
- upgrade_monitor.notify_start();
-
- return false;
- });
+
+ // don't use upgrade_monitor for new databases, as the upgrade should be near-
+ // instantaneous. Also, there's some issue with GTK when starting the progress
+ // monitor while GtkDialog's are in play:
+ // https://bugzilla.gnome.org/show_bug.cgi?id=726269
+ if (!new_db && !upgrade_monitor.is_in_progress) {
+ upgrade_monitor.notify_start();
+ }
}
-
+
protected override void completed_upgrade(int final_version) {
- // see starting_upgrade() for explanation why this is done in Idle loop
- Idle.add(() => {
- if (!new_db && upgrade_monitor.is_in_progress)
- upgrade_monitor.notify_finish();
-
- return false;
- });
+ if (!new_db && upgrade_monitor.is_in_progress) {
+ upgrade_monitor.notify_finish();
+ }
}
-
- protected override void post_upgrade(int version) {
+
+ protected async override void post_upgrade(int version,
+ Cancellable? cancellable)
+ throws Error {
switch (version) {
case 5:
- post_upgrade_populate_autocomplete();
+ yield post_upgrade_populate_autocomplete(cancellable);
break;
-
+
case 6:
- post_upgrade_encode_folder_names();
+ yield post_upgrade_encode_folder_names(cancellable);
break;
-
+
case 11:
- post_upgrade_add_search_table();
+ yield post_upgrade_add_search_table(cancellable);
break;
-
+
case 12:
- post_upgrade_populate_internal_date_time_t();
+ yield post_upgrade_populate_internal_date_time_t(cancellable);
break;
-
+
case 13:
- post_upgrade_populate_additional_attachments();
+ yield post_upgrade_populate_additional_attachments(cancellable);
break;
-
+
case 14:
- post_upgrade_expand_page_size();
+ yield post_upgrade_expand_page_size(cancellable);
break;
-
+
case 15:
- post_upgrade_fix_localized_internaldates();
+ yield post_upgrade_fix_localized_internaldates(cancellable);
break;
-
+
case 18:
- post_upgrade_populate_internal_date_time_t();
+ yield post_upgrade_populate_internal_date_time_t(cancellable);
break;
-
+
case 19:
- post_upgrade_validate_contacts();
+ yield post_upgrade_validate_contacts(cancellable);
break;
-
+
case 22:
- post_upgrade_rebuild_attachments();
+ yield post_upgrade_rebuild_attachments(cancellable);
break;
-
+
case 23:
- post_upgrade_add_tokenizer_table();
+ yield post_upgrade_add_tokenizer_table(cancellable);
break;
}
}
-
+
// Version 5.
- private void post_upgrade_populate_autocomplete() {
- try {
- Db.Result result = query("SELECT sender, from_field, to_field, cc, bcc FROM MessageTable");
- while (!result.finished) {
- MessageAddresses message_addresses =
+ private async void post_upgrade_populate_autocomplete(Cancellable? cancellable)
+ throws Error {
+ yield exec_transaction_async(Db.TransactionType.RW, (cx) => {
+ Db.Result result = cx.query(
+ "SELECT sender, from_field, to_field, cc, bcc FROM MessageTable"
+ );
+ while (!result.finished && !cancellable.is_cancelled()) {
+ MessageAddresses message_addresses =
new MessageAddresses.from_result(account_owner_email, result);
- foreach (Contact contact in message_addresses.contacts) {
- do_update_contact(get_master_connection(), contact, null);
+ foreach (Contact contact in message_addresses.contacts) {
+ do_update_contact(cx, contact, null);
+ }
+ result.next();
}
-
- result.next();
- }
- } catch (Error err) {
- debug("Error populating autocompletion table during upgrade to database schema 5");
- }
+ return Geary.Db.TransactionOutcome.COMMIT;
+ }, cancellable);
}
-
+
// Version 6.
- private void post_upgrade_encode_folder_names() {
- try {
- Db.Result select = query("SELECT id, name FROM FolderTable");
- while (!select.finished) {
- int64 id = select.int64_at(0);
- string encoded_name = select.nonnull_string_at(1);
-
- try {
- string canonical_name = Geary.ImapUtf7.imap_utf7_to_utf8(encoded_name);
-
- Db.Statement update = prepare("UPDATE FolderTable SET name=? WHERE id=?");
- update.bind_string(0, canonical_name);
- update.bind_int64(1, id);
- update.exec();
- } catch (Error e) {
- debug("Error renaming folder %s to its canonical representation: %s", encoded_name,
e.message);
+ private async void post_upgrade_encode_folder_names(Cancellable? cancellable)
+ throws Error {
+ yield exec_transaction_async(Db.TransactionType.RW, (cx) => {
+ Db.Result select = cx.query("SELECT id, name FROM FolderTable");
+ while (!select.finished && !cancellable.is_cancelled()) {
+ int64 id = select.int64_at(0);
+ string encoded_name = select.nonnull_string_at(1);
+
+ try {
+ string canonical_name = Geary.ImapUtf7.imap_utf7_to_utf8(encoded_name);
+
+ Db.Statement update = cx.prepare(
+ "UPDATE FolderTable SET name=? WHERE id=?"
+ );
+ update.bind_string(0, canonical_name);
+ update.bind_int64(1, id);
+ update.exec();
+ } catch (Error e) {
+ debug("Error renaming folder %s to its canonical representation: %s", encoded_name,
e.message);
+ }
+
+ select.next();
}
-
- select.next();
- }
- } catch (Error e) {
- debug("Error decoding folder names during upgrade to database schema 6: %s", e.message);
- }
+ return Geary.Db.TransactionOutcome.COMMIT;
+ }, cancellable);
}
-
+
// Version 11.
- private void post_upgrade_add_search_table() {
- try {
- string stemmer = find_appropriate_search_stemmer();
- debug("Creating search table using %s stemmer", stemmer);
-
- // This can't go in the .sql file because its schema (the stemmer
- // algorithm) is determined at runtime.
- exec("""
- CREATE VIRTUAL TABLE MessageSearchTable USING fts4(
+ private async void post_upgrade_add_search_table(Cancellable? cancellable)
+ throws Error {
+ yield exec_transaction_async(Db.TransactionType.RW, (cx) => {
+ string stemmer = find_appropriate_search_stemmer();
+ debug("Creating search table using %s stemmer", stemmer);
+
+ // This can't go in the .sql file because its schema (the stemmer
+ // algorithm) is determined at runtime.
+ cx.exec("""
+ CREATE VIRTUAL TABLE MessageSearchTable USING fts4(
body,
attachment,
subject,
@@ -244,16 +235,15 @@ private class Geary.ImapDB.Database : Geary.Db.VersionedDatabase {
receivers,
cc,
bcc,
-
+
tokenize=unicodesn "stemmer=%s",
prefix="2,4,6,8,10",
);
- """.printf(stemmer));
- } catch (Error e) {
- error("Error creating search table: %s", e.message);
- }
+ """.printf(stemmer));
+ return Geary.Db.TransactionOutcome.COMMIT;
+ }, cancellable);
}
-
+
private string find_appropriate_search_stemmer() {
// Unfortunately, the stemmer library only accepts the full language
// name for the stemming algorithm. This translates between the user's
@@ -280,27 +270,30 @@ private class Geary.ImapDB.Database : Geary.Db.VersionedDatabase {
case "tr": return "turkish";
}
}
-
+
// Default to English because it seems to be on average the language
// most likely to be present in emails, regardless of the user's
// language setting. This is not an exact science, and search results
// should be ok either way in most cases.
return "english";
}
-
+
// Versions 12 and 18.
- private void post_upgrade_populate_internal_date_time_t() {
- try {
- exec_transaction(Db.TransactionType.RW, (cx) => {
- Db.Result select = cx.query("SELECT id, internaldate FROM MessageTable");
+ private async void
+ post_upgrade_populate_internal_date_time_t(Cancellable? cancellable)
+ throws Error {
+ yield exec_transaction_async(Db.TransactionType.RW, (cx) => {
+ Db.Result select = cx.query(
+ "SELECT id, internaldate FROM MessageTable"
+ );
while (!select.finished) {
int64 id = select.rowid_at(0);
string? internaldate = select.string_at(1);
-
+
try {
time_t as_time_t = (internaldate != null ?
Geary.Imap.InternalDate.decode(internaldate).to_time_t() : -1);
-
+
Db.Statement update = cx.prepare(
"UPDATE MessageTable SET internaldate_time_t=? WHERE id=?");
update.bind_int64(0, (int64) as_time_t);
@@ -310,22 +303,19 @@ private class Geary.ImapDB.Database : Geary.Db.VersionedDatabase {
debug("Error converting internaldate '%s' to time_t: %s",
internaldate, e.message);
}
-
+
select.next();
}
-
+
return Db.TransactionOutcome.COMMIT;
- });
- } catch (Error e) {
- debug("Error populating internaldate_time_t column during upgrade to database schema 12: %s",
- e.message);
- }
+ }, cancellable);
}
-
+
// Version 13.
- private void post_upgrade_populate_additional_attachments() {
- try {
- exec_transaction(Db.TransactionType.RW, (cx) => {
+ private async void
+ post_upgrade_populate_additional_attachments(Cancellable? cancellable)
+ throws Error {
+ yield exec_transaction_async(Db.TransactionType.RW, (cx) => {
Db.Statement stmt = cx.prepare("""
SELECT id, header, body
FROM MessageTable
@@ -334,11 +324,12 @@ private class Geary.ImapDB.Database : Geary.Db.VersionedDatabase {
stmt.bind_int(0, Geary.Email.REQUIRED_FOR_MESSAGE);
stmt.bind_int(1, Geary.Email.REQUIRED_FOR_MESSAGE);
Db.Result select = stmt.exec();
+
while (!select.finished) {
int64 id = select.rowid_at(0);
Geary.Memory.Buffer header = select.string_buffer_at(1);
Geary.Memory.Buffer body = select.string_buffer_at(2);
-
+
try {
Geary.RFC822.Message message = new Geary.RFC822.Message.from_parts(
new RFC822.Header(header), new RFC822.Text(body));
@@ -350,53 +341,52 @@ private class Geary.ImapDB.Database : Geary.Db.VersionedDatabase {
} catch (Error e) {
debug("Error fetching inline Mime parts: %s", e.message);
}
-
+
select.next();
}
-
+
// additionally, because this schema change (and code changes as well) introduces
// two new types of attachments as well as processing for all MIME text sections
// of messages (not just the first one), blow away the search table and let the
// search indexer start afresh
cx.exec("DELETE FROM MessageSearchTable");
-
+
return Db.TransactionOutcome.COMMIT;
- });
- } catch (Error e) {
- debug("Error populating old inline attachments during upgrade to database schema 13: %s",
- e.message);
- }
+ }, cancellable);
}
-
+
// Version 14.
- private void post_upgrade_expand_page_size() {
- try {
- // When the MessageSearchTable is first touched, SQLite seems to
- // read the whole table into memory (or an awful lot of data,
- // either way). This was causing slowness when Geary first started
- // and checked for any messages not yet in the search table. With
- // the database's page_size set to 4096, the reads seem to happen
- // about 2 orders of magnitude quicker, probably because 4096
- // matches the default filesystem block size and/or Linux's default
- // memory page size. With this set, the full read into memory is
- // barely noticeable even on slow machines.
-
- // NOTE: these can't be in the .sql file itself because they must
- // be back to back, outside of a transaction.
- exec("""
- PRAGMA page_size = 4096;
- VACUUM;
- """);
- } catch (Error e) {
- debug("Error bumping page_size or vacuuming database; performance may be degraded: %s",
- e.message);
- }
+ private async void post_upgrade_expand_page_size(Cancellable? cancellable)
+ throws Error {
+ // When the MessageSearchTable is first touched,
+ // SQLite seems to read the whole table into memory
+ // (or an awful lot of data, either way). This was
+ // causing slowness when Geary first started and
+ // checked for any messages not yet in the search
+ // table. With the database's page_size set to 4096,
+ // the reads seem to happen about 2 orders of
+ // magnitude quicker, probably because 4096 matches
+ // the default filesystem block size and/or Linux's
+ // default memory page size. With this set, the full
+ // read into memory is barely noticeable even on slow
+ // machines.
+
+ // NOTE: these can't be in the .sql file itself because
+ // they must be back to back, outside of a transaction
+ Geary.Db.Connection cx = yield open_connection();
+ yield Nonblocking.Concurrent.global.schedule_async(() => {
+ cx.exec("""
+ PRAGMA page_size = 4096;
+ VACUUM;
+ """);
+ }, cancellable);
}
-
+
// Version 15
- private void post_upgrade_fix_localized_internaldates() {
- try {
- exec_transaction(Db.TransactionType.RW, (cx) => {
+ private async void
+ post_upgrade_fix_localized_internaldates(Cancellable? cancellable)
+ throws Error {
+ yield exec_transaction_async(Db.TransactionType.RW, (cx) => {
Db.Statement stmt = cx.prepare("""
SELECT id, internaldate, fields
FROM MessageTable
@@ -443,19 +433,15 @@ private class Geary.ImapDB.Database : Geary.Db.VersionedDatabase {
// reuse statment, overwrite invalid_id, fields only
stmt.reset(Db.ResetScope.SAVE_BINDINGS);
}
-
+
return Db.TransactionOutcome.COMMIT;
- });
- } catch (Error err) {
- debug("Error fixing INTERNALDATES during upgrade to schema 15 for %s: %s",
- this.path, err.message);
- }
+ }, cancellable);
}
-
+
// Version 19.
- private void post_upgrade_validate_contacts() {
- try {
- exec_transaction(Db.TransactionType.RW, (cx) => {
+ private async void post_upgrade_validate_contacts(Cancellable? cancellable)
+ throws Error {
+ yield exec_transaction_async(Db.TransactionType.RW, (cx) => {
Db.Result result = cx.query("SELECT id, email FROM ContactTable");
while (!result.finished) {
string email = result.string_at(1);
@@ -471,16 +457,13 @@ private class Geary.ImapDB.Database : Geary.Db.VersionedDatabase {
}
return Db.TransactionOutcome.COMMIT;
- });
- } catch (Error e) {
- debug("Error fixing up contacts table: %s", e.message);
- }
+ }, cancellable);
}
-
+
// Version 22
- private void post_upgrade_rebuild_attachments() {
- try {
- exec_transaction(Db.TransactionType.RW, (cx) => {
+ private async void post_upgrade_rebuild_attachments(Cancellable? cancellable)
+ throws Error {
+ yield exec_transaction_async(Db.TransactionType.RW, (cx) => {
Db.Statement stmt = cx.prepare("""
SELECT id, header, body
FROM MessageTable
@@ -492,34 +475,33 @@ private class Geary.ImapDB.Database : Geary.Db.VersionedDatabase {
Db.Result results = stmt.exec();
if (results.finished)
return Db.TransactionOutcome.ROLLBACK;
-
+
do {
int64 message_id = results.rowid_at(0);
Geary.Memory.Buffer header = results.string_buffer_at(1);
Geary.Memory.Buffer body = results.string_buffer_at(2);
-
+
Geary.RFC822.Message message;
try {
message = new Geary.RFC822.Message.from_parts(
new RFC822.Header(header), new RFC822.Text(body));
} catch (Error err) {
debug("Error decoding message: %s", err.message);
-
continue;
}
-
+
// build a list of attachments in the message itself
- Gee.List<GMime.Part> msg_attachments = message.get_attachments();
-
- // delete all attachments for this message
+ Gee.List<GMime.Part> msg_attachments =
+ message.get_attachments();
+
try {
Geary.ImapDB.Folder.do_delete_attachments(cx, message_id);
} catch (Error err) {
- debug("Error deleting existing attachments: %s", err.message);
-
+ debug("Error deleting existing attachments: %s",
+ err.message);
continue;
}
-
+
// rebuild all
try {
Geary.ImapDB.Folder.do_save_attachments_db(cx, message_id, msg_attachments,
@@ -530,35 +512,31 @@ private class Geary.ImapDB.Database : Geary.Db.VersionedDatabase {
// fallthrough
}
} while (results.next());
-
+
// rebuild search table due to potentially new attachments
cx.exec("DELETE FROM MessageSearchTable");
-
+
return Db.TransactionOutcome.COMMIT;
- });
- } catch (Error e) {
- debug("Error populating old inline attachments during upgrade to database schema 13: %s",
- e.message);
- }
+ }, cancellable);
}
-
+
// Version 23
- private void post_upgrade_add_tokenizer_table() {
- try {
- string stemmer = find_appropriate_search_stemmer();
- debug("Creating tokenizer table using %s stemmer", stemmer);
-
- // These can't go in the .sql file because its schema (the stemmer
- // algorithm) is determined at runtime.
- exec("""
- CREATE VIRTUAL TABLE TokenizerTable USING fts3tokenize(
- unicodesn,
- "stemmer=%s"
- );
- """.printf(stemmer));
- } catch (Error e) {
- error("Error creating tokenizer table: %s", e.message);
- }
+ private async void post_upgrade_add_tokenizer_table(Cancellable? cancellable)
+ throws Error {
+ yield exec_transaction_async(Db.TransactionType.RW, (cx) => {
+ string stemmer = find_appropriate_search_stemmer();
+ debug("Creating tokenizer table using %s stemmer", stemmer);
+
+ // These can't go in the .sql file because its schema (the stemmer
+ // algorithm) is determined at runtime.
+ cx.exec("""
+ CREATE VIRTUAL TABLE TokenizerTable USING fts3tokenize(
+ unicodesn,
+ "stemmer=%s"
+ );
+ """.printf(stemmer));
+ return Db.TransactionOutcome.COMMIT;
+ }, cancellable);
}
/**
@@ -612,5 +590,5 @@ private class Geary.ImapDB.Database : Geary.Db.VersionedDatabase {
cx.set_synchronous(Db.SynchronousMode.NORMAL);
sqlite3_unicodesn_register_tokenizer(cx.db);
}
-}
+}
diff --git a/src/engine/imap-db/imap-db-gc.vala b/src/engine/imap-db/imap-db-gc.vala
index 11ec3b1..b567cb4 100644
--- a/src/engine/imap-db/imap-db-gc.vala
+++ b/src/engine/imap-db/imap-db-gc.vala
@@ -205,9 +205,10 @@ private class Geary.ImapDB.GC {
// NOTE: VACUUM cannot happen inside a transaction, so to avoid blocking the main thread,
// run a non-transacted command from a background thread
+ Geary.Db.Connection cx = yield db.open_connection(cancellable);
yield Nonblocking.Concurrent.global.schedule_async(() => {
- db.open_connection(cancellable).exec("VACUUM", cancellable);
-
+ cx.exec("VACUUM", cancellable);
+
// it's a small thing, but take snapshot of time when vacuum completes, as scheduling
// of the next transaction is not instantaneous
last_vacuum_time = new DateTime.now_local();
@@ -220,7 +221,7 @@ private class Geary.ImapDB.GC {
// update last vacuum time and reset messages reaped since last vacuum ... don't allow this
// to be cancelled, really want to get this in stone so the user doesn't re-vacuum
// unnecessarily
- yield db.exec_transaction_async(Db.TransactionType.WO, (cx) => {
+ yield cx.exec_transaction_async(Db.TransactionType.WO, (cx) => {
Db.Statement stmt = cx.prepare("""
UPDATE GarbageCollectionTable
SET last_vacuum_time_t = ?, reaped_messages_since_last_vacuum = ?
diff --git a/test/CMakeLists.txt b/test/CMakeLists.txt
index f1ad286..cd96c03 100644
--- a/test/CMakeLists.txt
+++ b/test/CMakeLists.txt
@@ -26,9 +26,12 @@ set(TEST_ENGINE_SRC
engine/app/app-conversation-test.vala
engine/app/app-conversation-monitor-test.vala
engine/app/app-conversation-set-test.vala
+ engine/db/db-database-test.vala
+ engine/db/db-versioned-database-test.vala
engine/imap/command/imap-create-command-test.vala
engine/imap/response/imap-namespace-response-test.vala
engine/imap/transport/imap-deserializer-test.vala
+ engine/imap-db/imap-db-database-test.vala
engine/imap-engine/account-processor-test.vala
engine/mime-content-type-test.vala
engine/rfc822-mailbox-address-test.vala
diff --git a/test/engine/db/db-database-test.vala b/test/engine/db/db-database-test.vala
new file mode 100644
index 0000000..e79e8d4
--- /dev/null
+++ b/test/engine/db/db-database-test.vala
@@ -0,0 +1,125 @@
+/*
+ * Copyright 2018 Michael Gratton <mike vee net>
+ *
+ * This software is licensed under the GNU Lesser General Public License
+ * (version 2.1 or later). See the COPYING file in this distribution.
+ */
+
+class Geary.Db.DatabaseTest : TestCase {
+
+
+ public DatabaseTest() {
+ base("Geary.Db.DatabaseTest");
+ add_test("transient_open", transient_open);
+ add_test("open_existing", open_existing);
+ add_test("open_create_file", open_create_file);
+ add_test("open_create_dir", open_create_dir);
+ add_test("open_create_dir_existing", open_create_dir_existing);
+ }
+
+ public void transient_open() throws Error {
+ Database db = new Geary.Db.Database.transient();
+ db.open.begin(
+ Geary.Db.DatabaseFlags.NONE, null, null,
+ (obj, ret) => { async_complete(ret); }
+ );
+ db.open.end(async_result());
+
+ // Need to get a connection since the database doesn't
+ // actually get created until then
+ db.get_master_connection();
+ }
+
+ public void open_existing() throws Error {
+ GLib.FileIOStream stream;
+ GLib.File tmp_file = GLib.File.new_tmp(
+ "geary-db-database-test-XXXXXX", out stream
+ );
+
+ Database db = new Geary.Db.Database.persistent(tmp_file);
+ db.open.begin(
+ Geary.Db.DatabaseFlags.NONE, null, null,
+ (obj, ret) => { async_complete(ret); }
+ );
+ db.open.end(async_result());
+
+ // Need to get a connection since the database doesn't
+ // actually get created until then
+ db.get_master_connection();
+
+ tmp_file.delete();
+ }
+
+ public void open_create_file() throws Error {
+ GLib.File tmp_dir = GLib.File.new_for_path(
+ GLib.DirUtils.make_tmp("geary-db-database-test-XXXXXX")
+ );
+
+ Database db = new Geary.Db.Database.persistent(
+ tmp_dir.get_child("test.db")
+ );
+ db.open.begin(
+ Geary.Db.DatabaseFlags.CREATE_FILE, null, null,
+ (obj, ret) => { async_complete(ret); }
+ );
+ db.open.end(async_result());
+
+ // Need to get a connection since the database doesn't
+ // actually get created until then
+ db.get_master_connection();
+
+ db.file.delete();
+ tmp_dir.delete();
+ }
+
+ public void open_create_dir() throws Error {
+ GLib.File tmp_dir = GLib.File.new_for_path(
+ GLib.DirUtils.make_tmp("geary-db-database-test-XXXXXX")
+ );
+
+ Database db = new Geary.Db.Database.persistent(
+ tmp_dir.get_child("nonexistent").get_child("test.db")
+ );
+ db.open.begin(
+ Geary.Db.DatabaseFlags.CREATE_DIRECTORY |
+ Geary.Db.DatabaseFlags.CREATE_FILE,
+ null, null,
+ (obj, ret) => { async_complete(ret); }
+ );
+ db.open.end(async_result());
+
+ // Need to get a connection since the database doesn't
+ // actually get created until then
+ db.get_master_connection();
+
+ db.file.delete();
+ db.file.get_parent().delete();
+ tmp_dir.delete();
+ }
+
+ public void open_create_dir_existing() throws Error {
+ GLib.File tmp_dir = GLib.File.new_for_path(
+ GLib.DirUtils.make_tmp("geary-db-database-test-XXXXXX")
+ );
+
+ Database db = new Geary.Db.Database.persistent(
+ tmp_dir.get_child("test.db")
+ );
+ db.open.begin(
+ Geary.Db.DatabaseFlags.CREATE_DIRECTORY |
+ Geary.Db.DatabaseFlags.CREATE_FILE,
+ null, null,
+ (obj, ret) => { async_complete(ret); }
+ );
+ db.open.end(async_result());
+
+ // Need to get a connection since the database doesn't
+ // actually get created until then
+ db.get_master_connection();
+
+ db.file.delete();
+ tmp_dir.delete();
+ }
+
+
+}
diff --git a/test/engine/db/db-versioned-database-test.vala b/test/engine/db/db-versioned-database-test.vala
new file mode 100644
index 0000000..6d52abc
--- /dev/null
+++ b/test/engine/db/db-versioned-database-test.vala
@@ -0,0 +1,54 @@
+/*
+ * Copyright 2018 Michael Gratton <mike vee net>
+ *
+ * This software is licensed under the GNU Lesser General Public License
+ * (version 2.1 or later). See the COPYING file in this distribution.
+ */
+
+
+class Geary.Db.VersionedDatabaseTest : TestCase {
+
+
+ public VersionedDatabaseTest() {
+ base("Geary.Db.VersionedDatabaseTest");
+ add_test("open_new", open_new);
+ }
+
+ public void open_new() throws Error {
+ GLib.File tmp_dir = GLib.File.new_for_path(
+ GLib.DirUtils.make_tmp("geary-db-database-test-XXXXXX")
+ );
+
+ GLib.File sql1 = tmp_dir.get_child("version-001.sql");
+ sql1.create(
+ GLib.FileCreateFlags.NONE
+ ).write("CREATE TABLE TestTable (id INTEGER PRIMARY KEY, col TEXT);".data);
+
+ GLib.File sql2 = tmp_dir.get_child("version-002.sql");
+ sql2.create(
+ GLib.FileCreateFlags.NONE
+ ).write("INSERT INTO TestTable (col) VALUES ('value');".data);
+
+ VersionedDatabase db = new VersionedDatabase.persistent(
+ tmp_dir.get_child("test.db"), tmp_dir
+ );
+
+ db.open.begin(
+ Geary.Db.DatabaseFlags.CREATE_FILE, null, null,
+ (obj, ret) => { async_complete(ret); }
+ );
+ db.open.end(async_result());
+
+ Geary.Db.Result result = db.query("SELECT * FROM TestTable;");
+ assert_false(result.finished, "Row not inserted");
+ assert_string("value", result.string_for("col"));
+ assert_false(result.next(), "Multiple rows inserted");
+
+ db.file.delete();
+ sql1.delete();
+ sql2.delete();
+ tmp_dir.delete();
+ }
+
+
+}
diff --git a/test/engine/imap-db/imap-db-database-test.vala b/test/engine/imap-db/imap-db-database-test.vala
new file mode 100644
index 0000000..209694a
--- /dev/null
+++ b/test/engine/imap-db/imap-db-database-test.vala
@@ -0,0 +1,48 @@
+/*
+ * Copyright 2018 Michael Gratton <mike vee net>
+ *
+ * This software is licensed under the GNU Lesser General Public License
+ * (version 2.1 or later). See the COPYING file in this distribution.
+ */
+
+
+class Geary.ImapDB.DatabaseTest : TestCase {
+
+
+ public DatabaseTest() {
+ base("Geary.ImapDb.DatabaseTest");
+ add_test("open_new", open_new);
+ }
+
+ public void open_new() throws Error {
+ GLib.File tmp_dir = GLib.File.new_for_path(
+ GLib.DirUtils.make_tmp("geary-db-database-test-XXXXXX")
+ );
+
+ Database db = new Database(
+ tmp_dir,
+ GLib.File.new_for_path(_SOURCE_ROOT_DIR).get_child("sql"),
+ new Geary.SimpleProgressMonitor(Geary.ProgressType.DB_UPGRADE),
+ new Geary.SimpleProgressMonitor(Geary.ProgressType.DB_VACUUM),
+ "test example com"
+ );
+
+ db.open.begin(
+ Geary.Db.DatabaseFlags.CREATE_FILE, null,
+ (obj, ret) => { async_complete(ret); }
+ );
+ db.open.end(async_result());
+
+ // Need to get a connection since the database doesn't
+ // actually get created until then
+ db.get_master_connection();
+
+ // Need to close it again to stop the GC process running
+ db.close();
+
+ db.file.delete();
+ tmp_dir.delete();
+ }
+
+
+}
diff --git a/test/meson.build b/test/meson.build
index b844d0d..f2e5f44 100644
--- a/test/meson.build
+++ b/test/meson.build
@@ -22,9 +22,12 @@ geary_test_engine_sources = [
'engine/app/app-conversation-test.vala',
'engine/app/app-conversation-monitor-test.vala',
'engine/app/app-conversation-set-test.vala',
+ 'engine/db/db-database-test.vala',
+ 'engine/db/db-versioned-database-test.vala',
'engine/imap/command/imap-create-command-test.vala',
'engine/imap/response/imap-namespace-response-test.vala',
'engine/imap/transport/imap-deserializer-test.vala',
+ 'engine/imap-db/imap-db-database-test.vala',
'engine/imap-engine/account-processor-test.vala',
'engine/mime-content-type-test.vala',
'engine/rfc822-mailbox-address-test.vala',
diff --git a/test/test-engine.vala b/test/test-engine.vala
index ab4a0ef..071f781 100644
--- a/test/test-engine.vala
+++ b/test/test-engine.vala
@@ -30,10 +30,13 @@ int main(string[] args) {
engine.add_suite(new Geary.App.ConversationSetTest().get_suite());
// Depends on ConversationTest and ConversationSetTest passing
engine.add_suite(new Geary.App.ConversationMonitorTest().get_suite());
+ engine.add_suite(new Geary.Db.DatabaseTest().get_suite());
+ engine.add_suite(new Geary.Db.VersionedDatabaseTest().get_suite());
engine.add_suite(new Geary.HTML.UtilTest().get_suite());
engine.add_suite(new Geary.Imap.DeserializerTest().get_suite());
engine.add_suite(new Geary.Imap.CreateCommandTest().get_suite());
engine.add_suite(new Geary.Imap.NamespaceResponseTest().get_suite());
+ engine.add_suite(new Geary.ImapDB.DatabaseTest().get_suite());
engine.add_suite(new Geary.ImapEngine.AccountProcessorTest().get_suite());
engine.add_suite(new Geary.Inet.Test().get_suite());
engine.add_suite(new Geary.JS.Test().get_suite());
[
Date Prev][
Date Next] [
Thread Prev][
Thread Next]
[
Thread Index]
[
Date Index]
[
Author Index]
