BeSQLite.h

Go to the documentation of this file.

/*--------------------------------------------------------------------------------------+
|
|   Supplied under applicable software license agreement.
|
|   Copyright (c) 2018 Bentley Systems, Incorporated. All rights reserved.
|
+---------------------------------------------------------------------------------------*/
#pragma once
#include <Bentley/Bentley.h>
#include <Bentley/bset.h>
#if defined (SUPPORT_BRIEFCASE)
#include <Bentley/BeId.h>
#endif
#include <list>
#include <type_traits>

#ifndef NDEBUG
#include <Logging/bentleylogging.h>
#endif

/****
@page BeSQLiteOverview BeSQLite Overview

BeSQLite is a Bentley-authored layer to leverage and extend the unique features of SQLite
(http://www.sqlite.org/index.html). SQLite is a full-featured SQL database engine that can be embedded inside an
application. Unlike enterprise SQL database systems, SQLite has no client/server architecture, and a "SQLte Database" is
a disk file. This makes SQLite a wonderful solution for fast and flexible access to locally stored data, but
intrinsically limits its relevance for multi-user scenarios. In fact, there are very few scenarios where SQLite and
enterprise-level databases are interchangeable. Despite the fact that SQLite supports multiple connections to the same
database file from separate processes, BeSQLite is explicitly designed for use cases where the file is stored on a local
disk and is only accessed by a single process from a single connection.

It is important to recognize that while BeSQLite extends the capabilities of SQLite, <em>it is a strict superset of
SQLite</em> and a BeSQLite database file <b>is a</b> SQLite file. Therefore, all SQLite-based utilities and tools will
work on a BeSQLite file. Further, the SQL grammar, rules, and idiosyncrasies of SQLite apply equally to BeSQLite. To
find the answer to specific SQL-related questions, please consult the SQLite website at http://sqlite.org/lang.html. All
SQLite documentation is considered "incorporated by reference" in the BeSQLite documentation and is not repeated here.
Exceptions are clearly noted.

@section OVRBeSQLiteTransactions 1. Transactions in BeSQLite

BeSQLite's transactions are based on SQLite's transactions, and inherit the same guarantees of ACID compliance (see
http://sqlite.org/transactional.html). All changes to persistent data are made in the context of a SQLite transaction
and are not saved to the database unless and until committed. However, to help coordinate the synchronization of in-memory
application data with related persistent data, BeSQLite wraps the SQL- based transaction system with the SavePoint api.
Further, SQLite supports a concept called "implicit transactions" that is almost never desirable for a BeSQLite
application. Therefore, BeSQLite will start a "default transaction" unless explicitly directed otherwise. When
the default transaction is committed, it is automatically restarted.

<b>For this reason, it is important that you do not use any of the SQLite Transaction SQL statements BEGIN, END,
COMMIT, or ROLLBACK (http://www.sqlite.org/lang_transaction.html) directly in your code.</b> Doing so will cause
unpredictable results. Instead, use the methods #BeSQlite::Db::SaveChanges and #BeSQlite::Db::AbandonChanges.

Generally, the methods in BeSQLite do not do transaction management themselves, leaving the control of transaction to
the application. So, if you call method that changes persistent data, it may potentially make partial changes to the
data before encountering an error. When this happens, the active transaction can potentially hold inconsistent data
from the perspective of your application. When this happens, you may choose to abandon the entire transaction as appropriate
(note: changes are never permanently saved to disk until you call SaveChanges.)

@section OVRBeSQLiteRepositories 2. Repositories in BeSQLite

Many layers that use BeSQLite are designed to work with a single process with a single (exclusive) connection to a
database  file. That means that in a multi-user scenario, every user must have their own @i private copy of the
database. We refer to each user's copy of the database as a "briefcase" (following the convention of Mercurial and Git
for Distributed Revision Control, see http://en.wikipedia.org/wiki/Distributed_revision_control.) Each briefcase stores
its own 4-byte identifier, referred to as its @c BeBriefcaseId, that distinguishes it from all other repositories.
Changes made in each briefcase can then be merged together in an orderly fashion by coordinating with a team server.
Team servers are also responsible for assigning and tracking BeBriefcaseIds.

@section OVRBeSQLiteIds 3. Ids in BeSQLite

Since each user has a local copy of the briefcase, Ids in BeSQLite must be carefully managed. There are several
approaches:

    -# Use Globally Unique Identifiers (GUID, See #BeGuid). A globally unique identifier is a 16 byte randomly assigned
value. 16 bytes hold enough data that there is virtually no chance of collision. Unfortunately, 16 bytes will not fit
into registers of current generation computers, making them relatively expensive to handle programmatically. For large
tables, GUIDs can add significant overhead for lookups and are much more expensive than 8-byte values. This usually
makes them a <i>poor choice for primary keys.</i> However, as the name implies, when you use a GUID, you never need to
worry about uniqueness.

    -# Use #BeBriefcaseBasedIds. A BeBriefcaseBasedId is an 8-byte number that is designed to be unique by combining the
4-byte BeBriefcaseId with a unique-within-the-briefcase 4-byte value. This limits the number of potential values
within a BeBriefcase to 2^32 (4 billion). Of course it permits 4 billion different BeRepositories. It is often the best
compromise of performance and flexibility for tables with lots of activity.

    -# Use #BeServerIssuedId. A BeServerIssuedId is an 8-byte number that is unique because it is issued by a
synchronized id-administration server, enforced outside of the scope of BeSQLite. BeServerIssuedIds therefore cannot be
created locally and require a connection to the (one, globally administered) server to obtain a new value. Generally
they are useful for "rarely changed but often used" ids such as styles and fonts, etc.

@section OVRBeSQLiteProperties 4. Properties and Settings

Every BeSQlite database has a table named "be_Prop" that can be used to save name-id-value triplets, referred to as
Properties. Properties are often used for storing ad-hoc values that don't warrant the creation of a specific
class/table. The "name" for a Property is defined by a #PropertySpec that includes a two-part namespace/name to uniquely
identify the property. The id for a Property is also a two-part majorId/subId pair to permit arrays.

A PropertySpec has a flag that indicates that it is a "Setting." When Settings are changed, their value remains in
effect only for the duration of the session, unless an explicit call to #Db::SaveSettings is made. If the database is
closed without a call to Db::SaveSettings, the changes are not saved.

@section OVRBeSQLiteProperties 5. Briefcase Local Values

Every BeSQLite database has a table named "be_local" that can be used to save name-value pairs that are specific to the
BeBriefcase and not merged with the team server. They are used to keep track of the state of the local file and are not
considered part of the "real" database.

@section OVRBeSQLiteEmbeddedFiles 6. Embedded Files

Every BeSQLite database has a table named "be_EmbedFile" that holds copies of files that are "embedded in" the database.
These files are stored as blobs, and are not directly accessible by external applications. Instead, BeSQLite provides
methods to extract them into temporary locations.

@section OVRBeSQLiteLanguageSupport 7. Support for language-specific collation and case-folding

By default, BeSQLite does not support language-specific collation, and performs case-folding only for the ASCII
character set. However, applications can extend BeSQLite by implementing the #BeSQLiteLib::ILanguageSupport interface.

*/

#define BE_SQLITE_EXPORT IMPORT_ATTRIBUTE

#define TEMP_TABLE_Prefix "temp."
#define TEMP_TABLE(name) TEMP_TABLE_Prefix name

// the "unique temporary table" macros can be used for temporary tables that shadow a real table, but use a unique name.
#define TEMP_TABLE_UniquePrefix TEMP_TABLE_Prefix "t_"
#define TEMP_TABLE_UNIQUE(name) TEMP_TABLE_UniquePrefix name

#define BEDB_TABLE_Local        "be_Local"
#define BEDB_TABLE_Property     "be_Prop"
#define BEDB_TABLE_EmbeddedFile "be_EmbedFile"
#define BEDB_MemoryDb           ":memory:"

#define BEGIN_BENTLEY_SQLITE_NAMESPACE BEGIN_BENTLEY_NAMESPACE namespace BeSQLite {
#define END_BENTLEY_SQLITE_NAMESPACE   } END_BENTLEY_NAMESPACE
#define USING_NAMESPACE_BENTLEY_SQLITE using namespace Bentley::BeSQLite;

#include <Bentley/BeFile.h>
#include <Bentley/BeThread.h>
#include <Bentley/WString.h>
#include <Bentley/NonCopyableClass.h>
#include <Bentley/bvector.h>
#include <Bentley/bmap.h>
#include <Bentley/RefCounted.h>
#include <Bentley/DateTime.h>
#include <Bentley/BeVersion.h>


// this is used to quiet compiler warnings for variables only used in asserts
#define UNUSED_VARIABLE(x) (void)(x)

#define BESQLITE_TYPEDEFS(_name_) BEGIN_BENTLEY_SQLITE_NAMESPACE DEFINE_POINTER_SUFFIX_TYPEDEFS(_name_) END_BENTLEY_SQLITE_NAMESPACE

BESQLITE_TYPEDEFS(BeGuid);
BESQLITE_TYPEDEFS(Db);
BESQLITE_TYPEDEFS(DbFile);
BESQLITE_TYPEDEFS(Statement);
BESQLITE_TYPEDEFS(NamedParams);

#if !defined (DOCUMENTATION_GENERATOR)
#ifndef GUID_DEFINED
#define GUID_DEFINED
typedef struct _GUID
{
    uint32_t Data1;
    unsigned short Data2;
    unsigned short Data3;
    unsigned char  Data4[8];
} GUID;
#endif

typedef struct sqlite3_blob* SqlDbBlobP;
typedef struct sqlite3* SqlDbP;
typedef struct sqlite3& SqlDbR;
typedef struct sqlite3_stmt* SqlStatementP;
typedef struct sqlite3_session* SqlSessionP;
typedef struct sqlite3_changeset_iter* SqlChangesetIterP;
typedef struct Mem* SqlValueP;

#endif // DOCUMENTATION_GENERATOR

BEGIN_BENTLEY_SQLITE_NAMESPACE

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct BeGuid
{
    union{struct _GUID g; uint64_t u[2]; uint32_t i[4]; uint8_t b[16];} m_guid;

    explicit BeGuid(bool createValue=false) {if (createValue) Create(); else Invalidate();}

    BeGuid(uint64_t u0, uint64_t u1) {m_guid.u[0]=u0; m_guid.u[1]=u1;}

    BeGuid(uint32_t i0, uint32_t i1, uint32_t i2, uint32_t i3) {m_guid.i[0]=i0; m_guid.i[1]=i1; m_guid.i[2]=i2; m_guid.i[3]=i3;}

    bool operator==(BeGuid const& rhs) const {return rhs.m_guid.u[0]==m_guid.u[0] && rhs.m_guid.u[1]==m_guid.u[1];}

    bool operator!=(BeGuid const& rhs) const {return !(*this==rhs);}

    bool operator<(BeGuid const& rhs) const {return m_guid.u[0] < rhs.m_guid.u[0] || m_guid.u[0] == rhs.m_guid.u[0] && m_guid.u[1] < rhs.m_guid.u[1];}

    void Invalidate() {m_guid.u[0] = m_guid.u[1] = 0;}

    bool IsValid() const {return 0!=m_guid.u[0] && 0!=m_guid.u[1];}

    BE_SQLITE_EXPORT void Create();

    BE_SQLITE_EXPORT Utf8String ToString() const;

    BE_SQLITE_EXPORT BentleyStatus FromString(Utf8CP str);
};

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct BeBriefcaseId
{
protected:
    uint32_t m_id;

public:
    static uint32_t const MaxRepo() {return 1L<<24;} 
    static uint32_t const Master()  {return 0;} 
    static uint32_t const Standalone() {return 1;}
    static uint32_t const Illegal() {return (uint32_t)0xffffffff;}

    BeBriefcaseId GetNextBriefcaseId() const {return BeBriefcaseId(m_id+1);}
    BeBriefcaseId() {Invalidate();}             
    explicit BeBriefcaseId(uint32_t u) {m_id=u;} 
    void Invalidate() {m_id = Illegal();}  
    bool IsValid() const {return Illegal() != m_id;}  
    bool IsMasterId() const {return Master()==m_id;}  
    bool IsStandaloneId() const {return Standalone()==m_id;} 
    uint32_t GetValue() const {BeAssert(IsValid()); BeAssert(m_id<MaxRepo()); return m_id;} 
    bool operator==(BeBriefcaseId const& rhs) const {return rhs.m_id==m_id;}
    bool operator!=(BeBriefcaseId const& rhs) const {return !(*this == rhs);}
};

#if defined (SUPPORT_BRIEFCASE)
//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct BeBriefcaseBasedId : BeInt64Id
{
    BEINT64_ID_DECLARE_MEMBERS(BeBriefcaseBasedId,BeInt64Id)

public:
    static uint64_t const MaxLocal() {return 1LL<<40;} // top 24 bits are BeBriefcaseId, lower 40 bits are local id

    BeBriefcaseBasedId(BeBriefcaseId briefcaseId, uint64_t id) {BeAssert(id<MaxLocal()); m_id = ((briefcaseId.GetValue() * MaxLocal()) + id);}

    BeBriefcaseId GetBriefcaseId() const {return BeBriefcaseId((uint32_t) (m_id / MaxLocal()));} 

    BE_SQLITE_EXPORT void UseNext(Db&);

    BE_SQLITE_EXPORT BeBriefcaseBasedId(Db& db, Utf8CP tableName, Utf8CP columnName);
};

#define BEBRIEFCASEBASED_ID_SUBCLASS(classname,superclass) struct classname : superclass { \
    classname(BeSQLite::BeBriefcaseId briefcaseId, uint64_t id) : superclass(briefcaseId,id){} \
    classname(BeSQLite::Db& db, Utf8CP tableName, Utf8CP columnName) : superclass(db,tableName,columnName){} \
    BEINT64_ID_DECLARE_MEMBERS(classname,superclass) };

#define BEBRIEFCASEBASED_ID_CLASS(classname) BEBRIEFCASEBASED_ID_SUBCLASS(classname,BeSQLite::BeBriefcaseBasedId)

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct BeServerIssuedId : BeInt64Id
{
    BEINT64_ID_DECLARE_MEMBERS(BeServerIssuedId,BeInt64Id)
};

#define BESERVER_ISSUED_ID_SUBCLASS(classname,superclass) struct classname : superclass {BEINT64_ID_DECLARE_MEMBERS(classname,superclass)};
#define BESERVER_ISSUED_ID_CLASS(classname) BESERVER_ISSUED_ID_SUBCLASS(classname,BeSQLite::BeServerIssuedId)

#endif

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
enum DbConstants
{
    DbUserVersion           = 10,  
    NoCompressionLevel      = 0,   
    FastCompressionLevel    = 1,   
    DefaultCompressionLevel = 3,   
    MaxCompressionLevel     = 9,   
};

//=======================================================================================
// The profile version for BeSQLite database files created by this version of the BeSQLite library
// @bsiclass                                                     Bentley Systems
//=======================================================================================
enum DbProfileValues
    {
    BEDB_CURRENT_VERSION_Major = 3,
    BEDB_CURRENT_VERSION_Minor = 1,
    BEDB_CURRENT_VERSION_Sub1  = 0,
    BEDB_CURRENT_VERSION_Sub2  = 1,

    BEDB_SUPPORTED_VERSION_Major = BEDB_CURRENT_VERSION_Major,  // oldest version of the db schema supported by current api
    BEDB_SUPPORTED_VERSION_Minor = 0,
    BEDB_SUPPORTED_VERSION_Sub1  = 0,
    BEDB_SUPPORTED_VERSION_Sub2  = 0,
    };

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct ProfileVersion : BeVersion
{
public:
    ProfileVersion(uint16_t major, uint16_t minor, uint16_t sub1, uint16_t sub2) : BeVersion(major, minor, sub1, sub2) {}
    explicit ProfileVersion(Utf8CP json) {FromJson(json);}
    BE_SQLITE_EXPORT Utf8String ToJson() const;
    BE_SQLITE_EXPORT void FromJson(Utf8CP);
};

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
enum DbResult
{
    BE_SQLITE_OK          = 0,    
    BE_SQLITE_ERROR       = 1,    
    BE_SQLITE_INTERNAL    = 2,    
    BE_SQLITE_PERM        = 3,    
    BE_SQLITE_ABORT       = 4,    
    BE_SQLITE_BUSY        = 5,    
    BE_SQLITE_LOCKED      = 6,    
    BE_SQLITE_NOMEM       = 7,    
    BE_SQLITE_READONLY    = 8,    
    BE_SQLITE_INTERRUPT   = 9,    
    BE_SQLITE_IOERR       = 10,   
    BE_SQLITE_CORRUPT     = 11,   
    BE_SQLITE_NOTFOUND    = 12,   
    BE_SQLITE_FULL        = 13,   
    BE_SQLITE_CANTOPEN    = 14,   
    BE_SQLITE_PROTOCOL    = 15,   
    BE_SQLITE_EMPTY       = 16,   
    BE_SQLITE_SCHEMA      = 17,   
    BE_SQLITE_TOOBIG      = 18,   
    BE_SQLITE_CONSTRAINT_BASE  = 19,   
    BE_SQLITE_MISMATCH    = 20,   
    BE_SQLITE_MISUSE      = 21,   
    BE_SQLITE_NOLFS       = 22,   
    BE_SQLITE_AUTH        = 23,   
    BE_SQLITE_FORMAT      = 24,   
    BE_SQLITE_RANGE       = 25,   
    BE_SQLITE_NOTADB      = 26,   
    BE_SQLITE_ROW         = 100,  
    BE_SQLITE_DONE        = 101,  

    BE_SQLITE_IOERR_READ              = (BE_SQLITE_IOERR    | (1<<8)),
    BE_SQLITE_IOERR_SHORT_READ        = (BE_SQLITE_IOERR    | (2<<8)),
    BE_SQLITE_IOERR_WRITE             = (BE_SQLITE_IOERR    | (3<<8)),
    BE_SQLITE_IOERR_FSYNC             = (BE_SQLITE_IOERR    | (4<<8)),
    BE_SQLITE_IOERR_DIR_FSYNC         = (BE_SQLITE_IOERR    | (5<<8)),
    BE_SQLITE_IOERR_TRUNCATE          = (BE_SQLITE_IOERR    | (6<<8)),
    BE_SQLITE_IOERR_FSTAT             = (BE_SQLITE_IOERR    | (7<<8)),
    BE_SQLITE_IOERR_UNLOCK            = (BE_SQLITE_IOERR    | (8<<8)),
    BE_SQLITE_IOERR_RDLOCK            = (BE_SQLITE_IOERR    | (9<<8)),
    BE_SQLITE_IOERR_DELETE            = (BE_SQLITE_IOERR    | (10<<8)),
    BE_SQLITE_IOERR_BLOCKED           = (BE_SQLITE_IOERR    | (11<<8)),
    BE_SQLITE_IOERR_NOMEM             = (BE_SQLITE_IOERR    | (12<<8)),
    BE_SQLITE_IOERR_ACCESS            = (BE_SQLITE_IOERR    | (13<<8)),
    BE_SQLITE_IOERR_CHECKRESERVEDLOCK = (BE_SQLITE_IOERR    | (14<<8)),
    BE_SQLITE_IOERR_LOCK              = (BE_SQLITE_IOERR    | (15<<8)),
    BE_SQLITE_IOERR_CLOSE             = (BE_SQLITE_IOERR    | (16<<8)),
    BE_SQLITE_IOERR_DIR_CLOSE         = (BE_SQLITE_IOERR    | (17<<8)),
    BE_SQLITE_IOERR_SHMOPEN           = (BE_SQLITE_IOERR    | (18<<8)),
    BE_SQLITE_IOERR_SHMSIZE           = (BE_SQLITE_IOERR    | (19<<8)),
    BE_SQLITE_IOERR_SHMLOCK           = (BE_SQLITE_IOERR    | (20<<8)),
    BE_SQLITE_IOERR_SHMMAP            = (BE_SQLITE_IOERR    | (21<<8)),
    BE_SQLITE_IOERR_SEEK              = (BE_SQLITE_IOERR    | (22<<8)),
    BE_SQLITE_IOERR_DELETE_NOENT      = (BE_SQLITE_IOERR    | (23<<8)),

    BE_SQLITE_ERROR_FileExists        = (BE_SQLITE_IOERR | (1<<24)),  
    BE_SQLITE_ERROR_AlreadyOpen       = (BE_SQLITE_IOERR | (2<<24)),  
    BE_SQLITE_ERROR_NoPropertyTable   = (BE_SQLITE_IOERR | (3<<24)),  
    BE_SQLITE_ERROR_FileNotFound      = (BE_SQLITE_IOERR | (4<<24)),  
    BE_SQLITE_ERROR_NoTxnActive       = (BE_SQLITE_IOERR | (5<<24)),  
    BE_SQLITE_ERROR_BadDbProfile      = (BE_SQLITE_IOERR | (6 << 24)), 
    BE_SQLITE_ERROR_InvalidProfileVersion = (BE_SQLITE_IOERR | (7<<24)),  
    BE_SQLITE_ERROR_ProfileUpgradeFailed = (BE_SQLITE_IOERR | (8<<24)),  
    BE_SQLITE_ERROR_ProfileUpgradeFailedCannotOpenForWrite = (BE_SQLITE_IOERR | (9<<24)),  
    BE_SQLITE_ERROR_ProfileTooOld     = (BE_SQLITE_IOERR | (10<<24)),  
    BE_SQLITE_ERROR_ProfileTooNewForReadWrite = (BE_SQLITE_IOERR | (11<<24)),  
    BE_SQLITE_ERROR_ProfileTooNew     = (BE_SQLITE_IOERR | (12<<24)),  
    BE_SQLITE_ERROR_ChangeTrackError  = (BE_SQLITE_IOERR | (13<<24)),  
    BE_SQLITE_ERROR_InvalidRevisionVersion = (BE_SQLITE_IOERR | (14 << 24)), 
    
    BE_SQLITE_ERROR_SchemaReadFailed        = (BE_SQLITE_IOERR | 15 << 24), 
    BE_SQLITE_ERROR_SchemaNotFound          = (BE_SQLITE_ERROR | 16 << 24), 
    BE_SQLITE_ERROR_SchemaTooNew            = (BE_SQLITE_IOERR | 16 << 24), 
    BE_SQLITE_ERROR_SchemaTooOld            = (BE_SQLITE_IOERR | 17 << 24), 
        BE_SQLITE_ERROR_SchemaImportRequired    = (BE_SQLITE_IOERR | 18 << 24), 
    BE_SQLITE_ERROR_SchemaLockFailed        = (BE_SQLITE_IOERR | 19 << 24), 
    BE_SQLITE_ERROR_SchemaImportFailed      = (BE_SQLITE_IOERR | 20 << 24), 
    BE_SQLITE_ERROR_SchemaDomainMismatch    = (BE_SQLITE_IOERR | 21 << 24), 

    BE_SQLITE_LOCKED_SHAREDCACHE      = (BE_SQLITE_LOCKED   | (1<<8)),

    BE_SQLITE_BUSY_RECOVERY           = (BE_SQLITE_BUSY     | (1<<8)),

    BE_SQLITE_CANTOPEN_NOTEMPDIR      = (BE_SQLITE_CANTOPEN | (1<<8)),
    BE_SQLITE_CANTOPEN_ISDIR          = (BE_SQLITE_CANTOPEN | (2<<8)),
    BE_SQLITE_CANTOPEN_FULLPATH       = (BE_SQLITE_CANTOPEN | (3<<8)),

    BE_SQLITE_CORRUPT_VTAB            = (BE_SQLITE_CORRUPT | (1<<8)),

    BE_SQLITE_READONLY_RECOVERY       = (BE_SQLITE_READONLY | (1<<8)),
    BE_SQLITE_READONLY_CANTLOCK       = (BE_SQLITE_READONLY | (2<<8)),
    BE_SQLITE_READONLY_ROLLBACK       = (BE_SQLITE_READONLY | (3<<8)),

    BE_SQLITE_ABORT_ROLLBACK          = (BE_SQLITE_ABORT | (2<<8)),

    BE_SQLITE_CONSTRAINT_CHECK        = (BE_SQLITE_CONSTRAINT_BASE | (1<<8)),
    BE_SQLITE_CONSTRAINT_COMMITHOOK   = (BE_SQLITE_CONSTRAINT_BASE | (2<<8)),
    BE_SQLITE_CONSTRAINT_FOREIGNKEY   = (BE_SQLITE_CONSTRAINT_BASE | (3<<8)),
    BE_SQLITE_CONSTRAINT_FUNCTION     = (BE_SQLITE_CONSTRAINT_BASE | (4<<8)),
    BE_SQLITE_CONSTRAINT_NOTNULL      = (BE_SQLITE_CONSTRAINT_BASE | (5<<8)),
    BE_SQLITE_CONSTRAINT_PRIMARYKEY   = (BE_SQLITE_CONSTRAINT_BASE | (6<<8)),
    BE_SQLITE_CONSTRAINT_TRIGGER      = (BE_SQLITE_CONSTRAINT_BASE | (7<<8)),
    BE_SQLITE_CONSTRAINT_UNIQUE       = (BE_SQLITE_CONSTRAINT_BASE | (8<<8)),
    BE_SQLITE_CONSTRAINT_VTAB         = (BE_SQLITE_CONSTRAINT_BASE | (9<<8)),
};

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
enum class DbOpcode : int
{
    Delete  = 9,
    Insert  = 18,
    Update  = 23,
};

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
enum class DbValueType : int
{
    IntegerVal  = 1,
    FloatVal    = 2,
    TextVal     = 3,
    BlobVal     = 4,
    NullVal     = 5,
};

//=======================================================================================
// Utility class to initialize the BeSQLite library and directly use a few SQLite APIs.
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct BeSQLiteLib
{
public:
    //=======================================================================================
    // @bsiclass                                                     Bentley Systems
    //=======================================================================================
    struct ILanguageSupport
    {
        typedef void(*CollationUserDataFreeFunc)(void*);

        struct CollationEntry
        {
            AString m_name;     
            void* m_collator;   
        };

        virtual void _Lower(Utf16CP source, int sourceLen, Utf16CP result, int resultLen) = 0;

        virtual void _Upper(Utf16CP source, int sourceLen, Utf16CP result, int resultLen) = 0;

        virtual void _InitCollation(bvector<CollationEntry>& collations, CollationUserDataFreeFunc& collatorFreeFunc) = 0;

        virtual int _Collate(Utf16CP lhs, int lhsLen, Utf16CP rhs, int rhsLen, void* collator) = 0;

        virtual uint32_t _FoldCase(uint32_t) = 0;
    };

    enum class LogErrors : bool {Yes=1, No=0};

    BE_SQLITE_EXPORT static DbResult Initialize(BeFileNameCR tempDir, LogErrors wantLogging=LogErrors::No);

    BE_SQLITE_EXPORT static void Randomness(int numBytes, void* random);

    static void* MallocMem(int sz);

    static void* ReallocMem(void* p, int sz);

    static void FreeMem(void* p);

    BE_SQLITE_EXPORT static int CloseSqlDb(void* p);

    BE_SQLITE_EXPORT static void SetDownloadAdmin(struct IDownloadAdmin&);

    BE_SQLITE_EXPORT static struct IDownloadAdmin* GetDownloadAdmin();

    BE_SQLITE_EXPORT static void SetLanguageSupport(ILanguageSupport*);

    BE_SQLITE_EXPORT static ILanguageSupport* GetLanguageSupport();

    static int GetBaseDbResult(DbResult val) {return 0xff & val;}
    static bool TestBaseDbResult(DbResult val1, DbResult val2) {return GetBaseDbResult(val1) == GetBaseDbResult(val2);}
    static bool IsConstraintDbResult(DbResult val1) {return GetBaseDbResult(val1) == BE_SQLITE_CONSTRAINT_BASE;}
};

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct Statement : NonCopyableClass
{
private:
    SqlStatementP m_stmt;

    DbResult DoPrepare(DbFileCR, Utf8CP sql);

public:
    enum class MakeCopy : bool {No=0, Yes=1};
    explicit Statement(SqlStatementP stmt) {m_stmt=stmt;}

    Statement() {m_stmt=nullptr;}
    Statement(DbCR db, Utf8CP sql) {m_stmt=nullptr; Prepare(db, sql);}
    ~Statement() {Finalize();}

    SqlStatementP& GetStmtR() {return m_stmt;} 
    DbResult Prepare(DbFileCR, Utf8CP sql, bool suppressDiagnostics = false); 

    bool IsPrepared() const {return nullptr != m_stmt;}

    BE_SQLITE_EXPORT void Finalize();

    BE_SQLITE_EXPORT DbResult Prepare(DbCR db, Utf8CP sql);

    BE_SQLITE_EXPORT DbResult TryPrepare(DbCR db, Utf8CP sql);

    BE_SQLITE_EXPORT DbResult Step();

    BE_SQLITE_EXPORT DbResult Reset();

    BE_SQLITE_EXPORT DbResult ClearBindings();

    BE_SQLITE_EXPORT DbResult BindInt(int paramNum, int value);

    BE_SQLITE_EXPORT DbResult BindInt64(int paramNum, int64_t value);

    DbResult BindUInt64(int paramNum, uint64_t value) {return BindInt64(paramNum, (int64_t) value);}

    DbResult BindBoolean(int paramNum, bool value) {return BindInt(paramNum, value ? 1 : 0);}

#if defined (SUPPORT_BRIEFCASE)
    DbResult BindId(int paramNum, BeInt64Id value) {return value.IsValid() ? BindUInt64(paramNum,value.GetValue()) : BindNull(paramNum);}
#endif

    BE_SQLITE_EXPORT DbResult BindDouble(int paramNum, double value);

    DbResult BindText(int paramNum, Utf8StringCR stringValue, MakeCopy makeCopy) {return BindText(paramNum, stringValue.c_str(), makeCopy, (int)stringValue.size());}

    BE_SQLITE_EXPORT DbResult BindText(int paramNum, Utf8CP stringValue, MakeCopy makeCopy, int nBytes=-1);

    BE_SQLITE_EXPORT DbResult BindGuid(int paramNum, BeGuidCR value);

    BE_SQLITE_EXPORT DbResult BindZeroBlob(int paramNum, int size);

    BE_SQLITE_EXPORT DbResult BindBlob(int paramNum, void const* blobValue, int size, MakeCopy makeCopy);

    BE_SQLITE_EXPORT DbResult BindNull(int paramNum);

    BE_SQLITE_EXPORT DbResult BindVirtualSet(int paramNum, struct VirtualSet const& vSet);

    BE_SQLITE_EXPORT DbResult BindDbValue(int paramNum, struct DbValue const& dbVal);

    BE_SQLITE_EXPORT int GetColumnCount();

    BE_SQLITE_EXPORT DbValueType GetColumnType(int col);

    BE_SQLITE_EXPORT Utf8CP GetColumnTableName(int col);

    BE_SQLITE_EXPORT Utf8CP GetColumnDeclaredType(int col);

    bool IsColumnNull(int col) {return DbValueType::NullVal == GetColumnType(col);}

    BE_SQLITE_EXPORT Utf8CP GetColumnName(int col);

    BE_SQLITE_EXPORT int GetColumnBytes(int col);

    BE_SQLITE_EXPORT int GetColumnBytes16(int col);

    BE_SQLITE_EXPORT void const* GetValueBlob(int col);

    BE_SQLITE_EXPORT Utf8CP GetValueText(int col);

    BE_SQLITE_EXPORT int GetValueInt(int col);

    BE_SQLITE_EXPORT int64_t GetValueInt64(int col);

    uint64_t GetValueUInt64(int col) {return (uint64_t) GetValueInt64(col);}

    bool GetValueBoolean(int col) {return GetValueInt(col) != 0;}

    BE_SQLITE_EXPORT double GetValueDouble(int col);

    template <class T_Id> T_Id GetValueId(int col) {if (!IsColumnNull(col)) {return T_Id(GetValueUInt64(col));} return T_Id();}

    BE_SQLITE_EXPORT BeGuid GetValueGuid(int col);

    BE_SQLITE_EXPORT struct DbDupValue GetDbValue(int col);

    BE_SQLITE_EXPORT int GetParameterIndex(Utf8CP name);

    BE_SQLITE_EXPORT int GetParameterCount();

    BE_SQLITE_EXPORT Utf8CP GetSql() const;

    BE_SQLITE_EXPORT void DumpResults();

    SqlStatementP GetSqlStatementP() const {return m_stmt;}  // for direct use of sqlite3 api
    operator SqlStatementP(){return m_stmt;}                 // for direct use of sqlite3 api
};

#define DIAGNOSTICS_PREPARE_LOGGER_NAME L"Diagnostics.BeSQLite.Prepare"
#define DIAGNOSTICS_QUERYPLAN_LOGGER_NAME L"Diagnostics.BeSQLite.QueryPlan"
#define DIAGNOSTICS_QUERYPLANWITHTABLESCANS_LOGGER_NAME L"Diagnostics.BeSQLite.QueryPlanWithTableScans"

#ifdef NDEBUG
#define STATEMENT_DIAGNOSTICS_ON
#define STATEMENT_DIAGNOSTICS_OFF
#define STATEMENT_DIAGNOSTICS_LOGCOMMENT(comment)
#else
#define STATEMENT_DIAGNOSTICS_ON StatementDiagnostics::SetIsEnabled(true);
#define STATEMENT_DIAGNOSTICS_OFF StatementDiagnostics::SetIsEnabled(false);
#define STATEMENT_DIAGNOSTICS_LOGCOMMENT(comment) StatementDiagnostics::LogComment(comment)

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct StatementDiagnostics
    {
private:
    StatementDiagnostics();

public:
    BE_SQLITE_EXPORT static void SetIsEnabled(bool isEnabled);

    BE_SQLITE_EXPORT static void LogComment(Utf8CP comment);
    };
#endif

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct BlobIO
{
private:
    SqlDbBlobP m_blob;

public:
    SqlDbBlobP GetBlobP() {return m_blob;}
    BlobIO() {m_blob=0;}
    ~BlobIO() {Close();}

    BE_SQLITE_EXPORT DbResult Open(DbR db, Utf8CP tableName, Utf8CP columnName, uint64_t row, bool writable, Utf8CP dbName=0);

    BE_SQLITE_EXPORT DbResult ReOpen(uint64_t row);

    BE_SQLITE_EXPORT DbResult Close();

    BE_SQLITE_EXPORT DbResult Read(void* data, int numBytes, int offset);

    BE_SQLITE_EXPORT DbResult Write(const void* data, int numBytes, int offset);

    BE_SQLITE_EXPORT int GetNumBytes() const;

    bool IsValid() const {return nullptr != m_blob;}
};

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct DbValue
{
protected:
    SqlValueP m_val;

public:
    DbValue(SqlValueP val) : m_val(val) {}

    bool IsValid() const {return nullptr != m_val;}                    
    bool IsNull()  const {return DbValueType::NullVal == GetValueType();} 
    SqlValueP GetSqlValueP() const {return m_val;}  

    BE_SQLITE_EXPORT DbValueType GetValueType() const;      
    BE_SQLITE_EXPORT DbValueType GetNumericType() const;    
    BE_SQLITE_EXPORT int         GetValueBytes() const;     
    BE_SQLITE_EXPORT void const* GetValueBlob() const;      
    BE_SQLITE_EXPORT Utf8CP      GetValueText() const;      
    BE_SQLITE_EXPORT int         GetValueInt() const;       
    BE_SQLITE_EXPORT int64_t     GetValueInt64() const;     
    uint64_t GetValueUInt64() const {return (uint64_t) GetValueInt64();}
    BE_SQLITE_EXPORT double      GetValueDouble() const;    
    BE_SQLITE_EXPORT BeGuid      GetValueGuid() const;      
    template <class T_Id> T_Id   GetValueId() const {return T_Id(GetValueUInt64());}

    BE_SQLITE_EXPORT Utf8String Format(int detailLevel) const; 
};

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct DbDupValue : DbValue, NonCopyableClass
{
    BE_SQLITE_EXPORT DbDupValue(SqlValueP val);
    DbDupValue(DbDupValue&& other) : DbValue(other.m_val) {other.m_val = nullptr;}
    BE_SQLITE_EXPORT DbDupValue& operator=(DbDupValue&& other);
    BE_SQLITE_EXPORT ~DbDupValue();
};

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct DbFunction : NonCopyableClass
{
public:
    //=======================================================================================
    // @bsiclass                                                     Bentley Systems
    //=======================================================================================
    struct Context
        {
        enum class CopyData : int {No = 0, Yes = -1};                                     
        BE_SQLITE_EXPORT void SetResultBlob(void const* value, int length, CopyData copy=CopyData::Yes); 
        BE_SQLITE_EXPORT void SetResultDouble(double);                                
        BE_SQLITE_EXPORT void SetResultError(Utf8CP, int len=-1);                     
        BE_SQLITE_EXPORT void SetResultError_toobig();                                
        BE_SQLITE_EXPORT void SetResultError_nomem();                                 
        BE_SQLITE_EXPORT void SetResultError_code(int);                               
        BE_SQLITE_EXPORT void SetResultInt(int);                                      
        BE_SQLITE_EXPORT void SetResultInt64(int64_t);                                
        BE_SQLITE_EXPORT void SetResultNull();                                        
        BE_SQLITE_EXPORT void SetResultText(Utf8CP value, int length, CopyData);      
        BE_SQLITE_EXPORT void SetResultZeroblob(int length);                          
        BE_SQLITE_EXPORT void SetResultValue(DbValue);                                
        BE_SQLITE_EXPORT void* GetAggregateContext(int nbytes);
        };

private:
    Utf8String  m_name;
    int         m_nArgs;
    DbValueType m_returnType;

protected:
    DbFunction(Utf8CP name, int nArgs, DbValueType returnType) : m_name(name), m_nArgs(nArgs), m_returnType(returnType) {}

public:
    virtual bool _IsAggregate() {return false;}
    virtual ~DbFunction() {}
    Utf8CP GetName() const {return m_name.c_str();} 
    int GetNumArgs() const {return m_nArgs;}    
    DbValueType GetReturnType() const {return m_returnType;}

    void SetInputError(Context& ctx) {ctx.SetResultError(Utf8PrintfString("Illegal input to %s", GetName()).c_str());}
};

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct ScalarFunction : DbFunction
{
    virtual void _ComputeScalar(Context&, int nArgs, DbValue* args) = 0;   //<! see "xFunc" in sqlite3_create_function

    ScalarFunction(Utf8CP name, int nArgs, DbValueType returnType = DbValueType::NullVal) : DbFunction(name, nArgs, returnType) {}
};
//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct AggregateFunction : DbFunction
{
    bool _IsAggregate() override {return true;}
    virtual void _StepAggregate(Context&, int nArgs, DbValue* args) = 0; //<! see "xStep" in sqlite3_create_function
    virtual void _FinishAggregate(Context&) = 0;                         //<! see "xFinal" in sqlite3_create_function

    AggregateFunction(Utf8CP name, int nArgs, DbValueType returnType = DbValueType::NullVal) : DbFunction(name, nArgs, returnType) {}
};

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct RTreeMatchFunction : DbFunction
{
    enum class Within : int
    {
        Outside = 0,
        Partly = 1,
        Inside = 2,
    };

    //=======================================================================================
    // @bsiclass                                                     Bentley Systems
    //=======================================================================================
    struct QueryInfo
    {
        void*   m_context;
        int     m_nParam;
        double* m_param;
        void*   m_user;
        void    (*m_xDelUser)(void*);
        double* m_coords;
        unsigned int* m_nQueue;
        int     m_nCoord;
        int     m_level;
        int     m_maxLevel;
        int64_t m_rowid;
        double  m_parentScore;
        Within  m_parentWithin;
        mutable Within m_within;
        mutable double m_score;
        DbValue* m_args;       // SQL values of parameters
    };

    virtual int _TestRange(QueryInfo const&) = 0;

    RTreeMatchFunction(Utf8CP name, int nArgs) : DbFunction(name, nArgs, DbValueType::NullVal) {}
};

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct VirtualSet
{
    virtual bool _IsInSet(int nVals, DbValue const* vals) const = 0;
};

#if defined (SUPPORT_BRIEFCASE)
//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct BeIdSet : bset<BeInt64Id>
{
private:
    Utf8String ToReadableString() const;
    void FromCompactString(Utf8StringCR);
    void FromReadableString(Utf8StringCR);
public:
    enum class StringFormat {Compact=0, Readable=1};

    BE_SQLITE_EXPORT void FromString(Utf8StringCR in);
    BE_SQLITE_EXPORT Utf8String ToString() const;
    BE_SQLITE_EXPORT Utf8String ToCompactString() const;    // NB: This method is NOT backwards-compatible!
};

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
template<typename IdType> struct IdSet : VirtualSet
{
private:
    BeIdSet m_set;

    virtual bool _IsInSet(int nVals, DbValue const* vals) const override
        {
        BeAssert(nVals == 1);
        return Contains(IdType(vals[0].GetValueUInt64()));
        }
public:
    IdSet() {static_assert(std::is_base_of<BeInt64Id, IdType>::value && sizeof(BeInt64Id) == sizeof(IdType), "IdSet may only contain BeInt64Ids or subclasses of it of the same size.");}
    virtual ~IdSet() {}

    typedef IdSet<IdType> T_Self;
    typedef bset<IdType> T_SetType;
    typedef typename T_SetType::const_iterator const_iterator;
    typedef typename T_SetType::iterator iterator;

    const_iterator begin() const {return ((T_SetType&)m_set).begin();}
    const_iterator end() const {return ((T_SetType&)m_set).end();}
    const_iterator find(IdType id) const {return ((T_SetType&)m_set).find(id);}
    bool operator==(T_Self const& other) const {return m_set==other.m_set;}
    bool operator!=(T_Self const& other) const {return m_set!=other.m_set;}
    bool empty() const {return m_set.empty();}
    void clear() {m_set.clear();}
    size_t size() const {return m_set.size();}
    bpair<iterator,bool> insert(IdType const& val) {BeAssert(val.IsValid()); return ((T_SetType&)m_set).insert(val);}
    void insert(const_iterator first, const_iterator last) {((T_SetType&)m_set).insert(first,last);}
    size_t erase(IdType const& val) {return ((T_SetType&)m_set).erase(val);}
    iterator erase(iterator it) {return ((T_SetType&)m_set).erase(it);}
    bool Contains(IdType id) const {return end() != find(id);}
    void FromString(Utf8StringCR in) {m_set.FromString(in);}
    Utf8String ToString() const {return m_set.ToString();}

    BeIdSet const& GetBeIdSet() const {return m_set;}
};
#endif

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct SqlPrintfString
{
private:
    Utf8P m_str;
public:
    BE_SQLITE_EXPORT SqlPrintfString(Utf8CP fmt, ...);
    BE_SQLITE_EXPORT ~SqlPrintfString();
    operator Utf8CP(){return m_str;}
    Utf8CP GetUtf8CP() {return m_str;}
};

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct BeDbMutex : NonCopyableClass
{
private:
    void*  m_mux;
public:
    enum class MutexType : bool {Fast=0, Recursive=1};
    BE_SQLITE_EXPORT BeDbMutex(MutexType mutexType=MutexType::Fast);       
    BE_SQLITE_EXPORT ~BeDbMutex();      
    BE_SQLITE_EXPORT void Enter();      
    BE_SQLITE_EXPORT void Leave();      
#ifndef NDEBUG
    BE_SQLITE_EXPORT bool IsHeld();
#endif
};

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct BeDbMutexHolder : NonCopyableClass
{
    BeDbMutex& m_mutex;
    BeDbMutexHolder(BeDbMutex& mutex) : m_mutex(mutex) {m_mutex.Enter();}
    ~BeDbMutexHolder() {m_mutex.Leave();}
};

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct CachedStatement : Statement
{
    friend struct StatementCache;
private:
    mutable BeAtomic<uint32_t> m_refCount;
    bool m_inCache;
    struct StatementCache const& m_myCache;
    Utf8CP  m_sql;
    CachedStatement(Utf8CP sql, struct StatementCache const&);
    ~CachedStatement();

public:
    DEFINE_BENTLEY_NEW_DELETE_OPERATORS

    uint32_t AddRef() const {return m_refCount.IncrementAtomicPre();} 
    uint32_t GetRefCount() const {return m_refCount.load();}
    BE_SQLITE_EXPORT uint32_t Release();
    Utf8CP GetSQL() const {return m_sql;}
};

typedef RefCountedPtr<CachedStatement> CachedStatementPtr;

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct StatementCache : NonCopyableClass
{
    friend struct CachedStatement;
private:
    typedef std::list<CachedStatementPtr> Entries;
    mutable BeMutex m_mutex;
    mutable Entries m_entries;
    uint32_t m_size;
    Entries::iterator FindEntry(Utf8CP) const;
    BE_SQLITE_EXPORT void AddStatement(CachedStatementPtr& newEntry, Utf8CP sql) const;
    BE_SQLITE_EXPORT void FindStatement(CachedStatementPtr&, Utf8CP) const;

public:
    StatementCache(uint32_t size) {m_size=size;}
    ~StatementCache() {Empty();}

    BE_SQLITE_EXPORT DbResult GetPreparedStatement(CachedStatementPtr&, DbFile const& dbFile, Utf8CP sqlString) const;
    BE_SQLITE_EXPORT void Dump() const;
    BE_SQLITE_EXPORT void Empty();
    bool IsEmpty() const {return m_entries.empty();}
};

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct NamedParams
{
public:
    struct SqlParameter : RefCountedBase
    {
        Utf8String m_name;
        SqlParameter(Utf8CP name) : m_name(name) {}
        virtual void _Bind(Statement&) = 0;
    };

private:
    typedef RefCountedPtr<SqlParameter> SqlParameterPtr;
    Utf8String m_where;
    bvector<SqlParameterPtr>  m_params;

public:
    void AddSqlParameter(SqlParameter* param) {m_params.push_back(param);}

    BE_SQLITE_EXPORT void AddStringParameter(Utf8CP name, Utf8CP val);

    BE_SQLITE_EXPORT void AddIntegerParameter(Utf8CP name, uint64_t val);

    BE_SQLITE_EXPORT void AddDoubleParameter(Utf8CP name, double val);

    BE_SQLITE_EXPORT void AddBlobParameter(Utf8CP name, void const* data, int size, Statement::MakeCopy copy);

    NamedParams(Utf8CP where=nullptr) {SetWhere(where);}

    BE_SQLITE_EXPORT void SetWhere(Utf8CP where);

    Utf8StringCR GetWhere() const {return m_where;}

    BE_SQLITE_EXPORT void Bind(Statement& stmt) const;
};

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct DbTableIterator  : NonCopyableClass
{
protected:
    mutable DbP m_db;
    mutable CachedStatementPtr m_stmt;
    NamedParams m_params;

    explicit DbTableIterator(DbCR db) : m_db((DbP)&db) {}

    DbTableIterator(DbTableIterator&& rhs) : m_db(rhs.m_db), m_stmt(std::move(rhs.m_stmt)), m_params(rhs.m_params) {}

    BE_SQLITE_EXPORT Utf8String MakeSqlString(Utf8CP sql, bool hasWhere=false) const;

public:
    struct Entry
    {
    protected:
        bool  m_isValid;
        StatementP m_sql;
        Entry(StatementP sql, bool isValid) {m_sql=sql; m_isValid=isValid;}
        void Verify() const {BeAssert(nullptr != m_sql->GetSqlStatementP());}

    public:
        bool IsValid() const {return m_isValid && (nullptr!=m_sql->GetSqlStatementP());}
        void Invalidate() {m_isValid=false;}

        Entry& operator++() {m_isValid=(BeSQLite::BE_SQLITE_ROW == m_sql->Step()); return *this;}
        Entry const& operator* () const {return *this;}
        bool operator!=(Entry const& rhs) const {return (m_isValid != rhs.m_isValid);}
        bool operator==(Entry const& rhs) const {return (m_isValid == rhs.m_isValid);}
    };

    Statement* GetStatement() {return m_stmt.get();}

    NamedParams& Params() {return m_params;}
};

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct ICompressProgressTracker
{
    virtual StatusInt _Progress(uint64_t inSize, int64_t outSize) = 0;
};

#if defined (SUPPORT_EMBEDDED_FILE)
//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct DbEmbeddedFileTable
{
private:
    friend struct Db;
    DbR m_db;
    DbEmbeddedFileTable(DbR db) : m_db(db) {}

    BeBriefcaseBasedId GetNextEmbedFileId() const; 

public:
    //=======================================================================================
    // @bsiclass                                                     Bentley Systems
    //=======================================================================================
    struct Iterator : DbTableIterator
    {
    public:
        explicit Iterator(DbCR db) : DbTableIterator(db) {}

        struct Entry : DbTableIterator::Entry, std::iterator<std::input_iterator_tag, Entry const>
        {
        private:
            friend struct Iterator;
            Entry(StatementP sql, bool isValid) : DbTableIterator::Entry(sql,isValid) {}
        public:
            BE_SQLITE_EXPORT Utf8CP GetNameUtf8() const;          
            BE_SQLITE_EXPORT Utf8CP GetDescriptionUtf8() const;   
            BE_SQLITE_EXPORT Utf8CP GetTypeUtf8() const;          
            BE_SQLITE_EXPORT uint64_t GetFileSize() const;        
            BE_SQLITE_EXPORT uint32_t GetChunkSize() const;       
            BE_SQLITE_EXPORT DateTime GetLastModified() const;    
            BE_SQLITE_EXPORT BeBriefcaseBasedId GetId() const;   
            Entry const& operator* () const {return *this;}
        };

        typedef Entry const_iterator;
        typedef const_iterator iterator;
        BE_SQLITE_EXPORT const_iterator begin() const;
        const_iterator end() const {return Entry(nullptr, false);}
        BE_SQLITE_EXPORT size_t QueryCount() const;
    };
    Iterator MakeIterator() const {return Iterator(m_db);}

    BE_SQLITE_EXPORT BeBriefcaseBasedId QueryFile(Utf8CP name, uint64_t* totalSize = nullptr, uint32_t* chunkSize = nullptr, Utf8StringP descr = nullptr, Utf8StringP typeStr = nullptr, DateTime* lastModified = nullptr);

    BE_SQLITE_EXPORT BeBriefcaseBasedId ImportWithoutCompressing(DbResult* stat, Utf8CP name, Utf8CP localFileName, Utf8CP typeStr, Utf8CP description = nullptr, DateTime const* lastModified = nullptr, uint32_t chunkSize = 500 * 1024);

    BE_SQLITE_EXPORT BeBriefcaseBasedId ImportDbFile(DbResult& stat, Utf8CP name, Utf8CP localFileName, Utf8CP typeStr, Utf8CP description = nullptr, DateTime const* lastModified = nullptr, uint32_t chunkSize = 500 * 1024, bool supportRandomAccess = true);

    BE_SQLITE_EXPORT BeBriefcaseBasedId Import(DbResult* stat, Utf8CP name, Utf8CP localFileName, Utf8CP typeStr, Utf8CP description = nullptr, DateTime const* lastModified = nullptr, uint32_t chunkSize = 500 * 1024, bool supportRandomAccess = true);

    DbResult Import(Utf8CP name, Utf8CP localFileName, Utf8CP typeStr, Utf8CP description = nullptr, uint32_t chunkSize = 500 * 1024, bool supportRandomAccess = true)
        {
        DbResult stat = BE_SQLITE_OK;
        Import(&stat, name, localFileName, typeStr, description, nullptr, chunkSize, supportRandomAccess);
        return stat;
        }

    BE_SQLITE_EXPORT DbResult ExportDbFile(Utf8CP localFileName, Utf8CP name, ICompressProgressTracker* progress=nullptr);

    BE_SQLITE_EXPORT DbResult Export(Utf8CP localFileName, Utf8CP name, ICompressProgressTracker* progress=nullptr);

    BE_SQLITE_EXPORT DbResult Read(bvector<Byte>& buffer, Utf8CP name);

    BE_SQLITE_EXPORT DbResult Replace(Utf8CP name, Utf8CP localFileName, uint32_t chunkSize=500*1024, DateTime const* lastModified = nullptr);

    BE_SQLITE_EXPORT DbResult AddEntry(Utf8CP name, Utf8CP typeStr, Utf8CP description = nullptr, DateTime const* lastModified = nullptr);

    BE_SQLITE_EXPORT DbResult Save(void const* data, uint64_t size, Utf8CP name, DateTime const* lastModified = nullptr, bool compress=true, uint32_t chunkSize=500*1024);


    BE_SQLITE_EXPORT DbResult Remove(Utf8CP name);

    DbResult CreateTable() const;
};
#endif

enum class BeSQLiteTxnMode : int {None=0, Deferred=1, Immediate=2, Exclusive=3,};

enum class DefaultTxn : int
{
    No = (int) BeSQLiteTxnMode::None,
    Yes = (int) BeSQLiteTxnMode::Deferred,
    Immediate = (int) BeSQLiteTxnMode::Immediate,
    Exclusive = (int) BeSQLiteTxnMode::Exclusive
};

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct EXPORT_VTABLE_ATTRIBUTE Savepoint : NonCopyableClass
{
#if !defined (DOCUMENTATION_GENERATOR)
protected:
    friend struct DbFile;
    Db*         m_db;
    DbFile*     m_dbFile;
    int32_t     m_depth;
    BeSQLiteTxnMode  m_txnMode;
    Utf8String  m_name;

    virtual void _OnDeactivate(bool isCommit) {m_depth=-1;}
    BE_SQLITE_EXPORT virtual DbResult _Begin(BeSQLiteTxnMode);
    BE_SQLITE_EXPORT virtual DbResult _Commit(Utf8CP operation);
    BE_SQLITE_EXPORT virtual DbResult _Cancel();

    Savepoint(DbFile& db, Utf8CP name, BeSQLiteTxnMode txnMode) : m_dbFile(&db), m_name(name), m_txnMode(txnMode) {m_db=nullptr; m_depth = -1;}
#endif

public:
    BE_SQLITE_EXPORT Savepoint(Db& db, Utf8CP name, bool beginTxn=true, BeSQLiteTxnMode txnMode=BeSQLiteTxnMode::Deferred);

    virtual ~Savepoint() {Commit(nullptr);}

    bool IsActive() const {return GetDepth() >= 0;}

    int32_t GetDepth() const {return m_depth;}

    Utf8CP GetName() const {return m_name.c_str();}

    BeSQLiteTxnMode GetTxnMode() const {return m_txnMode;}

    void SetTxnMode(BeSQLiteTxnMode mode) {m_txnMode = mode;}

    BE_SQLITE_EXPORT DbResult Begin(BeSQLiteTxnMode mode);

    DbResult Begin() {return Begin(m_txnMode);}

    BE_SQLITE_EXPORT DbResult Commit(Utf8CP operation=nullptr);

    DbResult Save(Utf8CP operation) {Commit(operation); return Begin();}

    BE_SQLITE_EXPORT DbResult Cancel();
};

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct PropertySpec
{
public:
    enum class Compress
    {
        No=0,

        Yes=1
    };

    enum class Mode {Normal=0, Setting=1, Cached=2,};

private:
    Mode m_mode;
    Compress m_compress;
    bool   m_saveIfNull;
    Utf8CP m_name;
    Utf8CP m_namespace;

public:
    PropertySpec(Utf8CP name, Utf8CP nameSpace, Mode mode=Mode::Normal, Compress compress=Compress::Yes, bool saveIfNull=false) : m_name(name), m_namespace(nameSpace), m_compress(compress), m_mode(mode), m_saveIfNull(saveIfNull){}

    PropertySpec(PropertySpec const& other, Mode mode) {*this = other; m_mode=mode;}

    Utf8CP GetName() const {return m_name;}
    Utf8CP GetNamespace() const {return m_namespace;}

    bool IsSetting() const {return Mode::Setting == m_mode;}    
    bool IsCached()  const {return Mode::Cached == m_mode;}     
    bool SaveIfNull() const {return m_saveIfNull;}              
    bool IsCompress() const {return Compress::Yes==m_compress;} 
};

typedef PropertySpec const& PropertySpecCR;

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct BusyRetry : RefCountedBase
{
    virtual int _OnBusy(int count) const {if (count>4) return 0; BeThreadUtilities::BeSleep(1000); return 1;}
};

//=======================================================================================
// Cached "briefcase local values"
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct CachedBLV
{
private:
    Utf8String   m_name;
    bool         m_isUnset;
    mutable bool m_dirty;
    uint64_t     m_value;

public:
    explicit CachedBLV(Utf8CP name);
    Utf8CP GetName() const {return m_name.c_str();}
    uint64_t GetValue() const {BeAssert(!m_isUnset); return m_value;}
    void ChangeValue(uint64_t value, bool initializing = false);
    uint64_t Increment();
    bool IsUnset() const {return m_isUnset;}
    bool IsDirty() const {BeAssert(!m_isUnset); return m_dirty;}
    void SetIsNotDirty() const;
    void Reset();
};

//=======================================================================================
// Cache for uint64_t BriefcaseLocalValues. This is for BLV's that are of type uint64_t and are frequently
// accessed and/or modified. The BLVs are identified in the cache by "registering" their name are thereafter
// accessed by index. The cache is held in memory for performance. It is automatically saved whenever a transaction is committed.
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct BriefcaseLocalValueCache : NonCopyableClass
{
private:
    friend struct DbFile;
    friend struct Db;
    bvector<CachedBLV> m_cache;
    DbFile& m_dbFile;
    mutable BeMutex m_mutex;

    void Clear();
    bool TryQuery(CachedBLV*&, size_t rlvIndex);

public:
    BriefcaseLocalValueCache(DbFile& dbFile) : m_dbFile(dbFile) {}

    BE_SQLITE_EXPORT DbResult Register(size_t& rlvIndex, Utf8CP rlvName);

    BE_SQLITE_EXPORT bool TryGetIndex(size_t& rlvIndex, Utf8CP rlvName);

    BE_SQLITE_EXPORT DbResult SaveValue(size_t rlvIndex, uint64_t value);

    BE_SQLITE_EXPORT DbResult QueryValue(uint64_t& value, size_t rlvIndex);

    BE_SQLITE_EXPORT DbResult IncrementValue(uint64_t& newValue, size_t rlvIndex);
};

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct DbFile : NonCopyableClass
{ 
    friend struct Db;
    friend struct Statement;
    friend struct Savepoint;

protected:
    typedef RefCountedPtr<struct ChangeTracker> ChangeTrackerPtr;
    typedef RefCountedPtr<BusyRetry> BusyRetryPtr;
    typedef bvector<Savepoint*> DbTxns;
    typedef DbTxns::iterator DbTxnIter;

    bool m_settingsTableCreated;
    bool m_settingsDirty;
    bool m_allowImplicitTxns;
    bool m_inCommit;
    mutable bool m_readonly;
    uint64_t m_dataVersion; // for detecting changes from another process
    SqlDbP m_sqlDb;
    BusyRetryPtr m_retry;
    ChangeTrackerPtr m_tracker;
    mutable void* m_cachedProps;
    BriefcaseLocalValueCache m_blvCache;
    BeGuid m_dbGuid;
    Savepoint m_defaultTxn;
    BeBriefcaseId m_briefcaseId;
    StatementCache m_statements;
    DbTxns m_txns;

    explicit DbFile(SqlDbP sqlDb, BusyRetry* retry, BeSQLiteTxnMode defaultTxnMode);
    ~DbFile();
    DbResult StartSavepoint(Savepoint&, BeSQLiteTxnMode);
    DbResult StopSavepoint(Savepoint&, bool isCommit, Utf8CP operation);
    DbResult CreatePropertyTable(Utf8CP tablename, Utf8CP ddl, bool temp);
    DbResult SaveCachedProperty(PropertySpecCR spec, uint64_t id, uint64_t subId, Utf8CP stringData, void const* value, uint32_t size) const;
    struct CachedProperyMap& GetCachedPropMap() const;
    struct CachedPropertyValue& GetCachedProperty(PropertySpecCR spec, uint64_t id, uint64_t subId) const;
    struct CachedPropertyValue* FindCachedProperty(PropertySpecCR spec, uint64_t id, uint64_t subId) const;
    DbResult QueryCachedProperty(Utf8String*, void** value, uint32_t size, PropertySpecCR spec, uint64_t id, uint64_t subId) const;
    void DeleteCachedProperty(PropertySpecCR spec, uint64_t id, uint64_t subId);
    void DeleteCachedPropertyMap();
    void SaveCachedProperties(bool isCommit);
    Utf8String GetLastError(DbResult* lastResult) const;
    void SaveCachedBlvs(bool isCommit);
    BE_SQLITE_EXPORT DbResult SaveProperty(PropertySpecCR spec, Utf8CP strData, void const* value, uint32_t propsize, uint64_t majorId=0, uint64_t subId=0);
    BE_SQLITE_EXPORT bool HasProperty(PropertySpecCR spec, uint64_t majorId=0, uint64_t subId=0) const;
    BE_SQLITE_EXPORT DbResult QueryPropertySize(uint32_t& propsize, PropertySpecCR spec, uint64_t majorId=0, uint64_t subId=0) const;
    BE_SQLITE_EXPORT DbResult QueryProperty(void* value, uint32_t propsize, PropertySpecCR spec, uint64_t majorId=0, uint64_t subId=0) const;
    BE_SQLITE_EXPORT DbResult QueryProperty(Utf8StringR value, PropertySpecCR spec, uint64_t majorId=0, uint64_t subId=0) const;
    BE_SQLITE_EXPORT void SaveSettings();
    BE_SQLITE_EXPORT DbResult DeleteProperty(PropertySpecCR spec, uint64_t majorId=0, uint64_t subId=0);
    BE_SQLITE_EXPORT DbResult DeleteProperties(PropertySpecCR spec, uint64_t* majorId);
    BE_SQLITE_EXPORT int AddFunction(DbFunction& function) const;
    BE_SQLITE_EXPORT int AddRTreeMatchFunction(RTreeMatchFunction& function) const;
    BE_SQLITE_EXPORT int RemoveFunction(DbFunction&) const;
    BE_SQLITE_EXPORT BriefcaseLocalValueCache& GetBLVCache();

public:
    Utf8String ExplainQuery(Utf8CP sql, bool explainPlan, bool suppressDiagnostics) const;
    int OnCommit();
    bool UseSettingsTable(PropertySpecCR spec) const;
    void OnSettingsDirtied() {m_settingsDirty=true;}
    bool CheckImplicitTxn() const {return m_allowImplicitTxns || m_txns.size() > 0;}
    SqlDbP GetSqlDb() const {return m_sqlDb;}
};

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct EXPORT_VTABLE_ATTRIBUTE Db : NonCopyableClass
{                                           
    friend struct DbFile;
public:
    enum class Encoding {Utf8=0, Utf16=1};
    enum class PageSize : int
    {
        PAGESIZE_1K  = 1024,
        PAGESIZE_512 = PAGESIZE_1K / 2,
        PAGESIZE_2K  = PAGESIZE_1K * 2,
        PAGESIZE_4K  = PAGESIZE_1K * 4,
        PAGESIZE_8K  = PAGESIZE_1K * 8,
        PAGESIZE_16K = PAGESIZE_1K * 16,
        PAGESIZE_32K = PAGESIZE_1K * 32,
        PAGESIZE_64K = PAGESIZE_1K * 64
    };

    enum class OpenMode {Readonly = 1<<0, ReadWrite = 1<<1, Create = ReadWrite|(1<<2), SharedCache = 1<<17, };

    //=======================================================================================
    // @bsiclass                                                     Bentley Systems
    //=======================================================================================
    struct EXPORT_VTABLE_ATTRIBUTE OpenParams
    {
        mutable OpenMode m_openMode;
        DefaultTxn     m_startDefaultTxn;
        mutable bool  m_forSchemaUpgrade;
        bool          m_rawSQLite;
        BusyRetry*    m_busyRetry;

        BE_SQLITE_EXPORT virtual bool _ReopenForProfileUpgrade(Db&) const;

        BE_SQLITE_EXPORT explicit OpenParams(OpenMode openMode, DefaultTxn startDefaultTxn=DefaultTxn::Yes, BusyRetry* retry=nullptr);

        void SetRawSQLite() {m_rawSQLite=true;}

        bool IsRawSQLite() const {return m_rawSQLite;}

        bool IsReadonly() const {return m_openMode==OpenMode::Readonly;}

        void SetOpenMode(OpenMode openMode) {m_openMode = openMode;}

        void SetStartDefaultTxn(DefaultTxn val) {m_startDefaultTxn = val;}
    };

    //=======================================================================================
    // @bsiclass                                                     Bentley Systems
    //=======================================================================================
    struct CreateParams : OpenParams
    {
        Encoding m_encoding;
        PageSize m_pagesize;
        enum CompressedDb {CompressDb_None=0,CompressDb_Zlib=1,CompressDb_Snappy=2,} m_compressedDb;
        enum ApplicationId : uint64_t {APPLICATION_ID_BeSQLiteDb='BeDb',} m_applicationId;
        bool m_failIfDbExists;
        DateTime m_expirationDate;
        bool m_createStandalone;

        explicit CreateParams(PageSize pagesize=PageSize::PAGESIZE_4K, Encoding encoding=Encoding::Utf8, bool failIfDbExists=true,
                DefaultTxn defaultTxn=DefaultTxn::Yes, BusyRetry* retry=nullptr, bool createStandalone=false)
                : OpenParams(OpenMode::Create, defaultTxn, retry), m_encoding(encoding), m_pagesize(pagesize), m_compressedDb(CompressDb_None),
                m_failIfDbExists(failIfDbExists), m_applicationId(APPLICATION_ID_BeSQLiteDb), m_createStandalone(createStandalone) {;}

        void SetPageSize(PageSize pagesize) {m_pagesize = pagesize;}
        void SetEncoding(Encoding encoding) {m_encoding = encoding;}
        void SetFailIfDbExist(bool val) {m_failIfDbExists = val;}
        void SetCompressMode(CompressedDb val) {m_compressedDb=val;}
        void SetApplicationId(ApplicationId applicationId) {m_applicationId=applicationId;}
        void SetExpirationDate(DateTime xdate) {m_expirationDate=xdate;}
        void SetCreateStandalone(bool val) {m_createStandalone=val;}
    };

    //=======================================================================================
    // @bsiclass                                                     Bentley Systems
    //=======================================================================================
    struct AppData : RefCountedBase
    {
        struct Key : NonCopyableClass {};
    };

protected:
    DbFile* m_dbFile;
    StatementCache m_statements;
#if defined (SUPPORT_EMBEDDED_FILE)
    DbEmbeddedFileTable m_embeddedFiles;
#endif
    mutable bmap<AppData::Key const*, RefCountedPtr<AppData>, std::less<AppData::Key const*>, 8> m_appData;

    virtual DbResult _OnDbCreated(CreateParams const& params) {return BE_SQLITE_OK;}

    virtual void _OnSaveSettings() {}

    virtual DbResult _OnDbOpening() {return QueryDbIds();}

    virtual DbResult _OnDbOpened() {return BE_SQLITE_OK;}

    virtual void _OnDbClose() {}

    BE_SQLITE_EXPORT virtual void _OnDbChangedByOtherConnection();

    virtual DbResult _OnBriefcaseIdChanged(BeBriefcaseId newBriefcaseId) {return BE_SQLITE_OK;}

    virtual DbResult _VerifyProfileVersion(OpenParams const& params) {return BE_SQLITE_OK;}

    virtual int _OnAddFunction(DbFunction& func) const {return 0;}
    virtual void _OnRemoveFunction(DbFunction& func) const {}

    friend struct Statement;
    friend struct Savepoint;
    friend struct BeSQLiteProfileManager;

private:
    BE_SQLITE_EXPORT DbResult QueryDbIds();
    DbResult SaveBeDbGuid();
    DbResult SaveBriefcaseId();
    void DoCloseDb();
    DbResult DoOpenDb(Utf8CP dbName, OpenParams const& params);
    DbResult TruncateTable(Utf8CP tableName) const;
    DbResult ClearBriefcaseLocalValues();
    void SetupBlvSaveStmt(Statement& stmt, Utf8CP name);
    DbResult ExecBlvQueryStmt(Statement& stmt, Utf8CP name) const;

public:
    BE_SQLITE_EXPORT Db();
    BE_SQLITE_EXPORT virtual ~Db();

    DbFile* GetDbFile() {return m_dbFile;}

    BE_SQLITE_EXPORT void SetAllowImplictTransactions(bool val);

    StatementCache& GetStatementCache() const {return const_cast<StatementCache&>(m_statements);}

    BE_SQLITE_EXPORT DbResult GetCachedStatement(CachedStatementPtr& statement, Utf8CP sql) const;

    CachedStatementPtr GetCachedStatement(Utf8CP sql) const {CachedStatementPtr stmt; GetCachedStatement(stmt, sql); return stmt;}

    BE_SQLITE_EXPORT Savepoint* GetSavepoint(int32_t depth) const;

    BE_SQLITE_EXPORT int32_t GetCurrentSavepointDepth() const;

    bool IsTransactionActive() const {return 0 < GetCurrentSavepointDepth();}

    BE_SQLITE_EXPORT DbResult OpenBeSQLiteDb(Utf8CP dbName, OpenParams const& openParams);

    DbResult OpenBeSQLiteDb(BeFileNameCR dbName, OpenParams const& openParams) {return OpenBeSQLiteDb(dbName.GetNameUtf8().c_str(),openParams);}

    BE_SQLITE_EXPORT DbResult CreateNewDb(Utf8CP dbName, BeGuid dbGuid=BeGuid(), CreateParams const& params=CreateParams());

    DbResult CreateNewDb(BeFileNameCR dbName, BeGuid dbGuid=BeGuid(), CreateParams const& params=CreateParams())
                        {return CreateNewDb(dbName.GetNameUtf8().c_str(), dbGuid, params);}

    BE_SQLITE_EXPORT bool IsDbOpen() const;

    BE_SQLITE_EXPORT void CloseDb();

    BE_SQLITE_EXPORT Utf8CP GetDbFileName() const;

    BE_SQLITE_EXPORT void SaveProjectGuid(BeGuid);

    BE_SQLITE_EXPORT BeGuid QueryProjectGuid() const;

    BE_SQLITE_EXPORT DbResult SaveCreationDate();

    BE_SQLITE_EXPORT DbResult QueryCreationDate(DateTime& creationDate) const;

    BE_SQLITE_EXPORT DbResult AttachDb(Utf8CP filename, Utf8CP alias);

    BE_SQLITE_EXPORT DbResult DetachDb(Utf8CP alias);

    BE_SQLITE_EXPORT DbResult ExecuteSql(Utf8CP sql, int(*callback)(void*,int,char**,char**)=0, void* arg=0, char** errmsg=0) const;

    BE_SQLITE_EXPORT DbResult TryExecuteSql(Utf8CP sql, int (*callback)(void*,int,char**,char**)=0, void* arg=0, char** errmsg=0) const;

    BE_SQLITE_EXPORT Utf8String ExplainQuery(Utf8CP sql, bool plan=true) const;

    BE_SQLITE_EXPORT DbResult CreateTable(Utf8CP tableName, Utf8CP ddl) const;

    BE_SQLITE_EXPORT DbResult DropTable(Utf8CP tableName) const;

    BE_SQLITE_EXPORT bool TableExists(Utf8CP tableName) const;

    BE_SQLITE_EXPORT DbResult AddColumnToTable(Utf8CP tableName, Utf8CP columnName, Utf8CP columnDetails);

    BE_SQLITE_EXPORT bool ColumnExists(Utf8CP tableName, Utf8CP columnName) const;

    BE_SQLITE_EXPORT bool GetColumns(bvector<Utf8String>& columns, Utf8CP tableName) const;

    BE_SQLITE_EXPORT DbResult RenameTable(Utf8CP tableName, Utf8CP newTableName);

    BE_SQLITE_EXPORT DbResult CreateIndex(Utf8CP indexName, Utf8CP tableName, bool isUnique, Utf8CP columnName1, Utf8CP columnName2 = nullptr);

    BE_SQLITE_EXPORT void FlushPageCache();


    DbResult SaveProperty(PropertySpecCR spec, Utf8StringCR strData, void const* value, uint32_t propsize, uint64_t majorId=0, uint64_t subId=0) {return m_dbFile->SaveProperty(spec, strData.c_str(), value, propsize, majorId, subId);}

    DbResult SaveProperty(PropertySpecCR spec, void const* value, uint32_t propsize, uint64_t majorId=0, uint64_t subId=0) {return m_dbFile->SaveProperty(spec, nullptr, value, propsize, majorId, subId);}

    DbResult SavePropertyString(PropertySpecCR spec, Utf8CP value, uint64_t majorId=0, uint64_t subId=0) {return m_dbFile->SaveProperty(spec, value, nullptr, 0, majorId, subId);}

    DbResult SavePropertyString(PropertySpecCR spec, Utf8StringCR value, uint64_t majorId=0, uint64_t subId=0) {return SavePropertyString(spec, value.c_str(), majorId, subId);}

    bool HasProperty(PropertySpecCR spec, uint64_t majorId=0, uint64_t subId=0) const {return m_dbFile->HasProperty(spec, majorId, subId);}

    DbResult QueryPropertySize(uint32_t& propsize, PropertySpecCR spec, uint64_t majorId=0, uint64_t subId=0) const {return m_dbFile->QueryPropertySize(propsize, spec, majorId, subId);}

    DbResult QueryProperty(void* value, uint32_t propsize, PropertySpecCR spec, uint64_t majorId=0, uint64_t subId=0) const {return m_dbFile->QueryProperty(value, propsize, spec, majorId, subId);}

    DbResult QueryProperty(Utf8StringR value, PropertySpecCR spec, uint64_t majorId=0, uint64_t subId=0) const {return m_dbFile->QueryProperty(value, spec, majorId, subId);}

    DbResult DeleteProperty(PropertySpecCR spec, uint64_t majorId=0, uint64_t subId=0) {return m_dbFile->DeleteProperty(spec, majorId, subId);}

    DbResult DeleteProperties(PropertySpecCR spec, uint64_t* majorId) {return m_dbFile->DeleteProperties(spec, majorId);}

    BE_SQLITE_EXPORT void SaveSettings();


    BriefcaseLocalValueCache& GetBLVCache() {return m_dbFile->GetBLVCache();}

    BE_SQLITE_EXPORT DbResult QueryBriefcaseLocalValue(Utf8StringR value, Utf8CP name) const;
    BE_SQLITE_EXPORT DbResult QueryBriefcaseLocalValue(uint64_t& value, Utf8CP name) const;

    BE_SQLITE_EXPORT DbResult SaveBriefcaseLocalValue(Utf8CP name, Utf8StringCR value);
    BE_SQLITE_EXPORT DbResult SaveBriefcaseLocalValue(Utf8CP name, uint64_t value);

    BE_SQLITE_EXPORT DbResult DeleteBriefcaseLocalValue(Utf8CP name);

    BE_SQLITE_EXPORT int64_t GetLastInsertRowId() const;

    BE_SQLITE_EXPORT int GetModifiedRowCount() const;

    BE_SQLITE_EXPORT Utf8String GetLastError(DbResult* lastResult = nullptr) const;

    BE_SQLITE_EXPORT DbResult SaveChanges(Utf8CP changesetName=nullptr);
    DbResult SaveChanges(Utf8StringCR changesetName) {return SaveChanges(changesetName.c_str());}

    BE_SQLITE_EXPORT DbResult AbandonChanges();

    BE_SQLITE_EXPORT SqlDbP GetSqlDb() const;

    BE_SQLITE_EXPORT BeGuid GetDbGuid() const;

    BE_SQLITE_EXPORT BeBriefcaseId GetBriefcaseId() const;

#if defined (SUPPORT_BRIEFCASE)
    BE_SQLITE_EXPORT DbResult GetServerIssuedId(BeServerIssuedId& value, Utf8CP tableName, Utf8CP columnName, Utf8CP json=nullptr);
#endif

    BE_SQLITE_EXPORT bool IsReadonly() const;

#if defined (SUPPORT_EMBEDDED_FILE)
    BE_SQLITE_EXPORT DbEmbeddedFileTable& EmbeddedFiles();
#endif

    BE_SQLITE_EXPORT void AddAppData(AppData::Key const& key, AppData* appData) const;

    BE_SQLITE_EXPORT StatusInt DropAppData(AppData::Key const& key) const;

    BE_SQLITE_EXPORT AppData* FindAppData(AppData::Key const& key) const;

    BE_SQLITE_EXPORT void DumpSqlResults(Utf8CP sql);

    BE_SQLITE_EXPORT static Utf8CP InterpretDbResult(DbResult result);

    BE_SQLITE_EXPORT int SetLimit(int id, int newVal);

    BE_SQLITE_EXPORT int AddFunction(DbFunction& func) const;

    BE_SQLITE_EXPORT int RemoveFunction(DbFunction& func) const;

    BE_SQLITE_EXPORT int AddRTreeMatchFunction(RTreeMatchFunction& func) const;

    BE_SQLITE_EXPORT void ChangeDbGuid(BeGuid);

    BE_SQLITE_EXPORT DbResult ChangeBriefcaseId(BeBriefcaseId);

    BE_SQLITE_EXPORT void SetChangeTracker(ChangeTracker* tracker);

    BE_SQLITE_EXPORT bool IsSettingProperty(Utf8CP space, Utf8CP name, uint64_t id, uint64_t subId) const;
    Savepoint* GetDefaultTransaction() const {return (m_dbFile != nullptr) ? &m_dbFile->m_defaultTxn : nullptr;}

    BE_SQLITE_EXPORT static DbResult CheckProfileVersion(bool& fileIsAutoUpgradable, ProfileVersion const& expectedProfileVersion, ProfileVersion const& actualProfileVersion, ProfileVersion const& minimumUpgradableProfileVersion, bool openModeIsReadonly, Utf8CP profileName);

    BE_SQLITE_EXPORT DbResult UpgradeBeSQLiteProfile();

    BE_SQLITE_EXPORT bool IsExpired() const;

    BE_SQLITE_EXPORT DbResult QueryExpirationDate(DateTime& expirationDate) const;

    BE_SQLITE_EXPORT DbResult SaveExpirationDate(DateTime const& expirationDate);
};

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
enum ZipErrors
{
    ZIP_SUCCESS = 0,
    ZIP_ERROR_BASE              = 0x15000,
    ZIP_ERROR_WRITE_ERROR       = ZIP_ERROR_BASE + 0x01,
    ZIP_ERROR_COMPRESSION_ERROR = ZIP_ERROR_BASE + 0x02,
    ZIP_ERROR_END_OF_DATA       = ZIP_ERROR_BASE + 0x03,
    ZIP_ERROR_BUFFER_FULL       = ZIP_ERROR_BASE + 0x04,
    ZIP_ERROR_BLOB_READ_ERROR   = ZIP_ERROR_BASE + 0x05,
    ZIP_ERROR_FILE_DOES_NOT_EXIST = ZIP_ERROR_BASE + 0x06,
    ZIP_ERROR_CANNOT_OPEN_OUTPUT = ZIP_ERROR_BASE + 0x07,
    ZIP_ERROR_UNKNOWN            = ZIP_ERROR_BASE + 0x08,
    ZIP_ERROR_BAD_DATA           = ZIP_ERROR_BASE + 0x09,
    ZIP_ERROR_CANNOT_OPEN_INPUT  = ZIP_ERROR_BASE + 0x0A,
    ZIP_ERROR_ABORTED            = ZIP_ERROR_BASE + 0x0B,
    ZIP_ERROR_READ_ERROR         = ZIP_ERROR_BASE + 0x0C,
};

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct SnappyToBlob
{
private:
    struct SnappyChunk
    {
        uint16_t*  m_data;
        SnappyChunk(uint32_t size) {m_data = (uint16_t*) BeSQLiteLib::MallocMem(size + 2);} // 2 bytes for compressed size at start of data
        uint16_t GetChunkSize() {return m_data[0];}
        ~SnappyChunk() {BeSQLiteLib::FreeMem(m_data);}
    };

    bvector<SnappyChunk*> m_chunks;
    Byte*   m_rawBuf;
    uint32_t m_currChunk;
    uint32_t m_rawSize;
    uint32_t m_rawCurr;
    uint32_t m_rawAvail;
    uint32_t m_unsnappedSize;

public:
    BE_SQLITE_EXPORT SnappyToBlob();
    BE_SQLITE_EXPORT ~SnappyToBlob();
    uint32_t GetCurrChunk() {return m_currChunk;}
    Byte*  GetChunkData(int i) {return (Byte*) m_chunks[i]->m_data;}

    uint32_t GetUnCompressedSize() const {return m_unsnappedSize;}

    BE_SQLITE_EXPORT uint32_t GetCompressedSize();

    BE_SQLITE_EXPORT void Init();

    BE_SQLITE_EXPORT void Write(Byte const* data, uint32_t size);

    BE_SQLITE_EXPORT void Finish();

    void DoSnappy(Byte const* data, uint32_t size) {Init(); Write(data,size); Finish();}

    BE_SQLITE_EXPORT BentleyStatus SaveToRow(DbR db, Utf8CP tableName, Utf8CP column, int64_t rowId);

    BE_SQLITE_EXPORT BentleyStatus SaveToRow(BlobIO& blobIO);
    };

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct SnappyReader
{
    static const uint32_t SNAPPY_UNCOMPRESSED_BUFFER_SIZE = (34*1024);

    virtual ~SnappyReader() {}
    virtual ZipErrors _Read(Byte* data, uint32_t size, uint32_t& actuallyRead) = 0;

    static uint32_t GetUncompressedBufferSize() {return SNAPPY_UNCOMPRESSED_BUFFER_SIZE;}
};

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct SnappyFromMemory : SnappyReader
{
private:
    Byte*   m_uncompressed;
    Byte*   m_uncompressCurr;
    uint16_t m_uncompressAvail;
    uint16_t m_uncompressSize;
    Byte*   m_blobData;
    uint32_t m_blobOffset;
    uint32_t m_blobBytesLeft;

    ZipErrors ReadNextChunk();
    ZipErrors TransferFromBlob(void* data, uint32_t numBytes, int offset);

public:
    BE_SQLITE_EXPORT SnappyFromMemory(void* uncompressedBuffer, uint32_t uncompressedBufferSize);
    BE_SQLITE_EXPORT void Init(void* blobBuffer, uint32_t blobBufferSize);
    BE_SQLITE_EXPORT virtual ZipErrors _Read(Byte* data, uint32_t size, uint32_t& actuallyRead) override;
};

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct SnappyFromBlob : SnappyReader
{
private:
    Byte*   m_uncompressed;
    Byte*   m_uncompressCurr;
    Byte*   m_blobData;
    BlobIO  m_blobIO;
    uint32_t m_blobBufferSize;
    uint32_t m_blobOffset;
    uint32_t m_blobBytesLeft;
    uint16_t m_uncompressAvail;

    ZipErrors ReadNextChunk();

public:
    BE_SQLITE_EXPORT SnappyFromBlob();
    BE_SQLITE_EXPORT ZipErrors Init(DbCR db, Utf8CP tableName, Utf8CP column, int64_t rowId);
    BE_SQLITE_EXPORT ~SnappyFromBlob();
    BE_SQLITE_EXPORT virtual ZipErrors _Read(Byte* data, uint32_t size, uint32_t& actuallyRead) override;
    BE_SQLITE_EXPORT void Finish() ;
    ZipErrors ReadAndFinish(Byte* data, uint32_t bufSize, uint32_t& bytesActuallyRead) {auto stat=_Read(data, bufSize, bytesActuallyRead); Finish(); return stat;}
};

#define SQLITE_FORMAT_SIGNATURE     "SQLite format 3"
#define DOWNLOAD_FORMAT_SIGNATURE   "Download SQLite"
#define SQLZLIB_FORMAT_SIGNATURE    "ZV-zlib"
#define SQLSNAPPY_FORMAT_SIGNATURE  "ZV-snappy"

#define BEDB_PROPSPEC_NAMESPACE "be_Db"
#define BEDB_PROPSPEC_EMBEDBLOB_NAME "EmbdBlob"
//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct PropSpec : PropertySpec
{
    PropSpec(Utf8CP name, PropertySpec::Compress compress = PropertySpec::Compress::Yes) : PropertySpec(name, BEDB_PROPSPEC_NAMESPACE, Mode::Normal, compress) {}
};

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct Properties
{
    static PropSpec DbGuid()            {return PropSpec("DbGuid");}
    static PropSpec ProfileVersion()    {return PropSpec("SchemaVersion");}
    static PropSpec ProjectGuid()       {return PropSpec("ProjectGuid");}
    static PropSpec EmbeddedFileBlob()  {return PropSpec(BEDB_PROPSPEC_EMBEDBLOB_NAME, PropertySpec::Compress::No);}
    static PropSpec CreationDate()      {return PropSpec("CreationDate");}
    static PropSpec ExpirationDate()    {return PropSpec("ExpirationDate");}

    static PropSpec BeSQLiteBuild()     {return PropSpec("BeSQLiteBuild");}
};

//=======================================================================================
// @bsiclass                                                     Bentley Systems
//=======================================================================================
struct LzmaUtility
{
    static BE_SQLITE_EXPORT ZipErrors CompressDb(BeFileNameCR targetFile, BeFileNameCR sourceFile, ICompressProgressTracker* progressTracker, uint32_t dictionarySize, bool supportRandomAccess);

    // Decompress a Db file
    static BE_SQLITE_EXPORT ZipErrors DecompressDb(BeFileNameCR targetFile, BeFileNameCR sourceFile, ICompressProgressTracker* progress);

    static ZipErrors DecompressEmbeddedBlob(bvector<Byte>&out, uint32_t expectedSize, void const*inputBuffer, uint32_t inputSize, Byte*header, uint32_t headerSize); 
};

END_BENTLEY_SQLITE_NAMESPACE