User’s Guide to VelocityDB (Tuesday, August 12, 2014)

This guide compliments the sample programs and the API reference provided on our site.  Developers should review this in order to better understand how to a build a VelocityDB-integrated application.

Contents

Opening the samples solution, VelocityDB.sln   2

Project reference to VelocityDB.dll 3

NuGet packages for solution   4

Selecting the correct VelocityDB Session Class  5

Optimistic locking verses Pessimistic locking  6

Composite Object Identifier  6

DatabaseLocation   7

Page Compression   7

Moving Databases to a different host and/or directory  7

Databases  8

VelocityDB license database   8

Pages  8

Transactions  8

Why we need transaction for reads  9

How to enable persistent objects of some class  9

Register all types that you plan on persisting  9

Persistent placement of objects  10

Customizing object placement (most of you can skip this part)  10

Looking up objects  11

DO NOT reference persistent data using static variables  12

Updating persistent objects  12

Deleting (unpersisting) persistent objects  13

Using the provided BTree collections  13

Indexes  14

Class level index  14

Using a class level index  15

Index by a field   15

Using the index by field in a LINQ query  15

System.OutOfMemoryException   15

Limiting graph of objects in memory  16

Lazy load of object references  16

Specifying depth to load at object open   17

Session caching of databases, pages and slots  17

Diagnostics  17

Handling exceptions thrown by VelocityDB   17

Putting type definitions of persist able types in separate project class library  18

Controlling the in memory page and object caching  18

Verifying all objects and references  19

Database backup and restore   19

Backup   19

Restore   19

CopyAllDatabasesTo   19

ExportToCSV and ImportFromCsv  20

VelocityDbServer.exe   20

Changing the default SessionBase. BaseDatabasePath in a VelocityDbServer  21

Enabling Windows Authentication   21

Why installation ends up in Program Files (x86) instead of Program Files?  21

Windows Phone 8  21

Setting Up the sample Web Site (VelocityWeb) on a hosting web site (in this case GoDaddy)  22

Transfer all the files to your hosting account  22

Login to your hosting provider to enable write access to a few of the directories in the application   24

Create an application root virtual directory for the new web application   26

Wait a few minutes then point your browser at your web application   27

If you transferred your application directory with databases then install your databases in their new loacftion. 28

If all is well, you are done, access the application and the databases!  28

 

Opening the samples solution, VelocityDB.sln

Using Visual Studio 2012, open %USERPROFILE%\My Documents\VelocityDB\VelocityDB.sln

You can also start it by using the shortcut in the programs start menu.

Project reference to VelocityDB.dll

All sample projects should have a reference to VelocityDB.dll. The path used to VelocityDB.dll is C:\Program Files (x86)\VelocityDb\VelocityDB.dll, if you windows directory isn’t C: or the reference is broken then you need to remove each project reference to VelocityDB.dll and add a new one using the path to it in your installation.

NuGet packages for solution

A few of the samples including VelocityGraph project uses 3rd party NuGet libraries. These libraries are not part of the installation but will be downloaded automatically when you attempt to build such a project. To make this happen you need to allow NuGet to download missing packages. If it still does not download (firewall blocking?) then you may need to manually install the missing NuGets.

Selecting the correct VelocityDB Session Class

The most important class for users of VelocityDB is the Session class which contains the Transaction Control API, the Persistence API, the Data Cache API and more.  VelocityDB provides three session types and does not limit usage.  Your application can utilize all of them as necessary: 

·         ServerClientSession  - Used for distributed databases or when clients are hosted remotely.

                                                     // initial DatabaseLocation directory and hostname

using (ServerClientSession session = new ServerClientSession("c:\\Databases", "DbServer"))

{

  session.BeginRead();

  // your code here

  session.Commit();

}

·          SessionNoServer - Client and data are on the same host (unless it is a web application)

using (SessionNoServer session = new SessionNoServer("c:\\Databases"))

{

  session.BeginRead();

  // your code here

  session.Commit();

}

·         SessionNoServerShared - Client and data are on the same host (unless it is a web application) with mutex protection

using (SessionNoServerShared session = new SessionNoServerShared ("c:\\Databases"))

{

  session.BeginRead();

  // your code here

  session.Commit();

}

The session class ServerClientSession is appropriate if the application will distribute data and/or clients across multiple hosts (where the clients are not just clients of a web site).  Otherwise, SessionNoServer or SessionNoServerShared are appropriate.  Of the two, the best choice is dependent upon the architecture of the application.

 

Additional benefits of using ServerClientSession

ü  Granularity of locking is page instead of database (file).

ü  Backup feature option

ü  Shared cache for all users (on server side)

ü  Deadlock detection (when pessimistic locking is used, with optimistic locking deadlocks don’t happen)

ü  Change event subscription and notification

Benefits of using SessionNoServer or SessionNoServerShared

ü  No server installation required

ü  More stable, less can go wrong

ü  Can perform better with local files.

Our video talking about database concurrency control may help you decide what session to use.

Use SessionNoServerShared when the application must share a client-side cache between multiple threads.  This may be the case for a web site that has limited RAM resources while also having a large amount of persistent data to manage.  Otherwise, use SessionNoServer . The difference between these two classes is that the API on SessionNoServerShared is protected by lock(this){} blocks. If you are not sharing a session between multiple threads then you are slightly better-off using SessionNoServer as it avoids the performance penalty of SessionNoServerShared

It is recommended that a session is reused for multiple transactions since that will provide some caching benefits and also avoids some setup time, especially with ServerClientSession.

DO NOT pass objects between session instances. Once you read an object from a database, that object belongs to the session used to read it. Do not attempt to read an object using one session and the update it using another session. This will not work as expected and we may not detect it so it will fail silently.

Optimistic locking verses Pessimistic locking

By default VelocityDB uses optimistic locking. Pessimistic locking can be turned on by a session constructor parameter. With optimistic locking, reads are always possible except for uncommitted new databases and multiple updaters are allowed but only the first writer will succeed, the other writers of the same page (ServerClientSession) or database (SessionNoServer) will get an optimistic locking exception. Once you decide using optimistic/pessimistic concurrency control, stick with your choice. Do not mix sessions using optimistic concurrency control with sessions using pessimistic concurrency control.

Composite Object Identifier

All normal VelocityDB persistent objects have an associated composite object identifier. It is encoded as a UInt64 with three composite parts; a database number (upper 32 bits), a page number and a slot number.  The Id property returns an objects encoded object identifier and the Oid property returns the decoded object identifier as the struct Oid. A reference to a persistent object is persistently stored as an object identifier, it is normally a UInt64 but it can also be using a short object identifier, a UInt32, when the reference is to another object within the same database. The decoded short reference as a struct is OidShort. Use the special OidShort collection classes and tag object references with the attribute [UseOidShort] as in:

[Serializable]

[UseOidShort]

internal class Recovery : OptimizedPersistable

 

and for a specific member:

 

[UseOidShort]

public VelocityDbListOidShort<FreeSpace> theArray;

DatabaseLocation

This is a directory on some host. The initial DatabaseLocation is created when you create your first persistent object. You specify the directory when you create the session class. You can create additional database locations like:

using (ServerClientSession session = new ServerClientSession(systemDir, Dns.GetHostName()))

{

  session.BeginUpdate();

  DatabaseLocation otherLocation = new DatabaseLocation(Dns.GetHostName(), location2Dir, locationStartDbNum, locationEndDbNum, session, true, 0);

  otherLocation = session.NewLocation(otherLocation);

  session.Commit();

}

 

You need to commit the initial DatabaseLocation before other sessions (clients) can access it.

 

Page Compression

Page compression is now by default turned off. You can turn it on by setting the constructor parameter when you create a DatabaseLocation.

The initial/default DatabaseLocation is created when you run your first update transaction with a specified directory that does not already contain databases 0, 1, and 2 (0.odb, 1.odb, and 2.odb).

If you want page compression turned on for this DatabaseLocation, set SessionBase.DefaultCompressPages to true first. This static variable is also used when you create your own DatabaseLocation and not specifying the compressPages constructor parameter.

Moving Databases to a different host and/or directory

If you copy/move a directory containing your boot up database location then you need to update the DatabaseLocation containing the boot up host and path. We plan to introduce a convenience function for this but currently it can be done using the following code. Other database locations can be updated in a similar fashion but does not require the special parameters for BeginUpdate.

using (SessionNoServer session = new SessionNoServer(systemDirDeploy))

{

    session.BeginUpdate(false, true);

    DatabaseLocation bootLocation = session.DatabaseLocations.LocationForDb(0);

    DatabaseLocation locationNew = new DatabaseLocation(Dns.GetHostName(), systemDirDeploy, bootLocation.StartDatabaseNumber, bootLocation.EndDatabaseNumber, session,

                    bootLocation.CompressPages, bootLocation.PageEncryption, bootLocation.IsBackupLocation, bootLocation.BackupOfOrForLocation);

    bootLocation = session.NewLocation(locationNew);

    session.Commit(false);

}

Databases

A database corresponds to a file within a DatabaseLocation. The file name of a Database is <database number>.odb. When you create your first persistent data, three system databases are created:

·         0.odb
Contains a log of update transactions and the recovery mechanism data.

·         1.odb
Contains the schema objects

·         2.odb
Contains the DatabaseLocation objects.

These system databases must be committed by a session before other sessions can use them. This is true for any new database; a database must be committed before other sessions can access it.

A new uncommitted Database is named <database number>.new and an uncommitted deleted Database is named <database number>.del.

A Database can be created explicitly using session API or implicitly by placing a new persistent object with database part of the object identifier corresponding to an unallocated database number.

VelocityDB license database

Download your VelocityDB license database file from http://www.velocitydb.com/Secure/License.aspx

The license database file is named 4.odb. Copy this file to all database directories used for the system databases 1.odb … 9.odb. This is the directory you specify when creating the session instance. Some of the sample applications provided with the download will fail unless your license database, 4.odb, first is copied to the sample database directory.

Pages

A VelocityDB page can contains one or more persistent objects. The size of a Page can vary dynamically. A page is stored within a Database file. Each Page has a PageInfo header that contains information about a page. A Page can optionally be encrypted and/or compressed.

Transactions

All interaction with databases and persistent object require an active transaction. With VelocityDB we provide two kinds of transactions; update and read only. With an update transaction, you are permitted to update and add persistent data. With a read only transaction, an exception will be thrown by VelocityDB if you try to update persistent data. Only one concurrent transaction per session is permitted. A transaction is started and committed by API on the session classes.
An application may examine in memory persistent object without being in a transaction but an exception will be thrown if any persistent operation is requested like reading a page from a database.

public virtual void BeginRead(bool doRecoveryCheck = true)

 

public virtual void BeginUpdate()

 

public virtual void Commit(bool doRecoveryCheck = true)

 

public virtual void Abort()

Why we need transaction for reads

With optimistic locking option (the default) there is no locking reason for a transaction when only reading objects. If the other locking model is used, pessimistic locking, then read only transactions are needed because they define the scope of read locks. A session constructor parameter is used for requesting optimistic or pessimistic locking model. Another reason we need read only transaction is cache management and validation. Each Database, Page and Object is cached within a session instance. Each cached Database is validated on first use in a transaction, making sure cached version is up to date. If reads are frequent among multiple threads, it may make sense to use a shared session for the reads, SessionNoServerShared, and maintain an infinitely long open optimistic locking read transaction.  Call ForceDatabaseCacheValidation()frequently when there is possible other active database clients so that your cache stays up to date. Alternatively trigger validation of only selected databases by setting the Database property CachedVerified to false.

How to enable persistent objects of some class

There are two major choices for enabling persistence.

1.  Make your data model class a subclass of OptimizedPersistable

2.       Implement the interface IOptimizedPersistable. See the sample class PersistenceByInterfaceSnake as a template for how to implement the required interface API.

 

These two ways of enabling persistence can be mixed, some classes may implement the interface and others may be subclasses of OptimizedPersistable.

Objects of ValueType and arrays are embedded within a parent persistent object.

In addition, almost any type of object, except Delegate and Pointer instances, can be made persistent but this way is not very efficient due to requiring use of a fairly inefficient ConditionalWeakTable internally by VelocityDB due to such objects not maintaining an object identifier s as a field.

OptimizedPersistable implements IOptimizedPersistable.

Register all types that you plan on persisting

It is not mandatory but doing so you ensure that schema is created one way no matter in what order you persist objects and you avoid potential lock conflicts with the schema database (1.odb). For VelocityGraph, we do this the first time a Graph is persisted as:

    public override UInt64 Persist(Placement place, SessionBase session, bool persistRefs = true, bool disableFlush = false, Queue<IOptimizedPersistable> toPersist = null)

    {

      if (IsPersistent)

        return Id;

      session.RegisterClass(typeof(Graph));

      session.RegisterClass(typeof(BTreeMap<EdgeTypeId, EdgeTypeId>));

      session.RegisterClass(typeof(PropertyType));

      session.RegisterClass(typeof(VertexType));

      session.RegisterClass(typeof(VelocityDbList<VertexType>));

      session.RegisterClass(typeof(EdgeType));

      session.RegisterClass(typeof(UnrestrictedEdge));

      session.RegisterClass(typeof(VelocityDbList<Range<ElementId>>));

      session.RegisterClass(typeof(VelocityDbList<EdgeType>));

      session.RegisterClass(typeof(Range<VertexId>));

      session.RegisterClass(typeof(BTreeSet<Range<VertexId>>));

      session.RegisterClass(typeof(BTreeSet<EdgeType>));

      session.RegisterClass(typeof(BTreeSet<EdgeIdVertexId>));

      session.RegisterClass(typeof(BTreeMap<EdgeId, ulong>));

      session.RegisterClass(typeof(BTreeMap<EdgeId, UnrestrictedEdge>));

      session.RegisterClass(typeof(BTreeMap<string, PropertyType>));

      session.RegisterClass(typeof(BTreeMap<string, EdgeType>));

      session.RegisterClass(typeof(BTreeMap<string, VertexType>));

      session.RegisterClass(typeof(BTreeMap<VertexId, BTreeSet<EdgeIdVertexId>>));

      session.RegisterClass(typeof(BTreeMap<VertexType, BTreeMap<VertexId, BTreeSet<EdgeIdVertexId>>>));

      session.RegisterClass(typeof(BTreeMap<EdgeType, BTreeMap<VertexType, BTreeMap<VertexId, BTreeSet<EdgeIdVertexId>>>>));

      session.RegisterClass(typeof(BTreeMap<string, BTreeSet<ElementId>>));

      session.RegisterClass(typeof(BTreeMap<int, BTreeSet<ElementId>>));

      session.RegisterClass(typeof(BTreeMap<Int64, BTreeSet<ElementId>>));

      session.RegisterClass(typeof(PropertyTypeT<bool>));

      session.RegisterClass(typeof(PropertyTypeT<int>));

      session.RegisterClass(typeof(PropertyTypeT<long>));

      session.RegisterClass(typeof(PropertyTypeT<double>));

      session.RegisterClass(typeof(PropertyTypeT<DateTime>));

      session.RegisterClass(typeof(PropertyTypeT<string>));

      session.RegisterClass(typeof(PropertyTypeT<IComparable>));

      return base.Persist(place, session, persistRefs, disableFlush, toPersist);

    }

Persistent placement of objects

The placement (location) of persistent objects affects performance and locking. It is therefore important to make decisions about where to place an object when making it persistent. Once an object has been persisted, it remains in the same location for its persistent life time. You can decide how many objects you want on a single page. For slightly improved storage, require that a page only may contain objects of a specific type. Also fixed size objects (ones with no contained variable size arrays) can further improve object store efficiency. Several ways of controlling the placement when persisting object are provided. First on IOptimizedPersistable the following helps guide the placement:

UInt16 ObjectsPerPage

{

  get;

}

 

The recommended way of persisting objects is using the SessionBase api:

 

public UInt64 Persist(object obj)

 

When this api is used, each type is stored in its own database.

 

It is recommended that you make the following override in your OptimizedPersistable subclass for better performance:

    public override bool AllowOtherTypesOnSamePage

    {

      get

      {

        return false;

      }

    }

 

We may make this default but it could break existing code so it is not a trivial change.

Customizing object placement (most of you can skip this part)

In addition the IOptimizedPersistable interface contains API intended for customizing how fields of an object being persisted are to be persisted (including where to place).

 

UInt64 Persist(Placement place, SessionBase session, bool persistRefs = false, bool disableFlush = false);

 

UInt64 Persist(SessionBase session, IOptimizedPersistable placeHint, bool persistRefs = false, bool disableFlush = false);

 

 

for (int i = 0; i < numberOfPersons; i++)

{

   person = new Person();

   person.Persist(session, person);

}

for (int i = 0; i < numberOfPersons; i++)

{

   person = new Person();

   if (priorPerson == null)

    priorPerson = person;

   person.Persist(session, priorPerson); // use prior person as object to persist near

   priorPerson = person;

}

 

 

The second way of controlling the placement while persisting an object is by using persistent or transient instances of the Placement class.

 

public Placement(UInt32 db, UInt16 page = 1, UInt16 slot = 1, UInt16 objectsPerPage = 10000, UInt16 pagesPerDatabase = 10000, bool persistRefs = false, bool tryOtherDatabaseIfLockConflict = true, UInt32 maxNumberOfDatabases = UInt32.MaxValue, bool allowOtherTypesOnSamePage = true, bool flushFullPages = true)

 

public Placement(SessionBase session, IOptimizedPersistable placementProviderObject, IOptimizedPersistable objectToPlace, bool persistRefs = false, UInt32 maxNumberOfDatabases = UInt32.MaxValue, bool flushFullPages = true)

 

There is also additional API on Placement for fine tuning the placement. An instance of Placement is used as parameter to the IOptimizedPersistable Persist API mentioned above.

 

Sometimes it an advantage to put all related objects in a single database because then 32bit, OidShort, object references can be used instead of full 64 bit, Oid, object references. A short object reference contains only a page and slot part (16 bit each). Such references use less storage space and if only short references are used within a database, such a database can easily be cloned since it’s database number isn’t hard coded anywhere within the database. Short references are not automatically used when you place objects this way. The application must explicitly request it in the class definition by using the attribute [UseOidShort]. There are also special short references versions of the provided BTree collections. The application needs to use those instead of the long reference BTree collections when you want all objects within a database to use short references.

 

How to optimally place/persist objects is application dependent. The sample programs provided try to illustrate some of many use cases for object placement.

Looking up objects

The most efficient way is to have one or a few root objects that you look up by the object identifier as in:

ImdbRoot imdbRoot = (ImdbRoot)session.Open(ImdbRoot.PlaceInDatabase, 1, 1, false);

 

When you open an object this way, all objects referenced by the object is also connected to the object so then to reach related objects all you need to do is navigate to related objects such as in:

 

imdbRoot.ActingByNameSet

BTreeSet<Word> wordSet = indexRoot.lexicon.wordSet;

 

Another way to lookup objects is by using a LINQ query such as:

 

Database db = session.OpenDatabase(ComputerFileData.PlaceInDb);

var result = (from ComputerFileData computerFileData in db.AllObjects<ComputerFileData>()

                 where computerFileData.FileID == 500000

                 select computerFileData).First();

 

or you can accomplish the same lookup without using LINQ as:

 

ComputerFileData computerFileData = null;

Database db = session.OpenDatabase(ComputerFileData.PlaceInDb);

foreach (Page page in db)

  if (page.PageNumber > 0 && (computerFileData == null || computerFileData.FileID != 500000))

    foreach (IOptimizedPersistable o in page)

    {

      computerFileData = o as ComputerFileData;

      if (computerFileData != null)

         if (computerFileData.FileID == 500000)

             break;

    }

 

The third way is by looking up from a collection (usually a BTree)  as in:

 

doc.WordHit.TryGetValue(word, out wordHit)

DO NOT reference persistent data using static variables

It is not OK to have variables like

static VertexType movieType;

static PropertyType movieTitleType;

static PropertyType movieYearType;

Updating persistent objects

VelocityDB need to be notified when you want a change to an object to be persisted. The safest way to do this, is to create define property code for every field your application data objects have, such as:

public Person BestFriend

{

  get

  {

    return bestFriend;

  }

  set

  {

    Update(); // IMPORTANT, call Update() before updating object

    bestFriend = value;

  }

}

 

VelocityDB collection classes like VelocityDbList<T>, BTreeSet<Key> and BTreeMap<Key, Value> calls update automatically internally so you don’t need and should not call Update() when modifying such collections.

When updating objects that are not implementing IOptimizedPersistable, call session.UpdateObject. BindingList<MyItem> is such a case. Exception are: List<>, arrays and ValueType objects when embedded in an object that implements IOptimizedPersistable. For such lists call Update() on the object embedding the list.

public class MyContainer : OptimizedPersistable

{

    private BindingList<MyItem> _items;

    public BindingList<MyItem> Items {

            get { return _items; }

    }

 

    public MyContainer()

    {

        _items = new BindingList<MyItem>();

    }

 

    public bool UpdateBindingList(SessionBase session)

    {

        return session.UpdateObject(_items);

    }     

}

Deleting (unpersisting) persistent objects

Use OptimizedPersistable.Unpersist or Page.UnpersistObject or SessionBase.DeleteObject. You can override the default implementation of public virtual void Unpersist(SessionBase session, bool disableFlush = true), i.e.

    public override void Unpersist(SessionBase session, bool disableFlush = true)

    {

      if (id == 0)

        return;

      if (comparisonByteArrayId != 0)

      {ing)(((((((

        comparisonBytesTransient = (BTreeByteArray)session.Open(comparisonByteArrayId);

        comparisonBytesTransient.Unpersist(session, disableFlush);

        comparisonByteArrayId = 0;

      }

      nodeList.Unpersist(session, disableFlush);

      base.Unpersist(session, disableFlush);

    }

Using the provided BTree collections

Just about all object oriented applications need to use collections. VelocityDB provides BTree collections which are similar to BTree’s of the variety B*. A BTree is a collection where the added objects are sorted. An application can define the sort order by defining a subclass of VelocityDbComparer<Key> or by using the class CompareByField<Key>, a collection may also have a null comparator in which case the objects are ordered by the object identifier or by the ValueType ordering as defined by the objects public override int CompareTo(object obj) implementation. The BTree comes in a few varieties, a key only version and a key value version. They also have a long object Id (db-page-slot) version and a short Id (page-slot) version. A BTree can be used with comparisonByteArray data which is used to cache object key data within the BTree nodes so that when a binary search takes place we can avoid opening objects to compare. When you use the predefined class CompareByField<Key> it is easy to add comparisonByteArray data to the BTree nodes, you just specify how many bytes per object it should be and whether the cached node byte contains the entire data being compared when deciding if one object is less, equal or greater compared to another. If you customize building your own comparator, managing the comparisonByteArray becomes a little trickier; on the compare class you need to define SetComparisonArrayFromObject as in:

 

public override void SetComparisonArrayFromObject(Word key, byte[] comparisonArray, bool oidShort)

{

  Int32 hashCode = key.aWord.GetHashCode();

  Buffer.BlockCopy(BitConverter.GetBytes(IPAddress.HostToNetworkOrder(hashCode)), 0, comparisonArray, 0, comparisonArray.Length);

}

 

In this case we are sorting by the hash code of a string, the corresponding compare function in this case looks like:

 

public override int Compare(Word a, Word b)

{

  UInt32 aHash = (UInt32) a.aWord.GetHashCode();

  UInt32 bHash = (UInt32) b.aWord.GetHashCode();

  int value = aHash.CompareTo(bHash);

  if (value != 0)

    return value;

  return a.aWord.CompareTo(b.aWord);

}

 

A problem here is that a String GetHashCode() returns different values on a 32 bit platform then a 64 bit platform. To make your data cross platform compatible don’t use the string GetHashCode, instead build your own string hash code function. We do so in the VelocityDB build in class HashCodeComparer<T>.

 

Btree classes provided:

·         BTreeSet<Key>

·         BTreeSetOidShort<Key>

·         BTreeMap<Key, Value>

·         BTreeMapOidShort<Key, Value>

 

Sample usage:

public Lexicon(ushort nodeSize, HashCodeComparer<Word> hashComparer, SessionBase session)

{

   wordSet = new BTreeSet<Word>(hashComparer, session, nodeSize);

}

Indexes

Indexes is a simplified, automated, way of implicitly defining and keeping BTreeSet<Key>s up to date when objects are added, deleted and updated. An index is defined by using the class or field [Index] attribute.  Indexes for a persistent Type is stored in its own system selected database, the range of databases used is between 66000 up to 66000 + the number of Types and versions of a type that your application store persistently.  An object gets added to its indexes when the object is persisted. When an indexed object is updated, its indexes get updated when the page of the objects gets flushed to disk. An object is removed from its indexes when it is unpersisted and when Update() is called.  If you want to index objects separately for each Database, tag the class or field with the attribute [OnePerDatabase]. Before modifying an indexed field, it is important to call Update() on the object having the field before doing the update because the object needs to be removed from its indexes before updates or else the removal  code will fail to find the object in its indexes leading to an index corruption. Call FlushUpdates() or FlushUpdates(Database db) on the session after the changes have been made to add it back to indexes.

Class level index

When you want an index with compound keys, like order by lastName and then if two or more lastnames are equal by firstName and if two or more firstNames are equal, order these otherwise equal objects by yet another field name and so on. We currently only allow one class level index (by multiple compound keys) per class.

[Index("modelYear,brandName,modelName,color")]

  public abstract class Vehicle : OptimizedPersistable

  {

    string color;

    int maxPassengers;

    int fuelCapacity; // fuel capacity in liters  

    double litresPer100Kilometers; // fuel cunsumption

    DateTime modelYear;

    string brandName;

    string modelName;

    int maxSpeed; // km/h

    int odometer; // km

 

You can also use the class level Index attribute without specifying any field names; in that case the contained objects are sorted by the default ordering of the class which is normally by Oid (Id).

Using a class level index

To iterate all Cars in index sorted order

foreach (Car c in session.Index<Car>())

  Console.WriteLine(c.ToStringDetails(session));

Index by a field

This type of index sorts all persistent instances of a class by a field value. Note that in order to use this type of index in a LINQ query, you need to tell us what property that returns the value of the field. You do that by the FieldAccessor attribute as in sample class below. The [UniqueConstraint] attribute can be added when you don’t want multiple objects with the same field value in the index. An exception is raised if you add a second object with the same field value when [UniqueConstraint] is applied to the field. The [IndexStringByHashCode] attribute can also be added to string field indexes when you don’t care about the sort order. Sorting by hash code is faster than sorting by the normal string ordering.

 

  public class InsuranceCompany : OptimizedPersistable

  {

    [Index]

    [UniqueConstraint]

    [OnePerDatabase]

    string name;

    string phoneNumber;

 

    public InsuranceCompany(string name, string phoneNumber)

    {

      this.name = name;

      this.phoneNumber = phoneNumber;

    }

 

    [FieldAccessor("name")]

    public string Name

    {

      get

      {

        return name;

      }

    }

  }

Using the index by field in a LINQ query

var q = from company in session.Index<InsuranceCompany>("name")

where company.Name == "AAA" select company;

 

foreach (InsuranceCompany company in q)

  Console.WriteLine(company.ToStringDetails(session)); // only one will match

System.OutOfMemoryException

Make sure that your process is not running as a 32-bit process on a 64-bit Windows, as a 32-bit process you will get the OutOfMemoryException at around 1.5 GB. Use the Task Manager as a way to determine if your process runs as a 64bit process. 32-bit processes has their name appended with the string “(32 bit)”, also do not use the “Visual Studio Hosting Process” – it’s in your projects Debug options - if it is running as a 32 bit process. If your project is using .NET 4.5 make sure that you do not have the option “Prefer 32 bit” set. If this isn’t set but your process still is 32 bit then change to use .NET 4.0 as a work around. If you absolutely need to run your process as 32-bit then tell VelocityDB to limit its caching by setting:  DataCache.MaximumMemoryUse = 1100000000; to limit the memory usage.

Limiting graph of objects in memory

When an object is opened by a session object, all object referenced by that object are also brought into memory. In some cases that isn’t desired. You can limit the size of such graphs by using the BTree collections which avoids bringing in all the objects contained in a BTree. A BTree avoids bringing in all referenced objects by not having straight forward C# object references everywhere; instead some references are replaced by the object identifier of the referenced object, as in:

    internal UInt64 comparisonByteArrayId;

    internal UInt64[] keysArray;

    internal UInt64[] valuesArray;

 

Here each UInt64 is actually the Id of some persistent object. The BTree fetches such objects on demand:

public override Key GetKey(int index)

{

  if (IsPersistent && UseAlternateKeys == false)

  {

    SessionBase session = this.Session;

    IOptimizedPersistable pObj = session.Open(keysArray[index]);

    return (Key)(object)pObj;

  }

  else

    return keysArrayAlternate[index];

 }

 

This is the most efficient way of handling arrays of object references, for single non array referenced VelocityDB provides WeakIOptimizedPersistableReference<T> as in:

 

aMan.spouse = new WeakIOptimizedPersistableReference<VelocityDbSchema.Person>(aWoman);

 

to get the value use public T GetTarget(bool update, SessionBase session)

Lazy load of object references

Another way of limiting what gets loaded when an object is open is the LazyLoadMembers property on OptimizedPersistable

/// <summary>

/// By default all fields are loaded when opening a persistent object but an option is provided to load members on demand (lazy loading).

/// </summary>

public virtual bool LazyLoadFields

{

  get

  {

    return false;

  }

}

When a class uses lazy loading of fields, each field access must make sure the field is loaded first.

    public LazyLoadPropertyClass MyRef

    {

      get

      {

        LoadFields();

        return myRef;

      }

      set

      {

        LoadFields();

        Update();

        myRef = value;

      }

    }

Specifying depth to load at object open

An alternative to the lazy load property is to specify depth to load at object open.

LazyLoadByDepth lazy = (LazyLoadByDepth)session.Open(id, false, false, 0); // load only the root of the object graph

Session caching of databases, pages and slots

Each session object maintains a cache of databases, pages and slots. The caching is mostly using weak references. Database pages also have a strong reference cache which is released when available memory is low. Some objects are also cached with strong references. By default objects are not cached with strong references but if an object’s class overrides the Cache property, object caching may happen.

 

Strong reference caching can be disabled by creating the session instance with a parameter that disables caching.

Avoid having strong references to persistent object between transactions since a strong referenced object cannot be updated in case the object was updated by another session. Look up persistent objects from scratch in each new transaction so that stale objects can be avoided.

Diagnostics

When you notice that something isn’t the way it should be, maybe something is taking longer than expected, there is useful option you can turn on that logs all activities related to all database files or files of selected databases.

To turn on tracing for a specific database (in this case database 55), use SessionBase api:  session.SetTraceDbActivity(55);

To turn on tracing of all databases use: session.SetTraceAllDbActivity();

Handling exceptions thrown by VelocityDB

A VelocityDB application should handle exceptions thrown by the VelocityDB kernel.

try

{

  using (SessionNoServer session = new SessionNoServer(systemDir))

  {

    session.BeginRead();

    …

    session.Commit();

  }

}

catch (Exception ex)

{

  Console.WriteLine(ex.ToString());

}

 

Here is a list of the current possible VelocityDB exceptions:

AlreadyInTransactionException

DatabaseAlreadyExistsException

DatabaseReadLockException

DesKeyMissingException

FieldDoesNotExistException

IndexDatabaseNotSpecifiedException

IndexDatabaseOrBTreeMissingException

IndexDatabaseSpecifiedForGlobalIndexException

InternalErrorException

InvalidChangeOfDatabaseLocation

InvalidChangeOfDefaultLocationException

MaxNumberOfDatabasesException

NotInTransactionException

NullObjectException

ObjectNotInSameDatabaseAsOidShortCollectionException

OpenDatabaseException

OptimisticLockingFailed

PageDeadLockException

PageReadLockException

PageUpdateLockException

PersistedObjectExcpectedException

RequestedPlacementDatabaseNumberNotValidException

RequestedPlacementPageNumberNotValidException

SubscriptionsNotAvailableWithNoServerSessionException

SystemDatabaseNotFoundWithReadonlyTransactionException

TryingToBeginReadOnlyTransactionWhileInUpdateTransactionException

TryingToDeleteDeletedDatabaseException

UnexpectedException

UniqueConstraintException

UpdateLockFailedException

WeakReferenceMustBePersistentException

Putting type definitions of persist able types in separate project class library

We recommend that you put all type detentions of classes and other types that you plan to create persistent instances of in the project VelocityDbSchema. That way the VelocityDbBrowser gets access to your types so that it can display data of these types. If you don’t add the types to this project, provided with the sample VelocityDb.sln, then you need to add a reference from the VelocityDbBrowser to the project where the type definitions are placed. VelocityDbBrowser is provided with C# source, you can build it using the VerlocityDb.sln.

Do not define your persistent types in an .exe project because then you will not be able to browse your objects with VelocityDbBrowser. Always define these types in a library project. The Verify application also requires access to all your persistent type definitions.

Controlling the in memory page and object caching

Be default VelocityDB tries to cache database pages whenever there is enough available RAM memory. You can control how much enough RAM memory is by API on the DataCache object that is accessed from a session object by the property ClientCache. You can also completely turn off page caching by specifying this as one of the optional parameters when creating a session. Object caching is off by default but you can enable it for specific types by overriding the OptimizedPersistable public virtual CacheEnum Cache property.

 

Verifying all objects and references

The Verify.exe application provided in the sample solution can be used to verify your data. Run Verify.exe and specify as command line parameter the directory where your databases are located. Verify.exe walks through all objects and opens all their references and it iterates though all enumeration types such as BTreeSet and other collections. An exception will be thrown if a failure is found. You can also verify all objects by API using SessionBase.Verify().

Database backup and restore

Database backup is an option on each DatabaseLocation, you can request that all databases of a specified DatabaseLocation are backed up to a backup DatabaseLocation. This API is currently only supported with ServerClientSession.

Backup

The following code create a backup DatabaseLocation for the default DatabaseLocation (the one containing the system database 0, 1, 2, and 4)

 

using (ServerClientSession session = new ServerClientSession(systemDir, Dns.GetHostName()))

{

  const bool isBackupLocation = true;

  session.BeginUpdate();

  DatabaseLocation backupLocation = new DatabaseLocation(Dns.GetHostName(),

      "c:/NUnitTestDbsBackup",

      (uint)Math.Pow(2, 24),

      UInt32.MaxValue,

      session,

      false,

      PageInfo.encryptionKind.noEncryption,

      isBackupLocation,

     session.DatabaseLocations.Default());

  session.NewLocation(backupLocation);

  session.Commit();

}

 

From now on, every time a default DatabaseLocation database is created/updated, it will be backed up to the backup DatabaseLocation.

Restore

The following code restores the default DatabaseLocation from its backup.

using (SessionNoServer session = new SessionNoServer(systemDir))

{

  session.BeginUpdate();

  DatabaseLocation backupLocation = new DatabaseLocation(Dns.GetHostName(), "c:/NUnitTestDbsBackup", (uint)Math.Pow(2, 24), UInt32.MaxValue, session,

    false, PageInfo.encryptionKind.noEncryption, true, session.DatabaseLocations.Default())

  session.RestoreFrom(backupLocation, DateTime.Now);

  session.Commit(false, true);

}

CopyAllDatabasesTo

A fast and easy way to backup your databases is to use SessionBase.CopyAllDatabasesTo, as in

      using (ServerClientSession session = new ServerClientSession(systemDir))

      {

        session.CopyAllDatabasesTo(copyDbsDir);

      }

 

      using (SessionNoServer session = new SessionNoServer(copyDbsDir))

      {

        session.BeginRead();

        session.Verify();

        session.Commit();

      }

ExportToCSV and ImportFromCsv

SessionBase provides a to and from CSV file option. The CSV export files contains one csv file for each Type stored in the databases.

VelocityDbServer.exe

This is a server process that manages data transfer between client and server hosts. It also handles the page/database locking and manages a shared cache. The use of this server process is optional but is requires in order to distribute databases and the server is also required when page level locking is requested.

VelocityDbServer.exe is installed as a service unless you did the install choosing VelocityDbNoServer.exe. You can configure it using the Windows Computer Management

If you don’t want it running as a service, you can remove it after stopping by the command: sc delete VelocityDbServer. Or simply change the “Automatic” start to “Manual” start.

The server can be started from command line: VelocityDbServer true 25

Substitute “25” with how many worker threads you want it to use. The process runs as background process. A non-service VelocityDbServer is stopped by using the Task manager.

In order to distribute databases to multiple hosts, you need to install VelocityDb on each host where you want to place databases.

The VelocityDbServer is communicating on tcp/ip port number: 7031.

Make sure that your Firewall lets VelocityDbServer listen/talk to other hosts with VelocityDbServer running on them.

If you are experiencing issues with the VelocityDbServer, it may help to look at the VelocityDBServerLog in the Event log, as in

Changing the default SessionBase. BaseDatabasePath in a VelocityDbServer

Edit VelocityDbServer.exe.config (in Program Folder (x86)\VelocityDB)

<?xml version="1.0"?>

<configuration>

<startup><supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0"/></startup>

  <appSettings>

    <add key="BaseDatabasePath" value="c:\Databases"/>

    <add key="DoWindowsAuthentication" value="false"/>

    <add key="NumberOfWorkerThreads" value="30"/>

  </appSettings>

</configuration>

Enabling Windows Authentication

By default Windows Authentication is now disabled when connecting to a VelocityDBServer. It is disabled by default due to a slight performance cost when connecting to a server and also due to issues with making it work with Windows 8.1 clients.

Edit VelocityDbServer.exe.config (in Program Folder (x86)\VelocityDB)

<?xml version="1.0"?>

<configuration>

<startup><supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0"/></startup>

  <appSettings>

    <add key="BaseDatabasePath" value="c:\Databases"/>

    <add key="DoWindowsAuthentication" value="false"/>

    <add key="NumberOfWorkerThreads" value="30"/>

  </appSettings>

</configuration>

 In each of your clients set

SessionBase.DoWindowsAuthentication = true;

Why installation ends up in Program Files (x86) instead of Program Files?

An issue is that Install Shield LE 2013 does not support 64bit installers so installation ends up in Program Files (86) instead of Program Files.

We use Install Shield LE 2013 which comes with Visual Studio. For VelocityDbServer service install we create a merge module using WiX Toolset.

The latest version with Visual Studio 2013 is supposed to support 64 bit installers but we have not figured out how to do it yet. Be patient, we will solve it eventually or let us know how it’s accomplished!

Windows Phone 8

On this platform Microsoft doesn’t support the API that lets you instantiate an object without a default constructor. For this reason you need to add a default constructor to all your persistent classes, i.e.

[Serializable]

internal partial class FreeSpace : OptimizedPersistable

{

    public long offset;

    public long size;

#if WINDOWS_PHONE

    public FreeSpace()

    {

    }

#endif

….

Windows Phone 8 applications need to reference our DLL named VelocityDbWindowsPhone.dll.

You can store read only database files on a Windows 8 Phone SD card. To enable our code to look for such files, set session.UseExternalStorageApi = true;

VelocityDbServer, ServerClientSession and page encryption are not supported with Windows Phone 8.

A sample application named WindowsPhoneSDCard is provided in the download solution. We have not yet tested the SD card feature due all using iPhone, let us know if it works. Maybe emulator can support putting files on a fake SD card?

Setting Up the sample Web Site (VelocityWeb) on a hosting web site (in this case GoDaddy)

The VelocityDB sample solution contains a sample web application using VelocityDB, here we show you how to deploy this application.

Transfer all the files to your hosting account

Copy the entire directory named VelocityWeb to the root of your hosting directory. We use FileZilla (free software).

Login to your hosting provider to enable write access to a few of the directories in the application

Create an application root virtual directory for the new web application

Wait a few minutes then point your browser at your web application

If you transferred your application directory with databases then install your databases in their new loacftion.

If all is well, you are done, access the application and the databases!