User’s Guide to VelocityDB
This guide compliments the sample programs and the API reference provided on
our site. Developers should review this in order to better understand how to a
build a VelocityDB-integrated application.
Contents
Opening the samples solution,
VelocityDB.sln 2
Selecting the correct VelocityDB
Session Class 2
Composite Object Identifier 3
DatabaseLocation 3
Moving Databases to a different host
and/or directory 4
Databases 4
Pages 4
Transactions 4
How to enable persistent objects of
some class 5
Persistent placement of objects 5
Customizing object placement (most of
you can skip this part) 5
Looking up objects 6
Updating persistent objects 7
Deleting (unpersisting) persistent
objects 7
Using the provided BTree collections 8
Indexes 9
Class level index 9
Using
a class level index 9
Index
by a field 9
Using the index by field in a LINQ
query 10
Limiting graph of objects in memory 10
Lazy load of object references 10
Specifying depth to load at object
open 11
Session caching of databases, pages
and slots 11
Diagnostics 11
Handling exceptions thrown by
VelocityDB 11
Putting type definitions of persist
able types in separate project class library 12
Controlling the in memory page and
object caching 12
Database backup and restore 13
Backup 13
Restore 13
Using Visual Studio 2012, open %USERPROFILE%\My
Documents\VelocityDB\VelocityDB.sln
You can also start it by using the shortcut in the programs
start menu.
The most important class for users of VelocityDB is the Session
class which contains the Transaction Control API, the Persistence API,
the Data Cache API and more. VelocityDB provides three session types
and does not limit usage. Your application can utilize all of them as
necessary:
·
ServerClientSession
- Used for distributed databases or when clients are hosted remotely.
// initial DatabaseLocation directory and hostname
using (ServerClientSession
session = new ServerClientSession("c:\\Databases", "DbServer"))
{
session.BeginRead();
//
your code here
session.Commit();
}
·
SessionNoServer - Client and data are on the same host
(unless it is a web application)
using (SessionNoServer
session = new SessionNoServer("c:\\Databases"))
{
session.BeginRead();
//
your code here
session.Commit();
}
·
SessionNoServerShared - Client and data are on the same
host (unless it is a web application) with mutex protection
using (SessionNoServerShared
session = new SessionNoServerShared
("c:\\Databases"))
{
session.BeginRead();
//
your code here
session.Commit();
}
The session class ServerClientSession
is appropriate if the application will distribute data and/or clients across
multiple hosts (where the clients are not just clients of a web site).
Otherwise, SessionNoServer or SessionNoServerShared are
appropriate. Of the two, the best choice is dependent upon the architecture of
the application.
Use SessionNoServerShared when the application must
share a client-side cache between multiple threads. This may be the case for a
web site that has limited RAM resources while also having a large amount of persistent
data to manage. Otherwise, use SessionNoServer . The difference
between these two classes is that the API on SessionNoServerShared
is mutex-protected by lock(this){} blocks. If you are not
sharing a session between multiple threads then you are slightly better-off
using SessionNoServer
as it avoids the performance penalty of SessionNoServerShared
It is recommended that a session is reused for multiple
transactions since that will provide some caching benefits and also avoids some
setup time, especially with ServerClientSession.
The benefits of having the performance counter is
better tracking of memory used which effects page and object caching. Another
way to avoid the performance counter is to create the session with caching
disabled (see constructor parameters)
All normal VelocityDB persistent objects have an associated
composite object identifier. It is encoded as a UInt64 with
three composite parts; a database number (upper 32 bits), a page number and a
slot number. The Id property returns an objects encoded object identifier
and the Oid
property returns the decoded object identifier as the struct Oid.
A reference to a persistent object is persistently stored as an object
identifier, it is normally a UInt64 but it can also be using a short
object identifier, a UInt32, when the reference is to another object
within the same database. The decoded short reference as a struct is
OidShort.
Use the special OidShort collection classes and tag object references
with the attribute [UseOidShort] as in:
[Serializable]
[UseOidShort]
internal class Recovery : OptimizedPersistable
and for a specific member:
[UseOidShort]
public VelocityDbListOidShort<FreeSpace>
theArray;
This is a directory on some host. The initial DatabaseLocation
is created when you create your first persistent object. You specify the
directory when you create the session class. You can create additional database
locations like:
using (ServerClientSession session = new ServerClientSession(systemDir,
Dns.GetHostName()))
{
session.BeginUpdate();
DatabaseLocation otherLocation = new DatabaseLocation(Dns.GetHostName(), location2Dir,
locationStartDbNum, locationEndDbNum, session, true,
0);
otherLocation = session.NewLocation(otherLocation);
session.Commit();
}
You need to commit the initial DatabaseLocation before
other sessions (clients) can access it.
If you copy/move a directory containing your boot up
database location then you need to update the DatabaseLocation containing
the boot up host and path. We plan to introduce a convenience function for this
but currently it can be done using the following code. Other database locations
can be updated in a similar fashion but does not require the special parameters
for BeginUpdate.
using (SessionNoServer
session = new SessionNoServer(systemDirDeploy))
{
session.BeginUpdate(false, true);
DatabaseLocation
bootLocation = session.DatabaseLocations.LocationForDb(0);
DatabaseLocation
locationNew = new DatabaseLocation(Dns.GetHostName(),
systemDirDeploy, bootLocation.StartDatabaseNumber,
bootLocation.EndDatabaseNumber, session,
bootLocation.CompressPages,
bootLocation.PageEncryption, bootLocation.IsBackupLocation,
bootLocation.BackupOfOrForLocation);
bootLocation =
session.NewLocation(locationNew);
session.Commit(false);
}
A database corresponds to a file within a DatabaseLocation.
The file name of a Database is <database number>.odb. When
you create your first persistent data, three system databases are created:
·
0.odb
Contains a log of update transactions and the recovery mechanism data.
·
1.odb
Contains the schema objects
·
2.odb
Contains the DatabaseLocation objects.
These system databases must be committed by a session before
other sessions can use them. This is true for any new database; a database must
be committed before other sessions can access it.
A new uncommitted Database is named <database
number>.new
and an uncommitted deleted Database is named <database
number>.del.
A Database can be created explicitly using session
API or implicitly by placing a new persistent object with database part of the object
identifier corresponding to an unallocated database number.
A VelocityDB page can contains one or more persistent
objects. The size of a Page can vary dynamically. A page is
stored within a Database file. Each Page has a PageInfo
header that contains information about a page. A Page can optionally be
encrypted and/or compressed.
All interaction with databases and persistent object require
an active transaction. With VelocityDB we provide two kinds of transactions;
update and read only. With an update transaction, you are permitted to update
and add persistent data. With a read only transaction, an exception will be
thrown by VelocityDB if you try to update persistent data. Only one concurrent
transaction per session is permitted. A transaction is started and committed by
API on the session classes.
An application may examine in memory persistent object without being in a
transaction but an exception will be thrown if any persistent operation is
requested like reading a page from a database.
public virtual void
BeginRead(bool doRecoveryCheck = true)
public virtual void
BeginUpdate()
public virtual void Commit(bool doRecoveryCheck = true)
public virtual void Abort()
There are two major choices for enabling persistence.
1.
Make your data model class a subclass of OptimizedPersistable
2. Implement
the interface IOptimizedPersistable.
See the sample class PersistenceByInterfaceSnake as a template
for how to implement the required interface API.
These two ways of enabling persistence can be mixed, some
classes may implement the interface and others may be subclasses of OptimizedPersistable.
Objects of ValueType and arrays are embedded within a
parent persistent object.
In addition, almost any type of object, except Delegate and
Pointer instances, can be made persistent but this way is not very efficient
due to requiring use of a fairly inefficient ConditionalWeakTable
internally by VelocityDB due to such objects not maintaining an object
identifier s as a field.
OptimizedPersistable implements IOptimizedPersistable.
The placement (location) of persistent objects affects
performance and locking. It is therefore important to make decisions about
where to place an object when making it persistent. Once an object has been
persisted, it remains in the same location for its persistent life time. You
can decide how many objects you want on a single page. For slightly improved
storage, require that a page only may contain objects of a specific type. Also
fixed size objects (ones with no contained variable size arrays) can further improve
object store efficiency. Several ways of controlling the placement when
persisting object are provided. First on IOptimizedPersistable
the following helps guide the placement:
UInt16
ObjectsPerPage
{
get;
}
The recommended way of persisting objects is using
the SessionBase api:
public UInt64
Persist(object obj)
When this api is used, each type is stored in its
own database.
In addition the IOptimizedPersistable interface
contains API intended for customizing how fields of an object being persisted
are to be persisted (including where to place).
UInt64
Persist(Placement place, SessionBase session, bool
persistRefs = false, bool
disableFlush = false);
UInt64
Persist(SessionBase session, IOptimizedPersistable placeHint, bool persistRefs = false,
bool disableFlush = false);
|
|
for (int i
= 0; i < numberOfPersons; i++)
{
person = new Person();
person.Persist(session, person);
}
|
|
|
for (int i
= 0; i < numberOfPersons; i++)
{
person = new Person();
if
(priorPerson == null)
priorPerson = person;
person.Persist(session, priorPerson); //
use prior person as object to persist near
priorPerson = person;
}
|
The
second way of controlling the placement while persisting an object is by using
persistent or transient instances of the Placement
class.
public
Placement(UInt32 db, UInt16 page = 1, UInt16 slot
= 1, UInt16 objectsPerPage = 10000, UInt16 pagesPerDatabase = 10000, bool persistRefs = false,
bool tryOtherDatabaseIfLockConflict = true, UInt32
maxNumberOfDatabases = UInt32.MaxValue, bool allowOtherTypesOnSamePage = true, bool
flushFullPages = true)
public
Placement(SessionBase session, IOptimizedPersistable placementProviderObject, IOptimizedPersistable objectToPlace, bool persistRefs = false,
UInt32 maxNumberOfDatabases = UInt32.MaxValue, bool
flushFullPages = true)
There is also additional API on Placement
for fine tuning the placement. An instance of Placement is
used as parameter to the IOptimizedPersistable Persist API mentioned
above.
Sometimes it an advantage to put all related
objects in a single database because then 32bit, OidShort, object references can be
used instead of full 64 bit, Oid, object references. A short object reference contains
only a page and slot part (16 bit each). Such references use less storage space
and if only short references are used within a database, such a database can
easily be cloned since it’s database number isn’t hard coded anywhere within
the database. Short references are not automatically used when you place
objects this way. The application must explicitly request it in the class
definition by using the attribute [UseOidShort]. There are
also special short references versions of the provided BTree collections. The
application needs to use those instead of the long reference BTree collections
when you want all objects within a database to use short references.
How to optimally place/persist objects is application
dependent. The sample programs provided try to illustrate some of many use
cases for object placement.
The most efficient way is to have one or a few root objects
that you look up by the object identifier as in:
ImdbRoot imdbRoot = (ImdbRoot)session.Open(ImdbRoot.PlaceInDatabase,
1, 1, false);
When you open an object this way, all objects referenced
by the object is also connected to the object so then to reach related objects
all you need to do is navigate to related objects such as in:
imdbRoot.ActingByNameSet
BTreeSet<Word> wordSet = indexRoot.lexicon.wordSet;
Another way to lookup objects is by using a LINQ
query such as:
Database
db = session.OpenDatabase(ComputerFileData.PlaceInDb);
var
result = (from ComputerFileData
computerFileData in db.AllObjects<ComputerFileData>()
where computerFileData.FileID == 500000
select computerFileData).First();
or you can accomplish the same lookup without using
LINQ as:
ComputerFileData computerFileData = null;
Database
db = session.OpenDatabase(ComputerFileData.PlaceInDb);
foreach (Page page in db)
if (page.PageNumber > 0 &&
(computerFileData == null ||
computerFileData.FileID != 500000))
foreach (IOptimizedPersistable
o in page)
{
computerFileData = o as ComputerFileData;
if (computerFileData != null)
if (computerFileData.FileID == 500000)
break;
}
The third way is by looking up from a collection
(usually a BTree) as in:
doc.WordHit.TryGetValue(word,
out wordHit)
VelocityDB need to be notified when you want a change to an
object to be persisted. The safest way to do this, is to create define property
code for every field your application data objects have, such as:
public Person BestFriend
{
get
{
return bestFriend;
}
set
{
Update();
bestFriend = value;
}
}
Use OptimizedPersistable.Unpersist or Page.UnpersistObject or SessionBase.DeleteObject.
You can override the default implementation of public virtual void Unpersist(SessionBase
session, bool disableFlush = true), i.e.
public override void
Unpersist(SessionBase session, bool
disableFlush = true)
{
if (id
== 0)
return;
if
(comparisonByteArrayId != 0)
{ing)(((((((
comparisonBytesTransient = (BTreeByteArray)session.Open(comparisonByteArrayId);
comparisonBytesTransient.Unpersist(session, disableFlush);
comparisonByteArrayId = 0;
}
nodeList.Unpersist(session, disableFlush);
base.Unpersist(session,
disableFlush);
}
Just about all object oriented applications need to
use collections. VelocityDB provides BTree collections which are similar to
BTree’s of the variety B*. A BTree is a collection where the added objects are
sorted. An application can define the sort order by defining a subclass of VelocityDbComparer<Key> or by using the
class CompareByField<Key>, a collection may also
have a null comparator in which case the
objects are ordered by the object identifier or by the ValueType ordering as defined
by the objects public override
int CompareTo(object
obj) implementation. The BTree comes in a few varieties, a key only
version and a key value version. They also have a long object Id (db-page-slot)
version and a short Id (page-slot) version. A BTree can be used with comparisonByteArray data
which is used to cache object key data within the BTree nodes so that when a
binary search takes place we can avoid opening objects to compare. When you use
the predefined class CompareByField<Key> it is easy to add comparisonByteArray data to the
BTree nodes, you just specify how many bytes per object it should be and whether
the cached node byte contains the entire data being compared when deciding if
one object is less, equal or greater compared to another. If you customize
building your own comparstor, managing the comparisonByteArray becomes a little
trickier; on the compare class you need to define SetComparisonArrayFromObject as in:
public override void
SetComparisonArrayFromObject(Word key, byte[] comparisonArray, bool
oidShort)
{
Int32 hashCode = key.aWord.GetHashCode();
Buffer.BlockCopy(BitConverter.GetBytes(IPAddress.HostToNetworkOrder(hashCode)), 0,
comparisonArray, 0, comparisonArray.Length);
}
In this case we are sorting by the hash code of a
string, the corresponding compare function in this case looks like:
public override int Compare(Word a, Word b)
{
UInt32 aHash = (UInt32)
a.aWord.GetHashCode();
UInt32 bHash = (UInt32)
b.aWord.GetHashCode();
int value = aHash.CompareTo(bHash);
if (value != 0)
return value;
return a.aWord.CompareTo(b.aWord);
}
A problem here is that a String GetHashCode()
returns different values on a 32 bit platform then a 64 bit platform. To make
your data cross platform compatible don’t use the string GetHashCode, instead
build your own string hash code function. We do so in the VelocityDB build in
class HashCodeComparer<T>.
Btree
classes provided:
·
BTreeSet<Key>
·
BTreeSetOidShort<Key>
·
BTreeMap<Key, Value>
·
BTreeMapOidShort<Key, Value>
Sample usage:
public
Lexicon(ushort nodeSize, HashCodeComparer<Word>
hashComparer, SessionBase session)
{
wordSet = new BTreeSet<Word>(hashComparer, session, nodeSize);
}
Indexes is a simplified, automated, way of implicitly
defining and keeping BTreeSets up to date when objects are added, deleted and
updated. An index is defined by using the class or field Index attribute.
Indexes for a persistent Type is stored in its own system selected database,
the range of databases used is between 66000 up to 66000 + the number of Types
and versions of a type that your application store persistently. An object
gets added to its indexes when the object is persisted. When an indexed object
is updated, its indexes get updated when the page of the objects gets flushed to
disk. An object is removed from its indexes when it is unpersisted. If you
want to index objects separately for each Database, tag the class or field with
the attribute [OnePerDatabase]
When you want an index with compound keys, like order by lastName
and then if two or more lastnames are equal by firstName and if
two or more firstNames are equal, order these otherwise equal objects by
yet another field name and so on. We currently only allow one class level index
(by multiple compound keys) per class.
[Index("modelYear,brandName,modelName,color")]
public abstract class Vehicle : OptimizedPersistable
{
string
color;
int
maxPassengers;
int
fuelCapacity; // fuel capacity in liters
double
litresPer100Kilometers; // fuel cunsumption
DateTime
modelYear;
string
brandName;
string
modelName;
int
maxSpeed; // km/h
int
odometer; // km
You can also use the class level Index attribute
without specifying any field names; in that case the contained objects are
sorted by the default ordering of the class which is normally by Oid (Id).
To iterate all Cars in index
sorted order
foreach (Car c in
session.Index<Car>())
Console.WriteLine(c.ToStringDetails(session));
This type of index sorts all persistent instances
of a class by a field value. Note that in order to use this type of index in a
LINQ query, you need to tell us what property that returns the value of the
field. You do that by the FieldAccessor attribute as in sample
class below. The UniqueConstraint attribute can be added when you don’t
want multiple objects with the same field value in the index. The
IndexByHashCode attribute can also be added to string field indexes when you
don’t care about the sort order. Sorting by hash code is faster than sorting by
the normal string ordering.
public class InsuranceCompany : OptimizedPersistable
{
[Index]
[UniqueConstraint]
[OnePerDatabase]
string
name;
string
phoneNumber;
public
InsuranceCompany(string name, string
phoneNumber)
{
this.name
= name;
this.phoneNumber
= phoneNumber;
}
[FieldAccessor("name")]
public string Name
{
get
{
return
name;
}
}
}
var q = from
company in session.Index<InsuranceCompany>("name")
where company.Name == "AAA" select
company;
foreach (InsuranceCompany
company in q)
Console.WriteLine(company.ToStringDetails(session)); // only one will match
When an object is opened by a session object, all object
referenced by that object are also brought into memory. In some cases that
isn’t desired. You can limit the size of such graphs by using the BTree
collections which avoids bringing in all the objects contained in a BTree. A
BTree avoids bringing in all referenced objects by not having straight forward
C# object references everywhere; instead some references are replaced by the
object identifier of the referenced object, as in:
internal UInt64
comparisonByteArrayId;
internal UInt64[]
keysArray;
internal UInt64[]
valuesArray;
Here each UInt64 is actually the Id of some
persistent object. The BTree fetches such objects on demand:
public override Key GetKey(int
index)
{
if (IsPersistent && UseAlternateKeys ==
false)
{
SessionBase session = this.Session;
IOptimizedPersistable pObj =
session.Open(keysArray[index]);
return (Key)(object)pObj;
}
else
return keysArrayAlternate[index];
}
This is the most efficient way of handling arrays
of object references, for single non array referenced VelocityDB provides WeakReference<T> as in:
aMan.spouse
= new WeakReference<VelocityDbSchema.Person>(aWoman);
to
get the value use public T GetTarget(bool update, SessionBase
session)
Another way of limiting what gets loaded when an object is
open is the LazyLoadMembers property on OptimizedPersistable
/// <summary>
/// By default all fields are loaded when
opening a persistent object but an option is provided to load members on demand
(lazy loading).
/// </summary>
public virtual bool LazyLoadFields
{
get
{
return false;
}
}
When a class uses lazy loading of fields, each field access
must make sure the field is loaded first.
public LazyLoadPropertyClass
MyRef
{
get
{
LoadFields();
return
myRef;
}
set
{
LoadFields();
Update();
myRef = value;
}
}
An alternative to the lazy load property is to specify depth
to load at object open.
LazyLoadByDepth lazy = (LazyLoadByDepth)session.Open(id, false, false, 0); // load only the
root of the object graph
Each session object maintains a cache of databases,
pages and slots. The caching is mostly using weak references. Database pages
also have a strong reference cache which is released when available memory is
low. Some objects are also cached with strong references. By default objects
are not cached with strong references but if an object’s class overrides the
Cache property, object caching may happen.
Strong reference caching can be disabled by
creating the session instance with a parameter that disables caching.
Avoid having strong references to persistent object
between transactions since a strong referenced object cannot be updated in case
the object was updated by another session. Look up persistent objects from scratch
in each new transaction so that stale objects can be avoided.
When you notice that something isn’t the way it should be,
maybe something is taking longer than expected, there is useful option you can
turn on that logs all activities related to all database files or files of
selected databases.
To turn on tracing for a specific database (in this case
database 55), use SessionBase api: session.SetTraceDbActivity(55);
To turn on tracing of all databases use: session.SetTraceAllDbActivity();
A VelocityDB application should handle exceptions thrown by
the VelocityDB kernel.
try
{
using (SessionNoServer
session = new SessionNoServer(systemDir))
{
session.BeginRead();
…
session.Commit();
}
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
Here is a list of the current possible VelocityDB
exceptions:
DatabaseAlreadyExistsException
DatabaseReadLockException
DesKeyMissingException
InternalErrorException
InvalidChangeOfDatabaseLocation
InvalidChangeOfDefaultLocationException
MaxNumberOfDatabasesException
NotInTransactionException
NullObjectException
OpenDatabaseException
OptimisticLockingFailed
PageDeadLockException
PageReadLockException
PageUpdateLockException
PersistedObjectExcpectedException
RequestedPlacementDatabaseNumberNotValidException
RequestedPlacementPageNumberNotValidException
SystemDatabaseNotFoundWithReadonlyTransactionException
TryingToBeginReadOnlyTransactionWhileInUpdateTransactionException
TryingToDeleteDeletedDatabaseException
UnexpectedException
UpdateLockFailedException
WeakReferenceMustBePersistentException
We recommend that you put all type detentions of classes and
other types that you plan to create persistent instances of in the project VelocityDbSchema.
That way the VelocityDbBrowser gets access to your types so that it can display
data of these types. If you don’t add the types to this project, provided with
the sample VelocityDb.sln, then you need to add a reference from the
VelocityDbBrowser to the project where the type definitions are placed.
VelocityDbBrowser is provided with C# source, you can build it using the
VerlocityDb.sln.
Be default VelocityDB tries to cache database pages
whenever there is enough available RAM memory. You can control how much enough
RAM memory is by API on the DataCache object that is accessed from a session object
by the property ClientCache.
You can also completely turn off page caching by specifying this as one of the
optional parameters when creating a session. Object caching is off by default
but you can enable it for specific types by overriding the OptimizedPersistable public virtual
CacheEnum Cache property.
Database backup is an option on each DatabaseLocation,
you can request that all databases of a specified DatabaseLocation are backed up to a
backup DatabaseLocation.
This API is currently only supported with ServerClientSession.
The following code create a backup DatabaseLocation for
the default DatabaseLocation
(the one containing the system database 0, 1, 2, and 4)
using (ServerClientSession session = new ServerClientSession(systemDir,
Dns.GetHostName()))
{
const bool
isBackupLocation = true;
session.BeginUpdate();
DatabaseLocation backupLocation = new DatabaseLocation(Dns.GetHostName(),
"c:/NUnitTestDbsBackup",
(uint)Math.Pow(2,
24),
UInt32.MaxValue,
session,
false,
PageInfo.encryptionKind.noEncryption,
isBackupLocation,
session.DatabaseLocations.Default());
session.NewLocation(backupLocation);
session.Commit();
}
From
now on, every time a default DatabaseLocation database is created/updated, it will be
backed up to the backup DatabaseLocation.
The following code restores the default DatabaseLocation
from its backup.
using (SessionNoServer session = new
SessionNoServer(systemDir))
{
session.BeginUpdate();
DatabaseLocation backupLocation = new DatabaseLocation(Dns.GetHostName(), "c:/NUnitTestDbsBackup",
(uint)Math.Pow(2,
24), UInt32.MaxValue, session,
false, PageInfo.encryptionKind.noEncryption, true, session.DatabaseLocations.Default())
session.RestoreFrom(backupLocation, DateTime.Now);
session.Commit(false);
}