KeyValue User's Manual

Two C++ compilers are supported: Microsoft Visual C++ 2008 (for Windows) and GCC (for GNU/Linux).

Most of GNU/Linux distributions come with GCC already installed. KeyValue has been tested with version 4.x.x but other versions should work as well.

Microsoft provides different editions of Visual Studio C++ 2008. The Express Edition is available, free of charge, at

http://www.microsoft.com/express/download

Editions differ mainly in their IDEs. However, there are a few differences on compilers as well. During KeyValue's development we came across lines of code that the Professional Edition could compile while Express Edition failed. Some effort has been made to maintain compatibility with both editions.

We need additional build tools, notably, GNU make and the bash shell.

GNU/Linux users do not have to worry about most of these tools since they are probably installed by default. However, a less popular tool called makedepend is needed as well. Normally, it is part of the x11 or xorg packages. To check whether you have it or not, on a console window type:

$ makedepend

If not found, use your distribution's package system to install it or, alternatively, download and install from source code:

http://xorg.freedesktop.org/releases/individual/util

Windows users will also need those tools but, unfortunately, they are not directly available. Therefore, Cygwin (see Section 2.4) will be needed to have a GNU/Linux-compatibility layer.

Boost is a high quality set of C++ libraries for general purposes.

KeyValue depends on a few of Boost libraries notably Smart_Ptr (for shared pointers) and Date_Time (for date and time classes). All Boost libraries that KeyValue depends on are header-only. Therefore, all we need is to download and unpack Boost in the hard disk.

As of this writing, the latest Boost release is 1.44.0. KeyValue have been tested with version 1.38.0. Any newer version should work as well.

When compiling KeyValue with MSVC and Boost 1.44.0, one gets several annoying warnings which seems to be harmless. (See issue #3084655). Prefer using an earlier version of Boost (e.g. 1.43.0) or a latter one (when released) where this issue has been fixed.

Boost is available for download at its SourceForge page:

http://www.boost.org/users/download/

KeyValue is a cross platform library for GNU/Linux and Windows systems. Its build system relies on tools that are very popular on GNU/Linux systems but not on Windows. For that reason, Windows users must install Cygwin to have a GNU/Linux-like compatibility layer. Cygwin is available at

http://www.cygwin.com

During installation we have to make a few choices. Normally, default answers are fine. However, when choosing the packages to install, make sure the following items are selected:

Although installation procedures for KeyValue developers is not in the scope of this document, we anticipate here the list of extra Cygwin packages that developers must install:

Cygwin comes with a small tool called link to create file links (shortcuts). This tool is, probably, useless since there is a Windows native alternative and Cygwin also provides ln for the same purpose. Unfortunately, we must bother with link because it shares its name with the Microsoft linker, raising a conflict. A workaround is renaming link to, say, link-original. Open a Bash Shell by clicking on Start / Programs / Cygwin Gygwin Bash Shell and type the command below followed by Enter.

$ mv /usr/bin/link.exe /usr/bin/link-original.exe

On many occasions we need to type Bash Shell commands. Therefore, remember how to get a Bash shell console window and consider keeping it constantly open while working with KeyValue.

KeyValue comes with an OpenOffice Calc add-in for GNU/Linux and Windows systems. To build this add-in, one must install the OpenOffice SDK.

The OpenOffice Calc add-in has been tested with some 3.x.x versions of OpenOffice and OpenOffice SDK. It probably works with all 3.x.x versions. Users of versions 2.x.x are advised to upgrade their systems.

Download and install a OpenOffice SDK version compatible with your installed OpenOffice:

http://download.openoffice.org/sdk/index.html

KeyValue comes with an Excel add-in. To build this add-in, one must install the Excel SDK.

Only the Excel 2007 API is supported. If compatibility with this API is kept by new Excel releases, then the add-in should work with them as well. However, it does not work with Excel 2003.

Download Excel 2007 SDK from its website

http://www.microsoft.com/downloads/details.aspx?FamilyId=5272E1D1-93AB-4BD4-AF18-CB6BB487E1C4&displaylang=en

Microsoft Visual Studio 2008 users will find target sln very helpful. The command

$ make sln

creates a Microsoft Visual Studio 2008 solution (keyvalue.sln) and project files which allows for using Microsoft Visual Studio IDE, liberating users from direct calling make on Cygwin. Two configurations, debug and release, are set in keyvalue.sln.

Open keyvalue.sln. On the solution explorer (Ctrl+Alt+L), we see the projects. Initially, the start up project will be the first on alphabetic order. Change it to either excel-addin or openoffice-addin: Right click on the project name and then on Set as StartUp Project.

To configure excel-addin and openoffice-addin projects to call the appropriate applications under the debugger:

This pattern is composed by two parts: A single containing a text defining a key (i.e. verifying the necessary conditions) followed by either a single, vector or matrix, which will be the associated value. Those three possibilities are shown on the sheet Key-value patterns.

Figure 8. Key in single pattern.

There are two cases of this pattern. The first (the transpose of the second) is composed by a column vector and a matrix such that

Figure 9. Keys in vector pattern.

Furthermore, for each key in the vector, the corresponding row in the matrix defines a vector which is the value associated to the key.

There are two cases of this pattern. The first (the transpose of the second) is composed by a matrix such that

Figure 10. Keys in matrix pattern.

Furthermore, for each key in the first column, the remaining cells on the same row define a vector which is the value associated to the key.

Useful for tables, this pattern is made by one matrix M = M(i, j) , for i = 0, ..., m-1 and j = 0, ..., n-1 (with m>2 and n>2). In M we find three key-value pairs: row, column and table. There are two variants of this pattern:

The task performed on a data set is defined exclusively by its content. Indeed, excluding the Default data set (see Section 9), the value assigned to key Processor informs the action to be performed. More precisely, the bridge library implements a number of processors which perform different tasks on data sets. In any data set, the value assigned to key Processor (if present) names the processor which process the data set.

For instance, on sheet Reserved keys, the formula in B2 creates data set A which selects processor Polygon while the one in E2 creates an anonymous data set which selects processor Area. Recalculate B2 to verify on the logger the called processors:

[Debug  ] DataSet: A
  Size    : 4
  Key   #1: IsRegular
  Value #1: [Single] 1
  Key   #2: NumberOfEdges
  Value #2: [Single] 4
  Key   #3: Processor
  Value #3: [Single] Polygon
  Key   #4: Size
  Value #4: [Single] 1
[Debug  ] DataSet:
  Size    : 2
  Key   #1: Polygon
  Value #1: [Single] A
  Key   #2: Processor
  Value #2: [Single] Area

Processors that create objects are called builders (e.g. Polygon). Those that compute results are called calculators (e.g. Area).

Key Processor is optional. A data set which does not have such key is called data-only.

On sheet Reserved keys, the formula in B2 actually does not build any polygon. Indeed, for non anonymous data sets, by default KeyValue implements a lazy initialization strategy: It avoids to call processors until this is really necessary. In this case, all KEYVALUE does is creating the data set A which latterly might be used to build a polygon. In this example it will happen when we request its area in E2.

Key ProcessNow is used to change this behaviour. If ProcessNow is TRUE, then the data set is immediately processed and the result is returned to the front-end. Otherwise, KeyValue just creates and stores the data set for latter use and the result returned to the front-end is the data set name. Change cell C10 to TRUE and FALSE and check on the logger when the processor is called.

Anonymous data sets are always processed and, therefore, ProcessNow is ignored. Change F10 and check the logger.

This key is optional and when it cannot be resolved (see Section 9) assumes the value FALSE.

When the result of KEYVALUE is a vector the user may choose how this vector should be returned to the front-end: As a column vector, as a row vector or unchanged, i.e., as it is returned by the processor. For this purpose, the key VectorOutput might be assigned to "Row", "Column" or "AsIs".

This key is optional and when it cannot be resolved (see Section 9) assumes the value "AsIs".

Key Imports is optional. Its value is a vector of data set names whose keys and values are imported to the current data set. For more details see Section 9.2.

Key Export is reserved only in Default data set where it defines whether key-value pairs in Default participate in key resolution or not. (See Section 9.)

This processor builds a logger where KeyValue sends messages to. The input data set should contain the following keys:

This processor does not have any specific key. It returns the number of data sets currently stored by the repository.

This processor does not have any specific key. It returns a vector with the names of data sets currently stored in the repository.

Deletes a list of data sets from the repository. Only one key is expected:

This processor returns the number of data sets that were effectively deleted from the repository.

We can import the value of a key from another key. Moreover, the source key might be in a different data set. For this purpose, instead of providing the value for the key we should put a reference in the following format:

key-name@data-set-name

where key-name is the name of source key and data-set-name is the name of source data set. You can leave either key-name or data-set-name blank to refer to the same key or data set.

For instance, on sheet Key resolution and Default data set, key Size in data set Polygon #1 has the same value as Length in data set Small.

Figure 13. Polygon #1 imports key Size from key Length in data set Small.

Data set Polygon #2 imports key Size from data set Large.

Figure 14. Polygon #2 imports key Size from data set Large.

In data set Polygon #3 keys Size and NumberOfEdges have the same value.

Figure 15. Polygon #3 imports key Size from its own key NumberOfEdges.

We can import all key-value pairs from one or more data sets into the current one through the key Imports. The value associated to Imports must be a vector of data set names. All key-value pairs in any of those data sets are imported to the data set containing Imports.

Keys assigned locally, either directly or through references, take precedence over imported keys. Data sets assigned to key Imports are processed in the order they appear.

For instance, on sheet Key resolution and Default data set, Polygon #4 imports keys first from Large and second from Polygon #3. Only keys that are not found neither in Polygon #4 nor in Large will be imported from Polygon #3. Therefore, key NumberOfEdges is assigned locally, key Size is imported from Large and isRegular is imported from Polygon #3.

Figure 16. Use of key Imports.

After searching a key locally and in imported data sets, if the key is still not resolved, then KeyValue makes a last trial searching in a special data set named Default. To make this search effective, Default must have a key Export set to TRUE.

For instance, on sheet Key resolution and Default data set, Polygon #5 imports all keys, but Processor, from Default.

Figure 17. Polygon #5 imports all keys, but Processor, from Default.

This is the most typical example of key mapping: An object name is mapped to the object itself.

On sheet The KEYVALUE function of the example workbook, the formula in cell B8 returns the area of a certain polygon.

Figure 18. The value assigned to key Polygon, i.e., "Triangle" is mapped to an object (the triangle, itself).

Notice that value assigned to key Polygon is the text "Triangle". Rather than a text, the processor Area requires a polygon to computes its area. Therefore, when the processor asks for the value associated to key Polygon, KeyValue maps the text "Triangle" to a polygon which is returned to the processor.

More precisely, the text names a data set which is stored by the repository and defines an object. When an object is required the named data set is retrieved and passed to a processor (defined by key Processor) which creates the object. Then, the object is returned to the processor which has initiated the call.

A text is mapped to some other basic type. For instance, consider a key Month. The user might prefer to provide text values: "Jan", "Fev", ..., "Dec". On the other hand, for the processor, numbers 1, 2, ..., 12 might be more convenient.

This mapping is very similar to the lexical conversion from "Yes" to TRUE as discussed in section Section 10. The difference is that opposite to lexical conversions, flag map depends on the key. For instance, for a key Planet the text "Mar" might be mapped to something representing the planet Mars (e.g. the number 4 since Mars is the forth planet of our solar system) rather than the month of March.

Like flag map, a text is mapped into a number or date. However, the user can also provide the corresponding number or date instead of the text.

For instance, the key NumberOfEdges used in our example workbook implements a partial map. Its value must be an integer greater than 2. For some special values (not all) there correspond a polygon name (e.g "Triangle" for 3 or "Square" for 4). There is no special name for a regular polygon with 1111 edges. To see this mapping in action, go to sheet Key mappings and change the value of NumberOfEdges in data set Polygon #6 to "Triangle" or "Square" or 1111 and see its area on E2.

Figure 19. Key NumberOfEdges implements partial map. Assigning to it "Square" is the same as assign it to 4.

Finally, there is the identity map (a.k.a no map): The text which is assigned to the key is retrieved by KeyValue and passed to the caller as it is.

Inspired by spreadsheet applications, KeyValue uses five basic types:

The first two types above are C++ built-in types. The other three are library types.

KeyValue introduces value::Nothing to represent empty data.

To maximize portability, KeyValue uses ::std::string and ::boost::posix_time for strings and times, resp. These types are exported to namespace ::keyvalue where they are called string and ptime resp.

The single-value and multi-type container for basic types is value::Variant.

Values assigned to keys are not always single value::Variant object. They may be containers of value::Variant objects as well. KeyValue provides three such containers:

Actually, bridge and core library developers do not need to care about the containers above. Indeed, they are used exclusively inside KeyValue and, at some point, are converted to more standard types. More precisely, a value::Single object is converted into an appropriate basic type T while a value::Vector object becomes a ::std::vector<T> and a value::Matrix is transformed into a ::std::vector<std::vector<T> > object.

Class value::Value is a single-value and multi-type container for value::Single, value::Vector or value::Matrix objects.

return value::Value(value::Single(value::Variant(x)));

return x;

Initially a key is just a text labeling a value. However, there is more inside a key that just an string object can model. Consider the key Dates in the introductory example again. Its associated value is expected to verify certain conditions:

This kind of information is encapsulated by a certain class. In KeyValue terminology, those classes are called real keys and belong to namespace ::keyvalue::key.

The class key::Key is the base of all real keys. More precisely, real keys derive from key::Traits which derives from key::Key.

Actually, key::Traits is a template class depending on a few parameters:

Key-value pairs are stored in DataSets. This class implements methods getValue() and find() to retrieve values assigned to keys. Both methods receive a real key and processes all the information about the expected value encapsulated by the key. For instance, suppose the variable today holds the current date and consider a key BirthDates which corresponds to a vector of increasing dates, supposedly, in the past or today.

An appropriate real key is then:

key::MonotoneBoundedVector<ptime, key::Increasing, key::Leq>
  births("BrithDates", today);

Therefore, if key BirthDates belongs to a DataSet data, the result of

data.getValue(births);

is a ::boost::shared_ptr<const ::std::vector<ptime> > such that the elements of the pointed vector are in increasing order and before (less than or equal to) today. The caller does not need to check that.

Since the type returned by getValue() depends on the real key it receives, this method is a template function. The same is true for find().

The difference between getValue() and find() concerns what happens when the key is not resolved. The former method throws an exception to indicate the failure while the latter returns a null pointer. In practice, getValue() is used for compulsory keys and find() for optional ones. A typical use of find() follows:

bool foo(false);
if (bool* ptr = data.find(key::Single<bool>("Foo")))
  foo = *ptr;

In the code above foo is false unless key Foo is found in data, in which case, foo gets the given value.

All builders and calculators derive from class Processor. This class declares two pure virtual methods: getName() and getResult(). The former method returns the name under which the processor is recognized by key Processor. The second gets the result of processing a DataSet.

Actually, builders and calculators are specializations of template classes Builder and Calculator, resp. They depend on a parameter type named either OutputType (for Builder) or Tag (for Calculator). The primary role of these parameters is to distinguish between different specializations. For a Builder, OutputType also defines the type of object built.

Although a Builder specialization is uniquely identified by OutputType, it also needs to be assigned to a Tag for a unified registration process of Builders and Calculators into the global ProcessorMngr. (See Section 13 for more information on this registration process.)

Builder and Calculator specializations may implement different features which define their declaration and implementation. Therefore, different specializations of Builder (or of Calculator) might derive from different base classes and implement different methods. Rather than declare the specializations from scratch, providing all its base classes and declaring all its methods, the helper files keyvalue/mngt/DeclareBuilder.h and keyvalue/mngt/DeclareCalculator.h should be used. (See Builder and Calculator for details).

Builder and Calculator specializations might be Commands as explained below.

The abstract class Message defines the interface for all types of messages. MessageImpl is a template class which implements Message's pure virtual methods. There are six different specializations of MessageImpl with corresponding typedefs:

They define operator&() to append formated data to themselves. A typical use follows:

Info info(1);  // Create a level-1 Info message.
size_t i;
std::vector<double> x;
//...
info & "x[" & i & "] = " & x[i] & '\n';

Similarly, exception::Exception is an abstract class whose pure virtual methods are implemented by template class exception::ExceptionImpl. This template class has a member which is an instantiation of MessageImpl. The exact instantiation is provided as a template parameter of exception::ExceptionImpl. There are two specializations of exception::ExceptionImpl with corresponding typedefs:

RuntimeError indicates errors that can be detected only at runtime depending on user data. LogicError indicates errors that should be detected at development time. In other terms, a LogicError means a bug and is thrown when a program invariant fails. It is mainly used through macro KV_ASSERT as in

KV_ASSERT(i < getSize(), "Out of bound!");

To keep compatibility with exception handlers catching standard exceptions, RuntimeError derives from ::std::runtime_error while LogicError derives from ::std::logic_error.

Method exception::ExceptionImpl::operator&() provides the same functionality of MessageImpl::operator&(). Example:

if (price <= 0.0)
  throw RuntimeError() & "Invalid price. Expecting a positive number. Got " &
    price;

Other more specific exception classes are implemented to indicate errors that need special treatment. They all derive from either RuntimeError or LogicError.

Some methods of class Bridge are implemented by KeyValue itself. However there are three public methods which are left to the bridge developer. (See example in bridge-example/bridge-example/Bridge.cpp):

const char*
getCoreLibraryName() const;

Returns the name of the core library. The result also names the function called in OpenOffice Calc or Excel spreadsheets and, for that reason, must be a single word (no white spaces). Otherwise front-ends might get in trouble.

const char*
getSimpleInfo() const;

Returns a simple description (one line long) of the core library. This message is used by Excel add-in manager.

const char*
getCompleteInfo() const;

Returns a more detailed description of the core library. This message is presented by loggers when they are initialized.

Builder and Calculator specializations (the two flavors of Processor), are implemented in similar ways. Firstly, let us see how to implement the latter and then cover the differences for the former.

#include "keyvalue/mngt/Calculator.h"
// Include other required header files

namespace keyvalue {

#define TAG Foo

#define COMMAND // Must be defined if, and only if, the calculator is a Command

#include "keyvalue/mngt/DeclareCalculator.h"

const char*
Calculator<tag::Foo>::getName() const;

const char*
Calculator<tag::Foo>::getCommandName() const;

value::Value
Calculator<tag::Foo>::getValue(const DataSet& data) const;

} // namespace keyvalue

#define OBJECT_TYPE logger::Logger

shared_ptr<ObjectType>
Builder<ObjectType>::getObject(const bool& data) const;

shared_ptr<ObjectType>
Builder<ObjectType>::getObject(const DataSet& data) const;

Key functionalities belong to namespace ::keyvalue::key and all keys should be in this namespace as well.

The basics for implementing keys were explained in Section 12.3. In particular, we have seen that all keys derive from template key::Traits. For instance,

namespace keyvalue {
  namespace key {
    class MyKey : public Traits<double, StdSingle, NoMap> {
      // ...
    };
  }
}

is the prototype for a key accepting a single double value which is not mapped. (Actually, the second and third of key::Traits' parameters above are the default choices and, then, could be omitted.)

The choices of key::Traits parameters impose some methods to be implemented by derived classes. Those methods are divided in two categories: checking- and mapping- methods.

OutputType
get(const string& name) const = 0;

A help file in compressed HTML format can be associated to the Excel add-in. This file must be named manual.chm and be located in the directory containning the add-in. Then under the Excel function wizard for KEYVALUE function (or whatever is the name provided by the bridge), one can click on "Help on this function" to open manual.chm.

The file will be open by the program associated with extendion .chm at the position mapped to ID number 1000. For instance, the .chm of this user manual is called manual.chm and maps the ID number 1000 to Section 5.

Instructions on how to create .chm files and how to map ID number to anchors is outside of the scope of this document.

As explained in Section 7.1.1, Excel add-in provides special support for commands. Under the add-in menu on the menu bar, it presents a menu named after the core library from which one can call any command. The result of the command can be seen on the global logger.