KeyValue User's Manual

Two C++ compilers are supported: Microsoft Visual C++ 2008 (for Windows) and GCC (for Linux).

Most of Linux distributions come with GCC already installed. KeyValue has been tested with version 4.x.x but other versions should work as well.

Microsoft provides different editions of Visual Studio C++ 2008. The Express Edition is available, free of charge, at

http://www.microsoft.com/express/download

Editions differ mainly in their IDEs. However, there are a few differences on compilers as well. During KeyValue's development we came across lines of code that the Professional Edition could compile while Express Edition failed. Some effort has been made to maintain compatibility with both editions.

We need additional build tools, notably, GNU make and the bash shell.

Linux users do not have to worry about most of these tools since they are probably installed by default. However, a less popular tool called makedepend is needed as well. Normally, it is part of the x11 or xorg packages. To check whether we have it or not, on a console window type:

$ makedepend

If not found, use your distribution's package system to install it or, alternatively, download and install from source code:

http://xorg.freedesktop.org/releases/individual/util

Windows users will also need those tools but, unfortunately, they are not directly available. Therefore, Cygwin (see Section 2.4) will be needed to have a Linux-compatibility layer.

Boost is a high quality set of C++ libraries for general purposes.

KeyValue depends on a few of Boost libraries notably Smart Ptr (for shared pointers) and Date Time (for date and time classes). All Boost libraries that KeyValue depends on are header-only. Therefore, all we need is to download and unpack Boost in the hard disk.

As of this writing, the latest Boost release is 1.42.0. KeyValue have been tested with version 1.38.0. Any newer version should work as well.

Boost is available for download at its SourceForge page:

http://www.boost.org/users/download/

KeyValue is a cross platform library for Linux and Windows systems. Its build system relies on tools that are very popular on Linux systems but not on Windows. For that reason, Windows users must install Cygwin to have a Linux-like compatibility layer. Cygwin is available at

http://www.cygwin.com

During installation we have to make a few choices. Normally, default answers are fine. However, when selecting the packages to install, make sure the following items are selected:

Although installation procedures for KeyValue developers is not in the scope of this document, we anticipate here the list of extra Cygwin packages that developers must install:

Cygwin comes with a small tool called link to create file links (shortcuts). This tool is, probably, useless since there is a Windows native alternative and Cygwin also provides ln for the same purpose. Unfortunately, we must bother with link because it has the same name as the Microsoft linker, which raises a conflict. A workaround is renaming link to, say, link-original. Open a Bash Shell by clicking on Start / Programs / Cygwin Gygwin Bash Shell and type the following command followed by Enter.

$ mv /usr/bin/link.exe /usr/bin/link-original.exe

In many occasions we need Bash Shell commands. Therefore, remember how to get a Bash shell console window and consider keeping it constantly open while working with KeyValue.

KeyValue comes with an OpenOffice Calc add-in for Linux and Windows systems. To build this add-in, one must install the OpenOffice SDK.

The OpenOffice Calc add-in has been tested for versions 3.1.0 of OpenOffice and OpenOffice SDK. However, it probably works for all 3.x.x versions. Users of versions 2.4.x are advised to upgrade their systems.

Download and install a SDK version compatible with your installed OpenOffice version:

http://download.openoffice.org/sdk/index.html

KeyValue comes with an Excel add-in. To build this add-in, one must install the Excel SDK.

Only the Excel 2007 API is supported. If compatibility with this API is kept on new Excel releases, then the add-in should work for them as well. However, it does not work for Excel 2003.

Download Excel 2007 SDK from its website

http://www.microsoft.com/downloads/details.aspx?FamilyId=5272E1D1-93AB-4BD4-AF18-CB6BB487E1C4&displaylang=en

Microsoft Visual Studio 2008 users will find target sln very helpful. The command

$ make sln

creates a Microsoft Visual Studio 2008 solution (keyvalue.sln) and project files which allows for using Microsoft Visual Studio IDE, liberating users from direct calling make on Cygwin. Two configurations, debug and release, are set in keyvalue.sln.

Open keyvalue.sln. On the solution explorer (Ctrl+Alt+L), we see the projects. Initially, the start up project will be the first on alphabetic order. Change it to either excel-addin or openoffice-addin: Right click on the project name and then on Set as StartUp Project.

To configure excel-addin and openoffice-addin projects to call the appropriate applications under the debugger:

This pattern is composed by two parts: A single containing a text defining a key (i.e. verifying the necessary conditions) followed by a single or a vector or a matrix, which will be the associated value. Those three possibilities are shown in the sheet Key-value patterns .

Figure 8. Key in single pattern.

There are two cases of this pattern. The first (the transpose of the second) is composed by a column vector and a matrix such that

Figure 9. Keys in vector pattern.

Furthermore, for each key in the vector, the corresponding row in the matrix defines a vector which is the value associated to the key.

There are two cases of this pattern. The first (the transpose of the second) is composed by a matrix such that

Figure 10. Keys in matrix pattern.

Furthermore, for each key in the first column, the remaining cells on the same row define a vector which is the value associated to the key.

Useful for tables, this pattern is made by one matrix M = M(i, j) , for i = 0, ..., m-1 and j = 0, ..., n-1 (with m>2 and n>2). In M we find three key-value pairs: row, column and table. There are two variants of this pattern:

The task performed on a data set is defined exclusively by its content. Indeed, excluding the Default data set (see Section 9), the value assigned to key Processor informs the action to be performed. More precisely, the bridge library implements a number of processors which perform different tasks on data sets. The value assigned to key Processor gives the name of the processor for a data set.

For instance, in sheet Reserved keys, the formula in B2 selects processor Polygon while the one in E2 selects processor Area. Recalculate B2 to verify in the logger the called processors:

[Debug  ] DataSet: A
Size    : 4
Key   #1: IsRegular
Value #1: [Single] 1
Key   #2: NumberOfEdges
Value #2: [Single] 4
Key   #3: Size
Value #3: [Single] 1
Key   #4: Processor
Value #4: [Single] Polygon
[Debug  ] DataSet:
Size    : 2
Key   #1: Polygon
Value #1: [Single] A
Key   #2: Processor
Value #2: [Single] Area
[Info   ] Processor 'Area' called on anonymous data set.
[Info   ] Processor 'Polygon' called on data set 'A'.

Processors that create objects are called builders (e.g. Polygon). Those that compute results are called calculators (e.g. Area).

Key Processor is optional. A data set which does not have such key is called data-only.

In sheet Reserved keys, the formula in B2 actually does not build any polygon. Indeed, for named data sets, by default KeyValue implements a lazy initialization strategy: It avoids to call processors until this is really necessary. In this case, all KEYVALUE does is creating the data set A which latterly might be used to build a polygon. In this example it will happen when we request its area in E2.

For named data sets, key ProcessNow is used to change this behaviour. If ProcessNow is TRUE, then the data set is immediately processed and the result is returned to OpenOffice Calc. Otherwise, KeyValue just creates and stores the data set for latter use and the result returned to OpenOffice Calc is the data set name. Change cell C10 to TRUE and FALSE and check in the logger when the processor is called.

Anonymous data sets are always processed and ProcessNow has no effect. Change F10 and check the logger.

This key is optional and when it cannot be resolved (see Section 9) assumes the value FALSE.

When the result of KEYVALUE is a vector the user may choose how this vector should be presented on the spreadsheet: As a column vector, as a row vector or as it is returned by the processor. For this purpose, the key VectorOutput might be assigned to "Row", "Column" or "AsIs".

This key is optional and when it cannot be resolved (see Section 9) assumes the value "AsIs".

Key Imports is optional. Its value is a vector of data set names whose keys and values are imported to the current data set. For more details see Section 9.2.

Key Export is reserved only in Default data set where it defines whether key-value pairs in Default participate in key resolution or not. (See Section 9.)

This processor builds a logger where KeyValue sends messages to. The input data set should contain the following keys:

This processor does not have any specific key. It returns the number of data sets currently stored by the repository.

This processor does not have any specific key. It returns a vector with the names of data sets currently stored in the repository.

Deletes a list of data sets from the repository. Only one key is expected:

This processor returns the number of data sets that were effectively deleted from the repository.

We can import the value of a key from another key. Moreover, the source key might be in a different data set. For this purpose, instead of providing the value for the key we should put a reference in the following format:

key-name@data-set-name

where key-name is the source key and data-set-name is the source data set. You can leave either key-name or data-set-name blank to refer to the same key or data set.

For instance, in sheet Key resolution and Default data set , key Size in data set Polygon #1 has the same value as Length in data set Small.

Figure 13. Polygon #1 imports key Size from key Length in data set Small.

Data set Polygon #2 imports key Size from data set Large.

Figure 14. Polygon #2 imports key Size from data set Large.

In data set Polygon #3 keys Size and NumberOfEdges have the same value.

Figure 15. Polygon #3 imports key Size from its own key NumberOfEdges.

We can import all key-value pairs from one or more data sets to the current one through the key Imports. The value associated to Imports must be a vector of data set names. All key-value pairs in any of those data sets are imported to the data set containing Imports.

Keys assigned locally, either directly or through references, take precedence over imported keys. Data sets assigned to key Imports are processed in the order they appear.

For instance, in sheet Key resolution and Default data set , Polygon #4 imports keys first from Large and second from Polygon #3 . Only keys that are not found neither in Polygon #4 nor in Large will be imported from Polygon #3. Therefore, key NumberOfEdges is assigned locally, key Size is imported from Large and isRegular is imported from Polygon #3 .

Figure 16. Use of key Imports.

After searching a key locally and in imported data sets, if the key is still not resolved, then KeyValue makes a last trial searching in a special data set named Default. To make this search effective, Default must have a key Export set to TRUE.

For instance, in sheet Key resolution and Default data set , Polygon #5 imports all keys, but Processor, from Default.

Figure 17. Polygon #5 imports all keys, but Processor, from Default.

This is the most typical example of key mapping: An object name is mapped to the object itself.

In sheet The KEYVALUE function of the example workbook, the formula in cell B8 returns the area of a certain polygon.

Figure 18. The value assigned to key Polygon, i.e., "Triangle" is mapped to an object (the triangle, itself).

Notice that value assigned to key Polygon is the text "Triangle". Rather than a text, the processor Area requires a polygon to computes its area. Therefore, when the processor asks for the value associated to key Polygon, KeyValue maps the text "Triangle" to a polygon which is returned to the processor.

More precisely, the text names a data set which is stored by the repository and defines an object. When the object is required the named data set is retrieved and passed to a processor (defined by key Processor) which creates the object. Then, the object is returned to the processor which has initiated the call.

A text is mapped to some other basic type. For instance, consider a key Month. The user might prefer to provide text values: "Jan", "Fev", ..., "Dec". On the other hand, for the processor, numbers 1, 2, ..., 12 might be more convenient.

This mapping is very similar to the lexical conversion from "Yes" to TRUE as discussed in section Section 10. The difference is that opposite to lexical conversions, flag map depends on the key. For instance, for a key Planet the text "Mar" might be mapped to something representing the planet Mars (e.g. the number 4 since Mars is the forth planet of our solar system) rather than the month of March.

Like flag map, a text is mapped into a number or date. However, the user can also provide the corresponding number or date instead of the text.

For instance, the key NumberOfEdges used in our example workbook implements a partial map. Its value must be an integer greater than 2. For some special values (not all) there correspond a polygon name (e.g "Triangle" for 3 or "Square" for 4). There is no special name for a regular polygon with 1111 edges. To see this mapping in action, go to sheet Key mappings and change the value of NumberOfEdges in data set Polygon #6 to "Triangle" or "Square" or 1111 and see its area on E2.

Figure 19. Key NumberOfEdges implements partial map. Assigning to it "Square" is the same as assign it to 4.

Finally, there is the identity map (a.k.a no map): The text which is assigned to the key is retrieved by KeyValue and passed to the caller as it is.

Inspired by spreadsheet applications, KeyValue uses five basic types:

The first two types above are C++ built-in types. The other three are library types.

KeyValue introduces value::Nothing to represent empty data.

To maximize portability, KeyValue uses ::std::string and ::boost::posix_time for strings and times, resp. These types are exported to namespace ::keyvalue where they are called string and ptime resp.

The single-value and multi-type container for basic types is value::Variant.

Values assigned to keys are not always given by a single value::Variant. They may be containers of value::Variants. KeyValue provides three such containers:.

Actually, bridge and core library developers do not need to care neither about the containers above. Indeed, they are used exclusively inside KeyValue and at some point are converted to more standard types. More precisely, a value::Single is converted into an appropriate basic type a T while a value::Vector becomes a ::std::vector<T> and a value::Matrix is transformed into ::std::vector<std::vector<T> > .

Class value::Value is a single-value and multi-type container for value::Single, value::Vector or value::Matrix.

return value::Value(value::Single(value::Variant(x)));

return x;

Initially a key is just a string labeling a value. However, there is more inside a key that just a string can model. Consider the key Dates in the introductory example again. The value associated to it is expected to verify certain conditions:

This kind of information on keys is encapsulated by a certain class. In KeyValue terminology, those classes are called real key and belong to namespace ::keyvalue::key.

The class is key::Key is the base for all real keys. More precisely, real keys derive from key::Traits which derives from key::Key.

Actually, key::Traits is a template class depending on a few template parameters:

Key-value pairs are stored in DataSets. This class implements methods getValue() and also find() to retrieve values assigned to keys. Both methods receive a real key and processes all the information about the expected value encapsulated in it. For instance, suppose that the variable today holds the current date and consider a key BirthDates which corresponds to a vector of increasing dates, supposedly, in the past.

An appropriate real key is then:

key::MonotoneBoundedVector<ptime, key::Increasing, key::Leq>
  births("BrithDates", today);

Therefore, if key BirthDates is in a DataSet named data, the result of

data.getValue(births);

is a ::boost::shared_ptr< const ::std::vector<ptime> > such that the elements of the vector pointed by this pointer are in increasing order and before (less than or equal to) today. The caller does not need to check that.

Since the type returned by getValue() depends on the real key it receives, this method is a template function. The same is true for find().

The difference between getValue() and find() concerns what happens when the key is not resolved. The former method throws an exception to indicate the failure while the latter returns a null pointer. In practice, getValue() is used for compulsory keys and find() for optional keys. A typical use of find() is as follows:

bool foo(false);
if (bool* ptr = data.find(key::Single<bool>("Foo")))
  foo = *ptr;

In the code above foo is false unless key Foo is found, in which case, foo gets the given value.

All builders and calculators derive from protocol class Processor. This class declares three pure virtual methods: getName() and two overloads of getResult(). The former method returns the name under which the processor is recognized by key Processor. The first overload of getResult() takes a DataSet parameter while the second one takes a value::Variant. The difference between them is explained below.

In general, the information that processors need to perform their duties is so rich that must be stored in a DataSet. Nevertheless, in some particular cases, a single value::Variant might be enough. For instance, consider a builder that creates a curve given a few points on it. Normally, this processor needs the set of points together with interpolator and extrapolator methods. In this general case, a DataSet is necessary to hold all this information. However, when the curve is known to be constant, then a single number - the constant - is enough. Rather than creating a DataSet to store a single number, it would be more convenient if the processor could accept just this value (or more generally, a value::Variant). This is, indeed, the case.

Actually, builders and calculators are specializations of template classes Builder and Calculator, resp. They depend on a parameter type named either OutputType (for a builder) or Tag (for a calculator). The primary role of this parameter is to distinguish between different specializations. Additionally, in the case of a builder, it also defines the type of object constructed.

The header files of Builder and Calculator implement the two overloads of Processor::getResult(). The first overload forwards the call to either Builder<ObjectType>::getObject() or Calculator<Tag>::getValue(). These methods must be implemented by bridge developers whenever the template classes are used.

As explained earlier, processing a value::Variant does not always make sense. Therefore, the second overload of Processor::getResult() throws an exception. When this processing does make sense, then template classes XBuilder and XCalculator which extend their base classes Builder and Calculator, resp., must be used. Their header files reimplement the second overload of Processor::getResult() forwarding the call to XBuilder<ObjectType>::getObject() and XCalculator<Tag>::getValue(). As before, these two methods must be implemented by bridge developers whenever the extended template classes are used.

The abstract class Message defines the interface for all types of messages. MessageImpl is a template class which implements Message's pure virtual methods. There are six different specializations of MessageImpl with corresponding typedefs:

They define operator&() to append formated data to themselves. A typical use follows:

Info info(1);  // Create a level-1 Info message.
size_t i;
std::vector<double> x;
//...
info & "x[" & i & "] = " & x[i] & '\n';

Similarly, exception::Exception is an abstract class whose pure virtual methods are implemented in template class exception::ExceptionImpl. This template class has a member which is an instance of MessageImpl. The exact type of this instance is provided as a template parameter of exception::Exception. There are two specializations of exception::Exception with corresponding typedefs:

RuntimeError indicates errors that can be detected only at runtime depending on user data. LogicError indicates errors that should be detected at development time. In other terms, a LogicError means a bug and is thrown when a program invariant fails. It is mainly used through macro KV_ASSERT as in

KV_ASSERT(i < getSize(), "Out of bound!");

To keep compatibility with exception handlers catching standard exceptions, RuntimeError derives from ::std::runtime_error while LogicError derives from ::std::logic_error.

Method exception::ExceptionImpl::operator&() provides the same functionality of MessageImpl::operator&(). Example:

if (price <= 0.0)
  throw RuntimeError() & "Invalid price. Expecting a positive number. Got " &
    price;

Other more specific exception classes are implemented to indicate errors that need special treatment. They all derive from either RuntimeError or LogicError.

Some methods of class Bridge are implemented by KeyValue itself. However there are three public methods which are left to the bridge developer. (See example in bridge-example/bridge-example/Bridge.cpp):

const char*
getCoreLibraryName() const;

Returns the name of the core library. The result also names the function called in OpenOffice Calc or Excel spreadsheets and, for that reason, must be a single word (no white spaces). Otherwise front-ends might get in trouble.

const char*
getSimpleInfo() const;

Returns a simple description (one line long) of the core library. This message is used by Excel add-in manager.

const char*
getCompleteInfo() const;

Returns a more detailed description of the core library. This message is presented by loggers when they are initialized.

The two types of processors, builders and calculators, are implemented in similar ways. Let us see how to implement a calculator and then the differences for builders.

class Tag;          

template <>
const char* Calculator<Tag>::getName() const;

template <>
value::Value
Calculator<Tag>::getValue(const DataSet& data) const;

namespace keyvalue {
  namespace bridge {
    Processor* calculatorTag = &Calculator<Tag>::getInstance();
  }
}

template <>
value::Value
XCalculator<Tag>::getValue(const value::Variant& data) const;

template <>
const char* Builder<OutputType>::getName() const;

namespace keyvalue {
  namespace bridge {
    Processor* builderObjectType = &Builder<ObjectType>::getInstance();
  }
}

template <>
shared_ptr<ObjectType>
Builder<ObjectType>::getObject(const DataSet& data) const;

template <>
shared_ptr<Polygon>
Builder<ObjectType>::getObject(const DataSet& data) const {
  // ...
  if (data.mustUpdate())
    return shared_ptr<Polygon>(new Regular(nEdges, size));
  // ...
  return shared_ptr<Polygon>();
}

template <>
shared_ptr<ObjectType>
XBuilder<ObjectType>::getObject(const value::Variant& data) const;

Key functionalities belong to namespace ::keyvalue::key and all keys should be in this namespace as well.

The basics for implementing keys were explained in Section 12.3. In particular, we have seen that all keys derive from template key::Traits. For instance,

namespace keyvalue {
  namespace key {
    class MyKey : public Traits<double, StdSingle, NoMap> {
      // ...
    };
  }
}

is the prototype for a key accepting a single double value which is not mapped. (Actually, the second and third of key::Traits' parameters above are the default choices and, then, could be omitted.)

The choices of key::Traits parameters impose some methods to be implemented by derived classes. Those methods are divided in two categories: checking- and mapping- methods.

OutputType
get(const string& name) const = 0;