KeyValue-0.3 User's Manual

Two C++ compilers are supported: Microsoft Visual C++ 2008 (for Windows) and GCC (for GNU/Linux).

Most of GNU/Linux distributions come with GCC already installed. KeyValue has been tested with version 4.x.x but other versions should work as well.

Microsoft provides different editions of Visual Studio C++ 2008. The Express Edition is available, free of charge, at

http://www.microsoft.com/visualstudio/en-us/products/2008-editions/express

Editions differ mainly in their IDEs. However, there are also compiler differences. During KeyValue's development we came across lines of code that the Professional Edition did compile whereas the Express Edition failed. Some effort has been made to maintain compatibility with both editions.

We need additional build tools, notably, GNU make and the bash shell.

GNU/Linux users do not have to worry about most of these tools since they are probably installed by default. However, a less popular tool called makedepend is required. Normally, it is part of the x11 or xorg packages. To check whether you have it or not, on a console window type:

$ makedepend

If not found, use your distribution's package system to install it or, alternatively, download and install from source code:

http://xorg.freedesktop.org/releases/individual/util

Windows users will also need these tools but, unfortunately, they are not directly available. Therefore, Cygwin (see Section 2.4) will be required to provide a GNU/Linux-compatibility layer to Windows systems.

Boost is a high quality set of C++ libraries for general purposes.

KeyValue depends on a few of Boost libraries notably Smart_Ptr (for shared and intrusive pointers) and Date_Time (for date and time classes). All Boost libraries that KeyValue depends on are header-only. Therefore, all we need is to download and unpack Boost into the hard disk.

Boost is available for download at its SourceForge page:

http://sourceforge.net/projects/boost/files/boost

KeyValue is a cross platform library for GNU/Linux and Windows systems. Its build system relies on tools that are very popular on GNU/Linux systems but not on Windows. For that reason, Windows users must install Cygwin to have a GNU/Linux-like compatibility layer. Cygwin is available at

http://www.cygwin.com

During installation we have to make a few choices. Normally, default answers are fine. However, when choosing the packages to install, make sure that the following items are selected:

Although installation procedures for KeyValue developers is not in the scope of this document, we anticipate here the list of extra Cygwin packages that developers must install:

Cygwin comes with a small tool called link.exe to create file links (shortcuts). This tool is, probably, useless since there is a Windows native alternative and Cygwin also provides ln for the same purpose. Unfortunately, we must bother with link.exe because this is also the name of the Microsoft Visual Studio linker and, therefore, they conflict. A workaround is renaming Cygwin's link.exe to, say, link-original.exe. Open a bash shell by clicking on Start / Programs / Cygwin / Cygwin Bash Shell and type the command below followed by Enter.

$ mv /usr/bin/link.exe /usr/bin/link-original.exe

On many occasions we need to type bash shell commands. Therefore, remember how to get a bash shell console window and consider keeping it constantly open while working with KeyValue.

KeyValue comes with a LibreOffice Calc add-in for GNU/Linux and Windows systems. To build this add-in, one must install the LibreOffice SDK.

The LibreOffice Calc add-in has been tested with some 3.x.x versions of LibreOffice and LibreOffice SDK. It probably works with all 3.x.x versions.

Download and install a LibreOffice SDK version compatible with your installed LibreOffice:

http://www.libreoffice.org/download

KeyValue comes with an Excel add-in. To build this add-in, one must install the Excel SDK.

Only the Excel 2007 API is supported. If compatibility with this API is kept by new Excel releases, then the add-in should work with them as well. However, KeyValue does not work with Excel 2003.

Download Excel 2007 SDK from its website

http://www.microsoft.com/downloads/details.aspx?FamilyId=5272E1D1-93AB-4BD4-AF18-CB6BB487E1C4&displaylang=en

Microsoft Visual Studio 2008 users will find target sln very helpful. The command

$ make sln

creates a Microsoft Visual Studio 2008 solution (keyvalue.sln) and project files which allows for using Microsoft Visual Studio IDE, liberating users from direct calling make on Cygwin. Two configurations, debug and release, are set in keyvalue.sln.

Open keyvalue.sln. On the solution explorer (Ctrl+Alt+L), we see the projects. Initially, the start up project will be the first on alphabetic order. Change it to either excel-addin or openoffice-addin: Right click on the project name and then on Set as StartUp Project.

To configure excel-addin and openoffice-addin projects to call the appropriate applications under the debugger follow these steps:

This pattern is composed by two parts: A single containing a text defining a key (i.e. verifying the necessary conditions) followed by either a single, vector or matrix, which will be the associated value. Those three possibilities are shown on the sheet Key-value patterns.

Figure 8. Key in single pattern.

There are two cases of this pattern. The first (the transpose of the second) is composed by a column vector followed by a matrix such that

Figure 9. Keys in vector pattern.

Furthermore, for each key in the vector, the corresponding row in the matrix defines a vector which is the value associated to the key.

There are two cases of this pattern. The first (the transpose of the second) is composed by a matrix such that

Figure 10. Keys in matrix pattern.

Furthermore, for each key in the first column, the remaining cells on the same row define a vector which is the value associated to the key.

Useful for tables, this pattern is made by one matrix M = M(i, j) , for i = 0, ..., m-1 and j = 0, ..., n-1 (with m>2 and n>2). In M we find three key-value pairs: row, column and table. There are two variants of this pattern:

The task performed on a data set is defined exclusively by its content. Indeed, excluding the Default data set (see Section 9), the value assigned to key Processor informs the action to be performed. More precisely, the bridge library implements a number of processors which perform different tasks on data sets. In any data set, the value assigned to key Processor (if present) names the processor which process the data set.

For instance, on sheet Reserved keys, the formula in B2 creates data set A which selects processor Polygon while the one in E2 creates an anonymous data set which selects processor Area. Recalculate B2 to verify on the logger the called processors:

[Debug  ] DataSet: A
  Size    : 4
  Key   #1: IsRegular
  Value #1: [Single] 1
  Key   #2: NumberOfEdges
  Value #2: [Single] 4
  Key   #3: Processor
  Value #3: [Single] Polygon
  Key   #4: Size
  Value #4: [Single] 1
[Debug  ] DataSet:
  Size    : 2
  Key   #1: Polygon
  Value #1: [Single] A
  Key   #2: Processor
  Value #2: [Single] Area

Processors that create objects are called builders (e.g. Polygon). Those that compute results to be displayed on the spreadsheet are called calculators (e.g. Area).

Key Processor is optional. A data set which does not have such key is called data-only.

On sheet Reserved keys, the formula in B2 actually does not build any polygon. Indeed, for non anonymous data sets, by default KeyValue implements a lazy initialization strategy: It avoids to call processors until this is really necessary. In this case, all KEYVALUE does is creating the data set A which laterly might be used to build a polygon. In this example it will happen when we request its area in E2.

Key ProcessNow is used to change this behaviour. If ProcessNow is TRUE, then the data set is immediately processed and the result is returned to the front-end. Otherwise, KeyValue just creates and stores the data set for later use and the result returned to the front-end is the data set name. Change cell C10 to TRUE and FALSE and check the logger to see when the processor is called.

Anonymous data sets are always processed and, therefore, ProcessNow is ignored. Change F10 and check the logger.

This key is optional and when it cannot be resolved (see Section 9) assumes the value FALSE.

When the result of KEYVALUE is a vector the user may choose how this vector should be returned to the front-end: As a column vector, as a row vector or unchanged, i.e., as it is returned by the processor. For this purpose, the key VectorOutput might be assigned to "Row", "Column" or "AsIs".

This key is optional and when it cannot be resolved (see Section 9) assumes the value "AsIs".

Key Imports is optional. Its value is a vector of data set names whose keys and values are imported to the current data set. For more details see Section 9.2.

Key Export is reserved only in Default data (see Section 9) set where it defines whether key-value pairs in Default participate in key resolution or not. (See Section 9.)

This processor builds a logger where KeyValue sends messages to. The input data set should contain the following keys:

This processor does not have any specific key. It returns the number of data sets currently stored by the repository. This processor is a command and the Excel add-in provides a menu entry to call this processor.

This processor does not have any specific key. It returns a vector with the names of data sets currently stored in the repository.

Deletes a list of data sets from the repository. Only one key is expected:

This processor returns the number of data sets that were effectively deleted from the repository. This processor is a command and the Excel add-in provides a menu entry (named Reset Repository) to call this processor.

We can import the value of a key from another key. Moreover, the source key might be in a different data set. For this purpose, instead of providing the value for the key we should put a reference in the following format:

key-name@data-set-name

where key-name is the name of source key and data-set-name is the name of source data set. You can leave either key-name or data-set-name blank to refer to the current key or data set. For instance, on sheet Key resolution and Default data set, key Size in data set Polygon #1 has the same value as Length in data set Small.

Figure 13. Polygon #1 imports key Size from key Length in data set Small.

Data set Polygon #2 imports key Size from data set Large.

Figure 14. Polygon #2 imports key Size from data set Large.

In data set Polygon #3 keys Size and NumberOfEdges have the same value.

Figure 15. Polygon #3 imports key Size from its own key NumberOfEdges.

We can import all key-value pairs from one or more data sets into the current one through the key Imports. The value associated to Imports must be a vector of data set names. All key-value pairs in any of these data sets are imported to the data set containing Imports.

Keys assigned locally, either directly or through references, take precedence over imported keys. Data sets assigned to key Imports are processed in the order they appear.

For instance, on sheet Key resolution and Default data set, Polygon #4 imports keys first from Large and second from Polygon #3. Only keys that are not found neither in Polygon #4 nor in Large will be imported from Polygon #3. Therefore, key NumberOfEdges is assigned locally, key Size is imported from Large and isRegular is imported from Polygon #3.

Figure 16. Use of key Imports.

After searching a key locally and in imported data sets, if the key is still not resolved, then KeyValue makes a last trial searching in a special data set named Default. To make this search effective, Default must have a key Export set to TRUE.

For instance, on sheet Key resolution and Default data set, Polygon #5 imports all keys, but Processor, from Default.

Figure 17. Polygon #5 imports all keys, but Processor, from Default.

This is the most typical example of key mapping: An object name is mapped to the object itself.

On sheet The KEYVALUE function of the example workbook, the formula in cell B8 returns the area of a certain polygon.

Figure 18. The value assigned to key Polygon, i.e., "Triangle" is mapped to an object (the triangle, itself).

Notice that value assigned to key Polygon is the text "Triangle". Rather than a text, the processor Area requires a polygon to computes its area. Therefore, when the processor asks for the value associated to key Polygon, KeyValue maps the text "Triangle" to a polygon which is passed over to the processor.

More precisely, the text names a data set which is stored by the repository and defines an object. When an object is required the named data set is retrieved and passed over to a processor (defined by key Processor) which creates the object. Then, the object is returned to the processor which has initiated the call.

A text is mapped to some other basic type. For instance, consider the key Month. The user might prefer to provide text values: "Jan", "Fev", ..., "Dec". On the other hand, for the processor, numbers 1, 2, ..., 12 might be more convenient.

This mapping is very similar to the lexical conversion from "Yes" to TRUE as discussed in section Section 10. The difference is that opposite to lexical conversions, flag map depends on the key. For instance, for the key Planet the text "Mar" might be mapped to something representing the planet Mars (e.g. the number 4 since Mars is the forth planet of our solar system) rather than the month of March.

Like flag map, a text is mapped into a number or date. However, the user can also provide the corresponding number or date instead of the text.

For instance, the key NumberOfEdges used in our example workbook implements a partial map. Its value must be an integer greater than 2. For some special values (not all) there correspond a polygon name (e.g "Triangle" for 3 or "Square" for 4). There is no special name for a regular polygon with 1111 edges. To see this mapping in action, go to sheet Key mappings and change the value of NumberOfEdges in data set Polygon #6 to "Triangle" or "Square" or 1111 and see its area on E2.

Figure 19. Key NumberOfEdges implements partial map. Assigning to it "Square" is the same as assign it to 4.

Finally, there is the identity map (a.k.a no map): The text which is assigned to the key is retrieved by KeyValue and passed to the caller as it is.

The five so called basic types are:

Additionally, KeyValue introduces value::Nothing to represent empty data.

To maximize portability, KeyValue uses ::std::string and ::boost::posix_time for strings and times, resp. These types are exported to namespace ::keyvalue where they are called string and ptime resp.

The single-value and multi-type container for basic types (excluding unsigned int) is value::Variant.

The value assigned to a key is not necessarily a single value::Variant. It may be a container of value::Variants as well. KeyValue provides three such containers:

Actually, bridge and core library developers do not need to care about these containers. Indeed, they are used exclusively inside KeyValue and, at some point, are converted to more standard types. More precisely, a value::Single is converted into an appropriate basic type T while a value::Vector becomes a ::std::vector<T> and a value::Matrix is transformed into a ::std::vector<::std::vector<T>>.

Class value::Value is a single-value and multi-type container for value::Single, value::Vector or value::Matrix.

return value::Value(value::Single(value::Variant(x)));

return x;

Initially a key is just a text labeling a value. However, there is more inside a key that just a string can model. Consider the key Dates in the introductory example again. Its associated value is expected to verify certain conditions:

This kind of information is encapsulated by a certain class. In KeyValue terminology, these classes are called real keys and belong to namespace ::keyvalue::key.

The class key::Key is the base of all real keys. More precisely, real keys derive from key::Traits which, in turn, derives from key::Key.

Actually, key::Traits is a template class depending on a few parameters:

Key-value pairs are stored in DataSets. This class implements methods getValue() and find() to retrieve values assigned to keys. Both methods receive a real key and processes all the information about the expected value encapsulated by the key. For instance, suppose the variable today holds the current date and consider a key BirthDates which corresponds to a vector of increasing dates, supposedly, in the past or today.

An appropriate real key is then:

key::MonotoneBoundedVector<ptime, key::Increasing, key::Leq>
  births("BrithDates", today);

Therefore, if key BirthDates belongs to a DataSet data, the result of

data.getValue(births);

is a ::boost::shared_ptr<::std::vector<ptime>> such that the elements of the pointed vector are in increasing order and before (less than or equal to) today. If the input does not verify this constraint, then an exception is thrown to inform the user about the failure. Therefore, the caller does not need to check the constraints.

Since the type returned by getValue() depends on the real key it receives, this method is a template function. The same is true for find().

The difference between getValue() and find() concerns what happens when the key is not resolved. The former method throws an exception to indicate the failure whereas the latter returns a null pointer. In practice, getValue() is used for compulsory keys and find() for optional ones. A typical use of find() follows:

bool foo(false);
if (bool* ptr = data.find(key::Single<bool>("Foo")))
  foo = *ptr;

In the code above foo is false unless key Foo is found in data, in which case, foo gets the given value.

All builders and calculators derive from class Processor. This class declares two pure virtual methods: getName() and getResult(). The former method returns the name under which the processor is recognized by key Processor. The second gets the result of processing a DataSet.

Actually, builders and calculators are specializations of template classes Builder and Calculator, resp. They depend on a parameter type named Tag whose primary role is distinguishing different specializations.

Builder and Calculator specializations may implement different features which affects their declarations and implementations. Therefore, different specializations of Builders and Calculators might derive from different base classes and implement different methods. Rather than declaring the specializations from scratch, providing all their base classes and declaring all necessary methods, helper files keyvalue/mngt/DeclareBuilder.h and keyvalue/mngt/DeclareCalculator.h should be used. (See Builder and Calculator for details.)

Message is the abstract class that defines the interface for all types of messages sent to loggers. MessageImpl is a template class which implements Message's pure virtual methods. There are six different specializations of MessageImpl with corresponding typedefs:

They define operator &() to append formated data to themselves. A typical use follows:

Info info(1);  // Create a level-1 Info message.
size_t i;
::std::vector<double> x;
//...
info & "x[" & i & "] = " & x[i] & '\n';

Similarly, exception::Exception is an abstract class whose pure virtual methods are implemented by template class exception::ExceptionImpl. This template class has a member which is an instantiation of MessageImpl. The exact instantiation is provided as a template parameter of exception::ExceptionImpl. There are two specializations of exception::ExceptionImpl with corresponding typedefs:

RuntimeError indicates errors that can be detected only at runtime depending on user-provided data. LogicError indicates errors that should be detected at development time. In other terms, a LogicError means a bug and is thrown when a program invariant fails. It is mainly used indirectly through macro KV_ASSERT as in

KV_ASSERT(i < getSize(), "Out of bound!");

To keep compatibility with exception handlers catching standard exceptions, RuntimeError derives from ::std::runtime_error while LogicError derives from ::std::logic_error.

Method exception::ExceptionImpl::operator &() provides the same functionality of MessageImpl::operator &(). Example:

if (price <= 0.0)
  throw RuntimeError() & "Invalid price. Expecting a positive number. Got " &
    price;

Other more specific exception classes are implemented to indicate errors that need special treatment. They all derive from either RuntimeError or LogicError.

Some methods of class Bridge are implemented by KeyValue itself. However there are four public methods which are left to the bridge developer. (See example in bridge-example/bridge-example/Bridge.cpp):

Bridge();

The default constructor is declared by KeyValue (not by the compiler) and, therefore, it must be implemented. One can do whatever initialization it needs for the bridge and core libraries. For instance, Processors registration is suggested to be launched from here by calling function registerProcessors() as explained above.

const char*
getCoreLibraryName() const;

Returns the name of the core library. The result also names the function called in LibreOffice Calc or Excel spreadsheets and, for that reason, must be a single word (no white spaces). Otherwise front-ends might get in trouble.

const char*
getSimpleInfo() const;

Returns a simple description (one line long) of the core library. This message is used, for instance, by the Excel add-in manager.

const char*
getCompleteInfo() const;

Returns a more detailed description of the core library. This message is presented by loggers when they are initialized.

Builder and Calculator specializations (the two flavors of Processor) are implemented in similar ways. Firstly, let us see how to implement the latter and then cover the differences for the former.

// First #include the header file containing smart pointer information:
// (This is just an example. Each bridge library #includes its onw file.)
#include "bridge-example/PtrTraits.h"

// Now #include other required header files:
// ...
#include "keyvalue/mngt/Calculator.h"
// ...

namespace keyvalue {

#define TAG Foo

#define COMMAND // Must be defined if, and only if, the calculator is a Command

#include "keyvalue/mngt/DeclareCalculator.h"

const char*
Calculator<tag::Foo>::getName() const;

const char*
Calculator<tag::Foo>::getCommandName() const;

value::Value
Calculator<tag::Foo>::getValue(const DataSet& data) const;

} // namespace keyvalue

// First #include the header file containing smart pointer information:
// (This is just an example. Each bridge library #includes its onw file.)
#include "bridge-example/PtrTraits.h"

// Now #include other required header files:
// ...
#include "keyvalue/mngt/Builder.h"
// ...

namespace keyvalue {

#define TAG Logger

#define OBJECT_TYPE ::keyvalue::logger::Logger

#include "keyvalue/mngt/DeclareBuilder.h"

const char*
Builder<tag::Logger>::getName() const;

const char*
Builder<tag::Logger>::getCommandName() const;

value::PtrTraits<::keyvalue::logger::Logger>::Type_
Builder<tag::Logger>::getObject(const DataSet& data) const;

value::PtrTraits<::keyvalue::logger::Logger>::Type_
Builder<tag::Logger>::getObject(const double& data) const;

Key functionalities belong to namespace ::keyvalue::key and all keys should be in this namespace as well.

The basics for implementing keys were explained in Section 12.3. In particular, we have seen that all keys derive from template key::Traits. For instance,

namespace keyvalue {
namespace key {

class MyKey : public Traits<double, StdSingle, NoMap> {
  // ...
};

} // namespace key
} // namespace keyvalue

is the prototype for a key accepting a single double value which is not mapped. (Actually, the second and third parameters of key::Traits above match the default choices and then, could be omitted.)

The choices of key::Traits parameters impose some methods to be implemented by derived classes. Those methods are divided in two categories: mapping- and checking- methods.

OutputType
get(const string& name) const = 0;

This file provides the information on the smart pointer used by bridge and core libraries. The name of this file is flexible but, for sake of concreteness, in this manual it will be called PtrTraits.h.

PtrTraits.h must be included at the very beginning of all builder and calculator source files. (For instance, every processor implemented by the bridge-example library #includes the file bridge-example/PtrTraits.h.)

In the sequel, we shall see the four tasks that PtrTraits.h must acomplish. Section 14.1.5 presents examples of pointer traits files that come with KeyValue and can be used by bridge developers as samples for writing their onw.

namespace keyvalue {
namespace value {

template <typename ObjectType>
struct PtrTraits {
  typedef ::boost::shared_ptr<ObjectType> Type_;
};

} // namespace value
} // namespace keyvalue

namespace keyvalue {
namespace value {

template <>
struct PtrTraits<void> {
  typedef ::boost::shared_ptr< ::core::Polygon> Type_;
};

} // namespace value
} // namespace keyvalue

template <typename ObjectType>
value::PtrTraits<ObjectType>::Type_
dynamic_pointer_cast(const value::PtrTraits<void>::Type_& genericPointer);

template <typename ObjectType>
::boost::shared_ptr<ObjectType>
dynamic_pointer_cast(const ::boost::shared_ptr<::core::Polygon>& genericPointer);

#ifndef KEYVALUE_PTR_TRAITS_FILE
#define KEYVALUE_PTR_TRAITS_FILE __FILE__
#endif

The macro KEYVALUE_PTR_TRAITS_FILE must be set to the name of the pointer traits header file. Unfortunately, there are two places where this macro is set and both definitions must agree, otherwise weird errors can occur.

KeyValue has some expectations on the smart pointers received from builders. This section covers the conditions that smart pointer implentations must verify.

Most smart pointers are implemented as template classes parametrised on the type of object pointed to. Although this is not a KeyValue's requirement, to simplify the presentation, we will assume this is the case and we shall call this template MinimalPointer.

The interface that KeyValue requires MinimalPointer to implement is given by the skeleton below.

template <typename ObjectType>
class MinimalPointer {

public:

  MinimalPointer(const MinimalPtr& orig);

  ~MinimalPointer();

};

As we can see, MinimalPointer must have a public copy-constructor and a public destructor (virtual or not). We emphasize that this interface is the minimum requirement and smart pointers will certainly extend it. Having said that, we notice the omission of constructors (apart from the copy-constructor). Surely, there is no class without a constructor but, because KeyValue does not create these pointers (it only copies those created by the bridge), it does not require any other particular constructor to be implemented. Additionally, KeyValue does not dereference the pointer and, thus, does not require the implementation of operator ->(). The same holds for other methods usually implemented by smart pointers.

Recall that KeyValue receives pointers to different types of objects but, for uniform storage, casts them to a unique smart pointer type. For the sake of this presentation, we will assume that this smart pointer is a specialization of MinimalPointer when ObjectType is void. Its minimal interface is given below.

template <>
class MinimalPointer<void> {

public:

  MinimalPointer();

  ~MinimalPointer();

  MinimalPointer&
  operator =(const MinimalPointer& orig);

  bool
  operator ==(const MinimalPointer& rhs) const;

  template <typename ObjectType>
  MinimalPointer(const MinimalPtr<ObjectType>& orig);

};

The following public methods must be implemented: the default constructor, the destructor (virtual or not), the assignment operator, a comparison operator and a template constructor taking a generic smart pointer as argument. The first three methods do not need any further comments. Now, we shall consider how KeyValue uses the last two.

Usually smart pointers implement operator bool() returning false, if the pointer is NULL, or true, if it is not. KeyValue does not require that. Instead, it considers a pointer to be NULL if, and only if, the result of comparing the pointer (through operator ==()) with a default-constructed one gives true.

When KeyValue needs converting a MinimalPointer<ObjectType> into a MinimalPointer<void> it uses the template constructor above which can be explicit or not.

A help file in compressed HTML format can be associated to the Excel add-in. This file must be named manual.chm and be located in the directory containing the add-in. Then under the Excel function wizard for KEYVALUE function (or whatever is the name provided by the bridge), one can click on "Help on this function" to open manual.chm.

The file will be open by the program associated with extendion .chm at the position mapped to ID number 1000. For instance, the .chm of this user manual is called manual.chm and maps the ID number 1000 to Section 5.

Instructions on how to create .chm files and how to map ID numbers to anchors is outside of the scope of this document.

As explained in Section 7.1.1, Excel add-in provides special support for commands. Under the add-in menu on the menu bar, it presents a menu named after the core library from which one can call any command. The result of the command can be seen on the global logger.