|
|
DataPile structure
Our first approach used a verticalized database structure called DataPile, which had been designed to integrate heterogenous metadata into common database table. The first picture illustrates the database structure of our first middleware version. Area marked by red dashed curve contains the main DataPile tables:
| MW_RESOURCE
which contains list of all origin_id identifiers of resources as they appear in remote database together with related resource_id identifier which represents local identifiers inside the middleware's cache.
|
MW_ATTRIBUTE,
which contains list of all attributes (types of attributes) such as definition of title, definition of om_url etc.
MW_DATA
is the main DataPile table which contains values of attributes defined in MW_ATTRIBUTE related to resources stored in MW_RESOURCE.
The main drawbacks of this approach emerged soon during the development of the project and has also been described in the paper [url:DataPile].
The DataPile structure is too sensitive to updates. If the application used mainly SELECT sql commands, the indexes could vastly speed up the process, but each UPDATE command would degraded the performance by forcing reindexing of the verticalized table.
At the beginning of the project we decided to use DataPile structure because we thought we could transfer all data into middleware's cache. This idea has been rejected by OASIS consortium due to political reasons. Our partners did not want to store their valuable data into an intermediate repository. However, they agreed on cache-per-request nature of Middleware's cache. This new approach made the cache too dynamic for the DataPile structure and we had to use another technology to deal with frequent updates.
UNIX utilities
The new approach uses standard UNIX utilities such as cp, flock, grep, sort, sed, cut, tac, head, tail, nl, join to implement the cache-per-request functionality. Before we started with the implementation we had performed several benchmarks. As the result of the tests we discovered, that UNIX utilities are 20 times faster in certain circumstances than MySQL, especially in updates.
On the other hand, relational databases provide an easy way of indexing comparing to UNIX utilities. We solved this problem by designing appropriate directory structure which could substitute the indexes. Of course, this approach is much less flexible but we decided to sacrifice flexibility at the expense of efficiency.
|
|
→ Back to OASIS Database Adapter
→ Back to OASIS Middleware
→ Back to OASIS Platform
|
