Importing Data from different sources

There are many services in the www that were build around an import and export function so that the communication between different parties becomes easier. To give an example, since some years now there is a growing market of “e-metering” so to say, services that measure the energy consumption of a single unit/building, provide historical overviews of consumptions and help with the billing of customers. In this field there are many competitors with different architectures, devices and formats for representing their measurements/bills. To control this market segment a web service would need to be able to read all the different formats of the various suppliers and in turn present stored data in a format the customers can understand as well as an exchange format between different providers. How could the methods and features of an SOA be used best in such a scenario, or is there an architecture that is better suited?

Taggings:

Submitted by Michael Havlicek on Tue, 12/13/2011 - 06:29
1 answer

Importing Data from different sources - SOA approach description

To make full use of the basic principles of the service oriented architecture it is useful to define a public/generally accepted model of the data object to be stored. This model has to encompass all (or at least most) of the possible data such an object could have and is later handled in the lowest level of the architecture, the data layer. In this layer the data objects can be handled by a database, so let’s assume we use JAVA as the basic language, the data objects could then be stored in an HSQLDB, accessible by a basic CRUD-interface.

Above, in the service layer a general import/export function must be defined, and while this general function should not be bound to one format it will have to work through the data object already described. This is, most often than not, the hardest part as these functions must cover and handle model specific data in an open way as to not limit the presentation of the data in the next layer and to still allow the CRUD to handle the data object without loss of information.

It was already mentioned that the next layer will be the presentation layer. This is the first layer where actual information on the various formats is used. In object oriented programming languages it is quite easy to realize this (abstract classes in Java for example), once the underlying layers are working properly. Here is also the only location in the program where there is a true difference in the representation of the data. While it was quite abstract and handled only to store and retrieve in the layers below, here it can be manipulated to fit certain formats or vies-versa so that certain formats fit the underlying data object.

By this approach it becomes possible to expand (and reduce) the formats a service is able to manage without changing the underlying service. This has a positive effect on the work time as well as consistency throughout such projects.

Submitted by Michael Havlicek on Tue, 12/13/2011 - 07:07