Datamazed – analysing library data flows, manipulations and redundancies

ELAG 2015
By Lukas Koster, Library Systems Coordinator at the Library of the University of Amsterdam

View slides of presentation

It’s more about transformations than manipulations. Tried to build a dataflow repository for efficiencies and blueprint for improvement.Initial problem is that system environment is complex. Lots of things happening to maintain this environment. Data is all over the place. Labyrinths are easy, but a maze is much more complex and that’s what our systems look like. Worth spending time to develop new environments because currently it is all very fragmented. The data is hostage and we need to free it.

Goal of the project: describe the nature and content of all internal and external datastores and workflows between internal and external systems in terms of object types and data formats, thereby identifying overlap, redundancy and bottleneck that stand in the way of efficient data and service management.

Methodology used is enterprise architecture. Distinguish between business (what), enterprise (how) and technology. Looked at other similar experiences and knew of BIBNET Flemish Public Library Network and their Architecture Study, focusing on the big picture rather than dataflows.

DFD = dataflow diagramming is a fairly easy model. Also used tools such as data modelling, visualisation etc.  Chose Business System Modelling, a relatively open tool with a number of export/import and a lot of documentation and reports. 

Dataflow repository describes all elements, including the systems they use etc. Their Visual Paradigm Project Model is subdivided into meaningful folders that can also be used to generate reports. They also have made a data dictionary for all object types, data elements and so on.

Business layer top level
Business layer level 2
Business layer level 3: data management
Application layer: data exchange

Dataflows can be defined by type (they had 5). In all data flows there’s an element of selection on what you do and with what. It has to be documented to help for decisions and so that you know what to expect and what happens (especially if you’re going to change systems.) Same for transformations – has to be transparent. 

Data redundancy is also an important issue and can be caused for various reasons. The unique solution: linked data!

Mostly benefits of all of this is not only having a good overview of available data, dataflow dependencies and efficiencies but also experimenting with linked data. It may be the beginning of something else, such as data consolidation exchange. Descriptions of how things are more automated should also be recorded.

MIF and Europeana Inside

ELAG 2014

Metadata Interoperability Framework (MIF)
Naeem Muhammad, Software Architect at LIBIS KULeuven and Sam Alloing, Business Consultant at LIBIS KULeuven, Belgium

Made for Europeana inside. This is a technical project with different partners, content providers and software providers. It's to create a better integration of the different content in Europeana. So the goal is to create a component that developers of the different systems can directly add into their content management system and it will talk to Europeana. End of this project planned in Sept. 2014.

The content is enriched by Europeana and the content providers can get it back. This is still in discussion and development. The enriched metadata is not always correct so there are still issues to resolve.

ECK=Europeana Connection Kit. Technical providers use this to transform and push data to Europeana. The ECK local is the part to be integrated in the local system itself. Core ECK services include:

• Metadata definintion
• Set Mangaer
• Statistics
• PID generation
• Preview service
• Validation of metadata
• Data push (Sword) / data pull (OAI-PMH) because some content providers would rather push the data to Europeana rather than them taking it but Europeana is afraid of compabtibility issues so the data pull is still the one in use
• Mapping and Transformation

LibisCoDe (L content delivery to Europeana) is the tool used (?) for content providers to put their data in. It is then sent to Europeana. It will be developed in a way that users can decide what data they want to take back from Europeana.

Mapping and Transformation supports MARC to EDM and LIDO to EDM because that's what used in Europeana. LIDO =XML fomrat used by museums, EDM=RDF format from Europeana. So the input has to be MARC XML or LIDO. The output is only EDM at the moment. There are core classes and additional classes. EDM uses Dublin Core. For MARC, it works like this:
[command],[marc tag + subfield],[edm field], e.g. COPY, marc506a,dc:rights
doesn't use indicators at this moment, it could change.

Commands are: COPY, APPEND, SPLIT, COMBINE (multiple source fields can be combined in one target field), LIMIT (to limit the number of characters in a field), PUT, REPLACE, CONDITION (combine different actions and use a conditional flow; can be used with IF.

The plan is that EDM has to be as easy for users as possible, even though some understanding will help. The important is to know the EDM field, not the format.

It's a wservice, so no user interface. Meant to be integrated in CMS or use a REST client. Parameters: records (can be a zip file, XML), mappingRulesFile, sourceFormat (LIDO or MARC), targetFormat (EDM)

Future: add input formats, such as csv, filemaker xml, some custom xml etc. Update/add output formats (add EDM contextual classes, other formats...), extend/add update actions, add queuing (near future), add mapping interface (or integrate with  MINT, another Europeana project for mapping)