You are here
Frequently Asked Questions - Metadata in CLARIN: harvesting and VLO
If you have many metadata records or records that frequently change:
- Use OAI-PMH
- Provide them preferrably as CMDI (click here for details about how to serve CMDI over OAI-PMH)
- If that is not possible provide them as OLAC
- Send a mail to email@example.com to notify us about your OAI-PMH access point
- Depending on the situation we will add it to the centre registry (in case of a registered CLARIN centre) or add it manually to our harvester
If you have only a few static records and setting up an OAI-PMH access point is not feasible:
- Submit your records to the Language Resource Inventory (for data, corpora, lexica, web services, software, ...)
- All the records in the LRT inventory will be automatically converted into CMDI records. Note that this process can take a while.
Data provided over OAI-PMH or via the LRT inventory will be made available and searchable via the Virtual Language Observatory.
No, we are not. It is a term used for gathering metadata descriptions from several locations and storing it in a central database. You can find the results of such a harvesting process at https://vlo.clarin.eu
More information about harvesting metadata can be found at http://en.wikipedia.org/wiki/Open_Archives_Initiative_Protocol_for_Metad...
There are several packages available to setup an OAI provider, some popular examples:
file-based, Tomcat web application written in Java: jOAI (some centres have good experiences with this one)
database-based, written in PHP: oai-pmh-2
- library to connect to a Java application: OAIProvider
More tools to setup an OAI data provider can be found at the OAI webpage.
The metadata harvester runs with different configurations at different starting times:
- Monday and Thursday 20:00 CET/CEST: harvesting of CLARIN providers
- Friday 20:00 CET/CEST: harvesting of non-CLARIN providers
The VLO importer for the production VLO runs after every completed harvest (importing all current metadata, both CLARIN and non-CLARIN).
Combined, harvester/import runs typically take about 12-20 hours to complete, depending primarily on the response rate of the metadata providers.
Because of this schedule, it can take a couple of days before provided metadata becomes available in the VLO. If it takes longer than that, please send a message to firstname.lastname@example.org.
Be aware that the harvester and VLO schedule may be subject to change in the future!
If some or most of your records have been harvested and imported in the VLO fine, but some records seem to have been omitted, they were probably explicitly skipped by the VLO importer process. Records with no (valid) resource proxies are omitted from the import process. Make sure to always include at least one resource proxy, for example a landing page or link to a resource (see "How does the Resources section work?"). The VLO import process also skips records that are too large. The limit currently lies at 50 megabytes. If you have larger metadata records that you think should be included, please contact us at email@example.com.
Note that metadata files that are not in the CMDI 1.2 version get converted before import. The limit applies to the converted file, so certain records may get skipped even if the original file is within the size limit. Typically a converted record should not be more than twice the size of the original file, depending on the formatting. Typically the file size difference is 20% or less.
If files seem to be omitted from hierarchies, but do appear in the VLO in isolation, the most likely reason is that the Resource Proxy reference value (a URL or PID) used in the parent record does not match the self link value (in the MdSelfLink header item) in the referenced record. The VLO will only consider a record to be a parent of another record if it uses the exact self link of the latter to link to it.