Frequently Asked Questions

CLARIN does not create linguistic resources; its purpose is to offer rapid access to the existing resources and to facilitate their reuse in new contexts. When resources and tools are produced for individual usage interoperability and therefore the need to adhere to standards or best practices is of little relevance. The problem of interoperability only emerges when linguists are ready to offer their resources and tools to other researchers. One of the requirements of interoperability is to connect different resources to the same tool. This can be made using standards, but this would imply having all the resources standardized (this is an ideal situation, but cannot always be achieved in reality). When needed, a standard can also play the role of a pivot format (resources are converted to the standard before they are used).

An open list follows:

• character encoding: ISO -10646 UNICODE, UTF-8
• country codes: ISO 3166
• language codes: ISO 639-1/2/3
• codes for the representation of names of scripts: ISO 15924
• text format: XML
• text format: CSV (comma separated with "-quotes, with a header line and preferrably a line of ISOcat URIs for each column)
• feature structure representation: ISO 24610-1:2006
• representation of primary sources: TEI (Text Encoding Initiative)
• knowledge engeneering: RDF, RDF-S, SKOS, OWL
• audio/speech: PCM (Pulse Code Modulation) for digitizing sound waves, the Alphabet of the International Phonetic Association for phonetic transcriptions;
• video/multimodality: MJPEG2000 lossles as backend format, MPEG2 or H.264 for handling and processing
• data categories: ISO DCR and ISOcat
• annotation of temporal entities: TimeML (part of TC 37/SC 4)
• morpho-syntactic annotation: MAF (Morpho-syntactic Annotation Framework), ISO/DIS 24611
• syntactic annotation: SynAF (Syntactic Annotation Framework), ISO/CD 24615
• lexical annotation: LMF (Lexical Markup Framework), ISO 24613:2008
• linguistic annotation: LAF (Linguistic Annotation Framework), ISO/DIS 24612

CLARIN actively tracks a number of ongoing standardisation activities at two major levels: linguistic structures/formats and linguistic encoding. CLARIN as an infrastructure project has the duty to evaluate, test and comment these proposals in close relation with the relevant standardisation bodies. When necessary, CLARIN may take the lead in initiating new standardisation activities when a clear gap in coverage is identified. For more information see the CLARIN Standardization Action Plan.

No linguist should be required to read long documents about standards; it is primarily the task of the tool, service and converter developers to provide frameworks that help the researcher and that hide complex formalisms as much as possible.

Very good question. A researcher will always face this issue. The research moves a field on, and in no-man's-land there are no standards (yet). The standards remain behind the research. Industry stays always on ferm ground, therefore on well establishd conventions. Although it appears that reseach has no means to make use of standards, it should base itself on well-established foundations, which should be expressed in standardized form whenever possible. Only for the head of the arrow, the really fresh things, just invented, the researcher should look for his own ad-hoc conventions. Applied to the linguistic data, this means that in an annotated corpus, for example, one will find a mixture of standard and invented markings. CLARIN can be used for that part of processing that involves using existing tools and resources, that have been converted to a standard format.

• Download Arbil (2.3 or higher) and start it
• Go to Options > Templates & Profiles
• Select in "Clarin Profiles" which profile(s) you want to use as the basis for a CMDI file anc click on Close
• Right-click on Local corpus, choose Add and select the relevant profile (the CMDI profiles are marked with a icon)

Some profiles (obvious tests and the ones not intended for manual metadata creation) have been excluded from the default profile list in arbil (testing). You can see them disabling the "only load profiles selected for manual editing" in the Available Templates & Profiles dialog.

By default, if you create a new profile in the component registry, it will show up in Arbil.

as of Arbil 2.3  there is an item 'Insert Manual Resource Location' in the context menu of CMDI instances in the tree.

• they have the grey club icon
• the text is shown in grey
• after the node a number indicates how many times the component occurs

<ISO639>
   <iso-639-3-code>cat</iso-639-3-code>
</ISO639>
<ISO639>
   <iso-639-3-code>spa</iso-639-3-code>
</ISO639>

Centres will form the backbone of the persistent and stable CLARIN service infrastructure. For further information we refer to the Requirements Specification Document and to the Short Guides.

Researchers will only use certain resources and certain tools/services offered via the web when they are sure that they can access them also during a longer period. Currently, researchers mostly download resources first to their computer to create accessibility, but in the cyberinfrastructure scenario with the many big resources and collections this way is not suitable anymore. So availability and accessibility has to be guaranteed by institutions with a clear service oriented attitude that do pose as little restrictions as possible on the usage by the researchers. Only this new type of centres can give such guarantees.

Can everyone act as a CLARIN centre in the emerging network? No - since centres need to fulfill a number of criteria which have been mentioned in the documents. In particular these centres need to make a commitment statement that they will give their services for a defined period of time at a certain service level which is dependent on the type of service. Setting up centres that adhere to these requirements will cost some money, therefore it is obvious that centres need to have clear funding basis.

The answer  on this question is dependent on the country although some services are given at a European level. For detailed questions about the centres, see www.clarin.eu/centres

Persistent identifiers are increasingly often seen as core component for all the many references we are creating at various levels - this can range from references between metadata descriptions and their resources up to references between semantic assertions made by using the RDF (Resource Description Framework). For more information please read the requirements specification document or the short guide.

In the emerging cyberinfrastructure we are creating more and more references between resources, resource fragments and services. The creation of these references is very costly and often is essential for the interpretation of a resource. Therefore we need proper mechanisms to ensure that these references survive despite all the changes that happen in repositories for example. It is known that URLs are not appropriate - they are not persistent even when we believe that they are proper URIs. Therefore special PIDs come into place which identify an object and which are maintained by reliable institutions.

Handling PIDs is very simple. First you need to register a PID for a resource or service. You can do this very simply by providing the required information to the PID service site, in particular the path to access the resource such as a URL and you will receive back a PID which you can enter into the metadata description for example, so that everyone can use it for referencing. When a user finds such a PID in a resource, he/she can click on this reference and the service will resolve the PID and give access to (one of the copies of) the resource. Normally as user you don't see the intermediate transactions.

If the PIDs cannot be resolved at a certain moment one simply cannot access a resource. Think of a situation where hundreds of users are waiting on a resolution of a PID and nothing happens - a nightmare for any cyberinfrastructure scenario! Since this would not be acceptable, we need to make sure that the PID service is based (a) on a very robust and reliable software offering sufficient functionality, (b) on a proper service based on redundant centres with a high availability and persistency guarantee.

CLARIN has an arrangement with the EPIC consortium that CLARIN members will be able to register PIDs and of course resolve them. This consortium groups a number of reliable European service providers that want to participate in providing a redundant service for the research world, i.e. we are speaking about millions of PIDs and a service at very low costs. The service is based on the Handle System which according to our investigations is the only robust system meeting all requirements. No one is obliged to register Handles, but of course CLARIN centres will need to demonstrate that their PIDs can be resolved in a robust manner and offer the required functionality.

PIDs are as said unique and persistent identifiers of objects that are made available by proper repositories. For many resources there are additional characteristics such as multiple copies for preservation reasons, a string (such as MD5) that can be used to check authenticity, simple metadata for citation purposes, a reference to the access permission record etc. A proper PID system should offer such information immediately when resolving a PID. PURLs can't offer functionality, for URNs we do not know about well-proven and robust resolver, although the big libraries agreed on using URNs for their publications. DOIs are also based on the proven Handle System and it is certainly a proper service which is used in particular by the big publishing companies. However, DOI also comes with a business model that will not be acceptable for may research organizations.

• a prefix (e.g. 1839)
• a suffix (e.g. 00-0000-0000-0009-3C7E-F)

hdl:1839/00-0000-0000-0009-3C7E-F

To resolve such a handle (=make it a clickable link that redirects to the resource itself) use the following formula:

http://hdl.handle.net/prefix/suffix

e.g.: http://hdl.handle.net/1839/00-0000-0000-0009-3C7E-F

 

(Answer taken from the ISO citer draft, p. 11)

This International Standard supports different levels of granularity. The following recommendations are designed to encourage efficiency and promote interoperability with other naming schemes:

1) If there is an existing identifier scheme for a type of resources, for instance, ISBN, this level of granularity should be retained, which is to say that no new PIDs should be issued without very good reasons, such as for chapters. Chapters would preferably be addressed using part identifiers in conjunction with the PID of the book.

2) If the resource is associated with the complete content of a digital file, an individual PID should probably be assigned for this resource.

3) If the resource is autonomous and exists outside a larger context, an individual PID should probably be assigned for this resource.

4) If a resource should be citable apart from any containing resource, an individual PID should probably be assigned for this resource.

These recommendations are, however, subject to the needs of resource creators with respect to the level of granularity they deem suitable to the specific resource environment.

The rewriting behaviour of part identifiers can be configured per handle prefix (actually it can also be done per individual handle but this is not supported for EPIC at this point). For EPIC (version 1, so with prefix 11858) the choice was made to rewrite [suffix] to ?[suffix]

 So suppose that 11858/1234 resolves to http://clarin.eu then

 11858/1234@test=a will be resolved to http://clarin.eu?test=a

Please note that when you offer PIDs with part identifiers that you are responsible of maintaining the part identification fragment as well. Remember that users will use it to link to your resources and that the resulting end point should always be available.

Generally - a word, but also an abbreviation. In some aglutinative languages it could be just a morphem, part of a compound.
Used XML marking:

<tok tid="...">word</tok> or <w wid="...">word</w>.

Part of speech (POS) seem to occur in every natural language. The usual categories are: noun, verb, article, adjective, pronoun, preposition, adverb, conjunction, etc. Sometimes, by POS morphological and syntactic classes are also meant.

The canonical form of a word. It represent all the various forms of a morphological paradigm. It is marked on tokens with a specialized attribute, for instance: 

<tok id=“w10” pos=“det” lemma=“the”>the</tok>

<tok id=“w11” pos=“n” lemma=“wind”>winds</tok>

<tok id=“w12” pos=“prep” lemma=“of”>of</tok>

<tok id=“w13” pos=“n” lemma=“change”>change</tok>

A group of words acting as a unit surrounding at least one noun. One noun of the group acts as representative - the head. It gives the morpho-syntactic properties of the group. 
Common XML notation: 

<np np_id="np1" head_id="t3"><tok tid="t1">the</tok><tok tid="t2">black</tok><tok tid="t3">cat</tok></np>.

A complex tag delimiting sentence boundaries.
For instance: 

<seg sid="...">This is a sentence.</seg>.

Syntactic descriptions constitute a huge chapter of computational linguistics. A very general classification, however, sees a syntactic description notating either constituents or functional dependencies. In an FDG type of notation, for instance, one possibility is to mark on each word its parent and the type of the syntactic relation to its parent: 

<tok id=“w10” pos=“det” lem=“the” link=“w11” linktype=“det">the</tok>

<tok id=“w11” pos=“n” lem=“wind” link=“…” linktype=“…">winds</tok>

<tok id=“w12” pos=“prep” lem=“of” link=“w11” linktype="mod">of</tok>

<tok id=“w13” pos=“n” lem=“change” link=“w12” linktype=“pcomp">change</tok>

As words are often polisemous, only the contexts can define their senses. Accordingly, notations used to disambiguate word senses (WS) are indications of senses in context, as given by specialized repositories (for instance WordNet).
Common XML notation of WS are realized by an attribute (for instance, 'wsd' or 'sense') complementing a token marking: 

<tok tid="..." pos="v" lemma="run" wsd="s1">running</tok> or
<tok tid="..." pos="v" lemma="run" sense="s1">running</tok>.

We say that an anaphor and an antecedent are coreferential is both are text spans (usually NPs) refering the same discourse entity.
Bellow is an example of notation, as an attribute (coref) belonging to a 'np' element:
 

<np np_id="np1"><tok tid=“t1”>John</tok></np>
<tok tid=“t2”>hit</tok><np np_id="np2">
<tok tid=“t3” coref=“np1”>himself</tok></np>
 

The process of segmenting word tokens in texts. In all modern languages that use Latin, Cyrillic or Greek writting systems word tokens are recognized by the delimiting blank or punctuation. Numbers, alphanumerics and special format expressions (dates, measures, abbreviations) are also recognized as tokens, traditionally by using regular expressions. Tokenization in non-segmented languages, such as many Oriental languages, require more soffisticated algorithms (lexical look-up of longest matching sequences, hidden Markov models, n-gram methods and other statistical techniques).