Atramhasis

Introduction

Atramhasis is an online SKOS editor. It allows a user to create and edit an online thesaurus or vocabulary adhering to the SKOS specification through a simple web interface. This allows any user with access to a web browser to consult the thesauri and if so wanted, to edit them.

Atramhasis is also intended to be one of the focal points in a Service Oriented Architecture. It exposes as much of its functionalities as possible through REST services. Both reading from and writing to concept schemes is possible with Atramhasis.

Atramhasis tries to stick as closely as possible to the SKOS specification. Where this was not possible, we tried to follow other standards such as SKOS-THES.

Atramhasis is being developed by the Flanders Heritage Agency, an agency of the Flemish Government that deals with Archaeology, Monuments and Landscapes. As such, we mainly intend to use it with vocabularies and thesauri that are related to cultural heritage. We generally construct our own thesauri, specific to our own applications, but while always keeping an eye on other thesauri in the larger field of cultural heritage such as the Art and Architecture Thesaurus (AAT).

If you have questions about the project, want to help out, want to report a bug or just want to have a conversation with us, please get in touch. For general conversations, you can use our Google Group. If you have encountered a bug in Atramhasis or it’s documentation, or if you want to ask us to consider implementing a feature, feel free to use our issue tracker.

Features

Public HTML interface

When visiting the homepage of an Atramhasis instance, users are presented with a few different options. They can search for a certain label within a certain conceptscheme. Links are presented to all the conceptschemes present in the instance. For a list of (configurable) conceptschemes the most visited concepts are displayed for easy and quick access. Notice that by default Atramhasis comes with an English, French and Dutch public interface. Other languages can easily be added.

A concept detail page details one concept or collection. It lists the concept’s labels, notes and sources used in creating or researching the concept. Every concept has an id (an identifier within a conceptscheme) and a URI that can be custom generated. For interoperability with other applications, every detail has both RDF/XML and N3/Turtle downloads available. As Atramhasis tries to take the user’s preferred language settings into account, it will try to provide the title of the page in the user’s preferred language, whilst also listing the labels separately.

Scrolling further down on a detail page, we come to the relations between this concept or collection and other concepts or collections. All common SKOS relations (broader, narrower, related, member, …) are accounted for. Links to other conceptschemes (suh as the AAT) are supported with SKOS matches.

While every detail page presents the immediate relations for a certain concept or collection, there’s also a tree view available that presents all broader/narrower relations for all concepts and collections in one go. This can be reached on the conceptscheme page or from every detail page. As can be seen here, an Atramhasis instance can easily be reskinned for a certain organisation. The Flanders Heritage Thesaurus is an Atramhasis implementation with custom styling and authentication.

SKOS editor

The Atramhasis admin interface allows editors to find a certain concept or collection and edit or delete it. While it’s also possiblel to edit conceptscheme attributes this way, they always need some code configuration and thus need to be created by a system administrator.

While you can define concepts from scratch, it’s also possible to import a concept from another thesaurus, such as the AAT. It will import labels and notes from the original concept and add an automatic SKOS match back to the original. Bear in mind that this actually creates a new concept with a new URI, while referencing the original. If you want to import an entire thesaurus, you need to go beyond the User Interface.

Editing a concept or collection is done using one or more tabs, such as the notes tab to edit scopenotes, sourcenotes, … Adding HTML markup is supported.

The relations tab allows editing the relations of concepts or collections with other concepts or collectons. Dropdown lists are present to facilitate editing.

The matches tab allows an editorto match a local concept to a concept in a remote conceptscheme.

Using the skosprovider that powers a remote conceptscheme, matching concepts can be searched for and added to the local concept.

Once a concept has been matched with a concept from an external provider it’s possible to merge the two concepts. This is similar to importing a concept, but works for concepts that already exist in your local thesaurus. Merging with copy the labels and notes from the external concept. If you can link you concept with an external concept that has labels for the concept in different languages, this is a quick way to extend the number of languages supported by your local concept. Before saving the results of the merge, you are free to review the results of the merge and accept or reject certain labels and notes.

LDF server

With a little bit of effort, Atramhasis can setup an LDF server for you, allowing you to quickly serve RDF tripples.

Demo

Running a demo site

Atramhasis comes with a demo site include. This allows you to quickly evaluate and inspect the software. To get started, just download Atramhasis from pypi and install it. We recommend doing this in a virtualenvironment.

$ mkvirtualenv atramhasis_demo
$ pip install -U atramhasis

Once Atramhasis is installed, you can call upon a pyramid scaffold to generate the demo site.

$ pcreate -s atramhasis_demo atramhasis_demo
$ cd atramhasis_demo

This creates a local demo package you can run with just a few more commands:

# setup
$ pip install -r requirements-dev.txt
$ python setup.py develop
# download and install client side libraries
$ cd atramhasis_demo/static
$ npm install
$ cd ../..
# create or upgrade database
$ alembic upgrade head
# intialize sample data
$ initialize_atramhasis_db development.ini
# generate full RDF dumps (not necessary for basic functionality)
$ dump_rdf development.ini
# start server
$ pserve development.ini

The Atramhasis demo instance is now running on your localhost at port 6543. To reach it, open your browser and surf to the address http://localhost:6543.

You will be greeted by the Atramhasis front page. From this page you can start searching and browsing the thesauri. You can also start editing the thesauri by surfing to http://localhost:6543/admin. The demo instance does not requires you to login to access the admin module. If you want to run Atramhasis in a production environment, you can easily write your own security module. This enables you to use the security mechanisms (eg. LDAP, Active Directory, a custom users database, …) that your organisation requires. Please consult the documentation on Security customisation for further information on this topic.

Running a demo site with Docker

There is a Docker image available that allows you to quickly get a demo instance up and running. The Docker image contains the demo application and the LDF server.

After installing Docker for your operating system, you can simply pull the image and run a container. Once you’ve executed the following commands, you should be able to visit the demo application in your browser on http://localhost:6543. A LDF-server is also included in the demo, which is accessible on http://localhost:3000.

$ sudo docker pull atramhasis/demo
$ sudo docker run -p 6543:6543 -p 3000:3000 atramhasis/demo

Alternatively, you can run a specific version of Atramhasis (starting from atramhasis 0.6.4):

$ sudo docker pull atramhasis/demo:0.6.4
$ sudo docker run -p 6543:6543 -p 3000:3000 atramhasis/demo:0.6.4

While this is a fast and easy way to get a first impression of Atramhasis, please be aware that any edits you make when running the image, will be discarded when you stop the Docker container. If you want to test the application over a longer period of time, this is probably not what you’re looking for. If you need persistence, but still want to use Docker, you can customise the Dockerfile to suit your needs.

Running a demo site on Heroku

This section will tell you how to deploy an Atramhasis demo (or your own implementation) in the cloud. We’ll use Heroku, since this provider allows for a free Python instance (dyno) with a limited Postgresql database.

Create an account on Heroku and make sure you have Heroku Toolbelt installed. Prepare your local Heroku setup

Note

More information on running Python apps on Heroku can be found on the Heroku dev center.

Atramhasis scaffold

Create an Atramhasis scaffold (if you want to deploy an existing scaffold, skip this step)

$ mkvirtualenv atramhasis_heroku
$ pip install -U atramhasis
$ pcreate -s atramhasis_demo atramhasis_heroku
$ cd atramhasis_heroku

Git repository

Make sure your atramhasis_heroku folder is a git repository.

$ git init
$ git add .
$ git commit -m "initial commit"

requirements.txt

Update the requirements.txt file, make sure it contains a reference to atramhasis and to waitress.

Note

waitress has to be in the requirements.txt file for our Heroku deployment, requirements-dev.txt will be ignored.

Procfile

Generate Procfile with the following command.

$ echo "web: ./run" > Procfile

run file

Create run with the following content.

#!/bin/bash
set -e
python setup.py develop
python runapp.py

Note

Make sure to chmod +x run before continuing. The develop step is necessary because the current package must be installed before Paste can load it from the INI file.

runapp.py

Create a runapp.py file.

import os

from paste.deploy import loadapp
from waitress import serve

if __name__ == "__main__":
    port = int(os.environ.get("PORT", 5000))
    app = loadapp('config:production.ini', relative_to='.')

    serve(app, host='0.0.0.0', port=port)

Note

After creating the necessary files, commit them in your local git repository

Initialize the Heroku stack

$ heroku create

Deploy to Heroku

To deploy a new version, push it to Heroku.

$ git push heroku master

Postgresql

Attach an Heroku Postgres add-on to your application

$ heroku addons:add heroku-postgresql:hobby-dev

It can take a couple of minutes before your db is ready. You can wait for it to be ready using this command.

$ heroku pg:wait

When ready, check the connection url and copy paste it into your production.ini file

$ heroku config | grep HEROKU_POSTGRESQL

Also change the alembic.ini file to check your production.ini file instead of development.ini

ini_location = %(here)s/production.ini

Make sure to commit everything and push it to Heroku

$ git commit -a
$ git push heroku master

Note

More info on provisioning a database

Preparing the app

Open a remote console on your app

$ heroku run bash

This will start a console inside your remote Python virtualenv, so you can use all your libraries.

Run the commands to prepare your application

$ python setup.py develop
$ alembic upgrade head
$ initialize_atramhasis_db production.ini
$ dump_rdf production.ini

Note

Close the remote console!

Run the app

Run your app by starting one worker

$ heroku scale web=1

Check to see if your app is running.

$ heroku ps

Take a look at the logs to debug any errors if necessary.

$ heroku logs -t

Your app should now be available on the application url.

Development

Technology

Atramhasis is a python webapplication that is being developed within the pyramid framework. Other major technologies used are sqlalchemy as the ORM and Jinja2 as the templating framework.

Client side the main technologies being used are Zurb Foundation and Dojo toolkit.

While Atramhasis is an editor for creating and editing SKOS vocabularies, it uses other libraries that are more geared towards using a vocabulary in an application.

• skosprovider: This library defines a VocabularyProvider. This is an abstraction of usefull functionalities an application integrating SKOS needs. Different libraries can implement this interface for different datasources. This allows decoupling the interface from the concrete implementation. Out of the box this comes with a simple DictionaryProvider that serves a vocabulary based on a simple python dict as datasource.
• skosprovider_sqlalchemy: An implementation of the VocabularyProvider interface with a SQLAlchemy backend. This allows using a RDBMS for reading, but also writing, SKOS concepts.
• skosprovider_rdf: An implemenation of the VocabularyProvider interface with an RDF backend. Atramhasis uses this for exporting ConceptSchemes to RDF. It can also be used to get an existing SKOS vocabulary defined in RDF into Atramhasis.
• pyramid_skosprovider: A library that integrates pyramid and skosprovider. This libraries creates a skosprovider.registry.Registry and makes it accessible through the pyramid.request.Request. Is also exposes a set of readonly REST services on the registered providers.
• skosprovider_getty: An implemenation of the VocabularyProvider against the Linked Open Data vocabularies published by the Getty Research Institute at http://vocab.getty.edu such as the Art and Architecture Thesaurus (AAT) and the Thesaurus of Geographic Names (TGN).
• skosprovider_heritagedata: An implementation of the VocabularyProvider against the vocabularies published by EH, RCAHMS and RCAHMW at heritagedata.org.

Atramhasis can easily be set up with a Linked Data Fragments (LDF) server. This server allows posing simple triple pattern queries of your dataset. Combined with a Linked Data Fragments client, similar functionalities to a traditional SPARQL endpoint can be achieved. Atramhasis facilitates the setup of a Linked Data Fragments server by generating a suitable config file for the Javascript server. This server can use different backends. Out of the box, Atramhasis generates Turtle files that can be used by the LDF server. It’s also possible to configre Atramhasis with a link to the rdf2hdt command (requires a separate installation). In this case, everytime the conceptschemes are dumped to RDF, the dump files are also written in HDT format.

General installation

We recommend installing Atramhasis in a virtual environment.

$ mkvirtualenv atramhasis_dev

To install a fully working development environment a pip requirements-dev.txt file is provided. By passing this file to pip install -r all requirements for Atramhasis and development of the software (Sphinx, py.test, tox) will be installed.

The following step will help you get the python development environment up and running. If you also need to work on the javascript admin backend, please refer to the admin module documentation.

# Install dependencies
$ pip install -r requirements-dev.txt
# create or update database
$ alembic upgrade head
# insert sample data
$ initialize_atramhasis_db development.ini
# generate first RDF download
$ dump_rdf development.ini
# compile the Message Catalog Files
$ python setup.py compile_catalog

Once you’ve executed these steps, you can run a development server. This uses the standard pyramid server (Waitress) and should not be used as-is in a production environment.

# run a local development server
$ pserve --reload development.ini

Admin development

To work on the admin part, you’ll need npm installed. Consult your operating system documentation on how to install these. The following instructions will assume you’re running a recent Debian based Linux distribution.

# install npm and grunt-cli
$ sudo apt install nodejs
$ sudo apt install npm
$ sudo npm install -g grunt-cli
# install js dependencies for public site using npm
$ cd atramhasis/static
$ npm install
# install js dependencies for admin using npm
$ cd atramhasis/static/admin
$ npm install

These commands will install a couple of js libraries that Atramhasis uses in /atramhasis/static/node_modules and /atramhasis/static/admin/node_modules and a set of tools to be able to generate js builds. Builds are carried out through a simple grunt file:

# Build a dojo distribution
$ cd atramhasis/static/admin
$ grunt -v build

This will create a build a place the resulting files in atramhasis/static/admin/dist. The web application can be told to use this build by setting dojo.mode in development.ini to dist.

Frontend development

When updating the frontend templates, you might want to add extra translations. This can be done by placing {% trans %} tags in the templates

<h2>{% trans %}welcome_to{% endtrans %}</h2>

To update the message catalogs, do as follows:

$ python setup.py extract_messages
$ python setup.py update_catalog -l fr -i atramhasis/locale/atramhasis.pot -o atramhasis/locale/fr/LC_MESSAGES/atramhasis.po
$ python setup.py update_catalog -l nl -i atramhasis/locale/atramhasis.pot -o atramhasis/locale/nl/LC_MESSAGES/atramhasis.po
$ python setup.py update_catalog -l en -i atramhasis/locale/atramhasis.pot -o atramhasis/locale/en/LC_MESSAGES/atramhasis.po

Update the catalogs accordingly and run:

$ python setup.py compile_catalog

You might also want to add a new translation. Suppose you want to add a German translation.

$ python setup.py init_catalog -l de -i atramhasis/locale/atramhasis.pot -o atramhasis/locale/de/LC_MESSAGES/atramhasis.po

Edit atramhasis/locale/do/LC_MESSAGES/atramhasis.po and add the necessary translations. Just as with updating the catalogs, you need to recompile them.

$ python setup.py compile_catalog

At this moment, Atramhasis will still only show the default languages in it’s language switcher. If you want to add you new language, you need to edit your development.ini (or similar file). Look for the line that says available_languages and add your locale identifier.

available_languages = en nl fr de

After restarting your server you will now have the option of switching to German.

Running a Linked Data Fragments server

If you want to add a Linked Data Fragments server, Atramhasis makes it easy for you. First you need to decide if you want to run the server with HDT files. If not, you can always use raw Turtle files, but be aware that the HDT files offer much better performance.

If you want to use HDT files, please install hdt-cpp. Be aware that you might have to download the source files and compile them yourself. Once you have done so, add the rdf2hdt command to your development.ini file. Supposing you installed it in /opt/hdt-cpp/hdt-lib/tools/rdf2hdt:

# Location of rdf2hdt executable
atramhasis.rdf2hdt = /opt/hdt-cpp/hdt-lib/tools/rdf2hdt

Now, whenever Atramhasis creates rdf dumps it will also create HDT files. If you do not have rdf2hdt installed, you will still have Turtle datadumps that can be used by the LDF-server.

$ dump_rdf development.ini

Now you’re ready to generate the configuration for the LDF server. Out of the box this file will be generated in the same directory your development.ini is located in, but you can override this in your ini file by setting atramhasis.ldf.config_location or you can pass this on the command line

# Generate config
$ generate_ldf_config development.ini
# Generate config and override config_location
$ generate_ldf_config development.ini -l /opt/my/ldf/server

Now you’re ready to run your LDF server. First we need to install it. It requires Node.js 4.0 or higher and should run on OSX and Linux. Please refer to the LDF server documentation for troubleshooting.

# Install ldf-server
$ [sudo] npm install -g ldf-server
# Run ldf-server
$ ldf-server ldf_server_config.json

Now you have an LDF server running at http://localhost:3000. Browse there and have fun!

When deploying Atramhasis with an LDF server in production, we recommend runnig both behind eg. nginx. In case you want to do this, you might run Atramhasis on port 6543 and LDF server on port 3000, but serve both through nginx. You can easily do this by setting the atramhasis.ldf.baseurl in your ini file. Suppose you want to server both on the host demo.atramhasis.org with Atramhasis as the root of your domain and the LDF server at /ldf. In this case, set atramhasis.ldf.baseurl to http://demo.atramhasis.org/ldf.

Contributing

Atramhasis is being developed as open source software by the Flanders Heritage Agency. All development is done on the agency’s Github page for Atramhasis.

Since we place a lot of importance of code quality, we expect to have a good amount of code coverage present and run frequent unit tests. All commits and pull requests will be tested with Travis-ci. Code coverage is being monitored with Coveralls.

Locally you can run unit tests by using pytest or tox. Running pytest manually is good for running a distinct set of unit tests. For a full test run, tox is preferred since this can run the unit tests against multiple versions of python.

# Run unit tests for all environments
$ tox
# No coverage
$ py.test
# Coverage
$ py.test --cov atramhasis --cov-report term-missing
# Only run a subset of the tests
$ py.test atramhasis/tests/test_views.py

Every pull request will be run through Travis-ci. When providing a pull request, please run the unit tests first and make sure they all pass. Please provide new unit tests to maintain 100% coverage. If you send us a pull request and this build doesn’t function, please correct the issue at hand or let us know why it’s not working.

Distribution

For building a distribution use the prepare command before the distribution command. This will update the requirement files in the scaffolds.

$ python setup.py prepare sdist

Services

Atramhasis can be used fully with a SOA. While we provide a public and and an administrator’s interface out of the box, you can also write your own client side code that interacts with the Atramhasis services, either for reading information or writing it.

Pyramid_skosprovider

This library takes your skosproviders and makes them available as REST services. The pyramid_skosprovider serves JSON as a REST service so it can be used easily inside a AJAX webbrowser call or by an external program.

The following API can be used by clients:

GET /uris

Find more information on a certain URI. This can map to eiter a concept, collection or conceptscheme that is known by the current SKOS registry.

Example request:

GET /uris?uri=urn:x-skosprovider:trees HTTP/1.1
Host: localhost:6543
Accept: application/json

Example response:

HTTP/1.1 200 OK
Content-Type:  application/json; charset=UTF-8

{
    "id": "TREES",
    "uri": "urn:x-skosprovider:trees",
    "type": "concept_scheme"
}

Example request:

GET /uris/?uri=http://python.com/trees/larch HTTP/1.1
Host: localhost:6543
Accept: application/json

Example response:

HTTP/1.1 200 OK
Content-Type:  application/json; charset=UTF-8

{
    "id": "1",
    "uri": "http://python.com/trees/larch",
    "type": "concept",
    "concept_scheme": {
        "id": "TREES",
        "uri": "urn:x-skosprovider:trees"
    }
}

Query Parameters:  

• uri – The URI to search for.

Status Codes:

• 200 OK – The URI maps to something known by pyramid_skosprovider, either a conceptscheme, a concept or collection.
• 404 Not Found – The URI can’t be found by pyramid_skosprovider.

GET /c

Search for concepts or collections, no matter what scheme they’re a part of.

Although it is possible to search a single conceptscheme with just this endpoint, for performance reasons it is advised to use GET /conceptschemes/{scheme_id}/c.

Example request:

GET /c HTTP/1.1
Host: localhost:6543
Accept: application/json

Example response:

HTTP/1.1 200 OK
Content-Range:  items 0-2/232
Content-Type:  application/json; charset=UTF-8

[
    {
        "id": "1",
        "uri": "urn:x-skosprovider:TREES:1",
        "type": "concept",
        "label": "De Lariks"
    }, {
        "id": "2",
        "uri": "urn:x-skosprovider:TREES:2",
        "type": "concept",
        "label": "De Paardekastanje"
    }, {
        "id": 3,
        "uri": "urn:x-skosprovider:TREES:3",
        "type": "collection",
        "label": "Bomen per soort"
    }
]

Example request:

GET /c?type=concept&providers.subject=external&sort=uri HTTP/1.1
Host: localhost:6543
Accept: application/json

Query Parameters:  

• type – Define if you want to show concepts or collections. Leave blank to show both.
• mode – Allows for special processing mode for dijitFilteringSelect. Makes it possible to use wildcards in the label parameter.
• label – Shows all concepts and collections that have this search string in one of their labels.
• language – Returns the label with the corresponding language-tag if present. If the language is not present for this concept/collection, it falls back to 1) the default language of the provider. 2) ‘en’ 3) any label. Eg. ?language=nl to show the dutch labels of the concepts/collections.
• sort – Define if you want to sort the results by a given field. Otherwise items are returned in an indeterminate order. Prefix with ‘+’ to sort ascending, ‘-‘ to sort descending. eg. ?sort=-label to sort all results descending by label.
• providers.ids – A comma separated list of concept scheme id’s. The query will only be passed to the providers with these id’s. eg. ?providers.ids=TREES, PARROTS will only list concepts from these two providers.
• providers.subject – A subject can be registered with a skosprovider in the registry. Adding this search parameter means that the query will only be passed on to providers that have been tagged with this subject. Eg. ?providers.subject=external to only query the providers that have been marked with the subject external.

Request Headers:  

• Range – Can be used to request a certain set of results. eg. items=0-24 requests the first 25 results.

Response Headers:  

• Content-Range – Tells the client what set of results is being returned eg. items=0-24/306 means the first 25 out of 306 results are being returned.

Status Codes:

• 200 OK – The concepts in this conceptscheme were found.

GET /conceptschemes

Get all registered conceptschemes.

Example request:

GET /conceptschemes HTTP/1.1
Host: localhost:6543
Accept: application/json

Example response:

HTTP/1.1 200 OK
Content-Type:  application/json; charset=UTF-8
Date:  Mon, 14 Apr 2014 14:42:34 GMT

[
    {
        "id": "TREES",
        "uri": "urn:x-skosprovider:trees",
        "label": "Different types of trees."
    }
]

Status Codes:

• 200 OK – The list of conceptschemes was found.

GET /conceptschemes/{scheme_id}

Get information about a concept scheme.

Example request:

GET /conceptschemes/TREES HTTP/1.1
Host: localhost:6543
Accept: application/json

Example response:

HTTP/1.1 200 OK
Content-Length:  15
Content-Type:  application/json; charset=UTF-8
Date:  Mon, 14 Apr 2014 14:45:37 GMT
Server:  waitress

{
    "id": "TREES",
    "uri": "urn:x-skosprovider:trees",
    "label": "Different types of trees.",
    "labels": [
        {"type": "prefLabel", "language": "en", "label": "Different types of trees."},
        {"type": "prefLabel", "language": "nl", "label": "Verschillende soorten bomen."}
    ]
}

Example request:

-.. sourcecode:: http

GET /conceptschemes/PLANTS HTTP/1.1 Host: localhost:6543 Accept: application/json

Example response:

HTTP/1.1 404 Not Found
Content-Length:  775
Content-Type:  text/html; charset=UTF-8
Date:  Tue, 15 Apr 2014 20:32:52 GMT
Server:  waitress

Status Codes:

• 200 OK – The conceptscheme was found.
• 404 Not Found – The conceptscheme was not found.

GET /conceptschemes/{scheme_id}/topconcepts

Get all top concepts in a certain conceptscheme. These are all the concepts in the conceptscheme that have no broader concept.

Example request:

GET /conceptschemes/TREES/topconcepts HTTP/1.1
Host: localhost:6543
Accept: application/json

Example response:

HTTP/1.1 200 OK
Content-Type:  application/json; charset=UTF-8
Date:  Mon, 14 Apr 2014 14:47:33 GMT
Server:  waitress

[
    {
        "id": "1",
        "uri": "urn:x-skosprovider:TREES:1",
        "type": "concept",
        "label": "De Lariks"
    }, {
        "id": "2",
        "uri": "urn:x-skosprovider:TREES:2",
        "type": "concept",
        "label": "De Paardekastanje"
    }
]

Query Parameters:  

• language – Returns the label with the corresponding language-tag if present. If the language is not present for this concept/collection, it falls back to 1) the default language of the provider. 2) ‘en’ 3) any label. Eg. ?language=nl to show the dutch labels of the concepts/collections.

Status Codes:

• 200 OK – The topconcepts in this conceptscheme were found.
• 404 Not Found – The conceptscheme was not found.

GET /conceptschemes/{scheme_id}/displaytop

Get the top of a display hierarchy. Depending on the underlying provider this will be a list of Concepts and Collections.

Example request:

GET /conceptschemes/TREES/displaytop HTTP/1.1
Host: localhost:6543
Accept: application/json

Example response:

HTTP/1.1 200 OK
Content-Type:  application/json; charset=UTF-8
Date:  Mon, 14 Apr 2014 14:47:33 GMT
Server:  waitress

[
    {
        "id": "1",
        "uri": "urn:x-skosprovider:TREES:1",
        "type": "concept",
        "label": "De Lariks"
    }, {
        "id": "2",
        "uri": "urn:x-skosprovider:TREES:2",
        "type": "concept",
        "label": "De Paardekastanje"
    }
]

Query Parameters:  

• language – Returns the label with the corresponding language-tag if present. If the language is not present for this concept/collection, it falls back to 1) the default language of the provider. 2) ‘en’ 3) any label. Eg. ?language=nl to show the dutch labels of the concepts/collections.

Status Codes:

• 200 OK – The concepts and collections were found.
• 404 Not Found – The conceptscheme was not found.

GET /conceptschemes/{scheme_id}/c

Search for concepts or collections in a scheme.

Example request:

GET /conceptschemes/TREES/c HTTP/1.1
Host: localhost:6543
Accept: application/json

Example response:

HTTP/1.1 200 OK
Content-Length:  117
Content-Range:  items 0-2/3
Content-Type:  application/json; charset=UTF-8
Date:  Mon, 14 Apr 2014 14:47:33 GMT
Server:  waitress

[
    {
        "id": "1",
        "uri": "urn:x-skosprovider:TREES:1",
        "type": "concept",
        "label": "De Lariks"
    }, {
        "id": "2",
        "uri": "urn:x-skosprovider:TREES:2",
        "type": "concept",
        "label": "De Paardekastanje"
    }, {
        "id": 3,
        "uri": "urn:x-skosprovider:TREES:3",
        "type": "collection",
        "label": "Bomen per soort"
    }
]

Example request:

GET /conceptschemes/PLANTS/c HTTP/1.1
Host: localhost:6543
Accept: application/json

Example response:

HTTP/1.1 404 Not Found
Content-Length:  775
Content-Type:  text/html; charset=UTF-8
Date:  Tue, 15 Apr 2014 20:32:52 GMT
Server:  waitress

Query Parameters:  

• type – Define if you want to show concepts or collections. Leave blank to show both.
• mode – Allows for special processing mode for dijitFilteringSelect. Makes it possible to use wildcards in the label parameter.
• label – Shows all concepts and collections that have this search string in one of their labels.
• collection – Get information about the content of a collection. Expects to be passed an id of a collection in this scheme. Will restrict the search to concepts or collections that are a member of this collection or a narrower concept of a member.
• language – Returns the label with the corresponding language-tag if present. If the language is not present for this concept/collection, it falls back to 1) the default language of the provider. 2) ‘en’ 3) any label. Eg. ?language=nl to show the dutch labels of the concepts/collections.
• sort – Define if you want to sort the results by a given field. Otherwise items are returned in an indeterminate order. Prefix with ‘+’ to sort ascending, ‘-‘ to sort descending. eg. ?sort=-label to sort all results descending by label.

Request Headers:  

• Range – Can be used to request a certain set of results. eg. items=0-24 requests the first 25 results.

Response Headers:  

• Content-Range – Tells the client was set of results is being returned eg. items=0-24/306 means the first 25 out of 306 results are being returned.

Status Codes:

• 200 OK – The concepts in this conceptscheme were found.
• 404 Not Found – The conceptscheme was not found.

GET /conceptschemes/{scheme_id}/c/{c_id}

Get information about a concept or collection.

Example request:

GET /conceptschemes/TREES/c/1 HTTP/1.1
Host: localhost:6543
Accept: application/json

Example response:

HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Date: Mon, 14 Apr 2014 14:49:27 GMT
Server: waitress

{
    "broader": [],
    "narrower": [],
    "notes": [
        {"note": "A type of tree.", "type": "definition", "language": "en"}
    ],
    "labels": [
        {"type": "prefLabel", "language": "en", "label": "The Larch"},
        {"type": "prefLabel", "language": "nl", "label": "De Lariks"}
    ],
    "type": "concept",
    "id": "1",
    "uri": "urn:x-skosprovider:TREES:1",
    "related": [],
    "label": "The Larch",
    "matches": {
        "close": [
            "http://id.python.org/different/types/of/trees/nr/1/the/larch"
        ]
    },
    "concept_scheme": {
        "uri": "urn:x-foo:bar"
    }
}

Example request:

GET /conceptschemes/TREES/c/4 HTTP/1.1
Host: localhost:6543
Accept: application/json

Example response:

HTTP/1.1 404 Not Found
Content-Length:  775
Content-Type:  text/html; charset=UTF-8
Date:  Tue, 15 Apr 2014 20:06:12 GMT
Server:  waitress

Status Codes:

• 200 OK – The concept was found in the conceptscheme.
• 404 Not Found – The concept was not found in the conceptscheme or the conceptscheme was not found.

GET /conceptschemes/{scheme_id}/c/{c_id}/displaychildren

Get a list of Collections and Concepts that should be displayed as children of this Concept or Collection.

Example request:

GET /conceptschemes/TREES/c/3/displaychildren HTTP/1.1
Host: localhost:6543
Accept: application/json

Example response:

HTTP/1.1 200 OK
Content-Type:  application/json; charset=UTF-8
Date:  Mon, 14 Apr 2014 14:49:27 GMT
Server:  waitress

[
    {
        "id": "1",
        "uri": "urn:x-skosprovider:TREES:1",
        "type": "concept",
        "label": "De Lariks"
    }, {
        "id": "2",
        "uri": "urn:x-skosprovider:TREES:2",
        "type": "concept",
        "label": "De Paardekastanje"
    }
]

Query Parameters:  

• language – Returns the label with the corresponding language-tag if present. If the language is not present for this concept/collection, it falls back to 1) the default language of the provider. 2) ‘en’ 3) any label. Eg. ?language=nl to show the dutch labels of the concepts/collections.

Status Codes:

• 200 OK – The concept was found in the conceptscheme.
• 404 Not Found – The concept was not found in the conceptscheme or the conceptscheme was not found.

GET /conceptschemes/{scheme_id}/c/{c_id}/expand

Expand a concept or collection to all it’s narrower concepts.

This method should recurse and also return narrower concepts of narrower concepts.

If the id passed belongs to a skosprovider.skos.Concept, the id of the concept itself should be include in the return value.

If the id passed belongs to a skosprovider.skos.Collection, the id of the collection itself must not be present in the return value In this case the return value includes all the member concepts and their narrower concepts.

Returns A list of id’s or HTTPNotFound if the concept or collection doesn’t

exist.

Example request:

GET /conceptschemes/TREES/c/3/expand HTTP/1.1
Host: localhost:6543
Accept: application/json

Example response:

HTTP/1.1 200 OK
Content-Type:  application/json; charset=UTF-8
Date:  Mon, 14 Apr 2014 14:49:27 GMT
Server:  waitress

[1 , 2]

Status Codes:

• 200 OK – The concept/collection was found in the conceptscheme.
• 404 Not Found – The concept/collection was not found in the conceptscheme or the conceptscheme was not found.

Atramhasis

Concepts and collections

The main Atramhasis write services allow you to add concepts and collections, edit them and delete them.

POST /conceptschemes/{scheme_id}/c

Add a concept or collection to a conceptscheme. The response body will contain a representation of the concept or collection after is has been added to the conceptscheme.

Example request:

POST /conceptschemes/TREES/c HTTP/1.1
Host: demo.atramhasis.org
Accept: application/json
Content-Type: application/json

{
    "type": "concept",
    "broader": [],
    "narrower": [],
    "related": [],
    "labels": [
        {
            "type": "prefLabel",
            "language": "en",
            "label": "The Larch"
        }
    ],
    "notes": []
}

Example response:

HTTP/1.1 201 Created
Location: http://demo.atramhasis.org/conceptschemes/TREES/c/1
Content-Type: application/json

{
    "id": 1,
    "uri": "urn:x-atramhasis-demo:TREES:1",
    "type": "concept",
    "broader": [],
    "narrower": [],
    "related": [],
    "labels": [
        {
            "type": "prefLabel",
            "language": "en",
            "label": "The Larch"
        }
    ],
    "notes": []
}

Example request:

POST /conceptschemes/TAUNTS/c HTTP/1.1
Host: demo.atramhasis.org
Accept: application/json
Content-Type: application/json

{
    "type": "concept",
    "broader": [],
    "narrower": [],
    "related": [],
    "labels": [
        {
            "type": "tauntLabel",
            "language": "en-FR",
            "label": "Your mother was a Hamster!"
        }
    ],
    "notes": []
}

Example response:

HTTP/1.1 400 Bad Request
Location: http://demo.atramhasis.org/conceptschemes/TREES/c/1
Content-Type: application/json

{
    "errors": [
                {"labels": "Invalid labeltype."},
                {"labels": "Invalid language."}
              ],
    "message": "Concept could not be validated"
}

Parameters:

• scheme_id – The identifier for a certain concept scheme.

Request Headers:  

• Accept – The response content type depends on this header. Currently only application/json is supported.

Response Headers:  

• Content-Type – This service currently always returns application/json
• Location – The url where the newly added concept or collection can be found.

Status Codes:

• 201 Created – The concept or collection was added succesfully.
• 400 Bad Request – The concept or collection could not be added because the submitted json was invalid due to eg. validation errors.
• 404 Not Found – The conceptscheme scheme_id does not exist.
• 405 Method Not Allowed – The concept or collection could not be added because the conceptscheme scheme_id is a readonly conceptscheme.

PUT /conceptschemes/{scheme_id}/c/{c_id}

Edit the concept or collection with id c_id. The response body will contain a representation of the concept or collection after the modifications.

Example request:

PUT /conceptschemes/TREES/c/1 HTTP/1.1
Host: demo.atramhasis.org
Accept: application/json
Content-Type: application/json

{
    "type": "concept",
    "broader": [],
    "narrower": [],
    "related": [],
    "labels": [
        {
            "type": "prefLabel",
            "language": "en",
            "label": "The Larch"
        }, {
            "type": "prefLabel",
            "language": "nl",
            "label": "De Lariks"
        }
    ],
    "notes": []
}

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
    "id": 1,
    "uri": "urn:x-atramhasis-demo:TREES:1",
    "type": "concept",
    "broader": [],
    "narrower": [],
    "related": [],
    "labels": [
        {
            "type": "prefLabel",
            "language": "en",
            "label": "The Larch"
        }, {
            "type": "prefLabel",
            "language": "nl",
            "label": "De Lariks"
        }
    ],
    "notes": []
}

Parameters:

• scheme_id – The identifier for a certain concept scheme.
• c_id – The identifier for a certain concept or collection.

Request Headers:  

• Accept – The response content type depends on this header. Currently only application/json is supported.

Response Headers:  

• Content-Type – This service currently always returns application/json

Status Codes:

• 200 OK – The concept or collection was edited succesfully.
• 400 Bad Request – The concept or collection could not be edited because the submitted json was invalid due to eg. validation errors.
• 404 Not Found – The conceptscheme scheme_id or the concept or collection c_id does not exist.
• 405 Method Not Allowed – The concept or collection could not be edited because the conceptscheme scheme_id is a readonly conceptscheme.

DELETE /conceptschemes/{scheme_id}/c/{c_id}

Remove the concept with id c_id. The response body will contain the last representation known by the service.

Example request:

DELETE /conceptschemes/TREES/c/1 HTTP/1.1
Host: demo.atramhasis.org
Accept: application/json

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
    "id": 1,
    "uri": "urn:x-atramhasis-demo:TREES:1",
    "type": "concept",
    "broader": [],
    "narrower": [],
    "related": [],
    "labels": [
        {
            "type": "prefLabel",
            "language": "en",
            "label": "The Larch"
        }, {
            "type": "prefLabel",
            "language": "nl",
            "label": "De Lariks"
        }
    ],
    "notes": []
}

Parameters:

• scheme_id – The identifier for a certain concept scheme.
• c_id – The identifier for a certain concept or collection.

Request Headers:  

• Accept – The response content type depends on this header. Currently only application/json is supported.

Response Headers:  

• Content-Type – This service currently always returns application/json

Status Codes:

• 200 OK – The concept or collection was deleted succesfully.
• 400 Bad Request – The concept or collection could not be edited because the submitted json was invalid due to eg. validation errors.
• 404 Not Found – The conceptscheme scheme_id or the concept or collection c_id does not exist.
• 405 Method Not Allowed – The concept or collection could not be deleted because the conceptscheme scheme_id is a readonly conceptscheme.
• 409 Conflict – The concept or collection could not be deleted because Atramhasis has determined that it’s still being used somewhere else. The response body will contain a message and a list of URI’s that are using this concept.

Languages

Apart from the main services, Atramhasis exposes some secondary services that deal with languages.

GET /languages

List all languages known to this Atramhasis instance.

Please bear in mind that these are not all known IANA language tags, but a subset used in this Atramhasis instance. This is used to populate drop down lists and such.

Example request:

GET /languages HTTP/1.1
Host: demo.atramhasis.org
Accept: application/json

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

[
    {"id": "la", "name": "Latin"},
    {"id": "nl", "name": "Dutch"},
    {"id": "en", "name": "English"},
    {"id": "fr", "name": "French"},
    {"id": "de", "name": "German"}
]

Parameters:

• sort – Which field to sort on. Use - and + to indicate sort order. Eg. id or +id sort ascending on id, -name sort descending on name.

Request Headers:  

• Accept – The response content type depends on this header. Currently only application/json is supported.

Response Headers:  

• Content-Type – This service currently always returns application/json

Status Codes:

• 200 OK – The list of languages was returned.

GET /languages/{language_id}

Get information on a certain language.

Please bear in mind this will only work for languages known to this Atramhasis instance. Valid IANA languages not known to this instance will not work.

Example request:

GET /languages HTTP/1.1
Host: demo.atramhasis.org
Accept: application/json

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
    "id": "la",
    "name": "Latin"
}

Request Headers:  

• Accept – The response content type depends on this header. Currently only application/json is supported.

Response Headers:  

• Content-Type – This service currently always returns application/json

Status Codes:

• 200 OK – The language was found.
• 404 Not Found – The language was not found in this instance.

PUT /languages/{language_id}

Update the information on a certain language or create an entry for a new one.

The user is required to submit the language_id and this must be a valid IANA language tag.

Example request:

PUT /languages/nl-BE HTTP/1.1
Host: demo.atramhasis.org
Accept: application/json
Content-Type: application/json

{
    "id": "nl-BE",
    "name": "Dutch (Flanders)"
}

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
    "id": "nl-BE",
    "name": "Dutch (Flanders)"
}

Request Headers:  

• Accept – The response content type depends on this header. Currently only application/json is supported.

Response Headers:  

• Content-Type – This service currently always returns application/json

Status Codes:

• 200 OK – The language was updated or added.
• 400 Bad Request – The request could not be executed because of problems with the submitted data. Most likely you are submitting an invalid IANA langage code.

DELETE /languages/{language_id}

Delete a language from this Atramhasis instance.

Example request:

DELETE /languages/nl-BE HTTP/1.1
Host: demo.atramhasis.org
Accept: application/json

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
    "id": "nl-BE",
    "name": "Dutch (Flanders)"
}

Request Headers:  

• Accept – The response content type depends on this header. Currently only application/json is supported.

Response Headers:  

• Content-Type – This service currently always returns application/json

Status Codes:

• 200 OK – The language was deleted.
• 404 Not Found – The language was not found in this instance.

Customisation

Out of the box Atramhasis tries to make as few assumptions as possible about setup. We have taken care to ensure that significant parts of the application are easy to customise and expect most installations to have custom code. We’ve shipped Atramhasis with sane defaults so you can get a quick feel for the capabilities of the software. However, we do not advise running a production instance with only these default settings.

Creating your own project

Whenever you want to run an instance of Atramhasis, you start by creating your own project. This is the place where you will maintain and develop your own custom templates, static assets such as stylesheets, your security implementation and other general configuration. To make it easier on you to get started, we provide a scaffold just for this. As always, we advise working in a virtual environment.

$ mkvirtualenv my_thesaurus
$ pip install atramhasis
$ pcreate -s atramhasis_scaffold my_thesaurus
$ cd my_thesaurus
# Install dependencies
$ pip install -r requirements-dev.txt
# Download and install client side libraries
$ cd my_thesaurus/static
$ npm install

This gives you a clean slate to start your customisations on.

Database

By default the scaffold comes with a simple SQLite database. This is more than enough for your first experiments and can even be used in production environment if your needs are modest. You can always instruct Atramhasis to use some other database engine, as long as SQLAlchemy supports it. Configure the sqlalchemy.url configuration option in development.ini to change the database. See the documentation of SQLAlchemy for more information about this connection url.

Database initialisation

To initialise the database, simply run the following.

# Create or update database based on
# the configuration in development.ini
$ alembic upgrade head

Custom alembic revisions

If you have a need to create your own tables, or do custom database changes we suggest you do so in another alembic branch next to the atramhasis branch.

First edit the alembic.ini file so it contains the following:

script_location = alembic
version_locations = %(here)s/alembic/versions atramhasis:alembic/versions

Second, initialise alembic in your project:

$ alembic init alembic

This will create an alembic folder for your own revisions.

To create your first revision, the command is a little longer:

$ alembic revision -m "first revision" --head=base --branch-label=myproject \
--version-path=alembic/versions

Note

if you need your alembic revisions to run after the atramhasis - for example if you want to create foreign keys to atramhasis tables - you can use --depends-on <hash> where the hash is the latest revision hash from atramhasis. This hash can be found by using alembic heads. In this example it is 184f1bbcb916

$ alembic heads
184f1bbcb916 (atramhasis) (head)

Having created a revision like above will have created a second alembic branch. Your alembic should have 2 heads now:

$ alembic heads
184f1bbcb916 (atramhasis) (effective head)
975228f4f18c (myproject) (head)

Adding additional revisions will look like:

alembic revision -m "second revision" --head=myproject@head

Warning

Not using a seperate branch will add revisions to the atramhasis alembic branch. While this may work initially, this may create split branches and multiple heads when upgrading atramhasis in the future and this is ill-advised

Whenever you would use alembic upgrade head to upgrade your database, you now have to use heads plural instead.

# Create or update database based on
# the configuration in development.ini
$ alembic upgrade heads

Running a local server

Your custom version of Atramhasis can now be run. Run the following command and point your browser to http://localhost:6543 to see the result.

$ pserve development.ini

Creating conceptschemes

Atramhasis is now running but does not contain any ConceptSchemes. You will need to configure this by entering a database record for the ConceptScheme and writing a small piece of code.

Warning

Instantiating providers has changed between version 0.6.x and 0.7.0. Make sure to update your skos initialisation when updating. The old code is no longer supported, although the changes you need to make are minor.

To enter the database record, you need to enter a record in the table conceptscheme. In this table you need to register an id for the conceptscheme and a uri. The id is for internal database use and has no other meaning. The uri can be used externally. To register a new ConceptScheme in the sqlite database that was created:

$ sqlite3 my_thesaurus.sqlite

INSERT INTO conceptscheme VALUES (1, 'urn:x-my-thesaurus:stuff')

This takes care of the first step. Now you also need to tell Atramhasis where to find your conceptscheme and how to handle it. To do this, you need to edit the file called my_thesaurus/skos/__init__.py. This is the default location for creating a registry factory. Be default, this function is called create_registry, but this can be changed in your development.ini file. The function itself needs to receive the current request as a parameter and return the instantiated skosprovider.registry.Registry.

In this funcion you will register SQLAlchemyProvider instances to the SKOS registry. If not yet present, you need to tell Python where to find such a provider by adding this code to the top of the file:

from skosprovider_sqlalchemy.providers import SQLAlchemyProvider

Then you need to instantiate such a provider within the create_registry function in this file. This provider needs a few arguments: an id for the provider, an id for the conceptscheme it’s working with and a connectionb to a database session. The id for the provider is often a text string and will appear in certain url’s and might popup in the user interface from time to time. The database session is added to the Pyramid request that is passed to function and can be reached as request.db. Finally, you need to register this provider with the skosprovider.registry.Registry.

STUFF = SQLAlchemyProvider(
    {
        'id': 'STUFF',
        'conceptscheme_id': 1
    },
    request.db
)

registry.register_provider(STUFF)

After having registered your provider, the file should look more or less like this:

# -*- coding: utf-8 -*-

from skosprovider.registry import Registry
from skosprovider.uri import UriPatternGenerator
from skosprovider_sqlalchemy.providers import SQLAlchemyProvider

import logging
log = logging.getLogger(__name__)


def create_registry(request):
    # create the SKOS registry
    registry = Registry(instance_scope='threaded_thread')

    # create your own providers
    STUFF = SQLAlchemyProvider(
        {'id': 'STUFF', 'conceptscheme_id': 1},
        request.db
    )

    # Add your custom provider to the registry
    registry.register_provider(STUFF)

    # return the SKOS registry
    return registry

Now you can restart your server and then you front page will show you a new, but empty thesaurus.

Creating concepts and collections

You can now start creating concepts and collections by going to the admin interface at http://localhost:6543/admin.

You will notice that any concepts or collections you create wil get a URI similar to urn:x-skosprovider:STUFF:1. This is due to the fact that your SQLAlchemyProvider has a UriGenerator that generates uris for the provider. By default, the provider configures a DefaultUrnGenerator, but it’s expected that you will want to override this.

Warning

The UriGenerator that you configure only generates URI’s when creating new concepts or collections. When importing existing vocabularies, please be sure to create the URI’s before or during import (possbily by using a relevant generator yourself).

Suppose you have decided that your URI’s should look like this: http://id.mydata.org/thesauri/stuff/[id]. You can do this by registering a UriPatternGenerator with your provider:

STUFF = SQLAlchemyProvider(
    {
        'id': 'STUFF',
        'conceptscheme_id': 1
    },
    request.db,
    uri_generator=UriPatternGenerator(
        'http://id.mydata.org/thesauri/stuff/%s'
    )
)

Don’t forget to import the UriPatternGenerator at the top of your file:

from skosprovider.uri import UriPatternGenerator

Your final file should look similar to this:

# -*- coding: utf-8 -*-

from skosprovider.registry import Registry
from skosprovider.uri import UriPatternGenerator
from skosprovider_sqlalchemy.providers import SQLAlchemyProvider

import logging
log = logging.getLogger(__name__)


def create_registry(request):
    # create the SKOS registry
    registry = Registry(instance_scope='threaded_thread')

    # create your own providers
    STUFF = SQLAlchemyProvider(
        {'id': 'STUFF', 'conceptscheme_id': 1},
        request.db,
        uri_generator=UriPatternGenerator(
            'http://id.mydata.org/thesauri/stuff/%s'
        )
    )

    # Add your custom provider to the registry
    registry.register_provider(STUFF)

    # return the SKOS registry
    return registry

If you need more complicated URI’s, you can easily write you own generator with a small piece of python code. You just need to follow the interface provided by skosprovider.uri.UriGenerator.

Hiding a vocabulary

Atramhasis allows you to hide a vocabulary. This means the vocabulary is still there as far as services are concerned and you can still edit it. But it will not be visible in the public html user interface. You might want to use it for small and rather technical vocabularies you need but don’t want to draw attention to. The only thing you need to do, is tagging this provider with a subject. By adding the hidden subject to the provider, we let Atramhasis know that this vocabulary should not be present among your regular vocabularies.

Suppose we wanted to hide our stuff:

# -*- coding: utf-8 -*-

import logging
log = logging.getLogger(__name__)

from skosprovider.registry import Registry
from skosprovider_sqlalchemy.providers import SQLAlchemyProvider
from skosprovider.uri import UriPatternGenerator


def create_registry(request):
    # create the SKOS registry
    registry = Registry(instance_scope='threaded_thread')

    # create your own providers
    #
    STUFF = SQLAlchemyProvider(
        {
            'id': 'STUFF',
            'conceptscheme_id': 1,
            'subject': ['hidden']
        },
        request.db,
        uri_generator=UriPatternGenerator(
            'http://id.mydata.org/thesauri/stuff/%s'
        )
    )

    # Add your custom provider to the registry
    registry.register_provider(STUFF)

    # return the SKOS registry
    return registry

Now the STUFF thesaurus will not show up in the public web interface, but REST calls to this conceptscheme will function as normal and you will be able to maintain it from the admin interface.

Force a display language for a vocabulary

Under normal circumstances, Atramhasis tries to provide the most appropriate label for a certain concept or collection, based on some default configuration and the preferences of the end-user. Every provider can be marked as having a certain default language (English if not set), but Atramhasis also tries to read what the user wants. It does this through the user’s browser’s locale. This information can be read from the browser’s HTTP headers or cookies. Generally, Atramhasis just knows in what language a user is browsing the site and tries to return labels appropriate for that language. So, the same thesaurus visited from the US will return English labels, while it will return Dutch when visited from Gent (Belgium).

You might have a vocabulary with a strongly preferential relation to a certain language. We ran into this situation with a vocabulary of species: names for plants and trees commonly found in Flanders. Some of them have one or more local, Dutch, names. Most or all of them have an official name in Latin. The normal language handling mechanism created a weird situation. It led to a tree of names that was mostly in Latin, with the odd Dutch word thrown in for good measure. This was not as desired by our users. To that end, a special mechanism was created to force rendering labels of concepts and collections in a certain language, no matter what the end-user’s browser is requesting.

To set this, please edit the my_thesaurus/skos/__init__.py. Look for the thesaurus you want to override and add a setting atramhasis.force_display_label_language to the provider’s metadata. Set it to a language supported by the provider (there’s little sense to setting it to a language that isn’t present in the vocabulary). Now Atramhasis will try serving concepts from this provider with this language. All labels will still be shown, but the page title or current label will be set to the selected language as much as possible. The normal language determination mechanisms will keep on working, so if the concept has no label in the requested language, Atramhasis will fall back on other labels present.

Your provider should end up similar to this:

STUFF = SQLAlchemyProvider(
    {
        'id': 'STUFF',
        'conceptscheme_id': 1,
        'atramhasis.force_display_label_language': 'la'
    },
    request.db,
    uri_generator=UriPatternGenerator(
        'http://id.mydata.org/thesauri/stuff/%s'
    )
)

Beware that this will only affect the Atramhasis UI, not the Atramhasis REST services. We looked into some solutions for our problem that would have also changed the underlying service, but decided against that because it would have prevented you from making your own choices when interacting with Atramhasis. If you want to render the tree of concepts using a preferred language different from what a browser would advocate for, you can pass the language parameter in a url, eg. http://my.thesaurus.org/conceptschemes/STUFF/tree?language=la.

Internationalisation

When you create a new empty project with the atramhasis scaffold, you get an English only version. The standard version of Atramhasis has been translated in Dutch and French. If you desire, you can activate these by editing your project’s development.ini

# Edit and uncomment to activate nl and fr language support or other languages
# you have added yourself.
available_languages = en nl fr

Available languages should be a space separated list of IANA language codes. If you add new languages, please consider contributing them back to the project.

Appearance

By implementing a few simple techniques from the Pyramid web framework, it’s very easy to customise the look and feel of the public user interface. The default implementation is a very neutral implementation based on the basic elements in the Foundation framework. Customising and overriding this style is possible if you have a bit of knowledge about HTML and CSS.

You can also override the HTML templates that Atramhasis uses without needing to alter the originals so that future updates to the system will not override your modifications.

Overriding templates

One very easy technique to use, is Pyramid’s override assets mechanism. This allows you to override a core Atramhasis template with your own template. Suppose we want to change the text on the Atramhasis homepage to welcome visitors to your instances. This text can be found in atramhasis/templates/welcome.jinja2.

Assuming that you created your project as my_thesaurus, we can now create our own template in my_thesaurus/templates/my_welcome.jinja2. Please consult the Jinja2 documentation if you need help with this.

Once you’ve created your template file, you just need to tell your project to override the default welcome.jinja2 with your version. To do this you need to configure the Pyramid config object found in my_thesaurus.__init__.py.

config.override_asset(
    to_override='atramhasis:templates/welcome.jinja2',
    override_with='templates/my_welcome.jinja2'
)

Note

Normally, to see the effect of the changes you made, you would need to restart your webserver. When developing, you can make use of the pserve command’s auto-reload feature. To do this, start your server like this:

$ pserve --reload development.ini

Changing the focal conceptschemes

An Atramhasis instance should contain one or more conceptschemes. Four of your conceptschemes can be picked to receive a little more attention and focus than the other ones. These conceptschemes will appear on the homepagina with a list of recently visited concepts in those schemes.

Selecting which conceptschemes receive this focus is done in your development.ini file.

layout.focus_conceptschemes =
  STUFF

This should be a space or newline delimited list, limited to 4 entries.

Changing the CSS

Out of the box, Atramhasis, comes with the Zurb Foundation framework. We have created a custom style for this framework, but as always you are free to modify this style. Your custom instance contains a few extension points that make it easy to override and change style elements without having to rewrite to much css. All style related files can be found in the my_thesaurus/static folder. This project’s CSS is being maintained and generated by Compass. You will find a scss folder that contains three files that can be used for easy customisations: _my_thesaurus-settings.scss, _my_thesaurus.scss and _my_thesaurus-admin.scss. The first file is a settings file that allows you to override a lot of variables that are used in generating the css. Suppose you want to override the default row width and the default font. You would change _my_thesarus-settings.scss to the following:

// Custom SASS code for my_thesaurus

$row-width: rem-calc(1140);
$body-font-family: "museo-sans", "Open Sans", "Helvetica", Helvetica, Arial, sans-serif;

To have you changes take effect, you need to recompile the scss files and restart your webserver.

$ compass compile
    write css/app-admin.css
    write css/app.css

The other two files, _my_thesaurus.scss and _my_thesaurus-admin.scss are the final scss files loaded before compiling them and can be used to overwrite things in the public or admin interface.

Security

We assume that every deployment of Atramhasis has different needs when it comes to security. Some instances will run on a simple laptop for testing and evaluation purposes, others might need a simple standalone database of users and certain deployments might need to integrate with enterprise authentication systems like LDAP, Active Directory, Single Sign On, …

Atramhasis provides authorisation hooks for security. To edit, add or delete a concept or collection, a user is required to have the ‘editor’ pemission. Unless no authorisation policy has been configured.

To get started, consult the sections of the Pyramid documentation on security.

Prior to version 0.6.3, Atramhasis contained a demo scaffold that had a custom security implementation using Mozilla Persona. Since this service has been discontinued, the security configuration was removed as well. But you can still check out the old code in our Github repository to see how it works.

Sitemap

Since Atramhasis 0.7.0 it’s possible to generate a sitemap. It consists of a set of files (one per conceptscheme and an index file) you can submit to a search engine. It will help it index your thesaurus as efficiently as possible.

You can generate the sitemap using the following commands:

# remove any existing sitemaps
$ rm my_thesaurus/static/_sitemaps/*
$ sitemap_generator development.ini

The sitemap index xml will be visible at the root of your webserver, eg. http://localhost:6543/sitemap_index.xml. Depending on how often you edit conceptschemes, concepts or collections it’s a good idea to make this into a cron job. When recreating the sitemap it is best practice to remove existing files from the static/_sitemap directory. If the directory is not empty the script will overwrite existing sitemaps, but unused sitemaps will be retained. Unless the –no-input flag is used, the script wil ask the user to press [enter] before overwriting existing files. The sitemap index will always contain links to all the files (used and unused).

Since a sitemap needs to contain abolute URL’s, the script needs to know where the application is being hosted. This can be controlled with a setting atramhasis.url in the application’s ini file. Set this to the root of your webapplication, eg. http://my.thesaurus.org (no trailing slash needed).

Foreign Keys

Atramhasis will often function as a central part of a SOA in an organisation. Concept and maybe Collection objects will be used by other applications. One of the riskier aspects of this is that someone might delete a concept in a certain scheme that is still being used by another application. Even worse, the user approving the delete might not even have a clue that the concept is being used by some external application. While in the decentralised world that is the world wide web, we can never be sure that nobody is using our concept any more, we can take some steps to at least control what happens within other applications that are within our control.

Of course, within the framework that is Atramhasis it’s very difficult to know how or where your own resources might be and how they might be using concepts from Atramhasis. We have therefor provided the necessary hooks for you that can help you deal with the sort of situation. But the actual implementation is left up to you.

We have added a decorator protected_operation(). When you add this decorator to a view, this view will emit a ProtectedResourceEvent. By default we have added this decorator the delete_concept() view.

In you own code, you can subscribe to this ProtectedResourceEvent through the usual pyramid.events.subscriber(). In this event handler you are then free to implement whatever check you need to do. If you find that the resource in question is being used somewhere and this operation should thus not be allowed to proceed, you simply need to raise a atramhasis.protected_resources.ProtectedResourceException. Into this exception you can also pass a list of URI that might provide the user with some feedback as to where this concept might be used.

For example, a sample event handler that would make it impossible to delete concepts with a URI of less than 5 characters:

from pyramid.events import subscriber
from atramhasis.protected_resources import ProtectedResourceEvent

@subscriber(ProtectedResourceEvent)
def never_delete_a_short_uri(event):
    if len(event.uri) < 5:
        raise ProtectedResourceException(
            'resource {0} has a URI shorter than 5 characters, preventing this operation'.format(event.uri),
            []
        )

Adding Google Analytics

Out of the box, it’s very easy to add Google Analytics integration to Atramhasis. All you need to do is add you Web Property ID to development.ini.

# Enter your Google Analytics Web Property ID
ga.tracker_key = UA-12345678-9

This will add basic analytics to every page, using a Jinja2 macro. If you need more control over the code, you can override this macro in your own project. Suppose you always want to use SSL when sending data. First, you would create you own macro, eg. in my_macros.jinja2 in the templates directory of your own project.

{% macro ga_tracker(ga_key) %}
    <!-- Google Analytics -->
    <script type="text/javascript">
    (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
    (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
    m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
    })(window,document,'script','//www.google-analytics.com/analytics.js','ga');

    ga('create', '{{ ga_key }}', 'auto');
    ga('set', 'forceSSL', true);
    ga('send', 'pageview');
    </script>
    <!-- End Google Analytics -->
{% endmacro %}

Once that’s done, you need to override the the ga block in the base template. To do this, it’s easiest to override Atramhasis’ base.jinja2 in your own project. To do that, add the following to your project’s main function:

config.override_asset(
    to_override='atramhasis:templates/base.jinja2',
    override_with='templates/base.jinja2'
)

In this file, you can now choose what should appear within the ga block defined in staticbase.jinja2. Here we are just replacing one macro with another, but you are off course free to make further alterations.

{%- extends 'staticbase.jinja2' -%}

{% block ga %}
    {% set ga_key = ga_key|default(request.registry.settings["ga.tracker_key"]) %}
    {% from 'my_macros.jinja2' import ga_tracker %}
    {% if ga_key %}
        {{ ga_tracker(ga_key) }}
    {% endif %}
{% endblock %}

Adding external providers

Within your Atramhasis instance you can make use of external providers. These are other systems serving up thesauri that you can interact with. Within the admin interface you can create links to these thesauri as SKOS matches. This way you can state that a concept within your thesauri is the same as or similar to a concept in the external thesaurus. And, more interestingly, you can also import concepts from such a thesaurus into your own vocabulary. Importing a concept like this will automatically create a SKOS match for you. Once a match is in place, you can also update your local concept with information from the external concept by performing a merge.

To enable all this power, you again need to configure a provider in you application. Continuing with our example project, we need to go back to our my_thesaurus/skos/__init__.py. In this file you need to register other instances of skosprovider.providers.VocabularyProvider. Currently providers have already been written for Getty Vocabularies, English Heritage vocabularies and Flanders Heritage Vocabularies. Depending on the system you’re trying to interact with, writing a new provider is fairly simple. For this example, we’ll assume that you want to integrate the wealth of information that the Art and Architecture Thesaurus (AAT) vocabulary offers you.

The AATProvider for this (and other Getty vocabularies) is available as skosprovider_getty and is installed by default in an Atramhasis instance. All you need to do is configure it. First, we need to import the provider. Place this code at the top of my_thesaurus/skos/__init__.py.

from skosprovider_getty.providers import AATProvider

Once this is done, we need to instantiate the provider within the includeme function and register it with the skosprovider.registry.Registry. This is all quite similar to registering your own skosprovider_sqlalchemy.providers.SQLAlchemyProvider. One thing you do need to do, is tagging this provider with a subject. By adding the external subject to the provider, we let Atramhasis know that this is not a regular, internal provider that can be stored in our database, but a special external one that can only be used for making matches. As such, it will not be present and visible to the public among your regular vocabularies.

AAT = AATProvider(
    {'id': 'AAT', 'subject': ['external']},
)
registry.register_provider(AAT)

That’s all. You can do the same with the TGNProvider for the Thesaurus of Geographic Names (TGN) or any of the providers for heritagedata.org that can be found in skosprovider_heritagedata.

In the end your my_thesaurus/skos/__init__.py should look somewhat like this:

# -*- coding: utf-8 -*-

import logging
log = logging.getLogger(__name__)

from skosprovider.registry import Registry
from skosprovider_sqlalchemy.providers import SQLAlchemyProvider
from skosprovider_getty.providers import AATProvider
from skosprovider.uri import UriPatternGenerator


def create_registry(request):
    # create the SKOS registry
    registry = Registry(instance_scope='threaded_thread')

    STUFF = SQLAlchemyProvider(
        {
            'id': 'STUFF',
            'conceptscheme_id': 1
        },
        request.db,
        uri_generator=UriPatternGenerator(
            'http://id.mydata.org/thesauri/stuff/%s'
        )
    )

    AAT = AATProvider(
        {
            'id': 'AAT',
            'subject': ['external']
        }
    )

    registry.register_provider(STUFF)
    registry.register_provider(AAT)

    return registry

Now you’ll be able to import from the AAT to your heart’s delight. For an extended example that adds even more providers, you could have a look at the demo scaffold that comes with Atramhasis.

Import a controlled vocabulary

Atramhasis includes a script atramhasis/scripts/import_file.py which helps you import an existing vocabulary from a file. It supports a few different file types, but not every file type supports the full Atramhasis datamodel.

The supported file types:

• RDF using RDFProvider. This provider supports the full datamodel. Since the heavy lifting is done by RDFlib, most of the dialects supported by RDFlib should work. The full list can be found in rdflib.util.SUFFIX_FORMAT_MAP. Formats like rdf/xml and turtle should work.
• CSV (.csv) using SimpleCsvProvider. The provider only supports importing and id, a prefLabel, a note and a source. It will work well when importing a simple flat list, but not for complex hierarchies.
• JSON (.json) using DictionaryProvider. This provider supports the full datamodel.

Some things to take into account:

• Atramhasis only supports concepts with a numeric id. This ensures they can be auto-generated when adding new concepts or collections. These map to the concept_id attribute in the database, which is unique per conceptscheme as opposed to the id attribute that is unique for the entire database.
• When importing from an RDF vocabulary, the id will be read from a dc or dcterms identifier property if present. Please ensure this property contains a numeric id, not a string or a URI.
• When importing from RDF, the import file could possibly contain more than one conceptscheme. Please ensure only one conceptscheme is present or no conceptschemes are presents and specify the URI and label on the command line.
• When importing from CSV or JSON, the data file only contains the concepts and collections in the scheme, but not the conceptscheme itself. In this case, please specify the URI and label of the conceptscheme on the command line.

The script can be called through the commandline in the project virtual environment. Call it with the help argument to see the possible arguments.

$ workon my_thesarus
$ import_file --help

usage: import_file [--from path_input_file] [--to conn_string] [--conceptscheme_label cs_label]
 (example: "import_file --from atramhasis/scripts/my_file --to sqlite:///atramhasis.sqlite --conceptscheme_label Labels")

Import file to a database

optional arguments:
  -h, --help            show this help message and exit
  --from INPUT_FILE     local path to the input file
  --to TO               Connection string of the output database
  --conceptscheme_label CS_LABEL
                        Label of the conceptscheme

The from argument is required and details where the file you want to import is located, for example my_thesaurus/data/trees.json. It is relative to your current location.

The to argument contains the connection string of output database. Only PostGreSQL and SQLite are supported. The structure is either postgresql://username:password@host:port/db_name or either sqlite:///path/db_name.sqlite. The default value is sqlite:///atramhasis.sqlite.

The data is loaded in a ConceptScheme. With a RDFProvider the conceptscheme can be present in the RDF file. The other providers can specify it on the command line through the conceptscheme_label argument. If no conceptscheme_label is present, the default label is the name of the file.

Once the data is loaded in the database, the configuration of the added provider must be included in the my_thesaurus/skos/__init__.py. A successfull run of the script will give a suggestion of the code to add to this file. Make sure to use the same ConceptSchem ID since it is needed to connect your provider and the conceptscheme in the database.

For example, to insert this file:

[{"broader": [],
  "id": 1,
  "labels": [{"label": "The Larch",
               "language": "en",
               "type": "prefLabel"},
              {"label": "De Lariks",
               "language": "nl",
               "type": "prefLabel"}],
  "matches": {"broad": [],
               "close": [],
               "exact": [],
               "narrow": [],
               "related": []},
  "member_of": [3],
  "narrower": [],
  "notes": [{"language": "en",
              "note": "A type of tree.",
              "type": "definition"}],
  "related": [],
  "subordinate_arrays": [],
  "type": "concept",
  "uri": "http://id.trees.org/1"},
 {"broader": [],
  "id": 2,
  "labels": [{"label": "The Chestnut",
               "language": "en",
               "type": "prefLabel"},
              {"label": "De Paardekastanje",
               "language": "nl",
               "type": "altLabel"},
              {"label": "la châtaigne",
               "language": "fr",
               "type": "altLabel"}],
  "matches": {"broad": [],
               "close": [],
               "exact": [],
               "narrow": [],
               "related": []},
  "member_of": [3],
  "narrower": [],
  "notes": [{"language": "en",
              "note": "A different type of tree.",
              "type": "definition"}],
  "related": [],
  "subordinate_arrays": [],
  "type": "concept",
  "uri": "http://id.trees.org/2"},
 {"id": 3,
  "labels": [{"label": "Bomen per soort",
               "language": "nl",
               "type": "prefLabel"},
              {"label": "Trees by species",
               "language": "en",
               "type": "prefLabel"}],
  "member_of": [],
  "members": [1, 2],
  "notes": [],
  "superordinates": [],
  "type": "collection",
  "uri": "http://id.trees.org/3"}]

We run the following command:

$ workon my_thesarus
$ import_file --from my_thesaurus/data/trees.json --to sqlite:///my_thesaurus.sqlite --conceptscheme_label Trees

This will return output similar to this:

sqlalchemy.engine.base.Engine SELECT CAST('test plain returns' AS VARCHAR(60)) AS anon_1
sqlalchemy.engine.base.Engine ()
sqlalchemy.engine.base.Engine SELECT CAST('test unicode returns' AS VARCHAR(60)) AS anon_1
sqlalchemy.engine.base.Engine ()
sqlalchemy.engine.base.Engine BEGIN (implicit)
sqlalchemy.engine.base.Engine INSERT INTO note (note, notetype_id, language_id) VALUES (?, ?, ?)
sqlalchemy.engine.base.Engine ('A type of tree.', 'definition', 'en')
sqlalchemy.engine.base.Engine INSERT INTO note (note, notetype_id, language_id) VALUES (?, ?, ?)
sqlalchemy.engine.base.Engine ('A different type of tree.', 'definition', 'en')
sqlalchemy.engine.base.Engine INSERT INTO conceptscheme (uri) VALUES (?)
sqlalchemy.engine.base.Engine (None,)
sqlalchemy.engine.base.Engine INSERT INTO label (label, labeltype_id, language_id) VALUES (?, ?, ?)
sqlalchemy.engine.base.Engine ('Trees', 'prefLabel', 'nl')
sqlalchemy.engine.base.Engine INSERT INTO label (label, labeltype_id, language_id) VALUES (?, ?, ?)
sqlalchemy.engine.base.Engine ('The Larch', 'prefLabel', 'en')
sqlalchemy.engine.base.Engine INSERT INTO label (label, labeltype_id, language_id) VALUES (?, ?, ?)
sqlalchemy.engine.base.Engine ('De Lariks', 'prefLabel', 'nl')
sqlalchemy.engine.base.Engine INSERT INTO label (label, labeltype_id, language_id) VALUES (?, ?, ?)
sqlalchemy.engine.base.Engine ('The Chestnut', 'prefLabel', 'en')
sqlalchemy.engine.base.Engine INSERT INTO label (label, labeltype_id, language_id) VALUES (?, ?, ?)
sqlalchemy.engine.base.Engine ('De Paardekastanje', 'altLabel', 'nl')
sqlalchemy.engine.base.Engine INSERT INTO label (label, labeltype_id, language_id) VALUES (?, ?, ?)
sqlalchemy.engine.base.Engine ('la châtaigne', 'altLabel', 'fr')
sqlalchemy.engine.base.Engine INSERT INTO label (label, labeltype_id, language_id) VALUES (?, ?, ?)
sqlalchemy.engine.base.Engine ('Bomen per soort', 'prefLabel', 'nl')
sqlalchemy.engine.base.Engine INSERT INTO label (label, labeltype_id, language_id) VALUES (?, ?, ?)
sqlalchemy.engine.base.Engine ('Trees by species', 'prefLabel', 'en')
sqlalchemy.engine.base.Engine INSERT INTO conceptscheme_label (conceptscheme_id, label_id) VALUES (?, ?)
sqlalchemy.engine.base.Engine (11, 3548)
sqlalchemy.engine.base.Engine INSERT INTO concept (type, concept_id, uri, conceptscheme_id) VALUES (?, ?, ?, ?)
sqlalchemy.engine.base.Engine ('concept', 1, 'http://id.trees.org/1', 11)
sqlalchemy.engine.base.Engine INSERT INTO concept (type, concept_id, uri, conceptscheme_id) VALUES (?, ?, ?, ?)
sqlalchemy.engine.base.Engine ('concept', 2, 'http://id.trees.org/2', 11)
sqlalchemy.engine.base.Engine INSERT INTO concept (type, concept_id, uri, conceptscheme_id) VALUES (?, ?, ?, ?)
sqlalchemy.engine.base.Engine ('collection', 3, 'http://id.trees.org/3', 11)
sqlalchemy.engine.base.Engine INSERT INTO concept_label (concept_id, label_id) VALUES (?, ?)
sqlalchemy.engine.base.Engine ((2558, 3551), (2558, 3552), (2558, 3553), (2557, 3549), (2557, 3550), (2559, 3554), (2559, 3555))
sqlalchemy.engine.base.Engine INSERT INTO concept_note (concept_id, note_id) VALUES (?, ?)
sqlalchemy.engine.base.Engine ((2558, 3605), (2557, 3604))
sqlalchemy.engine.base.Engine SELECT concept.id AS concept_id_1, concept.type AS concept_type, concept.concept_id AS concept_concept_id, concept.uri AS concept_uri, concept.conceptscheme_id AS concept_conceptscheme_id
FROM concept
WHERE concept.conceptscheme_id = ? AND concept.concept_id = ? AND concept.type IN (?)
sqlalchemy.engine.base.Engine (11, 1, 'concept')
sqlalchemy.engine.base.Engine SELECT concept.id AS concept_id_1, concept.type AS concept_type, concept.concept_id AS concept_concept_id, concept.uri AS concept_uri, concept.conceptscheme_id AS concept_conceptscheme_id
FROM concept
WHERE concept.conceptscheme_id = ? AND concept.concept_id = ? AND concept.type IN (?)
sqlalchemy.engine.base.Engine (11, 2, 'concept')
sqlalchemy.engine.base.Engine SELECT concept.id AS concept_id_1, concept.type AS concept_type, concept.concept_id AS concept_concept_id, concept.uri AS concept_uri, concept.conceptscheme_id AS concept_conceptscheme_id
FROM concept
WHERE concept.conceptscheme_id = ? AND concept.concept_id = ? AND concept.type IN (?)
sqlalchemy.engine.base.Engine (11, 3, 'collection')
sqlalchemy.engine.base.Engine SELECT concept.id AS concept_id_1, concept.type AS concept_type, concept.concept_id AS concept_concept_id, concept.uri AS concept_uri, concept.conceptscheme_id AS concept_conceptscheme_id
FROM concept
WHERE concept.conceptscheme_id = ? AND concept.concept_id = ?
sqlalchemy.engine.base.Engine (11, 1)
sqlalchemy.engine.base.Engine SELECT concept.id AS concept_id_1, concept.type AS concept_type, concept.concept_id AS concept_concept_id, concept.uri AS concept_uri, concept.conceptscheme_id AS concept_conceptscheme_id
FROM concept, collection_concept
WHERE ? = collection_concept.collection_id AND concept.id = collection_concept.concept_id
sqlalchemy.engine.base.Engine (2559,)
sqlalchemy.engine.base.Engine INSERT INTO collection_concept (collection_id, concept_id) VALUES (?, ?)
sqlalchemy.engine.base.Engine (2559, 2557)
sqlalchemy.engine.base.Engine SELECT concept.id AS concept_id_1, concept.type AS concept_type, concept.concept_id AS concept_concept_id, concept.uri AS concept_uri, concept.conceptscheme_id AS concept_conceptscheme_id
FROM concept
WHERE concept.conceptscheme_id = ? AND concept.concept_id = ?
sqlalchemy.engine.base.Engine (11, 2)
sqlalchemy.engine.base.Engine INSERT INTO collection_concept (collection_id, concept_id) VALUES (?, ?)
sqlalchemy.engine.base.Engine (2559, 2558)
sqlalchemy.engine.base.Engine COMMIT
sqlalchemy.engine.base.Engine BEGIN (implicit)
sqlalchemy.engine.base.Engine SELECT label.id AS label_id, label.label AS label_label, label.labeltype_id AS label_labeltype_id, label.language_id AS label_language_id
FROM label JOIN conceptscheme_label ON label.id = conceptscheme_label.label_id
WHERE label.label = ?
 LIMIT ? OFFSET ?
sqlalchemy.engine.base.Engine ('Trees', 1, 0)
sqlalchemy.engine.base.Engine SELECT conceptscheme.id AS conceptscheme_id, conceptscheme.uri AS conceptscheme_uri
FROM conceptscheme, conceptscheme_label
WHERE ? = conceptscheme_label.label_id AND conceptscheme.id = conceptscheme_label.conceptscheme_id
sqlalchemy.engine.base.Engine (3548,)


*** The import of the my_thesaurus/data/trees.json file with conceptscheme label 'Trees' to sqlite:///my_thesaurus.sqlite was successfull. ***

To use the data in Atramhasis, you must edit the file my_thesaurus/skos/__init__.py.
Add next lines:

def includeme(config):
        TREES = SQLAlchemyProvider(
                {'id': 'TREES', 'conceptscheme_id': 11},
                config.registry.dbmaker
        )
        skosregis = config.get_skos_registry()
        skosregis.register_provider(TREES)

Just follow these instructions and edit your my_thesaurus/skos/__init__.py like this:

# -*- coding: utf-8 -*-

import logging
log = logging.getLogger(__name__)

from skosprovider.registry import Registry
from skosprovider_sqlalchemy.providers import SQLAlchemyProvider


def create_registry(request):
    # create the SKOS registry
    registry = Registry(instance_scope='threaded_thread')a

    TREES = SQLAlchemyProvider(
            {'id': 'TREES', 'conceptscheme_id': 11},
            request.db
    )
    registry.register_provider(TREES)

    return registry

Now your thesaurus has been successfully imported and is ready to be browsed, expanded and edited.

SessionFactory

You can change the default session factory in the __init__.py file.

# set default session factory
from pyramid.session import SignedCookieSessionFactory
atramhasis_session_factory = SignedCookieSessionFactory(settings['atramhasis.session_factory.secret'])
config.set_session_factory(atramhasis_session_factory)

API Documentation

atramhasis.data

atramhasis.data.datamanagers

This module adds DataManagers for Atramhasis. These are service layer objects that abstract all interactions with the database away from the views.

versionadded:0.4.1

class atramhasis.data.datamanagers.AuditManager(session)

A data manager for logging the visit.

get_most_popular_concepts_for_conceptscheme(conceptscheme_id, max=5, period='last_month')

get the most popular concepts for a conceptscheme :param conceptscheme_id: id of the conceptscheme :param max: maximum number of results, default 5 :param period: ‘last_day’ or ‘last_week’ or ‘last_month’ or ‘last_year’, default ‘last_month’ :return: List of the most popular concepts of a conceptscheme over a certain period

save(visit_log)

save a certain visit :param visit_log: log of visit to save :return: The saved visit log

class atramhasis.data.datamanagers.ConceptSchemeManager(session)

A DataManager for ConceptSchemes <skosprovider_sqlalchemy.models.ConceptScheme>.

find(conceptscheme_id, query)

Find concepts and collections in this concept scheme.

Parameters:

• conceptscheme_id – a concepscheme id
• query – A python dictionary containing query parameters.

Returns:

A list of skosprovider_sqlalchemy.models.Thing instances.

get(conceptscheme_id)

Parameters:conceptscheme_id – a concepscheme id Returns:the concepscheme for the given id

get_all(conceptscheme_id)

Get all concepts and collections in this concept scheme.

Parameters:conceptscheme_id – a concepscheme id Returns:A list of skosprovider_sqlalchemy.models.Thing instances.

get_collections_for_scheme_tree(conceptscheme_id)

Parameters:conceptscheme_id – a concepscheme id Returns:all collections for the scheme_tree

get_concepts_for_scheme_tree(conceptscheme_id)

Parameters:conceptscheme_id – a concepscheme id Returns:all concepts for the scheme_tree

save(conceptscheme)

Parameters:conceptscheme – conceptscheme to save Returns:saved conceptscheme

class atramhasis.data.datamanagers.CountsManager(session)

A data manager that deals with triple counts.

save(counts)

Save a certain counts object

Parameters:counts (atramhasis.data.models.ConceptschemeCounts) – Counts object to save Returns:The saved count

class atramhasis.data.datamanagers.DataManager(session)

A DataManager abstracts all interactions with the database for a certain model.

class atramhasis.data.datamanagers.LanguagesManager(session)

A DataManager for Languages <skosprovider_sqlalchemy.models.Language>.

delete(language)

Parameters:language – the language to delete

get_all()

Returns:list of all languages

get_all_sorted(sort_coll, sort_desc)

Parameters:

• sort_coll – sort on this column
• sort_desc – descending or not

Returns:

sorted list of languages

save(language)

Parameters:language – language to save Returns:saved language

class atramhasis.data.datamanagers.SkosManager(session)

A DataManager for Concepts and Collections <skosprovider_sqlalchemy.models.Thing>.

delete_thing(thing)

Parameters:thing – the thing to delete

get_by_list_type(list_type)

Parameters:list_type – a specific list type Returns:all results for the specific list type

get_thing(concept_id, conceptscheme_id)

Parameters:

• concept_id – a concept id
• conceptscheme_id – a conceptscheme id

Returns:

the selected thing (Concept or Collection)

save(thing)

Parameters:thing – thing to save Returns:saved thing

atramhasis.data.db

Module that sets up the datamanagers and the database connections.

atramhasis.data.db.data_managers(request)

Generate a datamanager with a database session and register a cleanup handler.

Parameters:request (pyramid.request.Request) – The request this db session will be tied to. Returns:A dictionary containing different datamanagers.

atramhasis.data.db.includeme(config)

Set up SQLAlchemy.

Parameters:config (pyramid.config.Configurator) – Pyramid configuration.

atramhasis.errors

Module containing errors generated by Atramhasis.

exception atramhasis.errors.ConceptNotFoundException(c_id)

A Concept or Collection could not be found.

exception atramhasis.errors.ConceptSchemeNotFoundException(scheme_id)

A ConceptScheme could not be found.

exception atramhasis.errors.DbNotFoundException(value='No database found, please check your application setup')

Atramhasis could not find a database.

exception atramhasis.errors.LanguageNotFoundException(scheme_id)

A Language could not be found.

exception atramhasis.errors.SkosRegistryNotFoundException(value='No SKOS registry found, please check your application setup')

Atramhasis could not find a SKOS registry.

exception atramhasis.errors.ValidationError(value, errors)

Some data that was validated is invalid.

atramhasis.mappers

Module containing mapping functions used by Atramhasis.

atramhasis.mappers.is_html(value)

Check if a value has html inside. Only tags checked <strong> <em> <a>.

Parameters:value – a string Returns:a boolean (True, HTML present | False, no HTML present)

atramhasis.mappers.map_concept(concept, concept_json, skos_manager)

Map a concept from json to the database.

Parameters:

• concept (skosprovider_sqlalchemy.models.Thing) – A concept or collection as known to the database.
• concept_json (dict) – A dict representing the json sent to our REST service.
• skos_manager – A skos_manager to acces db operations

Returns:

The skosprovider_sqlalchemy.models.Thing enhanced with the information from the json object.

atramhasis.mappers.map_conceptscheme(conceptscheme, conceptscheme_json)

Map a conceptscheme from json to the database.

Parameters:

• conceptscheme (skosprovider_sqlalchemy.models.ConceptScheme) – A conceptscheme as known to the database.
• conceptscheme_json (dict) – A dict representing the json sent to our REST service.

Returns:

The skosprovider_sqlalchemy.models.ConceptScheme enhanced with the information from the json object.

atramhasis.protected_resources

class atramhasis.protected_resources.ProtectedResourceEvent(uri, request)

Event triggered when calling a protected operation on a resource

exception atramhasis.protected_resources.ProtectedResourceException(value, referenced_in)

raise this exception when the resource is still used somewhere

referenced_in should contain locations where the resource is still referenced

atramhasis.protected_resources.protected_operation(fn)

use this decorator to prevent an operation from being executed when the related resource is still in use

atramhasis.routes

Routes for the Atramhasis views.

New in version 0.4.4.

atramhasis.routes.includeme(config)

Setup the routing for Atramhasis.

Parameters:config (pyramid.config.Configurator) – The application config.

atramhasis.utils

Module containing utility functions used by Atramhasis.

atramhasis.utils.from_thing(thing)

Map a skosprovider_sqlalchemy.models.Thing to a skosprovider.skos.Concept or a skosprovider.skos.Collection, depending on the type.

Parameters:thing (skosprovider_sqlalchemy.models.Thing) – Thing to map. Return type:Concept or Collection.

atramhasis.utils.internal_providers_only(fn)

aspect oriented way to check if provider is internal when calling the decorated function

Parameters:fn – the decorated function Returns:around advice Raises:pyramid.httpexceptions.HTTPMethodNotAllowed – when provider is not internal

atramhasis.validators

Module that validates incoming JSON.

class atramhasis.validators.Concept(*arg, **kw)

class atramhasis.validators.ConceptScheme(*arg, **kw)

class atramhasis.validators.Concepts(*args, **kw)

class atramhasis.validators.Label(*arg, **kw)

class atramhasis.validators.Labels(*args, **kw)

class atramhasis.validators.LanguageTag(*arg, **kw)

class atramhasis.validators.MatchList(*args, **kw)

class atramhasis.validators.Matches(*arg, **kw)

class atramhasis.validators.Note(*arg, **kw)

class atramhasis.validators.Notes(*args, **kw)

class atramhasis.validators.RelatedConcept(*arg, **kw)

class atramhasis.validators.Source(*arg, **kw)

class atramhasis.validators.Sources(*args, **kw)

atramhasis.validators.broader_hierarchy_rule(errors, node_location, skos_manager, conceptscheme_id, cstruct)

Checks that the broader concepts of a concepts are not alreadt narrower concepts of that concept.

atramhasis.validators.collection_members_unique_rule(errors, node_location, members)

Checks that a collection has no duplicate members.

atramhasis.validators.collection_type_rule(errors, node_location, skos_manager, conceptscheme_id, members)

Checks that the targets of member_of are collections and not concepts.

atramhasis.validators.concept_matches_rule(errors, node_location, matches, concept_type)

Checks that only concepts have matches.

atramhasis.validators.concept_matches_unique_rule(errors, node_location, matches)

Checks that a concept has not duplicate matches.

This means that a concept can only have one match (no matter what the type) with another concept. We don’t allow eg. a concept that has both a broadMatch and a relatedMatch with the same concept.

atramhasis.validators.concept_relations_rule(errors, node_location, relations, concept_type)

Checks that only concepts have narrower, broader and related relations.

atramhasis.validators.concept_schema_validator(node, cstruct)

This validator validates an incoming concept or collection

This validator will run a list of rules against the concept or collection to see that there are no validation rules being broken.

Parameters:

• node (colander.SchemaNode) – The schema that’s being used while validating.
• cstruct – The concept or collection being validated.

atramhasis.validators.concept_type_rule(errors, node_location, skos_manager, conceptscheme_id, items)

Checks that the targets of narrower, broader and related are concepts and not collections.

atramhasis.validators.conceptscheme_schema_validator(node, cstruct)

This validator validates the incoming conceptscheme labels

Parameters:

• node (colander.SchemaNode) – The schema that’s being used while validating.
• cstruct – The conceptscheme being validated.

atramhasis.validators.hierarchy_rule(errors, node_location, skos_manager, conceptscheme_id, cstruct, property1, property2, property2_list_name, concept_type, error_message)

Checks that the property1 of a concept are not already in property2 hierarchy

atramhasis.validators.html_preparer(value)

Prepare the value by stripping all html except certain tags.

Parameters:value – The value to be cleaned. Return type:str

atramhasis.validators.label_lang_rule(errors, node, languages_manager, labels)

Checks that languages of a label are valid.

Checks that they are valid IANA language tags. If the language tag was not already present in the database, it adds them.

atramhasis.validators.label_type_rule(errors, node, skos_manager, labels)

Checks that a label has the correct type.

atramhasis.validators.languagetag_checkduplicate(node, language_tag, languages_manager, errors)

Check that a languagetag isn’t duplicated.

atramhasis.validators.languagetag_isvalid_rule(node, language_tag, errors)

Check that a languagetag is a valid IANA language tag.

atramhasis.validators.languagetag_validator(node, cstruct)

This validator validates a languagetag.

The validator will check if a tag is a valid IANA language tag. The the validator is informed that this should be a new language tag, it will also check if the tag doesn’t already exist.

Parameters:

• node (colander.SchemaNode) – The schema that’s being used while validating.
• cstruct – The value being validated.

atramhasis.validators.max_preflabels_rule(errors, node, labels)

Checks that there’s only one prefLabel for a certain language.

atramhasis.validators.members_hierarchy_rule(errors, node_location, skos_manager, conceptscheme_id, cstruct)

Checks that a collection does not have members that are in themselves already “parents” of that collection.

atramhasis.validators.members_only_in_collection_rule(errors, node, concept_type, members)

Checks that only collections have members.

atramhasis.validators.min_labels_rule(errors, node, cstruct)

Checks that a label or collection always has a least one label.

atramhasis.validators.narrower_hierarchy_rule(errors, node_location, skos_manager, conceptscheme_id, cstruct)

Checks that the narrower concepts of a concept are not already broader concepts of that concept.

atramhasis.validators.semantic_relations_rule(errors, node_location, skos_manager, conceptscheme_id, members, collection_id)

Checks that the elements in a group of concepts or collections are not the the group itself, that they actually exist and are within the same conceptscheme.

atramhasis.validators.subordinate_arrays_hierarchy_rule(errors, node_location, skos_manager, conceptscheme_id, cstruct)

Checks that the subordinate arrays of a concept are not themselves parents of that concept.

atramhasis.validators.subordinate_arrays_only_in_concept_rule(errors, node, concept_type, subordinate_arrays)

Checks that only a concept has subordinate arrays.

atramhasis.validators.subordinate_arrays_type_rule(errors, node_location, skos_manager, conceptscheme_id, subordinate_arrays)

Checks that subordinate arrays are always collections.

atramhasis.validators.superordinates_hierarchy_rule(errors, node_location, skos_manager, conceptscheme_id, cstruct)

Checks that the superordinate concepts of a collection are not themselves members of that collection.

atramhasis.validators.superordinates_only_in_concept_rule(errors, node, concept_type, superordinates)

Checks that only collections have superordinates.

atramhasis.validators.superordinates_type_rule(errors, node_location, skos_manager, conceptscheme_id, superordinates)

Checks that superordinates are always concepts.

atramhasis.views

atramhasis.views.views

class atramhasis.views.views.AtramhasisAdminView(request)

This object groups HTML views part of the admin user interface.

class atramhasis.views.views.AtramhasisListView(request)

This object groups list views part for the user interface.

class atramhasis.views.views.AtramhasisView(request)

This object groups HTML views part of the public user interface.

conceptschemes_view()

This view displays a list of available conceptschemes.

favicon_view()

This view returns the favicon when requested from the web root.

home_view()

This view displays the homepage.

search_result()

This view displays the search results

set_locale_cookie()

This view will set a language cookie

atramhasis.views.crud

Module containing views related to the REST service.

class atramhasis.views.crud.AtramhasisCrud(context, request)

This object groups CRUD REST views part of the private user interface.

edit_conceptscheme()

Edit an existing concept

Raises:atramhasis.errors.ValidationError – If the provided json can’t be validated

atramhasis.views.exception_views

Module containing error views.

atramhasis.views.exception_views.data_integrity(exc, request)

View invoked when IntegrityError was raised.

atramhasis.views.exception_views.failed(exc, request)

View invoked when bad data was submitted to Atramhasis.

atramhasis.views.exception_views.failed_not_found(exc, request)

View invoked when a resource could not be found.

atramhasis.views.exception_views.failed_not_method_not_allowed(exc, request)

View invoked when a method is not allowed.

atramhasis.views.exception_views.failed_skos(exc, request)

View invoked when Atramhasis can’t find a SKOS registry.

atramhasis.views.exception_views.failed_validation(exc, request)

View invoked when bad data was submitted to Atramhasis.

atramhasis.views.exception_views.protected(exc, request)

when a protected operation is called on a resource that is still referenced

atramhasis.views.exception_views.provider_unavailable(exc, request)

View invoked when ProviderUnavailableException was raised.

History

0.7.0 (06-11-2020)

This releases is a new major release with some new features and some backwards incompatible changes that require a careful upgrade and some manual intervention. The 0.7.x releases will also be that last to support Python 2. If you haven’t upgraded to Python 3 yet, we advise you to do now.

BC break The major change in this version is no longer initiating the skosprovider.registry.Registry on starting the application, but when a request is created. The previous way of working created problems with SQLAlchemy providers in a webserver using mutiple threads. Please review the docs at https://atramhasis.readthedocs.io/en/latest/customisation.html#creating-conceptschemes to see how it works now. For more background, have a look at the pyramid_skosprovider docs at https://pyramid-skosprovider.readthedocs.io/en/0.9.0/install.html

• All requirements were updated to their latest versions. Python versions were fixed to 2.7, 3.6, 3.7 and 3.8. If you made custom changes, you might have to edit them. (#508, #519, #513, #566)
• Npm has replaced bower as the package manager for frontend packages and the build process was revised. If you made custom frontend changes, please check them thoroughly.(#511)
• Instantiation of the SKOS registry was changed to work on a per request basis. (#346, #490, #535)
• Fixed a major issue with generating the expanded version of a concept. By default the assumption was that concepts in a collection were also narrower concepts of the collection’s superordinate concept, but the implementation for this was incomplete and contained bugs. This has been changed to an boolean attribute infer_concept_relations. When set to true, concepts in a collection are considerd to be narrower concepts of that collections’s superordinate concept. This is especially important for a provider’s expand function and affects what is considered a narrower concept of a concept that uses thesaurus arrays or node labels.
• The docs were updated and now contain a part detailing what Atramhasis does with some screenshots. (#495, #583, #440)
• Default inclusion of skosprovider_heritagedata was removed because the service is unstable too often. (#537)
• Improve some SEO by adding canonicul URL’s, open graph info, Twitter cards and the ability to generate a sitemap through a script. (#530, #531, #496, #497)
• Clean up importing and exporting of conceptschemes to make it easier. (#452, #475, #476, #495)
• Provide a simple, printable version of a thesaurus tree. (#533, #532)
• Add a script to make removing a conceptscheme easier. Be careful as this will drop all concepts, collections and the conceptscheme itself. (#579)
• Lots of minor improvements and bug fixes.

0.6.7 (21-06-2019)

• Fix corrupt build
• Security updates

0.6.6 (01-03-2019)

• Update Colander and other dependencies. (#464)
• Remove old convert_oe script because it depends on an obsolete webservice. (#466)
• Fix an issue with circular dependencies in requirements files. (#463)
• Change the default GA macro to anonymizeIp and be more GDPR compliant. (#450)

0.6.5 (19-12-2018)

• Generate a default dump location in development.ini files. (#416)
• Update skosprovider_sqlalchemy to solve a problem with the tree cache. (#455)
• Update a lot of dependencies.

0.6.4 (22-12-2017)

0.6.3 was a brown bag release. Sorry!

This version is exactly the same as 0.6.3, but properly packaged.

0.6.3 (21-12-2017)

• This version updates a lot of the requirements to their latest versions. This might be an issued if you’ve written lots of code against older pyramid versions. (#418, #413, #412, #411, #410, #408, #407, #404, #403)
• Remove authentication from the demo version since Persona does not work anymore. (#361)
• Fixed the base HTML template and added a DOCTYPE declaration. (#429)
• Update the URI’s for licenses of the Flemish Government. (#430)
• Fixed an issues with sorting on Python 3. (#424)

0.6.2 (11-10-2017)

• When an LDF server is present, add a link to the HTML document to this server. (#394)
• Wrong expansion of SKOS namespace in LDF server. (#401)

0.6.1 (01-09-2017)

This release is a minor release, containing improvements regarding the Linked Data Fragments server.

• Also add hidden datasets to the LDF server. Only external ones are not added now. (#390)
• Make it possible to set the LDF server protocol when generating the config. (#391)
• When generating an LDF server config, add a composite source as well. (#393)
• When generating the dataset information, add hydra controls that link to the LDF server instance. (#392)

0.6.0 (23-08-2017)

This release is a major release containing new features.

• Added a script to generate nightly dumps. Instead of generating full downloads on demand, they can now be generated by a cron job (eg. once per night, week, …). This makes it possible to download a large conceptscheme at once. During these dumps, some statistics on every conceptscheme such as the number of triples in it will be generated as well. This was done to make it easier to embed a custom Python based LDF server, but currently only serves the purpose of keeping some score. (#337, #360)
• Added easy integration with a Linked Data Fragments server (https://linkeddatafragments.org). Atramhasis can now generate a config file for such a server that you can use to setup the server. By default this config will work with the Turtle files that can be generated every night. But if you have access to the HDT library, you can also work with HDT files for a masssive performance boost. See the section Running a Linked Data Fragments server in the docs for more information. (#365)
• Add some more information the HTML title tags for a concept detail. (#363)
• Changed the UI for doing a search so that you now get a proper warning when searching for a label without specifying the conceptscheme to search in. (#373)
• It is now possible to generate URI’s when importing from a file that does not contain them, eg. a JSON or CSV file. The import_file can now take a pattern_uri parameter than will be used to generate new URI’s with. (#372)
• Fixed some issues with the tree cache that came to light when running Atramhasis as two nodes. Where before it was not possible to configure the tree cache, it now is. Previously an in-memory cache would always be used. Now it’s possible to use a different type of cache. If you’re running more than one webserver, it is advised to run a shared cache. If you’re running a previous version of Atramhasis, you will need to configure your cache with cache.tree and cache.list settings. (#371)
• It is now possible to add sortLabels to concepts. These can be used in the REST service to arbitrarily sort concepts. The sortLabel works per language. This makes it possible to eg. sort historical periods in chronological order. Most of the functionality was already present in skosprovider 0.6.0, but it had not been properly included in Atramhasis. (#362)
• Added ‘und - undetermined’ to the default language set to support json file imports. (#386)
• Fixed a bug when editing concepts where data from previously opened concepts would bleed into the concept you were editing. (#367)
• Update several dependencies to the latest versions. (#380, #381, #376)
• Added ‘und - undetermined’ to the default language set to support json file imports (#386)

0.5.2 (07-10-2016)

This minor release fixes a bug with the protected resource event. The event should give the uri of a concept instead of the url path. In addition to the uri the request is added to the event. It also fixes the bug with removing relations and updates the requirements for skosprovider_sqlalchemy.

0.5.1 (04-10-2016)

This minor release fixes a bug with the tree browser. Before it wasn’t possible to zoom and pan the tree. With certain larger trees this would cause issues as content would run off the page.

0.5.0 (14-09-2016)

This release is a major update based on the skosprovider 0.6.0 line of libraries. The most visible change is with the public and admin interfaces. These have been completely overhauled to provide a more pleasing user experience. Among other things visitors are now pointed towards popular concepts and concepts they have recently visited. Browsing an entire conceptscheme tree has been redesigned.

The adming interface now offers users an option to edit certain aspects of a conceptscheme such as the labels, notes and sources. Editing in general has been update and improved. Links between the public interface and the admin interface have been added to make switching from one to the other easier. Notes and sources can now contain certain HTML tags, allowing greater flexibility in defining concepts and collections.

A command line script was added to make it easy to import an entire conceptscheme, eg. when migrating from another system. It is now possible to import a RDF, CSV or JSON file on the command line in your Atramhasis instance. With earlier versions you had to script this yourself.

As always, bugs have been fixed, code has been rewritten and documenation has been updated.

See https://github.com/OnroerendErfgoed/atramhasis/milestone/8?closed=1 for the full list of changes.

0.4.4 (04-06-2015)

• Added more sample datasets to get a better view of real data. These will make the demo more interesting.
• Fix a bug where it was possible to create a relation between a concept and itself causing all sorts of nasty things to happen.
• Minor refactoring. Move the pyramid routes to a new file.
• Added a CONTRIBUTING.md file. Contributions welcome!

0.4.3 (11-03-2015)

We had some packaging issues with the 0.4.2 release.

0.4.2 (11-03-2015)

This release of Atramhasis is mostly a bugfix update of the 0.4.1 release.

• Fix paths of db in scaffolds
• Add more information on exceptions
• Update skosprovider_getty and skosprovider_heritagedata (fix the problems when importing external thesauri)
• Documentation update

0.4.1 (04-03-2015)

This release of Atramhasis is a minor update of the 0.4.0 release, focussing on small corrections and improvements and improving the documentation. A few interesting non-invasive features were added, mostly to the editor’s admin interface and machine-readable exports of RDF data.

Upgrading from 0.4.0 should be simple and cause no or few problems.

• A conceptscheme, concept or collection can now be exported to RDF through skosprovider_rdf 0.3.1. These are individuals export endpoints that can be reached in one of two ways. Either by hitting a url like http://localhost:6543/conceptschemes/GEOGRAPHY/c/335 with a supported RDF mimetype (application/rdf+xml, application/x-turtle, text-turle). Or by using an RDF syntax specific suffix (.rdf or .ttl).
• When importing, allow the user to request more information on a concept or collection, before actually importing it.
• Allow merging a concept with other concepts it matches. This allows a user to compare a local concept with an external one it matches and import any notes or labels that are present in the external concept, but not the local one.
• Reworked some parts of the public interface to make everything a bit clearer and to make all pages easily reachable.
• Allow sorting the languages in the admin interface.
• Reorganised and extended the right click menu on the grid in the admin interface.
• Allow looking up a skos:match from within the admin interface.
• Some issues with the length of language ids were solved.
• Fixed some issues when importing a collection instead of a concept.
• Made it easy to add a Google Analytics tracker.
• Added instructions on how to deploy a demo site on heroku. These work just as well for deploying an actual production site to heroku.
• Lots of small updates and tweaks to the documentation.
• Updated some dependencies.
• Some code cleanup and reorganisation. Several smaller bugs in the admin interface were fixed.
• The data fixtures were updated with skos:note examples. Added a license for reuse of the fixture data.

0.4.0 (23-12-2014)

• Update to skosprovider 0.5.0. Among other things, this makes it possible to handle relations between Concepts and Collections using the subordinate_arrays and superordinates properties. Conceptschemes are now also much better integrated within the providers, thus making it possible to provider more context for a Concept. This version of skosprovider can also handle skos:matches.
• Add possibility to edit language tags. It’s now possible to use the admin interface to add, edit and delete languages in Atramhasis.
• When the REST service receives labels or notes in currently unavailable languages, it will validate those through language_tags. It the languages are valid according to the IANA registry, they will be added to the languages available in the application.
• Default length of language id changed to 64 characters. This is not available as an alembic migration. So only effective when creating a new database. If you already have a database created from an older version of Atramhasis, please modify accordingly. Modifying column length on SQLite is not possible (see http://www.sqlite.org/omitted.html ).
• Abiltity to match Concepts in an Atramhasis ConceptScheme to Concepts in external ConceptSchemes through properties such as skos:exactMatch and skos:closeMatch.
• Ability to import Concepts and Collections from external providers. This makes it possible to import Concepts from eg. the AAT (via skosprovider_getty), Flanders Heritage Thesauri (via skosprovider_oe), English Heritage Thesauri (via skosprovider_heritagedata) or any other SKOS vocabulary for which a skosprovider has been written. Currently only the concept or collection itself can be imported, without its relations to other concepts or collections.
• Add the ability to have a delete of a concept or collection fail if it is being used in other systems.
• Implement a delete permission.
• Add validation rule that a Concept must have at least one label.
• Update to skosprovider_sqlalchemy 0.4.1.
• Update to pyramid_skosprovider 0.5.0.
• Update to skosprovider_rdf 0.3.0. This update adds support for dumping ConceptScheme in an RDF file and also handles subordinate_arrays and superordinates.
• Update to language_tags 0.3.0.

0.3.1 (05-09-2014)

• Update to skosprovider_sqlalchemy 0.2.1.
• Update to skosprovider_rdf 0.1.3 This fixes an issue with RDF having some SKOS elements in the wrong namespace. Also added a missing dependency on skosprovider_rdf to setup.py
• Updated the Travis build file to run a basic dojo build and test for build failures.

0.3.0 (15-08-2014)

• Atramhasis now includes a working admin userinterface at /admin. Still needs some polish when it comes to error handling and reporting about validation errors.
• The admin module gets run through a dojo build to minimize page loads and download times
• Added RDF/XML en RDF/Turtle downloads to the public interface. Currently only dumps a full conceptscheme, not individual concepts.
• Added more docs.

0.2.0 (16-05-2014)

• Full public userinterface
• REST CRUD service
• Security integration
• CSV export
• demo using Mozilla Persona as sample security setup

0.1.0 (22-04-2014)

• Initial version
• Setup of the project: docs, unit testing, code coverage
• Scaffolding for demo and deployment packages
• Limited public user interface
• Basis i18n abilities present
• Integration of pyramid_skosprovider
• Integration of skosprovider
• Integration of skosprovider_sqlalchemy

Glossary

CSS

Cascading Style Sheet is a style specification used to add style and presentation to webpages.

HDT

HDT (Header, Dictionary, Triples) is a compact data structure and binary serialization format for RDF that keeps big datasets compressed to save space while maintaining search and browse operations without prior decompression. This makes it an ideal format for storing and sharing RDF datasets on the Web.

HTML

HyperText Markup Language is the markup language used to create webpage.

Jinja2

Jinja2 is a python templating engine. It’s used by Atramhasis for rendering HTML templates.

Pyramid

This webframework was used to implement the server side components of Atramhasis.

RDF

Resource Description Framework. A very flexible model for data definition organised around triples. These triples forms a directed, labeled graph, where the edges represent the named link between two resources, represented by the graph nodes.

REST

REST or REpresentational State Transfer is a way of data exchange that is very complimentary to the operations of the HTTP protcol.

SKOS

Simple Knowledge Organization System. An general specification for Knowledge Organisation Systems (thesauri, word lists, authority files, …) that is commonly serialised as RDF.

SKOS-THES

The ISO 25964 SKOS extension defines mappings between the ISO 25964 standard and the SKOS specification.

SOA

Service Oriented Architecture.

URI

A Uniform Resource Identifier.

URN

A URN is a specific form of a URI.

Indices and tables

• Index
• Module Index
• Search Page