Atramhasis¶
Introduction¶
Atramhasis is an online SKOS editor. It allows a user to create and edit an online thesaurus or vocabulary adhering to the SKOS specification through a simple web interface. This allows any user with access to a web browser to consult the thesauri and if so wanted, to edit them.
Atramhasis is also intended to be one of the focal points in a Service Oriented Architecture. It exposes as much of its functionalities as possible through REST services. Both reading from and writing to concept schemes is possible with Atramhasis.
Atramhasis tries to stick as closely as possible to the SKOS specification. Where this was not possible, we tried to follow other standards such as SKOS-THES.
Atramhasis is being developed by the Flanders Heritage Agency, an agency of the Flemish Government that deals with Archaeology, Monuments and Landscapes. As such, we mainly intend to use it with vocabularies and thesauri that are related to cultural heritage. We generally construct our own thesauri, specific to our own applications, but while always keeping an eye on other thesauri in the larger field of cultural heritage such as the Art and Architecture Thesaurus (AAT).
If you have questions about the project, want to help out, want to report a bug or just want to have a conversation with us, please get in touch. For general conversations, you can use our Google Group. If you have encountered a bug in Atramhasis or it’s documentation, or if you want to ask us to consider implementing a feature, feel free to use our issue tracker.
Demo¶
Running a demo site¶
Atramhasis comes with a demo site include. This allows you to quickly evaluate and inspect the software. To get started, just download Atramhasis from pypi and install it. We recommend doing this in a virtualenvironment.
$ mkvirtualenv atramhasis_demo
$ pip install -U atramhasis
Once Atramhasis is installed, you can call upon a pyramid scaffold to generate the demo site.
$ pcreate -s atramhasis_demo atramhasis_demo
$ cd atramhasis_demo
This creates a local demo package you can run with just a few more commands:
# setup
$ pip install -r requirements-dev.txt
$ python setup.py develop
# download and install client side libraries
$ cd atramhasis_demo/static
$ bower install
$ cd admin
$ bower install
$ cd ../../..
# create or upgrade database
$ alembic upgrade head
# intialize sample data
$ initialize_atramhasis_db development.ini
# compile translations
$ python setup.py compile_catalog
# generate full RDF dumps (not necessary for basic functionality)
$ dump_rdf development.ini
# start server
$ pserve development.ini
The Atramhasis demo instance is now running on your localhost at port 6543. To reach it, open your browser and surf to the address http://localhost:6543.
You will be greeted by the Atramhasis front page. From this page you can start searching and browsing the thesauri. You can also start editing the thesauri by surfing to http://localhost:6543/admin. The demo instance does not requires you to login to access the admin module. If you want to run Atramhasis in a production environment, you can easily write your own security module. This enables you to use the security mechanisms (eg. LDAP, Active Directory, a custom users database, …) that your organisation requires. Please consult the documentation on Security customisation for further information on this topic.
Running a demo site with Docker¶
There is a Docker image available that allows you to quickly get a demo instance up and running. The Docker image contains the demo application and the LDF server.
After installing Docker for your operating system, you can simply pull the image and run a container. Once you’ve executed the following commands, you should be able to visit the demo application in your browser on http://localhost:6543. A LDF-server is also included in the demo, which is accessible on http://localhost:3000.
$ sudo docker pull atramhasis/demo
$ sudo docker run -p 6543:6543 -p 3000:3000 atramhasis/demo
Alternatively, you can run a specific version of Atramhasis (starting from atramhasis 0.6.4):
$ sudo docker pull atramhasis/demo:0.6.4
$ sudo docker run -p 6543:6543 -p 3000:3000 atramhasis/demo:0.6.4
While this is a fast and easy way to get a first impression of Atramhasis, please be aware that any edits you make when running the image, will be discarded when you stop the Docker container. If you want to test the application over a longer period of time, this is probably not what you’re looking for. If you need persistence, but still want to use Docker, you can customise the Dockerfile to suit your needs.
Running a demo site on Heroku¶
This section will tell you how to deploy an Atramhasis demo (or your own implementation) in the cloud. We’ll use Heroku, since this provider allows for a free Python instance (dyno) with a limited Postgresql database.
Create an account on Heroku and make sure you have Heroku Toolbelt installed. Prepare your local Heroku setup
Note
More information on running Python apps on Heroku can be found on the Heroku dev center.
Atramhasis scaffold¶
Create an Atramhasis scaffold (if you want to deploy an existing scaffold, skip this step)
$ mkvirtualenv atramhasis_heroku
$ pip install -U atramhasis
$ pcreate -s atramhasis_demo atramhasis_heroku
$ cd atramhasis_heroku
Git repository¶
Make sure your atramhasis_heroku folder is a git repository.
$ git init
$ git add .
$ git commit -m "initial commit"
requirements.txt¶
Update the requirements.txt file, make sure it contains a reference to atramhasis and to waitress.
Note
waitress has to be in the requirements.txt file for our Heroku deployment, requirements-dev.txt will be ignored.
run file¶
Create run
with the following content.
#!/bin/bash
set -e
python setup.py develop
python runapp.py
Note
Make sure to chmod +x run
before continuing. The develop
step is
necessary because the current package must be installed before Paste can
load it from the INI file.
runapp.py¶
Create a runapp.py
file.
import os
from paste.deploy import loadapp
from waitress import serve
if __name__ == "__main__":
port = int(os.environ.get("PORT", 5000))
app = loadapp('config:production.ini', relative_to='.')
serve(app, host='0.0.0.0', port=port)
Note
After creating the necessary files, commit them in your local git repository
Initialize the Heroku stack¶
$ heroku create
Postgresql¶
Attach an Heroku Postgres add-on to your application
$ heroku addons:add heroku-postgresql:hobby-dev
It can take a couple of minutes before your db is ready. You can wait for it to be ready using this command.
$ heroku pg:wait
When ready, check the connection url and copy paste it into your production.ini file
$ heroku config | grep HEROKU_POSTGRESQL
Also change the alembic.ini file to check your production.ini file instead of development.ini
ini_location = %(here)s/production.ini
Make sure to commit everything and push it to Heroku
$ git commit -a
$ git push heroku master
Note
More info on provisioning a database
Preparing the app¶
Open a remote console on your app
$ heroku run bash
This will start a console inside your remote Python virtualenv, so you can use all your libraries.
Run the commands to prepare your application
$ python setup.py develop
$ alembic upgrade head
$ initialize_atramhasis_db production.ini
$ python setup.py compile_catalog
$ dump_rdf production.ini
Note
Close the remote console!
Run the app¶
Run your app by starting one worker
$ heroku scale web=1
Check to see if your app is running.
$ heroku ps
Take a look at the logs to debug any errors if necessary.
$ heroku logs -t
Your app should now be available on the application url.
Development¶
Technology¶
Atramhasis is a python webapplication that is being developed within the pyramid framework. Other major technologies used are sqlalchemy as the ORM and Jinja2 as the templating framework.
Client side the main technologies being used are Zurb Foundation and Dojo toolkit.
While Atramhasis is an editor for creating and editing SKOS vocabularies, it uses other libraries that are more geared towards using a vocabulary in an application.
- skosprovider: This library defines a
VocabularyProvider
. This is an abstraction of usefull functionalities an application integrating SKOS needs. Different libraries can implement this interface for different datasources. This allows decoupling the interface from the concrete implementation. Out of the box this comes with a simpleDictionaryProvider
that serves a vocabulary based on a simple pythondict
as datasource. - skosprovider_sqlalchemy: An implementation of the
VocabularyProvider
interface with a SQLAlchemy backend. This allows using a RDBMS for reading, but also writing, SKOS concepts. - skosprovider_rdf: An implemenation of the
VocabularyProvider
interface with an RDF backend. Atramhasis uses this for exporting ConceptSchemes to RDF. It can also be used to get an existing SKOS vocabulary defined in RDF into Atramhasis. - pyramid_skosprovider: A library that integrates pyramid and skosprovider.
This libraries creates a
skosprovider.registry.Registry
and makes it accessible through thepyramid.request.Request
. Is also exposes a set of readonly REST services on the registered providers. - skosprovider_getty:
An implemenation of the
VocabularyProvider
against the Linked Open Data vocabularies published by the Getty Research Institute at http://vocab.getty.edu such as the Art and Architecture Thesaurus (AAT) and the Thesaurus of Geographic Names (TGN). - skosprovider_heritagedata:
An implementation of the
VocabularyProvider
against the vocabularies published by EH, RCAHMS and RCAHMW at heritagedata.org.
Atramhasis can easily be set up with a Linked Data Fragments (LDF) server. This server allows posing simple triple pattern queries of your dataset. Combined with a Linked Data Fragments client, similar functionalities to a traditional SPARQL endpoint can be achieved. Atramhasis facilitates the setup of a Linked Data Fragments server by generating a suitable config file for the Javascript server. This server can use different backends. Out of the box, Atramhasis generates Turtle files that can be used by the LDF server. It’s also possible to configre Atramhasis with a link to the rdf2hdt command (requires a separate installation). In this case, everytime the conceptschemes are dumped to RDF, the dump files are also written in HDT format.
General installation¶
We recommend installing Atramhasis in a virtual environment.
$ mkvirtualenv atramhasis_dev
To install a fully working development environment a pip requirements-dev.txt file is provided. By passing this file to pip install -r all requirements for Atramhasis and development of the software (Sphinx, py.test, tox) will be installed.
The following step will help you get the python development environment up and running. If also need to work on the javascript admin backend, please refer to the admin module documentation.
# Install dependencies
$ pip install -r requirements-dev.txt
# create or update database
$ alembic upgrade head
# insert sample data
$ initialize_atramhasis_db development.ini
# generate first RDF download
$ dump_rdf development.ini
# compile the Message Catalog Files
$ python setup.py compile_catalog
Once you’ve executed these steps, you can run a development server. This uses the standard pyramid server (Waitress) and should not be used as-is in a production environment.
# run a local development server
$ pserve --reload development.ini
Admin development¶
To work on the admin part, you’ll need npm and bower installed. Consult your operating system documentation on how to install these. The following instructions will assume you’re running a recent Debian based Linux distribution.
# install npm, bower and grunt-cli
$ sudo apt-get install nodejs
$ sudo apt-get install npm
$ sudo npm install -g bower grunt-cli
# install js dependencies using bower
$ cd atramhasis/static/admin
$ bower install
# install dojo build tools
$ npm install
These commands will install a couple of js libraries that Atramhasis uses in
/atramhasis/static/admin/src
and a set of tools to be able to generate
js builds. Builds are carried out through a simple grunt file:
# Build a dojo distribution
$ cd atramhasis/static/admin
$ grunt -v build
This will create a build a place the resulting files in
atramhasis/static/admin/dist
. The web application can be told to use
this build by setting dojo.mode in development.ini
to dist.
Frontend development¶
When updating the frontend templates, you might want to add extra translations. This can be done by placing {% trans %} tags in the templates
<h2>{% trans %}welcome_to{% endtrans %}</h2>
To update the message catalogs, do as follows:
$ python setup.py extract_messages
$ python setup.py update_catalog -l fr -i atramhasis/locale/atramhasis.pot -o atramhasis/locale/fr/LC_MESSAGES/atramhasis.po
$ python setup.py update_catalog -l nl -i atramhasis/locale/atramhasis.pot -o atramhasis/locale/nl/LC_MESSAGES/atramhasis.po
$ python setup.py update_catalog -l en -i atramhasis/locale/atramhasis.pot -o atramhasis/locale/en/LC_MESSAGES/atramhasis.po
Update the catalogs accordingly and run:
$ python setup.py compile_catalog
You might also want to add a new translation. Suppose you want to add a German translation.
$ python setup.py init_catalog -l de -i atramhasis/locale/atramhasis.pot -o atramhasis/locale/de/LC_MESSAGES/atramhasis.po
Edit atramhasis/locale/do/LC_MESSAGES/atramhasis.po
and add the necessary
translations. Just as with updating the catalogs, you need to recompile them.
$ python setup.py compile_catalog
At this moment, Atramhasis will still only show the default languages in it’s
language switcher. If you want to add you new language, you need to edit your
development.ini
(or similar file). Look for the line that says
available_languages and add your locale identifier.
available_languages = en nl fr de
After restarting your server you will now have the option of switching to German.
Running a Linked Data Fragments server¶
If you want to add a Linked Data Fragments server, Atramhasis makes it easy for you. First you need to decide if you want to run the server with HDT files. If not, you can always use raw Turtle files, but be aware that the HDT files offer much better performance.
If you want to use HDT files, please install hdt-cpp. Be aware that
you might have to download the source files and compile them yourself. Once
you have done so, add the rdf2hdt command to your development.ini file.
Supposing you installed it in /opt/hdt-cpp/hdt-lib/tools/rdf2hdt
:
# Location of rdf2hdt executable
atramhasis.rdf2hdt = /opt/hdt-cpp/hdt-lib/tools/rdf2hdt
Now, whenever Atramhasis creates rdf dumps it will also create HDT files. If you do not have rdf2hdt installed, you will still have Turtle datadumps that can be used by the LDF-server.
$ dump_rdf development.ini
Now you’re ready to generate the configuration for the LDF server. Out of the
box this file will be generated in the same directory your
development.ini
is located in, but you can override this in your ini
file by setting atramhasis.ldf.config_location or you can pass this on the
command line
# Generate config
$ generate_ldf_config development.ini
# Generate config and override config_location
$ generate_ldf_config development.ini -l /opt/my/ldf/server
Now you’re ready to run your LDF server. First we need to install it. It requires Node.js 4.0 or higher and should run on OSX and Linux. Please refer to the LDF server documentation for troubleshooting.
# Install ldf-server
$ [sudo] npm install -g ldf-server
# Run ldf-server
$ ldf-server ldf_server_config.json
Now you have an LDF server running at http://localhost:3000. Browse there and have fun!
When deploying Atramhasis with an LDF server in production, we recommend runnig both behind eg. nginx. In case you want to do this, you might run Atramhasis on port 6543 and LDF server on port 3000, but serve both through nginx. You can easily do this by setting the atramhasis.ldf.baseurl in your ini file. Suppose you want to server both on the host demo.atramhasis.org with Atramhasis as the root of your domain and the LDF server at /ldf. In this case, set atramhasis.ldf.baseurl to http://demo.atramhasis.org/ldf.
Contributing¶
Atramhasis is being developed as open source software by the Flanders Heritage Agency. All development is done on the agency’s Github page for Atramhasis.
Since we place a lot of importance of code quality, we expect to have a good amount of code coverage present and run frequent unit tests. All commits and pull requests will be tested with Travis-ci. Code coverage is being monitored with Coveralls.
Locally you can run unit tests by using pytest or tox. Running pytest manually is good for running a distinct set of unit tests. For a full test run, tox is preferred since this can run the unit tests against multiple versions of python.
# Run unit tests for all environments
$ tox
# No coverage
$ py.test
# Coverage
$ py.test --cov atramhasis --cov-report term-missing
# Only run a subset of the tests
$ py.test atramhasis/tests/test_views.py
Every pull request will be run through Travis-ci. When providing a pull request, please run the unit tests first and make sure they all pass. Please provide new unit tests to maintain 100% coverage. If you send us a pull request and this build doesn’t function, please correct the issue at hand or let us know why it’s not working.
Distribution¶
For building a distribution use the prepare command before the distribution command. This will update the requirement files in the scaffolds.
$ python setup.py prepare sdist
Services¶
Atramhasis can be used fully with a SOA. While we provide a public and and an administrator’s interface out of the box, you can also write your own client side code that interacts with the Atramhasis services, either for reading information or writing it.
Pyramid_skosprovider¶
This library takes your skosproviders and makes them available as REST services. The pyramid_skosprovider serves JSON as a REST service so it can be used easily inside a AJAX webbrowser call or by an external program.
The following API can be used by clients:
-
GET
/uris
¶ Find more information on a certain URI. This can map to eiter a concept, collection or conceptscheme that is known by the current SKOS registry.
Example request:
GET /uris?uri=urn:x-skosprovider:trees HTTP/1.1 Host: localhost:6543 Accept: application/json
Example response:
HTTP/1.1 200 OK Content-Type: application/json; charset=UTF-8 { "id": "TREES", "uri": "urn:x-skosprovider:trees", "type": "concept_scheme" }
Example request:
GET /uris/?uri=http://python.com/trees/larch HTTP/1.1 Host: localhost:6543 Accept: application/json
Example response:
HTTP/1.1 200 OK Content-Type: application/json; charset=UTF-8 { "id": "1", "uri": "http://python.com/trees/larch", "type": "concept", "concept_scheme": { "id": "TREES", "uri": "urn:x-skosprovider:trees" } }
Query Parameters: - uri – The URI to search for.
Status Codes: - 200 OK – The URI maps to something known by pyramid_skosprovider, either a conceptscheme, a concept or collection.
- 404 Not Found – The URI can’t be found by pyramid_skosprovider.
-
GET
/c
¶ Search for concepts or collections, no matter what scheme they’re a part of.
Although it is possible to search a single conceptscheme with just this endpoint, for performance reasons it is advised to use
GET /conceptschemes/{scheme_id}/c
.Example request:
GET /c HTTP/1.1 Host: localhost:6543 Accept: application/json
Example response:
HTTP/1.1 200 OK Content-Range: items 0-2/232 Content-Type: application/json; charset=UTF-8 [ { "id": "1", "uri": "urn:x-skosprovider:TREES:1", "type": "concept", "label": "De Lariks" }, { "id": "2", "uri": "urn:x-skosprovider:TREES:2", "type": "concept", "label": "De Paardekastanje" }, { "id": 3, "uri": "urn:x-skosprovider:TREES:3", "type": "collection", "label": "Bomen per soort" } ]
Example request:
GET /c?type=concept&providers.subject=external&sort=uri HTTP/1.1 Host: localhost:6543 Accept: application/json
Query Parameters: - type – Define if you want to show concepts or collections. Leave blank to show both.
- mode – Allows for special processing mode for dijitFilteringSelect. Makes it possible to use wildcards in the label parameter.
- label – Shows all concepts and collections that have this search string in one of their labels.
- language – Returns the label with the corresponding language-tag if present.
If the language is not present for this concept/collection, it falls back to
1) the default language of the provider. 2) ‘en’ 3) any label.
Eg.
?language=nl
to show the dutch labels of the concepts/collections. - sort – Define if you want to sort the results by a given field. Otherwise items are returned
in an indeterminate order. Prefix with ‘+’ to sort ascending, ‘-‘ to sort descending.
eg.
?sort=-label
to sort all results descending by label. - providers.ids – A comma separated list of concept scheme id’s. The query
will only be passed to the providers with these id’s. eg.
?providers.ids=TREES, PARROTS
will only list concepts from these two providers. - providers.subject – A subject can be registered with a skosprovider in
the registry. Adding this search parameter means that the query will only
be passed on to providers that have been tagged with this subject. Eg.
?providers.subject=external
to only query the providers that have been marked with the subject external.
Request Headers: - Range – Can be used to request a certain set of results.
eg.
items=0-24
requests the first 25 results.
Response Headers: - Content-Range – Tells the client what set of results is being returned
eg.
items=0-24/306
means the first 25 out of 306 results are being returned.
Status Codes: - 200 OK – The concepts in this conceptscheme were found.
-
GET
/conceptschemes
¶ Get all registered conceptschemes.
Example request:
GET /conceptschemes HTTP/1.1 Host: localhost:6543 Accept: application/json
Example response:
HTTP/1.1 200 OK Content-Type: application/json; charset=UTF-8 Date: Mon, 14 Apr 2014 14:42:34 GMT [ { "id": "TREES", "uri": "urn:x-skosprovider:trees", "label": "Different types of trees." } ]
Status Codes: - 200 OK – The list of conceptschemes was found.
-
GET
/conceptschemes/{scheme_id}
¶ Get information about a concept scheme.
Example request:
GET /conceptschemes/TREES HTTP/1.1 Host: localhost:6543 Accept: application/json
Example response:
HTTP/1.1 200 OK Content-Length: 15 Content-Type: application/json; charset=UTF-8 Date: Mon, 14 Apr 2014 14:45:37 GMT Server: waitress { "id": "TREES", "uri": "urn:x-skosprovider:trees", "label": "Different types of trees.", "labels": [ {"type": "prefLabel", "language": "en", "label": "Different types of trees."}, {"type": "prefLabel", "language": "nl", "label": "Verschillende soorten bomen."} ] }
Example request:
-.. sourcecode:: http
GET /conceptschemes/PLANTS HTTP/1.1 Host: localhost:6543 Accept: application/jsonExample response:
HTTP/1.1 404 Not Found Content-Length: 775 Content-Type: text/html; charset=UTF-8 Date: Tue, 15 Apr 2014 20:32:52 GMT Server: waitress
Status Codes: - 200 OK – The conceptscheme was found.
- 404 Not Found – The conceptscheme was not found.
-
GET
/conceptschemes/{scheme_id}/topconcepts
¶ Get all top concepts in a certain conceptscheme. These are all the concepts in the conceptscheme that have no broader concept.
Example request:
GET /conceptschemes/TREES/topconcepts HTTP/1.1 Host: localhost:6543 Accept: application/json
Example response:
HTTP/1.1 200 OK Content-Type: application/json; charset=UTF-8 Date: Mon, 14 Apr 2014 14:47:33 GMT Server: waitress [ { "id": "1", "uri": "urn:x-skosprovider:TREES:1", "type": "concept", "label": "De Lariks" }, { "id": "2", "uri": "urn:x-skosprovider:TREES:2", "type": "concept", "label": "De Paardekastanje" } ]
Query Parameters: - language – Returns the label with the corresponding language-tag if present.
If the language is not present for this concept/collection, it falls back to
1) the default language of the provider. 2) ‘en’ 3) any label.
Eg.
?language=nl
to show the dutch labels of the concepts/collections.
Status Codes: - 200 OK – The topconcepts in this conceptscheme were found.
- 404 Not Found – The conceptscheme was not found.
- language – Returns the label with the corresponding language-tag if present.
If the language is not present for this concept/collection, it falls back to
1) the default language of the provider. 2) ‘en’ 3) any label.
Eg.
-
GET
/conceptschemes/{scheme_id}/displaytop
¶ Get the top of a display hierarchy. Depending on the underlying provider this will be a list of Concepts and Collections.
Example request:
GET /conceptschemes/TREES/displaytop HTTP/1.1 Host: localhost:6543 Accept: application/json
Example response:
HTTP/1.1 200 OK Content-Type: application/json; charset=UTF-8 Date: Mon, 14 Apr 2014 14:47:33 GMT Server: waitress [ { "id": "1", "uri": "urn:x-skosprovider:TREES:1", "type": "concept", "label": "De Lariks" }, { "id": "2", "uri": "urn:x-skosprovider:TREES:2", "type": "concept", "label": "De Paardekastanje" } ]
Query Parameters: - language – Returns the label with the corresponding language-tag if present.
If the language is not present for this concept/collection, it falls back to
1) the default language of the provider. 2) ‘en’ 3) any label.
Eg.
?language=nl
to show the dutch labels of the concepts/collections.
Status Codes: - 200 OK – The concepts and collections were found.
- 404 Not Found – The conceptscheme was not found.
- language – Returns the label with the corresponding language-tag if present.
If the language is not present for this concept/collection, it falls back to
1) the default language of the provider. 2) ‘en’ 3) any label.
Eg.
-
GET
/conceptschemes/{scheme_id}/c
¶ Search for concepts or collections in a scheme.
Example request:
GET /conceptschemes/TREES/c HTTP/1.1 Host: localhost:6543 Accept: application/json
Example response:
HTTP/1.1 200 OK Content-Length: 117 Content-Range: items 0-2/3 Content-Type: application/json; charset=UTF-8 Date: Mon, 14 Apr 2014 14:47:33 GMT Server: waitress [ { "id": "1", "uri": "urn:x-skosprovider:TREES:1", "type": "concept", "label": "De Lariks" }, { "id": "2", "uri": "urn:x-skosprovider:TREES:2", "type": "concept", "label": "De Paardekastanje" }, { "id": 3, "uri": "urn:x-skosprovider:TREES:3", "type": "collection", "label": "Bomen per soort" } ]
Example request:
GET /conceptschemes/PLANTS/c HTTP/1.1 Host: localhost:6543 Accept: application/json
Example response:
HTTP/1.1 404 Not Found Content-Length: 775 Content-Type: text/html; charset=UTF-8 Date: Tue, 15 Apr 2014 20:32:52 GMT Server: waitress
Query Parameters: - type – Define if you want to show concepts or collections. Leave blank to show both.
- mode – Allows for special processing mode for dijitFilteringSelect. Makes it possible to use wildcards in the label parameter.
- label – Shows all concepts and collections that have this search string in one of their labels.
- collection – Get information about the content of a collection. Expects to be passed an id of a collection in this scheme. Will restrict the search to concepts or collections that are a member of this collection or a narrower concept of a member.
- language – Returns the label with the corresponding language-tag if present.
If the language is not present for this concept/collection, it falls back to
1) the default language of the provider. 2) ‘en’ 3) any label.
Eg.
?language=nl
to show the dutch labels of the concepts/collections. - sort – Define if you want to sort the results by a given field. Otherwise items are returned
in an indeterminate order. Prefix with ‘+’ to sort ascending, ‘-‘ to sort descending.
eg.
?sort=-label
to sort all results descending by label.
Request Headers: - Range – Can be used to request a certain set of results.
eg.
items=0-24
requests the first 25 results.
Response Headers: - Content-Range – Tells the client was set of results is being returned
eg.
items=0-24/306
means the first 25 out of 306 results are being returned.
Status Codes: - 200 OK – The concepts in this conceptscheme were found.
- 404 Not Found – The conceptscheme was not found.
-
GET
/conceptschemes/{scheme_id}/c/{c_id}
¶ Get information about a concept or collection.
Example request:
GET /conceptschemes/TREES/c/1 HTTP/1.1 Host: localhost:6543 Accept: application/json
Example response:
HTTP/1.1 200 OK Content-Type: application/json; charset=UTF-8 Date: Mon, 14 Apr 2014 14:49:27 GMT Server: waitress { "broader": [], "narrower": [], "notes": [ {"note": "A type of tree.", "type": "definition", "language": "en"} ], "labels": [ {"type": "prefLabel", "language": "en", "label": "The Larch"}, {"type": "prefLabel", "language": "nl", "label": "De Lariks"} ], "type": "concept", "id": "1", "uri": "urn:x-skosprovider:TREES:1", "related": [], "label": "The Larch", "matches": { "close": [ "http://id.python.org/different/types/of/trees/nr/1/the/larch" ] }, "concept_scheme": { "uri": "urn:x-foo:bar" } }
Example request:
GET /conceptschemes/TREES/c/4 HTTP/1.1 Host: localhost:6543 Accept: application/json
Example response:
HTTP/1.1 404 Not Found Content-Length: 775 Content-Type: text/html; charset=UTF-8 Date: Tue, 15 Apr 2014 20:06:12 GMT Server: waitress
Status Codes: - 200 OK – The concept was found in the conceptscheme.
- 404 Not Found – The concept was not found in the conceptscheme or the conceptscheme was not found.
-
GET
/conceptschemes/{scheme_id}/c/{c_id}/displaychildren
¶ Get a list of Collections and Concepts that should be displayed as children of this Concept or Collection.
Example request:
GET /conceptschemes/TREES/c/3/displaychildren HTTP/1.1 Host: localhost:6543 Accept: application/json
Example response:
HTTP/1.1 200 OK Content-Type: application/json; charset=UTF-8 Date: Mon, 14 Apr 2014 14:49:27 GMT Server: waitress [ { "id": "1", "uri": "urn:x-skosprovider:TREES:1", "type": "concept", "label": "De Lariks" }, { "id": "2", "uri": "urn:x-skosprovider:TREES:2", "type": "concept", "label": "De Paardekastanje" } ]
Query Parameters: - language – Returns the label with the corresponding language-tag if present.
If the language is not present for this concept/collection, it falls back to
1) the default language of the provider. 2) ‘en’ 3) any label.
Eg.
?language=nl
to show the dutch labels of the concepts/collections.
Status Codes: - 200 OK – The concept was found in the conceptscheme.
- 404 Not Found – The concept was not found in the conceptscheme or the conceptscheme was not found.
- language – Returns the label with the corresponding language-tag if present.
If the language is not present for this concept/collection, it falls back to
1) the default language of the provider. 2) ‘en’ 3) any label.
Eg.
-
GET
/conceptschemes/{scheme_id}/c/{c_id}/expand
¶ Expand a concept or collection to all it’s narrower concepts.
This method should recurse and also return narrower concepts of narrower concepts.
If the id passed belongs to a
skosprovider.skos.Concept
, the id of the concept itself should be include in the return value.If the id passed belongs to a
skosprovider.skos.Collection
, the id of the collection itself must not be present in the return value In this case the return value includes all the member concepts and their narrower concepts.- Returns A list of id’s or
HTTPNotFound
if the concept or collection doesn’t - exist.
Example request:
GET /conceptschemes/TREES/c/3/expand HTTP/1.1 Host: localhost:6543 Accept: application/json
Example response:
HTTP/1.1 200 OK Content-Type: application/json; charset=UTF-8 Date: Mon, 14 Apr 2014 14:49:27 GMT Server: waitress [1 , 2]
Status Codes: - 200 OK – The concept/collection was found in the conceptscheme.
- 404 Not Found – The concept/collection was not found in the conceptscheme or the conceptscheme was not found.
- Returns A list of id’s or
Atramhasis¶
Concepts and collections¶
The main Atramhasis write services allow you to add concepts and collections, edit them and delete them.
-
POST
/conceptschemes/{scheme_id}/c
¶ Add a concept or collection to a conceptscheme. The response body will contain a representation of the concept or collection after is has been added to the conceptscheme.
Example request:
POST /conceptschemes/TREES/c HTTP/1.1 Host: demo.atramhasis.org Accept: application/json Content-Type: application/json { "type": "concept", "broader": [], "narrower": [], "related": [], "labels": [ { "type": "prefLabel", "language": "en", "label": "The Larch" } ], "notes": [] }
Example response:
HTTP/1.1 201 Created Location: http://demo.atramhasis.org/conceptschemes/TREES/c/1 Content-Type: application/json { "id": 1, "uri": "urn:x-atramhasis-demo:TREES:1", "type": "concept", "broader": [], "narrower": [], "related": [], "labels": [ { "type": "prefLabel", "language": "en", "label": "The Larch" } ], "notes": [] }
Example request:
POST /conceptschemes/TAUNTS/c HTTP/1.1 Host: demo.atramhasis.org Accept: application/json Content-Type: application/json { "type": "concept", "broader": [], "narrower": [], "related": [], "labels": [ { "type": "tauntLabel", "language": "en-FR", "label": "Your mother was a Hamster!" } ], "notes": [] }
Example response:
HTTP/1.1 400 Bad Request Location: http://demo.atramhasis.org/conceptschemes/TREES/c/1 Content-Type: application/json { "errors": [ {"labels": "Invalid labeltype."}, {"labels": "Invalid language."} ], "message": "Concept could not be validated" }
Parameters: - scheme_id – The identifier for a certain concept scheme.
Request Headers: - Accept – The response content type depends on this header. Currently only application/json is supported.
Response Headers: - Content-Type – This service currently always returns application/json
- Location – The url where the newly added concept or collection can be found.
Status Codes: - 201 Created – The concept or collection was added succesfully.
- 400 Bad Request – The concept or collection could not be added because the submitted json was invalid due to eg. validation errors.
- 404 Not Found – The conceptscheme scheme_id does not exist.
- 405 Method Not Allowed – The concept or collection could not be added because the conceptscheme scheme_id is a readonly conceptscheme.
-
PUT
/conceptschemes/{scheme_id}/c/{c_id}
¶ Edit the concept or collection with id c_id. The response body will contain a representation of the concept or collection after the modifications.
Example request:
PUT /conceptschemes/TREES/c/1 HTTP/1.1 Host: demo.atramhasis.org Accept: application/json Content-Type: application/json { "type": "concept", "broader": [], "narrower": [], "related": [], "labels": [ { "type": "prefLabel", "language": "en", "label": "The Larch" }, { "type": "prefLabel", "language": "nl", "label": "De Lariks" } ], "notes": [] }
Example response:
HTTP/1.1 200 OK Content-Type: application/json { "id": 1, "uri": "urn:x-atramhasis-demo:TREES:1", "type": "concept", "broader": [], "narrower": [], "related": [], "labels": [ { "type": "prefLabel", "language": "en", "label": "The Larch" }, { "type": "prefLabel", "language": "nl", "label": "De Lariks" } ], "notes": [] }
Parameters: - scheme_id – The identifier for a certain concept scheme.
- c_id – The identifier for a certain concept or collection.
Request Headers: - Accept – The response content type depends on this header. Currently only application/json is supported.
Response Headers: - Content-Type – This service currently always returns application/json
Status Codes: - 200 OK – The concept or collection was edited succesfully.
- 400 Bad Request – The concept or collection could not be edited because the submitted json was invalid due to eg. validation errors.
- 404 Not Found – The conceptscheme scheme_id or the concept or collection c_id does not exist.
- 405 Method Not Allowed – The concept or collection could not be edited because the conceptscheme scheme_id is a readonly conceptscheme.
-
DELETE
/conceptschemes/{scheme_id}/c/{c_id}
¶ Remove the concept with id c_id. The response body will contain the last representation known by the service.
Example request:
DELETE /conceptschemes/TREES/c/1 HTTP/1.1 Host: demo.atramhasis.org Accept: application/json
Example response:
HTTP/1.1 200 OK Content-Type: application/json { "id": 1, "uri": "urn:x-atramhasis-demo:TREES:1", "type": "concept", "broader": [], "narrower": [], "related": [], "labels": [ { "type": "prefLabel", "language": "en", "label": "The Larch" }, { "type": "prefLabel", "language": "nl", "label": "De Lariks" } ], "notes": [] }
Parameters: - scheme_id – The identifier for a certain concept scheme.
- c_id – The identifier for a certain concept or collection.
Request Headers: - Accept – The response content type depends on this header. Currently only application/json is supported.
Response Headers: - Content-Type – This service currently always returns application/json
Status Codes: - 200 OK – The concept or collection was deleted succesfully.
- 400 Bad Request – The concept or collection could not be edited because the submitted json was invalid due to eg. validation errors.
- 404 Not Found – The conceptscheme scheme_id or the concept or collection c_id does not exist.
- 405 Method Not Allowed – The concept or collection could not be deleted because the conceptscheme scheme_id is a readonly conceptscheme.
- 409 Conflict – The concept or collection could not be deleted because Atramhasis has determined that it’s still being used somewhere else. The response body will contain a message and a list of URI’s that are using this concept.
Languages¶
Apart from the main services, Atramhasis exposes some secondary services that deal with languages.
-
GET
/languages
¶ List all languages known to this Atramhasis instance.
Please bear in mind that these are not all known IANA language tags, but a subset used in this Atramhasis instance. This is used to populate drop down lists and such.
Example request:
GET /languages HTTP/1.1 Host: demo.atramhasis.org Accept: application/json
Example response:
HTTP/1.1 200 OK Content-Type: application/json [ {"id": "la", "name": "Latin"}, {"id": "nl", "name": "Dutch"}, {"id": "en", "name": "English"}, {"id": "fr", "name": "French"}, {"id": "de", "name": "German"} ]
Parameters: - sort – Which field to sort on. Use - and + to indicate sort order. Eg. id or +id sort ascending on id, -name sort descending on name.
Request Headers: - Accept – The response content type depends on this header. Currently only application/json is supported.
Response Headers: - Content-Type – This service currently always returns application/json
Status Codes: - 200 OK – The list of languages was returned.
-
GET
/languages/{language_id}
¶ Get information on a certain language.
Please bear in mind this will only work for languages known to this Atramhasis instance. Valid IANA languages not known to this instance will not work.
Example request:
GET /languages HTTP/1.1 Host: demo.atramhasis.org Accept: application/json
Example response:
HTTP/1.1 200 OK Content-Type: application/json { "id": "la", "name": "Latin" }
Request Headers: - Accept – The response content type depends on this header. Currently only application/json is supported.
Response Headers: - Content-Type – This service currently always returns application/json
Status Codes: - 200 OK – The language was found.
- 404 Not Found – The language was not found in this instance.
-
PUT
/languages/{language_id}
¶ Update the information on a certain language or create an entry for a new one.
The user is required to submit the language_id and this must be a valid IANA language tag.
Example request:
PUT /languages/nl-BE HTTP/1.1 Host: demo.atramhasis.org Accept: application/json Content-Type: application/json { "id": "nl-BE", "name": "Dutch (Flanders)" }
Example response:
HTTP/1.1 200 OK Content-Type: application/json { "id": "nl-BE", "name": "Dutch (Flanders)" }
Request Headers: - Accept – The response content type depends on this header. Currently only application/json is supported.
Response Headers: - Content-Type – This service currently always returns application/json
Status Codes: - 200 OK – The language was updated or added.
- 400 Bad Request – The request could not be executed because of problems with the submitted data. Most likely you are submitting an invalid IANA langage code.
-
DELETE
/languages/{language_id}
¶ Delete a language from this Atramhasis instance.
Example request:
DELETE /languages/nl-BE HTTP/1.1 Host: demo.atramhasis.org Accept: application/json
Example response:
HTTP/1.1 200 OK Content-Type: application/json { "id": "nl-BE", "name": "Dutch (Flanders)" }
Request Headers: - Accept – The response content type depends on this header. Currently only application/json is supported.
Response Headers: - Content-Type – This service currently always returns application/json
Status Codes: - 200 OK – The language was deleted.
- 404 Not Found – The language was not found in this instance.
Customisation¶
Out of the box Atramhasis tries to make as few assumptions as possible about setup. We have taken care to ensure that significant parts of the application are easy to customise and expect most installations to have custom code. We’ve shipped Atramhasis with sane defaults so you can get a quick feel for the capabilities of the software. However, we do not advise running a production instance with only these default settings.
Creating your own project¶
Whenever you want to run an instance of Atramhasis, you start by creating your own project. This is the place where you will maintain and develop your own custom templates, static assets such as stylesheets, your security implementation and other general configuration. To make it easier on you to get started, we provide a scaffold just for this. As always, we advise working in a virtual environment.
$ mkvirtualenv my_thesaurus
$ pip install atramhasis
$ pcreate -s atramhasis_scaffold my_thesaurus
$ cd my_thesaurus
# Install dependencies
$ pip install -r requirements-dev.txt
# compile the Message Catalog Files
$ python setup.py compile_catalog
# Download and install client side libraries
$ cd my_thesaurus/static
$ bower install
$ cd admin
$ bower install
This gives you a clean slate to start your customisations on. By default the
scaffold comes with a simple SQLite database. This is more than enough for
your first experiments and can even be used in production environment if your
needs are modest. You can always instruct Atramhasis to use
some other database engine, as long as SQLAlchemy supports it. Configure the
sqlalchemy.url configuration option in development.ini
to change
the database. See the documentation of SQLAlchemy for more information about
this connection url. After settings this url, run alembic to
initialise and migrate the database to the latest version.
# Create or update database based on
# the configuration in development.ini
$ alembic upgrade head
Your custom version of Atramhasis can now be run. Run the following command and point your browser to http://localhost:6543 to see the result.
$ pserve development.ini
Of course, this does not do very much since your Atramhasis is now running, but does not contain any ConceptSchemes. You will need to configure this by entering a database record for the ConceptScheme and writing a small piece of code.
To enter the database record, you need to enter a record in the table conceptscheme. In this table you need to register an id for the conceptscheme and a uri. The id is for internal database use and has no other meaning. The uri can be used externally. To register a new ConceptScheme in the sqlite database that was created:
$ sqlite3 my_thesaurus.sqlite
INSERT INTO conceptscheme VALUES (1, 'urn:x-my-thesaurus:stuff')
This take care of the first step. Now you also need to tell Atramhasis where
to find your conceptscheme and how to handle it. To do this, you need to edit
the file called my_thesaurus/skos/__init__.py
. In this file you need
to register SQLAlchemyProvider
instances. First you need to tell python where to find such a provider by adding
this code just below the logging configuration:
from skosprovider_sqlalchemy.providers import SQLAlchemyProvider
Then you need to instantiate such a provider within the includeme function in
this file. This provider needs a few arguments: an id for the provider, an id
for the conceptscheme it’s working with and a function that knows how the
provide a database session. The id for the provider is often a text string
and will appear in certain url’s and might popup in the user interface from
time to time. The database sessionmaker can be found at
config.registry.dbmaker. Finally, you need to register this provider with
the skosprovider.registry.Registry
.
STUFF = SQLAlchemyProvider(
{
'id': 'STUFF',
'conceptscheme_id': 1
},
config.registry.dbmaker
)
skosregis.register_provider(STUFF)
After having registered your provider, the file should look more or less like this:
# -*- coding: utf-8 -*-
import logging
log = logging.getLogger(__name__)
from skosprovider_sqlalchemy.providers import SQLAlchemyProvider
def includeme(config):
STUFF = SQLAlchemyProvider(
{
'id': 'STUFF',
'conceptscheme_id': 1
},
config.registry.dbmaker
)
skosregis = config.get_skos_registry()
skosregis.register_provider(STUFF)
Now you can restart your server and then you front page will show you a new, but empty thesaurus. You can now start creating concepts and collections by going to the admin interface at http://localhost:6543/admin.
You will notice that any concepts or collections you create wil get a
URI similar to urn:x-skosprovider:STUFF:1. This is due to the fact
that your SQLAlchemyProvider
has a UriGenerator
that generates uris for the
provider. By default, the provider configures a
DefaultUrnGenerator
, but it’s expected that you
will want to override this.
Warning
The UriGenerator
that you configure only generates
URI’s when creating new concepts or collections. When importing existing
vocabularies, please be sure to create the URI’s before or during import
(possbily by using a relevant generator yourself).
Suppose you have decided that your URI’s should look like this:
http://id.mydata.org/thesauri/stuff/[id]. You can do this by registering
a UriPatternGenerator
with your provider:
STUFF = SQLAlchemyProvider(
{
'id': 'STUFF',
'conceptscheme_id': 1
},
config.registry.dbmaker,
uri_generator=UriPatternGenerator(
'http://id.mydata.org/thesauri/stuff/%s'
)
)
Don’t forget to import the UriPatternGenerator
at the
top of your file:
from skosprovider.uri import UriPatternGenerator
Your final file should look similar to this:
# -*- coding: utf-8 -*-
import logging
log = logging.getLogger(__name__)
from skosprovider_sqlalchemy.providers import SQLAlchemyProvider
from skosprovider.uri import UriPatternGenerator
def includeme(config):
STUFF = SQLAlchemyProvider(
{
'id': 'STUFF',
'conceptscheme_id': 1
},
config.registry.dbmaker,
uri_generator=UriPatternGenerator(
'http://id.mydata.org/thesauri/stuff/%s'
)
)
skosregis = config.get_skos_registry()
skosregis.register_provider(STUFF)
If you need more complicated URI’s, you can easily write you own generator
with a small piece of python code. You just need to follow the interface
provided by skosprovider.uri.UriGenerator
.
Hiding a vocabulary¶
Atramhasis allows you to hide a vocabulary. This means the vocabulary is still there as far as services are concerned and you can still edit it. But it will not be visible in the public html user interface. You might want to use it for small and rather technical vocabularies you need but don’t want to draw attention to. The only thing you need to do, is tagging this provider with a subject. By adding the hidden subject to the provider, we let Atramhasis know that this vocabulary should not be present among your regular vocabularies.
Suppose we wanted to hide our stuff:
# -*- coding: utf-8 -*-
import logging
log = logging.getLogger(__name__)
from skosprovider_sqlalchemy.providers import SQLAlchemyProvider
from skosprovider.uri import UriPatternGenerator
def includeme(config):
STUFF = SQLAlchemyProvider(
{
'id': 'STUFF',
'conceptscheme_id': 1,
'subject': ['hidden']
},
config.registry.dbmaker,
uri_generator=UriPatternGenerator(
'http://id.mydata.org/thesauri/stuff/%s'
)
)
skosregis = config.get_skos_registry()
skosregis.register_provider(STUFF)
Now the STUFF thesaurus will not show up in the public web interface, but REST calls to this conceptscheme will function as normal and you will be able to maintain it from the admin interface.
Internationalisation¶
When you create a new empty project with the atramhasis scaffold, you get an
English only version. The standard version of Atramhasis has been
translated in Dutch and French. If you desire, you can activate these by editing
your project’s development.ini
# Edit and uncomment to activate nl and fr language support or other languages
# you have added yourself.
available_languages = en nl fr
Available languages should be a space separated list of IANA language codes. If you add new languages, please consider contributing them back to the project.
Appearance¶
By implementing a few simple techniques from the Pyramid web framework, it’s very easy to customise the look and feel of the public user interface. The default implementation is a very neutral implementation based on the basic elements in the Foundation framework. Customising and overriding this style is possible if you have a bit of knowledge about HTML and CSS.
You can also override the HTML templates that Atramhasis uses without needing to alter the originals so that future updates to the system will not override your modifications.
Overriding templates¶
One very easy technique to use, is Pyramid’s
override assets mechanism.
This allows you to override a core Atramhasis template with your own template.
Suppose we want to change the text on the Atramhasis homepage to welcome visitors
to your instances. This text can be found in atramhasis/templates/welcome.jinja2
.
Assuming that you created your project as my_thesaurus, we can now create our
own template in my_thesaurus/templates/my_welcome.jinja2
. Please consult
the Jinja2 documentation if you need help with this.
Once you’ve created your template file, you just need to tell your project to
override the default welcome.jinja2
with your version. To do this you
need to configure the Pyramid config object found in
my_thesaurus.__init__.py
.
config.override_asset(
to_override='atramhasis:templates/welcome.jinja2',
override_with='templates/my_welcome.jinja2'
)
Note
Normally, to see the effect of the changes you made, you would need to restart your webserver. When developing, you can make use of the pserve command’s auto-reload feature. To do this, start your server like this:
$ pserve --reload development.ini
Changing the focal conceptschemes¶
An Atramhasis instance should contain one or more conceptschemes. Four of your conceptschemes can be picked to receive a little more attention and focus than the other ones. These conceptschemes will appear on the homepagina with a list of recently visited concepts in those schemes.
Selecting which conceptschemes receive this focus is done in your
development.ini
file.
layout.focus_conceptschemes =
STUFF
This should be a space or newline delimited list, limited to 4 entries.
Changing the CSS¶
Out of the box, Atramhasis, comes with the Zurb Foundation framework. We have
created a custom style for this framework, but as always you are free to modify
this style. Your custom instance contains a few extension points that make it
easy to override and change style elements without having to rewrite to much
css. All style related files can be found in the my_thesaurus/static
folder. This project’s CSS is being maintained and generated by Compass. You
will find a scss
folder that contains three files that can be used for
easy customisations: _my_thesaurus-settings.scss
,
_my_thesaurus.scss
and _my_thesaurus-admin.scss
. The first file
is a settings file that allows you to override a lot of variables that are used
in generating the css. Suppose you want to override the default row width and
the default font. You would change _my_thesarus-settings.scss
to the
following:
// Custom SASS code for my_thesaurus
$row-width: rem-calc(1140);
$body-font-family: "museo-sans", "Open Sans", "Helvetica", Helvetica, Arial, sans-serif;
To have you changes take effect, you need to recompile the scss files and restart your webserver.
$ compass compile
write css/app-admin.css
write css/app.css
The other two files, _my_thesaurus.scss
and
_my_thesaurus-admin.scss
are the final scss files loaded before
compiling them and can be used to overwrite things in the public or admin
interface.
Security¶
We assume that every deployment of Atramhasis has different needs when it comes to security. Some instances will run on a simple laptop for testing and evaluation purposes, others might need a simple standalone database of users and certain deployments might need to integrate with enterprise authentication systems like LDAP, Active Directory, Single Sign On, …
Atramhasis provides authorisation hooks for security. To edit, add or delete a concept or collection, a user is required to have the ‘editor’ pemission. Unless no authorisation policy has been configured.
To get started, consult the sections of the Pyramid documentation on security.
Prior to version 0.6.3, Atramhasis contained a demo scaffold that had a custom security implementation using Mozilla Persona. Since this service has been discontinued, the security configuration was removed as well. But you can still check out the old code in our Github repository to see how it works.
Foreign Keys¶
Atramhasis will often function as a central part of a SOA in an
organisation. Concept
and maybe
Collection
objects will be used by other applications.
One of the riskier aspects of this is that someone might delete a concept in a
certain scheme that is still being used by another application. Even worse, the
user approving the delete might not even have a clue that the concept is being
used by some external application. While in the decentralised world that is the
world wide web, we can never be sure that nobody is using our concept any more,
we can take some steps to at least control what happens within other applications
that are within our control.
Of course, within the framework that is Atramhasis it’s very difficult to know how or where your own resources might be and how they might be using concepts from Atramhasis. We have therefor provided the necessary hooks for you that can help you deal with the sort of situation. But the actual implementation is left up to you.
We have added a decorator protected_operation()
.
When you add this decorator to a view, this view will emit a
ProtectedResourceEvent
. By default we
have added this decorator the delete_concept()
view.
In you own code, you can subscribe to this
ProtectedResourceEvent
through the
usual pyramid.events.subscriber()
. In this event handler you are then
free to implement whatever check you need to do. If you find that the resource
in question is being used somewhere and this operation
should thus not be allowed to proceed, you simply need to raise a
atramhasis.protected_resources.ProtectedResourceException
. Into this
exception you can also pass a list of URI that might provide the
user with some feedback as to where this concept might be used.
For example, a sample event handler that would make it impossible to delete concepts with a URI of less than 5 characters:
from pyramid.events import subscriber
from atramhasis.protected_resources import ProtectedResourceEvent
@subscriber(ProtectedResourceEvent)
def never_delete_a_short_uri(event):
if len(event.uri) < 5:
raise ProtectedResourceException(
'resource {0} has a URI shorter than 5 characters, preventing this operation'.format(event.uri),
[]
)
Adding Google Analytics¶
Out of the box, it’s very easy to add Google Analytics integration to Atramhasis.
All you need to do is add you Web Property ID to development.ini
.
# Enter your Google Analytics Web Property ID
ga.tracker_key = UA-12345678-9
This will add basic analytics to every page, using a Jinja2 macro. If you need
more control over the code, you can override this macro in your own project.
Suppose you always want to use SSL when sending data. First, you would create
you own macro, eg. in my_macros.jinja2
in the templates directory
of your own project.
{% macro ga_tracker(ga_key) %}
<!-- Google Analytics -->
<script type="text/javascript">
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', '{{ ga_key }}', 'auto');
ga('set', 'forceSSL', true);
ga('send', 'pageview');
</script>
<!-- End Google Analytics -->
{% endmacro %}
Once that’s done, you need to override the the ga
block in the base template. To
do this, it’s easiest to override Atramhasis’ base.jinja2
in your own
project. To do that, add the following to your project’s main function:
config.override_asset(
to_override='atramhasis:templates/base.jinja2',
override_with='templates/base.jinja2'
)
In this file, you can now choose what should appear within the ga block defined
in staticbase.jinja2
. Here we are just replacing one macro with another,
but you are off course free to make further alterations.
{%- extends 'staticbase.jinja2' -%}
{% block ga %}
{% set ga_key = ga_key|default(request.registry.settings["ga.tracker_key"]) %}
{% from 'my_macros.jinja2' import ga_tracker %}
{% if ga_key %}
{{ ga_tracker(ga_key) }}
{% endif %}
{% endblock %}
Adding external providers¶
Within your Atramhasis instance you can make use of external providers. These are other systems serving up thesauri that you can interact with. Within the admin interface you can create links to these thesauri as SKOS matches. This way you can state that a concept within your thesauri is the same as or similar to a concept in the external thesaurus. And, more interestingly, you can also import concepts from such a thesaurus into your own vocabulary. Importing a concept like this will automatically create a SKOS match for you. Once a match is in place, you can also update your local concept with information from the external concept by performing a merge.
To enable all this power, you again need to configure a provider in you
application. Continuing with our example project, we need
to go back to our my_thesaurus/skos/__init__.py
. In this file you need
to register other instances of
skosprovider.providers.VocabularyProvider
. Currently providers
have already been written for Getty Vocabularies, English Heritage vocabularies
and Flanders Heritage Vocabularies. Depending on the system you’re trying to
interact with, writing a new provider is fairly simple. For this example, we’ll
assume that you want to integrate the wealth of information that the
Art and Architecture Thesaurus (AAT) vocabulary offers you.
The AATProvider
for this
(and other Getty vocabularies) is available as skosprovider_getty and is
installed by default in an Atramhasis instance. All you need to do is configure
it. First, we need to import the provider. Place this code at the top
of my_thesaurus/skos/__init__.py
.
from skosprovider_getty.providers import AATProvider
Once this is done, we need to instantiate the provider within the includeme
function and register it with the skosprovider.registry.Registry
. This
is all quite similar to registering your own
skosprovider_sqlalchemy.providers.SQLAlchemyProvider
. One thing you do
need to do, is tagging this provider with a subject. By adding the external
subject to the provider, we let Atramhasis know that this is not a regular,
internal provider that can be stored in our database, but a special external
one that can only be used for making matches. As such, it will not be present
and visible to the public among your regular vocabularies.
AAT = AATProvider(
{'id': 'AAT', 'subject': ['external']},
)
skosregis.register_provider(AAT)
That’s all. You can do the same with the
TGNProvider
for the
Thesaurus of Geographic Names (TGN) or any of the providers for
heritagedata.org that can be found in
skosprovider_heritagedata.
In the end your my_thesaurus/skos/__init__.py
should look somewhat like
this:
# -*- coding: utf-8 -*-
import logging
log = logging.getLogger(__name__)
from skosprovider_sqlalchemy.providers import SQLAlchemyProvider
from skosprovider_getty.providers import AATProvider
from skosprovider.uri import UriPatternGenerator
def includeme(config):
STUFF = SQLAlchemyProvider(
{
'id': 'STUFF',
'conceptscheme_id': 1
},
config.registry.dbmaker,
uri_generator=UriPatternGenerator(
'http://id.mydata.org/thesauri/stuff/%s'
)
)
AAT = AATProvider(
{
'id': 'AAT',
'subject': ['external']
}
)
skosregis = config.get_skos_registry()
skosregis.register_provider(STUFF)
skosregis.register_provider(AAT)
Now you’ll be able to import from the AAT to your heart’s delight. For an extended example that adds even more providers, you could have a look at the demo scaffold that comes with Atramhasis.
Import a controlled vocabulary¶
Atramhasis includes a script atramhasis/scripts/import_file.py
which
helps you import an existing vocabulary from a file. It supports a few
different file types, but not every file type supports the full Atramhasis
datamodel.
The supported file types:
- RDF (.html, .hturtle, .mdata, .microdata, .n3, .nquads, .nt, .rdfa, .rdfa1.0, .rdfa1.1, .trix, .turtle, .xml)
using
RDFProvider
. This provider supports the full datamodel. - CSV (.csv) using
SimpleCsvProvider
. The provider only supports importing and id, a prefLabel, a note and a source. It will work well when importing a simple flat list, but not for complex hierarchies. - JSON (.json) using
DictionaryProvider
. This provider supports the full datamodel.
The script can be called through the commandline in the project virtual environment. Call it with the help argument to see the possible arguments.
$ workon my_thesarus
$ import_file --help
usage: import_file [--from path_input_file] [--to conn_string] [--conceptscheme_label cs_label]
(example: "import_file --from atramhasis/scripts/my_file --to sqlite:///atramhasis.sqlite --conceptscheme_label Labels")
Import file to a database
optional arguments:
-h, --help show this help message and exit
--from INPUT_FILE local path to the input file
--to TO Connection string of the output database
--conceptscheme_label CS_LABEL
Label of the conceptscheme
The from argument is required and details where the file you want to import is
located, for example my_thesaurus/data/trees.json
. It is relative to your
current location.
The to argument contains the connection string of output database. Only PostGreSQL and SQLite are supported. The structure is either postgresql://username:password@host:port/db_name or either sqlite:///path/db_name.sqlite. The default value is sqlite:///atramhasis.sqlite.
The data is loaded in a ConceptScheme
. The
conceptscheme needs a label. The label can be added to the conceptscheme_label
argument. The default label is the name of the file.
Once the data is loaded in the database, the configuration of the added provider must be
included in the my_thesaurus/skos/__init__.py
. A successfull run of the
script will give a suggestion of the code to add to this file. Make sure to use
the same ConceptSchem ID since it is needed to connect your provider and the
conceptscheme in the database.
For example, to insert this file:
[{"broader": [],
"id": 1,
"labels": [{"label": "The Larch",
"language": "en",
"type": "prefLabel"},
{"label": "De Lariks",
"language": "nl",
"type": "prefLabel"}],
"matches": {"broad": [],
"close": [],
"exact": [],
"narrow": [],
"related": []},
"member_of": [3],
"narrower": [],
"notes": [{"language": "en",
"note": "A type of tree.",
"type": "definition"}],
"related": [],
"subordinate_arrays": [],
"type": "concept",
"uri": "http://id.trees.org/1"},
{"broader": [],
"id": 2,
"labels": [{"label": "The Chestnut",
"language": "en",
"type": "prefLabel"},
{"label": "De Paardekastanje",
"language": "nl",
"type": "altLabel"},
{"label": "la châtaigne",
"language": "fr",
"type": "altLabel"}],
"matches": {"broad": [],
"close": [],
"exact": [],
"narrow": [],
"related": []},
"member_of": [3],
"narrower": [],
"notes": [{"language": "en",
"note": "A different type of tree.",
"type": "definition"}],
"related": [],
"subordinate_arrays": [],
"type": "concept",
"uri": "http://id.trees.org/2"},
{"id": 3,
"labels": [{"label": "Bomen per soort",
"language": "nl",
"type": "prefLabel"},
{"label": "Trees by species",
"language": "en",
"type": "prefLabel"}],
"member_of": [],
"members": [1, 2],
"notes": [],
"superordinates": [],
"type": "collection",
"uri": "http://id.trees.org/3"}]
We run the following command:
$ workon my_thesarus
$ import_file --from my_thesaurus/data/trees.json --to sqlite:///my_thesaurus.sqlite --conceptscheme_label Trees
This will return the following output:
sqlalchemy.engine.base.Engine SELECT CAST('test plain returns' AS VARCHAR(60)) AS anon_1
sqlalchemy.engine.base.Engine ()
sqlalchemy.engine.base.Engine SELECT CAST('test unicode returns' AS VARCHAR(60)) AS anon_1
sqlalchemy.engine.base.Engine ()
sqlalchemy.engine.base.Engine BEGIN (implicit)
sqlalchemy.engine.base.Engine INSERT INTO note (note, notetype_id, language_id) VALUES (?, ?, ?)
sqlalchemy.engine.base.Engine ('A type of tree.', 'definition', 'en')
sqlalchemy.engine.base.Engine INSERT INTO note (note, notetype_id, language_id) VALUES (?, ?, ?)
sqlalchemy.engine.base.Engine ('A different type of tree.', 'definition', 'en')
sqlalchemy.engine.base.Engine INSERT INTO conceptscheme (uri) VALUES (?)
sqlalchemy.engine.base.Engine (None,)
sqlalchemy.engine.base.Engine INSERT INTO label (label, labeltype_id, language_id) VALUES (?, ?, ?)
sqlalchemy.engine.base.Engine ('Trees', 'prefLabel', 'nl')
sqlalchemy.engine.base.Engine INSERT INTO label (label, labeltype_id, language_id) VALUES (?, ?, ?)
sqlalchemy.engine.base.Engine ('The Larch', 'prefLabel', 'en')
sqlalchemy.engine.base.Engine INSERT INTO label (label, labeltype_id, language_id) VALUES (?, ?, ?)
sqlalchemy.engine.base.Engine ('De Lariks', 'prefLabel', 'nl')
sqlalchemy.engine.base.Engine INSERT INTO label (label, labeltype_id, language_id) VALUES (?, ?, ?)
sqlalchemy.engine.base.Engine ('The Chestnut', 'prefLabel', 'en')
sqlalchemy.engine.base.Engine INSERT INTO label (label, labeltype_id, language_id) VALUES (?, ?, ?)
sqlalchemy.engine.base.Engine ('De Paardekastanje', 'altLabel', 'nl')
sqlalchemy.engine.base.Engine INSERT INTO label (label, labeltype_id, language_id) VALUES (?, ?, ?)
sqlalchemy.engine.base.Engine ('la châtaigne', 'altLabel', 'fr')
sqlalchemy.engine.base.Engine INSERT INTO label (label, labeltype_id, language_id) VALUES (?, ?, ?)
sqlalchemy.engine.base.Engine ('Bomen per soort', 'prefLabel', 'nl')
sqlalchemy.engine.base.Engine INSERT INTO label (label, labeltype_id, language_id) VALUES (?, ?, ?)
sqlalchemy.engine.base.Engine ('Trees by species', 'prefLabel', 'en')
sqlalchemy.engine.base.Engine INSERT INTO conceptscheme_label (conceptscheme_id, label_id) VALUES (?, ?)
sqlalchemy.engine.base.Engine (11, 3548)
sqlalchemy.engine.base.Engine INSERT INTO concept (type, concept_id, uri, conceptscheme_id) VALUES (?, ?, ?, ?)
sqlalchemy.engine.base.Engine ('concept', 1, 'http://id.trees.org/1', 11)
sqlalchemy.engine.base.Engine INSERT INTO concept (type, concept_id, uri, conceptscheme_id) VALUES (?, ?, ?, ?)
sqlalchemy.engine.base.Engine ('concept', 2, 'http://id.trees.org/2', 11)
sqlalchemy.engine.base.Engine INSERT INTO concept (type, concept_id, uri, conceptscheme_id) VALUES (?, ?, ?, ?)
sqlalchemy.engine.base.Engine ('collection', 3, 'http://id.trees.org/3', 11)
sqlalchemy.engine.base.Engine INSERT INTO concept_label (concept_id, label_id) VALUES (?, ?)
sqlalchemy.engine.base.Engine ((2558, 3551), (2558, 3552), (2558, 3553), (2557, 3549), (2557, 3550), (2559, 3554), (2559, 3555))
sqlalchemy.engine.base.Engine INSERT INTO concept_note (concept_id, note_id) VALUES (?, ?)
sqlalchemy.engine.base.Engine ((2558, 3605), (2557, 3604))
sqlalchemy.engine.base.Engine SELECT concept.id AS concept_id_1, concept.type AS concept_type, concept.concept_id AS concept_concept_id, concept.uri AS concept_uri, concept.conceptscheme_id AS concept_conceptscheme_id
FROM concept
WHERE concept.conceptscheme_id = ? AND concept.concept_id = ? AND concept.type IN (?)
sqlalchemy.engine.base.Engine (11, 1, 'concept')
sqlalchemy.engine.base.Engine SELECT concept.id AS concept_id_1, concept.type AS concept_type, concept.concept_id AS concept_concept_id, concept.uri AS concept_uri, concept.conceptscheme_id AS concept_conceptscheme_id
FROM concept
WHERE concept.conceptscheme_id = ? AND concept.concept_id = ? AND concept.type IN (?)
sqlalchemy.engine.base.Engine (11, 2, 'concept')
sqlalchemy.engine.base.Engine SELECT concept.id AS concept_id_1, concept.type AS concept_type, concept.concept_id AS concept_concept_id, concept.uri AS concept_uri, concept.conceptscheme_id AS concept_conceptscheme_id
FROM concept
WHERE concept.conceptscheme_id = ? AND concept.concept_id = ? AND concept.type IN (?)
sqlalchemy.engine.base.Engine (11, 3, 'collection')
sqlalchemy.engine.base.Engine SELECT concept.id AS concept_id_1, concept.type AS concept_type, concept.concept_id AS concept_concept_id, concept.uri AS concept_uri, concept.conceptscheme_id AS concept_conceptscheme_id
FROM concept
WHERE concept.conceptscheme_id = ? AND concept.concept_id = ?
sqlalchemy.engine.base.Engine (11, 1)
sqlalchemy.engine.base.Engine SELECT concept.id AS concept_id_1, concept.type AS concept_type, concept.concept_id AS concept_concept_id, concept.uri AS concept_uri, concept.conceptscheme_id AS concept_conceptscheme_id
FROM concept, collection_concept
WHERE ? = collection_concept.collection_id AND concept.id = collection_concept.concept_id
sqlalchemy.engine.base.Engine (2559,)
sqlalchemy.engine.base.Engine INSERT INTO collection_concept (collection_id, concept_id) VALUES (?, ?)
sqlalchemy.engine.base.Engine (2559, 2557)
sqlalchemy.engine.base.Engine SELECT concept.id AS concept_id_1, concept.type AS concept_type, concept.concept_id AS concept_concept_id, concept.uri AS concept_uri, concept.conceptscheme_id AS concept_conceptscheme_id
FROM concept
WHERE concept.conceptscheme_id = ? AND concept.concept_id = ?
sqlalchemy.engine.base.Engine (11, 2)
sqlalchemy.engine.base.Engine INSERT INTO collection_concept (collection_id, concept_id) VALUES (?, ?)
sqlalchemy.engine.base.Engine (2559, 2558)
sqlalchemy.engine.base.Engine COMMIT
sqlalchemy.engine.base.Engine BEGIN (implicit)
sqlalchemy.engine.base.Engine SELECT label.id AS label_id, label.label AS label_label, label.labeltype_id AS label_labeltype_id, label.language_id AS label_language_id
FROM label JOIN conceptscheme_label ON label.id = conceptscheme_label.label_id
WHERE label.label = ?
LIMIT ? OFFSET ?
sqlalchemy.engine.base.Engine ('Trees', 1, 0)
sqlalchemy.engine.base.Engine SELECT conceptscheme.id AS conceptscheme_id, conceptscheme.uri AS conceptscheme_uri
FROM conceptscheme, conceptscheme_label
WHERE ? = conceptscheme_label.label_id AND conceptscheme.id = conceptscheme_label.conceptscheme_id
sqlalchemy.engine.base.Engine (3548,)
*** The import of the my_thesaurus/data/trees.json file with conceptscheme label 'Trees' to sqlite:///my_thesaurus.sqlite was successfull. ***
To use the data in Atramhasis, you must edit the file my_thesaurus/skos/__init__.py.
Add next lines:
def includeme(config):
TREES = SQLAlchemyProvider(
{'id': 'TREES', 'conceptscheme_id': 11},
config.registry.dbmaker
)
skosregis = config.get_skos_registry()
skosregis.register_provider(TREES)
Just follow these instructions and edit your my_thesaurus/skos/__init__.py
like this:
# -*- coding: utf-8 -*-
import logging
from skosprovider_sqlalchemy.providers import SQLAlchemyProvider
log = logging.getLogger(__name__)
def includeme(config):
TREES = SQLAlchemyProvider(
{'id': 'TREES', 'conceptscheme_id': 11},
config.registry.dbmaker
)
skosregis = config.get_skos_registry()
skosregis.register_provider(TREES)
Now your thesaurus has been successfully imported and is ready to be browsed, expanded and edited.
SessionFactory¶
You can change the default session factory in the __init__.py file.
# set default session factory
from pyramid.session import SignedCookieSessionFactory
atramhasis_session_factory = SignedCookieSessionFactory(settings['atramhasis.session_factory.secret'])
config.set_session_factory(atramhasis_session_factory)
API Documentation¶
atramhasis.data¶
atramhasis.data.datamanagers¶
This module adds DataManagers for Atramhasis. These are service layer objects that abstract all interactions with the database away from the views.
versionadded: | 0.4.1 |
---|
-
class
atramhasis.data.datamanagers.
AuditManager
(session)[source]¶ A data manager for logging the visit.
-
get_most_popular_concepts_for_conceptscheme
(conceptscheme_id, max=5, period='last_month')[source]¶ get the most popular concepts for a conceptscheme :param conceptscheme_id: id of the conceptscheme :param max: maximum number of results, default 5 :param period: ‘last_day’ or ‘last_week’ or ‘last_month’ or ‘last_year’, default ‘last_month’ :return: List of the most popular concepts of a conceptscheme over a certain period
-
-
class
atramhasis.data.datamanagers.
ConceptSchemeManager
(session)[source]¶ A
DataManager
forConceptSchemes <skosprovider_sqlalchemy.models.ConceptScheme>.
-
find
(conceptscheme_id, query)[source]¶ Find concepts and collections in this concept scheme.
Parameters: - conceptscheme_id – a concepscheme id
- query – A python dictionary containing query parameters.
Returns: A
list
ofskosprovider_sqlalchemy.models.Thing
instances.
-
get
(conceptscheme_id)[source]¶ Parameters: conceptscheme_id – a concepscheme id Returns: the concepscheme for the given id
-
get_all
(conceptscheme_id)[source]¶ Get all concepts and collections in this concept scheme.
Parameters: conceptscheme_id – a concepscheme id Returns: A list
ofskosprovider_sqlalchemy.models.Thing
instances.
-
get_collections_for_scheme_tree
(conceptscheme_id)[source]¶ Parameters: conceptscheme_id – a concepscheme id Returns: all collections for the scheme_tree
-
-
class
atramhasis.data.datamanagers.
CountsManager
(session)[source]¶ A data manager that deals with triple counts.
-
class
atramhasis.data.datamanagers.
DataManager
(session)[source]¶ A DataManager abstracts all interactions with the database for a certain model.
-
class
atramhasis.data.datamanagers.
LanguagesManager
(session)[source]¶ A
DataManager
forLanguages <skosprovider_sqlalchemy.models.Language>.
-
class
atramhasis.data.datamanagers.
SkosManager
(session)[source]¶ A
DataManager
forConcepts and Collections <skosprovider_sqlalchemy.models.Thing>.
-
get_by_list_type
(list_type)[source]¶ Parameters: list_type – a specific list type Returns: all results for the specific list type
-
atramhasis.data.db¶
Module that sets up the datamanagers and the database connections.
-
atramhasis.data.db.
data_managers
(request)[source]¶ Generate a datamanager with a database session and register a cleanup handler.
Parameters: request (pyramid.request.Request) – The request this db session will be tied to. Returns: A dictionary containing different datamanagers
.
-
atramhasis.data.db.
includeme
(config)[source]¶ Set up SQLAlchemy.
Parameters: config (pyramid.config.Configurator) – Pyramid configuration.
atramhasis.errors¶
Module containing errors generated by Atramhasis.
-
exception
atramhasis.errors.
ConceptNotFoundException
(c_id)[source]¶ A Concept or Collection could not be found.
-
exception
atramhasis.errors.
ConceptSchemeNotFoundException
(scheme_id)[source]¶ A ConceptScheme could not be found.
-
exception
atramhasis.errors.
DbNotFoundException
(value='No database found, please check your application setup')[source]¶ Atramhasis could not find a database.
-
exception
atramhasis.errors.
LanguageNotFoundException
(scheme_id)[source]¶ A Language could not be found.
atramhasis.mappers¶
Module containing mapping functions used by Atramhasis.
-
atramhasis.mappers.
is_html
(value)[source]¶ Check if a value has html inside. Only tags checked <strong> <em> <a>.
Parameters: value – a string Returns: a boolean (True, HTML present | False, no HTML present)
-
atramhasis.mappers.
map_concept
(concept, concept_json, skos_manager)[source]¶ Map a concept from json to the database.
Parameters: - concept (skosprovider_sqlalchemy.models.Thing) – A concept or collection as known to the database.
- concept_json (dict) – A dict representing the json sent to our REST service.
- skos_manager – A skos_manager to acces db operations
Returns: The
skosprovider_sqlalchemy.models.Thing
enhanced with the information from the json object.
-
atramhasis.mappers.
map_conceptscheme
(conceptscheme, conceptscheme_json)[source]¶ Map a conceptscheme from json to the database.
Parameters: - conceptscheme (skosprovider_sqlalchemy.models.ConceptScheme) – A conceptscheme as known to the database.
- conceptscheme_json (dict) – A dict representing the json sent to our REST service.
Returns: The
skosprovider_sqlalchemy.models.ConceptScheme
enhanced with the information from the json object.
atramhasis.protected_resources¶
-
class
atramhasis.protected_resources.
ProtectedResourceEvent
(uri, request)[source]¶ Event triggered when calling a protected operation on a resource
atramhasis.routes¶
Routes for the Atramhasis views.
New in version 0.4.4.
-
atramhasis.routes.
includeme
(config)[source]¶ Setup the routing for Atramhasis.
Parameters: config (pyramid.config.Configurator) – The application config.
atramhasis.utils¶
Module containing utility functions used by Atramhasis.
-
atramhasis.utils.
from_thing
(thing)[source]¶ Map a
skosprovider_sqlalchemy.models.Thing
to askosprovider.skos.Concept
or askosprovider.skos.Collection
, depending on the type.Parameters: thing (skosprovider_sqlalchemy.models.Thing) – Thing to map. Return type: Concept
orCollection
.
-
atramhasis.utils.
internal_providers_only
(fn)[source]¶ aspect oriented way to check if provider is internal when calling the decorated function
Parameters: fn – the decorated function Returns: around advice Raises: pyramid.httpexceptions.HTTPMethodNotAllowed – when provider is not internal
atramhasis.validators¶
Module that validates incoming JSON.
-
atramhasis.validators.
broader_hierarchy_rule
(errors, node_location, skos_manager, conceptscheme_id, cstruct)[source]¶ Checks that the broader concepts of a concepts are not alreadt narrower concepts of that concept.
-
atramhasis.validators.
collection_members_unique_rule
(errors, node_location, members)[source]¶ Checks that a collection has no duplicate members.
-
atramhasis.validators.
collection_type_rule
(errors, node_location, skos_manager, conceptscheme_id, members)[source]¶ Checks that the targets of member_of are collections and not concepts.
-
atramhasis.validators.
concept_matches_rule
(errors, node_location, matches, concept_type)[source]¶ Checks that only concepts have matches.
-
atramhasis.validators.
concept_matches_unique_rule
(errors, node_location, matches)[source]¶ Checks that a concept has not duplicate matches.
This means that a concept can only have one match (no matter what the type) with another concept. We don’t allow eg. a concept that has both a broadMatch and a relatedMatch with the same concept.
-
atramhasis.validators.
concept_relations_rule
(errors, node_location, relations, concept_type)[source]¶ Checks that only concepts have narrower, broader and related relations.
-
atramhasis.validators.
concept_schema_validator
(node, cstruct)[source]¶ This validator validates an incoming concept or collection
This validator will run a list of rules against the concept or collection to see that there are no validation rules being broken.
Parameters: - node (colander.SchemaNode) – The schema that’s being used while validating.
- cstruct – The concept or collection being validated.
-
atramhasis.validators.
concept_type_rule
(errors, node_location, skos_manager, conceptscheme_id, items)[source]¶ Checks that the targets of narrower, broader and related are concepts and not collections.
-
atramhasis.validators.
conceptscheme_schema_validator
(node, cstruct)[source]¶ This validator validates the incoming conceptscheme labels
Parameters: - node (colander.SchemaNode) – The schema that’s being used while validating.
- cstruct – The conceptscheme being validated.
-
atramhasis.validators.
hierarchy_rule
(errors, node_location, skos_manager, conceptscheme_id, cstruct, property1, property2, property2_list_name, concept_type, error_message)[source]¶ Checks that the property1 of a concept are not already in property2 hierarchy
-
atramhasis.validators.
html_preparer
(value)[source]¶ Prepare the value by stripping all html except certain tags.
Parameters: value – The value to be cleaned. Return type: str
-
atramhasis.validators.
label_lang_rule
(errors, node, languages_manager, labels)[source]¶ Checks that languages of a label are valid.
Checks that they are valid IANA language tags. If the language tag was not already present in the database, it adds them.
-
atramhasis.validators.
label_type_rule
(errors, node, skos_manager, labels)[source]¶ Checks that a label has the correct type.
-
atramhasis.validators.
languagetag_checkduplicate
(node, language_tag, languages_manager, errors)[source]¶ Check that a languagetag isn’t duplicated.
-
atramhasis.validators.
languagetag_isvalid_rule
(node, language_tag, errors)[source]¶ Check that a languagetag is a valid IANA language tag.
-
atramhasis.validators.
languagetag_validator
(node, cstruct)[source]¶ This validator validates a languagetag.
The validator will check if a tag is a valid IANA language tag. The the validator is informed that this should be a new language tag, it will also check if the tag doesn’t already exist.
Parameters: - node (colander.SchemaNode) – The schema that’s being used while validating.
- cstruct – The value being validated.
-
atramhasis.validators.
max_preflabels_rule
(errors, node, labels)[source]¶ Checks that there’s only one prefLabel for a certain language.
-
atramhasis.validators.
members_hierarchy_rule
(errors, node_location, skos_manager, conceptscheme_id, cstruct)[source]¶ Checks that a collection does not have members that are in themselves already “parents” of that collection.
-
atramhasis.validators.
members_only_in_collection_rule
(errors, node, concept_type, members)[source]¶ Checks that only collections have members.
-
atramhasis.validators.
min_labels_rule
(errors, node, cstruct)[source]¶ Checks that a label or collection always has a least one label.
-
atramhasis.validators.
narrower_hierarchy_rule
(errors, node_location, skos_manager, conceptscheme_id, cstruct)[source]¶ Checks that the narrower concepts of a concept are not already broader concepts of that concept.
-
atramhasis.validators.
semantic_relations_rule
(errors, node_location, skos_manager, conceptscheme_id, members, collection_id)[source]¶ Checks that the elements in a group of concepts or collections are not the the group itself, that they actually exist and are within the same conceptscheme.
-
atramhasis.validators.
subordinate_arrays_hierarchy_rule
(errors, node_location, skos_manager, conceptscheme_id, cstruct)[source]¶ Checks that the subordinate arrays of a concept are not themselves parents of that concept.
-
atramhasis.validators.
subordinate_arrays_only_in_concept_rule
(errors, node, concept_type, subordinate_arrays)[source]¶ Checks that only a concept has subordinate arrays.
-
atramhasis.validators.
subordinate_arrays_type_rule
(errors, node_location, skos_manager, conceptscheme_id, subordinate_arrays)[source]¶ Checks that subordinate arrays are always collections.
-
atramhasis.validators.
superordinates_hierarchy_rule
(errors, node_location, skos_manager, conceptscheme_id, cstruct)[source]¶ Checks that the superordinate concepts of a collection are not themselves members of that collection.
atramhasis.views¶
atramhasis.views.views¶
-
class
atramhasis.views.views.
AtramhasisAdminView
(request)[source]¶ This object groups HTML views part of the admin user interface.
-
class
atramhasis.views.views.
AtramhasisListView
(request)[source]¶ This object groups list views part for the user interface.
atramhasis.views.crud¶
Module containing views related to the REST service.
-
class
atramhasis.views.crud.
AtramhasisCrud
(context, request)[source]¶ This object groups CRUD REST views part of the private user interface.
-
edit_conceptscheme
()[source]¶ Edit an existing concept
Raises: atramhasis.errors.ValidationError – If the provided json can’t be validated
-
atramhasis.views.exception_views¶
Module containing error views.
-
atramhasis.views.exception_views.
data_integrity
(exc, request)[source]¶ View invoked when IntegrityError was raised.
-
atramhasis.views.exception_views.
failed
(exc, request)[source]¶ View invoked when bad data was submitted to Atramhasis.
-
atramhasis.views.exception_views.
failed_not_found
(exc, request)[source]¶ View invoked when a resource could not be found.
-
atramhasis.views.exception_views.
failed_not_method_not_allowed
(exc, request)[source]¶ View invoked when a method is not allowed.
-
atramhasis.views.exception_views.
failed_skos
(exc, request)[source]¶ View invoked when Atramhasis can’t find a SKOS registry.
-
atramhasis.views.exception_views.
failed_validation
(exc, request)[source]¶ View invoked when bad data was submitted to Atramhasis.
-
atramhasis.views.exception_views.
protected
(exc, request)[source]¶ when a protected operation is called on a resource that is still referenced
View invoked when ProviderUnavailableException was raised.
History¶
0.6.7 (21-06-2019)¶
- Fix corrupt build
- Security updates
0.6.6 (01-03-2019)¶
- Update Colander and other dependencies. (#464)
- Remove old convert_oe script because it depends on an obsolete webservice. (#466)
- Fix an issue with circular dependencies in requirements files. (#463)
- Change the default GA macro to anonymizeIp and be more GDPR compliant. (#450)
0.6.5 (19-12-2018)¶
- Generate a default dump location in development.ini files. (#416)
- Update skosprovider_sqlalchemy to solve a problem with the tree cache. (#455)
- Update a lot of dependencies.
0.6.4 (22-12-2017)¶
0.6.3 was a brown bag release. Sorry!
This version is exactly the same as 0.6.3, but properly packaged.
0.6.3 (21-12-2017)¶
- This version updates a lot of the requirements to their latest versions. This might be an issued if you’ve written lots of code against older pyramid versions. (#418, #413, #412, #411, #410, #408, #407, #404, #403)
- Remove authentication from the demo version since Persona does not work anymore. (#361)
- Fixed the base HTML template and added a DOCTYPE declaration. (#429)
- Update the URI’s for licenses of the Flemish Government. (#430)
- Fixed an issues with sorting on Python 3. (#424)
0.6.2 (11-10-2017)¶
- When an LDF server is present, add a link to the HTML document to this server. (#394)
- Wrong expansion of SKOS namespace in LDF server. (#401)
0.6.1 (01-09-2017)¶
This release is a minor release, containing improvements regarding the Linked Data Fragments server.
- Also add hidden datasets to the LDF server. Only external ones are not added now. (#390)
- Make it possible to set the LDF server protocol when generating the config. (#391)
- When generating an LDF server config, add a composite source as well. (#393)
- When generating the dataset information, add hydra controls that link to the LDF server instance. (#392)
0.6.0 (23-08-2017)¶
This release is a major release containing new features.
- Added a script to generate nightly dumps. Instead of generating full downloads on demand, they can now be generated by a cron job (eg. once per night, week, …). This makes it possible to download a large conceptscheme at once. During these dumps, some statistics on every conceptscheme such as the number of triples in it will be generated as well. This was done to make it easier to embed a custom Python based LDF server, but currently only serves the purpose of keeping some score. (#337, #360)
- Added easy integration with a Linked Data Fragments server (https://linkeddatafragments.org). Atramhasis can now generate a config file for such a server that you can use to setup the server. By default this config will work with the Turtle files that can be generated every night. But if you have access to the HDT library, you can also work with HDT files for a masssive performance boost. See the section Running a Linked Data Fragments server in the docs for more information. (#365)
- Add some more information the HTML title tags for a concept detail. (#363)
- Changed the UI for doing a search so that you now get a proper warning when searching for a label without specifying the conceptscheme to search in. (#373)
- It is now possible to generate URI’s when importing from a file that does not contain them, eg. a JSON or CSV file. The import_file can now take a pattern_uri parameter than will be used to generate new URI’s with. (#372)
- Fixed some issues with the tree cache that came to light when running Atramhasis as two nodes. Where before it was not possible to configure the tree cache, it now is. Previously an in-memory cache would always be used. Now it’s possible to use a different type of cache. If you’re running more than one webserver, it is advised to run a shared cache. If you’re running a previous version of Atramhasis, you will need to configure your cache with cache.tree and cache.list settings. (#371)
- It is now possible to add sortLabels to concepts. These can be used in the REST service to arbitrarily sort concepts. The sortLabel works per language. This makes it possible to eg. sort historical periods in chronological order. Most of the functionality was already present in skosprovider 0.6.0, but it had not been properly included in Atramhasis. (#362)
- Added ‘und - undetermined’ to the default language set to support json file imports. (#386)
- Fixed a bug when editing concepts where data from previously opened concepts would bleed into the concept you were editing. (#367)
- Update several dependencies to the latest versions. (#380, #381, #376)
- Added ‘und - undetermined’ to the default language set to support json file imports (#386)
0.5.2 (07-10-2016)¶
This minor release fixes a bug with the protected resource event. The event should give the uri of a concept instead of the url path. In addition to the uri the request is added to the event. It also fixes the bug with removing relations and updates the requirements for skosprovider_sqlalchemy.
0.5.1 (04-10-2016)¶
This minor release fixes a bug with the tree browser. Before it wasn’t possible to zoom and pan the tree. With certain larger trees this would cause issues as content would run off the page.
0.5.0 (14-09-2016)¶
This release is a major update based on the skosprovider 0.6.0 line of libraries. The most visible change is with the public and admin interfaces. These have been completely overhauled to provide a more pleasing user experience. Among other things visitors are now pointed towards popular concepts and concepts they have recently visited. Browsing an entire conceptscheme tree has been redesigned.
The adming interface now offers users an option to edit certain aspects of a conceptscheme such as the labels, notes and sources. Editing in general has been update and improved. Links between the public interface and the admin interface have been added to make switching from one to the other easier. Notes and sources can now contain certain HTML tags, allowing greater flexibility in defining concepts and collections.
A command line script was added to make it easy to import an entire conceptscheme, eg. when migrating from another system. It is now possible to import a RDF, CSV or JSON file on the command line in your Atramhasis instance. With earlier versions you had to script this yourself.
As always, bugs have been fixed, code has been rewritten and documenation has been updated.
See https://github.com/OnroerendErfgoed/atramhasis/milestone/8?closed=1 for the full list of changes.
0.4.4 (04-06-2015)¶
- Added more sample datasets to get a better view of real data. These will make the demo more interesting.
- Fix a bug where it was possible to create a relation between a concept and itself causing all sorts of nasty things to happen.
- Minor refactoring. Move the pyramid routes to a new file.
- Added a CONTRIBUTING.md file. Contributions welcome!
0.4.3 (11-03-2015)¶
We had some packaging issues with the 0.4.2 release.
0.4.2 (11-03-2015)¶
This release of Atramhasis is mostly a bugfix update of the 0.4.1 release.
- Fix paths of db in scaffolds
- Add more information on exceptions
- Update skosprovider_getty and skosprovider_heritagedata (fix the problems when importing external thesauri)
- Documentation update
0.4.1 (04-03-2015)¶
This release of Atramhasis is a minor update of the 0.4.0 release, focussing on small corrections and improvements and improving the documentation. A few interesting non-invasive features were added, mostly to the editor’s admin interface and machine-readable exports of RDF data.
Upgrading from 0.4.0 should be simple and cause no or few problems.
- A conceptscheme, concept or collection can now be exported to RDF through
skosprovider_rdf 0.3.1. These are individuals export endpoints that can
be reached in one of two ways. Either by hitting a url like
http://localhost:6543/conceptschemes/GEOGRAPHY/c/335 with a supported RDF mimetype
(
application/rdf+xml
,application/x-turtle
,text-turle
). Or by using an RDF syntax specific suffix (.rdf or .ttl). - When importing, allow the user to request more information on a concept or collection, before actually importing it.
- Allow merging a concept with other concepts it matches. This allows a user to compare a local concept with an external one it matches and import any notes or labels that are present in the external concept, but not the local one.
- Reworked some parts of the public interface to make everything a bit clearer and to make all pages easily reachable.
- Allow sorting the languages in the admin interface.
- Reorganised and extended the right click menu on the grid in the admin interface.
- Allow looking up a skos:match from within the admin interface.
- Some issues with the length of language ids were solved.
- Fixed some issues when importing a collection instead of a concept.
- Made it easy to add a Google Analytics tracker.
- Added instructions on how to deploy a demo site on heroku. These work just as well for deploying an actual production site to heroku.
- Lots of small updates and tweaks to the documentation.
- Updated some dependencies.
- Some code cleanup and reorganisation. Several smaller bugs in the admin interface were fixed.
- The data fixtures were updated with skos:note examples. Added a license for reuse of the fixture data.
0.4.0 (23-12-2014)¶
- Update to skosprovider 0.5.0. Among other things, this makes it possible to handle relations between Concepts and Collections using the subordinate_arrays and superordinates properties. Conceptschemes are now also much better integrated within the providers, thus making it possible to provider more context for a Concept. This version of skosprovider can also handle skos:matches.
- Add possibility to edit language tags. It’s now possible to use the admin interface to add, edit and delete languages in Atramhasis.
- When the REST service receives labels or notes in currently unavailable languages, it will validate those through language_tags. It the languages are valid according to the IANA registry, they will be added to the languages available in the application.
- Default length of language id changed to 64 characters. This is not available as an alembic migration. So only effective when creating a new database. If you already have a database created from an older version of Atramhasis, please modify accordingly. Modifying column length on SQLite is not possible (see http://www.sqlite.org/omitted.html ).
- Abiltity to match Concepts in an Atramhasis ConceptScheme to Concepts in external ConceptSchemes through properties such as skos:exactMatch and skos:closeMatch.
- Ability to import Concepts and Collections from external providers. This makes it possible to import Concepts from eg. the AAT (via skosprovider_getty), Flanders Heritage Thesauri (via skosprovider_oe), English Heritage Thesauri (via skosprovider_heritagedata) or any other SKOS vocabulary for which a skosprovider has been written. Currently only the concept or collection itself can be imported, without its relations to other concepts or collections.
- Add the ability to have a delete of a concept or collection fail if it is being used in other systems.
- Implement a delete permission.
- Add validation rule that a Concept must have at least one label.
- Update to skosprovider_sqlalchemy 0.4.1.
- Update to pyramid_skosprovider 0.5.0.
- Update to skosprovider_rdf 0.3.0. This update adds support for dumping ConceptScheme in an RDF file and also handles subordinate_arrays and superordinates.
- Update to language_tags 0.3.0.
0.3.1 (05-09-2014)¶
- Update to skosprovider_sqlalchemy 0.2.1.
- Update to skosprovider_rdf 0.1.3 This fixes an issue with RDF having some SKOS elements in the wrong namespace. Also added a missing dependency on skosprovider_rdf to setup.py
- Updated the Travis build file to run a basic dojo build and test for build failures.
0.3.0 (15-08-2014)¶
- Atramhasis now includes a working admin userinterface at /admin. Still needs some polish when it comes to error handling and reporting about validation errors.
- The admin module gets run through a dojo build to minimize page loads and download times
- Added RDF/XML en RDF/Turtle downloads to the public interface. Currently only dumps a full conceptscheme, not individual concepts.
- Added more docs.
0.2.0 (16-05-2014)¶
- Full public userinterface
- REST CRUD service
- Security integration
- CSV export
- demo using Mozilla Persona as sample security setup
0.1.0 (22-04-2014)¶
- Initial version
- Setup of the project: docs, unit testing, code coverage
- Scaffolding for demo and deployment packages
- Limited public user interface
- Basis i18n abilities present
- Integration of pyramid_skosprovider
- Integration of skosprovider
- Integration of skosprovider_sqlalchemy
Glossary¶
- CSS
- Cascading Style Sheet is a style specification used to add style and presentation to webpages.
- HDT
- HDT (Header, Dictionary, Triples) is a compact data structure and binary serialization format for RDF that keeps big datasets compressed to save space while maintaining search and browse operations without prior decompression. This makes it an ideal format for storing and sharing RDF datasets on the Web.
- HTML
- HyperText Markup Language is the markup language used to create webpage.
- Jinja2
- Jinja2 is a python templating engine. It’s used by Atramhasis for rendering HTML templates.
- Pyramid
- This webframework was used to implement the server side components of Atramhasis.
- RDF
- Resource Description Framework. A very flexible model for data definition organised around triples. These triples forms a directed, labeled graph, where the edges represent the named link between two resources, represented by the graph nodes.
- REST
- REST or REpresentational State Transfer is a way of data exchange that is very complimentary to the operations of the HTTP protcol.
- SKOS
- Simple Knowledge Organization System. An general specification for Knowledge Organisation Systems (thesauri, word lists, authority files, …) that is commonly serialised as RDF.
- SKOS-THES
- The ISO 25964 SKOS extension defines mappings between the ISO 25964 standard and the SKOS specification.
- SOA
- Service Oriented Architecture.
- URI
- A Uniform Resource Identifier.
- URN
- A URN is a specific form of a URI.