Development¶

Technology¶

Atramhasis is a python webapplication that is being developed within the pyramid framework. Other major technologies used are sqlalchemy as the ORM and Jinja2 as the templating framework.

Client side the main technologies being used are Zurb Foundation and a Vue 3 application built with Vite.

While Atramhasis is an editor for creating and editing SKOS vocabularies, it uses other libraries that are more geared towards using a vocabulary in an application.

skosprovider: This library defines a VocabularyProvider. This is an abstraction of usefull functionalities an application integrating SKOS needs. Different libraries can implement this interface for different datasources. This allows decoupling the interface from the concrete implementation. Out of the box this comes with a simple DictionaryProvider that serves a vocabulary based on a simple python dict as datasource.
skosprovider_sqlalchemy: An implementation of the VocabularyProvider interface with a SQLAlchemy backend. This allows using a RDBMS for reading, but also writing, SKOS concepts.
skosprovider_rdf: An implemenation of the VocabularyProvider interface with an RDF backend. Atramhasis uses this for exporting ConceptSchemes to RDF. It can also be used to get an existing SKOS vocabulary defined in RDF into Atramhasis.
pyramid_skosprovider: A library that integrates pyramid and skosprovider. This libraries creates a skosprovider.registry.Registry and makes it accessible through the pyramid.request.Request. Is also exposes a set of readonly REST services on the registered providers.
skosprovider_getty: An implemenation of the VocabularyProvider against the Linked Open Data vocabularies published by the Getty Research Institute at http://vocab.getty.edu such as the Art and Architecture Thesaurus (AAT) and the Thesaurus of Geographic Names (TGN).
skosprovider_heritagedata: An implementation of the VocabularyProvider against the vocabularies published by EH, RCAHMS and RCAHMW at heritagedata.org.

Atramhasis can easily be set up with a Linked Data Fragments (LDF) server. This server allows posing simple triple pattern queries of your dataset. Combined with a Linked Data Fragments client, similar functionalities to a traditional SPARQL endpoint can be achieved. Atramhasis facilitates the setup of a Linked Data Fragments server by generating a suitable config file for the Javascript server. This server can use different backends. Out of the box, Atramhasis generates Turtle files that can be used by the LDF server. It’s also possible to configre Atramhasis with a link to the rdf2hdt command (requires a separate installation). In this case, everytime the conceptschemes are dumped to RDF, the dump files are also written in HDT format.

General installation¶

We recommend installing Atramhasis in a virtual environment.

$ python -m venv atramhasis_dev
$ . atramhasis_dev/bin/activate
# Make sure pip and pip-tools are up to date
$ pip install --upgrade pip pip-tools

To install a fully working development environment a pip requirements-dev.txt file is provided. By passing this file to pip sync all requirements for Atramhasis and development of the software (Sphinx, py.test) will be installed.

The following step will help you get the python development environment up and running. If you also need to work on the javascript admin backend, please refer to the admin module documentation.

# Install packages in dev mode. During a pip install, the JS files will also be built.
$ pip install -e ."[dev]"
# create or update database
$ alembic upgrade head
# insert sample data
$ initialize_atramhasis_db development.ini
# generate first RDF download
$ dump_rdf development.ini
# compile the Message Catalog Files
$ pybabel compile --directory 'atramhasis/locale' --domain atramhasis --statistics true

Once you’ve executed these steps, you can run a development server. This uses the standard pyramid server (Waitress) and should not be used as-is in a production environment.

# run a local development server
$ pserve --reload development.ini

Update requirements files¶

This project does not offer requirements files that are manually maintained. The dependencies are defined within pyproject.toml. There, you can add, modify, or remove libraries. Afterward, run pip-compile to generate the requirements files.

Admin and frontend development¶

Since version 3.3.0, the admin client uses Vue 3. To work on the frontend/admin client, install Node.js and pnpm.

Required version ranges are:

Node.js: >=22.12.0
pnpm: >=9

# Verify installed versions
$ node -v
$ pnpm -v

If Node.js is not installed yet, install it first. We recommend using fnm so you can manage Node.js versions per project:

# Install Node.js (LTS) with fnm
$ fnm install --lts
$ fnm use lts-latest

If pnpm is not installed yet, you can enable it through corepack:

$ corepack enable
$ corepack prepare pnpm@latest --activate

The frontend source lives in frontend.

$ cd frontend
$ pnpm install

# Start the Vue app in dev mode
$ pnpm dev

# Type-check the Vue + TS application
$ pnpm type-check

# Build production assets
$ pnpm build

The production build outputs static assets into atramhasis/static/dist. In development.ini, use vue.mode to control whether source (src) or built (dist) frontend assets are used.

When building a distribution with hatch build, the build hook installs frontend dependencies and runs the frontend build automatically.

Translations¶

Translations are handled differently for backend templates and the Vue admin frontend.

Backend/template translations¶

When updating the backend templates, add translation keys with {% trans %} tags in the templates.

<h2>{% trans %}welcome_to{% endtrans %}</h2>

To update the message catalogs, do as follows:

$ pybabel extract --add-comments 'TRANSLATORS:' --output-file 'atramhasis/locale/atramhasis.pot' --width 80 --mapping-file 'message-extraction.ini' atramhasis
$ pybabel update --input-file 'atramhasis/locale/atramhasis.pot' --output-dir 'atramhasis/locale' --previous true --domain atramhasis

Update the catalogs accordingly and run:

$ pybabel compile --directory 'atramhasis/locale' --domain atramhasis --statistics true

You might also want to add a new translation. Suppose you want to add a German translation.

$ pybabel init --locale de --input-file 'atramhasis/locale/atramhasis.pot' --output-dir atramhasis/locale --domain atramhasis

Edit atramhasis/locale/de/LC_MESSAGES/atramhasis.po and add the necessary translations. Just as with updating the catalogs, you need to recompile them.

$ pybabel compile --directory 'atramhasis/locale' --domain atramhasis --statistics true

At this moment, Atramhasis will still only show the default languages in it’s language switcher. If you want to add your new language, you need to edit your development.ini (or similar file). Look for the line that says available_languages and add your locale identifier.

available_languages = en nl fr de

After restarting your server you will now have the option of switching to German.

Admin frontend translations (Vue i18n)¶

The Vue admin frontend uses vue-i18n with locale JSON files in frontend/src/locales. Currently frontend/src/i18n.ts loads frontend/src/locales/en.json.

To add or update admin UI translations:

Edit translation keys in frontend/src/locales/en.json.
If adding another locale, create a new JSON file in frontend/src/locales.
Register that locale in frontend/src/i18n.ts by importing the file and adding it to messages.
Run frontend checks/build:

$ cd frontend
$ pnpm type-check
$ pnpm build

Update Cookiecutters¶

In case changes are needed for the cookiecutters, you may want to test them on an unreleased version of Atramhasis. You can test them on a specific branch by running the following commands:

# fe you are working on the branch feature/876_cookiecutters and you want to test the demo cookiecutter
$ cookiecutter gh:OnroerendErfgoed/atramhasis --directory cookiecutters/demo --checkout feature/876_cookiecutters
$ cd atramhasis_demo  # or whatever you named the root_folder of your project
$ pip install "atramhasis @ git+ssh://git@github.com/OnroerendErfgoed/atramhasis.git@feature/876_cookiecutters"
$ pip install -e .[dev]
$ alembic upgrade head
$ initialize_atramhasis_db development.ini  # (for demo only)
$ pserve development.ini

Running a Linked Data Fragments server¶

If you want to add a Linked Data Fragments server, Atramhasis makes it easy for you. First you need to decide if you want to run the server with HDT files. If not, you can always use raw Turtle files, but be aware that the HDT files offer much better performance.

If you want to use HDT files, please install hdt-cpp. Be aware that you might have to download the source files and compile them yourself. Once you have done so, add the rdf2hdt command to your development.ini file. Supposing you installed it in /opt/hdt-cpp/hdt-lib/tools/rdf2hdt:

# Location of rdf2hdt executable
atramhasis.rdf2hdt = /opt/hdt-cpp/hdt-lib/tools/rdf2hdt

Now, whenever Atramhasis creates rdf dumps it will also create HDT files. If you do not have rdf2hdt installed, you will still have Turtle datadumps that can be used by the LDF-server.

$ dump_rdf development.ini

Now you’re ready to generate the configuration for the LDF server. Out of the box this file will be generated in the same directory your development.ini is located in, but you can override this in your ini file by setting atramhasis.ldf.config_location or you can pass this on the command line

# Generate config
$ generate_ldf_config development.ini
# Generate config and override config_location
$ generate_ldf_config development.ini -l /opt/my/ldf/server

Now you’re ready to run your LDF server. First we need to install it. It requires Node.js 4.0 or higher and should run on OSX and Linux. Please refer to the LDF server documentation for troubleshooting.

# Install ldf-server
$ [sudo] pnpm add -g @ldf/server
# Run ldf-server
$ ldf-server ldf_server_config.json

Now you have an LDF server running at http://localhost:3000. Browse there and have fun!

When deploying Atramhasis with an LDF server in production, we recommend runnig both behind eg. nginx. In case you want to do this, you might run Atramhasis on port 6543 and LDF server on port 3000, but serve both through nginx. You can easily do this by setting the atramhasis.ldf.baseurl in your ini file. Suppose you want to server both on the host demo.atramhasis.org with Atramhasis as the root of your domain and the LDF server at /ldf. In this case, set atramhasis.ldf.baseurl to https://demo.atramhasis.org/ldf.

Contributing¶

Atramhasis is being developed as open source software by the Flanders Heritage Agency. All development is done on the agency’s Github page for Atramhasis.

Since we place a lot of importance of code quality, we expect to have a good amount of code coverage present and run frequent unit tests. All commits and pull requests will be tested with Github Workflow Actions tests. We monitor code coverage on all pull requests using Github Actions workflows and orgoro/coverage.

Locally you can run unit tests by using pytest.

# No coverage
$ py.test
# Coverage
$ py.test --cov atramhasis --cov-report term-missing
# Only run a subset of the tests
$ py.test atramhasis/tests/test_views.py

Every pull request will be run through Github Workflow Actions tests. When providing a pull request, please run the unit tests first and make sure they all pass. Please provide new unit tests to maintain 100% coverage. If you send us a pull request and this build doesn’t function, please correct the issue at hand or let us know why it’s not working.

Distribution¶

To build a distribution for your project, you can use the hatch build command. This command will generate the necessary distribution archives, such as wheels and source distributions.

In addition to building the Python distribution, the hatch build command will also compile the JavaScript code located in the static folder. This ensures that all static assets are properly built and included in the distribution package.

$ pip install hatch
$ hatch build

Alternatively, you can specify your build as a wheel or as a source distribution (sdist) using the -t or –type parameter.

$ hatch build -t wheel
$ hatch build -t sdist