RTV - Reference-Target Validator

A versatile, configurable, extendable tool for reference/target data validation.

---

Table Of Contents

1. Quickstart
   1. Prerequisites
   2. Get the source code
   3. Setup python environment
   4. Installing the package
   5. Running the examples
2. Extending

Quickstart

Prerequisites

1. Git
2. Python

Get the source code

By cloning a repository.

git clone https://gitlab.perfacct.eu/perfacct/naaice/rtv.git

Setup python environment

The end goal is to get a python environment with all the packages listed in requirements.txt. The set of actions/commands to execute may differ depending on your toolset for managing evnironments and python packages.

Here is an example using venv package for virtual environments and pip for package management.

Create a virtual environment:

python -m venv <path/to/environment/directory>

Activate created environment:

source <path/to/environment/directory>/bin/activate

Install reqruirements:

cd <path/to/rtv/source/directory>
pip install -r requirements.txt

After you've finished working withing the environment you would probably want to deactivate it. With venv you can do it by simply running:

deactivate

Installing the package

This step is optional, and also can be an alternative to the previous one (or you can also install the package to the virtual environment), since your package installer will most probably handle all requirements automatically based on pyproject.toml.

The actions/commands sequence may differ depending on your toolset for python package management.

Here is the example using pip:

pip install <path/to/rtv/source/directory>

You can also make an editable installation:

pip install -e <path/to/rtv/source/directory>

Running the examples

This repository holds an examples directory with some configuration files and scripts that you can use to test/explore the RTV's functionality and features. The directory contains nested README file describing specific example scripts and/or configuration files in detail.

First of all you need to set up an environment variable RTV_EXAMPLE_PATH with the absolute path to the example directory, like this:

export RTV_EXAMPLE_PATH=</path/to/rtv/source/directory>/example

From now you should be able to run example scripts and configuration files.

Running example scripts

With a script the approach is as simple as just running:

python <path/to/the/script.py>

Important: If you haven't install rtv as a package into your environment, you would probably need to modify PYTHONPATH by adding RTV's src directory to it, like this:

PYTHONPATH=</path/to/rtv/source/directory>/src python <path/to/the/script.py>

Running example configuration files

With running the RTV's example configuration files you have the following approaches available:

1. Execute main script from source (inplace PYTHONPATH modification in case you do not have rtv installed as a package).

   PYTHONPATH=<path/to/rtv/source/directory>/src \
   python <path/to/rtv/source/directory>/src/rtv/__main__.py \
   <path/to/example/configuration/file>

2. Execute as python module (export PYTHONPATH).

   export PYTHONPATH=<path/to/rtv/source/directory>/src
   python -m rtv <path/to/example/configuration/file> 

3. Execute as installed package.

   rtv <path/to/example/configuration/file>

Further details on specific examples can be found in the example directory's readme.

Troubleshooting

RTVs error output is written into the working directory as files with names like rtv-error-<timestamp>.log. You will also see some info logs and warnings in your stdout during the execution.

Extending

Define the rtv user code directory

export RTV_USER_CODE_PATH=<path/to/your/code/directory>

Create a directory with the python files you need

mkdir $RTV_USER_CODE_PATH
cd $RTV_USER_CODE_PATH
touch <filename>.py

Add code to the py-files and add custom pipeline steps.

Import the needed classes

e.g. Transformer needs at least

from rtv.core.base import BaseEntity
from rtv.interfaces import ITransformer

If some custom parameter are required add:

from pydantic import BaseModel

Minimum smallest custom class that uses params:

class MyTransformer(BaseEntity, ITransformer):
    class Params(BaseModel):    # for custom params
        param1:str = ''         # for custom params

    def transform(self, collection, params = None):
        # custom transformation code using `self.params.param1`
        return transformed_collection

Funding

The development of RTV is funded by the BMFTR Germany in the context of the NAAICE project (GreenHPC grant).