EVL - Execution Verification Lane

The EVL guide

• EVL - Execution Verification Lane
  • The EVL guide
  • What is EVL?
    • High level overview
  • Get it running
    • Get the source code
    • Setup the python environment
    • Install as package
    • Setup file structure
    • Add config file
    • Run command
  • Try the example
  • Further explanations
    • Environment variables
      • Description of the variables
      • Accessing environmental variables
    • Validation with RTV

What is EVL?

High level overview

The tool is meant to simplify/automate some routine sequences for certain types of applications. Let's say that you have a project where you run some applications that produce certain output data. After each run of the execution the batch of output data needs to be verified/validated against previous output or some static reference data.

EVL abstracts/decomposes the process into the following steps:

1. Download - get required project data executing the applications (e.g. from git repository or local disk).
2. Install - prepare (set up) the application(s) and/or the data for execution (e.g. cmake, unzip, etc.).
3. Execute - execute the applications (produce output).
4. Verify - validate the application(s) output against some references.

The main task of this app is to perform the result verification/validation. But it could also perform benchmarks. It is also possible to ommit/skip/mock some steps from the above list if it is needed and provides benefits to your specific project/setup/scenario.

Get it running

Get the source code

By cloning the repository.

git clone https://gitlab.perfacct.eu/perfacct/naaice/evl.git

Setup the python environment

The end goal is to get the python environment containing all the packages listed in requirements.txt. The set of commands can change depending on what tools you use for managing your python environments and packages.

For example if you are using pip you could simply run:

pip install -r requirements.txt

inside the EVL's root directory.

Here are some useful articles on the topic:

1. Python Package Managers
2. Environment Management Best Practices

Install as package

EVL has a pyproject.toml file and supports installation as a package. This can be a good alternative to the previous step, if you do not mean to modify, or debug the EVL you should probably just install the package, since it provides more convenient and simple way of execution EVL setups just with:

evl <CFG_PATH>

or

python -m evl <CFG_PATH>

The command to install may differ depending on your python package management toolset.

Here is an example with pip:

Normal:

cd <path/to/evl/source/directory>
pip install .

Editable:

cd <path/to/evl/source/directory>
pip install -e .

Setup file structure

The evl script looking for config files in a directory (<CFG_ROOT>). <CFG_ROOT> must to contain the following directories:

• versions/ - contains config files which describe how to download, install, execute and validate an application.
• scenarios/ - needs to provide the input data.
• scripts/ - can include scripts that are required for execution.

<CFG_ROOT>/
├── versions/
│   ├── <EVL_CFG_FILE>
│   └── ...
├── scenarios/
│   ├── <SCENARIO_DATA>
│   └── ...
└── scripts/
    ├── <SCRIPT_FILES>
    └── ...

Generate this file structure by using the following bash script:

root=<CFG_ROOT>
dirs="versions scenarios scripts"
for dir in dirs; do mkdir -p "$root/$dir"; done

Add config file

Here is a description of an EVL-config-file to be stored in versions/:

download: # Describes the download process.
  method: # Method to download (e.g. git).
  source: # Source for a given method (e.g. https://git.code.de/project/subproject.git).
  hash: # *Optional* - used with method `git` (e.g. 1234abcd).

install: # Describes the installation process.
  method: # Method to install (e.g. bash, script).
  script: # Path to a script (e.g. setup.sh, setup.py).
  arguments: # Arguments for the script as a list or string.

execute: # Describes the execution process.
  method: # Method to execute (e.g. bash, script, app).
  script: # Path to a script or a command (e.g. ./app, setup.sh, setup.py).
  # Method script execute scripts from <CFG_ROOT>/`scripts` by its name.
  arguments: # Arguments for the script as a list or string.
  output: # Describes files as regex or list (regex possible) where the output
  # of the application is located. The program output will be copied into
  # output dir.

verify: # Describes the verification step.
  source: # Path to an RTV configuration file, python or bash script.
  arguments: # Arguments as a list or string, if source is a python script.

Run command

Run the Execution Verification Lane with this command:

PYTHONPATH=<path/to/evl/source/directory>/src \
python src/evl/evl.py `<CFG_ROOT>`

If you have installed EVL as a package:

evl `<CFG_ROOT>`

or

python -m evl `<CFG_ROOT>`

Try the example

An example for the configuration is provided in ./example/config/:

• config file: ./example/config/versions/test.yaml
• scenario / input data: ./example/config/scenarios/ScenarioA/*
• user script: ./example/config/scripts/install.sh

Run the example with e.g.:

PYTHONPATH=<path/to/evl/source/directory>/src \
python src/evl/evl.py example/config/

If you have installed EVL as a package:

evl example/config/

or

python -m evl example/config/

Further explanations

There are some other functionalities provided to act with the EVL. For scripting and configuring, EVL deploys some environmental variables.

Environment variables

There are some environment variables to exchange/persist information between the scripts and/or configuration files and the EVL framework.

Description of the variables

These environment variables are available for scripts and configurations used and are only defined while EVL is running.

Name	Description
EVL_REF_OUTPUT_DIR	Path to a temporary directory, where the reference output is stored.
EVL_RUNNING_ENV	Name of the running environment (generated, e.g. env_0, env_1, ...).
EVL_RUNNING_SCENARIO	Name of the running scenario. Corresponds to the name of the scenario directory.
EVL_RUNNING_VERSION	Name of the running version. Corresponds to the name of the configuration file.
EVL_TEST_INSTALL_DIR	Path to a temporary directory, where the build of the project is stored.
EVL_TEST_OUTPUT_DIR	Path to a temporary directory, where the output is stored.
EVL_TEST_ROOT_DIR	Path to temporary directory used by EVL.
EVL_TEST_SOURCE_DIR	Path to a temporary directory, where the project is stored.
EVL_TEST_WORKING_DIR	Path to a temporary directory, where the code is executed from and input data is stored.
EVL_USER_DIR	Path to configuration directory, given by the user.
EVL_USER_SCENARIO_DIR	Path to scenarios in configuration directory.
EVL_USER_SCRIPT_DIR	Path to scripts in configuration directory.
EVL_USER_VERSION_DIR	Path to versions in configuration directory.

Accessing environmental variables

In the config file for the parameters script and source, the environmental variables will be resolved. But to resolve variables of script, method: app is required. With method: script scripts always starts with ${EVL_TEST_SOURCE_DIR}/scripts.

Summary:

• method: app - resolves environment variable and executes the application
• method: script - to execute bash or python scripts in ${EVL_TEST_SOURCE_DIR}/scripts
• method: bash - to execute bash commands (resolves environment variables)

Notice: The only accepted syntax is ${VAR_NAME}, e.g.:

download:
  method: copy
  source: ${EVL_USER_DIR}/../example.py

execute:
  method: app
  script: ${EVL_TEST_SOURCE_DIR}/example.py

If it will be passed to a bash script, the bash syntax for environment variables could be used, e.g.:

execute:
  arguments:
    - $USE_NP  # or ${USE_NP}
    - matrix_a
    - matrix_b

The same applies to scripts that are executed at runtime:

echo "User configuration located at: $EVL_USER_DIR"  # or ${EVL_USER_DIR}"

Validation with RTV

It is recommended to use RTV for validation. Visit the gitlab page of RTV and follow the instructions to install it. Remove the underscore at the end of the RTV config: np_rtv.yaml_ -> np_rtv.yaml located in evl/example/config/versions/. Just run the example to see if it works.

Funding

The development of EVL is funded by the BMFTR Germany in the context of the NAAICE project (GreenHPC grant).