Packaging

Packaging

In order to deploy archives on Data Fabric, it is easier to package them. For JVM based languages, it is easy as a jar can be easily created. In order to help packaging, we provide you some examples for other languages.

Python / PySpark

With a project structure like the following:

.
+-- build.gradle
+-- settings.gradle
+-- gradle.properties
+-- src
|   +-- main
|   |   +-- python
|   |   |   +-- __main__.py
|   |   |   +-- requirements.txt
|   |   |   +-- ...

You should add a task in your build.gradle

def zipName = "${project.name}-${version}.zip"

task zip(type: Zip) {
    from 'src/main/python'
    include ['**']
    archiveName "$projec.buildDir/python/$zipName
}

createJob.dependsOn zip
updateJob.dependsOn zip

The generated zip is compatible with the capsule and you are not bind to rewrite the command template.

R

With a project structure like the following:

.
+-- build.gradle
+-- settings.gradle
+-- gradle.properties
+-- src
|   +-- main
|   |   +-- r
|   |   |   +-- my_script.r
|   |   |   +-- lib
|   |   |   |   +-- ...
|   |   |   +-- ...

You should add a task in your build.gradle

def zipName = "${project.name}-${version}.zip"

task zip(type: Zip) {
    from 'src/main/r'
    include ['**']
    archiveName "$projec.buildDir/r/$zipName
}

createJob.dependsOn zip
updateJob.dependsOn zip

On the job configuration, you must change the job's template property to something like the following:

saagie {
    server {...}
    jobs {[{
        ...
        template = 'unzip {file} && Rscript my_script.r arg1 arg2'
        ...
    }]}
}