Supporting non-fpm packaging methods

[travisgroth]

Hi,

I've been looking around at using docker for packaging - its a great way to provide a clean, disposable build environment - in a few different use cases. Berth is fantastic when you don't already have a spec or debian directory, but I was thinking that its current design might be tweaked to allow for alternate packaging methods beyond fpm. A section under 'package' could potentially specify the type of package build you're doing (say spec:), specify some parameters such as spec file location, macros, pre-build steps and environment variables.

There's a few projects out there that sort of cover this use case but they seem a bit more complex (eg, they require a dockerfile that just builds a package). Berth does a great job of wrapping the volume and container management already so I feel like this is a space it could excel at. Thoughts?

[Tenzer]

Hello Travis, and thank you for the feedback.

As you can tell, Berth has been made with FPM in mind from the beginning, although we only have used it for DEB packages so far. The benefit of FPM is that there's only one tool to tailor Berth towards, and it basically behaves the same regardless of which type of package you are trying to build.

Adding support for another type of packaging tool would likely make the Berth code base quite bigger. As it is right now, you could potentially just use the build step, skip the packaging step, and do your own packaging inside of the build step with the shell script you can run. Alternately, you could split your package building into two different Berth jobs, again only making use of the build step, where the first Berth job would create the compiled/prepared files, and the second job then would handle whatever type of packaging you would like.

You do of course loose some convenience with having to maintain the build configuration in more of a shell script rather than a YAML file, but comparing it against the work it would need to add support for other packaging methods, it probably is the more reasonable choice (at least how I see it :)).

That being said, I'm not too familiar with the spec packaging format, it's requirements and how well it would fit into a YAML file. Could you perhaps provide some example of that? Then it would be easier to give some feedback on how much work it would be to add support for.

[travisgroth]

Ah I may not have made my intention clear. FPM is for when you don't already have a spec file (what RPMs are built from) but some projects have one and may benefit from them when the build process is complex. What I'm suggesting is rather than only knowing how to call fpm, call to the right build tool as specified by the user - so rpmbuild, for instance. berth wouldn't need to understand the format of the build files as it wouldn't be generating the file. It would just need to accept some common options similar to how it does so for fpm.

Just dropping the commands into the 'build' section and making the package section optional (it presently is required without at least a small change) is an option but as I was thinking about it I saw no reason one couldn't treat the various build tools as pluggable and fpm is just one implementation. It would allow developers to use berth for any type of project and reap the workflow benefits even when fpm doesn't make sense.

A spitball version of what the yaml might look like:

environment:
  VERSION: 1.0.0
  SPECFILE: myproject.spec


package:
  image: myrepo/centos-rpmbuild:7
  volumes:
    .: /data

  rpmbuild:
    spec: $SPECFILE
    srpm: false
    buildprep: true
    prebuild:
        #!/bin/bash
        echo "Do some extra non standard stuff before running rpmbuild"
        touch /tmp/foo
    with:
        - libzot
        - libfoo
    without:
        - libtoast
        - libspam
    define:
        - buildver: $VERSION
        - buildrelease: 1

This would run something like this:

echo "Do some extra non standard stuff before running rpmbuild"
touch /tmp/foo
yum-buildprep myproject.spec
rpmbuild -bb  --with=libzot --with=libfoo --without=libtoast --without=libspam --define "bulidver $VERSION" --define "buildrelease 1" myproject.spec

Unless you're completely opposed to adding other packaging commands beyond fpm I'm happy to take a stab at a PR or two. I mostly work with rpm packaging so I'd tackle that first.

[Tenzer]

Thinking about it, having a plugin like structure for the packaging part would probably not be too much work. Berth could then come with two plugins by default, an FPM version which basically is just a copy of the current functionality, and probably the default, and a shell plugin which works more like the current build part.

I would have to think about how the integration between Berth and the packaging plugin would work, I am thinking something like:

1. Berth imports a Python module provided in the YAML file.
2. The Python module has a predefined variable which tells Berth the default Docker image to use for this plugin, but it can be overridden in the YAML file as it is right now.
3. The Python module also has a method with a set name, which is called by Berth with the package part of the YAML file provided as an argument.
4. The method return either a command to run inside the container or a script like the one provided in the build section to run in the container, and any environment variables that has to be provided besides those in the environment section.
5. Berth execute the container and the result should be a package file.

There could potentially also be a method in the plugin which can be used to verify the configuration has the necessary settings, which are needed in order for the packaging step to work. This part could however be optional.

Does that sound reasonable to you?

[travisgroth]

I think so. Its a bit more abstracted than I was thinking of but it seems perfectly reasonable. I was really just envisioning a set of modules that ship with berth and a generic call to whichever is specified by the yaml but more flexibility to call any arbitrary tool doesn't seem like a bad thing if the interface is light enough.

I'm already picturing using berth to build an rpm of the berth-rpmbuild plugin :-)

[Tenzer]

I'll try to see if I can find some time to split the current FPM functionality into a separate Python module within the next couple of weeks. I won't promise too much though, as I'll be relocating to another country and starting on a new job in the same period, but it could be a good task to work on when I have some down time :)