Docker Manager

This project was started to take over the Ventiv Technology Environment Manager application. However, this one will be more generic, and will support ANY Dockerized project. It is heavily inspired by Fig, but is geared towards a multi-server environment. There are also several other configuration options (such as labelling ports / volumes) that enable certain things like URL derivation.

Running

In order to run this project, you'll need some configuration, see the Configuring section below. Generally, you'll want to simplify this process by storing the configuration in a VCS repository. That being said, the general process for running is to do the following:

1. Clone this repository
2. cd DockerManager
3. Prepare your configuration, generally in the 'config' directory. There is a sample config located in sample-config.
4. When running DockerManagerApplication.groovy, be sure to add your authentication from a spring profile. This will auto-load any properties from config/application-.yml. E.g. --spring.profiles.active=ldap will load config/application-ldap.yml Please Note, Default security is to accept ANY user / password.

Docker Configuration Model

Docker Manager is configured to adhere to a 4 step hierarchy to organize your Environments and Applications. From top to bottom, they are:

1. Tier - Intended to be a top level organization, often by network segment. Examples are typically Development, Quality, UAT, Production this is a nice feature, since one deployment of DockerManager may not be able to communicate (network wise) to different tiers. This will allow you to deploy several DockerManager instances for each network segment.
2. Environment - Intended to be a grouping of Applications that are related to a given environment. This allows you to have one Tier manage several different disassociated applications. This is typically helpful if you combine tiers by logical area. For example Development A, Development B, and Quality B may all be environments under the development tier.
3. Application - A grouping of services that is intended to run together to form a single application. This will generally have one single URL endpoint, and there are features supporting as such. An application definition simply defines what services, and how many of those services should be running. DockerManger figures out the rest.
4. Service / Service Instance - The lowest level of configuration, representing a single running service. This is equivalent to a docker container running on some server.

For illustration / testing purposes, a sample configuration has been included. This sample only contains one server (boot2docker) so it is not what a typical production instance would look like, but it's good enough for documentation purposes. The structure is as follows:

• Tier: localhost
  • Environment: development (Development Testing)
    • Application: activiti (Activiti)
      • Service: activiti (Activiti)
      • Service: mysql (MySQL Database)

For the best in example and documentation, these Yaml files have been carefully constructed to illustrate certain features. These files also have many of the lines commented so you can tell what they are for.

Workflows (with Activiti)

See [Workflow Documentation] (https://github.com/Ventiv-Technology/DockerManager/blob/master/docs/Workflow.md)

Configuring

Configuration is split up into two different types of files, the Service Definition file (services.yml) and the Environment Definition file (environment-name.yml). The first (Service Definition) contains all the information about a Docker Image that is intended to eventually become a Service instance (at deploy time). The second (Environment Definition) contains all the information about different servers and the applications. All three pieces of information come together to actually deploy / run an application.

There is a 'sample-config' directory to illustrate a starter configuration and some of the features of Docker Manager. This configuration only deals with one host (boot2docker) so it can be a runnable sample. If you want to run with this, rename 'sample-config' to 'config', and be sure that you have 'boot2docker' in your hosts file (typically: 192.168.59.103 boot2docker).

Service Definition File

As stated above, this file describes Docker Images so Docker Manager knows how to work with a given image. It is also where you can state additional information, like how to build a Docker Image, should one not exist yet. Each service in this file is mapped to the Groovy Bean 'org.ventiv.docker.manager.model.configuration.ServiceConfiguration'. Please look there for further documentation, or in the sample-config/env-config/services.yml file for examples. Here is a sample of a simple configuration:

services:
  - name: rabbit
    description: RabbitMQ
    image: rabbitmq:3.4.4-management
    url: http://${server}:${port.http}
    containerPorts:
      - type: http
        port: 15672
      - type: amqp
        port: 5672
    environment:
      RABBITMQ_NODENAME: rabbit1

  - name: docker_manager
    description: Docker Manager
    image: ventivtech/docker_manager
    url: http://${server}:${port.http}/
    containerPorts:
      - type: http
        port: 8080

This example describes TWO services, one named 'rabbit', that is pinned to a particular Docker Image and Version (rabbitmq:3.4.4-management). It also describes the ports that will be exposed later, and pins a type to them. This type is important, since it's how Docker Manager will do the port mapping later. That being said, you should never have two ports with the same type here. The nice thing here is you can use these types later to fill in variables in URL's and Environment Variables...thus easily tying services together, and avoiding linking containers together across servers.

The other service described here is for this Application, Docker Manager. This is here to illustrate that the image is NOT pinned to a particular version, and will then query the DockerHub registry for all available versions.

Version Selection

There are several different ways the UI can present the list of versions allowed for a given application. There is some logic to bubble up the version selection logic from the Service to the Application, but that will be described later. For any given Service, the following are valid ways to determine which versions are allowed:

• Full Docker Tag (with version). Example: rabbitmq:3.4.4-management
  • This will only allow that given version to be deployed. The UI will NOT use this service to pick a version to deploy.
• Docker Tag with no version provided. Example: ventivtech/docker_manager
  • This will query the registry associated with the image to determine which versions are available. The UI will present this to the user.
• Supply a build configuration
  • If there is a versionSelection clause under build, it will run that module to get the list of versions. This is useful when the docker image does NOT exist in the registry yet, but some sort of build exists somewhere. The example used by Ventiv is that we use Jenkins to build and keep ALL WAR artifacts in a repository. We then use the versionSelection to give a list of the WAR's that are in there, and use Docker Manager to build the image.
  • If there is no versionSelection clause, the registry will be queried for available images, and the UI will present a new option to the user 'New Build'. This will give the option of deploying a build that has already been created, or kick off a new one. Helpful if you are not using CI, but still have a build server.

Build

Docker Manager will allow you to build a new image, should it not exist already. A good example of this has been included in the sample-config location. Please pay attention to the Activiti service, and how it builds. First, it utilizes the versionSelection clause as stated above. This has been configured to query the GitHub API and pull out the tags. Next, it is configured with a build stage of 'DockerBuild'. This simply kicks off a build in the org.ventiv.docker.manager.build.DockerBuild class, and detailed information can be seen there. Very simply, it treats the specified Dockerfile as a template so you can inject configuration from DockerManager. For example, if you open up sample-config/dockerfiles/activiti/Dockerfile you can see that it uses #{buildContext.requestedBuildVersion} to set into an Environment Variable. This environment variable is then used in the curl statement to download the ZIP release that has been specified from the dropdown in the UI.

You can have as many build stages as needed, but typically a single Dockerfile can do it all. Each build stage is designed to allow for variables to be placed in the buildContext for use in all subsequent stages. Also, there are certain variables that are always allowed:

• userAuthentication - Authentication Object from Spring
• buildingVersion - Resolved building version, after it's been determined. Example, Jenkins build number, after the build has started
• requestedBuildVersion - User specified version requested to be built, from the UI.
• outputDockerImage - Name of the image that has been output from the job
• extraParameters - This is a Map of variables specific to a Build Stage

For more details on specific build stages, see the Markdown files in docs/build.

Container Ports

This is the section where you will describe what ports a particular container may expose. Often, these are described by EXPOSE statements in the Dockerfile, but this is not descriptive enough for Docker Manager. We also must attach a type to the port here, so that we can pair it up later, as well as use it in variables. These variables are very helpful if you need to set container Environment Variables that are resolved with information from another container's ports. It is also important to note that the ports listed here DO NOT have to be described in the Dockerfile, as you can still expose any ports.

Based on the container port type, if there is no URL specified for this particular service, it will be auto derived to be: ${port.getType()}://${server}:${port.http}

Container Volumes

This is the section where you describe what volumes a particular container may expose. Often, these are described by VOLUME statements in the Docker file, but this allows us to give it a type so we may tie that in at container creation time. This is very similar to how ports are done, but there is no automatic behavior based on special types. Also, like above, you can expose any volume in a container, it does not have to be described in the Dockerfile.

Environment Variables

Here, you can describe any environment variables that should be set when the container is created. Since this is the service definition file, you should only put environment variables here that will be used for services in ALL containers. You will see later that these environment variables may be overridden at an Application level. Examples in the example config include setting the root password of Couch / MySql, as here we want the same one no matter where the instance is created.

These environment variables have certain runtime variables exposed to them when the container is created, like ports / servers from other services in the application they get created from. The variables that are exposed are as follows:

• application: Instance of the org.ventiv.docker.manager.model.ApplicationDetails object. Example: ${application.tierName}
• instance: Instance of the org.ventiv.docker.manager.model.ServiceInstance object. Example: ${instance.serviceDescription}
• serviceInstances: All other service instance objects for this application. A map, by service name. NOTE: If there is more than one service of a given type, the last one will survive in this map.
  • server: Server name for that this instance is running under. Example: ${serviceInstances.mysql.server}
  • port: Ports mapped by type. Example: ${serviceInstances.mysql.port.mysql} or ${serviceInstances.couch.port.http}

Additional Metrics

Sometimes the built in statistics for a given service are not enough. This is where additional metrics comes into play. In the example configuration, the Rabbit service has been configured with additional metrics. The metrics simply hit the REST endpoint /api/overview of the server, and pass that information back to the UI. The UI will then ask for the HTML template as it's configured in the ui section.

The UI is divided up into two sections: Button and Details. The button will display inline in the Environment view, next to the button you would click on for service instance details. This button can be configured to show any data you want, such as "Number of Messages Queued" in Rabbit. If you've sepecified a 'Template', then the html may be inline in the YAML configuration.
This is generally helpful for the button, since the HTML is minimal. However, for the details, the HTML is typically much larger so you would want to specify it in a 'Partial'.

The HTML here is integrated fully into AngularJS and the controllers of the full application. This allows you to access several variables from within your html.

• Button
  • environment: Environment Details Object
  • application: Application Details Object
  • serviceInstance: Service Instance Object
  • metricName: Text Name of the metric, specified in the configuration
  • metricValue: Value of the metric retrieved by it's component. Example for Rabbit would be the response from /api/overview
• Details
  • serviceInstance: Service Instance Object
  • data: Value of the metric retrieved by it's component. Example for Rabbit would be the response from /api/overview

Environment Description File

In a directory structure described above (under tiers) is a file for each individual environment. This is where we describe the environment's servers and applications. Furthermore, you will see an application's service requirements, and further configuration of them (like environment variables).

Server Configuration

The first section of an environment configuration file is describing the servers (or hosts) that may be part of it. These must include fully qualified host names, and these hosts MUST be running the Docker daemon, otherwise they will be discarded on application startup. For full documentation, see comments in org.ventiv.docker.manager.model.configuration.ServerConfiguration. The most important section in here is the eligibleServices portion, which describes what services are allowed to run on this host.

Each eligible service will have it's ports described. It's important when you're describing the ports to use either the 'port' or 'ports' section, but not both. The rule of thumb is that if you want to expose multiple services, use the ports...and if you want to expose one service, use port. For example, if I want to allow 'Docker Manager' to run just once on this server, I would do something like this:

- type: docker_manager
  portMappings:
  - type: http
    port: 8080

However, if I want to allow 'Docker Manager' to run 5 times on this server (on ports 8080, 8081, 8082, 8083, 9000), I might do the following:

- type: docker_manager
  portMappings:
  - type: http
    port: 8080-8083,9000

NOTE: The port listed here are the ports that will be exposed on the physical host, and tied back to a port in the container. This mapping is done by using the port type.

Application Configuration

The last remaining part of configuration is for the application, where you describe the application and all of it's running services. For full documentation, please see: org.ventiv.docker.manager.model.configuration.ApplicationConfiguration. At the top level there are things like description and name, which are solely used for UI and API purposes. Also there is a section here to describe the URL of this application. You have the opportunity to specify it manually (via the 'url' attribute), or to use one of the service's url (via the 'serviceInstanceUrl' attribute).

Most importantly, you configure the serviceInstance's that will be part of this application, and how many of them you need. For instance, you may say the following if you wanted 5 of a given service to be running:

- type: awesome-server
  count: 5

Docker manager will then attempt to find 5 'slots' for the 'awesome-server' instance that can run on hosts configured in the section above. By default, it will attempt to spread these 5 across physical hosts first, then back down to running on whatever is provided an available. If you want to change this algorithm, do so via the serviceSelectionAlgorithm attribute. The following algorithms are available:

- org.ventiv.docker.manager.service.selection.DistributedServerServiceSelectionAlgorithm: Algorithm to spread across physical
  hosts as much as possible
- org.ventiv.docker.manager.service.selection.NextAvailableServiceSelectionAlgorithm: Simple algorithm to pick the next
  instance that has not yet been allocated.

Also in this section is the volume mapping. Similar to the port mapping, you will specify the type here, and this is how the system ties back to the services configuration file. Again, as with the port mapping, the volumes listed here are the ones that will exist on the docker host, and be tied back to the container via the type. This is a very handy way to ensure the data from a given container sticks around between deployments.

Finally, underneath the service instance configuration section, you have the opportunity to specify environment variables. This is the exact same as documented above, but will allow you to specify environment variables that are specific to a given application. This is the perfect place to tie servers together (e.g. Activiti -> MySql) or to specify what environment this application is running in (e.g. Development).

Properties Management

Sometimes, there is a need to inject properties (application configuration) into a container. For these cases, Docker Manager has the ability to maintain and inject such properties in several ways:

• File (Just Java Properties format right now). Every time a container is started, restarted, or deployed, this file is re-created.
• Environment (Not built yet - use environment)

To accomplish this, properties are read in priority order, with things lower in the following list taking precedence:

• Properties YAML: /env-config/properties/<serviceName>.yml
• Environment Configuration: /env-config/tiers/<tierName>/<environmentName>.yml (Alongside servers / application)
• Application Configuration: /env-config/tiers/<tierName>/<environmentName>.yml (Within the application)
• Service Instance Configuration: /env-config/tiers/<tierName>/<environmentName>.yml (Within the serviceInstances section)

Within all of these files / sections, the properties are designed to be simple, and as close to a Java Properties file as possible. Also, if there is some metadata about a particular property, it may also (optionally) be included. Here are some examples:

- db.url: jdbc:mysql://localhost:3306/sakila?profileSQL=true
- db.driver: com.mysql.jdbc.Driver
- db.password: BnzsGT5+B4XjgJO6kNvcAL/N8PFe+reQE2x/kVQh9d5Hy2BFFZMXXSr4DmBbGYKu3vS4tmhFyMdy7HO31UqI5t754R6B96ygDJqlOgV07djEAEPY0Nb2htEqsdgoEzMtrABK4WeOoXMlQCGHXWYY6VLUIvJpgfX/VH24D2aQ7EKr56KLj4q5KwWqTH2MfwPRaz4e2NsT39t2o2OnB4K+VzxOVvAvrDeUfmYa/A6RzXTD6/UGbFR2JByUcyZsK6PfmpPZ94aXjTVFzK0SBH4cPh8tM/YT2D3Gf4nPHBIHXvCLOQkLnp8wbMRIEdzTp4oNyCsxN6yW84XCmMhoHPnGjw==
  secure: true
- special.property: This is a special property
  comments: This property does really special things in the application
  propertySets: [ all ]
  tiers: [ localhost ]

The following metadata is allowed:

• secure: If true, it's assumed the property contents are encrypted. See Secure Properties below
• comments: Any comments about this particular property. Will render in the final property file, and allows for multi-line comments
• propertySets: List of property sets that this property is valid for. Null implies everything
• tiers: List of tiers this property is valid for. Null implies everything
• environments: List of environment names this property is valid for. Null implies everything
• applications: List of application Id's this property is valid for. Null implies everything

It is important to note that you may use Groovy to fill in any variables that may be necessary. To do so, set the value of the property to contain ${}, and it will be evaluated with Groovy and the following bindings:

• globalProperties: Map - See below
• applicationDetails: ApplicationDetails object for the current application. Contains all of the ServiceInstance objects for this application.
• applicationConfiguration: ApplicationConfiguration object for the current application.
• serviceInstance: The ServiceInstance that is being worked on.

Finally, there is an area for 'global' properties, that can be shared across applications. This is helpful if there is a shared password for many applications, and you still want it to be secure. However, please be warned, that if a user has permissions to view secure properties in ANY environment, they will then be able to get this property for ALL environments, so design security permissions in such a way to prevent loss of secure information. To do this, there is a special file called /env-config/properties/global-properties.yml that looks just like above. You can then use these properties in any other location by referencing them in a Groovy script and using the globalProperties variable like so:

- db.password: ${globalProperties['global.db.password']}
  secure: true

Property Set Configuration

Docker Manager has the full ability to control how these properties are injected into the container. For now, only File method is built out, with more possible in the future. Here is an example of how to configure a property set (in /env-config/services.yml - as part of the Service Configuration):

- properties:
    - setId: all
      method: File
      location: /tmp/test.properties
      overrideServiceName: (Optional) Service Instance Name to use instead of this one

As you can see, this is the Property Set with ID: all, that will include any property that has a propertySets metadata option of all (or none configured). This states that a File should be injected into the container, with the path /tmp/test.properties.

You can have as many Property Set's as you want, lending to the complete ability to configure as many properties files as deemed necessary.

File from Template (Filled w/ Properties)

It is also possible for Docker Manager to place a file within the container that conforms to a Groovy Template. In order to do so, simply configure your property set configuration like so:

- properties:
    - setId: all
      method: File
      location: /tmp/test.xml
      templateLocation: config/templates/test/test.xml

When this template is populated, the following variables are present:

• properties: Map of all properties described above
• applicationDetails: ApplicationDetails Object. Includes all ServiceInstances
• applicationConfiguration: ApplicationConfiguration Object
• serviceInstance: The ServiceInstance object that is being worked with (e.g. where this filled out template will go)

Secure Properties

Part of the design of properties management in Docker Manager is the ability to fully secure properties. In order to accomplish this, some configuration is needed in the Docker Manager application configuration (see sample in application.yml)

keystore:
  location: file:./config/keystore.jks
  storepass: changeit
  alias: DockerManager
  keypassword: changeit

This will look for a Java Keystore in the location specified, and attempt to load the RSA Public / Private key within that keystore. If it does not find the keystore in that location, it will create one, and self-sign a certificate for you. It is recommended that you commit this keystore to VCS, since losing it will render all of your secure properties unusable.

Docker Manager uses the Public Key to encrypt the values, allowing for ANYONE to encrypt a property. However, the Private Key is required to decrypt the property, and requires a special permission (see below).

In order to add a secure property, follow this process:

1. With Docker Manager running, go to the following URL: /api/properties/encrypt?value=<ValueToEncrypt>
2. Take the output from above and put it into the desired property location with the metadata property secure: true

Example:

- db.password: BnzsGT5+B4XjgJO6kNvcAL/N8PFe+reQE2x/kVQh9d5Hy2BFFZMXXSr4DmBbGYKu3vS4tmhFyMdy7HO31UqI5t754R6B96ygDJqlOgV07djEAEPY0Nb2htEqsdgoEzMtrABK4WeOoXMlQCGHXWYY6VLUIvJpgfX/VH24D2aQ7EKr56KLj4q5KwWqTH2MfwPRaz4e2NsT39t2o2OnB4K+VzxOVvAvrDeUfmYa/A6RzXTD6/UGbFR2JByUcyZsK6PfmpPZ94aXjTVFzK0SBH4cPh8tM/YT2D3Gf4nPHBIHXvCLOQkLnp8wbMRIEdzTp4oNyCsxN6yW84XCmMhoHPnGjw==
  secure: true

Properties Permissions

Regarding properties management, there are 3 permissions that can be configured at the application level (in authorization.yml):

• secrets - Does the user / group have permissions to decrypt and view secure properties. NOTE: This is strictly for viewing via REST or a future UI. W/o this permission, you can still get the secure properties injected into the container.
• propertiesRead - Can the user / group read the properties. Same note as with secrets.
• propertiesUpdate - Unused for now. In the future there will be a REST endpoint + UI to maintain the properties, and this permission will enable those functions.