[FEATURE] Multi service docker support by jakeberesford-palmetto · Pull Request #99 · palmetto/palm-cli

jakeberesford-palmetto · 2023-05-16T15:53:17Z

Pull request checklist

Before submitting your PR, please review the following checklist:

Consider adding a unit test if your PR resolves an issue.
All new and existing tests pass locally (palm test)
Lint (palm lint) has passed locally and any fixes were made for failures
Docs have been reviewed and added / updated if needed (for bug fixes / features)

Breaking changes

Check if this pull request introduces a breaking change

What does this implement/fix? Explain your changes.

Adds support for projects with multiple services in docker-compose.
Palm was originally built for very simple applications with a single 'primary' service. This has worked well for our needs, but doesn't allow folks to use palm for more complex projects.

Does this close any currently open issues?

#16

Any other comments?

I also added some new utilities to support this functionality:

choice prompt: easily create a 'choose from list' interface for any CLI command.
environment.get_plugin allows you to get a plugin by name, useful if a plugin provides a method for re-use
docker details: info about the docker-compose file is now available via environment.palm

Housekeeping:

Removed palm scaffold config - this was replaced by palm init long ago and is unused.

Note: This is still a WIP. I need to add tests and docs for all of this. Opening the PR now to get feedback on the functionality from users.

Testing instructions

To test this functionality, you will need a codebase with multiple docker services in your compose file.

Pull this branch and install palm from source with python3 -m pip install .
palm init to set your repo up for using palm if you haven't already done so.
Add the multi_service plugin to your repo's .palm/config.yaml it should look like this:

plugins:
  - multi_service

palm to see available commands, you should see a section for 'Multi Service' with up/down/logs/exec - try them out!
Try writing a custom palm command to execute something useful in one (or more) of your containers
Try to break it!

Don't forget to re-install palm from pypi after you're done testing! pip install -U palm

Where has this been tested?

Operating System: MacOS

Platform: Python 3.9

palm/docker_details.py

norton120

few nitts but great addition!

norton120 · 2023-09-27T15:06:30Z

docs/source/commands.rst

 own command, ensure you use the ``@click.pass_obj`` decorator for your command,
 then use ``environment.run_in_docker(command)``.

+Note that in multi-service applications, ``run_in_docker`` will run proxy your command


future iteration, should we have it say things like that with maybe an --plan flag? that instead of running the command, it describes in detail what the command would do?

norton120 · 2023-09-27T15:07:39Z

palm/cli.py

                    plugin_name = self.plugin_manager.plugin_command_dict.get(
                        subcommand
                    )
+                plugin_name = plugin_name.replace("_", " ")


❤️ this has bit me before trying to be smarter than the service

norton120 · 2023-09-27T15:09:26Z

palm/docker_details.py

+        Returns:
+            str: The name of the running service
+        """
+        cmd = 'docker ps -a --filter name={service} --format {{{{.Names}}}}'.format(


eventually it'd be nice to move to dockerpy, but at the same time it looks like Docker is (terrifyingly enough) slowing down support for dockerpy - so shelling may stay the safest option long-run

norton120 · 2023-09-27T15:10:42Z

palm/environment.py

+            click.secho(
+                "This command is only available in multi-service repos", fg="red"
+            )
+            raise Exception("exec_in_docker only available in multi-service repos")


nitt - should it link you to the docs on how to enable multiservice? lots of editors like vscode will give you an easy cmd+ to open the link from the response in the browser

norton120 · 2023-09-27T15:11:23Z

palm/environment.py

            is not found or does not have a config
        """
-        plugin = self.plugin_manager.plugins.get(plugin_name)
+        plugin = self.get_plugin(plugin_name)


way nicer interface here 💥

norton120 · 2023-09-27T15:11:59Z

palm/palm_exceptions.py

+
+    def __init__(
+        self,
+        message: str = "No running services found, start your services with `palm up` and try again",


norton120 · 2023-09-27T15:14:24Z

palm/plugins/multi_service/commands/cmd_down.py

+def cli(environment, flags: tuple):
+    """Bring up all services"""
+    # This sucks. Don't commit this.
+    flags = [f"-{flag}" if len(flag) == 1 else f"--{flag}" for flag in flags]


can you just do [f"--{flag}" for flag in flags] and let it join the single element?

norton120 · 2023-09-27T15:16:56Z

palm/plugins/multi_service/commands/cmd_up.py

+
+@click.command("up")
+@click.option(
+    "--no-detach", "-nd", is_flag=True, help="Run containers in the background."


should the help be "don't run containers in the background?"

norton120 · 2023-09-27T15:17:25Z

palm/prompts/choice.py

+from collections import OrderedDict
+
+
+def choice_prompt(prompt: str, options: List[str]) -> str:


this is cool

…m init` which is much easier to remember.

- Add new multi-service plugin - Add detection of multi-service docker setup - Add container selection to run_in_docker

Adds new UI to allow users to select from a list

This will be used to get a list of services to support multi-service projects, but may be extended in the future to provide additional information about the docker configuration

- default to showing last 50 log lines

Allows users to exec arbitrary commands in any running container.

Create shared abstraction for choice prompt, this is useful in many contexts, not only environment.

…ing.

This could be useful for other features and provides a consistent way to access the compose yml

…want. - Get a list of service names - Check to make sure some services are running - Make exec_in_docker work as expected

Auto installing could lead to regressions for projects already using palm some projects may have secondary services that most users do not interact with directly forcing them into multi-service makes the UX of palm weird

…e apps

…ould just use the docker compose cli

jakeberesford-palmetto requested a review from a user May 16, 2023 15:53

jakeberesford-palmetto added no merge and removed no merge labels May 16, 2023

jakeberesford-palmetto commented May 19, 2023

View reviewed changes

palm/docker_details.py Outdated Show resolved Hide resolved

jakeberesford-palmetto requested a review from mariahjrogers May 26, 2023 13:13

jakeberesford-palmetto changed the title ~~[WIP] [FEATURE] Multi service docker support~~ [FEATURE] Multi service docker support May 26, 2023

jakeberesford-palmetto requested a review from norton120 May 31, 2023 19:19

norton120 approved these changes Sep 27, 2023

View reviewed changes

jakeberesford-palmetto added 22 commits November 9, 2023 15:16

Remove palm scaffold config. The same thing can be achieved with `pal…

7b7901b

…m init` which is much easier to remember.

feat: Add support for docker-compose files with multiple services

dbe97e8

- Add new multi-service plugin - Add detection of multi-service docker setup - Add container selection to run_in_docker

Use modern docker compose

248abfc

Rename plugin and install plugin config

a69e58c

Add new environment method for choice prompts

368b2e3

Adds new UI to allow users to select from a list

Add DockerDetails class to pull info from docker into palm config

9f5f94c

This will be used to get a list of services to support multi-service projects, but may be extended in the future to provide additional information about the docker configuration

Implement logs command wrapper for multi-service apps

0e4670e

Add sensible defaults for log command

eb8e6ac

- default to showing last 50 log lines

Add exec command for multi-service apps

b71abc6

Allows users to exec arbitrary commands in any running container.

Add docs, types, and multi-service detection to docker details

ab25c36

Auto-configure multi-service plugin based on docker-compose file.

4836d61

Adds new exception for no running services

9800ac8

Update naming and verbiage

d46aef8

Refactor to make choosing service part of the plugin

78fb371

Create shared abstraction for choice prompt, this is useful in many contexts, not only environment.

Add echo to all multi service commands to make it clear what's happen…

ad55710

…ing.

Update build for multi-service support

31b6816

Make operation of choice prompt more explicit

774da3b

Make DockerDetails available through palm config

a9c0fa5

This could be useful for other features and provides a consistent way to access the compose yml

Make get running service name a utility of the plugin

7fb0f37

Better default message for NoRunningServicesError

3b8e5ea

Default to deteched mode for compose up

1d03c83

Remove dumb container name stuff. Service names are what we actually …

9d18d7d

…want. - Get a list of service names - Check to make sure some services are running - Make exec_in_docker work as expected

jakeberesford-palmetto added 19 commits November 9, 2023 15:20

Just use exec_in_docker here, no need for duplicate logic

f08396a

Do not auto install multi_service plugin.

d1644e6

Auto installing could lead to regressions for projects already using palm some projects may have secondary services that most users do not interact with directly forcing them into multi-service makes the UX of palm weird

Linting

5b2b477

Fix bad list -> List type

6e465b5

Remove test for palm scaffold config - this command was removed.

2ff3ef5

Add test for multi-word plugin formatting

8ce83f1

Add tests around exec_in_docker environment method

b0477b2

Add test around run_in_docker calling exec_in_docker for multi-servic…

086173c

…e apps

test get_plugin

d88cd0a

Test choice prompt

4ed9639

test palm config is_multi_service

fc60cc7

Simplify for testing in python v <3.9

fc3ef58

Fix import error in conftest

87488d4

Refactor for test-ability

741b52e

Linting

06f0a8c

Add minimal docs around multi-service support

7a08b8d

fix: Support yaml andyml extensions for docker-compose

0fb35b7

Linting

9dd8f38

Resolve that conflict properly.

0462082

jakeberesford-palmetto force-pushed the feature/multi-service-docker-support branch from 5bb0b99 to 0462082 Compare November 9, 2023 20:28

jakeberesford-palmetto requested a review from a team as a code owner November 9, 2023 20:28

jakeberesford-palmetto requested review from a user, david-jy-kim, emilypastewka and murphymoulds and removed request for a team November 9, 2023 20:28

jakeberesford-palmetto added 4 commits November 9, 2023 15:30

More fixing dumb conflict. Old branches are the worst.

80239c4

linting

2591c86

Correct help text for no-detatch

c8e0b61

Remove flags from down command. Anyone who understands those flags sh…

4f05da4

…ould just use the docker compose cli

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FEATURE] Multi service docker support#99

[FEATURE] Multi service docker support#99
jakeberesford-palmetto wants to merge 45 commits intopalmetto:developfrom
jakeberesford-palmetto:feature/multi-service-docker-support

jakeberesford-palmetto commented May 16, 2023 •

edited

Loading

Uh oh!

Uh oh!

norton120 left a comment

Uh oh!

norton120 Sep 27, 2023

Uh oh!

norton120 Sep 27, 2023

Uh oh!

norton120 Sep 27, 2023

Uh oh!

norton120 Sep 27, 2023

Uh oh!

norton120 Sep 27, 2023

Uh oh!

norton120 Sep 27, 2023

Uh oh!

norton120 Sep 27, 2023

Uh oh!

norton120 Sep 27, 2023

Uh oh!

norton120 Sep 27, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		from collections import OrderedDict


		def choice_prompt(prompt: str, options: List[str]) -> str:

Conversation

jakeberesford-palmetto commented May 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull request checklist

Breaking changes

What does this implement/fix? Explain your changes.

Does this close any currently open issues?

Any other comments?

Testing instructions

Where has this been tested?

Uh oh!

Uh oh!

norton120 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jakeberesford-palmetto commented May 16, 2023 •

edited

Loading