Add pipelines/deploy endpoint

[LucaPerotti]

Description

this PR refactors PipelineManager and _juturna_service modules
adds a pipelines/deploy endpoint

PR type
Select all the labels that apply to this PR.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Code refactoring
• Performance improvement
• Test update
• Build/CI configuration change
• Other (please describe):

Key modifications and changes

uniforms PipelineManager method responses with dedicated pydantic models
introduces a 'models' module with submodules for each scope
thrown exceptions on api calls are now returned with appropriate HTTP code and message
available new endpoint pipelines/deploy to perform creation, warmup and start of a pipeline with a single api call

Affected components

Juturna API Service

[b3by]

I believe having separate classes for different outcomes is the correct approach, and yet, at this stage of the project I'd probably keep it a bit simpler than this? A reasonable approach seems to only have a model for successful requests, and one for failed requests. In this scenario, a properly created pipeline would trigger a successful response where, for instance, the pipeline id and its status would be included in an optional data field, defaulted to an empty object when nothing is provided to the response.

[b3by]

Having the models all under the same /models folder is a good design, as it can act as collecting point for other models too. The catch here is that pydantic is not included in the set of base dependencies, but it rather gets installed when the httpwrapper group is added for the CLI. The CLI itself has a safe importing mechanism that prevents everything to explode when the available commands are scanned.

Because of this reason, I would keep these models still in the cli/commands folder, at least for the moment. I have the feeling the organisation of the commands will be subject to some form of rearrangement in the near future anyway, so is these models need moving again, it will be done then.

[LucaPerotti]

My fault, I missed that pydantic is not a base dependency.
I'll move the/models folder into the cli/commands module.

[b3by]

I might be wrong here, but I would keep the logic of the responses out of the pipeline manager. Assuming that each public method returns an object that can be used by the service to create the response, I'd do that there. This would respect separation of concerns better. I guess, the "optimal" way to do this is with a global exception handler, but I don't think we are anywhere near that.

[LucaPerotti]

I agree in principle that the response logic should not be tied to the PipelineManager class, so that other services are free to use it.
In line with the previous suggestion, I'd wrap the returned object (on creation methods) in the data field of the SuccessfulResponse model.
I'm biased towards the DTO approach so I would still create a dedicated Pydantic model to return instead of a dict, but the tradeoff is some extra boilerplate so I'll leave the choice to you.

To handle the unsuccessful response cases, the most straightforward way I see to approach this is to create custom exceptions and raise those instead of returning unsuccessful responses, then reworking the raise_on_unsuccessful_response util to raise the appropriate HTTPException (called raise_http_exception in the following example)

PipelineManager

def warmup_pipeline(
        self, pipeline_id: str
    ) -> None:
        if pipeline_id not in self._pipelines:
            raise InvalidPipelineIdException()

        if self._pipelines[pipeline_id].status['self'] == PipelineStatus.READY:
            raise PipelineAlreadyWarmedUpException()

        try:
            self._pipelines[pipeline_id].warmup()
        except Exception as e:
            logger.exception('pipeline exception on warmup')
            raise

_juturna_service

@app.post('/pipelines/{pipeline_id}/warmup')
def warmup_pipeline(pipeline_id: str):
    try:
        PipelineManager().warmup_pipeline(pipeline_id)
    except Exception as e:
        raise_http_exception_(e)
    return SuccessfulResponse(
        data=PipelineManager().pipeline_status(pipeline_id)
    )

This would remove the need for the UnsuccessfulResponse model and looks like a cleaner approach to me.

[b3by]

This suffers from the same issue as the models! I would put it in the cli/commands folder somewhere, just to be on the safe side!

[b3by]

Reasons are implemented with a string enum, then codes are included in a dictionary. Maybe things could be kept tidier by using a IntEnum? Something like this:

from enum import IntEnum


class MyCoolReason(IntEnum):
    ALREADY_RUNNING = 409
    INVALID_ID = 400

reason: str = MyCoolReason.ALREADY_RUNNING.name
code: int = MyCoolReason.ALREADY_RUNNING

This will save us some headaches later on if we have to update the status codes!