diff --git a/CHANGELOG.md b/CHANGELOG.md index 52e2da8..a22f49c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Added `description`, `description_kind`, and `deprecated` fields to Schema and Block. This allows resource, data sources, and providers to specify these fields for documentation. - **End-to-End Tests**: - Added comprehensive end-to-end tests against OpenTofu. This ensures that providers, resources, and data sources work correctly against the real OpenTofu CLI. +- **Tutorial**: + - Added a tutorial to the documentation to help new users get started with writing a provider using `tf`. ## 1.1.0 diff --git a/docs/api.rst b/docs/api.rst index 5db4f79..35e9f50 100644 --- a/docs/api.rst +++ b/docs/api.rst @@ -4,22 +4,37 @@ Stable API .. py:module:: tf +:code:`tf`'s API consists of: + +* Low-level protocol classes that library consumers are expected to implement (such as :class:`~tf.iface.Provider`, :class:`~tf.iface.Resource`, and :class:`~tf.iface.DataSource`). +* Mid-level "glue" for schema definitions, such as :class:`~tf.schema.Attribute` and :class:`~tf.schema.Schema`. +* High-level commonly-used types such as :class:`~tf.types.String`, and :class:`~tf.types.List`. +* A handful of utility functions and classes, such as :func:`~tf.runner.run_provider` and :class:`~tf.runner.install_provider`. + +As a library consumer and provider author, your primary interaction will be writing classes that satisfy :class:`~tf.iface.Provider`, :class:`~tf.iface.Resource`, and :class:`~tf.iface.DataSource`. +If your provider has a large number of elements, you will likely want to implement your own higher-level abstractions to stamp out boilerplate class definitions. +Protocols give you the flexibility to do this in a clean way. + +However, it's perfectly fine (if somewhat tedious) to implement each element against it's protocol individually. + Execution Interface =================== The execution interface provides program entrypoints for OpenTofu to run your provider. +You provider should have a `Console Script `_ entrypoint (named :code:`terraform-provider-myprovider`) pointing to your main function. +Your main function should create an instance of your provider and call :func:`~tf.runner.run_provider`. .. autofunction:: tf.runner.run_provider -An installation utility is provided to install your provider. +An installation utility is provided to install your provider into the plugins directory. + +.. warning:: + Running :func:`~tf.runner.install_provider`. is only required if you want to install your provider *in development mode* into your plugin directory. + There are easier ways to test your provider during development, such as setting the :code:`TF_CLI_CONFIG_FILE`. .. autofunction:: tf.runner.install_provider -You are expected to provide a `main.py` file in your provider package with a method -(set as an entrypoint) -that calls -:func:`~tf.runner.run_provider` and :func:`~tf.runner.install_provider`. Provider Interface ================== @@ -77,6 +92,14 @@ It is a dictionary of field names to values. :members: +Config +------ + +*Config* is used instead of State during validation. + +.. autoclass:: tf.iface.Config + :members: + Schemas ============ @@ -124,3 +147,13 @@ Several utility types are also provided: .. autoclass:: tf.types.NormalizedJson :show-inheritance: + +Unknown +------- +:class:`~tf.types.Unknown` is a special `value` that represents a value that is not known at plan time. +Unknown is a fundamental part of TF's design and is used extensively in the planning phase. + +TF makes a distinction between `null` and `unknown`. :code:`tf` represents them with :code:`None` and :code:`tf.types.Unknown` respectively. + +.. autoclass:: tf.types.Unknown + :members: diff --git a/docs/caveats.md b/docs/caveats.md index 880c84c..cef74e9 100644 --- a/docs/caveats.md +++ b/docs/caveats.md @@ -21,15 +21,17 @@ The biggest problem is that Terraform **expects the provider to be a single bina You need to get your entire artifact into one executable file that you can move to someone else's machine. This is VERY at odds with the default Python experience. -To get around this, you can use something like [Nuitka](https://nuitka.net/) to compile your Python code into a single binary with `--onefile` and `--standalone`. -This is very fickle, and you need to consider: +To get around this, you can use something like [Pex](https://docs.pex-tool.org/) to generate a single binary. +While this works for simple (pure Python) packages, you need to consider: * How many OS/Architecture combinations do you need to support? -* How old of a version of GLIBC are you are targeting? You might want to run nuitka in docker container for an older distro. +* How old of a version of GLIBC are you are targeting? Prepare to bake pre-prepared `--complete-platform` configuration sets for every combination you want to support. * Do any of your dependencies have C extensions? Or do dynamic linking? - This might be a dealbreaker -- you can't install these on the fly. + This might be a dealbreaker. -With a lot of project-specific configuration, you can get this working and building a single `main.bin`. But YMMV. +With a lot of project-specific configuration, you can get this working and building a single `terraform-provider-$providername`. But YMMV. + +See [Building a Binary](tips_tricks.html#building-a-binary) for some tips and tricks to get this working. #### Development mode already works, why not use that? diff --git a/docs/index.rst b/docs/index.rst index f000bf0..c183aa0 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -1,12 +1,5 @@ .. mdinclude:: ../README.md -.. warning:: - - There are serious hurdles to taking a Python provider to production. - Read :doc:`caveats` before considering a production use case. - - For development, testing, and proof-of-concept, continue on to :doc:`usage`. - .. toctree:: :maxdepth: 1 :caption: Contents: @@ -15,5 +8,7 @@ elements.md functions.md api.rst + tutorial.rst example.md caveats.md + tips_tricks.rst diff --git a/docs/tips_tricks.rst b/docs/tips_tricks.rst new file mode 100644 index 0000000..876bd71 --- /dev/null +++ b/docs/tips_tricks.rst @@ -0,0 +1,62 @@ +*************** +Tips and Tricks +*************** + +Using a Debugger +================ + +You pre-emptively launch your provider in a long-lived server mode by running ``terraform-provider-$providername --stable --dev`` in a terminal. +This will start the provider, dump a ``export TF_REATTACH_PROVIDERS=...`` command, and wait for connections. + +In another terminal, you can paste and execute the ``export TF_REATTACH_PROVIDERS=...`` command to set up your environment. +Then you can run ``tofu plan`` or ``terraform apply`` as usual. + +Tofu will connect to the already-running provider instead of starting it on-demand. +By launching your provider with ``--stable --dev`` in your debugger, you can set breakpoints and inspect state as needed. + + +Building a Binary +================== + +Ultimately, if you want to distribute your provider you'll need to build a single standalone binary. +The Terraform registry (and all custom provider registries) expect a binary executable, not a Python script. + +There are many ways to build a binary from Python code, but the easiest one to get working is `pex `_. + +Pex essentially bundles up your Python code and all its dependencies into a single file that can be executed as a binary. +When the binary is run, it sets up a virtual environment and runs your code in that environment. + +Another nice feature of pex is that it can generate binaries for multiple platforms (Linux, Windows, MacOS) from a single machine. + +We have had success building provider binaries with Pex and uploading and consuming them from Terraform private registries. +From tofu and the registry's perspective, the pex binary no different than a Go binary built using the official Terraform provider framework. + +The provider upload registry APIs are not standardized across hosting providers, so you'll need to explore bundling and uploading the binaries yourself for your choice of registry. + +Roughly, the steps to build a pex binary are: + +#. Install pex: `pip install pex`, preferably into a separate virtual environment or as a development dependency of your provider project. +#. Dump the dependencies of your provider into a requirements file: + + .. code-block:: shell + + pip freeze > requirements.txt + + Alternatively, you can use a package-manager specific command + such as ``uv export -f requirements.txt > requirements.txt``). + +#. Build the pex binary: + + .. code-block:: shell + + pex -r requirements.txt ./ \ + -o terraform-provider-$providername \ + --scie eager \ + -m providerpackage.main:main + + By using ``--scie eager``, pex will include a Python interpreter in the binary as well as all the dependencies. + This means everything needs to run the provider is contained in the single binary file. + +You will now have a binary executable named ``terraform-provider-$providername`` that you can upload to your registry and use with Terraform. + +At this point you have a minimum viable provider binary, but you will need to think about multiple platforms and architectures and ABIs. diff --git a/docs/tutorial.rst b/docs/tutorial.rst new file mode 100644 index 0000000..4c3a706 --- /dev/null +++ b/docs/tutorial.rst @@ -0,0 +1,342 @@ +******** +Tutorial +******** + +In this tutorial, we'll walk through using ``tf`` to create a simple math provider. + +We'll use ``uv``, a popular package manager. + +Project Setup +============= + +We're going to name our provider Python package ``mathprovider``. We'll create a directory for it right in our home directory -- ``mkdir ~/mathprovider && cd ~/mathprovider``. +From now on, we'll assume you're in that directory. + +First, let's start a package with ``uv``: + +.. code-block:: shell + + $ uv init --package + Initialized project `mathprovider` + +This will give us a couple of files we can work with, namely ``pyproject.toml`` and the ``src/`` directory. + +Let's add ``tf`` as a dependency. + +.. code-block:: shell + + $ uv add tf + Resolved 8 packages in 114ms + Built mathprovider @ file://~/mathprovider + Prepared 1 package in 4ms + Installed 8 packages in 2ms + + cffi==1.17.1 + + cryptography==45.0.6 + + grpcio==1.74.0 + + mathprovider==0.1.0 (from file://~/mathprovider) + + msgpack==1.1.1 + + protobuf==5.29.5 + + pycparser==2.22 + + tf==1.1.0 + + +Finally, we're going to have a main function in our package that needs to be the entrypoint. +We'll create a ``src/mathprovider/main.py`` file and a ``main`` function in it. + +.. code-block:: python + :caption: src/mathprovider/main.py + + import sys + from tf.runner import run_provider + + def main(): + provider = None + run_provider(provider, sys.argv) + + +Let's create a console script in ``pyproject.toml`` that points to our main function. +These need to have a specific name (``terraform-provider-``). +We add this to a new ``[project.scripts]`` section of ``pyproject.toml``. + +.. code-block:: toml + :caption: pyproject.toml + + ... + + [project.scripts] + terraform-provider-math = "mathprovider.main:main" + +Finally, we have ``uv`` create this for us with ``uv sync``. + +You will now find a ``.venv/`` directory in your project along with a ``.venv/bin/terraform-provider-math`` script. + +If we run it, we'll find a bunch of garbage output and our program hanging: + +.. code-block:: shell + + $ uv run terraform-provider-math + 1|6|unix|/tmp/tmp5hf5fmoy/py-tf-plugin.sock|grpc|XXX... + (hang) + +What's going on here? Our provider entrypoint is speaking the Go Plugin Protocol, and it's waiting for Terraform to connect to it. + +Let's ``ctrl-c`` out of it. +We now have the basic scaffolding of a provider, but it doesn't do anything yet. + +Creating the Provider +======================= + +Let's sketch out the basic provider class in our ``main.py``. + +First, for simplicity let's import everything we'll need later. + +.. code-block:: python + :caption: src/mathprovider/main.py + + import sys + from typing import Optional, Type + + from tf import runner + from tf import types as t + from tf.iface import ( + Config, + DataSource, + ReadDataContext, + Resource, + State, + ) + from tf.provider import Diagnostics, Provider + from tf.schema import Attribute, Schema + +Now, we can add our ``MathProvider`` class that implements the ``Provider`` protocol. + +.. code-block:: python + :caption: src/mathprovider/main.py + + class MathProvider(Provider): + def get_model_prefix(self) -> str: + return "math_" + + def get_provider_schema(self, diags: Diagnostics) -> Schema: + return Schema(attributes=[]) + + def full_name(self) -> str: + return "test.terraform.io/test/math" + + def validate_config(self, diags: Diagnostics, config: Config): + pass + + def configure_provider(self, diags: Diagnostics, config: Config): + pass + + def get_data_sources(self) -> list[Type[DataSource]]: + return [] + + def get_resources(self) -> list[Type[Resource]]: + return [] + +While we'll leave most of these empty for now, there are a few ones worth noting: + +- ``get_model_prefix`` returns a prefix that will be used for all attributes in this provider. + Tofu decides which resources map to which provider by using their type name prefix. + We'll use ``math_`` here, so our resources will be named similarly to ```math_divider``. + +- ``full_name`` returns the full name of the provider, which is used in Tofu configuration files. + This should be in the format ``/``. + If you ever upload your provider to the Terraform Registry, this should match the name you use there. + That provider name following the last slash should align with the model prefix (e.g. ``math`` and ``math_``). + +- ``get_data_sources`` and ``get_resources`` return lists of data source and resource classes that this provider implements. + We'll leave these empty for now, but we'll add to them later. + +Finally, we need to plug this provider class into our ``main`` function. +We can do that by instantiating it and passing it to ``run_provider``. + +.. code-block:: python + :caption: src/mathprovider/main.py + + def main(): + provider = MathProvider() + runner.run_provider(provider, sys.argv) + +Tofu Environment +================= + +To easily get started using our provider, we're going to create an example directory +for our ``.tf`` files and a ``tofu.rc`` file + +Let's create a ``example/`` directory in our project root and ``tofu.rc`` and ``main.tf`` files in it. + +.. code-block:: shell + + $ mkdir example && cd example + $ touch tofu.rc main.tf + +The ``tofu.rc`` file is a configuration file for Tofu itself. +As we are developing our provider, we want Tofu to find it in our project's virtual environment's ``bin`` directory. + +Uv has helpfully created a ``.venv/`` directory in our project root. The ``tofu.rc`` file needs to point to the absolute path of our ``.venv/bin`` directory. +You'll need to change ``/home/hunter/mathprovider`` to the absolute path of your own project directory. + +.. code-block:: hcl + :caption: example/tofu.rc + + provider_installation { + dev_overrides { + "test.terraform.io/test/math" = "/home/hunter/mathprovider/.venv/bin" + } + direct {} + } + +Let's also fill in our ``main.tf`` file to use our provider. +Right now we'll just specify the provider and configure it with no arguments. + +.. code-block:: hcl + :caption: example/main.tf + + terraform { + required_providers { + math = { + source = "test.terraform.io/test/math" + } + } + } + + provider "math" {} + +Finally, in our shell we'll need to have tofu use our custom ``tofu.rc`` file. +We can do this by setting the ``TF_CLI_CONFIG_FILE`` environment variable to point to our ``tofu.rc`` file. + +.. code-block:: shell + + $ export TF_CLI_CONFIG_FILE=/home/hunter/mathprovider/example/tofu.rc + +Now we can run ```tofu plan`` while we are in the ``example/`` directory. Our ``main.tf`` isn't doing much yet, but we should see our provider being started up. + +.. code-block:: shell + + $ tofu plan + │ Warning: Provider development overrides are in effect + │ + │ The following provider development overrides are set in the CLI configuration: + │ - test.terraform.io/test/math in /home/hunter/mathprovider/.venv/bin + │ + │ The behavior may therefore not match any released version of the provider and applying changes may cause the state to become + │ incompatible with published releases. + ╵ + + No changes. Your infrastructure matches the configuration. + + OpenTofu has compared your real infrastructure against your configuration and found no differences, so no changes are needed. + + +Adding a Data Source +====================== + +In this tutorial we'll only implement a single data source, ``math_divider``, which will take two numbers and return their quotient. + +Let's add another class to our ``main.py`` file that implements the ``DataSource`` protocol. +We'll also add some basic validation to ensure the divisor is not zero. + +.. code-block:: python + :caption: src/mathprovider/main.py + + class Divider(DataSource): + @classmethod + def get_name(cls) -> str: + return "divider" + + @classmethod + def get_schema(cls) -> Schema: + return Schema( + attributes=[ + Attribute("dividend", t.Number(), required=True), + Attribute("divisor", t.Number(), required=True), + Attribute("quotient", t.Number(), computed=True), + ] + ) + + def validate(self, diags: Diagnostics, type_name: str, config: Config): + super().validate(diags, type_name, config) + if config["divisor"] == 0: + diags.add_error( + "Invalid divisor", + "The 'divisor' attribute cannot be zero.", + ) + + def read(self, ctx: ReadDataContext, config: Config) -> Optional[State]: + return { + "dividend": config["dividend"], + "divisor": config["divisor"], + "quotient": config["dividend"] / config["divisor"], + } + + def __init__(self, provider): + pass + +Then we need to add this class to our provider's ``get_data_sources`` method. + +.. code-block:: python + :caption: src/mathprovider/main.py + + ... + + class MathProvider: + ... + + def get_data_sources(self) -> list[Type[DataSource]]: + return [Divider] + +Finally, let's use our new data source in our ``main.tf`` file. +We'll add an ``output`` block so we can see the result of our division. + +.. code-block:: hcl + :caption: example/main.tf + + terraform { + required_providers { + math = { + source = "test.terraform.io/test/math" + } + } + } + + provider "math" {} + + data "math_divider" "example" { + dividend = 10 + divisor = 2 + } + + output "result" { + value = data.math_divider.example.quotient + } + +Now if we run ``tofu plan`` again, we should see our data source being read and the output being computed. + +.. code-block:: shell + + $ tofu plan + │ Warning: Provider development overrides are in effect + │ + │ The following provider development overrides are set in the CLI configuration: + │ - test.terraform.io/test/math in /home/hunter/mathprovider/.venv/bin + │ + │ The behavior may therefore not match any released version of the provider and applying changes may cause the state to become + │ incompatible with published releases. + ╵ + data.math_divider.example: Reading... + data.math_divider.example: Read complete after 0s + + Changes to Outputs: + + result = 5 + + You can apply this plan to save these new output values to the OpenTofu state, without changing any real infrastructure. + + ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── + + Note: You didn't use the -out option to save this plan, so OpenTofu can't guarantee to take exactly these actions if you run + "tofu apply" now. + +Congratulations! You've created a simple Tofu provider with a data source! +Now you can experiment with adding more data sources and resources to your provider. diff --git a/src/tf/__init__.py b/src/tf/__init__.py new file mode 100644 index 0000000..480d164 --- /dev/null +++ b/src/tf/__init__.py @@ -0,0 +1,2 @@ +def main() -> None: + print("Hello from tf!") diff --git a/tf/iface.py b/tf/iface.py index 9da0ed8..b454124 100644 --- a/tf/iface.py +++ b/tf/iface.py @@ -8,21 +8,22 @@ if TYPE_CHECKING: # pragma: no cover from tf.function import Function + +State: TypeAlias = dict """ State is the current state of a resource. It is a dictionary where field names are mapped to Python values (or None, or Unknown). Resource operations are mostly just pushing around, mutating, and returning State. """ -State: TypeAlias = dict +Config: TypeAlias = dict """ Config is like State, except its used in configuration validation and the values are null when they are not bound to a value. This is because the configuration is not yet bound to a resource. This is merely for validating that set of input parameters or values are correct. """ -Config: TypeAlias = dict class AbstractResource(Protocol): diff --git a/tf/types.py b/tf/types.py index 2bd4361..f530aa2 100644 --- a/tf/types.py +++ b/tf/types.py @@ -185,5 +185,8 @@ def __deepcopy__(self, memo): return self -# Unknown is a meta type that can be used to represent an unknown value in a state plan Unknown = _Unknown() +""" +Unknown is a sentinel value that represents a value that is not yet known. +You will find these in a state plan. +"""