Add tutorial and update documentation

CHANGELOG.md

@@ -12,6 +12,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - Added `description`, `description_kind`, and `deprecated` fields to Schema and Block. This allows resource, data sources, and providers to specify these fields for documentation.
 - **End-to-End Tests**:
   - Added comprehensive end-to-end tests against OpenTofu. This ensures that providers, resources, and data sources work correctly against the real OpenTofu CLI.
+- **Tutorial**:
+  - Added a tutorial to the documentation to help new users get started with writing a provider using `tf`.
 
 ## 1.1.0
 

docs/api.rst

@@ -4,22 +4,37 @@ Stable API
 
 .. py:module:: tf
 
+:code:`tf`'s API consists of:
+
+* Low-level protocol classes that library consumers are expected to implement (such as :class:`~tf.iface.Provider`, :class:`~tf.iface.Resource`, and :class:`~tf.iface.DataSource`).
+* Mid-level "glue" for schema definitions, such as :class:`~tf.schema.Attribute` and :class:`~tf.schema.Schema`.
+* High-level commonly-used types such as :class:`~tf.types.String`, and :class:`~tf.types.List`.
+* A handful of utility functions and classes, such as :func:`~tf.runner.run_provider` and :class:`~tf.runner.install_provider`.
+
+As a library consumer and provider author, your primary interaction will be writing classes that satisfy :class:`~tf.iface.Provider`, :class:`~tf.iface.Resource`, and :class:`~tf.iface.DataSource`.
+If your provider has a large number of elements, you will likely want to implement your own higher-level abstractions to stamp out boilerplate class definitions.
+Protocols give you the flexibility to do this in a clean way.
+
+However, it's perfectly fine (if somewhat tedious) to implement each element against it's protocol individually.
+
 Execution Interface
 ===================
 
 The execution interface provides program entrypoints for OpenTofu to run your provider.
 
+You provider should have a `Console Script <https://setuptools.pypa.io/en/latest/userguide/entry_point.html#console-scripts>`_ entrypoint (named :code:`terraform-provider-myprovider`) pointing to your main function.
+Your main function should create an instance of your provider and call :func:`~tf.runner.run_provider`.
 
 .. autofunction:: tf.runner.run_provider
 
-An installation utility is provided to install your provider.
+An installation utility is provided to install your provider into the plugins directory.
+
+.. warning::
+    Running :func:`~tf.runner.install_provider`. is only required if you want to install your provider *in development mode* into your plugin directory.
+    There are easier ways to test your provider during development, such as setting the :code:`TF_CLI_CONFIG_FILE`.
 
 .. autofunction:: tf.runner.install_provider
 
-You are expected to provide a `main.py` file in your provider package with a method
-(set as an entrypoint)
-that calls
-:func:`~tf.runner.run_provider` and :func:`~tf.runner.install_provider`.
 
 Provider Interface
 ==================
@@ -77,6 +92,14 @@ It is a dictionary of field names to values.
    :members:
 
 
+Config
+------
+
+*Config* is used instead of State during validation.
+
+.. autoclass:: tf.iface.Config
+   :members:
+
 Schemas
 ============
 
@@ -124,3 +147,13 @@ Several utility types are also provided:
 .. autoclass:: tf.types.NormalizedJson
    :show-inheritance:
 
+
+Unknown
+-------
+:class:`~tf.types.Unknown` is a special `value` that represents a value that is not known at plan time.
+Unknown is a fundamental part of TF's design and is used extensively in the planning phase.
+
+TF makes a distinction between `null` and `unknown`. :code:`tf` represents them with :code:`None` and :code:`tf.types.Unknown` respectively.
+
+.. autoclass:: tf.types.Unknown
+   :members:

docs/caveats.md

@@ -21,15 +21,17 @@ The biggest problem is that Terraform **expects the provider to be a single bina
 You need to get your entire artifact into one executable file that you can move to someone else's machine.
 This is VERY at odds with the default Python experience.
 
-To get around this, you can use something like [Nuitka](https://nuitka.net/) to compile your Python code into a single binary with `--onefile` and `--standalone`.
-This is very fickle, and you need to consider:
+To get around this, you can use something like [Pex](https://docs.pex-tool.org/) to generate a single binary.
+While this works for simple (pure Python) packages, you need to consider:
 
 * How many OS/Architecture combinations do you need to support?
-* How old of a version of GLIBC are you are targeting? You might want to run nuitka in docker container for an older distro.
+* How old of a version of GLIBC are you are targeting? Prepare to bake pre-prepared `--complete-platform` configuration sets for every combination you want to support.
 * Do any of your dependencies have C extensions? Or do dynamic linking?
-  This might be a dealbreaker -- you can't install these on the fly.
+  This might be a dealbreaker.
 
-With a lot of project-specific configuration, you can get this working and building a single `main.bin`. But YMMV.
+With a lot of project-specific configuration, you can get this working and building a single `terraform-provider-$providername`. But YMMV.
+
+See [Building a Binary](tips_tricks.html#building-a-binary) for some tips and tricks to get this working.
 
 #### Development mode already works, why not use that?
 

docs/index.rst

@@ -1,12 +1,5 @@
 ..  mdinclude:: ../README.md
 
-.. warning::
-
-    There are serious hurdles to taking a Python provider to production.
-    Read :doc:`caveats` before considering a production use case.
-
-    For development, testing, and proof-of-concept, continue on to :doc:`usage`.
-
 ..  toctree::
     :maxdepth: 1
     :caption: Contents:
@@ -15,5 +8,7 @@
     elements.md
     functions.md
     api.rst
+    tutorial.rst
     example.md
     caveats.md
+    tips_tricks.rst

docs/tips_tricks.rst

@@ -0,0 +1,62 @@
+***************
+Tips and Tricks
+***************
+
+Using a Debugger
+================
+
+You pre-emptively launch your provider in a long-lived server mode by running ``terraform-provider-$providername --stable --dev`` in a terminal.
+This will start the provider, dump a ``export TF_REATTACH_PROVIDERS=...`` command, and wait for connections.
+
+In another terminal, you can paste and execute the ``export TF_REATTACH_PROVIDERS=...`` command to set up your environment.
+Then you can run ``tofu plan`` or ``terraform apply`` as usual.
+
+Tofu will connect to the already-running provider instead of starting it on-demand.
+By launching your provider with ``--stable --dev`` in your debugger, you can set breakpoints and inspect state as needed.
+
+
+Building a Binary
+==================
+
+Ultimately, if you want to distribute your provider you'll need to build a single standalone binary.
+The Terraform registry (and all custom provider registries) expect a binary executable, not a Python script.
+
+There are many ways to build a binary from Python code, but the easiest one to get working is `pex <https://docs.pex-tool.org/>`_.
+
+Pex essentially bundles up your Python code and all its dependencies into a single file that can be executed as a binary.
+When the binary is run, it sets up a virtual environment and runs your code in that environment.
+
+Another nice feature of pex is that it can generate binaries for multiple platforms (Linux, Windows, MacOS) from a single machine.
+
+We have had success building provider binaries with Pex and uploading and consuming them from Terraform private registries.
+From tofu and the registry's perspective, the pex binary no different than a Go binary built using the official Terraform provider framework.
+
+The provider upload registry APIs are not standardized across hosting providers, so you'll need to explore bundling and uploading the binaries yourself for your choice of registry.
+
+Roughly, the steps to build a pex binary are:
+
+#. Install pex: `pip install pex`, preferably into a separate virtual environment or as a development dependency of your provider project.
+#. Dump the dependencies of your provider into a requirements file:
+
+   .. code-block:: shell
+
+      pip freeze > requirements.txt
+
+   Alternatively, you can use a package-manager specific command
+   such as ``uv export -f requirements.txt > requirements.txt``).
+
+#. Build the pex binary:
+
+   .. code-block:: shell
+
+      pex -r requirements.txt ./ \
+        -o terraform-provider-$providername \
+        --scie eager \
+        -m providerpackage.main:main
+
+   By using ``--scie eager``, pex will include a Python interpreter in the binary as well as all the dependencies.
+   This means everything needs to run the provider is contained in the single binary file.
+
+You will now have a binary executable named ``terraform-provider-$providername`` that you can upload to your registry and use with Terraform.
+
+At this point you have a minimum viable provider binary, but you will need to think about multiple platforms and architectures and ABIs.