diff --git a/releases/determinism-plus-llms/README.md b/releases/determinism-plus-llms/README.md new file mode 100644 index 0000000..900adc7 --- /dev/null +++ b/releases/determinism-plus-llms/README.md @@ -0,0 +1,70 @@ +# Deterministic codegen meets llm codegen for client sdks + +Traditional codegen is fast and reliable but rigid - change your API spec and you get predictable, identical output every time. LLM codegen is adaptive and intelligent but inconsistent - ask it to generate the same SDK twice and you'll get different results. + +This new hybrid approach gives you both: the reliability of deterministic generation for your core SDK structure, with LLM intelligence layered on top for adaptive features like intelligent parameter handling, context-aware documentation, and smart error recovery. + +## How it works + +Don't care how it works? Skip to [get started](#get-started-with-your-apis). + +The system runs deterministic codegen to establish the SDK structure. Then LLMs enhance specific components where adaptability adds value - like generating contextual examples, or adding functions that chain together multiple API calls. + +The LLM can edit most the files (see [python rules](./assets/python/CLAUDE.md) and [typescript rules](./assets/typescript/CLAUDE.md)) generated by deterministic codegen, and changes will persist across regenerations. The system uses structured pattern matching queries—essentially SQL for source code syntax trees—to precisely target only the elements that need updating. Instead of overwriting entire files, it identifies specific patterns (function signatures, import statements, etc.) and surgically modifies those components while preserving all custom code around them. + +![Codegen Process](./assets/codegen-diagram.svg) + +Questions? Join our new [slack](https://join.slack.com/t/sideko-community/shared_invite/zt-3bx0d66ra-iUN69c6qwcd2rnQ5BCY7zQ). + +The LLM component operates through rules files (like CLAUDE.md) that define what the AI can modify, and coding standards to follow. SDK builders can open the SDKs in Cursor, Claude Code, Gemini, or GH Copilot, and the LLM will follow the guidelines to enhance the code. + +## Get started with your APIs + +### Pick an install method to get `sideko` cli +- `npm install -g @sideko/cli` +- `pip install sideko-py` +- `brew install sideko-inc/tap/sideko` +- `curl -fsSL https://raw.githubusercontent.com/Sideko-Inc/sideko/main/install.sh | sh` + +### Login (social or email) +`sideko login` + +### Create initial sdk +`sideko sdk init` + +(*NOTE: use this example OpenAPI if you do not have one: [Flights API](./assets/kong-air-flights.yaml)*) + +### Git init +Make your first commit before enhancing with the LLM so you can easily review it's work. + +`git init && git add . && git commit -m 'deterministic commit'` + + +### Start prompting + +You might have a workflow in mind (make a function that calls this endpoint and then calls that endpoint). See sample prompt for the example api for inspiration: +``` +create a flight tracking workflow + 1. get all flights + 2. select the next flight + 3. return "enjoy :)" if the flight has in-flight entertainment and ":(" if not +``` + +### Test and push +Once all tests are passing (the LLM should test itself against the mock server), push your code to github + +![Tests](./assets/test.png) + +### Set up auto openapi-to-sdk sync +(LINK to new docs) + +### Open source +The core network request libraries (and single dependency in each sdk) are public and open source: +- [javascript](https://github.com/Sideko-Inc/make-request-js) +- [python](https://github.com/Sideko-Inc/make-request-py) +- all other supported sdk languages open sourcing soon + +### What's next +- [join our new slack](https://join.slack.com/t/sideko-community/shared_invite/zt-3bx0d66ra-iUN69c6qwcd2rnQ5BCY7zQ) for questions + sharing your work +- we will be adding llm enhancement support to rust, go, java, and c# in the coming weeks +- explore other sideko features like [pretty api docs](https://docs.sideko.dev/building-documentation/getting-started) diff --git a/releases/determinism-plus-llms/assets/codegen-diagram.svg b/releases/determinism-plus-llms/assets/codegen-diagram.svg new file mode 100644 index 0000000..75a04b3 --- /dev/null +++ b/releases/determinism-plus-llms/assets/codegen-diagram.svg @@ -0,0 +1,168 @@ + + + + + + + + + + + Original Code + + + + + import + json + + def + custom_logger + (msg): + # Custom logging logic + print + ( + f"LOG: {msg}" + ) + + class + APIClient + : + + + + def + send_request + (self, data): + return + json.dumps(data) + + def + validate_response + (self, resp): + # Custom validation + return + resp + is not None + + + + + + + AST Node Updates + + + + + // AST node manipulation + MATCH + function + 'send_request' + GET + params_list = node.args + + APPEND + to params_list: + + ast.arg(arg= + 'timeout' + , + default=ast.Constant(30)) + + UPDATE + function_body + REPLACE + 'json.dumps(data)' + WITH + + 'json.dumps(data, timeout=timeout)' + + + + + + + Result + + + + + import + json + + def + custom_logger + (msg): + # Custom logging logic + print + ( + f"LOG: {msg}" + ) + + class + APIClient + : + + + + def + send_request + (self, data, + timeout=30): + return + json.dumps(data, + timeout=timeout) + + def + validate_response + (self, resp): + # Custom validation + return + resp + is not None + \ No newline at end of file diff --git a/releases/determinism-plus-llms/assets/kong-air-flights.yaml b/releases/determinism-plus-llms/assets/kong-air-flights.yaml new file mode 100644 index 0000000..7fe183b --- /dev/null +++ b/releases/determinism-plus-llms/assets/kong-air-flights.yaml @@ -0,0 +1,209 @@ +--- +# download this file for sdk generation: +# curl -o openapi.yaml https://raw.githubusercontent.com/Sideko-Inc/sideko/refs/heads/main/releases/determinism-plus-llm/kong-air-flights.yaml +openapi: 3.0.0 + +info: + description: KongAir Flights service provides the scheduled flights for KongAir + version: 0.1.0 + title: Flights Service + +servers: +- url: https://api.kong-air.com + description: KongAir API Server + +paths: + /health: + get: + summary: Health check endpoint + description: Endpoint that returns the service health status. + operationId: flights-health-check + responses: + '200': + description: Service is healthy + content: + application/json: + schema: + type: object + properties: + status: + type: string + example: "OK" + headers: + hostname: + description: "The hostname of the machine fulfilling the request." + schema: + type: string + '500': + description: Service is unhealthy + content: + application/json: + schema: + type: object + properties: + status: + type: string + example: "unhealthy" + "/flights": + get: + summary: Get KongAir planned flights + description: | + Returns all the scheduled flights for a given day + tags: + - flight-data + operationId: get-flights + parameters: + - name: date + in: query + description: Filter by date (defaults to current day) + required: false + style: form + schema: + type: string + format: date + responses: + '200': + description: Successful respone with scheduled flights + headers: + hostname: + description: "The hostname of the machine fulfilling the request." + schema: + type: string + content: + application/json: + schema: + type: array + items: + $ref: '#/components/schemas/Flight' + examples: + Example Flights List: + value: + - number: "KD924" + route_id: "LHR-SFO" + scheduled_departure: "2024-03-20T09:12:28Z" + scheduled_arrival: "2024-03-20T19:12:28Z" + - number: "KD925" + route_id: "SFO-LHR" + scheduled_departure: "2024-03-21T09:12:28Z" + scheduled_arrival: "2024-03-21T19:12:28Z" + + "/flights/{flightNumber}": + get: + summary: Get a specific flight by flight number + description: | + Returns a specific flight given its flight number + tags: + - flight-data + operationId: get-flight-by-number + parameters: + - name: flightNumber + in: path + description: The flight number + required: true + style: simple + schema: + type: string + responses: + '200': + description: Successful response with the requested flight + headers: + hostname: + description: "The hostname of the machine fulfilling the request." + schema: + type: string + content: + application/json: + schema: + $ref: '#/components/schemas/Flight' + examples: + Example Flight KD924: + value: + number: "KD924" + route_id: "LHR-SFO" + scheduled_departure: "2024-03-20T09:12:28Z" + scheduled_arrival: "2024-03-20T19:12:28Z" + '404': + description: Flight not found + content: + application/json: + schema: + type: object + properties: + message: + type: string + + "/flights/{flightNumber}/details": + get: + summary: Fetch more details about a flight + description: Fetch more details about a flight + tags: + - flight-data + operationId: get-flight-details + parameters: + - name: flightNumber + in: path + description: The flight number + required: true + style: simple + schema: + type: string + responses: + '200': + description: Successful response with the requested flight details + headers: + hostname: + description: "The hostname of the machine fulfilling the request." + schema: + type: string + content: + application/json: + schema: + $ref: '#/components/schemas/FlightDetails' + '404': + description: Flight not found + content: + application/json: + schema: + type: object + properties: + message: + type: string + +components: + schemas: + Flight: + type: object + properties: + number: + type: string + route_id: + type: string + scheduled_departure: + type: string + format: date-time + scheduled_arrival: + type: string + format: date-time + required: + - number + - route_id + - scheduled_departure + - scheduled_arrival + FlightDetails: + type: object + properties: + flight_number: + type: string + in_flight_entertainment: + type: boolean + meal_options: + type: array + items: + type: string + aircraft_type: + type: string + required: + - flight_number + - in_flight_entertainment + - meal_options + - aircraft_type \ No newline at end of file diff --git a/releases/determinism-plus-llms/assets/python/CLAUDE.md b/releases/determinism-plus-llms/assets/python/CLAUDE.md new file mode 100644 index 0000000..7515592 --- /dev/null +++ b/releases/determinism-plus-llms/assets/python/CLAUDE.md @@ -0,0 +1,346 @@ +## Bash Commands +- `poetry install`: Install the project and the dependencies +- `poetry run pytest`: Run the testing suite +- `poetry run mypy /`: Run the type checker on the main module +- `poetry run pytest tests/test_specific_file.py -v`: Run specific test file +- `poetry add (dependency_name)`: Add a new dependency + +## Definitions +- "the client": The main exported module e.g. `from my_sdk import MyClient` +- "client function": An SDK function that the client can access +- "root client" The sync and async client in `client.py` at the root of the main module + +## Code style +- All imports should be placed at the top of a file +- Add a type hint to every function argument +- Add a return type hint to every function +- Create a sync and an async version of all client functions + +## Types Guide +- bool, int, float, str for primitives +- typing_extensions.Literal["option1", "option2"] for string enums +- typing.List[T] for arrays +- typing.Union[Type1, Type2, ...] for unions +- "MyClass" (quoted) for self-referencing objects + +### Argument Types Guide +- Use `httpx._types.FileTypes` for binary data +- For object inputs, create a typed dict class in `types/params` (see [typed-dicts](#typeddict)) and import it and use it like this: `param_name: params.MyExampleParam` + +### Response Types Guide +- `BinaryResponse` (imported from core) for binary data +- For object responses, create a pydantic model class in `types/models` (see [pydantic model](#pydantic-model)) and use it like this: `models.MyExampleModel` + +## Client function signatures +- See [function signature example](#function-signature) +- Start with "self, *" to prevent positional args +- The last arg should always be `request_options: typing.Optional[RequestOptions] = None,` +- Complex input types go in the `types/params` folder. These are always typed dicts +- Complex response types go in the `types/models/` folder. These are always pydantic models + +## Client functions +- For functions that combine multiple API calls, import and instantiate specific client classes +- Example: `from module.resources.api.spec.client import SpecClient; spec_client = SpecClient(base_client=self._base_client)` +- Include appropriate `auth_names` list +- Use `cast_to=type(None)` for methods returning None + +## Tests +- Create tests for any new functionality in `tests/` +- Tests run against a stateless mock server, so never make any stateful assertions in the tests +- Create contract tests that tests that the request is successful when all parameters are given and when only required parameters are given +- Test behavior when all only required parameters test - Call method with only required parameters +- Always create Async versions using @pytest.mark.asyncio + + +## Documentation +- When creating a new resource, include a README file in the resource folder +- See the [example readme](#documentation) for the style + +## Workflow +- Always typecheck when you're done making a series of code changes +- Prefer running single tests, and not the whole test suite +- Reference the README.md in the root of the repo to understand the module structure of the repo when gathering context. Follow the internal README links to view parameter tables, example snippets, and response examples. +- When adding a new resource: create a new folder within `resources/` create `client.py`, create `__init__.py` create `README.md` add the resource to the sync and async base clients. +- When creating new typed dicts or models, remember to add them to the respective `__init__.py` files in both the import section and `__all__` list +- Always run tests after creating new functionality to ensure everything works correctly + +## IMPORTANT RULES +- Do not remove or change versions of pydantic, httpx in pyproject.toml +- Instantiate the specific client classes needed rather than trying to access them. Never access the base_client directly +- Prefer creating a new resource when chaining together multiple API calls +- Remove unused imports to avoid linting errors +- When creating new models and params, always create new files (1 type per file) +- Do not change any code in `/core` +- Do not change the code in `/environment.py` +- Always use the correct Environment enum value in tests (`Environment.MOCK_SERVER`) +- When adding to an existing resource's README.md, add new content and comment so that the code generator can retain the content in future versions. + +## Code Examples + +### TypedDict +```py +import pydantic +import typing_extensions + + +class Example(typing_extensions.TypedDict): + """ + Example type description + """ + + name: typing_extensions.Required[str] + """ + A great description of the name parameter on this example typed dict + """ + + +class _SerializerExample(pydantic.BaseModel): + """ + Serializer for Example handling case conversions + and file omissions as dictated by the API + """ + + model_config = pydantic.ConfigDict( + populate_by_name=True, + ) + + name: str = pydantic.Field( + alias="name", + ) +``` + +### Pydantic Model +```py +import pydantic + + +class ModelExample(pydantic.BaseModel): + """ + A description of the purpose of this example model + """ + + model_config = pydantic.ConfigDict( + arbitrary_types_allowed=True, + populate_by_name=True, + ) + + name: str = pydantic.Field( + alias="name", + ) + """ + A great description of the name field on this example model + """ +``` + +### Function signature +```py + def create( + self, + *, + asset: params.Asset, + name: typing.Union[ + typing.Optional[str], type_utils.NotGiven + ] = type_utils.NOT_GIVEN, + request_options: typing.Optional[RequestOptions] = None, + ) -> models.CreatedAssets: +``` + +### New resource +Example file location: `resources/your_resource/__init__.py` +```python +from .client import AsyncYourResourceClient, YourResourceClient + +__all__ = ["AsyncYourResourceClient", "YourResourceClient"] +``` + +Example file location: `resources/your_resource/client.py` +```py +import typing +from .core import ( + AsyncBaseClient, + RequestOptions, + SyncBaseClient, + type_utils, +) +from .types import models + +from .resources.custom import CustomClient + +class ResourceClient: + def __init__(self, *, base_client: SyncBaseClient) -> None: + self._base_client = base_client + + def your_new_method( + self, + *, + required_param: str, + optional_param: typing.Optional[str] = None, + request_options: typing.Optional[RequestOptions] = None + ) -> models.NewMethodResponse: + """ + Brief description + + Longer description explaining the functionality. + + HTTP_METHOD /api/endpoint/path + + Args: + required_param: Description of required parameter + optional_param: Description of optional parameter + request_options: Additional options to customize the HTTP request + + Returns: + Success response description + + Raises: + ApiError: A custom exception class that provides additional context + for API errors, including the HTTP status code and response body. + """ + your_client = CustomClient(base_client=self._base_client) + result = your_client.call_api( + required_param=param1, + optional_param=optional_param, + request_options=request_options + ) + return models.NewMethodResponse(data=result) + +class AsyncYourResourceClient: + def __init__(self, *, base_client: AsyncBaseClient) -> None: + self._base_client = base_client + + async def your_new_method( + self, + *, + required_param: str, + optional_param: typing.Optional[str] = None, + request_options: typing.Optional[RequestOptions] = None + ) -> models.NewMethodResponse: + """ + [Same docstring as sync version, but explain that it is async] + """ + your_client = AsyncCustomClient(base_client=self._base_client) + result = await your_client.call_api( + required_param=param1, + optional_param=optional_param, + request_options=request_options + ) + return models.NewMethodResponse(data=result) +``` + + +### Tests +```python +import io +import pydantic +import pytest +import typing + +from import AsyncResourceClient, ResourceClient +from .environment import Environment +from .types import models + + +def test_create_201_success_all_params(): + """Tests a POST request to the /project/{name} endpoint. + + Operation: create + Test Case ID: success_all_params + Expected Status: 201 + Mode: Synchronous execution + + Empty response expected + + Validates: + - Authentication requirements are satisfied + - All required input parameters are properly handled + - Response status code is correct + - Response data matches expected schema + + This test uses example data to verify the endpoint behavior. + """ + # tests calling sync method with example data + client = ResourceClient(api_key="API_KEY", environment=Environment.MOCK_SERVER) + response = client.resource.create(project="my-project", file=io.BytesIO(b"123")) + try: + pydantic.TypeAdapter(models.Project).validate_python(response) + is_valid_response_schema = True + except pydantic.ValidationError: + is_valid_response_schema = False + assert is_valid_response_schema, "failed response type check" + + +@pytest.mark.asyncio +async def test_await_create_201_success_all_params(): + """Tests a POST request to the /project/{name} endpoint. + + Operation: create + Test Case ID: success_all_params + Expected Status: 201 + Mode: Asynchronous execution + + Empty response expected + + Validates: + - Authentication requirements are satisfied + - All required input parameters are properly handled + - Response status code is correct + - Response data matches expected schema + + This test uses example data to verify the endpoint behavior. + """ + # tests calling async method with example data + client = ResourceClient(api_key="API_KEY", environment=Environment.MOCK_SERVER) + response = await client.resource.create(project="my-project", file=io.BytesIO(b"123")) + try: + pydantic.TypeAdapter(models.Project).validate_python(response) + is_valid_response_schema = True + except pydantic.ValidationError: + is_valid_response_schema = False + assert is_valid_response_schema, "failed response type check" + +``` + + +### Documentation +```markdown +### Method Name + +Brief description of what the method does. + +**API Endpoint**: `HTTP_METHOD /api/endpoint/path` (or all endpoints it will hit if it's a workflow) + +#### Parameters + +| Parameter | Required | Description | Example | +|-----------|:--------:|-------------|---------| +| `required_param` | ✓ | Description | `"example_value"` | +| `optional_param` | | Description | `"optional_value"` | + +#### Synchronous Client + +```python +from import +from os import getenv + +client = (api_key=getenv("API_KEY")) +res = client.your_resource.method_name(required_param="value") +``` + +#### Asynchronous Client + +```python +from import +from os import getenv + +client = (api_key=getenv("API_KEY")) +res = await client.your_resource.method_name(required_param="value") +``` + +#### Response + +##### Type +[YourResponseModel](/path/to/types/your_response_model.py) + +##### Example +`{"field": "value", "another_field": 123}` +``` \ No newline at end of file diff --git a/releases/determinism-plus-llms/assets/test.png b/releases/determinism-plus-llms/assets/test.png new file mode 100644 index 0000000..1f47e7f Binary files /dev/null and b/releases/determinism-plus-llms/assets/test.png differ diff --git a/releases/determinism-plus-llms/assets/typescript/CLAUDE.md b/releases/determinism-plus-llms/assets/typescript/CLAUDE.md new file mode 100644 index 0000000..627ef71 --- /dev/null +++ b/releases/determinism-plus-llms/assets/typescript/CLAUDE.md @@ -0,0 +1,212 @@ +## Bash Commands +- `npm i`: Install the project and the dependencies +- `npm test`: Run the entire testing suite +- `npm test -- --testNamePattern="client.exampleResource.exampleMethod"` +- `npm add (dependency_name)`: Add a new dependency + +## Definitions +- "the client": The main exported module e.g. `import MyClient from "my_sdk"` +- "client function": An SDK function that the client can access +- "root client": `client.ts` at the root of the main module + +## Code style +- All imports should be placed at the top of a file +- Add a type to every function argument +- Add a return type to every function +- Add a docstring to every new function + +## Types Guide +- boolean, number, string for primitives +- string literals with "|" for string enums +- [] for arrays +- "|" for unions + +### Argument Types Guide +- `import { type UploadFile } from "/core";` for binary data +- For object inputs, create a file called `argument-types.ts` at the same level as the client and export the created type from `index.ts` + +### Response Types Guide +- `import { BinaryResponse } from "/core";` for binary data +- For object responses, create a new file and type in the `src/types` folder. +- Export any newly created types from `src/types/index.ts` + +## Client functions +- Functions should take at max two inputs. The first is a type for input data, and the second is `opts?: RequestOptions` (`import { RequestOptions} from "/core";`) + +## Tests +- Create jest test for any new functionality in `test/` +- Tests run against a stateless mock server, so never make any stateful assertions in the tests +- Create contract tests that test that the request is successful when all parameters are given and when only required parameters are given + +## Documentation +- When creating a new resource, include a README file in the resource folder +- See the [example readme](#documentation) for the style + +## IMPORTANT RULES +- Do not remove or change the versions of `form-data` `form-url-encoded` `js-base64` `jsonpointer` `node-fetch` `qs` or `zod` +- Any imports that are node or browser dependent must be imported using the `require` keyword inline according to the runtime from (`import { RUNTIME } from "/core";`) +- Prefer creating a new resource when chaining together multiple API calls +- Do not change any code in `"src/core"`; +- Do not change `src/environment.ts` +- Never use this._client.makeRequest directly, always use existing functionalities via resource clients + +## Code Examples + +### Type +```ts +import * as z from "zod"; + +import { zodTransform } from "/portal-client/core"; + +/** + * Example Type + */ +export type Example = { + /** + * the unique identifier of this example + */ + id: string; +}; +``` + +### Function Signature + /** + * Functionality Description + */ + method( + request: requests.MethodRequest, + opts?: RequestOptions, + ): Promise { + } + +### New Resource +Files +- index: `src/resources/example-resource/index.ts` +- client: `src/resources/example-resource/resource-client.ts` +- request types: `src/resources/example-resource/request-types.ts` +- docs: `src/resources/example-resource/README.md` + +Example `index.ts` +```ts +export { + ExampleMethodRequest, +} from "./request-types"; +export { ExampleClient } from "./resource-client"; +``` + +Example `request-types.ts` +```ts +export type ExampleMethodRequest = { + id: string; + val?: string; +} +``` + +Example `resource-client.ts` +```ts +import type { types } from ""; +import { + CoreResourceClient, + type RequestOptions, +} from "/core"; +import type * as requests from "/resources/example-resource/request-types"; +import { ResourceClientA } from "/resources/a"; +import { ResourceClientB } from "/resources/b"; + + +export class ExampleClient extends CoreResourceClient { + /** + * Example Description + */ + async exampleMethod( + request: requests.ExampleMethodRequest = {}, + opts?: RequestOptions, + ): Promise { + const resourceA = new ResourceClientA(this._client, this._opts); + const resourceB = new ResourceClientB(this._client, this._opts); + + const resA = await resourceA.get({ id: request.id }); + const resB = await resourceB.create({ a: resA.id, val: request.val }); + + return { aId: resA.id, bId: resB.id }; + } +} + +``` + +### Tests +`test/example-resource.test.ts` +```ts +import Client, { Environment } from ""; + +describe("tests client.exampleResource.exampleMethod", () => { + test.concurrent( + "Desciption actions | testId: success_all_params | Desciption of test", + async () => { + const client = new Client({ + apiKey: "API_KEY", + environment: Environment.MockServer, + }); + const response = await client.exampleResource.exampleMethod( + { + id: "abc", + val: "data", + } + ); + expect(response).toBeDefined(); + } + ) + test.concurrent( + "Desciption actions | testId: success_required_only | Desciption of test", + async () => { + const client = new Client({ + apiKey: "API_KEY", + environment: Environment.MockServer, + }); + const response = await client.exampleResource.exampleMethod({ id: "abc" }); + expect(response).toBeDefined(); + } + ) +}); +``` + +### Documentation +`src/resources/example-resource/README.md` + +````markdown +# example-resource + +## Module Functions +### Example Method + +#### Parameters + +| Parameter | Required | Description | Example | +|-----------|:--------:|-------------|--------| +| `id` | ✓ | the unique idenifier | `"3e4666bf-d5e5-4aa7-b8ce-cefe41c7568a"` | +| `val` | ✗ | the example to send | `data` | + + +#### Example Snippet + +```typescript +import Client from ""; + +const client = new Client({ + apiKey: process.env["API_KEY"]!!, +}); +const res = await client.exampleResource.exampleMethod({ + id: "3e4666bf-d5e5-4aa7-b8ce-cefe41c7568a" +}); + +``` + +#### Response + +##### Type +[ExampleMethodOutput](/src/types/example-method-output.ts) + +##### Example +`{ aId: "abc", bId: "bcd" }` + +```` \ No newline at end of file