From b023e2ec840e0032a510bffb2ab6f0f3e6437716 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C5=82=20Olender?= <92638966+TC-MO@users.noreply.github.com> Date: Tue, 7 Jul 2026 20:55:36 +0200 Subject: [PATCH 1/2] refactor(storage): move schema pages into Storage, grouped by store type (#2746) Move the dataset and key-value store schema pages out of Actor development into the Storage section, grouped under Dataset and Key-value store categories. Add nginx redirects for the old URLs and repoint internal links; page slugs are unchanged. Also align the two schema docs to a parallel structure and headings, correct the Actor definition schema list to cover input and output schemas, and drop first-person voice from the storage pages. --- .../parameters/datasetParameters.yaml | 2 +- .../components/schemas/datasets/Dataset.yaml | 2 +- nginx.conf | 5 + .../how-to-build/actorization_playbook.mdx | 2 +- .../deploying_your_code/output_schema.md | 2 +- .../actor_definition/actor_json.md | 4 +- .../development/actor_definition/index.md | 4 +- .../actor_definition/output_schema/index.md | 4 +- .../web_server_schema/index.md | 2 +- .../development/builds_and_runs/runs.md | 2 +- .../programming_interface/basic_commands.md | 4 +- .../actors/publishing/publish-task.mdx | 4 +- .../actors/publishing/quality_score.mdx | 2 +- .../actors/running/monitoring/index.md | 2 +- .../actors/running/usage_and_resources.md | 4 +- .../platform/get-started/building/locally.md | 4 +- .../dataset/dataset_schema.md} | 6 +- .../storage/{dataset.md => dataset/index.md} | 18 +- .../dataset}/multiple_datasets.mdx | 14 +- .../dataset}/validation.md | 8 +- sources/platform/storage/index.md | 2 +- .../images/kv-store-schema-example-run.png | Bin .../kv-store-schema-example-storage.png | Bin .../index.md} | 26 +-- .../key_value_store_schema.md} | 172 ++++++++++-------- sources/platform/storage/request_queue.md | 2 +- 26 files changed, 163 insertions(+), 134 deletions(-) rename sources/platform/{actors/development/actor_definition/dataset_schema/index.md => storage/dataset/dataset_schema.md} (99%) rename sources/platform/storage/{dataset.md => dataset/index.md} (88%) rename sources/platform/{actors/development/actor_definition/dataset_schema => storage/dataset}/multiple_datasets.mdx (90%) rename sources/platform/{actors/development/actor_definition/dataset_schema => storage/dataset}/validation.md (96%) rename sources/platform/{actors/development/actor_definition/key_value_store_schema => storage/key_value_store}/images/kv-store-schema-example-run.png (100%) rename sources/platform/{actors/development/actor_definition/key_value_store_schema => storage/key_value_store}/images/kv-store-schema-example-storage.png (100%) rename sources/platform/storage/{key_value_store.md => key_value_store/index.md} (82%) rename sources/platform/{actors/development/actor_definition/key_value_store_schema/index.md => storage/key_value_store/key_value_store_schema.md} (69%) diff --git a/apify-api/openapi/components/parameters/datasetParameters.yaml b/apify-api/openapi/components/parameters/datasetParameters.yaml index e89eac2f8b..f7bf7efe89 100644 --- a/apify-api/openapi/components/parameters/datasetParameters.yaml +++ b/apify-api/openapi/components/parameters/datasetParameters.yaml @@ -236,7 +236,7 @@ view: description: | Defines the view configuration for dataset items based on the schema definition. This parameter determines how the data will be filtered and presented. - For complete specification details, see the [dataset schema documentation](/actors/development/actor-definition/dataset-schema). + For complete specification details, see the [dataset schema documentation](/storage/dataset-schema). schema: type: string example: overview diff --git a/apify-api/openapi/components/schemas/datasets/Dataset.yaml b/apify-api/openapi/components/schemas/datasets/Dataset.yaml index af8afcff41..1063ad9df1 100644 --- a/apify-api/openapi/components/schemas/datasets/Dataset.yaml +++ b/apify-api/openapi/components/schemas/datasets/Dataset.yaml @@ -50,7 +50,7 @@ properties: description: "" schema: type: [object, "null"] - description: Defines the schema of items in your dataset, the full specification can be found in [Apify docs](/actors/development/actor-definition/dataset-schema) + description: Defines the schema of items in your dataset, the full specification can be found in [Apify docs](/storage/dataset-schema) example: actorSpecification: 1 title: My dataset diff --git a/nginx.conf b/nginx.conf index 0c8e5eedb3..f9e308d4d2 100644 --- a/nginx.conf +++ b/nginx.conf @@ -808,6 +808,11 @@ server { # the rest of the way to /storage and /proxy. The post-flip cleanup PR collapses # that double-hop.) + # IA v3 follow-ups (#2746): storage schema pages moved from Actor development to Storage. + # 302 while the IA settles; promote to permanent once stable. + rewrite ^/actors/development/actor-definition/dataset-schema(/.*)?$ /storage/dataset-schema$1 redirect; + rewrite ^/actors/development/actor-definition/key-value-store-schema$ /storage/key-value-store-schema redirect; + # Soft-landings for deleted URLs (better UX than a 404) rewrite ^/platform/integrations/airtable/console-integration$ /integrations/airtable permanent; rewrite ^/platform/?$ / permanent; diff --git a/sources/academy/build-and-publish/how-to-build/actorization_playbook.mdx b/sources/academy/build-and-publish/how-to-build/actorization_playbook.mdx index c4e9251404..d575473f28 100644 --- a/sources/academy/build-and-publish/how-to-build/actorization_playbook.mdx +++ b/sources/academy/build-and-publish/how-to-build/actorization_playbook.mdx @@ -102,7 +102,7 @@ These steps are the bare minimum you need to run your code on Apify. The rest of Most Actors accept an input and produce an output. As part of Actorization, you need to define the input and output structure of your application. -For detailed information, read the docs for [input schema](/actors/development/actor-definition/input-schema), [dataset schema](/actors/development/actor-definition/dataset-schema), and general [storage](/storage). +For detailed information, read the docs for [input schema](/actors/development/actor-definition/input-schema), [dataset schema](/storage/dataset-schema), and general [storage](/storage). #### Design guidelines diff --git a/sources/academy/platform/deploying_your_code/output_schema.md b/sources/academy/platform/deploying_your_code/output_schema.md index 31d1fb6495..b02d492061 100644 --- a/sources/academy/platform/deploying_your_code/output_schema.md +++ b/sources/academy/platform/deploying_your_code/output_schema.md @@ -162,7 +162,7 @@ Great! Now that everything is set up, it's time to run the Actor and admire your :::tip Additional resources -Visit the [dataset schema documentation](/actors/development/actor-definition/dataset-schema) for more detailed information about how to implement this feature. +Visit the [dataset schema documentation](/storage/dataset-schema) for more detailed information about how to implement this feature. ::: diff --git a/sources/platform/actors/development/actor_definition/actor_json.md b/sources/platform/actors/development/actor_definition/actor_json.md index 83b848e79a..9338a1ed95 100644 --- a/sources/platform/actors/development/actor_definition/actor_json.md +++ b/sources/platform/actors/development/actor_definition/actor_json.md @@ -83,8 +83,8 @@ Actor `name`, `version`, `buildTag`, and `environmentVariables` are currently on | `input` | Optional | You can embed your [input schema](./input_schema/index.md) object directly in `actor.json` under the `input` field. You can also provide a path to a custom input schema. If not provided, the input schema at `.actor/INPUT_SCHEMA.json` or `INPUT_SCHEMA.json` is used, in this order of preference. You can also use the `inputSchema` alias interchangeably. | | `output` | Optional | You can embed your [output schema](./output_schema/index.md) object directly in `actor.json` under the `output` field. You can also provide a path to a custom output schema. [Read more](/actors/development/actor-definition/output-schema) about Actor output schemas. You can also use the `outputSchema` alias interchangeably. | | `changelog` | Optional | The path to the CHANGELOG file displayed in the Information tab of the Actor in Apify Console next to Readme. If not provided, the CHANGELOG at `.actor/CHANGELOG.md` or `CHANGELOG.md` is used, in this order of preference. Your Actor doesn't need to have a CHANGELOG but it is a good practice to keep it updated for published Actors. | -| `storages.dataset` | Optional | You can define the schema of the items in your dataset under the `storages.dataset` field. This can be either an embedded object or a path to a JSON schema file. [Read more](/actors/development/actor-definition/dataset-schema) about Actor dataset schemas. | -| `storages.datasets` | Optional | You can define multiple datasets for the Actor under the `storages.datasets` field. This can be an object containing embedded objects or paths to a JSON schema files. [Read more](/actors/development/actor-definition/dataset-schema/multiple-datasets) about multiple dataset schemas. | +| `storages.dataset` | Optional | You can define the schema of the items in your dataset under the `storages.dataset` field. This can be either an embedded object or a path to a JSON schema file. [Read more](/storage/dataset-schema) about Actor dataset schemas. | +| `storages.datasets` | Optional | You can define multiple datasets for the Actor under the `storages.datasets` field. This can be an object containing embedded objects or paths to a JSON schema files. [Read more](/storage/dataset-schema/multiple-datasets) about multiple dataset schemas. | | `defaultMemoryMbytes` | Optional | Specifies the default amount of memory in megabytes to be used when the Actor is started. Can be an integer or a [dynamic memory expression string](./dynamic_actor_memory/index.md). | | `minMemoryMbytes` | Optional | Specifies the minimum amount of memory in megabytes required by the Actor to run. Requires an _integer_ value. If both `minMemoryMbytes` and `maxMemoryMbytes` are set, then `minMemoryMbytes` must be equal or lower than `maxMemoryMbytes`. Refer to the [Usage and resources](https://docs.apify.com/actors/running/usage-and-resources#memory) for more details about memory allocation. | | `maxMemoryMbytes` | Optional | Specifies the maximum amount of memory in megabytes required by the Actor to run. It can be used to control the costs of run. Requires an _integer_ value. Refer to the [Usage and resources](https://docs.apify.com/actors/running/usage-and-resources#memory) for more details about memory allocation. | diff --git a/sources/platform/actors/development/actor_definition/index.md b/sources/platform/actors/development/actor_definition/index.md index 9df2034405..01b0d1b46b 100644 --- a/sources/platform/actors/development/actor_definition/index.md +++ b/sources/platform/actors/development/actor_definition/index.md @@ -14,10 +14,10 @@ Actors have the following elements: - The main **[actor.json](./actor_json.md)** file contains **metadata** such as the Actor name, description, author, version, and links pointing to the other definition files below. - **[Dockerfile](./docker.md)** which specifies where is the Actor's source code, how to build it, and run it. - **Documentation** in the form of a **README.md** file. -- **[Input](./input_schema/index.md)** and **[dataset schemas](/actors/development/actor-definition/dataset-schema)** that describe what input the Actor requires and what results it produces. +- **[Input](./input_schema/index.md)** and **[output](./output_schema/index.md)** schemas that describe what input the Actor requires and what output it produces. - Access to an out-of-box **[storage](../../../storage/index.md)** system for Actor data, results, and files. -The documentation and the input/dataset schemas make it possible for people to easily understand what the Actor does, enter the required inputs both in the user interface or API, and integrate the Actor's results with their other workflows. Actors can easily call and interact with each other, enabling building more complex systems on top of simple ones. +The documentation and the input and output schemas make it possible for people to easily understand what the Actor does, enter the required inputs both in the user interface or API, and integrate the Actor's results with their other workflows. Actors can easily call and interact with each other, enabling building more complex systems on top of simple ones. The Apify platform provides an open [API](/api/v2), cron-style [scheduler](/actors/running/schedules), [webhooks](../../../integrations/programming/webhooks/index.md), and [integrations](../../integrations) to services such as Zapier or Make, which make it easy for users to integrate Actors with their existing workflows. Anyone is welcome to [publish Actors](/actors/publishing) in [Apify Store](https://apify.com/store), and you can even [monetize your Actors](/actors/publishing/monetize). diff --git a/sources/platform/actors/development/actor_definition/output_schema/index.md b/sources/platform/actors/development/actor_definition/output_schema/index.md index 3c939e4939..a5c6b6971b 100644 --- a/sources/platform/actors/development/actor_definition/output_schema/index.md +++ b/sources/platform/actors/development/actor_definition/output_schema/index.md @@ -6,7 +6,7 @@ description: Define an output schema to specify where your Actor stores its resu slug: /actors/development/actor-definition/output-schema --- -The Actor output schema builds upon the schemas for the [dataset](/actors/development/actor-definition/dataset-schema) and [key-value store](/actors/development/actor-definition/key-value-store-schema). It specifies where an Actor stores its output and defines templates for accessing that output. Apify Console uses these output definitions to display run results, and the Actor run's `GET` endpoint includes them in the output property. +The Actor output schema builds upon the schemas for the [dataset](/storage/dataset-schema) and [key-value store](/storage/key-value-store-schema). It specifies where an Actor stores its output and defines templates for accessing that output. Apify Console uses these output definitions to display run results, and the Actor run's `GET` endpoint includes them in the output property. ## Why output schema matters @@ -263,7 +263,7 @@ After you define `views` and `collections` in `dataset_schema.json` and `key_val :::note Output schema complements dataset schema -The output schema defines *where* data is stored and how to access it. The [dataset schema](/actors/development/actor-definition/dataset-schema) defines *what* fields each item contains, including descriptions and examples. Use both schemas together: +The output schema defines *where* data is stored and how to access it. The [dataset schema](/storage/dataset-schema) defines *what* fields each item contains, including descriptions and examples. Use both schemas together: - Output schema: Declares that results are in the default dataset - Dataset schema: Describes each field with `title`, `description`, and `example` diff --git a/sources/platform/actors/development/actor_definition/web_server_schema/index.md b/sources/platform/actors/development/actor_definition/web_server_schema/index.md index f4628e9937..95faadb33e 100644 --- a/sources/platform/actors/development/actor_definition/web_server_schema/index.md +++ b/sources/platform/actors/development/actor_definition/web_server_schema/index.md @@ -90,7 +90,7 @@ Follow the standard [OpenAPI 3.x format](https://spec.openapis.org/oas/latest.ht ## Build and deploy -The build process validates `webServerSchema`, similar to other Actor schemas like [input schema](/actors/development/actor-definition/input-schema) and [dataset schema](/actors/development/actor-definition/dataset-schema). If the spec is malformed, the build fails with a validation error. +The build process validates `webServerSchema`, similar to other Actor schemas like [input schema](/actors/development/actor-definition/input-schema) and [dataset schema](/storage/dataset-schema). If the spec is malformed, the build fails with a validation error. Once deployed, the **Standby** tab appears automatically on the Actor's detail page when you enable [standby mode](/actors/development/programming-interface/standby). It renders your spec with [Swagger UI](https://swagger.io/tools/swagger-ui/) and handles authentication automatically - Actor users can send requests without configuring API tokens. diff --git a/sources/platform/actors/development/builds_and_runs/runs.md b/sources/platform/actors/development/builds_and_runs/runs.md index 2e2bb0964e..831a9dc46e 100644 --- a/sources/platform/actors/development/builds_and_runs/runs.md +++ b/sources/platform/actors/development/builds_and_runs/runs.md @@ -18,7 +18,7 @@ You can start an Actor in several ways: ## Input and environment variables -The run receives input via the `INPUT` record of its default [key-value store](../../../storage/key_value_store.md). Environment variables are also passed to the run. For more information about environment variables check the [Environment variables](../programming_interface/environment_variables.md) section. +The run receives input via the `INPUT` record of its default [key-value store](/storage/key-value-store). Environment variables are also passed to the run. For more information about environment variables check the [Environment variables](../programming_interface/environment_variables.md) section. ## Run duration and timeout diff --git a/sources/platform/actors/development/programming_interface/basic_commands.md b/sources/platform/actors/development/programming_interface/basic_commands.md index cf8979c033..f40e55bfa9 100644 --- a/sources/platform/actors/development/programming_interface/basic_commands.md +++ b/sources/platform/actors/development/programming_interface/basic_commands.md @@ -98,7 +98,7 @@ Usually, the file is called `INPUT`, but the exact key is defined in the `ACTOR_ ## Key-value store access -Use the [Key-value store](../../../storage/key_value_store.md) to read and write arbitrary files +Use the [Key-value store](/storage/key-value-store) to read and write arbitrary files @@ -148,7 +148,7 @@ async def main(): ## Push results to the dataset -Store larger results in a [Dataset](../../../storage/dataset.md), an append-only object storage +Store larger results in a [Dataset](/storage/dataset), an append-only object storage Note that Datasets can optionally be equipped with the schema that ensures only certain kinds of objects are stored in them. diff --git a/sources/platform/actors/publishing/publish-task.mdx b/sources/platform/actors/publishing/publish-task.mdx index dbb59b7f63..616bb18152 100644 --- a/sources/platform/actors/publishing/publish-task.mdx +++ b/sources/platform/actors/publishing/publish-task.mdx @@ -15,7 +15,7 @@ Before you publish a task, make sure you have: - A [published Actor](./publish.mdx) that you own or maintain. - A [saved task](/actors/running/tasks) with a complete input configuration. -- An [input schema](/actors/development/actor-definition/input-schema) and at least one [dataset schema view](/actors/development/actor-definition/dataset-schema) defined on the Actor. +- An [input schema](/actors/development/actor-definition/input-schema) and at least one [dataset schema view](/storage/dataset-schema) defined on the Actor. ## What tasks to publish @@ -121,7 +121,7 @@ The Dataset schema section selects which view of the Actor's dataset is rendered ![Dataset schema view selection.](./images/public-tasks/task-dataset-schema.webp) -For more on configuring views, see [Dataset schema](/actors/development/actor-definition/dataset-schema). +For more on configuring views, see [Dataset schema](/storage/dataset-schema). ## Task landing page diff --git a/sources/platform/actors/publishing/quality_score.mdx b/sources/platform/actors/publishing/quality_score.mdx index 19133413ff..876097e0a0 100644 --- a/sources/platform/actors/publishing/quality_score.mdx +++ b/sources/platform/actors/publishing/quality_score.mdx @@ -89,4 +89,4 @@ Developers with a proven track record of publishing successful Actors receive re ### Congruency -Congruency measures the consistency and coherence across your Actor's components. A well-designed Actor maintains alignment between its title, description, documentation, and schemas. Ensure that your [input schema](https://docs.apify.com/actors/development/actor-definition/input-schema), [dataset schema](https://docs.apify.com/actors/development/actor-definition/dataset-schema), [key-value store schema](https://docs.apify.com/actors/development/actor-definition/key-value-store-schema), and README documentation all reflect consistent terminology and accurately describe the Actor's behavior. This coherence reduces user confusion and improves the overall experience. +Congruency measures the consistency and coherence across your Actor's components. A well-designed Actor maintains alignment between its title, description, documentation, and schemas. Ensure that your [input schema](https://docs.apify.com/actors/development/actor-definition/input-schema), [dataset schema](https://docs.apify.com/storage/dataset-schema), [key-value store schema](https://docs.apify.com/storage/key-value-store-schema), and README documentation all reflect consistent terminology and accurately describe the Actor's behavior. This coherence reduces user confusion and improves the overall experience. diff --git a/sources/platform/actors/running/monitoring/index.md b/sources/platform/actors/running/monitoring/index.md index 88ae15a7df..b32333e446 100644 --- a/sources/platform/actors/running/monitoring/index.md +++ b/sources/platform/actors/running/monitoring/index.md @@ -45,7 +45,7 @@ When you set up an alert, you have four choices for how you want the metrics to 3. **Alert, when run status is one of following** - This type of alert is checked only after the run finishes. It makes possible to track the status of your finished runs and send an alert if the run finishes in a state you do not expect. If your Actor runs very often and suddenly starts failing, you will receive a single alert after the first failed run in 1 minute, and then aggregated alert every 20 minutes. -4. **Alert for dataset field statistics** - If you have a [dataset schema](/actors/development/actor-definition/dataset-schema/validation) set up, then you can use the field statistics to set up an alert. You can use field statistics for example to track if some field is filled in all records, if some numeric value is too low/high (for example when tracking the price of a product over multiple sources), if the number of items in an array is too low/high (for example alert on Instagram Actor if post has a lot of comments) and many other tasks like these. +4. **Alert for dataset field statistics** - If you have a [dataset schema](/storage/dataset-schema/validation) set up, then you can use the field statistics to set up an alert. You can use field statistics for example to track if some field is filled in all records, if some numeric value is too low/high (for example when tracking the price of a product over multiple sources), if the number of items in an array is too low/high (for example alert on Instagram Actor if post has a lot of comments) and many other tasks like these. :::important diff --git a/sources/platform/actors/running/usage_and_resources.md b/sources/platform/actors/running/usage_and_resources.md index 2aa61396e5..6cb03224c7 100644 --- a/sources/platform/actors/running/usage_and_resources.md +++ b/sources/platform/actors/running/usage_and_resources.md @@ -13,7 +13,7 @@ slug: /actors/running/usage-and-resources Assigning an Actor a specific **Memory** capacity, also determines the allocated CPU power and its disk size. -Check out the [Limits](../../limits.md) page for detailed information on Actor memory, CPU limits, disk size and other limits. +Check out the [Limits](/account/limits) page for detailed information on Actor memory, CPU limits, disk size and other limits. ### Memory @@ -24,7 +24,7 @@ When invoking an Actor, the caller can specify the memory allocation for the Act - The maximum allowed value is `32768MB` - Acceptable values include: `128MB`, `256MB`, `512MB`, `1024MB`, `2048MB`, `4096MB`, `8192MB`, `16384MB`, and `32768MB` -Additionally, each user has a certain total limit of memory for running Actors. The sum of memory allocated for all running Actors and builds needs to be within this limit, otherwise the user cannot start a new Actor. For more details, see [limits](../../limits.md). +Additionally, each user has a certain total limit of memory for running Actors. The sum of memory allocated for all running Actors and builds needs to be within this limit, otherwise the user cannot start a new Actor. For more details, see [limits](/account/limits). ### CPU diff --git a/sources/platform/get-started/building/locally.md b/sources/platform/get-started/building/locally.md index 37dba9c4a2..32c54251b4 100644 --- a/sources/platform/get-started/building/locally.md +++ b/sources/platform/get-started/building/locally.md @@ -113,8 +113,8 @@ The dataset stores a series of data objects from web scraping, crawling, or data You define the Actor output using the Output schema files: -- [Dataset Schema Specification](/actors/development/actor-definition/dataset-schema) -- [Key-value Store Schema Specification](/actors/development/actor-definition/key-value-store-schema) +- [Dataset Schema Specification](/storage/dataset-schema) +- [Key-value Store Schema Specification](/storage/key-value-store-schema) The system uses this to generate an immutable JSON file that tells users where to find the Actor's results. diff --git a/sources/platform/actors/development/actor_definition/dataset_schema/index.md b/sources/platform/storage/dataset/dataset_schema.md similarity index 99% rename from sources/platform/actors/development/actor_definition/dataset_schema/index.md rename to sources/platform/storage/dataset/dataset_schema.md index 5f0b1532e3..811fb2653a 100644 --- a/sources/platform/actors/development/actor_definition/dataset_schema/index.md +++ b/sources/platform/storage/dataset/dataset_schema.md @@ -1,8 +1,8 @@ --- title: Dataset schema specification -sidebar_position: 5 +sidebar_position: 1 description: Define a dataset schema to control how your Actor's results are structured, validated, and displayed in Apify Console, the API, and integrations. -slug: /actors/development/actor-definition/dataset-schema +slug: /storage/dataset-schema sidebar_label: Dataset schema --- @@ -221,7 +221,7 @@ Each view has two parts: The Output tab displays fields from `transformation.fields` in the specified order: -![Output tab UI](../images/output-schema-example.png) +![Output tab UI](../../actors/development/actor_definition/images/output-schema-example.png) ### Multiple views example diff --git a/sources/platform/storage/dataset.md b/sources/platform/storage/dataset/index.md similarity index 88% rename from sources/platform/storage/dataset.md rename to sources/platform/storage/dataset/index.md index 73148214e6..6d21dbecb9 100644 --- a/sources/platform/storage/dataset.md +++ b/sources/platform/storage/dataset/index.md @@ -21,11 +21,11 @@ Named datasets are retained indefinitely. Unnamed datasets expire after 7 days u Dataset storage is _append-only_ - data can only be added and cannot be modified or deleted once stored. -![Dataset graphic](./images/datasets-overview.png) +![Dataset graphic](../images/datasets-overview.png) ## Dataset schema -Actors can define a [dataset schema](/actors/development/actor-definition/dataset-schema) that describes the structure of each dataset item and controls how data appears in the Output tab. The schema has two components: +Actors can define a [dataset schema](/storage/dataset-schema) that describes the structure of each dataset item and controls how data appears in the Output tab. The schema has two components: - `fields` - JSON Schema describing each item's structure, enabling validation and providing metadata for AI agents - `views` - Display configurations that control field ordering, formatting, and presentation in the Console @@ -45,7 +45,7 @@ You can access your datasets in several ways: In [Apify Console](https://console.apify.com), you can view your datasets in the [Storage](https://console.apify.com/storage) section under the [Datasets](https://console.apify.com/storage?tab=datasets) tab. -![Datasets in app](./images/datasets-app.png) +![Datasets in app](../images/datasets-app.png) To view or download a dataset: @@ -55,17 +55,17 @@ To view or download a dataset: Utilize the **Actions** menu to modify the dataset's name, which also affects its [retention period](/storage#data-retention), and to adjust [access rights](/account/collaboration). The **API** button allows you to explore and test the dataset's [API endpoints](/api/v2/storage-datasets). -![Datasets detail view](./images/datasets-detail.png) +![Datasets detail view](../images/datasets-detail.png) ### Apify API The [Apify API](/api/v2/storage-datasets) enables you programmatic access to your datasets using [HTTP requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods). -If you are accessing your datasets using the `username~store-name` [store ID format](./index.md), you will need to use your secret API token. You can find the token (and your user ID) on the [API & Integrations](https://console.apify.com/settings/integrations) tab of **Settings** page of your Apify account. +If you are accessing your datasets using the `username~store-name` [store ID format](../index.md), you will need to use your secret API token. You can find the token (and your user ID) on the [API & Integrations](https://console.apify.com/settings/integrations) tab of **Settings** page of your Apify account. :::tip Pass tokens in the Authorization header -When providing your API authentication token, we recommend using the request's `Authorization` header, rather than the URL. [More info](../integrations/programming/api.md#authentication). +When providing your API authentication token, we recommend using the request's `Authorization` header, rather than the URL. [More info](../../integrations/programming/api.md#authentication). ::: @@ -185,7 +185,7 @@ Check out the [Python API client documentation](/api/client/python/reference/cla #### JavaScript SDK -When working with a JavaScript [Actor](../actors/index.mdx), the [JavaScript SDK](/sdk/js/docs/guides/result-storage#dataset) is an essential tool, especially for dataset management. It simplifies the tasks of storing and retrieving data, seamlessly integrating with the Actor's workflow. Key features of the SDK include the ability to append data, retrieve what is stored, and manage dataset properties effectively. Central to this functionality is the [`Dataset`](/sdk/js/reference/class/Dataset) class. This class allows you to determine where your data is stored - locally or in the Apify cloud. To add data to your chosen datasets, use the [`pushData()`](/sdk/js/reference/class/Dataset#pushData) method. +When working with a JavaScript [Actor](../../actors/index.mdx), the [JavaScript SDK](/sdk/js/docs/guides/result-storage#dataset) is an essential tool, especially for dataset management. It simplifies the tasks of storing and retrieving data, seamlessly integrating with the Actor's workflow. Key features of the SDK include the ability to append data, retrieve what is stored, and manage dataset properties effectively. Central to this functionality is the [`Dataset`](/sdk/js/reference/class/Dataset) class. This class allows you to determine where your data is stored - locally or in the Apify cloud. To add data to your chosen datasets, use the [`pushData()`](/sdk/js/reference/class/Dataset#pushData) method. Additionally the SDK offers other methods like [`getData()`](/sdk/js/reference/class/Dataset#getData), [`map()`](/sdk/js/reference/class/Dataset#map), and [`reduce()`](/sdk/js/reference/class/Dataset#reduce). For practical applications of these methods, refer to the [example](/sdk/js/docs/examples/map-and-reduce) section. @@ -263,7 +263,7 @@ Check out the [JavaScript SDK documentation](/sdk/js/docs/guides/result-storage# #### Python SDK -For Python [Actors](../actors/index.mdx), the [Python SDK](/sdk/python/docs/concepts/storages#working-with-datasets) is essential. The dataset is represented by a [`Dataset`](/sdk/python/reference/class/Dataset) class. You can use this class to specify whether your data is stored locally or in the Apify cloud and push data to the datasets of your choice using the [`push_data()`](/sdk/python/reference/class/Dataset#push_data) method. For further data manipulation you could also use other methods such as [`get_data()`](/sdk/python/reference/class/Dataset#get_data), [`map()`](/sdk/python/reference/class/Dataset#map) and [`reduce()`](/sdk/python/reference/class/Dataset#reduce). +For Python [Actors](../../actors/index.mdx), the [Python SDK](/sdk/python/docs/concepts/storages#working-with-datasets) is essential. The dataset is represented by a [`Dataset`](/sdk/python/reference/class/Dataset) class. You can use this class to specify whether your data is stored locally or in the Apify cloud and push data to the datasets of your choice using the [`push_data()`](/sdk/python/reference/class/Dataset#push_data) method. For further data manipulation you could also use other methods such as [`get_data()`](/sdk/python/reference/class/Dataset#get_data), [`map()`](/sdk/python/reference/class/Dataset#map) and [`reduce()`](/sdk/python/reference/class/Dataset#reduce). For datasets stored locally, the data is located at the following path: @@ -426,7 +426,7 @@ For one-off sharing of specific records when access is restricted, you can gener ### Share datasets between runs -You can access a dataset from any [Actor](../actors/index.mdx) or [task](../actors/running/tasks.md) run as long as you know its _name_ or _ID_. +You can access a dataset from any [Actor](../../actors/index.mdx) or [task](../../actors/running/tasks.md) run as long as you know its _name_ or _ID_. To access a dataset from another run using the [Apify SDK](/sdk), open it using the same method as you would with any other dataset. diff --git a/sources/platform/actors/development/actor_definition/dataset_schema/multiple_datasets.mdx b/sources/platform/storage/dataset/multiple_datasets.mdx similarity index 90% rename from sources/platform/actors/development/actor_definition/dataset_schema/multiple_datasets.mdx rename to sources/platform/storage/dataset/multiple_datasets.mdx index 4396ccd437..5746f868e7 100644 --- a/sources/platform/actors/development/actor_definition/dataset_schema/multiple_datasets.mdx +++ b/sources/platform/storage/dataset/multiple_datasets.mdx @@ -1,8 +1,8 @@ --- title: Multiple datasets description: Learn how to use multiple datasets within your Actors to organize and store different types of data separately. -sidebar_position: 2 -slug: /actors/development/actor-definition/dataset-schema/multiple-datasets +sidebar_position: 3 +slug: /storage/dataset-schema/multiple-datasets --- import Tabs from '@theme/Tabs'; @@ -52,7 +52,7 @@ Behavior differs when an SDK runs outside the platform. See the SDK notes below. ::: -See the full [Actor schema reference](../actor_json.md#reference). +See the full [Actor schema reference](/actors/development/actor-definition/actor-json#reference). ## Access datasets in Actor code @@ -127,11 +127,11 @@ To export programmatically: - Call the [Dataset API](/api/v2/dataset-items-get) with the dataset ID from `ACTOR_STORAGES_JSON`. The API returns items in any supported format via query parameters. - From inside an Actor, open the dataset (see [Access datasets in Actor code](#access-datasets-in-actor-code)), then call `getData` / `get_data` to read items into memory, or `exportTo` / `export_to` to write a JSON or CSV file to the key-value store. -See [Datasets](../../../../storage/dataset.md) for formats and query parameters. +See [Datasets](/storage/dataset) for formats and query parameters. ## Surface datasets on the run page -The Storage tab shows data but doesn't surface it clearly to end users. To present datasets more prominently on the run page, define an [output schema](../../actor_definition/output_schema/index.md) that references each dataset by alias: +The Storage tab shows data but doesn't surface it clearly to end users. To present datasets more prominently on the run page, define an [output schema](/actors/development/actor-definition/output-schema) that references each dataset by alias: ```json { @@ -152,10 +152,10 @@ The Storage tab shows data but doesn't surface it clearly to end users. To prese } ``` -[Read more](../output_schema/index.md#how-templates-work) about how templates work. +[Read more](/actors/development/actor-definition/output-schema#how-templates-work) about how templates work. ## Billing for non-default datasets When an Actor uses multiple datasets, only items pushed to the `default` dataset trigger the built-in `apify-default-dataset-item` event. Items in other datasets are not charged automatically. -To charge for items in other datasets, implement custom billing in your Actor code. Refer to the [billing documentation](../../../publishing/monetize/pay_per_event.mdx) for implementation details. +To charge for items in other datasets, implement custom billing in your Actor code. Refer to the [billing documentation](/actors/publishing/monetize/pay-per-event) for implementation details. diff --git a/sources/platform/actors/development/actor_definition/dataset_schema/validation.md b/sources/platform/storage/dataset/validation.md similarity index 96% rename from sources/platform/actors/development/actor_definition/dataset_schema/validation.md rename to sources/platform/storage/dataset/validation.md index 8102adbeb5..ccd3c8675f 100644 --- a/sources/platform/actors/development/actor_definition/dataset_schema/validation.md +++ b/sources/platform/storage/dataset/validation.md @@ -1,8 +1,8 @@ --- title: Dataset validation description: Specify the dataset schema within your Actors to add monitoring and validation at the field level, ensuring data quality and consistency across runs. -sidebar_position: 1 -slug: /actors/development/actor-definition/dataset-schema/validation +sidebar_position: 2 +slug: /storage/dataset-schema/validation --- import Tabs from '@theme/Tabs'; @@ -80,7 +80,7 @@ Dataset schema needs to be a valid JSON schema draft-07, so the `$schema` line i ## Dataset validation -When you define a schema of your default dataset, the schema is then always used when you insert data into the dataset to perform validation (we use [AJV](https://ajv.js.org/)). +When you define a schema of your default dataset, the schema is then always used when you insert data into the dataset to perform validation with [AJV](https://ajv.js.org/). If the validation succeeds, nothing changes from the current behavior, data is stored and an empty response with status code `201` is returned. @@ -236,7 +236,7 @@ You can also use [conversion tools](https://www.liquid-technologies.com/online-j ## Dataset field statistics -When you configure the dataset fields schema, we generate a field list and measure the following statistics: +When you configure the dataset fields schema, Apify generates a field list and measures the following statistics: - **Null count:** how many items in the dataset have the field set to null - **Empty count:** how many items in the dataset are `undefined` , meaning that for example empty string is not considered empty diff --git a/sources/platform/storage/index.md b/sources/platform/storage/index.md index da913f1041..5b465fdff2 100644 --- a/sources/platform/storage/index.md +++ b/sources/platform/storage/index.md @@ -195,7 +195,7 @@ If your storage resource is set to _restricted_, all API calls must include a va Storage can be accessed from any [Actor](../actors/index.mdx) or [task](../actors/running/tasks.md) run, provided you have its _name_ or _ID_. Use the same methods and endpoints you'd use for the current run's storages. -[Datasets](./dataset.md) and [key-value stores](./key_value_store.md) support concurrent use. Multiple Actors or tasks can write to the same dataset or key-value store, and multiple runs can read from them at the same time. +[Datasets](/storage/dataset) and [key-value stores](/storage/key-value-store) support concurrent use. Multiple Actors or tasks can write to the same dataset or key-value store, and multiple runs can read from them at the same time. [Request queues](./request_queue.md), on the other hand, only allow multiple runs to add new data. A request queue can only be processed by one Actor or task run at any one time. diff --git a/sources/platform/actors/development/actor_definition/key_value_store_schema/images/kv-store-schema-example-run.png b/sources/platform/storage/key_value_store/images/kv-store-schema-example-run.png similarity index 100% rename from sources/platform/actors/development/actor_definition/key_value_store_schema/images/kv-store-schema-example-run.png rename to sources/platform/storage/key_value_store/images/kv-store-schema-example-run.png diff --git a/sources/platform/actors/development/actor_definition/key_value_store_schema/images/kv-store-schema-example-storage.png b/sources/platform/storage/key_value_store/images/kv-store-schema-example-storage.png similarity index 100% rename from sources/platform/actors/development/actor_definition/key_value_store_schema/images/kv-store-schema-example-storage.png rename to sources/platform/storage/key_value_store/images/kv-store-schema-example-storage.png diff --git a/sources/platform/storage/key_value_store.md b/sources/platform/storage/key_value_store/index.md similarity index 82% rename from sources/platform/storage/key_value_store.md rename to sources/platform/storage/key_value_store/index.md index 5750740ab3..783dbe5a43 100644 --- a/sources/platform/storage/key_value_store.md +++ b/sources/platform/storage/key_value_store/index.md @@ -21,7 +21,7 @@ Named key-value stores are retained indefinitely. Unnamed key-value stores expir ::: -![Key-value store graphic](./images/key-value-overview.svg) +![Key-value store graphic](../images/key-value-overview.svg) ## Basic usage @@ -36,12 +36,12 @@ You can access key-value stores through several methods In [Apify Console](https://console.apify.com), you can view your key-value stores in the [Storage](https://console.apify.com/storage) section under the [Key-value stores](https://console.apify.com/storage?tab=keyValueStores) tab. -![Key-value stores in app](./images/key-value-stores-app.png) +![Key-value stores in app](../images/key-value-stores-app.png) To view a key-value store's content, click on its **Store ID**. Under the **Actions** menu, you can rename your store (which extends its [retention period](/storage#named-and-unnamed-storages)) and grant [access rights](/account/collaboration) using the **Share** button. Click on the **API** button to view and test a store's [API endpoints](/api/v2/storage-key-value-stores). -![Key-value stores detail](./images/key-value-stores-detail-header.png) +![Key-value stores detail](../images/key-value-stores-detail-header.png) At the bottom of the page, you can work with records in your key-value store: @@ -51,17 +51,17 @@ At the bottom of the page, you can work with records in your key-value store: - Copy shareable links to records. - Delete records. -![Key-value stores detail](./images/key-value-stores-detail-records.png) +![Key-value stores detail](../images/key-value-stores-detail-records.png) ### Apify API The [Apify API](/api/v2/storage-key-value-stores) enables you programmatic access to your key-value stores using [HTTP requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods). -If you are accessing your key-value stores using the `username~store-name` [store ID format](./index.md), you will need to use your secret API token. You can find the token (and your user ID) on the [API & Integrations](https://console.apify.com/settings/integrations) tab of **Settings** page of your Apify account. +If you are accessing your key-value stores using the `username~store-name` [store ID format](../index.md), you will need to use your secret API token. You can find the token (and your user ID) on the [API & Integrations](https://console.apify.com/settings/integrations) tab of **Settings** page of your Apify account. :::tip Authentication -When providing your API authentication token, we recommend using the request's `Authorization` header, rather than the URL. For more information, refer to the [API integration](../integrations/programming/api.md#authentication) documentation. +When providing your API authentication token, we recommend using the request's `Authorization` header, rather than the URL. For more information, refer to the [API integration](../../integrations/programming/api.md#authentication) documentation. ::: @@ -142,11 +142,11 @@ Check out the [Python API client documentation](/api/client/python/reference/cla #### JavaScript SDK -When working with a JavaScript [Actor](../actors/index.mdx), the [JavaScript SDK](/sdk/js/docs/guides/result-storage#key-value-store) is an essential tool, especially for key-value store management. The primary class for this purpose is the [`KeyValueStore`](/sdk/js/reference/class/KeyValueStore). This class allows you to decide whether your data will be stored locally or in the Apify cloud. For data manipulation, it offers the [`getValue()`](/sdk/js/reference/class/KeyValueStore#getValue) and [`setValue()`](/sdk/js/reference/class/KeyValueStore#setValue) methods to retrieve and assign values, respectively. +When working with a JavaScript [Actor](../../actors/index.mdx), the [JavaScript SDK](/sdk/js/docs/guides/result-storage#key-value-store) is an essential tool, especially for key-value store management. The primary class for this purpose is the [`KeyValueStore`](/sdk/js/reference/class/KeyValueStore). This class allows you to decide whether your data will be stored locally or in the Apify cloud. For data manipulation, it offers the [`getValue()`](/sdk/js/reference/class/KeyValueStore#getValue) and [`setValue()`](/sdk/js/reference/class/KeyValueStore#setValue) methods to retrieve and assign values, respectively. Additionally, you can iterate over the keys in your store using the [`forEachKey()`](/sdk/js/reference/class/KeyValueStore#forEachKey) method. -Every Actor run is linked to a default key-value store that is automatically created for that specific run. If you're running your Actors and opt to store data locally, you can easily supply the [input](../actors/running/input_and_output.md) by placing an _INPUT.json_ file in the corresponding directory of the default key-value store. This method ensures that your Actor has all the necessary data readily available for its execution. +Every Actor run is linked to a default key-value store that is automatically created for that specific run. If you're running your Actors and opt to store data locally, you can easily supply the [input](../../actors/running/input_and_output.md) by placing an _INPUT.json_ file in the corresponding directory of the default key-value store. This method ensures that your Actor has all the necessary data readily available for its execution. You can find _INPUT.json_ and other key-value store files in the location below. @@ -206,15 +206,15 @@ await Actor.setValue('OUTPUT', imageBuffer, { contentType: 'image/jpeg' }); await Actor.exit(); ``` -The `Actor.getInput()` method is not only a shortcut to `Actor.getValue('INPUT')`; it is also compatible with [`Actor.metamorph()`](../actors/development/programming_interface/metamorph.md). This is because a metamorphed Actor run's input is stored in the _INPUT-METAMORPH-1_ key instead of _INPUT_, which hosts the original input. +The `Actor.getInput()` method is not only a shortcut to `Actor.getValue('INPUT')`; it is also compatible with [`Actor.metamorph()`](../../actors/development/programming_interface/metamorph.md). This is because a metamorphed Actor run's input is stored in the _INPUT-METAMORPH-1_ key instead of _INPUT_, which hosts the original input. Check out the [JavaScript SDK documentation](/sdk/js/docs/guides/result-storage#key-value-store) and the `KeyValueStore` class's [API reference](/sdk/js/reference/class/KeyValueStore) for details on managing your key-value stores with the JavaScript SDK. #### Python SDK -For Python [Actor](../actors/index.mdx), the [Python SDK](/sdk/python/docs/concepts/storages#working-with-key-value-stores) is essential. The key-value store is represented by a [`KeyValueStore`](/sdk/python/reference/class/KeyValueStore) class. You can use this class to specify whether your data is stored locally or in the Apify cloud. For further data manipulation it offers [`get_value()`](/sdk/python/reference/class/KeyValueStore#get_value) and [`set_value()`](/sdk/python/reference/class/KeyValueStore#set_value) methods to retrieve and assign values, respectively. +For Python [Actor](../../actors/index.mdx), the [Python SDK](/sdk/python/docs/concepts/storages#working-with-key-value-stores) is essential. The key-value store is represented by a [`KeyValueStore`](/sdk/python/reference/class/KeyValueStore) class. You can use this class to specify whether your data is stored locally or in the Apify cloud. For further data manipulation it offers [`get_value()`](/sdk/python/reference/class/KeyValueStore#get_value) and [`set_value()`](/sdk/python/reference/class/KeyValueStore#set_value) methods to retrieve and assign values, respectively. -Every Actor run is linked to a default key-value store that is automatically created for that specific run. If you're running your Actors and opt to store data locally, you can easily supply the [input](../actors/running/input_and_output.md) by placing an _INPUT.json_ file in the corresponding directory of the default key-value store. This method ensures that your Actor has all the necessary data readily available for its execution. +Every Actor run is linked to a default key-value store that is automatically created for that specific run. If you're running your Actors and opt to store data locally, you can easily supply the [input](../../actors/running/input_and_output.md) by placing an _INPUT.json_ file in the corresponding directory of the default key-value store. This method ensures that your Actor has all the necessary data readily available for its execution. You can find _INPUT.json_ and other key-value store files in the location below. @@ -262,7 +262,7 @@ async def main(): await Actor.set_value(key='OUTPUT', value=image_buffer, content_type='image/jpeg') ``` -The `Actor.get_input()` method is not only a shortcut to `Actor.get_value('INPUT')`; it is also compatible with [`Actor.metamorph()`](../actors/development/programming_interface/metamorph.md). This is because a metamorphed Actor run's input is stored in the _INPUT-METAMORPH-1_ key instead of _INPUT_, which hosts the original input. +The `Actor.get_input()` method is not only a shortcut to `Actor.get_value('INPUT')`; it is also compatible with [`Actor.metamorph()`](../../actors/development/programming_interface/metamorph.md). This is because a metamorphed Actor run's input is stored in the _INPUT-METAMORPH-1_ key instead of _INPUT_, which hosts the original input. Check out the [Python SDK documentation](/sdk/python/docs/concepts/storages#working-with-key-value-stores) and the `KeyValueStore` class's [API reference](/sdk/python/reference/class/KeyValueStore) for details on managing your key-value stores with the Python SDK. @@ -284,7 +284,7 @@ For one-off sharing of specific records when access is restricted, you can gener ### Share key-value stores between runs -You can access a key-value store from any [Actor](../actors/index.mdx) or [task](../actors/running/tasks.md) run as long as you know its _name_ or _ID_. +You can access a key-value store from any [Actor](../../actors/index.mdx) or [task](../../actors/running/tasks.md) run as long as you know its _name_ or _ID_. To access a key-value store from another run using the [Apify SDK](/sdk), open it using the same method as you would do with any other store. diff --git a/sources/platform/actors/development/actor_definition/key_value_store_schema/index.md b/sources/platform/storage/key_value_store/key_value_store_schema.md similarity index 69% rename from sources/platform/actors/development/actor_definition/key_value_store_schema/index.md rename to sources/platform/storage/key_value_store/key_value_store_schema.md index 59d0527e52..50f88a6873 100644 --- a/sources/platform/actors/development/actor_definition/key_value_store_schema/index.md +++ b/sources/platform/storage/key_value_store/key_value_store_schema.md @@ -1,16 +1,102 @@ --- title: Key-value store schema specification sidebar_label: Key-value store schema -sidebar_position: 6 +sidebar_position: 1 description: Define a key-value store schema to organize records into collections with content type validation, making your Actor's key-value store easier to navigate. -slug: /actors/development/actor-definition/key-value-store-schema +slug: /storage/key-value-store-schema --- The key‑value store schema organizes keys into logical groups called collections, which can be used to filter and categorize data both in the API and the visual user interface. This organization helps users navigate and find specific data more efficiently, while schema‑defined rules (such as content types and JSON schema) ensure that stored values remain consistent and valid. -## Example +## Schema components -Consider an example Actor that calls `Actor.setValue()` to save a record into the key-value store: +A key-value store schema has one component: + +- `collections` _(required)_ - Named groups of keys, each defined by a shared key prefix or a single key. A collection can also validate the content type of its records and, for JSON, validate them against a JSON Schema. + +```json title=".actor/key_value_store_schema.json" +{ + "actorKeyValueStoreSchemaVersion": 1, + "title": "Key-value store schema", + "collections": { + "documents": { + "title": "Documents", + "keyPrefix": "document-" + } + } +} +``` + +## File structure + +Place the key-value store schema in the `.actor` folder in your Actor's root directory. You can organize it in two ways: + +### Inline in `actor.json` + +```json title=".actor/actor.json" +{ + "actorSpecification": 1, + "name": "this-is-book-library-scraper", + "title": "Book Library scraper", + "version": "1.0.0", + "storages": { + "keyValueStore": { + "actorKeyValueStoreSchemaVersion": 1, + "title": "Key-Value Store Schema", + "collections": { /* Define your collections here */ } + } + } +} +``` + +### Separate file + +```json title=".actor/actor.json" +{ + "actorSpecification": 1, + "name": "this-is-book-library-scraper", + "title": "Book Library scraper", + "version": "1.0.0", + "storages": { + "keyValueStore": "./key_value_store_schema.json" + } +} +``` + +```json title=".actor/key_value_store_schema.json" +{ + "actorKeyValueStoreSchemaVersion": 1, + "title": "Key-Value Store Schema", + "collections": { /* Define your collections here */ } +} +``` + +Use a separate file when your schema is complex or you want to keep `actor.json` concise. + +## Collections + +Collections group related keys so they are easier to find and filter in Apify Console and through the API. Define each collection by a shared key prefix or a single key, and optionally validate the records it holds. + +### Define collection membership + +Each collection defines its member keys using one of the following properties: + +- `keyPrefix` - Includes all keys that start with the specified prefix (for example, all keys starting with `document-`). +- `key` - Includes a single specific key. + +Use either `key` or `keyPrefix` for each collection, but not both. + +### Validate record content + +When you define a collection with specific `contentTypes`, the Apify platform validates any data stored in that collection against those specifications. For example, if you specify that a collection should only contain JSON data with content type `application/json`, attempts to store data with other content types in that collection are rejected. + +The validation happens automatically when you call `Actor.setValue()` or use the [Put record](https://docs.apify.com/api/v2/key-value-store-record-put) API endpoint. + +If you define a `jsonSchema` for a collection with content type `application/json`, the platform also validates that the JSON data conforms to the specified schema. This helps ensure data consistency and prevents storing malformed data. + +### Collection example + +Consider an Actor that calls `Actor.setValue()` to save records into the key-value store: ```javascript title="main.js" import { Actor } from 'apify'; @@ -30,7 +116,7 @@ await Actor.setValue(`image-${imageID}`, imageBuffer, { contentType: 'image/jpeg await Actor.exit(); ``` -To configure the key-value store schema, use the following template for the `.actor/actor.json` configuration: +To organize those records, define collections in the `.actor/actor.json` configuration: ```json title=".actor/actor.json" { @@ -60,15 +146,9 @@ To configure the key-value store schema, use the following template for the `.ac } ``` -The template above defines the configuration for the default key-value store. -Each collection can define its member keys using one of the following properties: +This defines two collections for the default key-value store: `documents` for text keys prefixed `document-`, and `images` for JPEG keys prefixed `image-`. -- `keyPrefix` - All keys starting with the specified prefix will be included in the collection (e.g., all keys starting with "document-"). -- `key` - A specific individual key that will be included in the collection. - -You must use either `key` or `keyPrefix` for each collection, but not both. - -Once the schema is defined, tabs for each collection will appear in the **Storage** tab of the Actor's run: +Once the schema is defined, tabs for each collection appear in the **Storage** tab of the Actor's run: ![Storages tab in Run](images/kv-store-schema-example-run.png) @@ -76,9 +156,9 @@ The tabs also appear in the storage detail view: ![Storage detail](images/kv-store-schema-example-storage.png) -### API Example +### List keys by collection via the API -With the key-value store schema defined, you can use the API to list keys from a specific collection by using the `collection` query parameter when calling the [Get list of keys](https://docs.apify.com/api/v2/key-value-store-keys-get) endpoint: +With the schema defined, use the API to list keys from a specific collection with the `collection` query parameter on the [Get list of keys](https://docs.apify.com/api/v2/key-value-store-keys-get) endpoint: ```http title="Get list of keys from a collection" GET https://api.apify.com/v2/key-value-stores/{storeId}/keys?collection=documents @@ -113,67 +193,11 @@ You can also filter by key prefix using the `prefix` parameter: GET https://api.apify.com/v2/key-value-stores/{storeId}/keys?prefix=document- ``` -### Schema Validation - -When you define a key-value store schema with specific `contentTypes` for collections, the Apify platform validates any data being stored against these specifications. For example, if you've specified that a collection should only contain JSON data with content type `application/json`, attempts to store data with other content types in that collection will be rejected. - -The validation happens automatically when you call `Actor.setValue()` or use the [Put record](https://docs.apify.com/api/v2/key-value-store-record-put) API endpoint. - -If you've defined a `jsonSchema` for a collection with content type `application/json`, the platform will also validate that the JSON data conforms to the specified schema. This helps ensure data consistency and prevents storing malformed data. - -## Structure - -Output configuration files need to be located in the `.actor` folder within the Actor's root directory. - -You have two choices of how to organize files within the `.actor` folder. - -### Single configuration file - -```json title=".actor/actor.json" -{ - "actorSpecification": 1, - "name": "this-is-book-library-scraper", - "title": "Book Library scraper", - "version": "1.0.0", - "storages": { - "keyValueStore": { - "actorKeyValueStoreSchemaVersion": 1, - "title": "Key-Value Store Schema", - "collections": { /* Define your collections here */ } - } - } -} -``` - -### Separate configuration files - -```json title=".actor/actor.json" -{ - "actorSpecification": 1, - "name": "this-is-book-library-scraper", - "title": "Book Library scraper", - "version": "1.0.0", - "storages": { - "keyValueStore": "./key_value_store_schema.json" - } -} -``` - -```json title=".actor/key_value_store_schema.json" -{ - "actorKeyValueStoreSchemaVersion": 1, - "title": "Key-Value Store Schema", - "collections": { /* Define your collections here */ } -} -``` - -Choose the method that best suits your configuration. - -## Key-value store schema structure definitions +## Reference The key-value store schema defines the collections of keys and their properties. It allows you to organize and validate data stored by the Actor, making it easier to manage and retrieve specific records. -### Key-value store schema object definition +### `KeyValueStoreSchema` object | Property | Type | Required | Description | | --------------------------------- | ------- | -------- | --------------------------------------------------------------------------------------------------------------- | @@ -182,7 +206,7 @@ The key-value store schema defines the collections of keys and their properties. | `description` | string | false | Description of the schema | | `collections` | Object | true | An object where each key is a collection ID and its value is a collection definition object (see below). | -### Collection object definition +### `Collection` object | Property | Type | Required | Description | | -------------- | ------------ | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ | diff --git a/sources/platform/storage/request_queue.md b/sources/platform/storage/request_queue.md index ef27d66195..08dd000fa8 100644 --- a/sources/platform/storage/request_queue.md +++ b/sources/platform/storage/request_queue.md @@ -287,7 +287,7 @@ Check out the [Python SDK documentation](/sdk/python/docs/concepts/storages#work Request queue is a storage type built with scraping in mind, enabling developers to write scraping logic efficiently and scalably. The Apify tooling, including [Crawlee](https://crawlee.dev/), [Apify SDK for JavaScript](https://docs.apify.com/sdk/js/), and [Apify SDK for Python](https://docs.apify.com/sdk/python/), incorporates all these features, enabling users to leverage them effortlessly without extra configuration. -In the following section, we will discuss each of the main features in depth. +The following sections cover each of the main features in depth. ### Persistence and retention From ea9182a7bc2f587b35da687d7b334aabdb9a1475 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C5=82=20Olender?= <92638966+TC-MO@users.noreply.github.com> Date: Tue, 7 Jul 2026 20:56:01 +0200 Subject: [PATCH 2/2] refactor: move Limits from top nav into Account (#2746) Move the Limits page from the top navigation into the Account section (/limits -> /account/limits) and add an nginx redirect. Remove the top-nav entry from the shared theme. --- apify-docs-theme/src/config.js | 8 -------- apify-docs-theme/src/theme/DocItemContent/index.js | 1 - apify-docs-theme/src/theme/SearchBar/index.js | 1 - nginx.conf | 2 ++ sources/platform/{ => account}/limits.md | 5 ++--- .../actors/development/programming_interface/metamorph.md | 2 +- 6 files changed, 5 insertions(+), 14 deletions(-) rename sources/platform/{ => account}/limits.md (98%) diff --git a/apify-docs-theme/src/config.js b/apify-docs-theme/src/config.js index 634b1ca7c9..9b8ceab6b9 100644 --- a/apify-docs-theme/src/config.js +++ b/apify-docs-theme/src/config.js @@ -67,14 +67,6 @@ const themeConfig = { target: '_self', rel: 'dofollow', }, - { - label: 'Limits', - href: `${absoluteUrl}/limits`, - activeBasePath: 'limits', - position: 'left', - target: '_self', - rel: 'dofollow', - }, { label: 'Security', href: `${absoluteUrl}/security`, diff --git a/apify-docs-theme/src/theme/DocItemContent/index.js b/apify-docs-theme/src/theme/DocItemContent/index.js index 219637daf8..e71749a587 100644 --- a/apify-docs-theme/src/theme/DocItemContent/index.js +++ b/apify-docs-theme/src/theme/DocItemContent/index.js @@ -57,7 +57,6 @@ export default function DocItemContent({ children }) { '/proxy', '/account', '/integrations', - '/limits', '/security', '/sdk', '/cli', diff --git a/apify-docs-theme/src/theme/SearchBar/index.js b/apify-docs-theme/src/theme/SearchBar/index.js index 70837cd564..eb17f666ca 100644 --- a/apify-docs-theme/src/theme/SearchBar/index.js +++ b/apify-docs-theme/src/theme/SearchBar/index.js @@ -25,7 +25,6 @@ function matchesCurrentInstance(href, baseUrl) { href.startsWith('/proxy') || href.startsWith('/account') || href.startsWith('/integrations') || - href.startsWith('/limits') || href.startsWith('/security') ); } diff --git a/nginx.conf b/nginx.conf index f9e308d4d2..d643a3cc95 100644 --- a/nginx.conf +++ b/nginx.conf @@ -812,6 +812,8 @@ server { # 302 while the IA settles; promote to permanent once stable. rewrite ^/actors/development/actor-definition/dataset-schema(/.*)?$ /storage/dataset-schema$1 redirect; rewrite ^/actors/development/actor-definition/key-value-store-schema$ /storage/key-value-store-schema redirect; + # Limits moved from top-nav (/limits) into Account. + rewrite ^/limits$ /account/limits redirect; # Soft-landings for deleted URLs (better UX than a 404) rewrite ^/platform/integrations/airtable/console-integration$ /integrations/airtable permanent; diff --git a/sources/platform/limits.md b/sources/platform/account/limits.md similarity index 98% rename from sources/platform/limits.md rename to sources/platform/account/limits.md index 8de676cf19..6e976e8059 100644 --- a/sources/platform/limits.md +++ b/sources/platform/account/limits.md @@ -1,9 +1,8 @@ --- title: Limits description: View default resource limits for the Apify platform, including max memory, disk size, and number of Actors and tasks per user per plan tier. -sidebar_position: 16 -category: platform -slug: /limits +sidebar_position: 3.5 +slug: /account/limits --- The tables below demonstrate the Apify platform's default resource limits. For API limits such as rate limits and max payload size, see the [API documentation](/api/v2#rate-limiting). diff --git a/sources/platform/actors/development/programming_interface/metamorph.md b/sources/platform/actors/development/programming_interface/metamorph.md index e470586622..13b92625d5 100644 --- a/sources/platform/actors/development/programming_interface/metamorph.md +++ b/sources/platform/actors/development/programming_interface/metamorph.md @@ -35,7 +35,7 @@ To make your Actor compatible with metamorph, use `Actor.getInput()` instead of :::note Runtime limits -There's a limit to how many times you can metamorph a single run. Refer to the [Actor runtime limits](/limits#actor-limits) for more details. +There's a limit to how many times you can metamorph a single run. Refer to the [Actor runtime limits](/account/limits#actor-limits) for more details. :::