diff --git a/docs/07-mypy-plugin.md b/docs/07-mypy-plugin.md new file mode 100644 index 0000000..bff891b --- /dev/null +++ b/docs/07-mypy-plugin.md @@ -0,0 +1,207 @@ +# 7. Type checking with mypy + +## What's the problem? + +Due to the way the library is designed, type checkers have a hard time with validataclasses. + +For example, if we have a typical validataclass definition like this... + +```python +from validataclass.dataclasses import validataclass, Default +from validataclass.validators import IntegerValidator + +@validataclass +class ExampleDataclass: + field1: int = IntegerValidator() + field2: int | None = IntegerValidator(), Default(None) +``` + +... type checkers will point out that the assignments in this class are incorrect. + +Because from the perspective of a type checker, this class doesn't make a lot of sense. We're assigning an object of +type `IntegerValidator` to `field1`, which is typed as `int` - but a validator is not an integer. It gets even worse +with field `field2` where we're assigning a tuple consisting of two seemingly random objects to an `int | None` field. +A `tuple[IntegerValidator, Default[None]]` is neither an `int` nor `None`. + +Of course at runtime, the class is still correct, because the `@validataclass` decorator transforms the class and its +weird assignments to a simple dataclass and stores the validataclass-specific information in metadata. + +Type checkers like mypy and pyright cannot know what the decorator does (except that the decorators are dataclass-like +transformers since they're decorated with `typing.dataclass_transform` - but sadly that's not enough here). + +However, mypy has a plugin system exactly for this purpose. This is very specific to mypy, though, so if you want to use +a different type checker than mypy, I'm afraid you're out of luck. + + +## How to use the mypy plugin + +The mypy plugin is included in the library, you only need to enable it in your mypy configuration. + +If you're using a `pyproject.toml` file (which is the recommended way nowadays to configure Python projects and tools), +this is an example for a mypy configuration: + +```toml +[tool.mypy] +# These lines are just an example and might not be needed or need to be adjusted in your project: +files = ["src/"] +mypy_path = "src/" +explicit_package_bases = true + +# This is the important part: +plugins = ["validataclass.mypy.plugin"] +``` + + +### Advanced plugin configuration + +There are some advanced settings for configuring the plugin itself. The plugin configuration currently only supports +`pyproject.toml` and should be defined in the section `[tool.validataclass_mypy]`. + +The configuration is expected to be in the same `pyproject.toml` file that is used to configure mypy. The plugin does +not have any discovery mechanisms for config files, it simply looks up which config file was read by mypy during +initialization, and if it's a `.toml` file, it reads the same file to get its configuration. + +#### Strictness / optional rules + +- `allow_incompatible_field_overrides` (bool, default `true`): Allow incompatible overrides of fields in validataclass + sub classes, i.e. changing the type of an existing field in a sub class. This is enabled by default because it is a + common use case to define sub classes where, for example, only the defaults are changed. The default might be changed + in a future version, so it's advisable to set this explicitly to true if you need it. + +#### Extensions + +The following settings can be used to enable support for custom extensions of the library, for example if you've +defined your own decorator that extends `@validataclass` or you're using another library that does this. + +All these settings require fully qualified names, e.g. `example.module.my_custom_decorator`. + +- `custom_validataclass_decorators` (list of strings): Custom decorators for creating validataclasses. + - These are basically treated as aliases to `@validataclass`, so their signature and behaviour should be compatible to + the original decorator. + - They are also required to be decorated with `@typing.dataclass_transform(kw_only_default=True)`. +- `custom_field_functions` (list of strings): Custom functions that create validataclass fields. + - These must have the a compatible signature to the built-in `validataclass_field()` function, e.g. they must accept + a validator (as the only positional argument or as a named argument `validator`). +- `ignore_custom_types_in_fields` (list of strings): Custom base classes that are ignored when analyzing a field tuple. + - By default, only instances of `Validator` or `BaseDefault` are valid types in a field definition. Other types will + be reported as errors. With this option, you can have custom objects that get ignored by the plugin. + - Keep in mind that the `@validataclass` decorator will still reject those objects at runtime. This option only makes + sense in combination with a custom decorator, see `custom_validataclass_decorators`. + - All subclasses of classes in this list will be ignored as well. + +#### Miscellaneous + +- `debug_mode` (bool, default `false`): If enabled, the plugin prints verbose logs. Intended for plugin debugging only. + + +### Error codes + +The plugin defines its own mypy error codes which can be disabled or ignored in the same way as the built-in error codes +(e.g. via config or for individual lines with `# type: ignore[CODE]`. + +Keep in mind that these are only for checks that are implemented by the plugin itself. Other errors, like assignments +of a wrong validator to a field, are still reported by mypy using its own error codes (e.g. `assignment`). + +All error codes are sub codes of `validataclass`, so disabling `validataclass` will disable the others as well. + +- `validataclass`: Various errors in field definitions, e.g. a missing or duplicate validator, an invalid type of object + in a field tuple and more. +- `validataclass-decorator`: Incorrectly defined custom decorator (see `custom_validataclass_decorators` in config) +- `validataclass-empty-type`: Field that has an empty type, i.e. the dataclass can never be validated because the + validator and default don't allow any value. This can happen when you're using a `RejectValidator` without a default. + The validator will reject every input, but without a default the field is still required. +- `validataclass-not-implemented`: Special error code for edge cases that are currently not supported by the plugin. + If you get this error, you found an edge case that was unknown to the developers. Please create a bug report for this + to help with the plugin development. Even if you believe it was a mistake on your part, you might have found a way to + cause an edge case that we thought can never happen. + + +## Common mistakes / migration guide + +### Incompatible return type in custom validators + +validataclass is designed with extensibility in mind: You can easily write your own validators or extend existing ones +to build on their existing functionality. + +A common use case for this are validators that accept input of a basic type (like strings) and convert them to objects. +Examples of built-in validators are the `DecimalValidator` or `DateValidator`: They use the `StringValidator` for the +first step to validate input as a valid string, then they try to create a `Decimal` or `date` object from that string. + +Before validataclass 0.12, the common way to do that was to inherit from a base validator (e.g. the `StringValidator`) +and extend its `validate()` method. See the following example: + +```python +from datetime import date +from typing import Any, override +from validataclass.exceptions import ValidationError +from validataclass.validators import StringValidator + +class DateValidator(StringValidator): + def __init__(self) -> None: + # Initialize base validator with some parameters + super().__init__(min_length=1, max_length=10) + + @override + def validate(self, input_data: Any, **kwargs: Any) -> date: + # Use base validator to validate input as a string + date_string = super().validate(input_data, **kwargs) + + try: + # Convert string to a date object + return date.fromisoformat(date_string) + except ValueError: + raise ValidationError(code='invalid_date') +``` + +This works as intended at runtime, but type checkers will complain about the return type of the `validate()` method: + +``` +error: Return type "date" of "validate" incompatible with return type "str" in supertype + "validataclass.validators.string_validator.StringValidator" [override] +``` + +The reason for this is called the Liskov substitution principle (LSP): Basically, if you override a method in a +subclass, the return type of the method must be the same or a subtype of the return type in the base class. The +`StringValidator` returns a `str`, but `date` is not a subtype of `str`. + +Luckily, the solution to this problem is really simple in the case of our validators: Composition over inheritance. +Instead of subclassing the `StringValidator`, you can create an instance of `StringValidator` within your custom +validator (that is solely based on the abstract base class `Validator`) and use it as a helper in your validate method: + +```python +from datetime import date +from typing import Any, override +from validataclass.exceptions import ValidationError +from validataclass.validators import StringValidator, Validator + +class DateValidator(Validator[date]): # Note the type parameter here + # Base validator for validating strings + string_validator: StringValidator + + def __init__(self) -> None: + # Initialize string validator with some parameters + self.string_validator = StringValidator(min_length=1, max_length=10) + + @override + def validate(self, input_data: Any, **kwargs: Any) -> date: + # Use string validator to validate input as a string + date_string = self.string_validator.validate(input_data, **kwargs) + + try: + # Convert string to a date object + return date.fromisoformat(date_string) + except ValueError: + raise ValidationError(code='invalid_date') +``` + +Now from a typing perspective, the `DateValidator` is simply a `Validator[date]`, i.e. a validator that returns a `date` +object. It just outsources the first part of the validation to a different validator to reuse its functionality. + + +## How the mypy plugin works + +The mypy plugin works in mysterious ways. + +There is some explanation about the plugin in the description of the pull request +[!140](https://github.com/binary-butterfly/validataclass/pull/140). A more detailed explanation in the docs will follow +at some point in the future. diff --git a/docs/index.md b/docs/index.md index 234cda1..5355331 100644 --- a/docs/index.md +++ b/docs/index.md @@ -32,3 +32,7 @@ This is the index of the documentation. 6. [Building new validators](06-build-your-own.md) - (To be written...) + +7. [Type checking with mypy](07-mypy-plugin.md) + - How to use the validataclass mypy plugin + - Common mistakes and how to fix them diff --git a/pytest.ini b/pytest.ini index 6d5b134..889d54b 100644 --- a/pytest.ini +++ b/pytest.ini @@ -5,7 +5,7 @@ addopts = --cov-config=.coveragerc --cov-context=test --cov-report= - --mypy-ini-file=tests/mypy/pytest_mypy.ini + --mypy-pyproject-toml-file=tests/mypy/pytest_pyproject.toml --mypy-only-local-stub # TODO: This is needed to include the mypy plugin in coverage. However the docs say: # TODO "Useful for debugging, will create problems with import cache" - better solution? diff --git a/setup.cfg b/setup.cfg index 6e26494..7d6bb7b 100644 --- a/setup.cfg +++ b/setup.cfg @@ -32,6 +32,7 @@ package_dir = packages = find: python_requires = ~=3.10 install_requires = + tomli; python_version<'3.11' typing-extensions ~= 4.15 [options.packages.find] diff --git a/src/validataclass/mypy/plugin/constants.py b/src/validataclass/mypy/plugin/constants.py index 0192c8c..bb1eb43 100644 --- a/src/validataclass/mypy/plugin/constants.py +++ b/src/validataclass/mypy/plugin/constants.py @@ -6,14 +6,15 @@ from typing import Final -# Decorators that turn a class to a validataclass -# (Not Final so it can be modified by other plugins that add their own validataclass-style decorators.) -VALIDATACLASS_DECORATORS = { +# Built-in decorators that turn a class into a validataclass. +# Use `PluginConfig.custom_validataclass_decorators` to get user-defined decorators from the plugin config. +VALIDATACLASS_DECORATORS: Final = { 'validataclass.dataclasses.validataclass.validataclass', } -# Functions that create validataclass fields (must have similar signature as validataclass_field) -VALIDATACLASS_FIELD_FUNCS = { +# Built-in functions that create validataclass fields. +# Use `PluginConfig.custom_field_functions` to get user-defined field functions from the plugin config. +VALIDATACLASS_FIELD_FUNCS: Final = { 'validataclass.dataclasses.validataclass_field.validataclass_field', } diff --git a/src/validataclass/mypy/plugin/error_codes.py b/src/validataclass/mypy/plugin/error_codes.py index 0d4511c..0c482d5 100644 --- a/src/validataclass/mypy/plugin/error_codes.py +++ b/src/validataclass/mypy/plugin/error_codes.py @@ -15,6 +15,13 @@ 'Plugin', ) +ERROR_CODE_VALIDATACLASS_DECORATOR: Final = ErrorCode( + 'validataclass-decorator', + 'Check that validataclass is decorated correctly', + 'Plugin', + sub_code_of=ERROR_CODE_VALIDATACLASS, +) + ERROR_CODE_VALIDATACLASS_EMPTY_TYPE: Final = ErrorCode( 'validataclass-empty-type', 'Check that validataclass field has a validator or default that can return a value', diff --git a/src/validataclass/mypy/plugin/field_type_resolver.py b/src/validataclass/mypy/plugin/field_type_resolver.py index aec5fb1..680be44 100644 --- a/src/validataclass/mypy/plugin/field_type_resolver.py +++ b/src/validataclass/mypy/plugin/field_type_resolver.py @@ -27,6 +27,7 @@ from .error_codes import ERROR_CODE_VALIDATACLASS, ERROR_CODE_VALIDATACLASS_EMPTY_TYPE from .debug_logger import DebugLogger from .parsed_field_cache import ParsedFieldCache, ParsedValidataclassField +from .plugin_config import PluginConfig class FieldTypeResolver: @@ -40,15 +41,26 @@ class FieldTypeResolver: # Interface to mypy's type checker _api: CheckerPluginInterface + # Plugin configuration + _plugin_config: PluginConfig + # Logger for plugin development and debugging _logger: DebugLogger # Internal cache for parsed validataclass fields (i.e. parsed types), shared across instances of this class _parsed_field_cache: ParsedFieldCache - def __init__(self, ctx: FunctionContext, logger: DebugLogger, parsed_field_cache: ParsedFieldCache): + def __init__( + self, + *, + ctx: FunctionContext, + plugin_config: PluginConfig, + logger: DebugLogger, + parsed_field_cache: ParsedFieldCache, + ): self._ctx = ctx self._api = ctx.api + self._plugin_config = plugin_config self._logger = logger self._parsed_field_cache = parsed_field_cache @@ -182,8 +194,9 @@ def _parse_field_definition( # entire field definition), we can shortcut the analysis here. # (Fields created with dataclasses.field() are skipped by the ValidataclassTransformer.) if isinstance(field_rhs_expr, CallExpr) and isinstance(field_rhs_expr.callee, RefExpr): - # Handle fields created explicitly using validataclass_field() - if field_rhs_expr.callee.fullname in VALIDATACLASS_FIELD_FUNCS: + # Handle fields created explicitly using validataclass_field() or a similar function + callee_name = field_rhs_expr.callee.fullname + if callee_name in VALIDATACLASS_FIELD_FUNCS or callee_name in self._plugin_config.custom_field_functions: return self._parse_validataclass_field_callexpr(field_rhs_expr) # This will hold the end result that's returned at the end of the function @@ -279,6 +292,12 @@ def _parse_rhs_item(self, item_type: Type, parsed_field: ParsedValidataclassFiel parsed_field.default_type = item_type return + # Check for instances of user-defined types that should be ignored (via plugin config) + ignored_custom_types = self._plugin_config.ignore_custom_types_in_fields + if any(item_type.type.has_base(custom_type) for custom_type in ignored_custom_types): + self._log_debug(f' - Ignore instance of user-defined type: {item_type}') + return + # (Everything else is probably an error!) # One easy mistake is writing a validator class name without parentheses (e.g. `field: int = IntegerValidator`). diff --git a/src/validataclass/mypy/plugin/plugin.py b/src/validataclass/mypy/plugin/plugin.py index bd0ffc8..aadad65 100644 --- a/src/validataclass/mypy/plugin/plugin.py +++ b/src/validataclass/mypy/plugin/plugin.py @@ -4,8 +4,8 @@ Use of this source code is governed by an MIT-style license that can be found in the LICENSE file. """ -import os from datetime import datetime +from pathlib import Path from typing import Any, Callable from mypy.options import Options @@ -18,11 +18,27 @@ from .debug_logger import DebugLogger from .field_type_resolver import FieldTypeResolver from .parsed_field_cache import ParsedFieldCache +from .plugin_config import PluginConfig, PluginConfigParser, PluginConfigParseError from .validataclass_transformer import ValidataclassTransformer -def _is_debug_mode() -> bool: - return bool(os.getenv('VALIDATACLASS_MYPY_DEBUG', False)) +def _load_plugin_config(config_file: str | None) -> PluginConfig: + """ + Load configuration for the validataclass mypy plugin from the mypy configuration, i.e. the same config file + that mypy is currently using. + + If there is an error parsing the config, print that error message and exit with a non-zero exit code + """ + # Fallback to empty config if there is no mypy config + if config_file is None: # pragma: nocover + return PluginConfig() + + config_parser = PluginConfigParser() + try: + return config_parser.load_config(Path(config_file)) + except PluginConfigParseError as exc: # pragma: nocover + print(f'Error while parsing plugin config for validataclass mypy plugin:\n{exc}') + exit(1) class ValidataclassPlugin(Plugin): @@ -30,20 +46,33 @@ class ValidataclassPlugin(Plugin): Custom mypy plugin for validataclass support. """ + # Plugin config + _plugin_config: PluginConfig + # Logger class for easier debugging _logger: DebugLogger # Internal cache for parsed validataclass fields, shared between all instances of FieldTypeResolver _parsed_field_cache: ParsedFieldCache + # Set of all validataclass decorators, including the built-in ones and user-defined ones from the plugin config + _validataclass_decorator_fullnames: set[str] + def __init__(self, options: Options): super().__init__(options) - self._logger = DebugLogger( - debug_mode=_is_debug_mode(), - ) + # Load plugin config + self._plugin_config = _load_plugin_config(options.config_file) + + # Initialize dependencies + self._logger = DebugLogger(debug_mode=self._plugin_config.debug_mode) self._parsed_field_cache = ParsedFieldCache() + # Cache some values for easier access + self._validataclass_decorator_fullnames = ( + VALIDATACLASS_DECORATORS | self._plugin_config.custom_validataclass_decorators + ) + @override def report_config_data(self, ctx: ReportConfigContext) -> Any: """ @@ -83,8 +112,8 @@ def get_class_decorator_hook(self, fullname: str) -> Callable[[ClassDefContext], In this plugin, we use this hook to tag all validataclasses so that we can recognize them later. We also call the corresponding callback of the dataclass plugin because it wouldn't be called otherwise. """ - # Tag all classes decorated with `@validataclass` for later. - if fullname in VALIDATACLASS_DECORATORS: + # Tag all classes decorated with `@validataclass` or a similar decorator for later. + if fullname in self._validataclass_decorator_fullnames: return self._validataclass_decorator_tag_callback return None @@ -103,8 +132,8 @@ def get_class_decorator_hook_2(self, fullname: str) -> Callable[[ClassDefContext In this plugin, we transform the definition of validataclasses by wrapping each field definition (e.g. tuples of validator and default) in a virtual wrapper function which is analyzed later in `get_function_hook`. """ - # Update classes decorated with `@validataclass` or similar. - if fullname in VALIDATACLASS_DECORATORS: + # Update classes decorated with `@validataclass` or a similar decorator. + if fullname in self._validataclass_decorator_fullnames: return self._validataclass_decorator_transform_callback return None @@ -161,7 +190,11 @@ def _validataclass_decorator_transform_callback(self, ctx: ClassDefContext) -> b Transform all `@validataclass`-decorated classes for later analysis. """ - transformer = ValidataclassTransformer(ctx, self._logger) + transformer = ValidataclassTransformer( + ctx=ctx, + plugin_config=self._plugin_config, + logger=self._logger, + ) return transformer.transform() def _virtual_field_wrapper_callback(self, ctx: FunctionContext) -> Type: @@ -170,7 +203,12 @@ def _virtual_field_wrapper_callback(self, ctx: FunctionContext) -> Type: Analyze type of field definition and adjust function return type. """ - resolver = FieldTypeResolver(ctx, self._logger, self._parsed_field_cache) + resolver = FieldTypeResolver( + ctx=ctx, + plugin_config=self._plugin_config, + logger=self._logger, + parsed_field_cache=self._parsed_field_cache, + ) return resolver.resolve() diff --git a/src/validataclass/mypy/plugin/plugin_config.py b/src/validataclass/mypy/plugin/plugin_config.py new file mode 100644 index 0000000..7c29379 --- /dev/null +++ b/src/validataclass/mypy/plugin/plugin_config.py @@ -0,0 +1,154 @@ +""" +validataclass +Copyright (c) 2026, binary butterfly GmbH and contributors +Use of this source code is governed by an MIT-style license that can be found in the LICENSE file. +""" + +import os +import sys +from pathlib import Path +from typing import Any + +if sys.version_info >= (3, 11): + import tomllib +else: # pragma: nocover + import tomli as tomllib + + +# TODO: Add unit tests for config parsing and error handling (and remove "pragma: nocover"s). + +class PluginConfig: + """ + Plugin configuration for the validataclass mypy plugin. + """ + + # Whether debug mode is enabled, which results in a lot of debug log messages. (default: false) + # Can also be set via environment variable VALIDATACLASS_MYPY_DEBUG. + debug_mode: bool + + # Whether to allow incompatible overrides for fields in validataclass sub classes. For example, the base class could + # have a field typed as `int`, but the subclass adds a `Default(None)` and therefore changes the field type to + # `int | None`. This is technically an incompatible override. However, it's also commonly done in validataclasses, + # so this option exists for compatibility to allow these overrides. (default: true, might be changed in the future) + allow_incompatible_field_overrides: bool + + # Custom decorators that turn a class into a validataclass (see also .constants.VALIDATACLASS_DECORATORS) + custom_validataclass_decorators: set[str] + + # Custom functions that create validataclass fields, must have a signature compatible to validataclass_field() + custom_field_functions: set[str] + + # Custom types of objects that are allowed (read: ignored) in a field definition. This can be used in combination + # with the custom decorators to allow additional options in a field. + ignore_custom_types_in_fields: set[str] + + def __init__( + self, + *, + debug_mode: bool = False, + allow_incompatible_field_overrides: bool = True, + custom_validataclass_decorators: set[str] | list[str] | None = None, + custom_field_functions: set[str] | list[str] | None = None, + ignore_custom_types_in_fields: set[str] | list[str] | None = None, + ): + self.debug_mode = debug_mode + self.allow_incompatible_field_overrides = allow_incompatible_field_overrides + self.custom_validataclass_decorators = set(custom_validataclass_decorators or []) + self.custom_field_functions = set(custom_field_functions or []) + self.ignore_custom_types_in_fields = set(ignore_custom_types_in_fields or []) + + +class PluginConfigParseError(Exception): + """ + Exception raised when there is an error while parsing the plugin config. + """ + + +class PluginConfigParser: + """ + Parser for plugin config. + """ + + def load_config(self, config_file: Path) -> PluginConfig: + """ + Load plugin configuration for the validataclass mypy plugin from a file (currently, only pyproject.toml files + are supported). + """ + # Currently only pyproject.toml format is supported to configure the validataclass mypy plugin + if not config_file.suffix.lower() == '.toml': # pragma: nocover + # Return default config + return PluginConfig() + + parsed_config = self._parse_pyproject_toml(config_file) + + # Allow overriding some settings via environment variables (for now, only debug mode) + if (env_debug := os.getenv('VALIDATACLASS_MYPY_DEBUG', None)) is not None: # pragma: nocover + parsed_config.debug_mode = bool(env_debug) + + return parsed_config + + def _parse_pyproject_toml(self, config_file: Path) -> PluginConfig: + """ + Parse a TOML file in the style of pyproject.toml with the given filename. + """ + try: + # Load TOML file into dictionary + with config_file.open('rb') as toml_file: + toml_data = tomllib.load(toml_file) + except tomllib.TOMLDecodeError as exc: # pragma: nocover + raise PluginConfigParseError(f'{config_file}: {exc}') from exc + + # Check for [tool.validataclass_mypy] section. + # For tests, we also need to support [tool.mypy.x_validataclass] as an alternative, due to this line: + # https://github.com/typeddjango/pytest-mypy-plugins/blob/3.3.0/pytest_mypy_plugins/configs.py#L57 + tool_sections = toml_data.get('tool', {}) + if 'validataclass_mypy' in tool_sections: # pragma: nocover + config_section = 'tool.validataclass_mypy' + raw_config = tool_sections['validataclass_mypy'] + elif 'x_validataclass' in tool_sections.get('mypy', {}): + config_section = 'tool.mypy.x_validataclass' + raw_config = tool_sections['mypy']['x_validataclass'] + else: + return PluginConfig() + + try: + # Parse raw config + return self._parse_raw_config(raw_config) + except (ValueError, TypeError) as exc: # pragma: nocover + raise PluginConfigParseError(f'{config_file}: [{config_section}]: {exc}') from exc + + def _parse_raw_config(self, raw_config: dict[str, Any]) -> PluginConfig: + """ + Parse the content of the `[tool.validataclass_mypy]` section of a pyproject.toml file. + """ + return PluginConfig( + debug_mode=self._parse_as_bool(raw_config, 'debug_mode', False), + allow_incompatible_field_overrides=self._parse_as_bool( + raw_config, 'allow_incompatible_field_overrides', True + ), + custom_validataclass_decorators=self._parse_as_str_list(raw_config, 'custom_validataclass_decorators'), + custom_field_functions=self._parse_as_str_list(raw_config, 'custom_field_functions'), + ignore_custom_types_in_fields=self._parse_as_str_list(raw_config, 'ignore_custom_types_in_fields'), + ) + + @staticmethod + def _parse_as_bool(raw_config: dict[str, Any], key: str, default: bool) -> bool: + """ + Retrieve the value with the given key from the raw config dictionary, parse it as a boolean and raise an + exception if it's invalid. + """ + raw = raw_config.get(key, default) + if not isinstance(raw, bool): # pragma: nocover + raise TypeError(f'{key}: Must be a boolean') + return raw + + @staticmethod + def _parse_as_str_list(raw_config: dict[str, Any], key: str) -> list[str]: + """ + Retrieve the value with the given key from the raw config dictionary, parse it as a list of strings and raise + an exception if it's invalid. + """ + raw = raw_config.get(key, []) + if not isinstance(raw, list) or any(not isinstance(item, str) for item in raw): # pragma: nocover + raise TypeError(f'{key}: Must be a list of strings') + return raw diff --git a/src/validataclass/mypy/plugin/validataclass_transformer.py b/src/validataclass/mypy/plugin/validataclass_transformer.py index b98f02b..72d8587 100644 --- a/src/validataclass/mypy/plugin/validataclass_transformer.py +++ b/src/validataclass/mypy/plugin/validataclass_transformer.py @@ -27,13 +27,19 @@ ) from mypy.plugin import ClassDefContext, SemanticAnalyzerPluginInterface from mypy.plugins.dataclasses import dataclass_class_maker_callback, DATACLASS_FIELD_SPECIFIERS +from mypy.semanal_shared import find_dataclass_transform_spec from mypy.server.trigger import make_wildcard_trigger from mypy.types import AnyType, CallableType, TypeOfAny, UnboundType from typing_extensions import override from .constants import VIRTUAL_FIELD_WRAPPER_FUNC, VIRTUAL_FIELD_WRAPPER_FUNC_NAME -from .error_codes import ERROR_CODE_VALIDATACLASS, ERROR_CODE_VALIDATACLASS_NOT_IMPLEMENTED +from .error_codes import ( + ERROR_CODE_VALIDATACLASS, + ERROR_CODE_VALIDATACLASS_DECORATOR, + ERROR_CODE_VALIDATACLASS_NOT_IMPLEMENTED, +) from .debug_logger import DebugLogger +from .plugin_config import PluginConfig class ValidataclassField: @@ -119,17 +125,21 @@ class ValidataclassTransformer: # Interface to mypy's semantic analyzer _api: SemanticAnalyzerPluginInterface + # Plugin configuration + _plugin_config: PluginConfig + # Logger for plugin development and debugging _logger: DebugLogger # FuncDef for the virtual field wrapper function. Created on first use and cached here as a class variable. _virtual_field_wrapper_funcdef: FuncDef | None = None - def __init__(self, ctx: ClassDefContext, logger: DebugLogger): + def __init__(self, *, ctx: ClassDefContext, plugin_config: PluginConfig, logger: DebugLogger): self._ctx = ctx self._class_def = ctx.cls self._decorator = ctx.reason self._api = ctx.api + self._plugin_config = plugin_config self._logger = logger def _get_logger_context(self, context: Context | None) -> str: @@ -188,6 +198,11 @@ def transform(self) -> bool: Called for every class that is decorated with `@validataclass` (or an equivalent decorator). """ + # For custom validataclass decorators (see config), we need to ensure that they are properly defined with the + # `@typing.dataclass_transform()` decorator, otherwise the dataclass plugin will raise an exception later. + if not self._ensure_valid_decorator(): + return True + # Plugin hooks may be called several times, so we need to check if we have already processed this class. # TODO: I'm not sure how to test this. There is a comment in the mypy.plugin module that recommends using a # forward reference to a class which should force the module to be processed multiple times, but this doesn't @@ -226,6 +241,35 @@ def transform(self) -> bool: # function, or generate the __init__ function all by ourselves. return dataclass_class_maker_callback(self._ctx) + def _ensure_valid_decorator(self) -> bool: + """ + Ensure that the decorator used on the class is a valid validataclass decorator. + This means the decorator must be decorated itself with `@typing.dataclass_transform(kw_only_default=True)`. + + Return False if the decorator is not decorated with `dataclass_transform`, which means the class will not be + further processed. Return True if processing can be continued (even if errors are reported). + """ + # Get the dataclass transform spec (see `@dataclass_transform()`) for the decorator used on the class + spec = find_dataclass_transform_spec(self._ctx.reason) + + # Ensure that the decorator was itself decorated with `@dataclass_transform` + if spec is None: + self._fail( + 'Custom validataclass decorator was not decorated with typing.dataclass_transform', + self._ctx.reason, code=ERROR_CODE_VALIDATACLASS_DECORATOR, + ) + # Cannot continue processing the class + return False + + # Ensure that `@dataclass_transform` was used with `kw_only_default=True` (not a critical error, continue) + if spec.kw_only_default is not True: + self._fail( + 'Custom validataclass decorator should be defined with kw_only_default=True', + self._ctx.reason, code=ERROR_CODE_VALIDATACLASS_DECORATOR, + ) + + return True + def _get_assignment_statements_from_block(self, block: Block) -> Iterator[AssignmentStmt]: """ Iterate over all assignment statements of a block. @@ -351,11 +395,14 @@ def _transform_field_in_class(self, field: ValidataclassField) -> None: """ assert field.assignment_stmt is not None - # Allow incompatible overrides of fields in validataclasses - # TODO: This is necessary because historically we've allowed to override the type of a field in an incompatible - # way in a subclass. This actually isn't very type-safe, though. We probably should discourage this and - # provide an option to allow incompatible overrides for compatibility. (Maybe a strict mode for this plugin?) - field.lvalue_var.allow_incompatible_override = True + # Allow incompatible overrides of fields in validataclasses (unless disabled via plugin config). + # This is necessary because historically we've allowed to override the type of a field in an incompatible way + # in a subclass. This actually isn't very type-safe, though. We may change the default behaviour in the future, + # but we should also provide a way to achieve the same thing (easily reusing and extending validataclasses) + # without misusing class inheritance (e.g. by allowing to "decouple" subclasses from their base class by + # modifying the class MROs). + if self._plugin_config.allow_incompatible_field_overrides: + field.lvalue_var.allow_incompatible_override = True # Wrap the right-hand side of the assignment in a call to the virtual field wrapper function, changing the # assignment statement in the class body in-place. diff --git a/tests/mypy/dataclasses/test_validataclass.yml b/tests/mypy/dataclasses/test_validataclass.yml index a0cf078..6333140 100644 --- a/tests/mypy/dataclasses/test_validataclass.yml +++ b/tests/mypy/dataclasses/test_validataclass.yml @@ -17,7 +17,7 @@ - case: simple_validataclass main: | from validataclass.dataclasses import validataclass - from validataclass.validators import ListValidator, IntegerValidator, Noneable, StringValidator + from validataclass.validators import IntegerValidator, ListValidator, Noneable, StringValidator @validataclass class SimpleDataclass: diff --git a/tests/mypy/plugin/test_plugin_config.yml b/tests/mypy/plugin/test_plugin_config.yml new file mode 100644 index 0000000..a999936 --- /dev/null +++ b/tests/mypy/plugin/test_plugin_config.yml @@ -0,0 +1,191 @@ +# yaml-language-server: $schema=https://raw.githubusercontent.com/typeddjango/pytest-mypy-plugins/master/pytest_mypy_plugins/schema.json + +# Test parsing a complete plugin config without errors +- case: complete_plugin_config + mypy_config: | + [tool.mypy.x_validataclass] + debug_mode = true + custom_validataclass_decorators = [] + main: | + pass + +# Test plugin config with allow_incompatible_field_overrides=false (i.e. strict rules for field type overrides) +- case: disallow_incompatible_field_overrides + mypy_config: | + [tool.mypy.x_validataclass] + allow_incompatible_field_overrides = false + main: | + from validataclass.dataclasses import validataclass, Default, DefaultUnset + from validataclass.helpers import UnsetValueType + from validataclass.validators import IntegerValidator, RejectValidator, StringValidator + + @validataclass + class Base: + field1: int = IntegerValidator() + field2: int = IntegerValidator(), Default(0) + field3: str | None = StringValidator(), Default(None) + field4: int | UnsetValueType = IntegerValidator() + + @validataclass + class GoodSub(Base): + # Unchanged field type, default with same type + field1: int = Default(0) + + # Unchanged field type, compatible validator (default is still 0): `Never | int` is the same as `int` + field2: int = RejectValidator() + + # Unchanged field type, compatible default: `str` is contained in `str | None` + field3: str | None = Default('') + + # Unchanged field type, compatible validator and default: `UnsetValueType` is contained in `int | UnsetValueType` + field4: int | UnsetValueType = RejectValidator(), DefaultUnset + + # NOTE: The following sub class is "good" in terms of the assignment check, but violate the mutable-override check. + # Those are optional rules in mypy (not even included in strict mode), so we're checking them separately. + # Unless you enable that rule, the following class is "good". + + @validataclass + class MostlyGoodSub(Base): + # Type change: `str` is contained in `str | None` + field3: str = Default('') # E: Covariant override of a mutable attribute (base class "Base" defined the type as "str | None", expression has type "str") [mutable-override] + + # Type change: `UnsetValueType` is contained in `int | UnsetValueType` + field4: UnsetValueType = RejectValidator(), DefaultUnset # E: Covariant override of a mutable attribute (base class "Base" defined the type as "int | UnsetValueType", expression has type "UnsetValueType") [mutable-override] + + @validataclass + class BadSub(Base): + # Incompatible type change: Base class does not allow `None` + field1: int | None = Default(None) # E: Incompatible types in assignment (expression has type "int | None", base class "Base" defined the type as "int") [assignment] + + # Incompatible type change: Base class is `int`, does not allow `UnsetValueType` + field2: UnsetValueType = RejectValidator(), DefaultUnset # E: Incompatible types in assignment (expression has type "UnsetValueType", base class "Base" defined the type as "int") [assignment] + + # Incompatible type change: Base class is `str | None`, does not allow `int` + field3: int | None = IntegerValidator() # E: Incompatible types in assignment (expression has type "int | None", base class "Base" defined the type as "str | None") [assignment] + + # Compatible type change, but incompatible validator: Field still has `IntegerValidator` but does not allow `int` + field4: UnsetValueType = DefaultUnset # (line 50, multiple errors see below) + out: | + main:50: error: Incompatible types in assignment (expression has type "int | UnsetValueType", variable has type "UnsetValueType") [assignment] + main:50: error: Covariant override of a mutable attribute (base class "Base" defined the type as "int | UnsetValueType", expression has type "UnsetValueType") [mutable-override] + +# Test plugin config with a custom validataclass-like decorator and custom validataclass field functions +- case: custom_validataclass_decorator_and_field_functions + mypy_config: | + [tool.mypy.x_validataclass] + custom_validataclass_decorators = [ + "main.custom_validataclass_decorator", + ] + custom_field_functions = [ + "main.custom_validataclass_field", + ] + main: | + from typing import Any, TypeVar + from typing_extensions import dataclass_transform + from validataclass.dataclasses import Default, validataclass, validataclass_field + from validataclass.validators import IntegerValidator, Validator + + T = TypeVar('T') + + @dataclass_transform(kw_only_default=True) + def custom_validataclass_decorator(cls: type[T]) -> type[T]: + return validataclass(cls) + + def custom_other_decorator(cls: type[T]) -> type[T]: + return cls + + def custom_validataclass_field(validator: Validator[Any], **kwargs: Any) -> Any: + return validataclass_field(validator) + + @custom_validataclass_decorator + class CustomValidataclass: + # Valid fields + field1: int = IntegerValidator() + field2: int | None = IntegerValidator(), Default(None) + explicit_field: int = custom_validataclass_field(IntegerValidator()) + + # Validataclass field with incorrect type + invalid_type: str = IntegerValidator(), Default(None) # E: Incompatible types in assignment (expression has type "int | None", variable has type "str") [assignment] + invalid_explicit: str = custom_validataclass_field(IntegerValidator()) # E: Incompatible types in assignment (expression has type "int", variable has type "str") [assignment] + + # Not a valid validataclass field + invalid_field: int = 42 # E: Unexpected type "Literal[42]?" in field definition (expected Validator or BaseDefault) [validataclass] + + @custom_other_decorator + class DecoratedRegularClass: + # Regular class attribute (valid, this is not a validataclass!) + regular_attr: int = 42 + + # Validataclass field in a class that's not a validataclass + invalid_field: int = IntegerValidator() # E: Incompatible types in assignment (expression has type "IntegerValidator", variable has type "int") [assignment] + +# Test plugin config with incorrectly defined custom validataclass-like decorators +- case: invalid_custom_validataclass_decorators + mypy_config: | + [tool.mypy.x_validataclass] + custom_validataclass_decorators = [ + "main.custom_validataclass_decorator1", + "main.custom_validataclass_decorator2", + ] + main: | + from typing import Any, TypeVar + from typing_extensions import dataclass_transform + from validataclass.dataclasses import Default, validataclass + from validataclass.validators import IntegerValidator + + T = TypeVar('T') + + # No @dataclass_transform! + def custom_validataclass_decorator1(cls: type[T]) -> type[T]: + return validataclass(cls) + + # Missing kw_only_default=True + @dataclass_transform() + def custom_validataclass_decorator2(cls: type[T]) -> type[T]: + return validataclass(cls) + + @custom_validataclass_decorator1 # E: Custom validataclass decorator was not decorated with typing.dataclass_transform [validataclass-decorator] + class CustomValidataclass1: + field1: int = IntegerValidator() # E: Incompatible types in assignment (expression has type "IntegerValidator", variable has type "int") [assignment] + + @custom_validataclass_decorator2 # E: Custom validataclass decorator should be defined with kw_only_default=True [validataclass-decorator] + class CustomValidataclass2: + field1: int = IntegerValidator() + field2: int | None = IntegerValidator(), Default(None) + invalid_type: str = IntegerValidator(), Default(None) # E: Incompatible types in assignment (expression has type "int | None", variable has type "str") [assignment] + +# Test plugin config with custom user-defined types that should be ignored in field definitions +- case: validataclass_with_custom_types_in_fields + mypy_config: | + [tool.mypy.x_validataclass] + custom_validataclass_decorators = [ + "main.custom_validataclass_decorator", + ] + ignore_custom_types_in_fields = [ + "main.BaseCustomType", + ] + main: | + from typing import Any, TypeVar + from typing_extensions import dataclass_transform + from validataclass.dataclasses import Default, validataclass + from validataclass.validators import IntegerValidator + + T = TypeVar('T') + + @dataclass_transform(kw_only_default=True) + def custom_validataclass_decorator(cls: type[T]) -> type[T]: + return validataclass(cls) + + class BaseCustomType: + value: str + def __init__(self, value: str): + self.value = value + + class ExampleCustomType(BaseCustomType): + pass + + @custom_validataclass_decorator + class CustomValidataclass: + field1: int = IntegerValidator() + field2: int = IntegerValidator(), BaseCustomType('banana') + field3: int | None = ExampleCustomType('banana'), IntegerValidator(), Default(None) diff --git a/tests/mypy/pytest_mypy.ini b/tests/mypy/pytest_mypy.ini deleted file mode 100644 index f24241f..0000000 --- a/tests/mypy/pytest_mypy.ini +++ /dev/null @@ -1,24 +0,0 @@ -# This mypy config is used exclusively by the pytest-mypy-plugins plugin. -[mypy] -strict = true - -# Enable further checks that are not included in strict mode -disallow_any_unimported = true -strict_equality_for_none = true -warn_unreachable = true -enable_error_code = - deprecated, - explicit-override, - ignore-without-code, - # TODO: Maybe enable this is in the future (when we have the mypy plugin) - # mutable-override, - possibly-undefined, - redundant-expr, - redundant-self, - truthy-bool, - truthy-iterable, - unused-awaitable - -# Enable validataclass mypy plugin -plugins = - validataclass.mypy.plugin diff --git a/tests/mypy/pytest_pyproject.toml b/tests/mypy/pytest_pyproject.toml new file mode 100644 index 0000000..132cbf8 --- /dev/null +++ b/tests/mypy/pytest_pyproject.toml @@ -0,0 +1,25 @@ +# This mypy config is used exclusively by the pytest-mypy-plugins plugin. +[tool.mypy] +strict = true + +# Enable further checks that are not included in strict mode +disallow_any_unimported = true +strict_equality_for_none = true +warn_unreachable = true +enable_error_code = [ + "deprecated", + "explicit-override", + "ignore-without-code", + "mutable-override", + "possibly-undefined", + "redundant-expr", + "redundant-self", + "truthy-bool", + "truthy-iterable", + "unused-awaitable", +] + +# Enable validataclass mypy plugin +plugins = [ + "validataclass.mypy.plugin", +]