Shrink the textual mime set and default text to utf-8-sig

The set of application/* types treated as text is now just json and xml,
the two ubiquitous un-suffixed textual types the structural rules
(text/*, +json/+xml) can't reach. application/javascript and
application/ecmascript are gone: RFC 9239 made text/javascript canonical,
which the text/* rule already covers.

The inferred text encoding is now utf-8-sig, so a byte-order mark is
dropped rather than surfacing as a leading U+FEFF that JSON parsers
reject. A declared charset= is still used as-is.

The encoding is validated with codecs.lookup at construction, so a typo
fails at registration instead of on the first read. The extra="forbid"
change gets its own migration section, since it applies to every
Resource class, not just FileResource.

Author: maxisbey

docs/migration.md

@@ -826,13 +826,19 @@ Reading a missing resource now returns JSON-RPC error code `-32602` (invalid par
 
 The underlying lookups now raise typed exceptions instead of `ValueError`. `ResourceManager.get_resource()` raises `ResourceNotFoundError` when no resource or template matches the URI, and `ResourceTemplate.create_resource()` raises `ResourceError` when the template function fails. Neither subclasses `ValueError`, so callers catching `ValueError` should switch to `ResourceNotFoundError` / `ResourceError` (both importable from `mcp.server.mcpserver.exceptions`; `ResourceNotFoundError` subclasses `ResourceError`).
 
+### `Resource` classes reject unknown keyword arguments
+
+The `Resource` base class now sets `extra="forbid"`, so every resource class — `TextResource`, `BinaryResource`, `FunctionResource`, `FileResource`, `HttpResource`, `DirectoryResource`, and your own subclasses — raises `ValidationError` on an unrecognised keyword argument instead of silently dropping it. Previously a typo'd or since-removed parameter (such as `FileResource(is_binary=...)`, below) was accepted and ignored. Remove any stray keyword arguments; if a subclass needs to accept arbitrary extras, set its own `model_config = ConfigDict(extra="allow")`.
+
 ### `FileResource.is_binary` replaced by `encoding`
 
 `FileResource` used to take `is_binary: bool` and guess its default from `mime_type` (`text/*` → text, anything else → bytes). Two problems fell out of that: `is_binary=False` could not actually be set — `False` doubled as the "not given" sentinel, so `mime_type="application/json"` always came back as a base64 blob — and text reads used `Path.read_text()` with no encoding, i.e. the platform locale (cp1252 on Windows).
 
-The field is now `encoding: str | None`. A string means "decode with this encoding and serve as text"; `None` means "read bytes and serve as a blob". When omitted it defaults to the `charset` declared in `mime_type` if there is one, otherwise `"utf-8"` for textual mime types (`text/*`, `application/json`, `application/xml`, `application/javascript`, and any `+json`/`+xml` suffix) and `None` for everything else, so JSON and XML files are now served as text without any configuration.
+The field is now `encoding: str | None`. A string means "decode with this encoding and serve as text"; `None` means "read bytes and serve as a blob". When omitted it defaults to the `charset` declared in `mime_type` if there is one, otherwise `"utf-8-sig"` for textual mime types (`text/*`, `application/json`, `application/xml`, and any `+json`/`+xml` suffix) and `None` for everything else, so JSON and XML files are now served as text without any configuration. `utf-8-sig` is plain UTF-8 that also drops a byte-order mark if the file has one; a declared `charset=` is used as-is.
+
+Passing the removed `is_binary=` argument now raises a `ValidationError` at construction (see the section above) rather than being silently ignored. A misspelled `encoding` also fails at construction rather than on the first read.
 
-Passing the removed `is_binary=` argument raises a `ValidationError` at construction rather than being silently ignored — all `Resource` classes now reject unknown keyword arguments.
+One case flips the other way: a non-UTF-8 file with a textual mime type (say UTF-16 XML) previously shipped byte-exact as a blob and now fails to decode. Set `encoding` to the file's real encoding, or `encoding=None` to keep serving it as a blob.
 
 **Before (v1):**
 
@@ -845,7 +851,7 @@ FileResource(uri="file:///notes.txt", path=notes)  # text, decoded with the loca
 
 ```python
 FileResource(uri="file:///logo.png", path=logo, mime_type="image/png")  # bytes, from mime_type
-FileResource(uri="file:///notes.txt", path=notes)  # text, decoded as UTF-8
+FileResource(uri="file:///notes.txt", path=notes)  # text, decoded as UTF-8 (BOM tolerated)
 FileResource(uri="file:///data.json", path=data, mime_type="application/json")  # now text, not a blob
 ```
 

src/mcp/server/mcpserver/resources/types.py

@@ -2,6 +2,7 @@
 
 from __future__ import annotations
 
+import codecs
 import json
 from collections.abc import Callable
 from functools import partial
@@ -20,21 +21,17 @@
 from mcp.shared._callable_inspection import is_async_callable
 from mcp.shared.exceptions import MCPError
 
-_TEXTUAL_APPLICATION_TYPES = frozenset(
-    {
-        "application/json",
-        "application/xml",
-        "application/javascript",
-        "application/ecmascript",
-    }
-)
+# `application/*` types that are textual but predate the `+json`/`+xml`
+# structured-syntax suffixes, so the suffix rule below can't catch them.
+_TEXTUAL_APPLICATION_TYPES = frozenset({"application/json", "application/xml"})
 
 
 def _default_file_encoding(mime_type: str) -> str | None:
     """The encoding a file of this mime type is decoded with by default.
 
-    A declared `charset=` parameter wins. Otherwise textual types (`text/*`,
-    JSON, XML, JavaScript) are UTF-8 and everything else is bytes (None).
+    A declared `charset=` parameter wins. Otherwise textual types (`text/*`, JSON,
+    XML) are `utf-8-sig` — UTF-8 that also tolerates a byte-order mark — and
+    everything else is bytes (None).
     """
     essence, *params = (part.strip() for part in mime_type.split(";"))
     for param in params:
@@ -43,7 +40,7 @@ def _default_file_encoding(mime_type: str) -> str | None:
             return value.strip().strip('"')
     essence = essence.lower()
     if essence.startswith("text/") or essence.endswith(("+json", "+xml")) or essence in _TEXTUAL_APPLICATION_TYPES:
-        return "utf-8"
+        return "utf-8-sig"
     return None
 
 
@@ -151,9 +148,9 @@ class FileResource(Resource):
 
     The file is decoded with `encoding` and served as text, or read as bytes and
     served as a base64 blob when `encoding` is None. When `encoding` is omitted it
-    defaults to the `charset` declared in `mime_type`, else `"utf-8"` for textual
-    mime types (`text/*`, JSON, XML, JavaScript) and None for everything else;
-    pass it explicitly to override either way.
+    defaults to the `charset` declared in `mime_type`, else `"utf-8-sig"` for
+    textual mime types (`text/*`, JSON, XML) and None for everything else; pass
+    it explicitly to override either way.
     """
 
     path: Path = Field(description="Path to the file")
@@ -170,6 +167,17 @@ def validate_absolute_path(cls, path: Path) -> Path:
             raise ValueError("Path must be absolute")
         return path
 
+    @pydantic.field_validator("encoding")
+    @classmethod
+    def validate_known_encoding(cls, encoding: str | None) -> str | None:
+        """Ensure the encoding names a known codec, so a typo fails at construction not at read."""
+        if encoding is not None:
+            try:
+                codecs.lookup(encoding)
+            except LookupError as e:
+                raise ValueError(f"unknown encoding: {encoding}") from e
+        return encoding
+
     async def read(self) -> str | bytes:
         """Read the file content."""
         try:

tests/server/mcpserver/resources/test_file_resources.py

@@ -1,3 +1,4 @@
+import codecs
 import os
 from pathlib import Path
 from tempfile import NamedTemporaryFile
@@ -37,7 +38,7 @@ def test_file_resource_creation(temp_file: Path):
     assert resource.description == "test file"
     assert resource.mime_type == "text/plain"
     assert resource.path == temp_file
-    assert resource.encoding == "utf-8"
+    assert resource.encoding == "utf-8-sig"
 
 
 def test_file_resource_str_path_conversion(temp_file: Path):
@@ -79,17 +80,16 @@ async def test_encoding_none_reads_bytes(temp_file: Path):
     "mime_type",
     [
         "text/plain",
-        "text/html; charset=utf-8",
+        "text/html",
         "application/json",
         "application/xml",
         "application/vnd.api+json",
         "image/svg+xml",
-        "application/javascript",
     ],
 )
-def test_textual_mime_types_default_to_utf8(temp_file: Path, mime_type: str):
+def test_textual_mime_types_default_to_utf8_sig(temp_file: Path, mime_type: str):
     resource = FileResource(uri=temp_file.as_uri(), path=temp_file, mime_type=mime_type)
-    assert resource.encoding == "utf-8"
+    assert resource.encoding == "utf-8-sig"
 
 
 @pytest.mark.parametrize("mime_type", ["image/png", "application/octet-stream", "application/pdf"])
@@ -113,6 +113,17 @@ def test_removed_is_binary_kwarg_is_rejected(temp_file: Path):
         FileResource.model_validate({"uri": temp_file.as_uri(), "path": temp_file, "is_binary": True})
 
 
+def test_unknown_encoding_is_rejected(temp_file: Path):
+    """A codec typo fails at construction rather than on the first read."""
+    with pytest.raises(ValidationError, match="unknown encoding: not-a-codec"):
+        FileResource(uri=temp_file.as_uri(), path=temp_file, encoding="not-a-codec")
+
+
+def test_unknown_declared_charset_is_rejected(temp_file: Path):
+    with pytest.raises(ValidationError, match="unknown encoding"):
+        FileResource(uri=temp_file.as_uri(), path=temp_file, mime_type="text/plain; charset=not-a-codec")
+
+
 @pytest.mark.anyio
 async def test_json_file_is_served_as_text_by_default(temp_file: Path):
     """The mime type that motivated the encoding field: JSON must not become a base64 blob."""
@@ -121,6 +132,14 @@ async def test_json_file_is_served_as_text_by_default(temp_file: Path):
     assert await resource.read() == '{"a": 1}'
 
 
+@pytest.mark.anyio
+async def test_utf8_bom_is_stripped_by_default(temp_file: Path):
+    """The default utf-8-sig decoding drops a byte-order mark that would otherwise break JSON parsers."""
+    temp_file.write_bytes(codecs.BOM_UTF8 + b'{"a": 1}')
+    resource = FileResource(uri=temp_file.as_uri(), path=temp_file, mime_type="application/json")
+    assert await resource.read() == '{"a": 1}'
+
+
 @pytest.mark.anyio
 async def test_explicit_encoding_overrides_default(temp_file: Path):
     """An explicit encoding wins over the mime-type default and is what decodes the file."""