[BUG] Leading slashes in v2/image "file_url" #885
Description
Bug description
While working on #881, after installing the Django Debug Toolbar, I noticed that one of our tests was failing. The URL being tested was /v2/conditions. The full stack trace can be viewed below.
Investigating the /v2/conditions endpoint and was greated with an error page that read:
SuspiciousFileOperation at /v2/images/
The joined path (/img/object_icons/elderberry-inn-icons/conditions/blinded.svg) is located
outside of the base path component (/Users/calum/Documents/code/open5e-api/static)
Which pointed me toward the /v2/images endpoint where I experienced the same error.
Digging into this error I believe that it is being caused by the leading slash in the "file_path" in our V2 Image data. This is used to specify an absolute path rather, which I believe is causing the Django development server to throw a SuspiciousFileOperation error when Django tries to reach 'up' belong to repository's root director to the computer's root. The fix here is to remove the leading '/' on the "file_url" fields.
The error does not appear if DEBUG mode is off. So this shouldn't cause any issues on the live API, only during local testing.
I can only assume that this error has emerged because the DDT is somehow interferring with how the other middleware (WhiteNoise?) is handling our static files, specifically on sanitising those leading slashes.
So while there is a deeper issue to unpack here about dev/prod parity, the leading slash in the "file_path" field of our image data is nonetheless a data error and ought to be fixed.
Code
Pytest log
___________________________________________________________________ TestObjects.test_condition_example ___________________________________________________________________
self = <Response [400]>, kwargs = {}
def json(self, **kwargs):
r"""Decodes the JSON response body (if any) as a Python object.
This may return a dictionary, list, etc. depending on what is in the response.
:param \*\*kwargs: Optional arguments that ``json.loads`` takes.
:raises requests.exceptions.JSONDecodeError: If the response body does not
contain valid json.
"""
if not self.encoding and self.content and len(self.content) > 3:
# No encoding set. JSON RFC 4627 section 3 states we should expect
# UTF-8, -16 or -32. Detect which one to use; If the detection or
# decoding fails, fall back to `self.text` (using charset_normalizer to make
# a best guess).
encoding = guess_json_utf(self.content)
if encoding is not None:
try:
return complexjson.loads(self.content.decode(encoding), **kwargs)
except UnicodeDecodeError:
# Wrong UTF codec detected; usually because it's not UTF-8
# but some other 8-bit codec. This is an RFC violation,
# and the server didn't bother to tell us what codec *was*
# used.
pass
except JSONDecodeError as e:
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
try:
> return complexjson.loads(self.text, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
../../../.local/share/virtualenvs/open5e-api-zZJBumHW/lib/python3.11/site-packages/requests/models.py:976:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/Cellar/python@3.11/3.11.6/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/__init__.py:346: in loads
return _default_decoder.decode(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^
/usr/local/Cellar/python@3.11/3.11.6/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/decoder.py:337: in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <json.decoder.JSONDecoder object at 0x10cac0450>
s = 'SuspiciousFileOperation at /v2/conditions/stunned/\nThe joined path (/img/object_icons/elderberry-inn-icons/condition...ile. Change that to False, and Django will\ndisplay a standard page generated by the handler for this status code.\n\n'
idx = 0
def raw_decode(self, s, idx=0):
"""Decode a JSON document from ``s`` (a ``str`` beginning with
a JSON document) and return a 2-tuple of the Python
representation and the index in ``s`` where the document ended.
This can be used to decode a JSON document from a string that may
have extraneous data at the end.
"""
try:
obj, end = self.scan_once(s, idx)
except StopIteration as err:
> raise JSONDecodeError("Expecting value", s, err.value) from None
E json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
/usr/local/Cellar/python@3.11/3.11.6/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/decoder.py:355: JSONDecodeError
During handling of the above exception, another exception occurred:
self = <test_objects.TestObjects object at 0x1295c7750>
def test_condition_example(self):
path="/v2/conditions/stunned/"
> self._verify(path)
api_v2/tests/test_objects.py:171:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
api_v2/tests/test_objects.py:49: in _verify
response = requests.get(API_BASE + endpoint, allow_redirects=True, headers = {'Accept': 'application/json'}).json()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <Response [400]>, kwargs = {}
def json(self, **kwargs):
r"""Decodes the JSON response body (if any) as a Python object.
This may return a dictionary, list, etc. depending on what is in the response.
:param \*\*kwargs: Optional arguments that ``json.loads`` takes.
:raises requests.exceptions.JSONDecodeError: If the response body does not
contain valid json.
"""
if not self.encoding and self.content and len(self.content) > 3:
# No encoding set. JSON RFC 4627 section 3 states we should expect
# UTF-8, -16 or -32. Detect which one to use; If the detection or
# decoding fails, fall back to `self.text` (using charset_normalizer to make
# a best guess).
encoding = guess_json_utf(self.content)
if encoding is not None:
try:
return complexjson.loads(self.content.decode(encoding), **kwargs)
except UnicodeDecodeError:
# Wrong UTF codec detected; usually because it's not UTF-8
# but some other 8-bit codec. This is an RFC violation,
# and the server didn't bother to tell us what codec *was*
# used.
pass
except JSONDecodeError as e:
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
try:
return complexjson.loads(self.text, **kwargs)
except JSONDecodeError as e:
# Catch JSON-related errors and raise as requests.JSONDecodeError
# This aliases json.JSONDecodeError and simplejson.JSONDecodeError
> raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
E requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
../../../.local/share/virtualenvs/open5e-api-zZJBumHW/lib/python3.11/site-packages/requests/models.py:980: JSONDecodeError
======================================================================== short test summary info =========================================================================
FAILED api_v2/tests/test_objects.py::TestObjects::test_condition_example - requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
===================================================================== 1 failed, 59 passed in 15.19s ======================================================================