Skip to content

Fix metadata grain KeyError on http.query error responses (#65184)#69471

Open
dwoz wants to merge 1 commit into
saltstack:3006.xfrom
dwoz:fix/issue-65184
Open

Fix metadata grain KeyError on http.query error responses (#65184)#69471
dwoz wants to merge 1 commit into
saltstack:3006.xfrom
dwoz:fix/issue-65184

Conversation

@dwoz

@dwoz dwoz commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Guards salt.grains.metadata._search() against KeyError: 'headers' when
salt.utils.http.query() returns an error-shaped response (4xx/5xx with a
body but no headers key — the shape the tornado backend has produced on
HTTPError since 3006.3). The grain now treats the missing headers key as
"no Content-Type information" and falls through to the existing parsing path
instead of letting the whole grain load fail.

What issues does this PR fix or reference?

Fixes #65184

Related: #62061 (open against 3006.x) fixes the contributing user-data
=-splitter bug that triggers the bogus IMDS sub-path lookup in the
reporter's environment. The two fixes are complementary, not competing: even
with #62061 merged, any other 4xx/5xx response from IMDS (bad token, sporadic
server error, etc.) would still produce the same KeyError. Merging both
gives users a complete fix.

Previous Behavior

With metadata_server_grains: True, the grain crashed on load with:

[CRITICAL] Failed to load grains defined in grain file metadata.metadata in function <LoadedFunc name='metadata.metadata'>, error:
Traceback (most recent call last):
  ...
  File ".../salt/grains/metadata.py", line 64, in _search
    ret[line] = _search(prefix=os.path.join(prefix, line + "/"))
  File ".../salt/grains/metadata.py", line 53, in _search
    linedata["headers"].get("Content-Type", "text/plain")
KeyError: 'headers'

Root cause: between 3006.2 and 3006.3, commit 43b7fb52842 ("Return body when
error with tornado backend salt.utils.http") made http.query populate body
on HTTPError without also populating headers. The metadata grain indexed
linedata["headers"] unconditionally and crashed on any recursive sub-path
lookup that the IMDS rejected.

New Behavior

_search() returns a dict on the same recursive path; the missing-headers
leaf is parsed without Content-Type info and recursion continues. The whole
grain no longer fails to load.

Merge requirements satisfied?

  • Docs (no doc behavior changed)
  • Changelog (changelog/65184.fixed.md)
  • Tests written/updated (tests/pytests/unit/grains/test_metadata.py,
    three regression cases: missing-headers on recursion, missing-headers on
    the initial query, and a sanity guard for the existing
    application/octet-stream short-circuit)

Commits signed with GPG?

No (matches base-branch policy on 3006.x).

Since 3006.3 salt.utils.http.query (tornado backend) returns ``body`` on
HTTPError but does not populate ``headers``. salt.grains.metadata._search()
indexed ``linedata["headers"]`` unconditionally, so any recursive lookup that
hit a 4xx/5xx (e.g. IMDS rejecting a sub-path produced by the legacy
``=``-splitter on an EC2 user-data line) crashed the whole grain load with::

    [CRITICAL] Failed to load grains defined in grain file metadata.metadata
    ...
    KeyError: 'headers'

Guard the headers indexing the same way ``body`` is already guarded: when
``headers`` is absent, treat it as "no Content-Type information" and fall
through to the existing parsing path. Add three pytest regressions covering
the missing-headers case (both on the initial query and on recursion) and a
sanity guard for the existing ``application/octet-stream`` short-circuit.

Fixes saltstack#65184
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:full Run the full test suite

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant