fix(#868)!: replace oggtechnologies api with own wikia scraping to set vault information#870
Merged
TobiTenno merged 8 commits intoWFCD:masterfrom Mar 6, 2026
Merged
fix(#868)!: replace oggtechnologies api with own wikia scraping to set vault information#870TobiTenno merged 8 commits intoWFCD:masterfrom
TobiTenno merged 8 commits intoWFCD:masterfrom
Conversation
…o set vault information
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughSwitch vault sourcing from the Ogg API to a new Warframe Wiki scraper; add VaultScraper and sleep helper; integrate vaultData into fetchWikiaData; update parser to consume Changes
Sequence DiagramsequenceDiagram
participant Build as Build/App
participant Scraper as Scraper Module
participant VaultScraper as VaultScraper
participant Wiki as Warframe Wiki
participant Parser as Parser Module
participant Data as All.json
Build->>Scraper: fetchWikiaData()
Scraper->>VaultScraper: VaultScraper.scrape()
VaultScraper->>Wiki: GET Prime Vault page
Wiki-->>VaultScraper: HTML with vault tables
VaultScraper->>VaultScraper: parse tables → Array<VaultData]
VaultScraper-->>Scraper: return vaultData
Scraper->>Scraper: assemble WikiaData (includes vaultData)
Scraper-->>Build: return WikiaData
Build->>Parser: addVaultData(item, category, wikiaData)
Parser->>Parser: lookup wikiaData.vaultData by name
Parser->>Parser: set vaulted / vaultDate / estimatedVaultDate
Parser-->>Build: enriched items
Build->>Data: write updated All.json
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Author
TobiTenno
requested changes
Mar 6, 2026
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (2)
build/parser.mjs (1)
89-98: Typedef not updated forvaultDataproperty.The
RawItemDatatypedef at line 95 still referencesVaultData(singular), but the scraper now returnsArray<VaultData>. This should be updated for consistency with the JSDoc at line 1049.Suggested fix
* `@typedef` {Object} RawItemData * `@property` {Array<Partial<Item>>} api Raw api data * `@property` {ImageManifest.Manifest} manifest of image data * `@property` {Array<DropRate>} drops drop rates * `@property` {PatchlogWrap} patchlogs patch data * `@property` {WikiaData} wikia warframe wikia data - * `@property` {VaultData} vaultData Ogg vault data + * `@property` {Array<VaultData>} vaultData vault data from wiki * `@property` {Array<module:`@wfcd/relics.TitaniaRelic`>} relics Generated relic data * `@property` {Array<Partial<Item>>} i18n i18n data🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@build/parser.mjs` around lines 89 - 98, The RawItemData JSDoc typedef is incorrect for the vaultData property: update the `@property` declaration for vaultData in the RawItemData typedef so it reflects the actual returned type (Array<VaultData>) instead of VaultData; locate the RawItemData typedef and change the vaultData property type to an array type (e.g., Array<VaultData> or Array<module:.../VaultData> to match existing JSDoc conventions used elsewhere in the file).build/scraper.mjs (1)
268-273: Typedef inconsistency:estimatedVaultDatemarked as required but not always provided.The
VaultDatatypedef showsestimatedVaultDateas a required field, butextractNotVaultedItems(lines 305-316) creates objects without it. The parser compensates by computing it fromreleaseDate, but the typedef should reflect the actual contract.Suggested typedef fix
* `@typedef` {Object} VaultData * `@property` {string} name name of the vaulted item * `@property` {boolean} vaulted whether the item is vaulted or not - * `@property` {VaultDateStamp} estimatedVaultDate estimated vault date + * `@property` {VaultDateStamp} [estimatedVaultDate] estimated vault date * `@property` {VaultDateStamp} [vaultDate] vault date, only available if the item is vaulted🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@build/scraper.mjs` around lines 268 - 273, The typedef VaultData declares estimatedVaultDate as required but extractNotVaultedItems sometimes constructs VaultData without it; update the contract by making estimatedVaultDate optional in the typedef (e.g., mark as [estimatedVaultDate]) or alternatively ensure extractNotVaultedItems always populates estimatedVaultDate (compute from releaseDate) so the typedef matches the actual objects; reference the VaultData typedef and the extractNotVaultedItems function to locate and fix the mismatch.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@build/scraper.mjs`:
- Around line 285-288: The check that currently logs an error and returns []
when vaultedItems, formerlyVaulted, notYetVaulted, or neverVaulted are missing
should instead fail the build by throwing an Error; locate the block that
contains the variables vaultedItems, formerlyVaulted, notYetVaulted, and
neverVaulted in build/scraper.mjs and replace the console.error + return []
behavior with throwing a descriptive Error (e.g., "Failed to load vaulted items
tables from wiki"), so callers cannot continue with empty/incorrect vault data.
In `@data/json/All.json`:
- Line 296904: The estimatedVaultDate for frame-bound Prime weapons (e.g., "Dex
Pixia Prime", "Shadow Claws Prime", "Landslide Fists Prime") is being calculated
independently; update the scraping/parsing logic so that when an item is marked
as frame-bound (look for the "isFrameBound" or owner reference fields) the code
in the estimator function (e.g., estimateVaultDate / parsePrimeItems) instead
inherits the owning Warframe's vault date (use the owner's
release/estimatedVaultDate) rather than computing a separate date; ensure this
change is applied wherever "estimatedVaultDate" is assigned for derived/exalted
items so they follow the owning frame's vault timeline and add tests/validation
for cases referenced by item names in the data.
- Line 18640: Remove estimatedVaultDate for primes that never enter the Prime
Vault by adding a whitelist check in the generator: in the PrimeGenerator (or
functions generatePrimeEntry / buildPrimeData / populateVaultInfo) detect if the
prime's identifier or name is one of the permanently-available legacy primes
(Akbronco Prime, Braton Prime, Bronco Prime, Burston Prime, Fang Prime, Lex
Prime, Orthos Prime, Paris Prime and the IDs 211588, 216479, 222600, 354624,
557500, 1050716, 1064805) and if so do not set estimatedVaultDate (remove/null
it) when building the JSON; then re-run the generator to regenerate All.json.
---
Nitpick comments:
In `@build/parser.mjs`:
- Around line 89-98: The RawItemData JSDoc typedef is incorrect for the
vaultData property: update the `@property` declaration for vaultData in the
RawItemData typedef so it reflects the actual returned type (Array<VaultData>)
instead of VaultData; locate the RawItemData typedef and change the vaultData
property type to an array type (e.g., Array<VaultData> or
Array<module:.../VaultData> to match existing JSDoc conventions used elsewhere
in the file).
In `@build/scraper.mjs`:
- Around line 268-273: The typedef VaultData declares estimatedVaultDate as
required but extractNotVaultedItems sometimes constructs VaultData without it;
update the contract by making estimatedVaultDate optional in the typedef (e.g.,
mark as [estimatedVaultDate]) or alternatively ensure extractNotVaultedItems
always populates estimatedVaultDate (compute from releaseDate) so the typedef
matches the actual objects; reference the VaultData typedef and the
extractNotVaultedItems function to locate and fix the mismatch.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 1f3a7743-ca6f-4411-a0d4-e4c63799bd6b
📒 Files selected for processing (16)
build/parser.mjsbuild/scraper.mjsdata/json/All.jsondata/json/Arch-Gun.jsondata/json/Archwing.jsondata/json/Melee.jsondata/json/Misc.jsondata/json/Mods.jsondata/json/Pets.jsondata/json/Primary.jsondata/json/Secondary.jsondata/json/SentinelWeapons.jsondata/json/Sentinels.jsondata/json/Warframes.jsondata/json/i18n.jsondata/warnings.json
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@build/parser.mjs`:
- Around line 1082-1085: The current code parses item.releaseDate then uses
local-time month arithmetic (getMonth/setMonth), which causes timezone-dependent
shifts; instead parse the input as UTC and perform UTC-based month arithmetic.
Replace the Date construction and setMonth call with a UTC-safe approach (e.g.
build the date with Date.UTC or append "T00:00:00Z" to item.releaseDate) and
call setUTCMonth (or construct a new Date from Date.UTC(year, month + 21, day))
so the computed value stored in item.estimatedVaultDate is derived using UTC and
then use toISOString().split('T')[0] as before.
In `@build/scraper.mjs`:
- Around line 332-334: The rows coming from neverVaulted are being emitted
identically to notYetVaulted, causing Parser.addVaultData to synthesize
estimatedVaultDate for items that should never be vaulted; modify
extractNotVaultedItems (or its call site where
$(neverVaulted).find(...).each(...)) to tag entries from neverVaulted (e.g., add
a boolean like neverVaulted: true or a source: 'neverVaulted' field in the
produced VaultData) and then update Parser.addVaultData to only synthesize
estimatedVaultDate for unvaulted entries that are not tagged neverVaulted (i.e.,
gate the fallback on the new flag/source).
- Around line 283-284: The current code assigns vaulted status by DOM order
using const tables = $('#mw-customcollapsible-vaulted > div > div >
table').toArray() and const [vaultedItems, formerlyVaulted, notYetVaulted,
neverVaulted] = tables; which is fragile; instead, find each table by a stable
identifier (e.g., the nearest heading text, table caption, or a sibling element
ID/class) by scanning the same container for tables and mapping them by their
heading/caption content (e.g., match strings like "Vaulted", "Formerly Vaulted",
"Not Yet Vaulted", "Never Vaulted") to variables
vaultedItems/formerlyVaulted/notYetVaulted/neverVaulted, and add an assertion
that all four expected sections were found before further processing so you fail
fast if the structure changed.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: fe25b9ae-e0bf-4e1e-a05f-df74fbd82afb
📒 Files selected for processing (2)
build/parser.mjsbuild/scraper.mjs
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (4)
build/wikia/scrapers/VaultScraper.mjs (3)
71-76: Preserve distinction between "not yet vaulted" and "never vaulted" items.Both table types produce identical
{ name, vaulted: false }output. If the parser synthesizes anestimatedVaultDatefor unvaulted entries, items that should never be vaulted (like certain Baro exclusives) will incorrectly appear as future vault candidates. Consider adding a flag (e.g.,neverVaulted: true) or updating the typedef to distinguish these categories.♻️ Suggested approach
+ function extractNeverVaultedItems(row) { + const $row = $(row); + if ($row.find('th').length) { + return; + } + const name = + $row.find('td:nth-child(1) > span').attr('data-param-name') ?? $row.find('td:nth-child(1) > a').text().trim(); + if (name) { + vaultData.push({ name, vaulted: false, neverVaulted: true }); + } + } $(notYetVaulted) .find('tbody > tr') .each((_, row) => extractNotVaultedItems(row)); $(neverVaulted) .find('tbody > tr') - .each((_, row) => extractNotVaultedItems(row)); + .each((_, row) => extractNeverVaultedItems(row));Then update the
VaultDatatypedef to include@property {boolean} [neverVaulted].🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@build/wikia/scrapers/VaultScraper.mjs` around lines 71 - 76, The parser currently treats both notYetVaulted and neverVaulted rows the same (using extractNotVaultedItems), producing { name, vaulted: false } and allowing estimatedVaultDate to be synthesized for items that should never be vaulted; update the logic so items coming from the neverVaulted selector are marked with an explicit neverVaulted: true (either by adding a parameter to extractNotVaultedItems or calling a small wrapper like extractNeverVaultedItem that sets neverVaulted), ensure vaulted remains false, and update the VaultData typedef to include an optional boolean property neverVaulted so consumers can distinguish never-vaulted entries from not-yet-vaulted ones.
17-20: JSDoc return type should reflect async signature.The
@returnsshould bePromise<Array<VaultData>>sincescrape()is an async function./** * Get (estimated) vault dates from wiki. - * `@returns` {Array<VaultData>} + * `@returns` {Promise<Array<VaultData>>} */🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@build/wikia/scrapers/VaultScraper.mjs` around lines 17 - 20, Update the JSDoc for the async scrape() method in VaultScraper to reflect its promise return type: change the `@returns` annotation from Array<VaultData> to Promise<Array<VaultData>> (or Promise<VaultData[]>), ensuring the documented type matches the async signature of scrape().
24-26: DOM order reliance is fragile—use stable identifiers to match tables to categories.The destructuring assignment on line 26 relies entirely on table order within the selector. If the wiki reorders tabs, items could be silently misclassified—for example, if "Not Yet Vaulted" moves to index 0, those rows would be extracted as
vaulted: true. The null check only validates that 4 tables exist, not which table is which.Instead of relying on position, match tables to their categories using stable identifiers: select each
.tabbertabby itsdata-titleattribute ("Vaulted Items", "Formerly Vaulted", etc.) and then extract the table within that tab.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@build/wikia/scrapers/VaultScraper.mjs` around lines 24 - 26, The current code in VaultScraper.mjs destructures tables from $('#mw-customcollapsible-vaulted > div > div > table') into [vaultedItems, formerlyVaulted, notYetVaulted, neverVaulted], which relies on DOM order and is fragile; instead, query the tab elements by their stable data-title attribute (select each .tabbertab[data-title="Vaulted Items"], .tabbertab[data-title="Formerly Vaulted"], .tabbertab[data-title="Not Yet Vaulted"], .tabbertab[data-title="Never Vaulted"]) and for each found tab extract the contained table element to assign to the variables vaultedItems, formerlyVaulted, notYetVaulted, neverVaulted (fall back to null if not present) so that table-to-category mapping is explicit and order-independent.build/scraper.mjs (1)
219-219: Consider importing VaultData typedef for JSDoc resolution.The
VaultDatatype is defined inVaultScraper.mjs. For better IDE support and type resolution, you could add a JSDoc import:/** `@typedef` {import('./wikia/scrapers/VaultScraper.mjs').VaultData} VaultData */This is optional—the code works without it, but tooling may not resolve the type.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@build/scraper.mjs` at line 219, Add a JSDoc typedef import for the VaultData type so IDEs can resolve the type; specifically, near the top of the scraper module add a JSDoc line that imports VaultData from the VaultScraper module (reference the VaultData typedef from VaultScraper.mjs) so the existing JSDoc property annotation "@property {Array<VaultData>} vaultData" can be resolved by tooling.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@build/wikia/scrapers/VaultScraper.mjs`:
- Around line 27-30: Replace the silent failure in the VaultScraper parsing
check by throwing an error instead of logging and returning []; in the block
that inspects vaultedItems, formerlyVaulted, notYetVaulted and neverVaulted (the
variables in VaultScraper.mjs), throw a new Error with a clear message (include
which tables/variables are missing if possible) so the build fails and the
exception propagates rather than proceeding with an empty array.
---
Nitpick comments:
In `@build/scraper.mjs`:
- Line 219: Add a JSDoc typedef import for the VaultData type so IDEs can
resolve the type; specifically, near the top of the scraper module add a JSDoc
line that imports VaultData from the VaultScraper module (reference the
VaultData typedef from VaultScraper.mjs) so the existing JSDoc property
annotation "@property {Array<VaultData>} vaultData" can be resolved by tooling.
In `@build/wikia/scrapers/VaultScraper.mjs`:
- Around line 71-76: The parser currently treats both notYetVaulted and
neverVaulted rows the same (using extractNotVaultedItems), producing { name,
vaulted: false } and allowing estimatedVaultDate to be synthesized for items
that should never be vaulted; update the logic so items coming from the
neverVaulted selector are marked with an explicit neverVaulted: true (either by
adding a parameter to extractNotVaultedItems or calling a small wrapper like
extractNeverVaultedItem that sets neverVaulted), ensure vaulted remains false,
and update the VaultData typedef to include an optional boolean property
neverVaulted so consumers can distinguish never-vaulted entries from
not-yet-vaulted ones.
- Around line 17-20: Update the JSDoc for the async scrape() method in
VaultScraper to reflect its promise return type: change the `@returns` annotation
from Array<VaultData> to Promise<Array<VaultData>> (or Promise<VaultData[]>),
ensuring the documented type matches the async signature of scrape().
- Around line 24-26: The current code in VaultScraper.mjs destructures tables
from $('#mw-customcollapsible-vaulted > div > div > table') into [vaultedItems,
formerlyVaulted, notYetVaulted, neverVaulted], which relies on DOM order and is
fragile; instead, query the tab elements by their stable data-title attribute
(select each .tabbertab[data-title="Vaulted Items"],
.tabbertab[data-title="Formerly Vaulted"], .tabbertab[data-title="Not Yet
Vaulted"], .tabbertab[data-title="Never Vaulted"]) and for each found tab
extract the contained table element to assign to the variables vaultedItems,
formerlyVaulted, notYetVaulted, neverVaulted (fall back to null if not present)
so that table-to-category mapping is explicit and order-independent.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 1c5fff7d-cd46-48e5-891c-1282d4708396
📒 Files selected for processing (8)
build/build.mjsbuild/parser.mjsbuild/scraper.mjsbuild/wikia/scrapers/VaultScraper.mjsdata/json/All.jsondata/json/Misc.jsondata/json/Resources.jsondata/json/i18n.json
💤 Files with no reviewable changes (1)
- build/build.mjs
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@build/sleep.mjs`:
- Around line 6-8: The default exported function in build/sleep.mjs currently
passes ms straight to setTimeout which lets undefined, NaN, or negative values
become immediate timeouts; update the exported function (the anonymous default
export) to validate ms is a finite non-negative number (Number.isFinite(ms) &&
ms >= 0) and throw a TypeError (or RangeError) for invalid inputs so bad
durations fail fast instead of changing runtime behavior; after validation, call
setTimeout(resolve, ms) as before.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: b61b5936-dd6d-4c7a-93cd-2afac5b78582
📒 Files selected for processing (5)
build/scraper.mjsbuild/sleep.mjsdata/json/All.jsondata/json/Resources.jsondata/json/i18n.json
🚧 Files skipped from review as they are similar to previous changes (1)
- build/scraper.mjs
TobiTenno
approved these changes
Mar 6, 2026
Collaborator
|
🎉 This PR is included in version 1.1272.131 🎉 The release is available on: Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What did you fix?
closes #868
Reproduction steps
oggtechnologies api went down, so we needed a new solution to include vault information for items
Evidence/screenshot/link to line
I updated/edited the following parts of the code:
https://github.com/DopiGFX/warframe-items/blob/d68c8fd7a0ac70728eee6e8ed4123b956c42a519/build/parser.mjs#L1076-L1086
https://github.com/DopiGFX/warframe-items/blob/d68c8fd7a0ac70728eee6e8ed4123b956c42a519/build/scraper.mjs#L263-L338
This here may be categorized as "opinionated data formatting or manual data entry" and needs review:
https://github.com/DopiGFX/warframe-items/blob/d68c8fd7a0ac70728eee6e8ed4123b956c42a519/build/scraper.mjs#L318-L321
Considerations
Summary by CodeRabbit
New Features
Bug Fixes
Chores