diff --git a/docs/source/tutorials/options.md b/docs/source/tutorials/options.md index 1fa675d0b4..b1c2315b32 100644 --- a/docs/source/tutorials/options.md +++ b/docs/source/tutorials/options.md @@ -140,6 +140,8 @@ It defaults to `false`. For example, when set to `true`, a blank string would ev **ownPropertyOnly** hides scope variables from prototypes, useful when you're passing a not sanitized object into LiquidJS or need to hide prototypes from templates. Defaults to `true`. +Built-in DoS limits and host isolation guidance are documented in [Security Model](./security-model.html). + {% note info Nonexistent Tags %} Nonexistent tags always throw errors during parsing and this behavior cannot be customized. {% endnote %} diff --git a/docs/source/tutorials/render-tag-content.md b/docs/source/tutorials/render-tag-content.md index 33195b0901..cf655207fe 100644 --- a/docs/source/tutorials/render-tag-content.md +++ b/docs/source/tutorials/render-tag-content.md @@ -22,7 +22,7 @@ Expected output: ``` -Firstly, [register][register-tags] a tag named `wrap` and parse the content into `this.tpls`. Here in `parse(tagToken, remainTokens)`: +Firstly, [register][register-tags] a tag named `wrap` and parse the content into `this.tpls`. In the tag `constructor(tagToken, remainTokens, liquid)`: - `tagToken` is current token `{%raw%}{% wrap %}{%endraw%}`, and - `remainTokens` is an array of all tokens following `{%raw%}{% wrap %}{%endraw%}` until the end of this template file. @@ -30,11 +30,14 @@ Firstly, [register][register-tags] a tag named `wrap` and parse the content into Basically, what we need to do is take/`.shift()` enough tags from `remainTokens` until we get an `endwrap` token (the name can be arbitrary, but by convention it should be `endwrap`). And if there's no `endwrap` until the end of the template file, we need to throw a tag-not-closed `Error`. ```javascript -engine.registerTag('wrap', { - parse(tagToken, remainTokens) { - this.tpls = [] +const { Tag } = require('liquidjs') + +engine.registerTag('wrap', class WrapTag extends Tag { + tpls = [] + constructor(tagToken, remainTokens, liquid) { + super(tagToken, remainTokens, liquid) let closed = false - while(remainTokens.length) { + while (remainTokens.length) { let token = remainTokens.shift() // we got the end tag! stop taking tokens if (token.name === 'endwrap') { @@ -44,11 +47,11 @@ engine.registerTag('wrap', { // parse token into template // parseToken() may consume more than 1 tokens // e.g. {% if %}...{% endif %} - let tpl = this.liquid.parser.parseToken(token, remainTokens) + let tpl = liquid.parser.parseToken(token, remainTokens) this.tpls.push(tpl) } if (!closed) throw new Error(`tag ${tagToken.getText()} not closed`) - }, + } * render(context, emitter) { emitter.write("
") yield this.liquid.renderer.renderTemplates(this.tpls, context, emitter) @@ -57,16 +60,17 @@ engine.registerTag('wrap', { }) ``` -`.renderTemplates()` can be async; we need `yield` to wait for it to complete. For more details on async in LiquidJS, see [Sync and Async][async]. Other parts of the `render()` method are quite straightforward. Here's a JSFiddle version: +`.renderTemplates()` can be async; we need `yield` to wait for it to complete. For more details on async in LiquidJS, see [Sync and Async][async]. Here's a JSFiddle version: ## Using ParseStream -When it comes to complex tags like [for][for] and [if][if], the `parse()` can be very complicated. There's a [ParseStream][ParseStream] utility to organize the `parse()` in event-based style. Following is a re-written `parse()` using `ParseStream` that does exactly the same as the example above. +For more complex tags such as [for][for] and [if][if], constructor parsing can get unwieldy. [ParseStream][ParseStream] offers an event-based API for this. The constructor below is equivalent to the example above: ```javascript -parse(tagToken, remainTokens) { - this.tpls = [] - this.liquid.parser.parseStream(remainTokens) +tpls = [] +constructor(tagToken, remainTokens, liquid) { + super(tagToken, remainTokens, liquid) + liquid.parser.parseStream(remainTokens) .on('template', tpl => this.tpls.push(tpl)) // note that we cannot use arrow function because we need `this` .on('tag:endwrap', function () { this.stop() }) @@ -103,15 +107,18 @@ As you've noticed, there's an additional `repeat.i` in the context of `repeat`. Each time we enter a new *Context*, we need to push a new *Scope*. And when we finish rendering and exit the *Context*, we pop the *Scope* from the *Context*. As you can see in the following implementation: ```javascript -engine.registerTag('repeat', { - parse(tagToken, remainTokens) { - this.tpls = [] - this.liquid.parser.parseStream(remainTokens) +const { Tag } = require('liquidjs') + +engine.registerTag('repeat', class RepeatTag extends Tag { + tpls = [] + constructor(tagToken, remainTokens, liquid) { + super(tagToken, remainTokens, liquid) + liquid.parser.parseStream(remainTokens) .on('template', tpl => this.tpls.push(tpl)) .on('tag:endrepeat', function () { this.stop() }) .on('end', () => { throw new Error(`tag ${tagToken.getText()} not closed`) }) .start() - }, + } * render(context, emitter) { const repeat = { i: 1 } context.push({ repeat }) @@ -123,7 +130,7 @@ engine.registerTag('repeat', { }) ``` -The `parse()` is exactly the same as `wrap` tag, we repeat the content simply by calling `.renderTemplates(this.tpls)` twice during `render()`. Here's the JSFiddle: +The constructor is the same as in the `wrap` tag; we repeat the content by calling `.renderTemplates(this.tpls)` twice during `render()`. Here's the JSFiddle: {% note warn Use Push & Pop in Pairs %} `context.push()` and `context.pop()` have to be used in pairs. Failing to `pop()` the *Scope* you pushed will leak the *Scope* to latter templates and may corrupt the *Context* stack. diff --git a/docs/source/tutorials/security-model.md b/docs/source/tutorials/security-model.md index 5d65a01967..3c6b4aa746 100644 --- a/docs/source/tutorials/security-model.md +++ b/docs/source/tutorials/security-model.md @@ -2,21 +2,19 @@ title: Security Model --- -LiquidJS provides DoS-oriented limits (`parseLimit`, `renderLimit`, `memoryLimit`) to reduce risk. This page summarizes those limits, [`ownPropertyOnly`][ownPropertyOnly], custom [`Drop`][drop] usage, and the security boundary to assume in production. +LiquidJS provides DoS-oriented limits (`parseLimit`, `templateLimit`, `outputLengthLimit`, `maxDepth`) to reduce risk. This page summarizes those limits, [`ownPropertyOnly`][ownPropertyOnly], custom [`Drop`][drop] usage, and the security boundary to assume in production. -## Security boundary +## At a glance -The built-in limits are cooperative safeguards, not strict runtime isolation. - -- They do **not** equal process RSS/heap usage. -- They do **not** sandbox JavaScript execution. -- They should be combined with process/container limits and request timeouts for defense in depth. - -## Limits at a glance +LiquidJS ships a thin cooperative DoS layer: - [parseLimit][parseLimit]: limit total template size per `parse()` call. -- [renderLimit][renderLimit]: limit total render time per `render()` call. -- [memoryLimit][memoryLimit]: cooperatively limit memory-sensitive allocations counted by LiquidJS. +- [templateLimit][templateLimit]: limit total tag/HTML/output nodes rendered per `render()` call. +- [outputLengthLimit][outputLengthLimit]: limit total output length per `render()` call. +- [maxDepth][maxDepth]: limit nesting depth of `{% render %}`, `{% include %}`, and `{% layout %}`. +- Strftime numeric pad widths in the `date` filter are capped at `1_000_000` (1M) per conversion. + +These are cooperative safeguards, not runtime isolation—see [Production guidance](#production-guidance) below for host-level limits and online-service hardening. ## Limit details @@ -26,9 +24,9 @@ The built-in limits are cooperative safeguards, not strict runtime isolation. A typical PC handles `1e8` (100M) characters without issues. -### renderLimit +### templateLimit -Restricting template size alone is insufficient because dynamic loops with large counts can occur during rendering. [renderLimit][renderLimit] mitigates this by limiting the time consumed by each `render()` call. +Restricting template size alone is insufficient because dynamic loops with large counts can occur during rendering. [templateLimit][templateLimit] mitigates this by limiting the number of tag, HTML literal, and output nodes rendered in each `render()` call. ```liquid {%- for i in (1..10000000) -%} @@ -36,29 +34,19 @@ Restricting template size alone is insufficient because dynamic loops with large {%- endfor -%} ``` -Render time is checked on a per-template basis (before rendering each template). In the above example, there are 2 templates in the loop: `order: ` and `{{i}}`, render time will be checked 10000000x2 times. - -`renderLimit` is not a hard CPU limiter. It is checked between template renders, so compute-intensive filters/tags/user-defined functions or deeply nested template execution between checks can still cause DoS. +Each template node (the `for` tag, literal `order: `, output `{{i}}`, and so on) counts toward the limit. In the above example, a limit of `30000000` would be exceeded before the loop finishes. -### memoryLimit +`templateLimit` is checked before each node render, so compute-intensive filters/tags/user-defined functions between checks can still cause DoS. -`memoryLimit` only limits operations that LiquidJS explicitly counts. +### outputLengthLimit -- Counted: memory-sensitive LiquidJS operations that call internal memory accounting. -- Not guaranteed counted: arbitrary user object behavior such as custom `toValue()`/`toString()` chains, or other host-side code that allocates outside LiquidJS accounting points. +[outputLengthLimit][outputLengthLimit] caps the cumulative length of output written during a `render()` call, including output from partials rendered via `{% render %}`. -In other words, `memoryLimit` limits what LiquidJS counts, not every byte your process may allocate. +### maxDepth -Even with a small number of templates and iterations, memory usage can grow exponentially. In the following example, memory doubles with each iteration: +[maxDepth][maxDepth] limits how deeply `{% render %}`, `{% include %}`, and `{% layout %}` can nest. Defaults to `128`. In sync rendering (`renderSync`), nested tags are driven by `toValueSync`, which recursively resumes each yielded generator on the call stack—deep nesting can overflow it, and `maxDepth` caps that depth. Async `render()` resumes the same tag generators via `toPromise`/`yield` without a deep synchronous call chain, so stack overflow is not a concern there (the limit still applies as a DoS guard). -```liquid -{% assign array = "1,2,3" | split: "," %} -{% for i in (1..32) %} - {% assign array = array | concat: array %} -{% endfor %} -``` - -As [JavaScript uses GC to manage memory](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Memory_management), `memoryLimit` may not reflect the actual memory footprint. +The `memoryLimit` option was removed in v11; enforce memory limits at the host or process level instead. ## `ownPropertyOnly` and scope data @@ -68,20 +56,21 @@ With [`ownPropertyOnly`][ownPropertyOnly] `true`, plain scope objects only expos [`Drop`][drop] values are not restricted the same way: LiquidJS still reads the prototype chain and may call [`liquidMethodMissing`][liquidMethodMissing]. **You** control what a drop exposes; narrow APIs and never feed unsafe data into drops unless the class is built for template access. `ownPropertyOnly` alone does not harden custom drops—audit them like any privileged code. -## Online service guidance - -If you run an online service, avoid rendering fully user-defined templates whenever possible. +## Production guidance -- Prefer curated templates or a restricted template subset. -- If user-defined templates are required, isolate rendering (worker/process/container), enforce OS/container memory and CPU limits, and apply request rate limits. -- Treat `parseLimit`/`renderLimit`/`memoryLimit` as one layer in a broader DoS defense strategy. +LiquidJS does not sandbox template code—custom filters, tags, and scope helpers run as ordinary JavaScript with your process privileges. Built-in DoS limits are one layer; production deployments, especially online services that accept template input, need additional hardening: -For heavy single-template operations, process-level isolation is still recommended (for example with [paralleljs][paralleljs]). +- **Prefer curated templates** over fully user-defined Liquid when possible; if users need customization, offer a restricted subset rather than open template editing. +- Run each render in a **worker thread or child process** with a wall-clock timeout; **kill** the worker on expiry. Libraries such as [paralleljs][paralleljs] can help for heavy single-template work. +- Enforce **container/Kubernetes cgroup limits**, `ulimit`, or equivalent on the renderer process for memory and CPU. +- Apply **request rate limits** at the API or gateway layer. +- **`node:vm`, `isolated-vm`, and Jinja/Twig-style sandbox modes are not a security boundary**—template logic runs in the same JS runtime as your app, with your privileges. [paralleljs]: https://www.npmjs.com/package/paralleljs [parseLimit]: /api/interfaces/LiquidOptions.html#parseLimit -[renderLimit]: /api/interfaces/LiquidOptions.html#renderLimit -[memoryLimit]: /api/interfaces/LiquidOptions.html#memoryLimit +[templateLimit]: /api/interfaces/LiquidOptions.html#templateLimit +[outputLengthLimit]: /api/interfaces/LiquidOptions.html#outputLengthLimit +[maxDepth]: /api/interfaces/LiquidOptions.html#maxDepth [ownPropertyOnly]: /api/interfaces/LiquidOptions.html#ownPropertyOnly [renderOwnPropertyOnly]: /api/interfaces/RenderOptions.html#ownPropertyOnly [strictVariables]: /api/interfaces/LiquidOptions.html#strictVariables diff --git a/docs/themes/navy/source/js/main.js b/docs/themes/navy/source/js/main.js index 1d4e5eb4cd..e55e35ea07 100644 --- a/docs/themes/navy/source/js/main.js +++ b/docs/themes/navy/source/js/main.js @@ -42,8 +42,7 @@ if (!/\/playground(?:\.html)?$/.test(location.pathname)) return; updateVersion(liquidjs.version); const engine = new liquidjs.Liquid({ - memoryLimit: 1e5, - renderLimit: 1e5 + templateLimit: 1e5 }); const colorScheme = window.matchMedia('(prefers-color-scheme: dark)'); const editor = createEditor('editorEl', 'liquid'); diff --git a/src/context/context.ts b/src/context/context.ts index 9dcb711ea7..fbc17f8d8c 100644 --- a/src/context/context.ts +++ b/src/context/context.ts @@ -1,4 +1,3 @@ -import { getPerformance } from '../util/performance' import { Drop } from '../drop/drop' import { __assign } from 'tslib' import { NormalizedFullOptions, defaultOptions, RenderOptions } from '../liquid-options' @@ -36,17 +35,19 @@ export class Context { */ public strictVariables: boolean; public ownPropertyOnly: boolean; - public memoryLimit: Limiter; - public renderLimit: Limiter; - public constructor (env: object = {}, opts: NormalizedFullOptions = defaultOptions, renderOptions: RenderOptions = {}, { memoryLimit, renderLimit }: { [key: string]: Limiter } = {}) { + public templateLimit: Limiter; + public outputLengthLimit: Limiter; + public depthLimit: Limiter; + public constructor (env: object = {}, opts: NormalizedFullOptions = defaultOptions, renderOptions: RenderOptions = {}, { templateLimit, outputLengthLimit, depthLimit }: { templateLimit?: Limiter, outputLengthLimit?: Limiter, depthLimit?: Limiter } = {}) { this.sync = !!renderOptions.sync this.opts = opts this.globals = renderOptions.globals ?? opts.globals this.environments = isObject(env) ? env : Object(env) this.strictVariables = renderOptions.strictVariables ?? this.opts.strictVariables this.ownPropertyOnly = renderOptions.ownPropertyOnly ?? opts.ownPropertyOnly - this.memoryLimit = memoryLimit ?? new Limiter('memory alloc', renderOptions.memoryLimit ?? opts.memoryLimit) - this.renderLimit = renderLimit ?? new Limiter('template render', getPerformance().now() + (renderOptions.renderLimit ?? opts.renderLimit)) + this.templateLimit = templateLimit ?? new Limiter('template', renderOptions.templateLimit ?? opts.templateLimit) + this.outputLengthLimit = outputLengthLimit ?? new Limiter('output length', renderOptions.outputLengthLimit ?? opts.outputLengthLimit) + this.depthLimit = depthLimit ?? new Limiter('template depth', opts.maxDepth) } public getRegister (key: string, defaultValue: T = undefined as T): T { return (this.registers[key] = this.registers[key] || defaultValue) @@ -109,8 +110,9 @@ export class Context { strictVariables: this.strictVariables, ownPropertyOnly: this.ownPropertyOnly }, { - renderLimit: this.renderLimit, - memoryLimit: this.memoryLimit + templateLimit: this.templateLimit, + outputLengthLimit: this.outputLengthLimit, + depthLimit: this.depthLimit }) } private findScope (key: string | number) { diff --git a/src/emitters/simple-emitter.ts b/src/emitters/simple-emitter.ts index f1d0482068..3de1e368f5 100644 --- a/src/emitters/simple-emitter.ts +++ b/src/emitters/simple-emitter.ts @@ -1,10 +1,17 @@ -import { stringify } from '../util' +import { stringify, Limiter } from '../util' import { Emitter } from './emitter' export class SimpleEmitter implements Emitter { public buffer = ''; + private outputLengthLimit?: Limiter + + constructor (outputLengthLimit?: Limiter) { + this.outputLengthLimit = outputLengthLimit + } public write (html: any) { - this.buffer += stringify(html) + const str = stringify(html) + this.outputLengthLimit?.use(str.length) + this.buffer += str } } diff --git a/src/emitters/streamed-emitter.ts b/src/emitters/streamed-emitter.ts index 6750ea3b4e..a9c93e4c5b 100644 --- a/src/emitters/streamed-emitter.ts +++ b/src/emitters/streamed-emitter.ts @@ -1,12 +1,20 @@ -import { stringify } from '../util' +import { stringify, Limiter } from '../util' import { Emitter } from './emitter' import { PassThrough } from 'stream' export class StreamedEmitter implements Emitter { public buffer = ''; public stream: NodeJS.ReadWriteStream = new PassThrough() + private outputLengthLimit?: Limiter + + constructor (outputLengthLimit?: Limiter) { + this.outputLengthLimit = outputLengthLimit + } + public write (html: any) { - this.stream.write(stringify(html)) + const str = stringify(html) + this.outputLengthLimit?.use(str.length) + this.stream.write(str) } public error (err: Error) { this.stream.emit('error', err) diff --git a/src/filters/array.ts b/src/filters/array.ts index bf16b59511..2a5cb8a0e0 100644 --- a/src/filters/array.ts +++ b/src/filters/array.ts @@ -8,9 +8,6 @@ import { EmptyDrop } from '../drop' export const join = argumentsToValue(function (this: FilterImpl, v: any[], arg: string) { const array = toArray(v) const sep = isNil(arg) ? ' ' : stringify(arg) - let outputSize = sep.length * Math.max(array.length - 1, 0) - for (let i = 0; i < array.length; i++) outputSize += String(array[i]).length - this.context.memoryLimit.use(outputSize) return Array.prototype.join.call(array, sep) }) export const last = argumentsToValue(function (this: FilterImpl, v: any) { @@ -21,14 +18,12 @@ export const first = argumentsToValue(function (this: FilterImpl, v: any) { }) export const reverse = argumentsToValue(function (this: FilterImpl, v: any[]) { const array = toArray(v) - this.context.memoryLimit.use(array.length) return [...array].reverse() }) function * sortBy (this: FilterImpl, arr: T[], property: string | undefined, comparator: (a: unknown, b: unknown) => number): IterableIterator { const values: [T, unknown][] = [] const array = toArray(arr) - this.context.memoryLimit.use(array.length) for (const item of array) { values.push([ item, @@ -51,7 +46,6 @@ export const size = (v: string | any[]) => v?.length || 0 export function * map (this: FilterImpl, arr: Scope[], property: string): IterableIterator { const results = [] const array = toArray(arr) - this.context.memoryLimit.use(array.length) for (const item of array) { results.push(yield this.context._getFromScope(item, stringify(property), false)) } @@ -70,14 +64,12 @@ export function * sum (this: FilterImpl, arr: Scope[], property?: string): Itera export function compact (this: FilterImpl, arr: T[]) { const array = toArray(arr) - this.context.memoryLimit.use(array.length) return Array.prototype.filter.call(array, x => !isNil(toValue(x))) } export function concat (this: FilterImpl, v: T1[], arg: T2[] = []): (T1 | T2)[] { const lhs = toArray(v) const rhs = toArray(arg) - this.context.memoryLimit.use(lhs.length + rhs.length) return Array.prototype.concat.call(lhs, rhs) } @@ -87,7 +79,6 @@ export function push (this: FilterImpl, v: T[], arg: T): T[] { export function unshift (this: FilterImpl, v: T[], arg: T): T[] { const array = toArray(v) - this.context.memoryLimit.use(array.length) const clone = [...array] clone.unshift(arg) return clone @@ -95,7 +86,6 @@ export function unshift (this: FilterImpl, v: T[], arg: T): T[] { export function pop (this: FilterImpl, v: T[]): T[] { const array = toArray(v) - this.context.memoryLimit.use(array.length) const clone = [...array] clone.pop() return clone @@ -103,7 +93,6 @@ export function pop (this: FilterImpl, v: T[]): T[] { export function shift (this: FilterImpl, v: T[]): T[] { const array = toArray(v) - this.context.memoryLimit.use(array.length) const clone = [...array] clone.shift() return clone @@ -114,7 +103,6 @@ export function slice (this: FilterImpl, v: T[] | string, begin: number, leng if (isNil(v)) return [] if (!isArray(v)) v = stringify(v) begin = begin < 0 ? v.length + begin : begin - this.context.memoryLimit.use(length) return isArray(v) ? Array.prototype.slice.call(v, begin, begin + length) : String.prototype.slice.call(v, begin, begin + length) @@ -133,7 +121,6 @@ function expectedMatcher (this: FilterImpl, expected: any): (v: any) => boolean function * filter (this: FilterImpl, include: boolean, arr: T[], property: string, expected: any): IterableIterator { const values: unknown[] = [] arr = toArray(arr) - this.context.memoryLimit.use(arr.length) const token = new Tokenizer(stringify(property)).readScopeValue() for (const item of arr) { values.push(yield evalToken(token, this.context.spawn(item))) @@ -146,7 +133,6 @@ function * filter_exp (this: FilterImpl, include: boolean, arr const filtered: unknown[] = [] const keyTemplate = new Value(stringify(exp), this.liquid) const array = toArray(arr) - this.context.memoryLimit.use(array.length) for (const item of array) { this.context.push({ [itemName]: item }) const value = yield keyTemplate.value(this.context) @@ -176,7 +162,6 @@ export function * group_by (this: FilterImpl, arr: T[], proper const map = new Map() arr = toEnumerable(arr) const token = new Tokenizer(stringify(property)).readScopeValue() - this.context.memoryLimit.use(arr.length) for (const item of arr) { const key = yield evalToken(token, this.context.spawn(item)) if (!map.has(key)) map.set(key, []) @@ -189,7 +174,6 @@ export function * group_by_exp (this: FilterImpl, arr: T[], it const map = new Map() const keyTemplate = new Value(stringify(exp), this.liquid) arr = toEnumerable(arr) - this.context.memoryLimit.use(arr.length) for (const item of arr) { this.context.push({ [itemName]: item }) const key = yield keyTemplate.value(this.context) @@ -253,7 +237,6 @@ export function * find_exp (this: FilterImpl, arr: T[], itemNa export function uniq (this: FilterImpl, arr: T[]): T[] { arr = toArray(arr) - this.context.memoryLimit.use(arr.length) return [...new Set(arr)] } @@ -261,7 +244,6 @@ export function sample (this: FilterImpl, v: T[] | string, count = 1): T | st v = toValue(v) if (isNil(v)) return [] if (!isArray(v)) v = stringify(v) - this.context.memoryLimit.use(v.length) const shuffled = [...v].sort(() => Math.random() - 0.5) if (count === 1) return shuffled[0] return shuffled.slice(0, count) diff --git a/src/filters/base64.ts b/src/filters/base64.ts index 29ef84e52c..5324c74732 100644 --- a/src/filters/base64.ts +++ b/src/filters/base64.ts @@ -10,16 +10,13 @@ import { base64Encode, base64Decode } from './base64-impl' export function base64_encode (this: FilterImpl, value: string | Buffer): string { if (typeof Buffer !== 'undefined' && Buffer.isBuffer(value)) { - this.context.memoryLimit.use(value.byteLength) return value.toString('base64') } const str = stringify(value) - this.context.memoryLimit.use(str.length) return base64Encode(str) } export function base64_decode (this: FilterImpl, value: string): string { const str = stringify(value) - this.context.memoryLimit.use(str.length) return base64Decode(str) } diff --git a/src/filters/crypto.ts b/src/filters/crypto.ts index 7210523bde..26caebd267 100644 --- a/src/filters/crypto.ts +++ b/src/filters/crypto.ts @@ -10,13 +10,11 @@ import { sha256 as sha256Impl, hmacSha256 as hmacSha256Impl } from './crypto-imp export function sha256 (this: FilterImpl, value: unknown): string | Promise { const str = stringify(value) - this.context.memoryLimit.use(str.length) return sha256Impl(str) } export function hmac_sha256 (this: FilterImpl, value: unknown, key: unknown): string | Promise { const str = stringify(value) const keyStr = stringify(key) - this.context.memoryLimit.use(str.length + keyStr.length) return hmacSha256Impl(str, keyStr) } diff --git a/src/filters/date.ts b/src/filters/date.ts index 07665c9579..e2567421b1 100644 --- a/src/filters/date.ts +++ b/src/filters/date.ts @@ -3,14 +3,11 @@ import { FilterImpl } from '../template' import { NormalizedFullOptions } from '../liquid-options' export function date (this: FilterImpl, v: string | Date, format?: string, timezoneOffset?: number | string) { - const size = ((v as string)?.length ?? 0) + ((timezoneOffset as string)?.length ?? 0) - this.context.memoryLimit.use(size) const date = parseDate(v, this.context.opts, timezoneOffset) if (!date) return v format = toValue(format) format = isNil(format) ? this.context.opts.dateFormat : stringify(format) - this.context.memoryLimit.use(format.length) - return strftime(date, format, this.context.memoryLimit) + return strftime(date, format) } export function date_to_xmlschema (this: FilterImpl, v: string | Date) { @@ -32,14 +29,13 @@ export function date_to_long_string (this: FilterImpl, v: string | Date, type?: function stringify_date (this: FilterImpl, v: string | Date, month_type: string, type?: string, style?: string) { const date = parseDate(v, this.context.opts) if (!date) return v - const ml = this.context.memoryLimit if (type === 'ordinal') { const d = date.getDate() return style === 'US' - ? strftime(date, `${month_type} ${d}%q, %Y`, ml) - : strftime(date, `${d}%q ${month_type} %Y`, ml) + ? strftime(date, `${month_type} ${d}%q, %Y`) + : strftime(date, `${d}%q ${month_type} %Y`) } - return strftime(date, `%d ${month_type} %Y`, ml) + return strftime(date, `%d ${month_type} %Y`) } function parseDate (v: string | Date, opts: NormalizedFullOptions, timezoneOffset?: number | string): LiquidDate | undefined { diff --git a/src/filters/html.ts b/src/filters/html.ts index b52f1b3da2..01cd67d654 100644 --- a/src/filters/html.ts +++ b/src/filters/html.ts @@ -18,7 +18,6 @@ const unescapeMap: Record = { export function escape (this: FilterImpl, str: string) { str = stringify(str) - this.context.memoryLimit.use(str.length) return str.replace(/&|<|>|"|'/g, m => escapeMap[m]) } @@ -28,7 +27,6 @@ export function xml_escape (this: FilterImpl, str: string) { function unescape (this: FilterImpl, str: string) { str = stringify(str) - this.context.memoryLimit.use(str.length) return str.replace(/&(amp|lt|gt|#34|#39);/g, m => unescapeMap[m]) } @@ -38,7 +36,6 @@ export function escape_once (this: FilterImpl, str: string) { export function newline_to_br (this: FilterImpl, v: string) { const str = stringify(v) - this.context.memoryLimit.use(str.length) return str.replace(/\r?\n/gm, '
\n') } @@ -46,7 +43,6 @@ export function newline_to_br (this: FilterImpl, v: string) { // equivalent is O(n^2) in V8 on unclosed openers. export function strip_html (this: FilterImpl, v: string) { const str = stringify(v) - this.context.memoryLimit.use(str.length) const blocks = new Map([[''], [''], [''], ['<', '>']]) let out = '' let i = 0 diff --git a/src/filters/misc.ts b/src/filters/misc.ts index 4473765508..8a24ba7545 100644 --- a/src/filters/misc.ts +++ b/src/filters/misc.ts @@ -2,18 +2,6 @@ import { isFalsy } from '../render/boolean' import { identify, isArray, isString, toValue } from '../util/underscore' import { FilterImpl } from '../template' -function chargeJsonReplacerValue (memoryLimit: { use(count: number): void }, val: unknown) { - if (typeof val === 'string') { - memoryLimit.use(val.length) - } else if (val === null || typeof val === 'number' || typeof val === 'boolean') { - memoryLimit.use(JSON.stringify(val).length) - } else if (Array.isArray(val)) { - memoryLimit.use(val.length + 1) - } else if (typeof val === 'object') { - memoryLimit.use(2) - } -} - function defaultFilter (this: FilterImpl, value: T1, defaultValue: T2, ...args: Array<[string, any]>): T1 | T2 { value = toValue(value) if (isArray(value) || isString(value)) return value.length ? value : defaultValue @@ -22,29 +10,21 @@ function defaultFilter (this: FilterImpl, value: T1, def } function json (this: FilterImpl, value: any, space = 0) { - const memoryLimit = this.context.memoryLimit - return JSON.stringify(value, (_key, val) => { - chargeJsonReplacerValue(memoryLimit, val) - return val - }, space) + return JSON.stringify(value, undefined, space) } function inspect (this: FilterImpl, value: any, space = 0) { - const memoryLimit = this.context.memoryLimit const ancestors: object[] = [] return JSON.stringify(value, function (this: unknown, _key: unknown, value: any) { if (typeof value !== 'object' || value === null) { - chargeJsonReplacerValue(memoryLimit, value) return value } // `this` is the object that value is contained in, i.e., its direct parent. while (ancestors.length > 0 && ancestors[ancestors.length - 1] !== this) ancestors.pop() if (ancestors.includes(value)) { - memoryLimit.use('[Circular]'.length) return '[Circular]' } ancestors.push(value) - chargeJsonReplacerValue(memoryLimit, value) return value }, space) } diff --git a/src/filters/string.ts b/src/filters/string.ts index d0c59708ed..3d19ace6a2 100644 --- a/src/filters/string.ts +++ b/src/filters/string.ts @@ -22,7 +22,6 @@ export function append (this: FilterImpl, v: string, arg: string) { assert(arguments.length === 2, 'append expect 2 arguments') const lhs = stringify(v) const rhs = stringify(arg) - this.context.memoryLimit.use(lhs.length + rhs.length) return lhs + rhs } @@ -30,16 +29,13 @@ export function prepend (this: FilterImpl, v: string, arg: string) { assert(arguments.length === 2, 'prepend expect 2 arguments') const lhs = stringify(v) const rhs = stringify(arg) - this.context.memoryLimit.use(lhs.length + rhs.length) return rhs + lhs } export function lstrip (this: FilterImpl, v: string, chars?: string) { const str = stringify(v) - this.context.memoryLimit.use(str.length) if (chars) { chars = stringify(chars) - this.context.memoryLimit.use(chars.length) for (let i = 0, set = new Set(chars); i < str.length; i++) { if (!set.has(str[i])) return str.slice(i) } @@ -50,34 +46,29 @@ export function lstrip (this: FilterImpl, v: string, chars?: string) { export function downcase (this: FilterImpl, v: string) { const str = stringify(v) - this.context.memoryLimit.use(str.length) return str.toLowerCase() } export function upcase (this: FilterImpl, v: string) { const str = stringify(v) - this.context.memoryLimit.use(str.length) return stringify(str).toUpperCase() } export function remove (this: FilterImpl, v: string, arg: string) { const str = stringify(v) arg = stringify(arg) - this.context.memoryLimit.use(str.length + arg.length) return str.split(arg).join('') } export function remove_first (this: FilterImpl, v: string, l: string) { const str = stringify(v) l = stringify(l) - this.context.memoryLimit.use(str.length + l.length) return str.replace(l, '') } export function remove_last (this: FilterImpl, v: string, l: string) { const str = stringify(v) const pattern = stringify(l) - this.context.memoryLimit.use(str.length + pattern.length) const index = str.lastIndexOf(pattern) if (index === -1) return str return str.substring(0, index) + str.substring(index + pattern.length) @@ -85,10 +76,8 @@ export function remove_last (this: FilterImpl, v: string, l: string) { export function rstrip (this: FilterImpl, str: string, chars?: string) { str = stringify(str) - this.context.memoryLimit.use(str.length) if (chars) { chars = stringify(chars) - this.context.memoryLimit.use(chars.length) for (let i = str.length - 1, set = new Set(chars); i >= 0; i--) { if (!set.has(str[i])) return str.slice(0, i + 1) } @@ -99,7 +88,6 @@ export function rstrip (this: FilterImpl, str: string, chars?: string) { export function split (this: FilterImpl, v: string, arg: string) { const str = stringify(v) - this.context.memoryLimit.use(str.length) const arr = str.split(stringify(arg)) // align to ruby split, which is the behavior of shopify/liquid // see: https://ruby-doc.org/core-2.4.0/String.html#method-i-split @@ -109,10 +97,8 @@ export function split (this: FilterImpl, v: string, arg: string) { export function strip (this: FilterImpl, v: string, chars?: string) { const str = stringify(v) - this.context.memoryLimit.use(str.length) if (chars) { const set = new Set(stringify(chars)) - this.context.memoryLimit.use(set.size) let i = 0 let j = str.length - 1 while (set.has(str[i])) i++ @@ -124,13 +110,11 @@ export function strip (this: FilterImpl, v: string, chars?: string) { export function strip_newlines (this: FilterImpl, v: string) { const str = stringify(v) - this.context.memoryLimit.use(str.length) return str.replace(/\r?\n/gm, '') } export function capitalize (this: FilterImpl, str: string) { str = stringify(str) - this.context.memoryLimit.use(str.length) return str.charAt(0).toUpperCase() + str.slice(1).toLowerCase() } @@ -139,8 +123,6 @@ export function replace (this: FilterImpl, v: string, pattern: string, replaceme pattern = stringify(pattern) replacement = stringify(replacement) const parts = str.split(pattern) - const outputSize = str.length + (parts.length - 1) * (replacement.length - pattern.length) - this.context.memoryLimit.use(outputSize) return parts.join(replacement) } @@ -148,7 +130,6 @@ export function replace_first (this: FilterImpl, v: string, arg1: string, arg2: const str = stringify(v) arg1 = stringify(arg1) arg2 = stringify(arg2) - this.context.memoryLimit.use(str.length + arg1.length + arg2.length) return str.replace(arg1, () => arg2) } @@ -156,7 +137,6 @@ export function replace_last (this: FilterImpl, v: string, arg1: string, arg2: s const str = stringify(v) const pattern = stringify(arg1) const replacement = stringify(arg2) - this.context.memoryLimit.use(str.length + pattern.length + replacement.length) const index = str.lastIndexOf(pattern) if (index === -1) return str return str.substring(0, index) + replacement + str.substring(index + pattern.length) @@ -165,7 +145,6 @@ export function replace_last (this: FilterImpl, v: string, arg1: string, arg2: s export function truncate (this: FilterImpl, v: string, l = 50, o = '...') { const str = stringify(v) o = stringify(o) - this.context.memoryLimit.use(str.length + o.length) if (str.length <= l) return v return str.substring(0, l - o.length) + o } @@ -173,7 +152,6 @@ export function truncate (this: FilterImpl, v: string, l = 50, o = '...') { export function truncatewords (this: FilterImpl, v: string, words = 15, o = '...') { const str = stringify(v) o = stringify(o) - this.context.memoryLimit.use(str.length + o.length) const arr = str.split(/\s+/) if (words <= 0) words = 1 let ret = arr.slice(0, words).join(' ') @@ -183,13 +161,11 @@ export function truncatewords (this: FilterImpl, v: string, words = 15, o = '... export function normalize_whitespace (this: FilterImpl, v: string) { const str = stringify(v) - this.context.memoryLimit.use(str.length) return str.replace(/\s+/g, ' ') } export function number_of_words (this: FilterImpl, input: string, mode?: 'cjk' | 'auto') { const str = stringify(input) - this.context.memoryLimit.use(str.length) input = str.trim() if (!input) return 0 switch (mode) { @@ -209,9 +185,6 @@ export function number_of_words (this: FilterImpl, input: string, mode?: 'cjk' | export function array_to_sentence_string (this: FilterImpl, array: unknown[], connector = 'and') { connector = stringify(connector) - let outputSize = connector.length + array.length * 2 - for (let i = 0; i < array.length; i++) outputSize += stringify(array[i]).length - this.context.memoryLimit.use(outputSize) switch (array.length) { case 0: return '' diff --git a/src/liquid-options.ts b/src/liquid-options.ts index f41ba1ec81..f0fc1bff68 100644 --- a/src/liquid-options.ts +++ b/src/liquid-options.ts @@ -87,10 +87,12 @@ export interface LiquidOptions { orderedFilterParameters?: boolean; /** For DoS handling, limit total length of templates parsed in one `parse()` call. A typical PC can handle 1e8 (100M) characters without issues. */ parseLimit?: number; - /** For DoS handling, limit total time (in ms) for each `render()` call. */ - renderLimit?: number; - /** For DoS handling, limit new objects creation, including array concat/join/strftime, etc. A typical PC can handle 1e9 (1G) memory without issue. */ - memoryLimit?: number; + /** For DoS handling, limit total renders of tag/HTML/output in one `render()` call. */ + templateLimit?: number; + /** For DoS handling, limit total output length in one `render()` call. */ + outputLengthLimit?: number; + /** For DoS handling, limit nesting depth of `{% render %}`, `{% include %}`, and `{% layout %}` tags. Defaults to `128`. */ + maxDepth?: number; } export interface RenderOptions { @@ -110,12 +112,10 @@ export interface RenderOptions { * Same as `ownPropertyOnly` on LiquidOptions, but only for current render() call */ ownPropertyOnly?: boolean; - /** For DoS handling, limit total renders of tag/HTML/output in one `render()` call. A typical PC can handle 1e5 renders of typical templates per second. */ + /** For DoS handling, limit total renders of tag/HTML/output in one `render()` call. */ templateLimit?: number; - /** For DoS handling, limit total time (in ms) for each `render()` call. */ - renderLimit?: number; - /** For DoS handling, limit new objects creation, including array concat/join/strftime, etc. A typical PC can handle 1e9 (1G) memory without issue.. */ - memoryLimit?: number; + /** For DoS handling, limit total output length in one `render()` call. */ + outputLengthLimit?: number; } export interface RenderFileOptions extends RenderOptions { @@ -160,8 +160,9 @@ export interface NormalizedFullOptions extends NormalizedOptions { globals: object; operators: Operators; parseLimit: number; - renderLimit: number; - memoryLimit: number; + templateLimit: number; + outputLengthLimit: number; + maxDepth: number; } export const defaultOptions: NormalizedFullOptions = { @@ -194,9 +195,10 @@ export const defaultOptions: NormalizedFullOptions = { lenientIf: false, globals: {}, operators: defaultOperators, - memoryLimit: Infinity, parseLimit: Infinity, - renderLimit: Infinity + templateLimit: Infinity, + outputLengthLimit: Infinity, + maxDepth: 128 } export function normalize (options: LiquidOptions): NormalizedFullOptions { diff --git a/src/render/expression.ts b/src/render/expression.ts index 62041ae174..a05aca395e 100644 --- a/src/render/expression.ts +++ b/src/render/expression.ts @@ -67,7 +67,6 @@ export function evalQuotedToken (token: QuotedToken) { function * evalRangeToken (token: RangeToken, ctx: Context) { const low: number = yield evalToken(token.lhs, ctx) const high: number = yield evalToken(token.rhs, ctx) - ctx.memoryLimit.use(high - low + 1) return range(+low, +high + 1) } diff --git a/src/render/render.ts b/src/render/render.ts index 15d6ab8655..57aaa0699a 100644 --- a/src/render/render.ts +++ b/src/render/render.ts @@ -1,4 +1,3 @@ -import { getPerformance } from '../util/performance' import { toPromise, RenderError, LiquidErrors, LiquidError } from '../util' import { Context } from '../context' import { Template } from '../template' @@ -6,23 +5,17 @@ import { Emitter, StreamedEmitter, SimpleEmitter } from '../emitters' export class Render { public renderTemplatesToNodeStream (templates: Template[], ctx: Context): NodeJS.ReadableStream { - const emitter = new StreamedEmitter() + const emitter = new StreamedEmitter(ctx.outputLengthLimit) Promise.resolve().then(() => toPromise(this.renderTemplates(templates, ctx, emitter))) .then(() => emitter.end(), err => emitter.error(err)) return emitter.stream } - public * renderTemplates (templates: Template[], ctx: Context, emitter?: Emitter): IterableIterator { - if (!emitter) { - emitter = new SimpleEmitter() - } - ctx.renderLimit.check(getPerformance().now()) + public * renderTemplates (templates: Template[], ctx: Context, emitter: Emitter = new SimpleEmitter(ctx.outputLengthLimit)): IterableIterator { const errors = [] for (const tpl of templates) { - ctx.renderLimit.check(getPerformance().now()) + ctx.templateLimit.use(1) try { - // if tpl.render supports emitter, it'll return empty `html` const html = yield tpl.render(ctx, emitter) - // if not, it'll return an `html`, write to the emitter for it html && emitter.write(html) if (ctx.breakCalled || ctx.continueCalled) break } catch (e) { diff --git a/src/tags/for.ts b/src/tags/for.ts index 1987501160..ec964ac6ac 100644 --- a/src/tags/for.ts +++ b/src/tags/for.ts @@ -43,13 +43,6 @@ export default class extends Tag { } * render (ctx: Context, emitter: Emitter): Generator { const r = this.liquid.renderer - let collection = toEnumerable(yield evalToken(this.collection, ctx)) - - if (!collection.length) { - yield r.renderTemplates(this.elseTemplates, ctx, emitter) - return - } - const continueKey = 'continue-' + this.variable + '-' + this.collection.getText() ctx.push(createScope({ continue: ctx.getRegister(continueKey, {}) })) const hash = (yield this.hash.render(ctx)) as Record @@ -59,6 +52,7 @@ export default class extends Tag { ? Object.keys(hash).filter(x => MODIFIERS.includes(x)) : MODIFIERS.filter(x => hash[x] !== undefined) + let collection = toEnumerable(yield evalToken(this.collection, ctx)) collection = modifiers.reduce((collection, modifier: valueOf) => { if (modifier === 'offset') return offset(collection, hash['offset']) if (modifier === 'limit') return limit(collection, hash['limit']) @@ -66,6 +60,14 @@ export default class extends Tag { }, collection) ctx.setRegister(continueKey, (hash['offset'] || 0) + collection.length) + + if (!collection.length) { + yield r.renderTemplates(this.elseTemplates, ctx, emitter) + return + } + + if (!this.templates.length) return + const scope = createScope({ forloop: new ForloopDrop(collection.length, this.collection.getText(), this.variable) }) ctx.push(scope) for (const item of collection) { diff --git a/src/tags/include.ts b/src/tags/include.ts index a00dda4e26..41507757d7 100644 --- a/src/tags/include.ts +++ b/src/tags/include.ts @@ -28,6 +28,7 @@ export default class extends Tag { this.hash = new Hash(tokenizer, liquid.options.jekyllInclude || liquid.options.keyValueSeparator) } * render (ctx: Context, emitter: Emitter): Generator { + ctx.depthLimit.use(1) const { liquid, hash, withVar } = this const { renderer } = liquid const filepath = (yield renderFilePath(this.file, ctx, liquid)) as string @@ -43,6 +44,7 @@ export default class extends Tag { yield renderer.renderTemplates(templates, ctx, emitter) ctx.pop() ctx.restoreRegister(saved) + ctx.depthLimit.release(1) } public * children (partials: boolean, sync: boolean): Generator { diff --git a/src/tags/layout.ts b/src/tags/layout.ts index 156ab0ee42..91e512ecb0 100644 --- a/src/tags/layout.ts +++ b/src/tags/layout.ts @@ -26,6 +26,7 @@ export default class extends Tag { yield renderer.renderTemplates(this.templates, ctx, emitter) return } + ctx.depthLimit.use(1) const filepath = (yield renderFilePath(this.file, ctx, liquid)) as string assert(filepath, () => `illegal file path "${filepath}"`) const templates = (yield liquid._parseLayoutFile(filepath, ctx.sync, this.currentFile)) as Template[] @@ -43,6 +44,7 @@ export default class extends Tag { ctx.push(createScope((yield args.render(ctx)) as Scope)) yield renderer.renderTemplates(templates, ctx, emitter) ctx.pop() + ctx.depthLimit.release(1) } public * children (partials: boolean): Generator { diff --git a/src/tags/render.ts b/src/tags/render.ts index bf86a28cb5..8ac279e943 100644 --- a/src/tags/render.ts +++ b/src/tags/render.ts @@ -55,6 +55,7 @@ export default class extends Tag { this.hash = new Hash(tokenizer, liquid.options.keyValueSeparator) } * render (ctx: Context, emitter: Emitter): Generator { + ctx.depthLimit.use(1) const { liquid, hash } = this const filepath = (yield renderFilePath(this.file, ctx, liquid)) as string assert(filepath, () => `illegal file path "${filepath}"`) @@ -81,6 +82,7 @@ export default class extends Tag { const templates = (yield liquid._parsePartialFile(filepath, childCtx.sync, this.currentFile)) as Template[] yield liquid.renderer.renderTemplates(templates, childCtx, emitter) } + ctx.depthLimit.release(1) } public * children (partials: boolean, sync: boolean): Generator { diff --git a/src/tags/tablerow.ts b/src/tags/tablerow.ts index 1676fe46df..563b44392b 100644 --- a/src/tags/tablerow.ts +++ b/src/tags/tablerow.ts @@ -45,6 +45,10 @@ export default class extends Tag { const limit = (args.limit === undefined) ? collection.length : args.limit collection = collection.slice(offset, offset + limit) + if (!collection.length) return + + if (!this.templates.length) return + const cols = args.cols || collection.length const r = this.liquid.renderer diff --git a/src/util/limiter.ts b/src/util/limiter.ts index f241f5e530..b5b5ae299c 100644 --- a/src/util/limiter.ts +++ b/src/util/limiter.ts @@ -14,6 +14,11 @@ export class Limiter { this.base += +count } } + release (count: number) { + if (+count > 0) { + this.base -= +count + } + } check (count: number) { if (+count > 0) { assert(+count <= this.limit, this.message) diff --git a/src/util/strftime.spec.ts b/src/util/strftime.spec.ts index b629b6d985..9de7731ba7 100644 --- a/src/util/strftime.spec.ts +++ b/src/util/strftime.spec.ts @@ -188,6 +188,14 @@ describe('util/strftime', function () { it('should have higher priority than H', () => { expect(t(then, '%0H')).toBe('03') }) + it('should allow pad width up to MAX_STRFTIME_PAD', () => { + expect(t(now, '%100000d').length).toBe(100000) + expect(t(now, `%${1_000_000}d`).length).toBe(1_000_000) + }) + it('should throw when pad width exceeds MAX_STRFTIME_PAD', () => { + expect(() => t(now, `%${1024 * 1024 + 1}d`)).toThrow('strftime pad width limit exceeded') + expect(() => t(now, '%5000000d')).toThrow('strftime pad width limit exceeded') + }) }) describe('modifier field', () => { it('should ignore E modifier', () => { diff --git a/src/util/strftime.ts b/src/util/strftime.ts index 8bf8b09d11..9a97857bb2 100644 --- a/src/util/strftime.ts +++ b/src/util/strftime.ts @@ -1,13 +1,15 @@ import { changeCase, padStart, padEnd } from './underscore' import { LiquidDate } from './liquid-date' -import type { Limiter } from './limiter' +import { assert } from './assert' + +/** Per-conversion numeric width cap for strftime (%N, %15d, …). */ +export const MAX_STRFTIME_PAD = 1024 * 1024 const rFormat = /%([-_0^#:]+)?(\d+)?([EO])?(.)/ interface FormatOptions { flags: Record; width?: string; modifier?: string; - memoryLimit?: Pick; } // prototype extensions @@ -98,8 +100,8 @@ const formatCodes: Record = { M: (d: LiquidDate) => d.getMinutes(), N: (d: LiquidDate, opts: FormatOptions) => { const width = Number(opts.width) || 9 + assertPadWidth(width) const str = String(d.getMilliseconds()).slice(0, width) - opts.memoryLimit?.use(width - str.length) return padEnd(str, width, '0') }, p: (d: LiquidDate) => (d.getHours() < 12 ? 'AM' : 'PM'), @@ -123,25 +125,29 @@ const formatCodes: Record = { } formatCodes.h = formatCodes.b -export function strftime (d: LiquidDate, formatStr: string, memoryLimit?: Pick) { +export function strftime (d: LiquidDate, formatStr: string) { let output = '' let remaining = formatStr let match while ((match = rFormat.exec(remaining))) { output += remaining.slice(0, match.index) remaining = remaining.slice(match.index + match[0].length) - output += format(d, match, memoryLimit) + output += format(d, match) } return output + remaining } -function format (d: LiquidDate, match: RegExpExecArray, memoryLimit?: Pick) { +function assertPadWidth (width: number) { + assert(width <= MAX_STRFTIME_PAD, 'strftime pad width limit exceeded') +} + +function format (d: LiquidDate, match: RegExpExecArray) { const [input, flagStr = '', width, modifier, conversion] = match const convert = formatCodes[conversion] if (!convert) return input const flags: Record = {} for (const flag of flagStr) flags[flag] = true - let ret = String(convert(d, { flags, width, modifier, memoryLimit })) + let ret = String(convert(d, { flags, width, modifier })) let padChar = padSpaceChars.has(conversion) ? ' ' : '0' let padWidth = Number(width) || padWidths[conversion] || 0 if (flags['^']) ret = ret.toUpperCase() @@ -149,6 +155,6 @@ function format (d: LiquidDate, match: RegExpExecArray, memoryLimit?: Pick { - const engine = new Liquid({ - memoryLimit: 1e5 - }) - const tpl = `{% for i in (1..1000000000) %} {{'a'}} {% endfor %}` - expect(() => engine.parseAndRenderSync(tpl)).toThrow('memory alloc limit exceeded, line:1, col:1') - }) it('group_by_exp fails with object as input #785', () => { const site = { tags: { diff --git a/test/integration/filters/date.spec.ts b/test/integration/filters/date.spec.ts index 9928769061..f36090561e 100644 --- a/test/integration/filters/date.spec.ts +++ b/test/integration/filters/date.spec.ts @@ -204,31 +204,21 @@ describe('filters/date', function () { return test('{{ "1990-12-31T23:00:00Z" | date: "%Y-%m-%dT%H:%M:%S" }}', '1991-01-01T04:30:00', undefined, optsWithDateFormat) }) }) - describe('strftime width / memoryLimit', () => { - it('should charge memoryLimit for huge numeric strftime widths', () => { - const liquid = new Liquid({ memoryLimit: 500 }) - expect(() => liquid.parseAndRenderSync('{{ d | date: f }}', { d: 'now', f: '%5000000d' })) - .toThrow('memory alloc limit exceeded') - }) - it('should charge memoryLimit for array format PoC', () => { - const liquid = new Liquid({ memoryLimit: 50, renderLimit: 1e9 }) - expect(() => liquid.parseAndRenderSync('{{ d | date: f }}', { d: 'now', f: ['a'.repeat(2000000)] })) - .toThrow('memory alloc limit exceeded') - }) - it('should charge memoryLimit for object toString format PoC', () => { - const liquid = new Liquid({ memoryLimit: 50, renderLimit: 1e9 }) - const huge = 'a'.repeat(2000000) - const f = { toString: () => huge } - expect(() => liquid.parseAndRenderSync('{{ d | date: f }}', { d: 'now', f })) - .toThrow('memory alloc limit exceeded') - }) - it('should honor numeric strftime pad width when memoryLimit allows', () => { - const liquid = new Liquid({ memoryLimit: 1e7 }) + describe('strftime width', () => { + it('should honor numeric strftime pad width', () => { + const liquid = new Liquid() const out = liquid.parseAndRenderSync('{{ d | date: f }}', { d: 'now', f: '%5000d' }) expect(out.length).toBe(5000) - const tight = new Liquid({ memoryLimit: 100 }) - expect(() => tight.parseAndRenderSync('{{ d | date: f }}', { d: 'now', f: '%5000d' })) - .toThrow('memory alloc limit exceeded') + }) + it('should honor large numeric strftime pad width up to the cap', () => { + const liquid = new Liquid() + const out = liquid.parseAndRenderSync('{{ d | date: f }}', { d: 'now', f: '%100000d' }) + expect(out.length).toBe(100000) + }) + it('should throw when numeric strftime pad width is too large', () => { + const liquid = new Liquid() + expect(() => liquid.parseAndRenderSync('{{ d | date: f }}', { d: 'now', f: '%5000000d' })) + .toThrow('strftime pad width limit exceeded') }) }) }) diff --git a/test/integration/liquid/dos.spec.ts b/test/integration/liquid/dos.spec.ts index 65d78adb52..7d630b2c16 100644 --- a/test/integration/liquid/dos.spec.ts +++ b/test/integration/liquid/dos.spec.ts @@ -4,15 +4,18 @@ import { mock, restore } from '../../stub/mockfs' describe('DoS related', function () { describe('#parseLimit', function () { afterEach(restore) + it('should throw when parse limit exceeded', async () => { const noLimit = new Liquid() const limit10 = new Liquid({ parseLimit: 10 }) const limit90 = new Liquid({ parseLimit: 90 }) const template = '{% capture bar %}{{ foo | bar: 3, a[3] }}{% endcapture %}' + await expect(noLimit.parseAndRender(template)).resolves.toBe('') await expect(limit10.parseAndRender(template)).rejects.toThrow('parse length limit exceeded') await expect(limit90.parseAndRender(template)).resolves.toBe('') }) + it('should take included template into account', async () => { mock({ '/small': 'Lorem ipsum', @@ -23,119 +26,151 @@ describe('DoS related', function () { await expect(liquid.parseAndRender('{% include "large" %}')).rejects.toThrow('parse length limit exceeded') }) }) - describe('#renderLimit', () => { + + describe('#templateLimit', () => { it('should throw when rendering too many templates', async () => { const src = '{% for i in (1..1000) %}{{i}},{% endfor %}' const noLimit = new Liquid() - const limitSmall = new Liquid({ renderLimit: 0.01 }) - const limitLarge = new Liquid({ renderLimit: 2e4 }) + const limitSmall = new Liquid({ templateLimit: 100 }) + const limitLarge = new Liquid({ templateLimit: 2001 }) await expect(noLimit.parseAndRender(src)).resolves.toMatch(/^1,2,3,4,5,.*,999,1000,$/) - await expect(limitSmall.parseAndRender(src)).rejects.toThrow('template render limit exceeded') + await expect(limitSmall.parseAndRender(src)).rejects.toThrow('template limit exceeded') await expect(limitLarge.parseAndRender(src)).resolves.toMatch(/^1,2,3,4,5,.*,999,1000,$/) }) + it('should support reset when calling render', async () => { const src = '{% for i in (1..1000) %}{{i}},{% endfor %}' - const liquid = new Liquid({ renderLimit: 0.01 }) - await expect(liquid.parseAndRender(src)).rejects.toThrow('template render limit exceeded') - await expect(liquid.parseAndRender(src, {}, { renderLimit: 1e6 })).resolves.toMatch(/^1,2,3,4,5,.*,999,1000,$/) + const liquid = new Liquid({ templateLimit: 100 }) + await expect(liquid.parseAndRender(src)).rejects.toThrow('template limit exceeded') + await expect(liquid.parseAndRender(src, {}, { templateLimit: 2001 })).resolves.toMatch(/^1,2,3,4,5,.*,999,1000,$/) }) + it('should take partials into account', async () => { mock({ '/small': '{% for i in (1..5) %}{{i}}{% endfor %}', '/large': '{% for i in (1..50000000) %}{{i}}{% endfor %}' }) - const liquid = new Liquid({ root: '/', renderLimit: 1000 }) - await expect(liquid.parseAndRender('{% render "large" %}')).rejects.toThrow('template render limit exceeded') + const liquid = new Liquid({ root: '/', templateLimit: 1000 }) + await expect(liquid.parseAndRender('{% render "large" %}')).rejects.toThrow('template limit exceeded') await expect(liquid.parseAndRender('{% render "small" %}')).resolves.toBe('12345') }) - it('should enforce renderLimit when for body has no template nodes', () => { - const liquid = new Liquid({ memoryLimit: 1e9, renderLimit: 1 }) - expect(() => liquid.parseAndRenderSync('{%- for i in (1..5000000) -%}{%- endfor -%}', {})) - .toThrow('template render limit exceeded') + }) + + describe('#outputLengthLimit', () => { + it('should throw when output length exceeded', async () => { + const src = '{% for i in (1..1000) %}{{i}},{% endfor %}' + const noLimit = new Liquid() + const limitSmall = new Liquid({ outputLengthLimit: 10 }) + const limitLarge = new Liquid({ outputLengthLimit: 5000 }) + await expect(noLimit.parseAndRender(src)).resolves.toMatch(/^1,2,3,4,5,.*,999,1000,$/) + await expect(limitSmall.parseAndRender(src)).rejects.toThrow('output length limit exceeded') + await expect(limitLarge.parseAndRender(src)).resolves.toMatch(/^1,2,3,4,5,.*,999,1000,$/) + }) + + it('should support reset when calling render', async () => { + const src = '{% for i in (1..1000) %}{{i}},{% endfor %}' + const liquid = new Liquid({ outputLengthLimit: 10 }) + await expect(liquid.parseAndRender(src)).rejects.toThrow('output length limit exceeded') + await expect(liquid.parseAndRender(src, {}, { outputLengthLimit: 5000 })).resolves.toMatch(/^1,2,3,4,5,.*,999,1000,$/) + }) + + it('should take partials into account', async () => { + mock({ + '/small': 'abc', + '/large': '{% for i in (1..1000) %}{{i}}{% endfor %}' + }) + const liquid = new Liquid({ root: '/', outputLengthLimit: 10 }) + await expect(liquid.parseAndRender('{% render "small" %}')).resolves.toBe('abc') + await expect(liquid.parseAndRender('{% render "large" %}')).rejects.toThrow('output length limit exceeded') + }) + + it('should enforce outputLengthLimit in sync render', () => { + const liquid = new Liquid({ outputLengthLimit: 5 }) + expect(() => liquid.parseAndRenderSync('{% for i in (1..100) %}{{i}}{% endfor %}')) + .toThrow('output length limit exceeded') }) - it('should enforce renderLimit when tablerow body has no template nodes', () => { - const liquid = new Liquid({ memoryLimit: 1e9, renderLimit: 1 }) - expect(() => liquid.parseAndRenderSync('{%- tablerow i in (1..1000000) cols:1 -%}{%- endtablerow -%}', {})) - .toThrow('template render limit exceeded') + + it('should enforce outputLengthLimit in stream render', async () => { + const liquid = new Liquid({ outputLengthLimit: 5 }) + const tpl = liquid.parse('{% for i in (1..100) %}{{i}}{% endfor %}') + const stream = liquid.renderToNodeStream(tpl) + await expect(new Promise((resolve, reject) => { + stream.on('error', reject) + stream.on('end', resolve) + })).rejects.toThrow('output length limit exceeded') }) }) - describe('#memoryLimit', () => { - it('should throw for too many array creation in filters', async () => { - const array = Array(1e3).fill(0) - const liquid = new Liquid({ memoryLimit: 100 }) - await expect(liquid.parseAndRender('{{ array | slice: 0, 3 | join }}', { array })).resolves.toBe('0 0 0') - await expect(liquid.parseAndRender('{{ array | slice: 0, 300 | join }}', { array })).rejects.toThrow('memory alloc limit exceeded, line:1, col:1') + + describe('#maxDepth', () => { + function chain (depth: number, tag: string) { + const templates: Record = {} + for (let i = 0; i < depth; i++) { + templates[`t${i}`] = i === depth - 1 ? 'done' : `{% ${tag} "t${i + 1}" %}` + } + return templates + } + + it('should throw when include depth exceeded', async () => { + const liquid = new Liquid({ templates: chain(3, 'include'), maxDepth: 2 }) + await expect(liquid.parseAndRender('{% include "t0" %}')).rejects.toThrow('template depth limit exceeded') }) - it('should support reset when calling render', async () => { - const array = Array(1e3).fill(0) - const liquid = new Liquid({ memoryLimit: 100 }) - await expect(liquid.parseAndRender('{{ array | slice: 0, 300 | join }}', { array })).rejects.toThrow('memory alloc limit exceeded, line:1, col:1') - await expect(liquid.parseAndRender('{{ array | slice: 0, 300 | join }}', { array }, { memoryLimit: 1e3 })).resolves.toBe(Array(300).fill(0).join(' ')) - }) - it('should throw for too many array iteration in tags', async () => { - const array = ['a'] - const liquid = new Liquid({ memoryLimit: 100 }) - const src = '{% for i in (1..count) %}{% assign array = array | concat: array %}{% endfor %}{{ array | join }}' - await expect(liquid.parseAndRender(src, { array, count: 3 })).resolves.toBe('a a a a a a a a') - await expect(liquid.parseAndRender(src, { array, count: 100 })).rejects.toThrow('memory alloc limit exceeded, line:1, col:26') - }) - it('should charge pop allocation to memoryLimit', async () => { - const array = Array(1e3).fill(0) - const liquid = new Liquid({ memoryLimit: 100 }) - await expect(liquid.parseAndRender('{{ array | pop | size }}', { array })).rejects.toThrow('memory alloc limit exceeded') - }) - it('should charge sample allocation to memoryLimit', async () => { - const array = Array(1e3).fill(0) - const liquid = new Liquid({ memoryLimit: 100 }) - await expect(liquid.parseAndRender('{{ array | sample: 1 | size }}', { array })).rejects.toThrow('memory alloc limit exceeded') - }) - it('should charge join by produced output size, not element count', () => { - const array = ['a'.repeat(100), 'b'.repeat(100)] - const liquid = new Liquid({ memoryLimit: 100 }) - expect(() => liquid.parseAndRenderSync('{{ array | join: "" }}', { array })) - .toThrow('memory alloc limit exceeded') - }) - it('should allow join within memoryLimit', () => { - const array = ['a'.repeat(20), 'b'.repeat(20)] - const liquid = new Liquid({ memoryLimit: 100 }) - expect(liquid.parseAndRenderSync('{{ array | join: "" }}', { array })).toBe('a'.repeat(20) + 'b'.repeat(20)) - }) - it('should prevent concat doubling from bypassing join memoryLimit', () => { - const liquid = new Liquid({ memoryLimit: 1e4 }) - const src = '{%- assign a = s | split: "NOSEP" -%}' + - '{%- assign a = a | concat: a -%}{%- assign a = a | concat: a -%}{%- assign a = a | concat: a -%}' + - '{{ a | join: "" | size }}' - expect(() => liquid.parseAndRenderSync(src, { s: 'a'.repeat(5000) })) - .toThrow('memory alloc limit exceeded') - }) - it('should charge array_to_sentence_string by produced output size', () => { - const array = ['a'.repeat(100), 'b'.repeat(100), 'c'.repeat(100)] - const liquid = new Liquid({ memoryLimit: 100 }) - expect(() => liquid.parseAndRenderSync('{{ array | array_to_sentence_string }}', { array })) - .toThrow('memory alloc limit exceeded') - }) - it('should charge json serialization of concat-doubled arrays', () => { - const liquid = new Liquid({ memoryLimit: 1e4 }) - const src = '{%- assign a = s | split: "NOSEP" -%}' + - '{%- assign a = a | concat: a -%}{%- assign a = a | concat: a -%}{%- assign a = a | concat: a -%}' + - '{{ a | json | size }}' - expect(() => liquid.parseAndRenderSync(src, { s: 'a'.repeat(5000) })) - .toThrow('memory alloc limit exceeded') - }) - it('should charge inspect serialization of concat-doubled arrays', () => { - const liquid = new Liquid({ memoryLimit: 1e4 }) - const src = '{%- assign a = s | split: "NOSEP" -%}' + - '{%- assign a = a | concat: a -%}{%- assign a = a | concat: a -%}{%- assign a = a | concat: a -%}' + - '{{ a | inspect | size }}' - expect(() => liquid.parseAndRenderSync(src, { s: 'a'.repeat(5000) })) - .toThrow('memory alloc limit exceeded') - }) - it('should charge strip_html input length to memoryLimit', () => { - const liquid = new Liquid({ memoryLimit: 100 }) - expect(() => liquid.parseAndRenderSync('{{ s | strip_html }}', { s: 'a'.repeat(200) })) - .toThrow('memory alloc limit exceeded') + + it('should allow include within maxDepth', async () => { + const liquid = new Liquid({ templates: chain(2, 'include'), maxDepth: 2 }) + await expect(liquid.parseAndRender('{% include "t0" %}')).resolves.toBe('done') + }) + + it('should throw when render depth exceeded', async () => { + const liquid = new Liquid({ templates: chain(3, 'render'), maxDepth: 2 }) + await expect(liquid.parseAndRender('{% render "t0" %}')).rejects.toThrow('template depth limit exceeded') + }) + + it('should allow render within maxDepth', async () => { + const liquid = new Liquid({ templates: chain(2, 'render'), maxDepth: 2 }) + await expect(liquid.parseAndRender('{% render "t0" %}')).resolves.toBe('done') + }) + + it('should throw when layout depth exceeded', async () => { + const liquid = new Liquid({ + templates: { + a: '{% layout "b" %}body-a', + b: '{% layout "c" %}body-b', + c: 'body-c' + }, + maxDepth: 2 + }) + await expect(liquid.parseAndRender('{% layout "a" %}root')).rejects.toThrow('template depth limit exceeded') + }) + + it('should allow layout within maxDepth', async () => { + const liquid = new Liquid({ + templates: { + a: '{% layout "b" %}body-a', + b: 'body-b' + }, + maxDepth: 2 + }) + await expect(liquid.parseAndRender('{% layout "a" %}root')).resolves.toBe('body-b') + }) + + it('should not count layout none toward depth', async () => { + const liquid = new Liquid({ maxDepth: 0 }) + await expect(liquid.parseAndRender('{% layout none %}ok')).resolves.toBe('ok') + }) + + it('should default maxDepth to 128', async () => { + const liquid = new Liquid({ templates: chain(128, 'include') }) + await expect(liquid.parseAndRender('{% include "t0" %}')).resolves.toBe('done') + const overflow = new Liquid({ templates: chain(129, 'include') }) + await expect(overflow.parseAndRender('{% include "t0" %}')).rejects.toThrow('template depth limit exceeded') + }) + + it('should enforce maxDepth in sync render', () => { + const liquid = new Liquid({ templates: chain(3, 'include'), maxDepth: 2 }) + expect(() => liquid.parseAndRenderSync('{% include "t0" %}')).toThrow('template depth limit exceeded') }) }) + describe('strip_html ReDoS', () => { // Regression for O(n^2) backtracking on unclosed ` { const liquid = new Liquid() const payload = ' but no in linear time', () => { const liquid = new Liquid() const payload = '