From 430626fd2e4435fb308e9f4e137f92ed55d7d365 Mon Sep 17 00:00:00 2001 From: Denis Defreyne Date: Sun, 3 Jan 2016 14:36:40 +0100 Subject: [PATCH 1/3] Add RFC: lazy-load items --- active/0000-lazy-load-items.adoc | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 active/0000-lazy-load-items.adoc diff --git a/active/0000-lazy-load-items.adoc b/active/0000-lazy-load-items.adoc new file mode 100644 index 0000000..b9b5338 --- /dev/null +++ b/active/0000-lazy-load-items.adoc @@ -0,0 +1,30 @@ += Lazy-load items +:start_date: 2016-01-03 +:rfc_issue: (leave this empty) +:nanoc_issue: (leave this empty) + +== Summary + +Do not load all items into memory, but compile them one-by-one. + +== Motivation + +* Loading all items in memory is problematic in terms of memory pressure. For large sites, Nanoc can run out of memory, or spend a large amount of time in the garbage collector footnote:[CRuby 2.2 has a generational garbage collector, which helps alleviate this problem. I have not compared performance of CRuby 2.1 and 2.2 in terms of GC performance yet, however.]. + +* Loading items one-by-one allows compilation to start right away, rather than waiting for all data to have been loaded into memory. With data sources that pre-fetch data, I/O wait time could be reduced to a minimum. This approach will also get us a step closer towards parallelism footnote:[CRuby’s global interpreter lock will prevent us from gaining much here, but it will benefit other Ruby implementations that do not have such a lock, such as JRuby.]. + +== Detailed design + +(wip) + +== Drawbacks + +(wip) + +== Alternatives + +(wip) + +== Unresolved questions + +(wip) From 5816096ff325bc2ca45e2f1737dcba1edf44fc5d Mon Sep 17 00:00:00 2001 From: Denis Defreyne Date: Sun, 3 Jan 2016 14:43:09 +0100 Subject: [PATCH 2/3] Describe lazy-loading content alternative --- active/0000-lazy-load-items.adoc | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/active/0000-lazy-load-items.adoc b/active/0000-lazy-load-items.adoc index b9b5338..8b7a57c 100644 --- a/active/0000-lazy-load-items.adoc +++ b/active/0000-lazy-load-items.adoc @@ -23,7 +23,9 @@ Do not load all items into memory, but compile them one-by-one. == Alternatives -(wip) +* Rather than lazy-loading items, load all items into memory but lazy-load their content and attributes instead. This likely will not give much of a performance gain (if any at all), since the content and attributes for each item needs to be loaded at some point anyway, in order for the checksum to be calculated. ++ +To get around this problem, at the end of compilation, the previous checksum of an item could be reused if neither their content nor attributes were loaded. == Unresolved questions From 919f8dd17748f11f413c6c060e4fbf956118f063 Mon Sep 17 00:00:00 2001 From: Denis Defreyne Date: Sun, 3 Jan 2016 14:49:37 +0100 Subject: [PATCH 3/3] Document dependency on #1 --- active/0000-lazy-load-items.adoc | 2 ++ 1 file changed, 2 insertions(+) diff --git a/active/0000-lazy-load-items.adoc b/active/0000-lazy-load-items.adoc index 8b7a57c..4880dcc 100644 --- a/active/0000-lazy-load-items.adoc +++ b/active/0000-lazy-load-items.adoc @@ -15,6 +15,8 @@ Do not load all items into memory, but compile them one-by-one. == Detailed design +This depends on letting data sources find objects. For details, see https://github.com/nanoc/rfcs/pull/1[RFC #1]. + (wip) == Drawbacks