Feature: Generation vs crawling

[leonstafford]

tracking in branch https://github.com/leonstafford/wp2static/tree/feature/generation-vs-crawling

originally discussed/tested in #760 (comment)

can simulate running this within PHP, aka plugin, by entering wp shell and running:

$_SERVER['REQUEST_URI'] = '/author/admin/';
include 'index.php'

will output the admin author page's HTML.

Pros:

bypass webserver (don't need one at all)
reduce network latencies
esier to write caching logic

Cons:

lose webserver's request based process pooling?
need to simulate some WP bits, ie these var errors below:

Constant AUTH_KEY already defined in /usr/html/wp-config.php on line 49            
Constant AUTH_SALT already defined in /usr/html/wp-config.php on line 53           
Constant DB_CHARSET already defined in /usr/html/wp-config.php on line 35          
Constant DB_COLLATE already defined in /usr/html/wp-config.php on line 38          
Constant DB_HOST already defined in /usr/html/wp-config.php on line 32             
Constant DB_NAME already defined in /usr/html/wp-config.php on line 23                  
Constant DB_PASSWORD already defined in /usr/html/wp-config.php on line 29         
Constant DB_USER already defined in /usr/html/wp-config.php on line 26                                                                                          PHP Warning:  Constant LOGGED_IN_KEY already defined in /usr/html/wp-config.php on line 51 
Constant LOGGED_IN_SALT already defined in /usr/html/wp-config.php on line 55   
Constant NONCE_KEY already defined in /usr/html/wp-config.php on line 52           
Constant NONCE_SALT already defined in /usr/html/wp-config.php on line 56          
Constant SECURE_AUTH_KEY already defined in /usr/html/wp-config.php on line 50  
Constant SECURE_AUTH_SALT already defined in /usr/html/wp-config.php on line 54  
Constant WP_CACHE_KEY_SALT already defined in /usr/html/wp-config.php on line 57
Undefined array key "SERVER_NAME" in /usr/html/wp-includes/general-template.php on line 3413
Undefined array key "host" in /usr/html/wp-includes/canonical.php on line 687   
Undefined array key "host" in /usr/html/wp-includes/canonical.php on line 687   
Undefined array key "host" in /usr/html/wp-includes/canonical.php on line 696   
Undefined array key "host" in /usr/html/wp-includes/canonical.php on line 696   
Undefined array key "host" in /usr/html/wp-includes/canonical.php on line 699   
Undefined array key "host" in /usr/html/wp-includes/canonical.php on line 699   
Undefined array key "scheme" in /usr/html/wp-includes/canonical.php on line 720 
Undefined array key "scheme" in /usr/html/wp-includes/canonical.php on line 720 

[leonstafford]

how to deal with binary files? just do a straight copy?

[leonstafford]

doing the include() seems messier than calling directly from php index.php, we're trying to reload a bunch of stuff we already have instantiated... should be able to call a WP method higher up the chain.... let's see...

exec'ing to call PHP seems like overhead, but trying to do within WP, we have too much false context where the plugin is running from...

[leonstafford]

note, to tail requests output, can put ~ line 73 in wp-includes/canonical.php: error_log(print_r($requested_url, true));

[leonstafford]

for our generation, we don't need a LOT of WP stuff, we need a read-only system to generate the frontend, no wp-admin, etc. could be used in Lokl just for generation, else likely impractical...

[leonstafford]

still, could be fun project to build a tiny, fast WP just for generation stage....

[leonstafford]

not happy with this little test. Really want to avoid exec'ing and do the generation programmatically. with the standalone WP URL Generator library I've started, we'll have the seed URL list to start with.

I'm less worried about having any on-file-change auto regeneration of specific site sections, rather, just have the fastest way to generate a whole WP site statically, without a webserver.

ideas:

simple outside-in approach

• get URL list independently (from WordPressURLDetector library)
• open mass number of threads, each calling index.php with vars set to simulate webserver request (still slow, loading whole WP each time)
• use php process manager or something similar to avoid boostrap overhead per php process
• use stripped down WP version (override functions majority of sites don't need)

inside-out approach

• we know what pages need authoring (authors, categories, custom posts, sitemaps, etc) and know their intended URLs.
• avoid plugin overhead by including in theme's functions.php, no composer, etc?
• options to cache all header and/or footer renders
• need to allow all WP hooks/filters to run, so do we need to run within WP?
• dequeue/override all non-essential filters, like the cringey capital_P_dangit
• apply other optimizations like disabling emojis, look for things in Little Bizzy, WP-Strip-Naked, etc.

[leonstafford]

more thoughts:

• set site_url / home programmatically, to avoid rewriting before later
• wp-load.php doesn't do much, we can start further up if loading from external script

define( 'ABSPATH', __DIR__ . '/' );
// toggle error reporting with a DEBUG mode for this script. We have developed WP site, mindful of errors, so this should just be catching issues with our generator

define('THISSCRIPT_DEBUG', false);

if(THISSCRIPT_DEBUG === true)
{
   error_reporting(E_ALL);
   display_errors(true);
   log_errors(true);
}
else
{
   error_reporting(0);
   display_errors(false);
   log_errors(false);
}

//wp-config.php:
define( 'DB_NAME', 'database_name_here' );
define( 'DB_USER', 'username_here' );
define( 'DB_PASSWORD', 'password_here' );
define( 'DB_HOST', 'localhost' );
define( 'DB_CHARSET', 'utf8' );
define( 'DB_COLLATE', '' );

define( 'AUTH_KEY',         '' );
define( 'SECURE_AUTH_KEY',  '' );
define( 'LOGGED_IN_KEY',    '' );
define( 'NONCE_KEY',        '' );
define( 'AUTH_SALT',        '' );
define( 'SECURE_AUTH_SALT', '' );
define( 'LOGGED_IN_SALT',   '' );
define( 'NONCE_SALT',       '' );

$table_prefix = 'wp_';

//wp-settings.php:
define( 'WPINC', 'wp-includes' );
// rm the tinymce, required php, mysql checks
// ie https://github.com/WordPress/WordPress/search?q=required_php_version not used anywhere of relevance to us
// wp_version is used in a few places, ie Etag header, but really, shouldn't need it for generating site frontend, I'd remove it
// global $wp_version, $wp_db_version, $tinymce_version, $required_php_version, $required_mysql_version, $wp_local_package;
// so we can ignore this: require ABSPATH . WPINC . '/version.php';


// require ABSPATH . WPINC . '/load.php'; <-- much of this can be skipped. We need to set the REQUEST_URI type stuff, make a decision on language support and load the DB. a lot of this file is unneeded utility code.

// review load.php further, then continue with wp-settings.php review from line 34

The normal "boot sequence" of WordPress (started in wp-blog-header.php) loads the plugins, parses the URL, executes a post query based on the URL, and calls the theme. This main post query is typically used in "The Loop". So if you want all the advantages of URL parsing but not display it using the site theme, you can set WP_USE_THEMES to false and it will not execute that final step