From 855686c87723ca3c24355aabb0cdead092f16cdc Mon Sep 17 00:00:00 2001 From: "dominik.matic" Date: Wed, 22 Jan 2025 16:20:53 +0100 Subject: [PATCH 1/7] update magento2 s3 obj storage doc --- ...onfigure-remote-storage-for-magento-2-x.md | 21 +++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md b/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md index fc1bfa0e..b8ecd509 100644 --- a/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md +++ b/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md @@ -72,26 +72,43 @@ Magento's S3 implementation creates a test file called `storage.flag`, which is To start serving media assets from your S3 bucket, you need to make some adjustments to your nginx configuration. +We recommend that you create a configuration file defining the cache storage location, structure, size constraints, and cache expiration policies. +```bash +echo "proxy_cache_path /data/var/nginx-asset-cache levels=1:2 keys_zone=asset_cache:10m max_size=1g inactive=1w;" > /data/web/nginx/http.asset_proxy_cache.conf +``` + +Then update your nginx configuration in the following manner. + ```nginx location /media { # ... location ~* \.(ico|jpg|jpeg|png|gif|svg|js|css|swf|eot|ttf|otf|woff|woff2)$ { resolver 8.8.8.8; set $bucket ""; - proxy_pass https://s3.amazonaws.com/$bucket$uri; + proxy_pass https://$bucket.s3.amazonaws.com$uri; proxy_pass_request_body off; proxy_pass_request_headers off; proxy_intercept_errors on; proxy_hide_header "x-amz-id-2"; proxy_hide_header "x-amz-request-id"; proxy_hide_header "x-amz-storage-class"; + proxy_hide_header "x-amz-server-side-encryption"; proxy_hide_header "Set-Cookie"; proxy_ignore_headers "Set-Cookie"; + + # only include the following if you followed the previous step + proxy_cache_key "$bucket$uri"; + proxy_cache_valid 200 302 7d; + proxy_cache_methods GET HEAD; + proxy_cache_background_update on; + proxy_cache_use_stale updating; + proxy_cache asset_cache; + } # ... } ``` - +Keep in mind your bucket URL might be different depending on your AWS region. For example, you might need to change it to `https://s3.amazonaws.com/$bucket$uri` instead. Also make sure your S3 bucket policies are configured correctly, so that only `/media` is publicly readable. For example: ```json From 4065cb78addae32772f5e1dd0a46bfabfb6254e2 Mon Sep 17 00:00:00 2001 From: "dominik.matic" Date: Wed, 22 Jan 2025 16:28:11 +0100 Subject: [PATCH 2/7] apply mdformat --- ...ow-to-configure-remote-storage-for-magento-2-x.md | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md b/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md index b8ecd509..bc92a301 100644 --- a/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md +++ b/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md @@ -1,10 +1,6 @@ ---- -myst: - html_meta: - description: Configure remote storage for Magento 2.x. Learn how to configure - Magento 2 to start storing files in your bucket using a single command. - title: How to Configure Remote Storage for Magento 2.x | Hypernode ---- +______________________________________________________________________ + +## myst: html_meta: description: Configure remote storage for Magento 2.x. Learn how to configure Magento 2 to start storing files in your bucket using a single command. title: How to Configure Remote Storage for Magento 2.x | Hypernode # How to Configure Remote Storage for Magento 2.x @@ -73,6 +69,7 @@ Magento's S3 implementation creates a test file called `storage.flag`, which is To start serving media assets from your S3 bucket, you need to make some adjustments to your nginx configuration. We recommend that you create a configuration file defining the cache storage location, structure, size constraints, and cache expiration policies. + ```bash echo "proxy_cache_path /data/var/nginx-asset-cache levels=1:2 keys_zone=asset_cache:10m max_size=1g inactive=1w;" > /data/web/nginx/http.asset_proxy_cache.conf ``` @@ -108,6 +105,7 @@ location /media { # ... } ``` + Keep in mind your bucket URL might be different depending on your AWS region. For example, you might need to change it to `https://s3.amazonaws.com/$bucket$uri` instead. Also make sure your S3 bucket policies are configured correctly, so that only `/media` is publicly readable. For example: From 65a77399be826b1cbcb93683c73f5712b88a7a37 Mon Sep 17 00:00:00 2001 From: "dominik.matic" Date: Wed, 22 Jan 2025 16:38:22 +0100 Subject: [PATCH 3/7] something like this? --- .../how-to-configure-remote-storage-for-magento-2-x.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md b/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md index bc92a301..66cd1c37 100644 --- a/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md +++ b/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md @@ -68,10 +68,10 @@ Magento's S3 implementation creates a test file called `storage.flag`, which is To start serving media assets from your S3 bucket, you need to make some adjustments to your nginx configuration. -We recommend that you create a configuration file defining the cache storage location, structure, size constraints, and cache expiration policies. +We recommend that you create a configuration file `/data/web/nginx/http.asset_proxy_cache.conf` defining the cache storage location, structure, size constraints, and cache expiration policies. -```bash -echo "proxy_cache_path /data/var/nginx-asset-cache levels=1:2 keys_zone=asset_cache:10m max_size=1g inactive=1w;" > /data/web/nginx/http.asset_proxy_cache.conf +```nginx +proxy_cache_path /data/var/nginx-asset-cache levels=1:2 keys_zone=asset_cache:10m max_size=1g inactive=1w; ``` Then update your nginx configuration in the following manner. @@ -93,7 +93,7 @@ location /media { proxy_hide_header "Set-Cookie"; proxy_ignore_headers "Set-Cookie"; - # only include the following if you followed the previous step + # include the following if you defined proxy_cache_path previously proxy_cache_key "$bucket$uri"; proxy_cache_valid 200 302 7d; proxy_cache_methods GET HEAD; From e21ddf4cc482df4ea6391f7853046be1eef6e9e3 Mon Sep 17 00:00:00 2001 From: "dominik.matic" Date: Wed, 22 Jan 2025 16:42:55 +0100 Subject: [PATCH 4/7] remove line --- .../magento-2/how-to-configure-remote-storage-for-magento-2-x.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md b/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md index 66cd1c37..e0edcea2 100644 --- a/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md +++ b/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md @@ -100,7 +100,6 @@ location /media { proxy_cache_background_update on; proxy_cache_use_stale updating; proxy_cache asset_cache; - } # ... } From 6d7fe38a0adbdb2466e57347755058f026d3fc14 Mon Sep 17 00:00:00 2001 From: "dominik.matic" Date: Wed, 22 Jan 2025 16:48:00 +0100 Subject: [PATCH 5/7] undo bad mdformat change --- .../how-to-configure-remote-storage-for-magento-2-x.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md b/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md index e0edcea2..26e0b2b3 100644 --- a/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md +++ b/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md @@ -1,6 +1,10 @@ -______________________________________________________________________ - -## myst: html_meta: description: Configure remote storage for Magento 2.x. Learn how to configure Magento 2 to start storing files in your bucket using a single command. title: How to Configure Remote Storage for Magento 2.x | Hypernode +--- +myst: + html_meta: + description: Configure remote storage for Magento 2.x. Learn how to configure + Magento 2 to start storing files in your bucket using a single command. + title: How to Configure Remote Storage for Magento 2.x | Hypernode +--- # How to Configure Remote Storage for Magento 2.x From dd262fd6e7a8f907b7c0c00ec8926fe686138177 Mon Sep 17 00:00:00 2001 From: "dominik.matic" Date: Mon, 27 Jan 2025 11:32:11 +0100 Subject: [PATCH 6/7] remove python 3.7 from test matrix --- .github/workflows/test.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml index 372caa6c..c64c08d0 100644 --- a/.github/workflows/test.yaml +++ b/.github/workflows/test.yaml @@ -8,7 +8,7 @@ jobs: runs-on: ubuntu-latest strategy: matrix: - python-version: ['3.7', '3.8', '3.9', '3.10', '3.11'] + python-version: ['3.8', '3.9', '3.10', '3.11'] steps: - uses: actions/checkout@v3 From 2f218187a0bf82b6575f8435dab973eb99c0b0de Mon Sep 17 00:00:00 2001 From: "dominik.matic" Date: Mon, 27 Jan 2025 12:01:30 +0100 Subject: [PATCH 7/7] update docs --- ...onfigure-remote-storage-for-magento-2-x.md | 128 +++++++++++++----- 1 file changed, 95 insertions(+), 33 deletions(-) diff --git a/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md b/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md index 26e0b2b3..0209c1dd 100644 --- a/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md +++ b/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md @@ -70,47 +70,109 @@ Magento's S3 implementation creates a test file called `storage.flag`, which is ## Serving assets from your S3 bucket -To start serving media assets from your S3 bucket, you need to make some adjustments to your nginx configuration. - -We recommend that you create a configuration file `/data/web/nginx/http.asset_proxy_cache.conf` defining the cache storage location, structure, size constraints, and cache expiration policies. +To start serving media assets from your S3 bucket, you need to make some adjustments to your nginx configuration. Create the following file at `/data/web/nginx/example.com/server.assets.conf` for each relevant vhost: ```nginx -proxy_cache_path /data/var/nginx-asset-cache levels=1:2 keys_zone=asset_cache:10m max_size=1g inactive=1w; -``` +set $backend "haproxy"; + +location @object_storage_fallback { + # Proxy to object storage + set $bucket "my_bucket_name"; + proxy_cache_key "$bucket$uri"; + proxy_cache_valid 200 302 7d; + proxy_cache_methods GET HEAD; + proxy_cache_background_update on; + proxy_cache_use_stale updating; + proxy_cache asset_cache; + resolver 8.8.8.8; + proxy_pass https://$bucket.s3.amazonaws.com$uri; + proxy_pass_request_body off; + proxy_pass_request_headers off; + proxy_intercept_errors on; + proxy_hide_header "x-amz-id-2"; + proxy_hide_header "x-amz-request-id"; + proxy_hide_header "x-amz-storage-class"; + proxy_hide_header "x-amz-server-side-encryption"; + proxy_hide_header "Set-Cookie"; + proxy_ignore_headers "Set-Cookie"; + add_header Cache-Control "public"; + add_header X-Frame-Options "SAMEORIGIN"; + add_header X-Cache-Status $upstream_cache_status; + expires +1y; + + # If object storage fails, fallback to PHP handler + error_page 404 = @asset_fallback; + error_page 403 = @asset_fallback; +} -Then update your nginx configuration in the following manner. +location @php_asset_fallback { + # Handle with phpfpm + rewrite ^/media /get.php?$args last; + rewrite ^/static/(version\d*/)?(.*)$ /static.php?resource=$2 last; + echo_exec @phpfpm; +} -```nginx -location /media { - # ... - location ~* \.(ico|jpg|jpeg|png|gif|svg|js|css|swf|eot|ttf|otf|woff|woff2)$ { - resolver 8.8.8.8; - set $bucket ""; - proxy_pass https://$bucket.s3.amazonaws.com$uri; - proxy_pass_request_body off; - proxy_pass_request_headers off; - proxy_intercept_errors on; - proxy_hide_header "x-amz-id-2"; - proxy_hide_header "x-amz-request-id"; - proxy_hide_header "x-amz-storage-class"; - proxy_hide_header "x-amz-server-side-encryption"; - proxy_hide_header "Set-Cookie"; - proxy_ignore_headers "Set-Cookie"; - - # include the following if you defined proxy_cache_path previously - proxy_cache_key "$bucket$uri"; - proxy_cache_valid 200 302 7d; - proxy_cache_methods GET HEAD; - proxy_cache_background_update on; - proxy_cache_use_stale updating; - proxy_cache asset_cache; +location @haproxy { + # Handle with haproxy + include /etc/nginx/proxy_to_haproxy.conf; + proxy_pass http://127.0.0.1:8080; +} + +location @asset_fallback { + try_files "" $asset_fallback_handler; +} + +location ~ ^/static/ { + expires max; + + # Remove signature of the static files that is used to overcome the browser cache + location ~ ^/static/version\d*/ { + rewrite ^/static/version\d*/(.*)$ /static/$1 last; + } + + location ~* \.(ico|jpg|jpeg|png|gif|svg|svgz|webp|avif|avifs|js|css|eot|ttf|otf|woff|woff2|html|json|webmanifest)$ { + add_header Cache-Control "public"; + add_header X-Frame-Options "SAMEORIGIN"; + expires +1y; + + try_files $uri $uri/ @asset_fallback; + } + location ~* \.(zip|gz|gzip|bz2|csv|xml)$ { + add_header Cache-Control "no-store"; + add_header X-Frame-Options "SAMEORIGIN"; + expires off; + + try_files $uri $uri/ @asset_fallback; + } + try_files $uri $uri/ @asset_fallback; + add_header X-Frame-Options "SAMEORIGIN"; +} + +location /media/ { + try_files $uri $uri/ @asset_fallback; + + location ~ ^/media/theme_customization/.*\.xml { + deny all; + } + + location ~* \.(ico|jpg|jpeg|png|gif|svg|svgz|webp|avif|avifs|js|css|swf|eot|ttf|otf|woff|woff2)$ { + add_header Cache-Control "public"; + add_header X-Frame-Options "SAMEORIGIN"; + expires +1y; + try_files $uri $uri/ @object_storage_fallback; + } + location ~* \.(zip|gz|gzip|bz2|csv|xml)$ { + add_header Cache-Control "no-store"; + add_header X-Frame-Options "SAMEORIGIN"; + expires off; + try_files $uri $uri/ @object_storage_fallback; } - # ... + add_header X-Frame-Options "SAMEORIGIN"; } ``` -Keep in mind your bucket URL might be different depending on your AWS region. For example, you might need to change it to `https://s3.amazonaws.com/$bucket$uri` instead. -Also make sure your S3 bucket policies are configured correctly, so that only `/media` is publicly readable. For example: +Make sure to change the string `my_bucket_name` to the name of your bucket and keep in mind that your bucket URL might be different depending on your AWS region. For example, you might need to change it from `https://$bucket.s3.amazonaws.com$uri` to `https://s3.amazonaws.com/$bucket$uri` instead. +Furthermore, ensure that your S3 bucket policies are configured correctly, so that only `/media` is publicly readable. For example: ```json {