diff --git a/docs/deployment/configure.md b/docs/deployment/configure.md index df7c5c79e4..759ec09a9f 100644 --- a/docs/deployment/configure.md +++ b/docs/deployment/configure.md @@ -34,7 +34,6 @@ export LD_LIBRARY_PATH=/usr/local/cloudberry-db/lib:LD_LIBRARY_PATH ./configure --prefix=/usr/local/cloudberry-db \ --disable-external-fts \ --enable-debug \ - --enable-cassert \ --enable-debug-extensions \ --enable-gpcloud \ --enable-ic-proxy \ @@ -42,7 +41,7 @@ export LD_LIBRARY_PATH=/usr/local/cloudberry-db/lib:LD_LIBRARY_PATH --enable-orafce \ --enable-orca \ --enable-pax \ - --enable-pxf \ + --disable-pxf \ --enable-tap-tests \ --with-gssapi \ --with-ldap \ @@ -81,7 +80,6 @@ cd ~/cloudberry ./configure --prefix=/usr/local/cloudberry-db \ --disable-external-fts \ --enable-debug \ - --enable-cassert \ --enable-debug-extensions \ --enable-gpcloud \ --enable-ic-proxy \ @@ -89,7 +87,7 @@ cd ~/cloudberry --enable-orafce \ --enable-orca \ --enable-pax \ - --enable-pxf \ + --disable-pxf \ --enable-tap-tests \ --with-gssapi \ --with-ldap \ @@ -123,7 +121,7 @@ Also, some packages names vary between different Linux distributions. |--|--|--| | `--prefix=PREFIX` |Installation directory. `/usr/local/cbdb` is the default value. `make install` will install all the files in `/usr/local/cbdb/bin`, `/usr/local/cbdb/lib` etc. | You can specify an installation prefix other than `/usr/local/cbdb` using `--prefix`. In this guide, we use `/usr/local/cloudberry-db` as the installation directory. | | `--disable-gpfdist` | Do not use gpfdist | Enable gpfdist by default. This requires apr lib and libevent to be installed.| -| `--disable-pxf` | Do not build PXF. | Enable PXF by default. |PXF is a query federation engine that accesses data residing in external systems such as Hadoop, Hive, HBase, relational databases, S3, Google Cloud Storage, among other external systems.| +| `--disable-pxf` | Do not build PXF. | Enable PXF by default. PXF is a query federation engine that accesses data residing in external systems such as Hadoop, Hive, HBase, relational databases, S3, Google Cloud Storage, among other external systems. Now the [cloudberry-pxf](https://github.com/apache/cloudberry-pxf/tree/main/fdw) will be kept as the latest version of `pxf_fdw`.| | `--enable-orafce` | Build with Oracle compatibility functions. | | | `--enable-debug` | Build all programs and libraries with debugging symbols.| This means that you can run the programs in a debugger to analyze problems. This enlarges the size of the installed executables considerably, and on non-GCC compilers it usually also disables compiler optimization, causing slowdowns. However, having the symbols available is extremely helpful for dealing with any problems that might arise. Currently, this option is recommended for production installations only if you use GCC. But you should always have it on if you are doing development work or running a beta version.| | `--enable-profiling` | Build with profiling enabled.|This option is for use only with GCC and when doing development work.| diff --git a/docs/deployment/install-required-packages.md b/docs/deployment/install-required-packages.md index b6c38ec915..7f2f2c9f8d 100644 --- a/docs/deployment/install-required-packages.md +++ b/docs/deployment/install-required-packages.md @@ -170,5 +170,29 @@ sudo apt install -y bison \ python3-setuptools \ rsync ``` + +### Extra dependencies for building PAX + +In the latest main branch, we have introduced a new dependency `liburing` for building PAX. + +* For Ubuntu 22.04 + +```bash +sudo apt install -y liburing-dev +``` + +* For Ubuntu 20.04 + +```bash +sudo apt install -y git build-essential +wget https://github.com/axboe/liburing/archive/refs/tags/liburing-2.1.tar.gz +tar -xzf liburing-2.1.tar.gz +rm "liburing-2.1.tar.gz" +cd liburing-liburing-2.1 +make -j$(nproc) +sudo make install +sudo ldconfig +``` + \ No newline at end of file diff --git a/docs/deployment/quick-build.md b/docs/deployment/quick-build.md index 1f728a7469..fc019aadb7 100644 --- a/docs/deployment/quick-build.md +++ b/docs/deployment/quick-build.md @@ -10,7 +10,7 @@ In the following sections, we'll provide detailed, step-by-step instructions for By this, you will get an Apache Cloudberry environment with a demo cluster ready for testing and development. - + ```bash # Install sudo & git @@ -141,7 +141,6 @@ export LD_LIBRARY_PATH=/usr/local/cloudberry-db/lib:LD_LIBRARY_PATH ./configure --prefix=/usr/local/cloudberry-db \ --disable-external-fts \ --enable-debug \ - --enable-cassert \ --enable-debug-extensions \ --enable-gpcloud \ --enable-ic-proxy \ @@ -149,7 +148,7 @@ export LD_LIBRARY_PATH=/usr/local/cloudberry-db/lib:LD_LIBRARY_PATH --enable-orafce \ --enable-orca \ --enable-pax \ - --enable-pxf \ + --disable-pxf \ --enable-tap-tests \ --with-gssapi \ --with-ldap \ @@ -184,7 +183,7 @@ psql -P pager=off template1 -c 'SELECT * from gp_segment_configuration' psql template1 -c 'SELECT version()' ``` - + ```bash @@ -257,7 +256,6 @@ sudo apt install -y bison \ libprotobuf-dev \ libreadline-dev \ libssl-dev \ - liburing-dev \ libuv1-dev \ liblz4-dev \ libxerces-c-dev \ @@ -273,6 +271,19 @@ sudo apt install -y bison \ python3-setuptools \ rsync +# For PAX build, you need to install liburing +## For Ubuntu 22.04 + sudo apt install -y liburing-dev +## For Ubuntu 20.04 + sudo apt install -y git build-essential + wget https://github.com/axboe/liburing/archive/refs/tags/liburing-2.1.tar.gz + tar -xzf liburing-2.1.tar.gz + rm "liburing-2.1.tar.gz" + cd liburing-liburing-2.1 + make -j$(nproc) + sudo make install + sudo ldconfig + # Use the gpadmin user from now on sudo su - gpadmin @@ -292,7 +303,6 @@ cd ~/cloudberry ./configure --prefix=/usr/local/cloudberry-db \ --disable-external-fts \ --enable-debug \ - --enable-cassert \ --enable-debug-extensions \ --enable-gpcloud \ --enable-ic-proxy \ @@ -300,7 +310,7 @@ cd ~/cloudberry --enable-orafce \ --enable-orca \ --enable-pax \ - --enable-pxf \ + --disable-pxf \ --enable-tap-tests \ --with-gssapi \ --with-ldap \ diff --git a/docs/deployment/sandbox.md b/docs/deployment/sandbox.md index dfe3abd3f4..0132c9cde3 100644 --- a/docs/deployment/sandbox.md +++ b/docs/deployment/sandbox.md @@ -22,8 +22,9 @@ When building and deploying Cloudberry in Docker, you will have 2 different depl **Deployment Options** -1. **Single Container** (Default) - With the single container option, you will have the coordinator as well as the Cloudberry segments all running on a single container. This is the default behavior when deploying using the `run.sh` script provided. -2. **Multi-Container** - Deploying with the multi-container option will give you a more realistic deployment of what actual production Cloudberry clusters look like. With multi-node, you will have the coordinator, the standby coordinator, and 2 segment hosts all on their own respective containers. This is to both highlight the distributed nature of Apache Cloudberry as well as highlight how high availability (HA) features work in the event of a server (or in this case a container) failing. This is enabled by passing the `-m` flag to the `run.sh` script which will be highlighted below. +1. **Recommended for most users** – Build directly from your current local source code using `-c local`. This is the fastest way to get started as it reuses your existing checkout, avoiding the need to download the code again inside the container. It is also ideal for developers testing local changes. +2. Compile with the source code of the latest Apache Cloudberry (released in [Apache Cloudberry Release Page](https://github.com/apache/cloudberry/releases)). The base OS will be Rocky Linux 9 Docker image. +3. Compile with the latest Apache Cloudberry [main](https://github.com/apache/cloudberry/tree/main) branch. The base OS will be Rocky Linux 9 Docker image. ![Apache Cloudberry Sandbox Deployments](/img/bootcamp/sandbox-deployment.jpg) @@ -36,43 +37,63 @@ Build and deploy steps: 1. Start Docker Desktop and make sure it is running properly on your host platform. -2. Download the repository [apache/cloudberry-bootcamp](https://github.com/apache/cloudberry-bootcamp) to the target machine. +2. Clone the Apache Cloudberry repository to the target machine. ```shell - git clone https://github.com/apache/cloudberry-bootcamp.git + git clone https://github.com/apache/cloudberry.git ``` -3. Enter the repository and run the `run.sh` script to start the Docker container. This will start the automatic installation process. Depending on your environment, you may need to run this with `sudo` command. +3. Enter the repository and run the `run.sh` script to start the Docker container. This will start the automatic installation process. Depending on your environment, you may need to run this with 'sudo' command. - - For latest Cloudberry release running on a single container: + - **Recommended: Build from your current local source code (single container)** + + This is the most efficient option for both new users and developers. It uses your local checkout directly, saving time by skipping the code download step inside the container. It also allows you to immediately test any local code modifications. + + ```shell + cd cloudberry/devops/sandbox + ./run.sh -c local + ``` + + - **Recommended: Build from your current local source code (multi-container)** + + Same as above, but deploys a multi-container cluster. Ideal for testing distributed features or high availability with your local code. ```shell - cd cloudberry-bootcamp/000-cbdb-sandbox - ./run.sh + cd cloudberry/devops/sandbox + ./run.sh -c local -m ``` - - For latest Cloudberry release running across multiple containers: + + - For latest Apache Cloudberry release running on a single container + + ```shell + cd cloudberry/devops/sandbox + ./run.sh -c 2.0.0 + ``` + + - For latest Apache Cloudberry release running across multiple containers ```shell - cd cloudberry-bootcamp/000-cbdb-sandbox - ./run.sh -m + cd cloudberry/devops/sandbox + ./run.sh -c 2.0.0 -m ``` - - For latest main branch running on a single container: + + - For latest main branch running on a single container ```shell - cd cloudberry-bootcamp/000-cbdb-sandbox + cd cloudberry/devops/sandbox ./run.sh -c main ``` - - For latest main branch running across multiple containers: + - For latest main branch running across multiple containers ```shell - cd cloudberry-bootcamp/000-cbdb-sandbox + cd cloudberry/devops/sandbox ./run.sh -c main -m ``` - Once the script finishes without error, the sandbox is built and running successfully. The `docker run` and `docker compose` commands use the `--detach` option allowing you to ssh or access the running Cloudberry instance remotely. + Once the script finishes without error, the sandbox is built and running successfully. The `docker run` and `docker compose` commands use the --detach option allowing you to ssh or access the running Apache Cloudberry instance remotely. - Please review `run.sh` script for additional options (e.g. setting Timezone in running container, only building container). You can also execute `./run.sh -h` to see the usage. + Please review run.sh script for additional options (e.g. setting Timezone in running container, only building container). You can also execute `./run.sh -h` to see the usage. ## Connect to the database diff --git a/docs/tutorials/crash-course.md b/docs/tutorials/crash-course.md index f2edd69e41..2478474dc1 100644 --- a/docs/tutorials/crash-course.md +++ b/docs/tutorials/crash-course.md @@ -33,7 +33,7 @@ Topics include: ## Lesson 0. Prerequisite -Before starting this crash course, spend some time going through the [Apache Cloudberry Tutorials Based on Single-Node Installation](https://github.com/apache/cloudberry-bootcamp/tree/main/000-cbdb-sandbox/configs) to get familiar with Apache Cloudberry and how it works. +Before starting this crash course, spend some time going through the [Apache Cloudberry Tutorials Based on Single-Node Installation](https://github.com/apache/cloudberry/tree/main/devops/sandbox) to get familiar with Apache Cloudberry and how it works. ## Lesson 1. Where to read the official documentation @@ -43,7 +43,7 @@ Take a quick look at the official [Cloudberry Documentation](https://cloudberry. To begin your journey with Cloudberry, you are expected to install Cloudberry in your preferred environment. The following options are available: -- For testing or trying out Cloudberry in a sandbox environment, see [Install Cloudberry in a Sandbox](https://github.com/apache/cloudberry-bootcamp/tree/main/000-cbdb-sandbox/configs). +- For testing or trying out Cloudberry in a sandbox environment, see [Install Cloudberry in a Sandbox](https://github.com/apache/cloudberry/tree/main/devops/sandbox). - For deploying Cloudberry in other environments (including the production environment) and the prerequisite software/hardware configuration, see [Cloudberry Deployment Guide](http://localhost:3000/docs/next/deployment/). ## Lesson 3. Cluster architecture diff --git a/docs/tutorials/quick-trial-lessons/create-users-and-roles.md b/docs/tutorials/quick-trial-lessons/create-users-and-roles.md index 6ab18c3680..a1208c912e 100644 --- a/docs/tutorials/quick-trial-lessons/create-users-and-roles.md +++ b/docs/tutorials/quick-trial-lessons/create-users-and-roles.md @@ -13,7 +13,7 @@ Permissions can be granted to users or groups. Initially, only the `gpadmin` rol You can follow the examples below to create users and roles. -Before moving on to the operations, make sure that you have installed Apache Cloudberry by following [Install a Apache Cloudberry](https://github.com/apache/cloudberry-bootcamp/tree/main/000-cbdb-sandbox). +Before moving on to the operations, make sure that you have installed Apache Cloudberry by following [Install a Apache Cloudberry](https://github.com/apache/cloudberry/tree/main/devops/sandbox). ### Create a user using the CREATE USER command diff --git a/versioned_docs/version-2.x/deployment/build-based-on-docker.md b/versioned_docs/version-2.x/deployment/build-based-on-docker.md index 027fed8705..4fd74f4c53 100644 --- a/versioned_docs/version-2.x/deployment/build-based-on-docker.md +++ b/versioned_docs/version-2.x/deployment/build-based-on-docker.md @@ -95,3 +95,34 @@ Memory .............. : 15Gi total - [Build and Install Apache Cloudberry and contrib extensions](./build-and-install.md) - [Set up a Cloudberry Demo Cluster](./set-demo-cluster.md) (start with the step `Set up Cloudberry environment variables`) - [Post installation](./post-installation.md) + + +Or just running the following command more quickly: + +:::note +Available since version 2.1.0. +::: + +```bash +## Download the source code +git clone --recurse-submodules --branch REL_2_STABLE --single-branch --depth=1 https://github.com/apache/cloudberry.git + +## Change directory +cd /home/gpadmin/cloudberry + +## Set up the SRC_DIR environment variable and create a directory for build logs +export SRC_DIR=/home/gpadmin/cloudberry +mkdir -p ${SRC_DIR}/build-logs + +## Configure, build, and install Cloudberry +./devops/build/automation/cloudberry/scripts/configure-cloudberry.sh +./devops/build/automation/cloudberry/scripts/build-cloudberry.sh + +## Set up a Cloudberry Demo Cluster +./devops/build/automation/cloudberry/scripts/create-cloudberry-demo-cluster.sh + +## Start a Cloudberry Demo Cluster +source /usr/local/cloudberry-db/cloudberry-env.sh +source /home/gpadmin/cloudberry/gpAux/gpdemo/gpdemo-env.sh +psql -p 7000 postgres +``` \ No newline at end of file diff --git a/versioned_docs/version-2.x/deployment/configure.md b/versioned_docs/version-2.x/deployment/configure.md index ee715463c9..f64e60be83 100644 --- a/versioned_docs/version-2.x/deployment/configure.md +++ b/versioned_docs/version-2.x/deployment/configure.md @@ -34,7 +34,6 @@ export LD_LIBRARY_PATH=/usr/local/cloudberry-db/lib:LD_LIBRARY_PATH ./configure --prefix=/usr/local/cloudberry-db \ --disable-external-fts \ --enable-debug \ - --enable-cassert \ --enable-debug-extensions \ --enable-gpcloud \ --enable-ic-proxy \ @@ -42,7 +41,7 @@ export LD_LIBRARY_PATH=/usr/local/cloudberry-db/lib:LD_LIBRARY_PATH --enable-orafce \ --enable-orca \ --enable-pax \ - --enable-pxf \ + --disable-pxf \ --enable-tap-tests \ --with-gssapi \ --with-ldap \ @@ -81,7 +80,6 @@ cd ~/cloudberry ./configure --prefix=/usr/local/cloudberry-db \ --disable-external-fts \ --enable-debug \ - --enable-cassert \ --enable-debug-extensions \ --enable-gpcloud \ --enable-ic-proxy \ @@ -89,7 +87,7 @@ cd ~/cloudberry --enable-orafce \ --enable-orca \ --enable-pax \ - --enable-pxf \ + --disable-pxf \ --enable-tap-tests \ --with-gssapi \ --with-ldap \ @@ -123,7 +121,7 @@ Also, some packages names vary between different Linux distributions. |--|--|--| | `--prefix=PREFIX` |Installation directory. `/usr/local/cbdb` is the default value. `make install` will install all the files in `/usr/local/cbdb/bin`, `/usr/local/cbdb/lib` etc. | You can specify an installation prefix other than `/usr/local/cbdb` using `--prefix`. In this guide, we use `/usr/local/cloudberry-db` as the installation directory. | | `--disable-gpfdist` | Do not use gpfdist | Enable gpfdist by default. This requires apr lib and libevent to be installed.| -| `--disable-pxf` | Do not build PXF. | Enable PXF by default. |PXF is a query federation engine that accesses data residing in external systems such as Hadoop, Hive, HBase, relational databases, S3, Google Cloud Storage, among other external systems.| +| `--disable-pxf` | Do not build PXF. | Enable PXF by default. PXF is a query federation engine that accesses data residing in external systems such as Hadoop, Hive, HBase, relational databases, S3, Google Cloud Storage, among other external systems. Now the [cloudberry-pxf](https://github.com/apache/cloudberry-pxf/tree/main/fdw) will be kept as the latest version of `pxf_fdw`.| | `--enable-orafce` | Build with Oracle compatibility functions. | | | `--enable-debug` | Build all programs and libraries with debugging symbols.| This means that you can run the programs in a debugger to analyze problems. This enlarges the size of the installed executables considerably, and on non-GCC compilers it usually also disables compiler optimization, causing slowdowns. However, having the symbols available is extremely helpful for dealing with any problems that might arise. Currently, this option is recommended for production installations only if you use GCC. But you should always have it on if you are doing development work or running a beta version.| | `--enable-profiling` | Build with profiling enabled.|This option is for use only with GCC and when doing development work.| diff --git a/versioned_docs/version-2.x/deployment/create-gpadmin-user.md b/versioned_docs/version-2.x/deployment/create-gpadmin-user.md index 61bb224086..89a1d84c48 100644 --- a/versioned_docs/version-2.x/deployment/create-gpadmin-user.md +++ b/versioned_docs/version-2.x/deployment/create-gpadmin-user.md @@ -62,6 +62,35 @@ This script performs three main tasks as the `gpadmin` user: The script uses a heredoc (EOF) block to execute multiple commands under the `gpadmin` user context. This will be used multiple time throughout these instructions. + + +```bash +sudo -u gpadmin bash <<'EOF' +# Add Cloudberry environment setup to .bashrc +echo -e '\n# Add Cloudberry entries +if [ -f /usr/local/cloudberry-db/cloudberry-env.sh ]; then + source /usr/local/cloudberry-db/cloudberry-env.sh +fi + +# US English with UTF-8 character encoding +export LANG=en_US.UTF-8 +' >> /home/gpadmin/.bashrc + +# Set up SSH for passwordless access +mkdir -p /home/gpadmin/.ssh +if [ ! -f /home/gpadmin/.ssh/id_rsa ]; then + ssh-keygen -t rsa -b 2048 -C 'apache-cloudberry-dev' -f /home/gpadmin/.ssh/id_rsa -N "" +fi +cat /home/gpadmin/.ssh/id_rsa.pub >> /home/gpadmin/.ssh/authorized_keys + +# Set proper SSH directory permissions +chmod 700 /home/gpadmin/.ssh +chmod 600 /home/gpadmin/.ssh/authorized_keys +chmod 644 /home/gpadmin/.ssh/id_rsa.pub +EOF +``` + + ```bash sudo -u gpadmin bash <<'EOF' # Add Cloudberry environment setup to .bashrc @@ -86,4 +115,6 @@ chmod 700 /home/gpadmin/.ssh chmod 600 /home/gpadmin/.ssh/authorized_keys chmod 644 /home/gpadmin/.ssh/id_rsa.pub EOF -``` \ No newline at end of file +``` + + \ No newline at end of file diff --git a/versioned_docs/version-2.x/deployment/download-source-code.md b/versioned_docs/version-2.x/deployment/download-source-code.md index e6a2691b0a..b0edd10744 100644 --- a/versioned_docs/version-2.x/deployment/download-source-code.md +++ b/versioned_docs/version-2.x/deployment/download-source-code.md @@ -18,7 +18,13 @@ Clone the release source code for Apache Cloudberry into the `gpadmin` user's ho git clone https://github.com/apache/cloudberry.git ~/cloudberry cd ~/cloudberry git fetch --tags + +# For Apache Cloudberry 2.0.0 git checkout tags/2.0.0-incubating + +# For Apache Cloudberry 2.1.0 +git checkout tags/2.1.0-incubating + git submodule update --init --recursive ``` @@ -45,11 +51,21 @@ export GNUTLS_CPUID_OVERRIDE=0x1 Alternatively, you can download the source code archive from the [Apache Cloudberry releases page](/releases). + +- For Apache Cloudberry 2.0.0 + ```bash tar xvzf apache-cloudberry-2.0.0-incubating-src.tar.gz mv apache-cloudberry-2.0.0-incubating cloudberry ``` +- For Apache Cloudberry 2.1.0 + +```bash +tar xvzf apache-cloudberry-2.1.0-incubating-src.tar.gz +mv apache-cloudberry-2.1.0-incubating cloudberry +``` + :::note -The submodules are already included in the latest 2.0.0 release source code archive, so you don't need to download the submodules manually after extracting the archive. +The submodules are already included in the latest 2.x.0 release source code archive, so you don't need to download the submodules manually after extracting the archive. ::: \ No newline at end of file diff --git a/versioned_docs/version-2.x/deployment/install-required-packages.md b/versioned_docs/version-2.x/deployment/install-required-packages.md index f6ba2c7bb2..41db3656e9 100644 --- a/versioned_docs/version-2.x/deployment/install-required-packages.md +++ b/versioned_docs/version-2.x/deployment/install-required-packages.md @@ -60,11 +60,29 @@ On Rocky Linux, they are disabled by default and must be explicitly enabled. + +- For Apache Cloudberry 2.1.0 + +```bash +sudo dnf install -y --enablerepo=devel liburing-devel libuv-devel libyaml-devel perl-IPC-Run protobuf-devel +``` + +- For Apache Cloudberry 2.0.0 + ```bash sudo dnf install -y --enablerepo=devel libuv-devel libyaml-devel perl-IPC-Run protobuf-devel ``` + +- For Apache Cloudberry 2.1.0 + +```bash +sudo dnf install -y --enablerepo=crb liburing-devel libuv-devel libyaml-devel perl-IPC-Run protobuf-devel +``` + +- For Apache Cloudberry 2.0.0 + ```bash sudo dnf install -y --enablerepo=crb libuv-devel libyaml-devel perl-IPC-Run protobuf-devel ``` @@ -75,9 +93,13 @@ sudo dnf install -y --enablerepo=crb libuv-devel libyaml-devel perl-IPC-Run prot In Red Hat Enterprise Linux (RHEL), this repository is called "PowerTools." ::: -### Install gcc/gcc-c++ 11+ for Rocky Linux 8 (Optional) +### Install gcc and gcc-c++ 11+ for Rocky Linux 8 (Optional) + +:::tip +This step is only required for Apache Cloudberry 2.0.0. You can skip this step when building Apache Cloudberry 2.1.0. +::: -For Rocky Linux 8, we need to install the higher version of gcc and gcc-c++ to build Apache Cloudberry with PAX support: +For Rocky Linux 8, we need to install the higher version of gcc and gcc-c++ to build Apache Cloudberry 2.0.0 with PAX support: ```bash sudo yum install -y gcc-toolset-11-gcc gcc-toolset-11-gcc-c++ @@ -187,5 +209,32 @@ sudo apt install -y bison \ python3-setuptools \ rsync ``` + +### Extra dependencies for building PAX (Optional) + +:::tip +Only required for Apache Cloudberry 2.1.0. You can skip this step when building Apache Cloudberry 2.0.0. +::: + +In Apache Cloudberry 2.1.0, we have introduced a new dependency `liburing` for building PAX. + +* For Ubuntu 22.04 + +```bash +sudo apt install -y liburing-dev +``` + +* For Ubuntu 20.04 + +```bash +sudo apt install -y git build-essential +wget https://github.com/axboe/liburing/archive/refs/tags/liburing-2.1.tar.gz +tar -xzf liburing-2.1.tar.gz +rm "liburing-2.1.tar.gz" +cd liburing-liburing-2.1 +make -j$(nproc) +sudo make install +sudo ldconfig +``` diff --git a/versioned_docs/version-2.x/deployment/quick-build.md b/versioned_docs/version-2.x/deployment/quick-build.md index 5f38f3eabc..8b7d244c1f 100644 --- a/versioned_docs/version-2.x/deployment/quick-build.md +++ b/versioned_docs/version-2.x/deployment/quick-build.md @@ -10,7 +10,188 @@ In the following sections, we'll provide detailed, step-by-step instructions for By this, you will get an Apache Cloudberry environment with a demo cluster ready for testing and development. - + + +Below are the instructions for building Apache Cloudberry 2.0.0 and Apache Cloudberry 2.1.0 from source code on Rocky Linux 8/9: +- [For Apache Cloudberry 2.1.0](#for-apache-cloudberry-210) +- [For Apache Cloudberry 2.0.0](#for-apache-cloudberry-200) + +### For Apache Cloudberry 2.1.0 + +```bash +# Install sudo & git +dnf install -y sudo git + +# Create and configure the gpadmin user +sudo useradd -U -m -s /bin/bash gpadmin +echo 'gpadmin ALL=(ALL) NOPASSWD:ALL' | sudo tee /etc/sudoers.d/90-gpadmin +sudo -u gpadmin sudo whoami # if the output is root, the configuration is correct + + +# Required configuration for gpadmin user +sudo -u gpadmin bash <<'EOF' +## Add Cloudberry environment setup to .bashrc +echo -e '\n# Add Cloudberry entries +if [ -f /usr/local/cloudberry-db/cloudberry-env.sh ]; then + source /usr/local/cloudberry-db/cloudberry-env.sh +fi +## US English with UTF-8 character encoding +export LANG=en_US.UTF-8 +' >> /home/gpadmin/.bashrc +## Set up SSH for passwordless access +mkdir -p /home/gpadmin/.ssh +if [ ! -f /home/gpadmin/.ssh/id_rsa ]; then + ssh-keygen -t rsa -b 2048 -C 'apache-cloudberry-dev' -f /home/gpadmin/.ssh/id_rsa -N "" +fi +cat /home/gpadmin/.ssh/id_rsa.pub >> /home/gpadmin/.ssh/authorized_keys +## Set proper SSH directory permissions +chmod 700 /home/gpadmin/.ssh +chmod 600 /home/gpadmin/.ssh/authorized_keys +chmod 644 /home/gpadmin/.ssh/id_rsa.pub +EOF + + +# Configure system settings +sudo tee /etc/security/limits.d/90-db-limits.conf << 'EOF' +## Core dump file size limits for gpadmin +gpadmin soft core unlimited +gpadmin hard core unlimited +## Open file limits for gpadmin +gpadmin soft nofile 524288 +gpadmin hard nofile 524288 +## Process limits for gpadmin +gpadmin soft nproc 131072 +gpadmin hard nproc 131072 +EOF + +# Verify resource limits. +sudo -u gpadmin bash -c "ulimit -a" + +# Install required packages +sudo dnf install -y apr-devel \ + bison \ + bzip2-devel \ + curl \ + cmake3 \ + diffutils \ + flex \ + gcc \ + gcc-c++ \ + glibc-langpack-en \ + glibc-locale-source \ + iproute \ + krb5-devel \ + libcurl-devel \ + libevent-devel \ + libxml2-devel \ + libuuid-devel \ + libzstd-devel \ + lz4-devel \ + net-tools \ + openldap-devel \ + openssl-devel \ + openssh-server \ + pam-devel \ + perl \ + perl-ExtUtils-Embed \ + perl-Test-Simple \ + perl-Env \ + python3-devel \ + python3-pip \ + readline-devel \ + rsync \ + wget \ + which \ + zlib-devel + +# Enable additional development tools and libraries +## For Rocky Linux 8 +sudo dnf install -y --enablerepo=devel liburing-devel libuv-devel libyaml-devel perl-IPC-Run protobuf-devel +## For Rocky Linux 9 +sudo dnf install -y --enablerepo=crb liburing-devel libuv-devel libyaml-devel perl-IPC-Run protobuf-devel + +# Build Xerces-C source code +XERCES_LATEST_RELEASE=3.3.0 +XERCES_INSTALL_PREFIX="/usr/local/xerces-c" +wget -nv "https://dlcdn.apache.org//xerces/c/3/sources/xerces-c-${XERCES_LATEST_RELEASE}.tar.gz" +echo "$(curl -sL https://dlcdn.apache.org//xerces/c/3/sources/xerces-c-${XERCES_LATEST_RELEASE}.tar.gz.sha256)" | sha256sum -c - +tar xf "xerces-c-${XERCES_LATEST_RELEASE}.tar.gz" +rm "xerces-c-${XERCES_LATEST_RELEASE}.tar.gz" +cd xerces-c-${XERCES_LATEST_RELEASE} +./configure --prefix="${XERCES_INSTALL_PREFIX}-${XERCES_LATEST_RELEASE}" +make -j$(nproc) +make check +sudo make install +sudo ln -s ${XERCES_INSTALL_PREFIX}-${XERCES_LATEST_RELEASE} ${XERCES_INSTALL_PREFIX} + +# Switch to the gpadmin user from now on +sudo su - gpadmin + +# Download Source Code (2.x branch) +git clone https://github.com/apache/cloudberry.git ~/cloudberry +cd ~/cloudberry +git fetch --tags +git checkout tags/2.1.0-incubating +git submodule update --init --recursive + +# Prepare the build environment for Apache Cloudberry +sudo rm -rf /usr/local/cloudberry-db +sudo chmod a+w /usr/local +mkdir -p /usr/local/cloudberry-db/lib +sudo cp -v /usr/local/xerces-c/lib/libxerces-c.so \ + /usr/local/xerces-c/lib/libxerces-c-3.*.so \ + /usr/local/cloudberry-db/lib +sudo chown -R gpadmin:gpadmin /usr/local/cloudberry-db + +# Run configure +cd ~/cloudberry +export LD_LIBRARY_PATH=/usr/local/cloudberry-db/lib:LD_LIBRARY_PATH +./configure --prefix=/usr/local/cloudberry-db \ + --disable-external-fts \ + --enable-debug \ + --enable-debug-extensions \ + --enable-gpcloud \ + --enable-ic-proxy \ + --enable-mapreduce \ + --enable-orafce \ + --enable-orca \ + --enable-pax \ + --disable-pxf \ + --enable-tap-tests \ + --with-gssapi \ + --with-ldap \ + --with-libxml \ + --with-lz4 \ + --with-pam \ + --with-perl \ + --with-pgport=5432 \ + --with-python \ + --with-pythonsrc-ext \ + --with-ssl=openssl \ + --with-uuid=e2fs \ + --with-includes=/usr/local/xerces-c/include \ + --with-libraries=/usr/local/cloudberry-db/lib + +# Build and install Cloudberry and its contrib modules +make -j$(nproc) -C ~/cloudberry +make -j$(nproc) -C ~/cloudberry/contrib +make install -C ~/cloudberry +make install -C ~/cloudberry/contrib + +# Verify the installation +/usr/local/cloudberry-db/bin/postgres --gp-version +/usr/local/cloudberry-db/bin/postgres --version +ldd /usr/local/cloudberry-db/bin/postgres + +# Set up a Cloudberry demo cluster +source /usr/local/cloudberry-db/cloudberry-env.sh +make create-demo-cluster -C ~/cloudberry +source ~/cloudberry/gpAux/gpdemo/gpdemo-env.sh +psql -P pager=off template1 -c 'SELECT * from gp_segment_configuration' +psql template1 -c 'SELECT version()' +``` + +### For Apache Cloudberry 2.0.0 ```bash # Install sudo & git @@ -105,6 +286,8 @@ sudo dnf install -y --enablerepo=devel libuv-devel libyaml-devel perl-IPC-Run pr sudo dnf install -y --enablerepo=crb libuv-devel libyaml-devel perl-IPC-Run protobuf-devel # Only for Rocky Linux 8, install the higher version of gcc and gcc-c++ +# Only needed for Apache Cloudberry 2.0.0 +# You can skip this step if you are using Apache Cloudberry 2.1.0 or later sudo yum install -y gcc-toolset-11-gcc gcc-toolset-11-gcc-c++ scl enable gcc-toolset-11 bash # for temprory use sudo echo "source /opt/rh/gcc-toolset-11/enable" >> /etc/profile.d/gcc.sh @@ -149,7 +332,6 @@ export LD_LIBRARY_PATH=/usr/local/cloudberry-db/lib:LD_LIBRARY_PATH ./configure --prefix=/usr/local/cloudberry-db \ --disable-external-fts \ --enable-debug \ - --enable-cassert \ --enable-debug-extensions \ --enable-gpcloud \ --enable-ic-proxy \ @@ -157,7 +339,7 @@ export LD_LIBRARY_PATH=/usr/local/cloudberry-db/lib:LD_LIBRARY_PATH --enable-orafce \ --enable-orca \ --enable-pax \ - --enable-pxf \ + --disable-pxf \ --enable-tap-tests \ --with-gssapi \ --with-ldap \ @@ -192,7 +374,175 @@ psql -P pager=off template1 -c 'SELECT * from gp_segment_configuration' psql template1 -c 'SELECT version()' ``` - + + +Below are the instructions for building Apache Cloudberry 2.0.0 and Apache Cloudberry 2.1.0 from source code on Ubuntu 20.04/22.04: +- [For Apache Cloudberry 2.1.0](#for-apache-cloudberry-210-1) +- [For Apache Cloudberry 2.0.0](#for-apache-cloudberry-200-1) + +### For Apache Cloudberry 2.1.0 + +```bash +# Install sudo & git +apt update && apt install -y sudo git + +# Create and configure the gpadmin user +sudo useradd -U -m -s /bin/bash gpadmin +echo 'gpadmin ALL=(ALL) NOPASSWD:ALL' | sudo tee /etc/sudoers.d/90-gpadmin +sudo -u gpadmin sudo whoami # if the output is root, the configuration is correct + +# Required configuration +sudo -u gpadmin bash <<'EOF' +## Add Cloudberry environment setup to .bashrc +echo -e '\n# Add Cloudberry entries +if [ -f /usr/local/cloudberry-db/cloudberry-env.sh ]; then + source /usr/local/cloudberry-db/cloudberry-env.sh +fi +## US English with UTF-8 character encoding +export LANG=en_US.UTF-8 +' >> /home/gpadmin/.bashrc +## Set up SSH for passwordless access +mkdir -p /home/gpadmin/.ssh +if [ ! -f /home/gpadmin/.ssh/id_rsa ]; then + ssh-keygen -t rsa -b 2048 -C 'apache-cloudberry-dev' -f /home/gpadmin/.ssh/id_rsa -N "" +fi +cat /home/gpadmin/.ssh/id_rsa.pub >> /home/gpadmin/.ssh/authorized_keys +## Set proper SSH directory permissions +chmod 700 /home/gpadmin/.ssh +chmod 600 /home/gpadmin/.ssh/authorized_keys +chmod 644 /home/gpadmin/.ssh/id_rsa.pub +EOF + +# Configure system settings +sudo tee /etc/security/limits.d/90-db-limits.conf << 'EOF' +## Core dump file size limits for gpadmin +gpadmin soft core unlimited +gpadmin hard core unlimited +## Open file limits for gpadmin +gpadmin soft nofile 524288 +gpadmin hard nofile 524288 +## Process limits for gpadmin +gpadmin soft nproc 131072 +gpadmin hard nproc 131072 +EOF + +# Verify resource limits +sudo -u gpadmin bash -c "ulimit -a" + +# Install basic system packages +sudo apt install -y bison \ + bzip2 \ + cmake \ + curl \ + flex \ + gcc \ + g++ \ + iproute2 \ + iputils-ping \ + language-pack-en \ + locales \ + libapr1-dev \ + libbz2-dev \ + libcurl4-gnutls-dev \ + libevent-dev \ + libkrb5-dev \ + libipc-run-perl \ + libldap2-dev \ + libpam0g-dev \ + libprotobuf-dev \ + libreadline-dev \ + libssl-dev \ + libuv1-dev \ + liblz4-dev \ + libxerces-c-dev \ + libxml2-dev \ + libyaml-dev \ + libzstd-dev \ + libperl-dev \ + make \ + pkg-config \ + protobuf-compiler \ + python3-dev \ + python3-pip \ + python3-setuptools \ + rsync + +# For PAX build, you need to install liburing +## For Ubuntu 22.04 + sudo apt install -y liburing-dev +## For Ubuntu 20.04 + sudo apt install -y git build-essential + wget https://github.com/axboe/liburing/archive/refs/tags/liburing-2.1.tar.gz + tar -xzf liburing-2.1.tar.gz + rm "liburing-2.1.tar.gz" + cd liburing-liburing-2.1 + make -j$(nproc) + sudo make install + sudo ldconfig + +# Use the gpadmin user from now on +sudo su - gpadmin + +# Clone the Apache Cloudberry repository (2.x branch) +git clone https://github.com/apache/cloudberry.git ~/cloudberry +cd ~/cloudberry +git fetch --tags +git checkout tags/2.1.0-incubating +git submodule update --init --recursive + +# Prepare the build environment for Apache Cloudberry +sudo rm -rf /usr/local/cloudberry-db +sudo chmod a+w /usr/local +mkdir -p /usr/local/cloudberry-db +sudo chown -R gpadmin:gpadmin /usr/local/cloudberry-db + +# Run configure +cd ~/cloudberry +./configure --prefix=/usr/local/cloudberry-db \ + --disable-external-fts \ + --enable-debug \ + --enable-debug-extensions \ + --enable-gpcloud \ + --enable-ic-proxy \ + --enable-mapreduce \ + --enable-orafce \ + --enable-orca \ + --enable-pax \ + --disable-pxf \ + --enable-tap-tests \ + --with-gssapi \ + --with-ldap \ + --with-libxml \ + --with-lz4 \ + --with-pam \ + --with-perl \ + --with-pgport=5432 \ + --with-python \ + --with-pythonsrc-ext \ + --with-ssl=openssl \ + --with-uuid=e2fs \ + --with-includes=/usr/include/xercesc + +# Build and install Cloudberry and its contrib modules +make -j$(nproc) -C ~/cloudberry +make -j$(nproc) -C ~/cloudberry/contrib +make install -C ~/cloudberry +make install -C ~/cloudberry/contrib + +# Verify the installation +/usr/local/cloudberry-db/bin/postgres --gp-version +/usr/local/cloudberry-db/bin/postgres --version +ldd /usr/local/cloudberry-db/bin/postgres + +# Set up a Cloudberry demo cluster +source /usr/local/cloudberry-db/cloudberry-env.sh +make create-demo-cluster -C ~/cloudberry +source ~/cloudberry/gpAux/gpdemo/gpdemo-env.sh +psql -P pager=off template1 -c 'SELECT * from gp_segment_configuration' +psql template1 -c 'SELECT version()' +``` + +### For Apache Cloudberry 2.0.0 ```bash # Install sudo & git @@ -300,7 +650,6 @@ cd ~/cloudberry ./configure --prefix=/usr/local/cloudberry-db \ --disable-external-fts \ --enable-debug \ - --enable-cassert \ --enable-debug-extensions \ --enable-gpcloud \ --enable-ic-proxy \ @@ -308,7 +657,7 @@ cd ~/cloudberry --enable-orafce \ --enable-orca \ --enable-pax \ - --enable-pxf \ + --disable-pxf \ --enable-tap-tests \ --with-gssapi \ --with-ldap \ diff --git a/versioned_docs/version-2.x/deployment/sandbox.md b/versioned_docs/version-2.x/deployment/sandbox.md index dfe3abd3f4..b6bcc652f7 100644 --- a/versioned_docs/version-2.x/deployment/sandbox.md +++ b/versioned_docs/version-2.x/deployment/sandbox.md @@ -3,6 +3,10 @@ title: Get Started with Sandbox description: Learn how to quickly set up and connect to a Cloudberry in a Docker environment. --- +:::tip +Please note that the sandbox files have been migrated from `apache/cloudberry-bootcamp` to [`apache/cloudberry/devops/sandbox`](https://github.com/apache/cloudberry/tree/main/devops/sandbox). The local mode is available from version 2.1.0. +::: + This document guides you on how to quickly set up and connect to a Apache Cloudberry in a Docker environment. You can try out Apache Cloudberry by performing some basic operations and running SQL commands. :::caution @@ -22,8 +26,9 @@ When building and deploying Cloudberry in Docker, you will have 2 different depl **Deployment Options** -1. **Single Container** (Default) - With the single container option, you will have the coordinator as well as the Cloudberry segments all running on a single container. This is the default behavior when deploying using the `run.sh` script provided. -2. **Multi-Container** - Deploying with the multi-container option will give you a more realistic deployment of what actual production Cloudberry clusters look like. With multi-node, you will have the coordinator, the standby coordinator, and 2 segment hosts all on their own respective containers. This is to both highlight the distributed nature of Apache Cloudberry as well as highlight how high availability (HA) features work in the event of a server (or in this case a container) failing. This is enabled by passing the `-m` flag to the `run.sh` script which will be highlighted below. +1. **Recommended for most users** – Build directly from your current local source code using `-c local`. This is the fastest way to get started as it reuses your existing checkout, avoiding the need to download the code again inside the container. It is also ideal for developers testing local changes. +2. Compile with the source code of the latest Apache Cloudberry (released in [Apache Cloudberry Release Page](https://github.com/apache/cloudberry/releases)). The base OS will be Rocky Linux 9 Docker image. +3. Compile with the latest Apache Cloudberry [main](https://github.com/apache/cloudberry/tree/main) branch. The base OS will be Rocky Linux 9 Docker image. ![Apache Cloudberry Sandbox Deployments](/img/bootcamp/sandbox-deployment.jpg) @@ -36,43 +41,63 @@ Build and deploy steps: 1. Start Docker Desktop and make sure it is running properly on your host platform. -2. Download the repository [apache/cloudberry-bootcamp](https://github.com/apache/cloudberry-bootcamp) to the target machine. +2. Clone the Apache Cloudberry repository to the target machine. + + ```shell + git clone https://github.com/apache/cloudberry.git + ``` + +3. Enter the repository and run the `run.sh` script to start the Docker container. This will start the automatic installation process. Depending on your environment, you may need to run this with 'sudo' command. + + - **Recommended: Build from your current local source code (single container)** + + This is the most efficient option for both new users and developers. It uses your local checkout directly, saving time by skipping the code download step inside the container. It also allows you to immediately test any local code modifications. ```shell - git clone https://github.com/apache/cloudberry-bootcamp.git + cd cloudberry/devops/sandbox + ./run.sh -c local ``` -3. Enter the repository and run the `run.sh` script to start the Docker container. This will start the automatic installation process. Depending on your environment, you may need to run this with `sudo` command. + - **Recommended: Build from your current local source code (multi-container)** - - For latest Cloudberry release running on a single container: + Same as above, but deploys a multi-container cluster. Ideal for testing distributed features or high availability with your local code. ```shell - cd cloudberry-bootcamp/000-cbdb-sandbox - ./run.sh + cd cloudberry/devops/sandbox + ./run.sh -c local -m ``` - - For latest Cloudberry release running across multiple containers: + + - For latest Apache Cloudberry release running on a single container ```shell - cd cloudberry-bootcamp/000-cbdb-sandbox - ./run.sh -m + cd cloudberry/devops/sandbox + ./run.sh -c 2.0.0 ``` - - For latest main branch running on a single container: + + - For latest Apache Cloudberry release running across multiple containers + + ```shell + cd cloudberry/devops/sandbox + ./run.sh -c 2.0.0 -m + ``` + + - For latest main branch running on a single container ```shell - cd cloudberry-bootcamp/000-cbdb-sandbox + cd cloudberry/devops/sandbox ./run.sh -c main ``` - - For latest main branch running across multiple containers: + - For latest main branch running across multiple containers ```shell - cd cloudberry-bootcamp/000-cbdb-sandbox + cd cloudberry/devops/sandbox ./run.sh -c main -m ``` - Once the script finishes without error, the sandbox is built and running successfully. The `docker run` and `docker compose` commands use the `--detach` option allowing you to ssh or access the running Cloudberry instance remotely. + Once the script finishes without error, the sandbox is built and running successfully. The `docker run` and `docker compose` commands use the --detach option allowing you to ssh or access the running Apache Cloudberry instance remotely. - Please review `run.sh` script for additional options (e.g. setting Timezone in running container, only building container). You can also execute `./run.sh -h` to see the usage. + Please review run.sh script for additional options (e.g. setting Timezone in running container, only building container). You can also execute `./run.sh -h` to see the usage. ## Connect to the database diff --git a/versioned_docs/version-2.x/deployment/set-demo-cluster.md b/versioned_docs/version-2.x/deployment/set-demo-cluster.md index fee5d0a3bb..f5faf160b2 100644 --- a/versioned_docs/version-2.x/deployment/set-demo-cluster.md +++ b/versioned_docs/version-2.x/deployment/set-demo-cluster.md @@ -30,10 +30,18 @@ ssh $(hostname) date ## Set up Cloudberry environment variables +:::warning +Please note that the `greenplum_path.sh` has changed to `cloudberry-env.sh` since Cloudberry 2.1.0. You can learn more about the change in this [blog post](/blog/from-greenplum-path.sh-to-cloudberry-env.sh). +::: + Load Cloudberry environment variables that set up paths for binaries, libraries, and other essential components: ```bash +# For Apache Cloudberry 2.0.0 source /usr/local/cloudberry-db/greenplum_path.sh + +# For Apache Cloudberry 2.1.0 +source /usr/local/cloudberry-db/cloudberry-env.sh ``` ## Create development cluster