Improve PXF build when in special network environments.

[MisterRaindrop]

https://github.com/apache/cloudberry-pxf/blob/main/ci/docker/pxf-cbdb-dev/ubuntu/script/build_pxf.sh

# mirror
# If the download fails, you can uncomment the line to switch to another mirror address.
# Configure Gradle to use Aliyun mirror
# mkdir -p ~/.gradle
# cat > ~/.gradle/init.gradle <<'EOF'
# allprojects {
#     repositories {
#         maven { url 'https://maven.aliyun.com/repository/public/' }
#         maven { url 'https://maven.aliyun.com/repository/gradle-plugin' }
#         mavenCentral()
#     }
#     buildscript {
#         repositories {
#             maven { url 'https://maven.aliyun.com/repository/public/' }
#             maven { url 'https://maven.aliyun.com/repository/gradle-plugin' }
#             mavenCentral()
#         }
#     }
# }
# EOF

and

# export GOPROXY=https://goproxy.cn,direct

I want to improve the go and gradle proxy settings in the script here. In special network environments, compilation often fails. It's troublesome to adjust the proxy every time I compile. Anybody have any better suggestions?

[MisterRaindrop]

@tuhaihe I think you have experience in this kind of problem.

[tuhaihe]

@tuhaihe I think you have experience in this kind of problem.

Want to apply these proxy setting here? We should balance the global and local users' experience. Any auto mirrors policy here?

[ostinru]

export GOPROXY=https://goproxy.cn,direct

I would assume that we don't need golang and pxf-cli inside pxf-cbdb-dev container.

~/.gradle/init.gradle

Alternatevly I can imagine following redising of automation tests: Currently we run automation(java) + PXF inside docker:

block-beta
columns 1
  block:Docker["docker"]
    columns 1
    block:Batch
        Cloudberry
        PXF
        block:Storage
            columns 1
            Hadoop
            S3
        end
    end
    space
    Automation
  end
  

Cloudberry --> PXF
PXF --> Hadoop
PXF --> S3

Loading

And, I think that we can reorganize it to run Automation on host, Possibly run PXF as embedded to automation (so, debug became trivial). Then, Cloudberry / Hadoop / S3 / other services under management of TestContainer.

block-beta
columns 1
  block:Docker["Host"]
    columns 1
    block:Batch["Docker"]
        Cloudberry
        space
        block:Storage
            columns 1
            Hadoop
            S3
        end
    end
    space
    block:Dom0
        
        Automation
        PXF
    end
  end
  

Cloudberry --> PXF
PXF --> Hadoop
PXF --> S3

Loading

In this case gradle will use you host's configuration. However it also has downsides - in tests we will run PXF with all dependencies.

[ostinru]

@MisterRaindrop , do you have access to gradle main repositories, but it slow? What do you think about sharing gradle/maven cache?
https://github.com/open-gpdb/pxf/blob/MDB_STABLE/automation/docker-compose.yml#L34-L36

[MisterRaindrop]

Accessing Go or Gradle often fails, if I don't make special network settings.

[MisterRaindrop]

My initial idea was that, like other projects , the proxy URL could be configured through environment variables. This depends on the user's settings. But how can Gradle be configured better?

[ostinru]

But how can Gradle be configured better?

It seems that gradle build cache & your config both stored in ~/.gradle/. Then mounting host's ~/.gradle to container ~/.gradle should both fix your problem and speed up builds.
Probably, there will be an issue with file permissions.

[MisterRaindrop]

If I mount a local Gradle and configure aliyun it accordingly, it should work. But I'm wondering if it could be made smarter, so that users don't have to configure it themselves? Because you still need to modify your Docker file.