Skip to content

ci: add retry for flaky network operations (cargo test, docker compose) #7703

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
TennyZhuang wants to merge 2 commits into
base: main
Choose a base branch
from
Open

ci: add retry for flaky network operations (cargo test, docker compose) #7703

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2d51668
ci: add retry for flaky network operations (cargo test, docker compose)
TennyZhuang Jun 7, 2026
f7b18e4
fix: quote heredoc delimiter to prevent bash expansion in generated a…
TennyZhuang Jun 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions .github/actions/test_behavior_core/action.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ runs:
shell: bash
run: |
mkdir -p ./dynamic_test_core &&
cat <<EOF >./dynamic_test_core/action.yml
cat <<'EOF' >./dynamic_test_core/action.yml
runs:
using: composite
steps:
Expand All @@ -41,7 +41,19 @@ runs:
- name: Run Test Core
shell: bash
working-directory: core
run: cargo test behavior --features tests,${{ inputs.feature }}
run: |
# Retry cargo test up to 3 times to handle transient network issues
# when fetching crates from crates.io.
for i in $(seq 1 3); do
echo "--- cargo test behavior attempt $i/3 ---"
if cargo test behavior --features tests,${{ inputs.feature }}; then
exit 0
fi
if [ "$i" -lt 3 ]; then
sleep $((i * 10))
fi
done
exit 1
env:
OPENDAL_TEST: ${{ inputs.service }}
EOF
Expand Down
15 changes: 14 additions & 1 deletion .github/services/hdfs_native/hdfs_native_cluster/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,20 @@ runs:
- name: Setup HDFS Native cluster
shell: bash
working-directory: fixtures/hdfs
run: docker compose -f docker-compose-hdfs-cluster.yml up -d --wait
run: |
# Retry docker compose up to 3 times to handle transient network issues

@erickguan erickguan Jun 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea. For handling pull failure, we can have a separate step (or a nested group) calling docker compose pull --quiet --ignore-pull-failures.

This will not conceal container problems.

# (e.g., Docker Hub pull timeouts).
for i in $(seq 1 3); do
echo "--- docker compose up attempt $i/3 ---"
if docker compose -f docker-compose-hdfs-cluster.yml up -d --wait; then
exit 0
fi
if [ "$i" -lt 3 ]; then
sleep $((i * 10))
docker compose -f docker-compose-hdfs-cluster.yml down 2>/dev/null || true
fi
done
exit 1
- name: Setup opendal env
shell: bash
run: |
Expand Down