Fix ATXP-654: Include hidden files in subdirectories during deploy#3
Conversation
Fixed ATXP-654 where cloud sandboxes were not getting the current working directory due to overly aggressive file exclusion pattern. The zip command in /deploy was using `-x "*/.*"` which excluded ALL hidden files in subdirectories, including critical configuration files like: - .atxp/.env.production (environment variables for sandbox) - config/.env (application configuration) - Other necessary dotfiles in subdirectories This caused deployed sandboxes to initialize with incomplete project files, making them non-functional. Changes: - Removed `-x "*/.*"` exclusion pattern from deploy command - Kept specific exclusions for .git, node_modules, .DS_Store, .atxp-instance - Verified all project files are now included while unwanted files still excluded Testing: - Created test project with hidden files in subdirectories - Confirmed .atxp/.env.production and config/.env are now included - Verified exclusions for .git, node_modules, etc. still work correctly - Tested extraction produces correct directory structure Fixes https://linear.app/circuitandchisel/issue/ATXP-654
Code Review: Fix ATXP-654 - Include hidden files in subdirectories during deploySummaryThis PR fixes a critical bug where the ✅ Strengths1. Problem Identification
2. Minimal, Surgical Fix
3. Testing
🔍 Code Quality & Best PracticesOverall Assessment: Good ✅ The change itself is straightforward and appropriate. The remaining exclusion patterns are:
These patterns are specific and appropriate for the use case. 🐛 Potential Issues1. Pattern Precision
Current pattern Recommendation: Test this specifically or change to 2. Root-level Hidden Files
Question: Should root-level 🔒 Security ConcernsThe fix now allows Concerns:
Recommendations:
Mitigation: The current approach assumes developers understand which files contain secrets appropriate for cloud deployment vs local development. This should be clearly documented. ⚡ Performance ConsiderationsNo Issues ✅ The change actually slightly improves performance by:
Including hidden config files in subdirectories will have negligible impact on zip file size. 🧪 Test CoverageCurrent State: Manual Testing Only The PR includes excellent manual testing with concrete before/after results. However: Gaps:
Recommendations:
📋 RecommendationsBefore Merging:
Future Improvements:
🎯 Overall AssessmentApproval Status: ✅ Approve with Minor Recommendations This is a good fix for a real problem. The core change is correct and solves the immediate issue. The security and pattern precision concerns are worth addressing, but they don't block this PR since:
Recommended Action:
Great job identifying and fixing this issue! The testing approach was thorough and the fix is appropriately minimal. 🎉 |
The previous use of mktemp with a .zip extension template created an empty file that caused zip command to fail with "invalid zip data" errors when the file was uploaded and extracted. Changes: - Changed from mktemp /tmp/deploy-XXXXXX.zip to /tmp/deploy-TIMESTAMP-PID.zip - This ensures the zip file doesn't pre-exist when the zip command runs - Prevents "Zip file structure invalid" and "invalid zip data" errors The mktemp command with a .zip suffix creates an empty placeholder file, which confuses the zip utility. Using timestamp+PID ensures unique filenames without pre-creating empty files.
PR Review: Fix ATXP-654 - Include hidden files in subdirectories during deploySummaryThis PR addresses a critical bug where the deploy command was excluding all hidden files in subdirectories, causing deployed sandboxes to be non-functional. The fix is well-targeted and appropriate. ✅ Code Quality & Best PracticesStrengths:
Observations:
🐛 Potential Issues1. Temporary file naming change (Minor) The PR changes the temporary file creation from: ZIP_FILE=$(mktemp /tmp/deploy-XXXXXX.zip)to: ZIP_FILE="/tmp/deploy-$(date +%s)-$$.zip"Analysis:
Recommendation: Consider reverting to 2. Hidden file exposure (Security consideration) Removing Considerations:
Recommendation: Consider a more targeted allowlist approach: # Option 1: Explicitly include needed hidden directories
zip -r "$ZIP_FILE" . \
-x "*.git/*" \
-x "node_modules/*" \
-x ".atxp-instance" \
-x "*.DS_Store" \
-x "*/.*" \
-i "*/.atxp/*" \
-i "*/config/.env*" \
> /dev/nullOr add explicit exclusions: # Option 2: Exclude specific sensitive patterns
zip -r "$ZIP_FILE" . \
-x "*.git/*" \
-x "node_modules/*" \
-x ".atxp-instance" \
-x "*.DS_Store" \
-x "*/.npmrc" \
-x "*/.aws/*" \
-x "*/.ssh/*" \
> /dev/null⚡ Performance ConsiderationsNo significant performance impact:
🔒 Security ConcernsCritical consideration: Unintended file exposure The broadening of included files from removing
Mitigation strategies:
🧪 Test CoverageExisting testing (from PR description):
Recommended additional testing:
📋 Test Plan ReviewThe provided test plan is good but could be enhanced: Current test plan:
Suggested additions:
💡 Overall RecommendationApproval with suggestions: This PR solves a real problem and the core fix is correct. However, I recommend addressing the security concern before merging:
The fix is functional, but the security implications of including all hidden files should be carefully considered for a production deployment system. 📝 Suggested Code ChangesOption A: Keep it simple with critical exclusions zip -r "$ZIP_FILE" . \
-x "*.git/*" \
-x "node_modules/*" \
-x ".atxp-instance" \
-x "*.DS_Store" \
-x "*/.env.local" \
-x "*/.env.development" \
-x "*/.env.test" \
-x "*/.npmrc" \
-x "*/.aws/*" \
> /dev/nullOption B: Use mktemp for safety ZIP_FILE=$(mktemp /tmp/deploy-XXXXXX.zip)Would you like me to help implement any of these suggestions? |
Implements PR review feedback to improve security and reliability: 1. Fixed .DS_Store pattern: - Changed from `*.DS_Store` to `.DS_Store` and `*/.DS_Store` - Previous pattern wouldn't match hidden .DS_Store files properly 2. Added security exclusions for sensitive files: - .env.local, .env.development, .env.test (local env files) - .npmrc (npm credentials) - .aws/* (AWS credentials) - Prevents accidental leakage of local secrets in deployments 3. Fixed mktemp usage for better security: - Changed from timestamp+PID to proper mktemp with unique filename - mktemp guarantees uniqueness and prevents race conditions - Remove the temp file before using the .zip extension Testing: - Created test project with 9 files including sensitive data - Verified 6 files included (.atxp/.env.production, config/.env, index.js, src/) - Verified 6 sensitive files excluded (.env.local, .env.development, .npmrc, .aws/credentials, .DS_Store files) - All security exclusions working correctly Addresses feedback from PR review comments.
PR Review: Fix ATXP-654 - Include hidden files in subdirectories during deployOverall AssessmentStatus: ✅ Approved with minor suggestions This is a well-crafted PR that effectively solves a critical deployment issue. The iterative improvements through multiple commits show strong attention to security and code quality. The problem and solution are clearly documented. Code Quality & Best PracticesStrengths ✅
Observations 🔍
Security ConcernsImprovements Made ✅
Potential Issues
|
Implements all "must fix before merge" items from PR review: 1. Fixed .git exclusion pattern: - Changed from `*.git/*` to `.git/*` - Previous pattern would match `foo.git/`, not `.git/` 2. Added nested node_modules exclusion: - Added `*/node_modules/*` pattern - Now excludes node_modules in subdirectories (e.g., packages/foo/node_modules) 3. Improved mktemp security: - Changed from `rm + reuse` to `mv + rm` approach - Creates unique temp file, renames to .zip, then removes empty file - Eliminates security window where file could be hijacked 4. Expanded credential file exclusions: - Added .env.*.local pattern (covers .env.production.local, etc.) - Added .netrc (curl credentials) - Added certificate files: .pem, .key, .p12, .pfx - Comprehensive protection against accidental credential leakage Testing: - Created test project with 18 files including all sensitive file types - Verified 8 files included (app files and necessary config) - Verified 10 sensitive files excluded: * .git/config * node_modules (root and nested) * .env.local, .env.production.local, .env.development * .npmrc, .netrc * .aws/credentials * Certificate files (.pem, .key, .p12, .pfx) * .DS_Store files All security exclusions working correctly. Addresses all feedback from PR review comment #3530020630.
Created comprehensive test script to verify file inclusion/exclusion patterns work correctly. This addresses the need for repeatable testing instead of manual verification. Test Coverage: - 4 inclusion tests (files that should be deployed) - 17 exclusion tests (sensitive files that should NOT be deployed) - Total: 21 automated tests Features: - Self-contained test with automatic setup and cleanup - Colored output for easy reading - Clear pass/fail indicators - Summary report with test counts - Exit code 0 for success, 1 for failure (CI/CD friendly) Tests verify all security exclusions: - Environment files (.env.local, .env.*.local, .env.development, .env.test) - Credentials (.npmrc, .netrc, .aws/*) - Certificates (*.pem, *.key, *.p12, *.pfx) - Git files (.git/*) - Dependencies (node_modules, including nested) - macOS metadata (.DS_Store) - Instance tracking (.atxp-instance) Usage: ./cloud/tests/test-deploy-exclusions.sh Can be integrated into CI/CD pipelines to prevent regressions.
PR Review: Fix ATXP-654 - Include hidden files in subdirectories during deployOverall AssessmentThis is an excellent PR that addresses a critical bug while significantly improving the security posture of the deployment process. The fix is well-documented, thoroughly tested, and includes comprehensive automated testing. The iterative improvements based on review feedback demonstrate attention to detail. Code Quality and Best Practices ✅Strengths:
Suggestions:Minor: Consider grouping patterns for readability # Instead of individual -x lines, could document in comments above groups
# Security - Environment files
-x ".env.local" \
-x ".env.*.local" \
-x "*/.env.local" \
-x "*/.env.*.local" \
-x ".env.development" \
-x "*/.env.development" \
-x ".env.test" \
-x "*/.env.test" \Consider: Add a comment explaining why .env.production is NOT excluded Potential Bugs or Issues
|
Removed unnecessary mv and rm steps. Now uses mktemp directly with .zip
extension appended to the filename.
Before (4 lines):
ZIP_FILE=$(mktemp /tmp/deploy-XXXXXX)
mv "$ZIP_FILE" "${ZIP_FILE}.zip"
ZIP_FILE="${ZIP_FILE}.zip"
rm "$ZIP_FILE"
After (1 line):
ZIP_FILE="$(mktemp /tmp/deploy-XXXXXX).zip"
Benefits:
- Simpler and more readable
- Still uses mktemp for security (unique filenames)
- No intermediate files to clean up
- Same functionality with less code
Updated both deploy.md and test script.
All tests still passing (21/21).
Pull Request Review: Fix ATXP-654 - Include hidden files in subdirectories during deploySummaryThis PR fixes a critical bug where the ✅ Strengths1. Excellent Problem IdentificationThe root cause analysis is clear and accurate. The 2. Comprehensive Test CoverageThe addition of
3. Security ImprovementsThe new exclusion patterns significantly improve security by explicitly excluding:
This is actually a security enhancement beyond just fixing the bug. 4. Good Documentation
🐛 Critical Issue: Certificate/Key Exclusion IncompleteLocation: Issue: The patterns Example Scenario: Fix Required: Add subdirectory patterns in both occurrences (lines 63-66 and 184-187): -x "*.pem" \
-x "*/*.pem" \
-x "*.key" \
-x "*/*.key" \
-x "*.p12" \
-x "*/*.p12" \
-x "*.pfx" \
-x "*/*.pfx"Severity: HIGH - Private keys in subdirectories would be deployed to cloud.atxp.ai. 🎯 RecommendationsMust Fix (Before Merge)
Should Consider
📝 Final VerdictStatus: This PR solves the reported bug and adds significant value through comprehensive testing and improved security. However, there's a critical gap in the certificate/key exclusion patterns that must be fixed before merge. Impact Assessment:
Estimated Effort to Fix: ~10 minutes 🙏 Positive Notes
Great work overall! Just need to close that certificate exclusion gap and this will be ready to merge. |
Addresses security concern from PR review about orphaned temp files.
Previous approach left the mktemp base file as an orphan:
ZIP_FILE="$(mktemp /tmp/deploy-XXXXXX).zip"
# Creates /tmp/deploy-abc123 (unused) and uses /tmp/deploy-abc123.zip
Fixed approach explicitly removes the placeholder:
TEMP_BASE=$(mktemp /tmp/deploy-XXXXXX)
ZIP_FILE="${TEMP_BASE}.zip"
rm "$TEMP_BASE" # Clean up the placeholder
Benefits:
- No orphaned temp files left in /tmp
- Clearer intent - shows we're using mktemp just for unique name generation
- Still secure - maintains unique filename from mktemp
- Matches Option 3 from PR review feedback
All tests still passing (21/21).
Pull Request Review: Fix ATXP-654SummaryThis PR addresses a critical bug where deployed sandboxes were missing configuration files due to overly aggressive file exclusion patterns. The solution removes the problematic Code Quality: ExcellentStrengths
Minor Suggestions
Security: StrongExcellent Security AdditionsThe PR significantly improves security by excluding sensitive files:
Considerations
Performance: Good
Potential Improvement
Potential Bugs: Minor Concerns1. Pattern Matching Edge CasesThe exclusion patterns may not cover all cases. For example, does Recommendation: Document the expected 2. Test Script ReliabilityLine 37 in 3. Race Condition (Very Minor)Lines 38-40 in Test Coverage: ExcellentTest Quality
Test Coverage AnalysisCovers: Environment file patterns, credential files, git files, node modules (root and nested), certificates, macOS metadata, and instance tracking files. Suggestions for Enhanced Testing
Best Practices: Followed
RecommendationsMust Fix (None!)There are no blocking issues. The code is ready to merge. Should Fix (Optional Improvements)
Nice to Have
Final Verdict: APPROVEThis is an excellent PR that:
Recommendation: Approve and merge. The optional improvements can be addressed in follow-up PRs if desired. Additional CommentsThe progression through the commits shows excellent engineering practices. The author iteratively:
This iterative approach with clear commit messages makes the PR easy to review and understand. Great work! |
Added test case for deeply nested .env.local file to verify that the */.env.local pattern correctly excludes files at all depths, not just one level. Test case: - Created deep/nested/directory/.env.local - Verified it's excluded by the */. env.local pattern - Confirms zip's -x patterns match at all depths (unlike shell globs) This addresses the "Minor Issue" from PR review about wildcard pattern coverage. The patterns already work correctly for deep nesting - zip's -x behavior matches paths at any depth with the */ prefix. All tests passing: 22/22 (added 1 new deep nesting test)
Pull Request ReviewSummaryThis PR fixes a critical bug where the deploy command was excluding necessary hidden files in subdirectories (like ✅ Code Quality & Best PracticesStrengths:
Improvements:
🐛 Potential Bugs
⚡ Performance ConsiderationsGood:
Observations:
🔒 Security ConcernsExcellent security improvements:
Remaining concerns:
🧪 Test CoverageExcellent test coverage:
Suggestions:
📝 DocumentationGood:
Improvements:
🎯 Overall AssessmentRating: Good with Important Security Fixes Needed This PR successfully solves the immediate problem of excluding too many hidden files. However, the security improvements are incomplete due to wildcard pattern limitations that only exclude sensitive files in the root directory. Must Fix Before Merge:
Recommended for Follow-up:
Testing Recommendation:Run the test script and verify it passes: cd cloud/tests
./test-deploy-exclusions.shThen create a test with mkdir -p test-dir/nested/certs
echo "PRIVATE" > test-dir/nested/certs/private.key
cd test-dir
# Run deploy zip command
# Check if private.key is excluded (it won't be with current patterns)Great work on identifying and fixing this critical bug! The test coverage is particularly impressive. Once the wildcard patterns are fixed for recursive exclusion, this will be a solid improvement to the deploy command's security. |
Addressed: mktemp Race Condition & Pattern DepthThanks for the detailed review! Both issues have been fixed: ✅ Critical Issue: mktemp Race Condition (FIXED)Implemented Option 3 as suggested: TEMP_BASE=$(mktemp /tmp/deploy-XXXXXX)
ZIP_FILE="${TEMP_BASE}.zip"
rm "$TEMP_BASE" # Remove the temporary placeholderThis approach:
Fixed in commits: a7e3ae9, 92006c0 ✅ Minor Issue: Wildcard Pattern Coverage (VERIFIED)Tested and confirmed that Test Results:
The current patterns already handle deep nesting properly. The All tests passing: 22/22 (including new deep nesting test) Fixed in commit: 2265302 |
Added automated testing workflow that runs on: - Push to main branch - Pull requests to main - Changes to deploy.md, test scripts, or workflow file The workflow: - Runs on ubuntu-latest - Executes all 22 deploy exclusion tests - Reports pass/fail status - Fails the build if any tests fail This ensures deploy command exclusions remain correct and prevents regressions from being merged. Benefits: - Automatic verification on every PR - Catches breaking changes before merge - No manual testing needed for reviews - Clear pass/fail indicators in PR checks
Pull Request ReviewSummaryThis PR fixes a critical bug in the ✅ Strengths1. Problem Identification & Solution
2. Security ImprovementsThe new exclusion patterns significantly enhance security by explicitly blocking sensitive files:
This is a major security win! 🔒 3. Test CoverageOutstanding test implementation (
4. Code Quality
🔍 Issues & Recommendations1. SECURITY CONCERN - Critical: Missing Pattern Depth (Lines 47, 148)Issue: Some exclusion patterns may not work at arbitrary depths. The pattern
Recommendation: Use glob patterns with proper depth or consider using 2. SECURITY CONCERN - Medium: Incomplete .env ExclusionsIssue: The patterns exclude Current behavior:
Questions:
3. TESTING GAP - High: Test Only Validates Patterns, Not Real DeploymentIssue: The test script uses the exact zip command but doesn't verify that extracted files work in actual cloud.atxp.ai sandboxes or that environment variables are properly loaded from Recommendation: Add integration test that deploys to staging and verifies sandbox initialization. 4. CODE QUALITY - Low: Pattern DuplicationIssue: The zip exclusion patterns appear twice (lines 44-68 and 162-186) and are identical. This creates maintenance burden. Recommendation: Extract to a shared function to avoid duplication. 5. BUG - Low: Temporary File Cleanup (Lines 38-40, 162-164)Issue: The pattern creates a temp file, deletes it, then uses the path with Recommendation: Use mktemp directly with suffix: 6. TESTING - Medium: GitHub Actions Workflow Issue (Lines 34-39)Issue: The test results summary checks Fix: Use 📊 Performance ConsiderationsPositive: The exclusion patterns should perform well since Consideration: For large projects, consider progress indicators or compression level optimization. 🧪 Test Plan SuggestionsThe PR includes a test plan checklist. I recommend adding: 🎯 Final RecommendationAPPROVE with minor changes requested This PR successfully fixes the critical bug and adds valuable security improvements. The test coverage is excellent. However, please address:
Risk Assessment: LOW
Impact: HIGH POSITIVE
Great work on thorough testing and security improvements! 🎉 |
|
FYI @napoleond Going to merge in the AM. |
|
@robdimarco-atxp I'm peaceful with this change but I don't think it addresses the issue I was seeing, which was that no files were being extracted into the sandbox correctly I anticipate a bug in the cloud repo |
Summary
Problem
The
/deploycommand was using the exclusion pattern-x "*/.*"which excluded all hidden files in subdirectories, including:.atxp/.env.production- Environment variables for the sandboxconfig/.env- Application configuration filesThis caused deployed sandboxes to initialize with incomplete project files, making them non-functional. The working directory technically existed at
/tmp/workspace, but was missing critical configuration.Solution
Removed the
-x "*/.*"pattern from the zip command in both occurrences (lines 47 and 148) ofcloud/commands/deploy.md. The remaining exclusion patterns are sufficient and specific:.git/*- Git repository filesnode_modules/*- Node dependencies.atxp-instance- Local instance tracking file.DS_Store- macOS metadata filesTesting
Created a test project with:
.atxp/.env.production,config/.env).git/,node_modules/,.DS_Store)Before fix:
.atxp/.env.production❌ EXCLUDEDconfig/.env❌ EXCLUDEDAfter fix:
.atxp/.env.production✅ INCLUDEDconfig/.env✅ INCLUDEDTest plan
.atxp/.env.productionto cloud.atxp.ai.atxp/.env.productionare available in the sandboxFixes https://linear.app/circuitandchisel/issue/ATXP-654