Skip to content

Normboy1/Cloudlink

BranchesTags

Repository files navigation

🌐 CloudLink - Distributed Cloud Platform

CloudLink is a comprehensive distributed cloud platform featuring two powerful systems:

  1. CloudNode - Enterprise-grade distributed storage with Raft consensus (Go)
  2. CloudLink-mini - AWS-powered personal cloud mesh network (Python)

📦 Projects Overview

🔷 CloudNode - Distributed Storage System

Enterprise-grade distributed storage with automatic leader election and fault tolerance.

Key Features:

  • 🗳️ Raft Consensus - Automatic leader election, prevents split-brain
  • 🔄 Fault Tolerance - Seamless coordinator failover
  • 📊 Real-time Monitoring - Web dashboard with cluster health metrics
  • ⚖️ Load Balancing - Intelligent request routing
  • 🔒 Security - Ed25519 signatures, TLS encryption
  • 💾 Persistent Storage - SQLite backend with automatic backups
  • 🎯 High Performance - Concurrent operations, connection pooling
  • 🐳 Easy Deployment - Docker, systemd, cloud-ready

Tech Stack: Go, Raft, WebSocket, SQLite, Next.js

🔶 CloudLink-mini - Personal Cloud Mesh

Turn any computer into a cloud node with P2P mesh networking and AWS control plane.

Key Features:

  • ☁️ AWS Control Plane - API Gateway, Lambda, DynamoDB, S3
  • 🕸️ P2P Mesh Network - WebRTC data channels with NAT traversal
  • 📊 Live Metrics - CPU, RAM, disk, bandwidth monitoring
  • 🎯 Task Execution - SHA256, image resize, custom jobs
  • 🧠 Smart Routing - Lowest RTT neighbor selection
  • 📱 Live Dashboard - React-based real-time monitoring
  • 🔐 Secure - Pre-signed URLs, WebRTC encryption

Tech Stack: Python, AWS SAM, WebRTC, React, Vite, TypeScript

📁 Repository Structure

CloudLink/
├── agent/                     # CloudNode: Core Go agent
│   ├── main.go               # Dual coordinator/storage roles
│   ├── election/             # Raft consensus
│   ├── coordinator/          # Coordinator service
│   └── connect/              # WebSocket client
├── client/                    # CloudNode: Go client CLI
├── backend/                   # CloudNode: Legacy Node.js coordinator
├── dashboard/                 # CloudNode: Next.js monitoring
├── deployment/                # CloudNode: Production deployment
├── monitoring/                # CloudNode: Health checks
├── loadbalancer/              # CloudNode: Load balancer
├── storage/                   # CloudNode: SQLite backend
├── testing/                   # CloudNode: Test suite
└── cloudlink-mini/            # CloudLink-mini: Complete AWS mesh system
    ├── agent/                 # Python agent with WebRTC
    ├── backend/               # AWS SAM Lambda functions
    ├── dashboard/             # React dashboard
    └── examples/              # Usage examples

⚡ Quick Start

🔷 CloudNode (Distributed Storage)

1. Build Agent

cd agent/
# macOS
./build_mac.sh

# Linux  
./build_linux.sh

# Windows
powershell .\build_win.ps1

2. Single Node (Development)

# Start with default config (becomes leader automatically)
./cloudnode-agent-macos

3. Multi-Node Cluster (Production)

# Node 1 (Bootstrap)
cat > agent_config.json <<EOF
{
  "enable_election": true,
  "node_id": "node1",
  "raft_bind_addr": "192.168.1.10:7000",
  "peers": [],
  "coordinator_port": "8080"
}
EOF
./cloudnode-agent-linux

# Node 2 (Join cluster)
cat > agent_config.json <<EOF
{
  "enable_election": true,
  "node_id": "node2", 
  "raft_bind_addr": "192.168.1.11:7000",
  "peers": ["192.168.1.10:7000"],
  "coordinator_port": "8080"
}
EOF
./cloudnode-agent-linux

🔶 CloudLink-mini (AWS Mesh Network)

1. Deploy Backend

cd cloudlink-mini/backend
sam build
sam deploy --guided
# Note the API endpoint URL

2. Run Agent

cd cloudlink-mini/agent
pip install -r requirements.txt
export API_URL="https://xxx.execute-api.region.amazonaws.com/prod"
python agent.py

3. Launch Dashboard

cd cloudlink-mini/dashboard
npm install
npm run dev
# Open http://localhost:5173

🖥️ Monitoring Dashboards

CloudNode Dashboard

Access at http://localhost:3000:

cd dashboard/
npm install
npm run dev

Features:

  • 📊 Real-time cluster status
  • 🖥️ Node health monitoring
  • 💾 Storage metrics and usage
  • ⚡ Performance analytics
  • 📝 Live activity logs

CloudLink-mini Dashboard

Access at http://localhost:5173:

cd cloudlink-mini/dashboard/
npm install
npm run dev

Features:

  • 🕸️ P2P mesh topology visualization
  • 📊 Real-time node metrics
  • 🎯 Task queue and execution status
  • 🔗 WebRTC connection status
  • 📈 Performance graphs

🔧 Production Deployment

CloudNode Deployment

Automated

sudo ./deployment/deploy.sh

Docker

docker build -t cloudnode:latest .
docker run -d \
  --name cloudnode-node1 \
  -p 8080:8080 \
  -p 7000:7000 \
  -v /opt/cloudnode:/data \
  cloudnode:latest

CloudLink-mini Deployment

AWS SAM

cd cloudlink-mini/backend
sam build
sam deploy --guided

Dashboard (Vercel)

cd cloudlink-mini/dashboard
npm run build
vercel deploy

See cloudlink-mini/DEPLOYMENT_CHECKLIST.md for detailed AWS setup.

🧪 Testing

CloudNode Tests

# Comprehensive test suite
python3 testing/cluster_test.py localhost:8080

# Multi-node cluster
python3 testing/cluster_test.py \
  192.168.1.10:8080 \
  192.168.1.11:8080 \
  192.168.1.12:8080

# Manual testing
curl http://localhost:8080/cluster/status
curl http://localhost:8080/hosts
python3 test_push.py

CloudLink-mini Tests

cd cloudlink-mini

# Test workflow
./test-workflow.sh

# Run multiple agents
./run-multiple-agents.sh

# Submit test task
python examples/submit-task.py

⚙️ Configuration

CloudNode Configuration

Core Settings (agent_config.json):

{
  "enable_election": true,
  "node_id": "unique-node-name",
  "raft_bind_addr": "0.0.0.0:7000",
  "raft_data_dir": "/opt/cloudnode/raft",
  "peers": ["node2:7000", "node3:7000"],
  "coordinator_port": "8080",
  "enable_storage": true,
  "chunk_dir": "/opt/cloudnode/chunks"
}

See deployment/production_config.json for full production settings.

CloudLink-mini Configuration

Agent Settings (.env):

API_URL=https://xxx.execute-api.region.amazonaws.com/prod
API_KEY=your-api-key
NODE_NAME=my-node

Backend Settings (backend/samconfig.toml):

[default.deploy.parameters]
stack_name = "cloudlink-mini"
region = "us-east-1"

See cloudlink-mini/.env.example for all options.

🔒 Security

CloudNode Security

TLS Configuration:

openssl req -x509 -newkey rsa:4096 -keyout server.key -out server.crt -days 365 -nodes

Firewall:

sudo ufw allow 7000:7002/tcp comment "CloudNode Raft"
sudo ufw allow 8080/tcp comment "CloudNode HTTP"

CloudLink-mini Security

  • ✅ API Gateway with API keys
  • ✅ Pre-signed S3 URLs (5-min expiry)
  • ✅ WebRTC DTLS/SRTP encryption
  • ✅ Optional: Ed25519 node signatures
  • ✅ Optional: Cognito user pool

📊 API Reference

CloudNode API

Cluster Management:

  • GET /cluster/status - Cluster health and leadership
  • GET /cluster/leader - Current leader information
  • GET /hosts - Available storage hosts
  • GET /metrics - Performance metrics

File Operations:

  • POST /push_chunk_direct - Upload file chunk
  • GET /files/{id} - Download file metadata
  • DELETE /files/{id} - Delete file

CloudLink-mini API

Node Management:

  • POST /register - Register new node
  • POST /heartbeat - Update node metrics
  • GET /list-nodes - Get all nodes
  • POST /get-peers - Get neighbor nodes

Task Management:

  • POST /create-task - Create new task
  • POST /request-task - Request task assignment
  • POST /complete-task - Mark task complete
  • GET /list-tasks - Get all tasks

WebRTC Signaling:

  • POST /signal-offer - Send WebRTC offer
  • POST /signal-answer - Send WebRTC answer
  • GET /signal-inbox - Get pending signals

🔧 Troubleshooting

CloudNode Issues

Split Brain Prevention:

# Always use odd number of nodes (3, 5, 7)
journalctl -u cloudnode-agent | grep -i raft

Network Connectivity:

telnet node2 7000
sudo ufw status

Storage Issues:

df -h /opt/cloudnode
ls -la /opt/cloudnode/chunks

CloudLink-mini Issues

Agent Won't Connect:

# Check API URL
echo $API_URL

# Test endpoint
curl $API_URL/list-nodes

# Check logs
python agent.py --verbose

WebRTC Connection Fails:

# Check STUN server
ping stun.l.google.com

# Verify NAT type
# Use TURN server if behind symmetric NAT

Task Execution Errors:

# Check task logs in agent output
# Verify S3 permissions
# Check pre-signed URL expiry

📈 Scaling

CloudNode Scaling

Horizontal:

  • Add nodes by including existing peers in configuration
  • Cluster automatically rebalances data
  • Zero-downtime scaling

Vertical:

  • Increase memory/CPU on existing nodes
  • Adjust connection limits and buffer sizes

CloudLink-mini Scaling

Nodes:

  • Unlimited nodes (DynamoDB auto-scales)
  • Each node maintains K=3 neighbors
  • Automatic peer discovery

Tasks:

  • Queue via SQS for FIFO (optional)
  • S3 standard or Glacier for storage
  • Lambda concurrent execution limits

📚 Documentation

CloudNode

  • ARCHITECTURE.md - System architecture overview
  • PROJECT_SUMMARY.md - Project statistics and achievements
  • DEPLOYMENT_STATUS.md - Deployment guide
  • deployment/production_config.json - Production configuration

CloudLink-mini

  • cloudlink-mini/README.md - Detailed project documentation
  • cloudlink-mini/ARCHITECTURE.md - Architecture deep dive
  • cloudlink-mini/QUICKSTART.md - Quick start guide
  • cloudlink-mini/DEPLOYMENT_CHECKLIST.md - AWS deployment steps
  • cloudlink-mini/SHOWCASE.md - Feature showcase
  • cloudlink-mini/WHAT_WE_BUILT.md - Build journey

🎯 Use Cases

CloudNode

  • Enterprise file storage
  • Distributed databases
  • Content delivery networks
  • Backup and archival systems
  • Multi-datacenter replication

CloudLink-mini

  • Personal cloud storage
  • Distributed computing tasks
  • P2P file sharing
  • Edge computing
  • Home lab clusters

🤝 Contributing

  1. Fork the repository
  2. Create feature branch (git checkout -b feature/amazing-feature)
  3. Commit changes (git commit -m 'Add amazing feature')
  4. Push to branch (git push origin feature/amazing-feature)
  5. Open Pull Request

See cloudlink-mini/CONTRIBUTING.md for detailed guidelines.

📄 License

MIT License - see the LICENSE file for details.

🙏 Acknowledgments

CloudNode

CloudLink-mini

CloudLink - Distributed cloud platform for the modern era. 🌐

Built with ❤️ for enterprise reliability and personal cloud freedom.

About

No description, website, or topics provided.

Resources

Readme

License

MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages