Skip to content

nicitaacom/AWS-scraper

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

1,105 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

🧱 Infrastructure Setup (Coolify + Layer7)

This project is deployed on a VPS (8 vCPU / 24 vGB RAM) using:

  • 🧊 Coolify: a self-hosted PaaS (like Heroku UI) to manage apps, Docker, environment, and volumes — no terminal needed
  • 🌐 Layer7: DNS + SSL + proxy management (acts like a user-friendly Traefik/Cloudflare layer)

You can look at it's like VPS + hesiaCP or cPanel

📦 VPS Specs

  • CPU: 8 vCPU
  • RAM: 24 vGB
  • OS: Debian 6.1 (kernel 6.1.0-13-amd64)
  • DNS Provider: Layer7 (reverse proxy / dashboard access via vps.yourdomain.com)

🚀 Coolify Setup

Coolify gives you a full UI-based experience to deploy and manage containers without typing terminal commands.

To install Coolify:

curl -fsSL https://cdn.coollabs.io/coolify/install.sh | bash
docker start coolify-proxy

Then visit:

https://vps.yourdomain.com

If it's not working you need to login (193.24... it's a VPS IP)

http://193.24.210.188:8000/

This opens the Coolify dashboard (secured by Layer7)
→ From there you can deploy frontend/backend containers with Docker, set up environment variables, auto-restart, persistent volumes, and more.


Scraper setup on coolify

Note: this is for users who completed outreach-tool setup
If you not done this yet it's nothing for you to do here...
DM me on linkedin: https://www.linkedin.com/in/nicitaacom/
And I help you setup outreach-tool (only if you have difficuties)

  1. Create a new project

  2. Click button "+new" near "Resources"

  3. Click "Public Repository" then paste https://github.com/nicitaacom/AWS-scraper

  4. In order to avoid conflicts with existing projects e.g main + backend + support + auth etc I recommend port 3011
    port-3011

  5. Click "Continue"

  6. Update "Domains" to your domain https://scraper.your-domain.com - sub-domain should be "scraper"

  7. Update Base Directory to /backend

  8. Build Pack is Dockerfile

  9. Go in "Git source" -> Change Branch to "production" prod-branch

  10. Click environment variables then Developer view Paste

FRONTEND_URL='https://www.outreach-tool.com'
BACKEND_URL='https://scraper.your-domain.com'

BACKEND_PORT='3011'
FRONTEND_PORT='3000'

# just keep this - I decided to simplify it and give credentials to everyone intentionaly
PUSHER_APP_ID='2001834'
NEXT_PUBLIC_PUSHER_APP_KEY='f70f62d28edcae738bac'
PUSHER_SECRET='8c204d0ed22307ed0ad9'


# separaed supabase account
# scraper@outreach-tool.com
# Fk3_f2kalfAF
NEXT_PUBLIC_SUPABASE_URL=''
NEXT_PUBLIC_SUPABASE_ANON_KEY=''
# DB password: 

# telegram channel (for notifications about high load and re-deployments )
TELEGRAM_BOT_TOKEN=''
TELEGRAM_CHAT_ID=''

# your AWS creds
ACCESS_KEY_ID=''
SECRET_ACCESS_KEY=''
REGION='eu-central-1'
S3_BUCKET='scraper-files-eu-central-1'

# copy paste user id from settings (for live-progress with pusher)
USER_ID=''

GLOBAL_MAX_CONCURRENT_JOBS='5'
MAX_CONCURRENT_JOBS='5'
MAX_JOBS='2000'
MAX_LOCATIONS_PER_GROUP='10'

# for (your VPS/domain name)
OPENAI_KEY=''

# separated upstash account for VPS name
# login: 
# password: 
UPSTASH_REDIS_REST_URL=''
UPSTASH_REDIS_REST_TOKEN=''
UPSTASH_REDIS_URL=''
  1. Refer to very end of this guide: https://gist.github.com/nicitaacom/24245a4c6281aa120411d6c9bbc89edf/
    Section called "Optional - scraper"
    Complete setup there

  2. Click "Deploy" (if fails debug with AI - then if still fails - DM me on linkedin)

  3. Deployment Setup 12.1. Fork this repo 12.2. In your fork: Settings → Secrets → Actions 12.3. COpy COOLIFY_WEBHOOK_URL = copy-here 12.4. Paste COOLIFY_WEBHOOK_URL = paste-here 12.5. Copy COOLIFY_WEBHOOK_TOKEN = copy-here 12.6. Paste COOLIFY_WEBHOOK_TOKEN = paste-here

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors