Build N8N server on AWS and automate tasks including scraping and email
1 day ago
Job description
Project OverviewWe currently use + Apify to scrape product data. While functional, the process is too slow for large datasets.
We need to move to a faster, scalable architecture built in N8N, capable of efficiently scraping:
-5,000 – 20,000 SKUs per batch
- Large datasets with parallelized execution
- Modular architecture for future scraper expansion
We believe the solution may require:
- Distributed workflows
- Multiple concurrent scraping actors/workers
- Queue based batching system
- Rate limiting + proxy handling
- Smart retry logic
- We are open to the best architectural recommendation from you.
Phase 1:
Infrastructure
- Deploy and configure N8N on our AWS environment
- Recommend AWS architecture (EC2, ECS, Docker, scaling strategy)
- Implement proper logging and error tracking
- Ensure system stability and scalability
Phase 2:
High-Performance Scraper (Initial Use Case)
- Build first large scale scraping workflow
- Accept large SKU list input (5k–20k SKUs)
- Parallelize requests efficiently
Implement:
- Proxy support (if required)
- Rotation logic
- Retry/backoff handling
- Captcha/anti bot mitigation strategy
- Structured output (JSON/CSV/database ready)
- Success Criteria
- Scraping 5,000+ SKUs reasonably fast
- Scalable architecture for future scrapers
- Clean, modular N8N workflows
- Clear documentation for internal team
Strong experience with:
- N8N (required)
- AWS infrastructure (EC2, ECS, Docker)
- High scale scraping architectures
- Proxy rotation systems
- Anti bot mitigation strategies
- Experience building distributed or parallelized workflows
- Understands rate limiting, queues, and fault tolerance
- Can recommend performance optimized solutions
- This is the first of multiple scraping projects
- Long term work available for the right candidate
- Must be comfortable working with large datasets
- Must be able to think architecturally, not just build a simple scraper
Please include:
- Examples of scraping systems you've built
- Scale handled (number of records per run)
- N8N experience (live examples preferred)
- Suggested high level architecture for this project
Contract duration of 1 to 3 months. with 30 hours per week.
Mandatory skills:
Automation, Data Scraping
Similar jobs
We are seeking a skilled freelancer to scrape emails from the website. · The ideal candidate will have experience in web scraping and data extraction, ensuring accurate and efficient collection of email addresses. · ...
1 month ago
We're looking for a reliable freelancer to research and compile a targeted contact list of music blogs. · Research and compile at least 300 music blogs across specific regions. · ...
4 weeks ago
I have a list of P&C insurance agencies based in Texas. I need to find their website, key contacts of owners / CEOs / Principals, their emails, their phone numbers, and their addresses It should be relatively straightforward data scraping · ...
3 weeks ago
I need someone to help me scrape the internet to find emails for all of the NYC restaurants I have in an excel file. · ...
6 days ago
We are seeking a skilled freelancer to assist us with scraping email addresses from a specified directory. · Data Scraping · Data Mining · Data Entry · ...
1 month ago
I'm looking for a detail-oriented researcher to build a Google Sheet with company names and contact persons for the first 100 results on Google for the search query: behang. · ...
1 month ago
Data Enrichment Specialist for Email Scraping
Only for registered members
We are seeking a Data Enrichment Specialist to gather verified business contact emails and names from a list of 3,110 domains. Emails should be added to a provided spreadsheet. · ...
1 month ago
We are seeking a skilled freelancer to scrape verified email addresses of real estate agents. This information will be used for bulk outreach campaigns to promote our services.The ideal candidate should have experience in web scraping and data collection. · Attention to detail an ...
1 month ago
We need someone to find provide around 400 email addresses for recruiters/HR representitives/etc. · ...
1 month ago
We are looking for a Backlink Scraping Specialist to help us identify and extract relevant SaaS and real business websites that already link out or are link-relevant. The ideal candidate will have strong experience scraping backlinks/prospects from competitor backlink profiles, S ...
1 month ago
I am seeking a detail-oriented freelancer to help me scrape my email inbox for receipts and invoices. · ...
1 week ago
Email Address Scraping for Medical Businesses
Only for registered members
I have a CPR company with 10 locations, 7 in Tennessee and 3 in Georgia, and I need someone to scrape medical colleges... · ...
1 month ago
Email Scraping for Real Estate Companies in Nigeria
Only for registered members
I need someone to scrape business emails of real estate companies based in Nigeria. · ...
3 weeks ago
Email Scraping and Data Collection Expert Needed
Only for registered members
We are seeking a detail-oriented freelancer to assist with email scraping and data collection. · Data Scraping · Data Mining · Data Entry · ...
2 weeks ago
We are looking for a skilled web scraper to collect realtor email addresses from various online sources across five specific states. The goal is to compile a comprehensive list to support our cold email campaign. · ...
1 week ago
Python Developer for Email Archive Data Scraping
Only for registered members
We need you to scrape each URL and extract the 3 main article blocks from each email. · I will provide:the CSV of campaign URLs (+ campaign IDs / sent dates) · 5–10 example URLs to calibrate the extraction · clarification on what counts as the "3 main articles" vs other links (so ...
2 weeks ago
Email and Data Scraping for UGC TikTok Campaign
Only for registered members
I am looking for a skilled freelancer to assist with email and data scraping for an upcoming Gen Z User-Generated Content (UGC) TikTok campaign. · ...
1 month ago
Email Scraping and Verification for Automotive Repair Shops
Only for registered members
We are seeking a detail-oriented freelancer to scrape email addresses from automotive repair shops. · Data Scraping · Lead Generation · ...
2 weeks ago
Email and Data Scraping for UGC TikTok Campaign
Only for registered members
I am looking for a skilled freelancer to assist with email and data scraping for an upcoming User-Generated Content (UGC) TikTok campaign. The ideal candidate should have experience in data scraping techniques and tools, ensuring high accuracy and compliance with data protection ...
1 month ago
Email scraping for UGC Creators and Micro Influencers
Only for registered members
Necesitamos un especialista en Generación de Leads para ayudarnos a crear grandes listados organizados de creadores de contenido (micro e influyentes) para enviar correos electrónicos. · ...
1 month ago
Data Scraping and Automated Email Outreach for Romanian Doctors
Only for registered members
We are seeking a skilled freelancer to assist in gathering a comprehensive list of doctors based in Romania through data scraping techniques. · Data Scraping · Web Automation · Email Marketing · ...
3 weeks ago