Deploy Multi-Threaded Python Data Script to AWS EC2
8 hours ago
Job description
Job Title:
Deploy Multi-Threaded Python Data Script to AWS EC2 (High-Core Spot Instance, S3 File Handling, Auto-Shutdown) - Fixed Price
Category:
Development and IT DevOps Engineering
Skills:
AWS EC2, Python, Amazon S3, IAM, Spot Instances, Automation, DevOps, Linux, Scripting and Automation, Git, boto3
Operating System:
Ubuntu
Description:
We are a small Toronto-based sustainability research organization looking to migrate a CPU-intensive standalone Python script (CKResolve) from a local Windows machine to AWS EC2 for faster execution on a high-core-count instance.
The script currently takes about 2 hours locally and uses numpy, pandas, openpyxl, xlsxwriter, and matplotlib. It is pure Python (no GPU needed), multi-threaded, and processes Excel and CSV files.
Goal:
Run it much faster (about 15-30 min target) on spot pricing, with easy file upload/download for non-technical users, and automatic shutdown to minimize costs.
Key RequirementsSelect and configure an appropriate EC2 instance (example: c7i family, spot preferred for 70-90 percent savings).
Set up Python 3.12+ environment on Ubuntu or Amazon Linux (install dependencies, handle any Windows-to-Linux tweaks like paths or matplotlib backend).
Implement simple S3-based file handling: non-tech users upload inputs to S3 bucket, script downloads, processes, and uploads outputs.Create a one-command or one-click trigger (example: AWS CLI batch file, SSM Run Command, or simple script) for non-technical team members to start runs.
Ensure the instance auto-shuts down after each run (example: poweroff at script end) to avoid idle costs.Provide brief, clear documentation (Google Doc or Markdown) so our team can run and maintain it independently.
About the Script
No external APIs or GUI dependencies beyond listed libs.
Input/output:
Excel (.xlsx) and CSV files.
May need minor testing/fixes for Linux compatibility (example: , encoding).
One-time setup; occasional updates via S3 or script tweaks.
Ideal Freelancer
Proven experience with AWS EC2, Spot Instances, S3, IAM roles, SSM (or similar).
Strong Python scripting (including boto3 for AWS integration).
Experience building user-friendly automation for non-technical clients.
Able to complete in 1-2 days (quick turnaround preferred).
Budget and Terms
Fixed-price project. Please quote based on your estimate (we expect 8-12 hours; happy to discuss range 300 to 500 USD). Toronto time zone alignment a plus for quick communication.
To Apply
In your proposal, please include:
Brief examples of similar AWS/Python automation projects you have done.
Your estimated hours, timeline, and fixed-price quote.
Any initial questions (example:
about script details or Linux compatibility).
We are excited to get this running efficiently. Looking forward to your proposals
Project type:
One-time project
Experience level:
Expert or Intermediate
Contract duration of less than 1 month.
Mandatory skills:
Python, Amazon Web Services, Amazon EC2, Ubuntu, macOS, Microsoft Windows, Linux, CentOS, Git, Amazon ECS
Similar jobs
DevOps Specialist Needed for AWS Documentation and Scripting
Only for registered members
We are seeking a specialized DevOps professional to create comprehensive technical documentation and scripts for migrating our AWS instance. · ...
3 weeks ago
Python Programmer Needed for Raspberry Pi MQTT Script
Only for registered members
We are seeking a skilled Python programmer to develop a script for a Raspberry Pi device. · ...
1 month ago
AWS Designer for API, Lambda, and DynamoDB Integration with Second Life
Only for registered members
We are seeking an experienced AWS Designer to integrate API, AWS Lambda and DynamoDB to connect our application with Second Life using LSL scripting. · ...
1 month ago
We are looking for an experienced Python developer to create a demo application that integrates with AWS services. The project aims to showcase the capabilities of AWS using Python, so familiarity with AWS SDKs and cloud architecture is essential. · ...
1 month ago
AWS Infrastructure Architect in London required to have strong expertise in Terraform and AWS CloudFormation with hands-on experience with AWS services. · Strong expertise in Terraform and AWS CloudFormation. · ...
1 month ago
Terraform and AWS CloudFormation expertise. · Hands-on experience with AWS services, networking and security. · ...
1 month ago
We are looking for an ambitious and proactive Junior AWS DevOps Engineer to join our team. The ideal candidate will have foundational knowledge of DevOps principles, a strong interest in Amazon Web Services (AWS), and either relevant cloud certification or practical experience in ...
1 week ago
Necesito crear un script en Pine para Tradingview que integre con mi bot de trading automático. · Pine Script con indicadores y estrategia personalizados. · ...
3 weeks ago
We are looking for an ambitious and proactive Junior AWS DevOps Engineer to join our team. Assist in the provisioning, management, and maintenance of core AWS services (EC2, S3, VPC, RDS, IAM) following established best practices. · Contribute to the development, configuration, ...
2 weeks ago
We have pentaho script running in aws batch. Basically this job comparing the dataload from Postgres and snowflake. · Now the 1st requirement is, currently we are using basic username and password to access snowflake. But it's need to changed with okta integration. The okta integ ...
2 days ago
Python Developer Needed for High-Speed Encoder Monitoring Script
Only for registered members
We are looking for an experienced Python developer to create a script that monitors an encoder using a high-speed counter. · ...
1 month ago
Amazon SP-API Developer Needed to Automate Daily FBA Inventory Report Export
Only for registered members
I am looking for an experienced Amazon SP-API developer who can set up an automated workflow to download the Manage FBA Inventory report in CSV format daily. · ...
1 month ago
Objective · We are seeking an experienced Kubernetes and cloud-native architecture consultant to conduct an independent, hands-on review of our containerized deployment architecture and documentation. The objective is to test our containerized applications to ensure that our solu ...
2 days ago
AWS | Linux | RHEL | Bash | Scripting | MedTech | Amazon Web Services | Docker | Kubernetes | Ansible | Helm | Support | Ticketing · Want to be involved in a very genuine Tech4Good? · Do you want to further your exposure to AWS, Cloud and modern SaaS technologies? · Looking for s ...
5 hours ago
Google Sheets Automation: Daily Import from AWS SFTP Feed
Only for registered members
We have an inventory feed hosted on AWS SFTP (AWS Transfer Family). We need a simple, reliable automation that pulls the latest file once per day and updates a Google Sheet automatically. · ...
1 month ago
Job summary: · Design and automate the deployment of Cloudera CDP components using Terraform. · Collaborate with data engineering teams to integrate CDP with existing Big Data workflows. · Migrating the cluster to new Cloudera cloud platform · ...
1 month ago
PowerShell Diagnostic Script for Windows Service S3 Upload Failures
Only for registered members
We need a PowerShell diagnostic script to identify the failure point of a Windows service that collects logs locally and uploads them to S3 via presigned URLs. · Work on PowerShell 5.1 (no PS7 features) · Be read-only and safe for customers to run · Output a report file to Deskto ...
2 weeks ago
We are looking for a skilled developer with expertise in Python Terraform AWS services to help us implement and manage serverless applications. · writing Python scripts for AWS Lambda functions · setting up infrastructure using Terraform · ...
2 weeks ago
AWS DevOps Specialist required to support build out of AWS infrastructure at a leading systematic hedge fund. · ...
1 month ago
AWS Infrastructure Design and Implementation with Terraform
Only for registered members
+We are seeking an experienced cloud engineer to design and implement a multi-environment AWS infrastructure using Terraform. + · +Your responsibilities will include setting up VPCs, subnets, security groups, and other AWS components across multiple environments.A successful proj ...
2 weeks ago