ETL/DB/Data Extraction Engineer

We are seeking an exceptionally skilled Senior Data Engineer with extensive experience in ETL development and bulk data extraction through web scraping and crawling technologies. The ideal candidate will have a solid background in building, optimizing, and maintaining ETL pipelines, with strong expertise in data management and cloud technologies. This is a challenging role designed for a detail-oriented professional who is passionate about delivering high-quality, scalable data solutions.

Responsibilities

• Develop, optimize, and troubleshoot ETL pipelines across production, QA, and UAT environments.
• Handle large datasets and manage server infrastructure effectively.
• Implement advanced web scraping techniques for rapid and efficient data extraction from large-scale sources across hundreds of websites.
• Apply techniques such as parallelism, concurrency, and threading to improve data extraction and processing efficiency.
• Utilize tools such as requests, xpath, ActionChain, Urllib, Numpy, Pandas, and lxml.
• Work with frameworks like Selenium and Scrapy to build scalable scraping solutions.
• Manage and optimize both relational databases (Postgres, MySQL) and NoSQL databases, including schemas, triggers, and queries.
• Implement cloud best practices for security, scalability, and performance using AWS, GCP, or Azure, including services such as Lambda, EC2, and S3.
• Integrate external APIs and develop REST APIs.
• Participate in data warehousing, architecture, and ETL pipeline development at an enterprise scale.
• Create and optimize SQL procedures and data pipelines for seamless integration.
• Work cross-functionally with teams to deliver end-to-end data solutions.
• Ensure source data quality, develop mapping and transformations, and validate jobs across various environments.

Requirements

• Proficiency in ETL pipeline development and optimization for large datasets.
• Experience with bulk data extraction through web scraping and crawling technologies.
• Strong understanding of parallelism, concurrency, and threading.
• Expertise in Python-based libraries like Numpy, Pandas, Selenium, and Scrapy.
• Proficiency in Postgres, MySQL, and NoSQL databases.
• Experience with AWS, GCP, or Azure, including hands-on knowledge of AWS Lambda, EC2, and S3.
• Experience developing and integrating REST APIs.
• Skilled in SQL procedures, query optimization, and pipeline creation.
• Familiarity with code versioning tools like Git and SVN.
• Solid understanding of the Software Development Life Cycle (SDLC), testing methodologies, and CI/CD processes.
• Exceptional analytical and problem-solving abilities with a focus on delivering scalable data solutions.

Qualifications

• Minimum 4-5 years of experience in ETL development, including experience with bulk data extraction through web scraping and crawling technologies.
• Proven track record of delivering large-scale data engineering solutions.
• Prior experience with cloud environments like AWS, GCP, or Azure is highly preferred.

Benefits

• Opportunity to work on high-impact, large-scale data projects.
• Collaborative, growth-oriented environment where innovation is encouraged.
• Competitive compensation and benefits package.