Software Engineer III

Responsibilities

• Spider Development: Design and deploy robust, distributed spiders and crawlers to extract data from a variety of web architectures (SPAs, SSR, etc.).
  • Reverse-engineer web applications to identify data points.
  • Implement evasion techniques to bypass IP rate-limiting and bot detection.
• Bot Evasion Engineering: Research and implement strategies to bypass anti-scraping technologies, including proxy rotation, browser fingerprinting, and CAPTCHA solving.
  • Utilize asynchronous Python programming to maximize the throughput and efficiency of data collection pipelines.
• Database Management: Create and optimize SQL schemas for large-scale data storage and perform complex data transformations and validation.
  • Manage relational databases like PostgreSQL or MySQL.
• System Maintenance: Proactively monitor the health of extraction agents and refactor code quickly in response to target website updates or layout changes.
  • Collaborate with data analysts to define and validate data requirements and output formats.

Requirements

• Advanced Python: Mastery of Python 3.x with deep experience in extraction frameworks (Scrapy, Playwright, Selenium, or Puppeteer).
• Technical Resilience: Proven ability to bypass high-level bot detection (e.g., Cloudflare, Akamai, or PerimeterX).
• Database Mastery: Expert-level SQL skills and experience managing relational databases like PostgreSQL or MySQL.
• Network Proficiency: Expert understanding of HTTP/S, TCP/IP, TLS fingerprinting, and browser-header manipulation.
• Problem Solving: A specialized ability to reverse-engineer JavaScript-heavy websites and hidden API endpoints.
• Able to write, debug, and deploy complex Python code in a distributed environment.
• Able to analyze and interpret complex web structures and network traffic using browser developer tools.
• Able to design and maintain relational database tables containing millions of rows.
• Able to pivot and respond quickly to technical "break-fixes" to ensure data continuity for the business.

Education & Experience

• Bachelor’s degree in Computer Science, Information Systems, or a related field (or equivalent professional experience).
• Minimum of 5+ years of experience in Software Engineering, with at least 2–3 years focused specifically on large-scale web scraping or data extraction.

Benefits

• Health Benefits: Babel Street covers 85-100% monthly premium costs for Medical, Dental, Vision, Life & Disability insurances – for you and your family!
• Roth 401(K): Babel Street offers both a Traditional and Roth 401(K) with a very competitive match.
• Unlimited Flexible Leave: We trust our employees to manage their own time and balance their personal and work lives.
• Holidays: Babel Street provides employees with 12 paid Federal Holidays
• Tuition Reimbursement: We are committed to investing in our employees. One way we do that is with our Tuition Reimbursement Program for continuing education.

Pay Range

$110,000 - $135,000 USD