Software Engineer (Ray Data)
About Anyscale
At Anyscale, we're on a mission to democratize distributed computing and make it accessible to software developers of all skill levels. We’re commercializing Ray, a popular open-source project that's creating an ecosystem of libraries for scalable machine learning. Companies like OpenAI, Uber, Spotify, Instacart, Cruise, and many more, have Ray in their tech stacks to accelerate the progress of AI applications out into the real world. With Anyscale, we’re building the best place to run Ray, so that any developer or data scientist can scale an ML application from their laptop to the cluster without needing to be a distributed systems expert. Proud to be backed by Andreessen Horowitz, NEA, and Addition with $250+ million raised to date.
About Ray Data Team
Ray Data is a Python-native data processing engine that is a one-stop shop for all AI data processing needs. Ray Data provides performant, first-class integration with cutting-edge AI frameworks using both multi-modal and structured data. The Ray Data team currently develops and maintains Ray Data. We are a team of engineers passionate about building a data processing engine that serves as a one-stop shop for all ML/AI needs.
Responsibilities
- Improve the performance of Ray Data and multi-modal batch inference use cases.
- Ensure efficient scaling across different stages of the data pipeline in a heterogeneous environment.
- Build data loading solutions for production training workloads.
- Focus on stability and fault tolerance at high scale.
- Work with customers and new-age AI-native companies in scaling their AI workloads.
Requirements
- At least 3-4 years of relevant work experience.
- Solid background in building scalable and fault-tolerant distributed systems.
- Experience with data processing, database internals.
- Passionate about large-scale systems and performance for AI.