Sr. Big Data Analyst / Developer

About the role

The scope of the project is to build an enterprise data warehouse to integrate property, lease, unit and other important data related to ProLogis globally and analyze it to understand the overall performance of the organization and to improve the same. It involves extracting data from various source systems and storing it in a single data store after applying required business transformations.

Responsibilities

• Develop batch and streaming analytics pipelines using Apache Spark and Scala programming to materialize and refine the event results and improve its processing time
• Write advanced Hive, Spark SQL queries against data warehouses utilizing large datasets
• Built object-oriented and functional Scala programs with Apache Spark clean and transform features that performs aggregations to integrate to machine learning models
• Ensure that all feature deliverables meet quality objectives in functionality, methodologies, performance, stability, security, accessibility and data quality
• Design and develop advanced analytics software components using Apache Spark, Scala, AWS and Hadoop Ecosystem and integrate it with various machine learning models that are architected for re-use
• Create complex data extracts, applications and ad-hoc queries as requested by internal and external customers using Hive and Scala transformations
• Develop Hive scripts and EMR (Elastic Map Reduce) mappings using Amazon Web Services EMR to integrate data from different sources and apply required transformations before loading to target databases
• Manage the implementation of statistical data quality procedures on new data sources by applying rigorous iterative data analytics to improve scaling and system performance
• Develop ETL code in various databases like Snowflake Computing and Denodo
• Work on performance tuning and resource optimization of databases like SQL Server to improve the overall performance of the system
• Develop and maintain data engineering best practices and contribute to Insights on data analytics and visualization concepts, methods and techniques to make the system robust and compatible to adhere the changes in data requirements in future
• Work closely with other Data Scientists, analysts, designers and project teams to identify new areas of research
• Analyze the production data issues to find the root cause and provide permanent fix to avoid similar issues in future
• Create and maintain Big Data Analytic pipeline engine specifications and process documentation to produce the required data deliverables (data transformation/actions, data profiling, source to target maps, ETL flows)
• Collaborate with business analysts and stakeholders to understand the business requirements and ensure the quality of data
• Create solutions to real world problems and verify the results of the work deployed
• Prepare project and system related documentation consistent with standards and procedures outlined in the data architecture approach including data quality, security and availability requirements
• Work with DBA’s and QA team to move the code developed to production environment using Git version control and Bitbucket
• Work with cross-functional teams, such as Product Management, Platform, Operations and Mobile Device Engineering to elaborate on functional requirements/designs of the mobile wireless analytics modules, based on the product level requirements and designs
• Use Jira or Confluence for defect tracking and documentation
• Implement unit test cases for various modules and coordinate with QA team for production deployments/releases
• Participate in Daily Sprint Meetings; Coordinate with other team members in understanding the requirements and provide best analysis for continuing with the development

Requirements

Master's degree in Computer Science Engineering, Information Technology, Computer Applications, any other related field.

Qualifications

No specific qualifications mentioned beyond the educational requirement.

Benefits

No specific benefits mentioned.

Schedule

No specific schedule mentioned.

Pay

No specific pay mentioned.