Senior Data Service Developer

We need to output HTML fragment with sections: h3 headings for sections like About the role, Responsibilities, Requirements, Qualifications, Skills, Benefits, Pay, Schedule. Only include sections that have real specific content. We need to parse the posting. First, the posting includes description of OTI, ODS, role description. We need to extract substantive info. We must not repeat job title as heading (the page already shows it). The job title is "Senior Data Service Developer". Not to put as heading. We need to include sections: maybe "About the role" (description), "Responsibilities" (list of primary responsibilities), "Requirements" (minimum qualifications?), "Qualifications" (maybe preferred skills?), "Skills" (list of skills), "Benefits" (mention of benefits: "great benefits", "Public Service Loan Forgiveness", maybe "NYC Residency not required"? Also "OTI offers great benefits" etc). "Pay" - not mentioned. "Schedule" - mention of HOURS/SHIFT: Day - due to necessary technical duties, candidate may be required to work various shifts such as weekends and/or nights/evenings. So schedule info present. We must not include sections with placeholder like "commensurate with experience". Not present. We must remove boilerplate: cookie/navigation/legal/EEO/equal-opportunity/scam-warning, duplicated lines, application instructions. So we need to strip out things like "TO APPLY", "Please go to www.cityjobs/jobs/search and search for Job ID #801794", "SUBMISSION OF A RESUME IS NOT A GUARANTEE...", "APPOINTMENTS ARE SUBJECT TO OVERSIGHT APPROVAL", "OTI participates in E-Verify", "Public Service Loan Forgiveness" maybe keep as benefit? It's a benefit info. "Residency Requirement New York City Residency is not required for this position" maybe keep as note under Benefits or maybe separate? Could be part of Benefits or Additional Info. We'll include under Benefits maybe. Also "Additional Information The City of New York is an inclusive equal opportunity employer..." that's EEO boilerplate, remove. We need to preserve all substantive information. Let's extract: About the role: description of OTI, ODS, role purpose. From text: "The Office of Technology and Innovation (OTI) leverages technology to drive opportunity, improve public safety, and help government run better across New York City. From delivering affordable broadband to protecting against cybersecurity threats and building digital government services, OTI is at the forefront of how the city delivers for New Yorkers in the 21st century. Watch our welcome video to see our work in action, follow us on social media NYCOfficeofTech, and visit www.oti.nyc.gov to learn more. At OTI, we offer great benefits, and the chance to work on projects that have a meaningful impact on millions of people. You'll have the opportunity to work with cutting-edge technology and collaborate with other passionate professionals who share your drive and commitment to making a difference through technology. The Office of Data Strategy (ODS) works with City agencies and their data to help serve New Yorkers more equitably and effectively. Historically, this work has mainly taken the form of analyzing agency data to improve City agency operations and working to make Open Data more accessible to all New Yorkers. ODS has also assumed a greater role in determining the direction of the infrastructure that powers non-public data sharing between City agencies, and an expansion of our role in automating data feeds, providing more robust quality assurance, and improving the user experience of the City’s data sharing platforms. We are seeking a versatile and driven Senior Data Service Developer to join our innovative Integration Services - Data Analytics team. This role ensures the efficient and successful implementation and support of complex data engineering solutions for City agencies. The Senior Data Service Developer should demonstrate a solid understanding of industry-standard implementation methodologies using data engineering technologies, tools, and processes; and will be responsible for developing, supporting and expanding our extensive data products and will collaborate closely with cross-functional teams to translate business requirements into technical implementations that drive our agency’s growth and success. The Senior Data Service Developer will report to the Director of Integration Services, and work closely with cross-functional teams and ODS office on projects." That's about the role. Responsibilities: list after "Primary responsibilities will include:" then bullet list. Let's extract each bullet: - Create and maintain optimal data pipeline architecture that is coherent and scalable, based on best practices of integrating data into a consolidated repository. - Build analytics tools that utilize the data pipeline to provide actionable insights into customer engagement and experience, operational efficiency, and other key business performance metrics. - Develop new or build against existing APIs for data access or landing data as output for further downstream consumption in the appropriate target data store. - Perform the technical design, development, and component testing of repository changes. - Build the infrastructure required for optimal extraction, transformation, and loading (ETL) of data from a wide variety of data sources using SQL, cloud, and “big data” technologies. - Develop ETLs to move data securely from source to target systems. - Contribute to new technology projects as assigned and provide operations & maintenance support for existing applications. - Troubleshoot errors, defects, and other system issues. - Create, update, and maintain system documentation. - Work with City’s Cyber Command unit to pass application security accreditation. - Perform special projects and initiatives assigned. That's responsibilities. Requirements: Minimum Qualifications section. We have "Minimum Qualifications" with four options (baccalaureate + 4 years, associate + 6 years, baccalaureate + 8 years, or equivalent). Also "Preferred Skills" list. We need to decide sections: maybe "Requirements" for minimum qualifications, "Qualifications" for preferred skills? Or combine? The instruction: use headings for section headings (e.g. About the role, Responsibilities, Requirements, Qualifications, Skills, Benefits, Pay, Schedule). So we can have Requirements (minimum qualifications) and Qualifications (preferred skills) and Skills (maybe list of specific skills). But we have a long list of preferred skills; could treat as Skills section. We need to avoid duplication. Let's extract minimum qualifications: - A baccalaureate degree from an accredited college in computer science, engineering, human computer interaction, interactive media, digital and graphics design, data visualization, communication or a related field, and four years of satisfactory full-time experience related to the area(s) required by the particular position and a specialization in a relevant technology, process, methodology and/or domain; or - An associate degree from an accredited college in computer science, engineering, human computer interaction, interactive media, digital and graphics design, data visualization, communication or a related field, and six years of satisfactory full-time experience related to the area(s) required by the particular position and a specialization in a relevant technology, process, methodology and/or domain; or - A baccalaureate degree from an accredited college, and eight years of satisfactory full-time experience related to the area(s) required by the particular position and a specialization in a relevant technology, process, methodology and/or domain; or - Education and/or experience which is equivalent to "1", "2", or "3" above. We'll keep that as Requirements. Preferred Skills: list of many items. Could be placed under Skills or Qualifications. Since we have both Qualifications and Skills headings allowed, we could put Preferred Skills under Qualifications (maybe "Qualifications" heading) and then list specific technical skills under Skills. But the preferred skills list includes both qualifications (e.g., Bachelor's degree, 8+ years Java, etc.) and skills (frameworks, tools). Might be better to treat the whole "Preferred Skills" as Skills section, and keep Requirements as minimum qualifications. However we also have "Qualifications" heading; we could use it for something like "Preferred qualifications" (like degree, years). But the preferred skills list includes many items that are more like skills. To avoid thin sections, we could combine: Requirements (minimum qualifications) and Skills (preferred skills). And maybe omit Qualifications heading if not needed. But we need to include sections only when posting has real specific content. We have content for Requirements and Skills. We could also have Benefits, Schedule. Thus sections: About the role, Responsibilities, Requirements, Skills, Benefits, Schedule. We must not include Pay (no info). Also not include Qualifications if we don't have separate content. Now extract Benefits: From text: "At OTI, we offer great benefits, and the chance to work on projects that have a meaningful impact on millions of people." Also later: "Public Service Loan Forgiveness As a prospective employee of the City of New York, you may be eligible for federal loan forgiveness programs and state repayment assistance programs. For more information, please visit the U.S. Department of Education’s website at https://studentaid.gov/pslf/." Also "Residency Requirement New York City Residency is not required for this position". Also maybe "great benefits" generic but we have specifics: Public Service Loan Forgiveness, residency not required. Also maybe "chance to work on projects that have a meaningful impact". That's more about role. We'll include benefits: great benefits (generic), Public Service Loan Forgiveness, NYC residency not required. We should not include application instructions or EEO. Now Schedule: "HOURS/SHIFT Day - Due to the necessary technical duties of this position in a 24/7 operation, candidate may be required to work various shifts such as weekends and/or nights/evenings." Also "WORK LOCATION Brooklyn, NY". Location maybe not schedule but could be included under Schedule? The instruction: Use Schedule heading for schedule info. We could also include work location maybe under About the role? But we have no specific heading for location. Could include under Schedule as additional info? Might be better to include under Schedule: "Day shift, may require weekends/nights/evenings; Work location: Brooklyn, NY." That seems reasonable. Now Skills: extract the preferred skills list. It's a long bullet list after "Preferred Skills The ideal candidate will have a passion for both front-end and back-end development, with a keen eye for detail and a commitment to delivering high-quality solutions. An interest and/or understanding of New York City policy and operations or strong policy or operational analysis capabilities is preferred but not required. The successful candidate should possess the following:" then bullet list with dash items. We need to preserve all items. Let's list them: - Bachelor’s degree in computer science, engineering, or a related field. - 8+ years of experience in Java application development. - 8+ years of experience in writing SQL and Python - 8+ years of experience in copying, transferring, manipulating, and automating data operations that were manual processes. - Experience in developing applications using Java Spring Boot and RabbitMQ - Solid understanding of web development fundamentals such as HTTP, RESTful APIs, and web security principles. - Experience with front-end frameworks/libraries (e.g., React, Angular, Vue.js) and proficiency in HTML5, CSS3, JavaScript. - Proficiency in server-side languages and frameworks (e.g., Node.js, Python/Django). - Experience with tools and components of data architecture such as Informatica Power Center, IICS, SSIS, or similar ETL tools. - Experience working with Amazon Web Services or Microsoft Azure cloud computing platform and services. - Experience with WSDL creation. - In-depth knowledge of SQL and other database solutions. - Experience with data pipelines and warehousing (Snowflake, Redshift etc.). - Familiarity with cloud platforms (e.g., AWS, Azure, GCP) and containerization technologies (e.g., Docker, Kubernetes). - Knowledge of modeling database schemas for large datasets. Knowledge of JSON, XML, XSD, WSDL, JDBC, MQ, SOAP concepts - Experience with tools and components of MDM (Master Data Management). - Experience developing cloud-ready applications. - Proven experience as a Data and Software Developer or in a similar role, with a strong proficiency in both front-end and back-end technologies. - Experience with databases (e.g., SQL, NoSQL) and version control systems (e.g., Git). - Experience