Data Engineer This individual will be responsible for creating products to host Supply Chain Analytics algorithms. Looking for someone with full stack experience, even if the specialty is in one area. This person to have 5-7+ years of experience with software engineering and testing, experience working in an Agile Environment, and experience with Rally. This person will have interactions with leadership team and needs to have good communication (both written and oral) and should feel comfortable not waiting and being told what to do. Work as part of an implementation team from concept to operations, providing deep technical subject matter expertise for successful deployment of companies
Data Platform Implement methods for standardization of all parts of the pipeline to maximize data usability and consistency Identify, develop, evaluate, and summarize Proof of Concepts to prove out solutions Test and compare competing solutions and report out a point of view on the best solution Design and build data engineering solutions to deliver our pipeline patterns using CI/CD services, Python, PySpark and GCP Native tools. Develop Frameworks and Tools for scheduling Jobs on Astronomer Design and build reusable utilities and tools for automation of data ingestion and transformation jobs
Skills Required:
Design, Develop and Implement data engineering solutions using standards, templates, patterns and best practices in the Google Cloud Platform o Full stack data engineering solutions utilizing both structured and unstructured data: development, ingestion, curation, implementation, deployment, automation and monitoring o Collaborates with the Data Factory Engineering Organization, Data Architecture, PEM, GDIA, Information Technology and Data Consumers to drive data engineering capabilities, product design and proof of concepts, MVPs, to expand understanding, define technical optimization, explore configurations and overcome challenges o Create high quality, elegant data engineering solutions that focus on cloud-first, encapsulation, repeatability, automation and auditability o Work as an individual contributor and part of a team to build, test, maintain and troubleshoot data solutions Continuously integrates and deploys data solutions via CI/CD Use of Test Driven Development and Code Pairing Practices
Skills Preferred:
2+ years Extensive knowledge and understanding of GCP offerings, bundled services, especially those associated with data operations Cloud Console, BigQuery, Data Flow, Data Fusion, PubSub / Kafka, Looker Studio and VertexAI Experience with Teradata, Hadoop, Hive, Spark and other on-premise/legacy data solutions Experience optimizing data solutions and data science/analytical workflows: re-coding / re-developing / re-factoring Data Governance concepts including GDPR (General Data Protection Regulation), PoLP and how these can impact technical architecture. Data Manipulation - GCP (Big Query, Cloud Storage, Data Proc, Terraform, Airflow etc.)
Experience Required:
Bachelor s degree in a Technical Field: Computer Science, Data Science, Computational Finance, Statistics, Economics and/or Mathematics: Masters Preferred 5+ years of experience in data solution, pipeline, mart and/or warehouse development and delivery using agile development methodology Critical thinking skills to propose solutions, test, and make them a reality. 2+ years of experience in a Data Engineering Competency on a public cloud - Google, MS Azure, AWS 2+ years Integration and Configuration Scripting in Tekton and Terraform 3+ years of experience writing complex SQL 3+ years of experience writing data solutions in Python, Java, Scala or Go Deep understanding of data service ecosystems including data warehouses, lakes, metadata, meshes, fabrics and analytical use cases User experience advocacy through empathetic stakeholder relationship Excellent Communication Skills, Verbal and Written, for both internal and external team members
Experience Preferred:
Highly effective in working with other technical experts, Product Managers, UI/UX Designers and business stakeholders Delivered products that include web front-end development; JavaScript, CSS, frameworks like Angular, etc. Comfortable with Continuous Integration/Continuous Delivery tools and pipelines e.g. Tekton, Cloud Build, etc. Experience with machine learning, mathematical modeling, and data analysis is a plus Experience with CA Agile Central (Rally), backlogs, iterations, user stories, or similar Agile Tools Experience in the development of microservices Understanding of fundamental data modeling Strong analytical and problem-solving skills
