This role is for one of the Weekday's clients
In this role, you will be responsible for operating and maintaining a cloud data platform that facilitates efficient storage, retrieval, and analysis of large volumes of data. You will collaborate closely with customer success representatives, analysts, and other stakeholders to ensure a seamless flow of data from customer source systems through the pipeline to the final destination. The ideal candidate will possess a strong background in Extract, Load, Transform (ELT) processes, relational and non-relational databases, and data strategy on the cloud. A willingness to thrive in a dynamic startup environment is essential.
Responsibilities
Customer Onboarding:
- Integrate data from various customer sources into the data pipeline, ensuring consistency, accuracy, and reliability.
- Analyze data for consistency, completeness, and quality.
- Collaborate with customer success representatives to engage with customers as needed.
- Automate processes to minimize manual steps whenever possible.
Operational Management
- Execute workflows and jobs to move data through various processing stages to the destination.
- Monitor workflow and job execution, taking corrective actions in case of failures, and automate steps to eliminate bottlenecks over time.
- Implement data partitioning, indexing, and other optimization techniques to enhance query performance.
- Develop and maintain data models to ensure optimal performance and adherence to best practices.
ELT (Extract, Load, Transform) Processes
- Design, develop, and maintain ELT processes to transform raw data into a usable format for analysis.
- Monitor and troubleshoot data pipeline issues, ensuring data integrity and availability.
Data Security And Compliance
- Implement and enforce data security measures to safeguard sensitive information.
- Ensure compliance with data governance policies and industry regulations.
Collaboration And Documentation
- Work with data scientists, analysts, and other stakeholders to understand data requirements and provide timely support.
- Document data engineering processes, standards, and best practices.
Qualifications
- Bachelor's or Master's degree in Computer Science, Statistics, Data Science, or a related field.
- Industry experience working with data pipelines in the cloud.
- Proficiency in analytical SQL, including window functions, aggregates, CTEs, unions, and sub-queries.
Preferred Skills
- Experience with data management and querying for efficient retrieval using BigQuery (preferred), RedShift, PostgreSQL, or SQL Server.
- Familiarity with managing and accessing document databases such as Firestore (preferred), MongoDB, or DocumentDB.
- Cloud platform expertise for automation in Google Cloud Platform (GCP) or AWS.
- Knowledge of GCP Integration Connectors or other ETL tools.
- Mid-level proficiency in Python.
Skills: data engineering,data strategy,aws,documentdb,postgresql,mongodb,firestore,google cloud platform (gcp),analytical sql,relational databases,non-relational databases,data management,python,sql,pipeline,sql server,data security,bigquery,redshift,extract, load, transform (elt)
