About the role:
The Staff Data Engineer will architect, build, and maintain data lake and data warehousing solutions for a B2B multi-tenant SaaS solution. Position requires a demonstrated ability to design processes from end to end and a strong technical understanding of modern data pipelines and strategies. In depth knowledge of multiple data sources, data design patterns, ETL transformation, SQL and performance tuning are required. Ability to train and mentor team members and collaborate with cross functional team members.
Role and Responsibilities:
- Participate in the entire data pipeline lifecycle, focusing on building data lakes, data warehouses and related processes
- Contribute to the design and documentation of our data lake platform
- Partner with Sr staff on the development and maintenance of data governance policies and procedures that ensure data quality, accuracy, consistency, and privacy.
- Mentor other data engineers, providing technical guidance, and ensuring that team members have the necessary skills and resources to perform their duties.
- Conduct code and design reviews with junior and peer developers
- Write clean code to develop data pipelines and necessary infrastructure as code
- Aid in setting coding standards for team members to follow
- Partner with cross functional teams to understand requirements and effectively design appropriate solutions.
- Monitor and troubleshoot performance, scalability, and security issues, making improvements as needed
- Aid in determining and implementing KPI's for performance and stability of platform
- Participate in the review of new technologies and tools for our data teams
- Partner with engineering and other cross functional teams to collect and implement technical and design requirements
- Communicate effectively with technical and non-technical team members to ensure alignment and successful execution of projects.
- Participate in cross-functional initiatives, promoting a culture of data-driven decision-making and continuous improvement.
- Participate in an on call rotation.
Job Requirement :
- 8 years of related experience; or 6 years and an advanced degree
- Bachelor's degree in engineering, computer science, math, data engineering or another related field or equivalent work experience
- Experience in the development of modern data architecture, analytics, data governance, AI/ML, or related areas.
- Strong Experience with AWS big data technologies including EMR, Glue, S3, EKS, Lambda, Athena, RDS, MKS/Kafka/Kinesis
- Expertise with relational and NoSQL databases such as PostgresQL, Cassandra, DynamoDB, MongoDb etc and data modeling principles
- Knowledge of data pipeline tools and workflow engines, e.g. Apache Airflow and Spark.
- Experience with DevOps, including CI/CD pipelines, containerized deployment/Kubernetes, and infrastructure-as-code/AWS Cloud Formation/Terraform.
- Successfully contributes to the entire development lifecycle of teams products
- Exposure to secure coding practices, access controls, authentication, and audit. AWS Lake formation preferred.
- Strong experience with SQL (SQL Server, Postgres) and Python, Scala
- Identify opportunities for process improvement as well as efficiency in solutions
- Demonstrated ability to mentor junior team members
- Excellent analytical and time management skills, with a proven ability to deliver value independently
- Strong written and verbal communication skills, with demonstrated experience providing technical input to technical and non-technical stakeholders
- Ability to participate in on-call rotation
Preferred:
FinTech industry experience
Understanding of MPP Data Warehouse (Redshift, Greenplum)
Experience with Graph database
