Role Purpose
We are looking for a highly motivated and experienced Data Engineer to join our team of data experts. The ideal candidate will have a strong background in designing, Development processes using technologies such as Python, Spark, PySpark, SQL and NoSQL. As a Data Engineer, you will work closely with the Lead Data Engineer and Data Architect to implement end-to-end data solutions, build, and maintain data pipelines, and ensure the quality and integrity of our organization's data.
Role Responsibilities
- Collaborate with the Lead Data Engineer and Data Architect to design and implement end-to-end data solutions
- Develop and maintain data pipelines and ETL processes using Python, Spark, and SQL
- Ensure data quality and integrity by developing and implementing data validation and testing strategies
- Implement data security measures to protect the confidentiality, integrity, and availability of our organization's data
- Continuously monitor and evaluate the performance of data systems and make recommendations for improvements
- Stay up-to-date with the latest data technologies and trends and provide recommendations for their adoption
- Design and create processes for contextualizing, cleaning, aggregating, and transforming data, making it better and more useful to the business
- Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency, and other key business performance metrics
Skills & Experience
- 5+ years of experience in data engineering and data architecture
- Strong Python/Java programming
- Strong SQL knowledge and good understanding of different databases (SQL and NoSQL)
- Good understanding of Data Able to transform a business entity into a database entity.
- Knowledge of SDLC Source code control (git process), unit testing, release process, CI/CD pipelines
- Knowledge of any cloud Storage, Network, Security and Compute
- Getting hands dirty attitude towards learning new technologies and exploring the current trends
- Good to have:
- Experience with data pipelines Spark/PySpark
- Experience with Databricks or Snowflake
- Experience with data streaming Kafka
- Experience with data modelling.
- Excellent problem-solving and analytical skills
- Ability to work independently and as part of a team
- Experience working in an Agile environment
- Excellent written and oral communication skills; Ability to communicate effectively with technical and non-technical staff.
