Job Description
Big Data Engineer ( 4 to 6 yrs)
Location - Chennai
About the role
The successful candidate who will take this role will serve as an expert big data engineer.
The big data engineer will integrate with multiple data sources and systems, develop data engineering workflows, and will work with rapidly evolving technology-based solutions and teams.
You will work closely with data architects, data scientists, data analysts and business stakeholders for achieving strategic business goals.
To succeed in this new position, you will: Work closely with data architects, data analysts and data science teams to come up with technical solutions to solve specific business problems.
Ability to deal with different data sources and data types. Integrate and Ingest data from multiple source systems using into big data environment.
Create end to end data transformations and workflows for the data engineering transformations that serve the business needs along with the proper logging and recovery procedures for these workflows.
Flexible to relocate to Middle East (Gulf Countries)
We are looking for a big data engineer with a successful track record for building scalable data solutions in a fast-paced environment.
You will need 4+ years of experience in developing big data solutions, including experience with the following:
Solid Experience in ingesting the data from multiple data sources through APIs into HDFS.
Solid Experience in creating data pipelines using NiFi.
Solid Experience in Data integrationTalend is a must, other data integration tools as Informatica Power Center, Informatica BDM, ODI, Denodo, Altryx ..etc is preferred.
Solid Experience dealing with different data types [structured, unstructured, semi structured] along with solid experience in dealing with different file formats [Text, Json, Avro, Parquet, ORC, etc]
Solid experience in creating transformation using Spark, Hive on Tez and different components existing in the Hadoop eco system Proven Experience with Hive, Impala, NoSQL data bases querying.
Experience with NoSQL, Hive, Impala. Experience in Tuning bigdata transformations using Apache Spark and Hive QL.
Experience solving for complex data integrations across multiple systems.
Experience working with Data Science teams to improve actionable data.
Solid experience in Python, Java, Scala and SQL (including Spark SQL)
Solid experience in creating multithreading Python, Java applications to ingest data
. Solid experience in UNIX Bash Commands and Scripts. Experience with databases like Postgres, Greenplum, mySQL, and Oracle.
Experience in creating batch, real-time and near-real-time data pipelines.
Experience with streaming data technologies using [ Kafka, Spark Streaming, Apache Flink, etc.] Provide top-quality data engineering solutions.
Provide support in defining the scope and sizing of work. Align the organization's big data solutions with their client initiatives as requested
. Engage with clients to understand strategic requirements. Responsible for translating business requirements into technology solutions.
Work with domain experts to put together a delivery plan with and stay on track. Utilize Big Data technologies to design, develop, and evolve scalable and fault-tolerant distributed components.
