Data Engineer

Calibo

Early Applicant

5 months ago
Be among the first 50 applicants

Exp: 5-7 Years

Full time

Bengaluru / Bangalore, Pune, India

Job Description

As a Big Data Engineer, you will develop, maintain, evaluate, and test big data solutions. You will be involved in data engineering activities such as the following:
Creating pipelines or workflows for the source-to-target data journey
Designing data solutions by using big-data-based technologies along with Hadoop, MongoDB, Casandra, and Azure HDInsight for Cloudera-based data lake by using Scala Programming
Monitoring data pipelines and DataOps
Ingesting data from files, streams, and databases

Processing the data with Hadoop, Scala, SQL Database, Spark, ML, and IoT devices, among others
Developing programs in Scala and Python as a part of data cleaning and processing
Designing and developing distributed, high volume, high-velocity multi-threaded event processing systems
Developing efficient software code for multiple use cases leveraging Python and big data technologies for various use cases built on the platform
Providing high operational excellence guaranteeing high availability and platform stability
Implementing scalable solutions to meet the ever-increasing data volumes, using big data/cloud technologies such as PySpark, Kafka, cloud computing, and others

Requirements:

Minimum 5+ years of experience in big data technologies High-level proficiency in any of the programming languages - Python, Scala, or Java Mid- to expert-level experience in programming capabilities in a large-scale enterprise In-depth experience in modern data platform components such as Hadoop, Hive, Pig, Spark, Python, Scala, Databricks, and Snowflake, among others

Experience with Distributed Version Control environments such as GIT Familiarity with development tools - experience on either IntelliJ or Eclipse or VSCode IDE and Maven, among others Exposure to data visualization tools (Qlik Sense/Tableau/Metabase) Demonstrated experience in modern API platform design including how modern UIs are built consuming services and APIs

Experience in data quality governance, data security, data lake, data warehouse, lineage, Drools, metadata management, Apache workflow, and big data service in the cloud (AWS/Azure/GCP)

Experience on Azure cloud including Data Factory, Databricks, Data Lake Storage Solid experience in all phases of software development life cycle - plan, design, develop, test, release, maintain, support, and decommission