Position - Big Data Engineer

Location - Pune/Hyderabad/Any

Experience - 5+



Job Summary

As a Big data engineer responsibilities include - data ingestions to Hadoop
datalake from text/json files and relational databases, data pipeline
implementation using sqoop and pyspark, sql to analyze the data from hive
tables using impala.
You will be part of a team responsible for delivering the highest quality
data processing drawing upon your engineering and coding expertise, whilst
being open minded to the opportunities the cloud provides. You will deliver
code, facilitate solution designs and identify potential technical impediments
then work to resolve them.Roles & Responsibilities

Devise technical solution using Cloudera Hadoop and Spark ecosystem
technologies
Data pipeline design, implementation and monitoring
Data ingestions from text and json files using pyspark programs
Data ingestions from relational databases using sqoop commands
Data manipulation stored in Hive tables and HDFS file system
Data transformations using pyspark programs and impala queries
Monitoring the data pipelines through Airflow
Wrapping python, pyspark , impala, spark-submit commands into shell scripts
Help to ensure work carried out is in line with quality standards
Essential Skills:

Strong knowledge of Hadoop/Big Data ecosystem technologies
Good working knowledge of Apache Spark with Python
Experience in working with Cloudera cluster
Must have hands on experience in HDFS commands and Hive table creation
Must have hands on experience in data ingestions to Hadoop datalake using
sqoop and pyspark
Must have hands on experience in analysing data sets using Hive/Impala SQL
though Hue interface
Must have hands on experience in writing transformations using PySpark and
SQL
Experience in monitoring data pipelines through airflow
Experience in debugging and troubleshooting Spark Jobs, sql queries and
shell scripts
Experience in Azure Databrics and ADF is a plus

Skills:
Cloudera Hadoop, Spark, HDFS, Hive, Impala, SQL, Airflow, Python, PySpark,
Shell scripting, Cloudera manager, Yarn, Parquet

Key Skills