PySpark with ADB Developer Hyderabad | NA Posted On: 12-11-24 job description
Design, develop, and deploy PySpark applications and workflows on Azure Databricks for data transformation, cleansing, and aggregation. Implement data pipelines using Azure Data Factory (ADF) to orchestrate ETL/ELT processes across heterogeneous data sources. Collaborate with Data Engineers and Data Scientists to integrate and process structured and unstructured data sets into actionable insights. Optimize PySpark jobs and data pipelines for performance, scalability, and reliability. Conduct regular financial risk assessments to identify potential vulnerabilities in data processing workflows. Ensure data quality and integrity throughout all stages of data processing. Develop and implement strategies to mitigate financial risks associated with data transformation and aggregation. Troubleshoot and debug issues related to data pipelines and processing. Ensure compliance with regulatory requirements and industry standards in all data processing activities. Implement best practices for data security, compliance, and privacy within Azure environment. Document technical specifications, data flows, and solution architecture. Skills Proven experience as a PySpark Developer or similar role with a strong understanding of Apache Spark internals Experience designing and optimizing data pipelines for ETL/ELT processes. Hands-on experience with Azure Databricks (ADB) and Azure Data Factory (ADF). Proficiency in Python programming language and solid understanding of SQL. Certification in Azure Data Engineering or related field. Knowledge of other big data technologies such as Hadoop, Hive, or Kafka Familiarity with machine learning frameworks and techniques. Experience in Financial, Risk, Compliance, or Banking domains is a plus. Experience identifying and mitigating financial risks in data processes. Ability to analyse data for potential risk factors and develop strategies to minimize financial risk. Ensure all data processes comply with relevant regulatory requirements and industry standards. Employment Type: Full Time, Permanent
Role Category: Software Development
Education
UG: B.Tech/B.E. in Any Specialization
PG: Any Postgraduate
Key Skills
Skills highlighted with ‘
Java Python PHP SQL Hadoop Spark Machine Learning AWS Azure Scala Data Warehousing Data Engineering Risk Assessment Scalability 
