Rahul Upadhyay

Roles & Responsibility

Role involves developing and maintaining real-time data pipelines using Apache Spark Streaming to ingest and process data from Apache Kafka. Responsibilities include applying transformation and aggregation functions to streaming data for analytics and business insights. Expertise required in optimizing Spark jobs for performance and scalability within Databricks environment. Additionally, implement robust data writing strategies to efficiently load aggregated results into PostgreSQL databases ensuring data integrity and consistency. Collaborate with cross-functional teams to support data-driven decision making and maintain system reliability.