Job duties and Responsibilities:

• Provide guidance to business and tech partners on best methods to engineer data processes.
• Code and build data pipeline frameworks to automate high-volume and real-time data delivery.
• Develop applications from the ground up using a modern technology stack, such as Scala, Spark, Postgres, and Shell.
• Engineer capabilities and pipelines for big data, machine learning solutions, and running high-end data analytics.
• Work product owners, tech leads, and other cross platform teams to deliver data products in a collaborative and agile environment.
• Code and engineer scalable and resilient data driven solutions with current new data technologies to meet the needs of the organization and business customers.
• Break down complex data issues and resolve them.
• Establish and drive DevOps techniques and practices, like Continuous Integration, Continuous Deployment, Build Automation, and Test-Driven Development to enable the rapid delivery of working code utilizing tools like Jenkins, Maven, Ansible, Git, and Docker.

Work experience / Technologies required for the position :

• Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases.
• Experience building and optimizing Big Data, Data pipelines, architectures and data sets.
• Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
• Strong analytic skills related to working with unstructured datasets.
• Build processes supporting data transformation, data structures, meta data, dependency and workload management.
• A successful history of manipulating, processing and extracting value from large disconnected data sets.
• Working knowledge of message queuing, stream processing, and highly scalable Big Data, data stores.
• Strong project management and organizational skills.
• Experience supporting and working with cross-functional teams in a dynamic environment.
• Experience in a Data Engineer or similar roles, who has attained a Bachelor’s degree in Bachelor’s degree in Engineering, Computer Science, Information Technology, related field or equivalent work experience.
• Experience with big data tools: Hadoop, Spark, Kafka, etc.
• Experience with relational SQL and NoSQL databases, including Postgres and Cassandra.
• Experience with data pipeline and workflow management tools: Azkaban, Luigi, Airflow, etc.
• Experience with AWS cloud services: EC2, EMR, RDS, Redshift
• Experience with stream-processing systems: Storm, Spark-Streaming, etc.
• Experience with object-oriented/object function scripting languages: Python, Java, C++, Scala, etc.