LEAD BIG DATA DEVELOPER | DC/MD/VA AND NYC (100% REMOTE) Contract

ROLE: LEAD BIG DATA DEVELOPER

LOCATION: DC/MD/VA AND NYC (100% REMOTE)

DURATION: 6-8 MONTH

VISA: , GC-EAD, GC- (W2/1099) CAN APPLY. (NO H1B)

*8 -12+ YEARS OF EXPERIENCE REQUIRED

JOB DESCRIPTION:

• Understand complex business requirements
• Design and develop ETL pipeline for collecting, validating and transforming data according to the specification
• Develop automated unit tests, functional tests and performance tests.
• Maintain optimal data pipeline architecture
• Design ETL jobs for optimal execution in the AWS cloud environment
• Reduce processing time and cost of ETL workloads
• Lead peer reviews and design/code review meetings
• Provide support for the production support operations team
• Implement data quality checks.
• Identify areas where machine learning can be used to identify data anomalies

Experience & Qualifications

• 7+ years of experience in programming language Java or Scala
• 7+ years of experience in ETL projects
• 5+ years of experience in big data projects
• 3+ years of experience with API development (REST API’s)
• Believes in Scrum/Agile, and has deep experience delivering software when working on teams that use Scrum/Agile methodology
• Strong and creative analytical and problem-solving skills

Required Technical Skills & Knowledge

• Strong experience in Java or Scala
• Strong experience in big data technologies like AWS EMR, AWS EKS, Apache Spark
• Strong experience with serverless technologies like AWS Dynamo DB, AWS Lambda
• Strong experience in processing with JSON and csv files
• Must be able to write complex SQL queries
• Experience in performance tuning and optimization
• Familiar with columnar storage formats (ORC, Parquet) and various compression techniques
• Experience in writing Unix shell scripts
• Unit testing using JUnit or ScalaTest
• Git/Maven/Gradle
• Code Reviews
• Experience with CI/CD pipelines
• Agile

The following skills a plus:

• AWS Cloud
• BPM/ AWS Step Functions
• Python scripting
• Performance testing tools like Gatling or JMeter

Top Skills’ Details

• Lead/Senior Big Data Developer
• This person will not be leading the team or have any direct reports but will be a senior developer on the team who can provide technical mentorship to other engineers.
  Preference is to start with candidates local to DC/MD/VA and NYC.
• This is a 100% remote role and candidates will not be required onsite regularly.
• Prior to an interview with the client, candidates must complete a Glider assessment, which I will send to candidates directly.
• Exp programming in Scala.
• Strong experience in big data technologies like AWS EMR and Apache Spark
• Strong experience with serverless technologies like AWS DynamoDB and AWS Lambda
• The chosen database is AWS Aurora

Technical experience in all the areas listed below:

• Experience working with JSON files as data will be coming in as JSON files
• Ability to write complex SQL queries
• Strong experience in performance tuning and optimization
• Strong unit testing using JUnit or ScalaTest is the minimum expectation, data testing experience would be great
• Git/Maven/Gradle

Tech Stack:

• Scala is the main programming language for the team.
• Aurora is a database.
• SQL is the backend database
• ETL process is Scala/Spark on EMR clusters.
• Code reviews are a large aspect of team culture.
• Agile environment with 2-week sprints
• Lead Big Data Developer
• Are you passionate about data? Do you like working in a challenging environment where a massive volume of data is ingested and processed every day? Are you a continuous learner who wants to learn new tools and technologies in evolving big data and data science technology? Do you have a passion for detecting data anomalies in large datasets? Do you expect the best from yourself and those around you?

We are looking for a lead big data developer for an ETL project in our enterprise Transparency Services group. Big data developers will work on designing, ingesting, storing, validating and disseminating, after transforming data in a consumable format, for business intelligence teams and data analysts to get deeper business insight from the data.

Nice to have skills:

• AWS Aurora
• Data testing