Senior Software Engineer, Big Data (

San Francisco - California

Date Posted: Sep. 19, 2018

Requisition ID: MAC12245


Job Overview:

Macy's is looking to hire experienced Big Data Developers to play a key role in designing and developing real time data acquisition and analytics platform using latest open source big data technologies.

The Senior Big Data Developer role requires you to actively participate in setting up a low latency, secure data pipeline which meets our machine learning and analytics requirements at enterprise scale. You will work extensively with application teams, data scientists and platform architects to setup and iteratively refine anomaly detection, alerting, key metric computation and visualizations that help site reliability and monitoring teams to isolate and troubleshoot issues. Perform other duties as assigned.

Essential Functions:

  • Design and develop scalable data pipelines for optimal ingestion, transformation, storage and computation using latest big data technologies.
  • Work with data science team to develop, deliver and validate solutions that meet the analytics and machine learning requirements.
  • Design, develop and maintain automated ETL processes for efficient batch records-matching of multiple large-scale datasets.
  • Translate complex business requirements into a scalable, efficient and high availability data platform.
  • Work with business teams/engineers to define instrumentation and data requirements.
  • Lead projects independently and also mentor other engineers in the team.
  • Prototype ideas quickly using cutting edge and new generation technologies.
  • Regular, dependable attendance & punctuality  




  • Bachelor’s or Master’s degree or equivalent in computer science or related field with minimum. of 5 years of directly related work experience.
  • Good understanding of data structures and algorithms.
  • Programming experience in Python, Java or Scala.
  • Extensive experience with Elastic stack, Kafka and Spark.
  • Experience with big data technologies like Hadoop, Map Reduce, Hive, Spark/Storm.
  • Experience with time series and NoSQL databases is a plus.
  • Experience working with containers and kubernetes is good to have.
  • Experience designing, implementing and scaling big data solutions on public cloud.
  • Experience with Unix/ Linux and shell scripting.
  • Experience working with various data sources like unstructured data files, flat files, message queues, JSON based events, databases.
  • Experience driving architectural discussions, and proposing solutions for system and product changes.
  • Proven track record of strong analytical and problem-solving skills.
  • Experience with large distributed services is a plus as is building/operating highly available systems.

Communication Skills

  • Able to effectively communicate across teams and roles
  • Excellent verbal and written communication skills
  • Ability to produce clear, well-documented designs for product features.
  • Experience writing high-performance, reliable and maintainable code.

Mathematical Skills

  • Basic math functions such as addition, subtraction, multiplication, division, and analytical skills.

Reasoning Ability

  • Proven ability as a problem-solver

Physical Demands:

  • This position involves regular walking, standing, sitting for extended periods of time, hearing, and talking. 
  • May occasionally involve stooping, kneeling, or crouching. 
  • May involve close vision, color vision, depth perception, focus adjustment, and viewing computer monitor for extended periods of time.  
  • Involves manual dexterity for using keyboard, mouse, and other office equipment. 
  • May involve moving or lifting items under 10 pounds.

 Other Skills

  • Strong estimating and planning skills
  • Ability and desire to thrive in a proactive, high-pressure, client-services environment
  • Ability to adapt to new development environments, changing business requirements.
  • Proven ability to effectively work across multiple teams on solutions that have complex dependencies and requirements in a fast-paced environment.

Work Hours

Ability to work a flexible schedule based on department and store/company needs.

This job overview is not all inclusive.  In addition, Macy’s, Inc. reserves the right to amend this job overview at any time.  Macy’s is an Equal Opportunity Employer, committed to a diverse and inclusive work environment.  Macy’s, Inc. – including Macy’s and Bloomingdale’s – will consider for employment qualified applicants with criminal convictions in a manner consistent with SFPC Art. 49 and LA MC ch.XVIII Art. 9.