Data Engineer - AWS Data Lake Engineer

Company: Air Products
Location: Allentown
Posted on: August 3, 2022

Job Description:

Job Description and QualificationsAir Products is a thriving Fortune 500 global company that is growing and looking for talented, driven Digital Technology professionals to join our team! With over 20,000 employees and operations in over 50 countries, Air Products is committed to its Higher Purpose of bringing people together to collaborate and innovate solutions to the world s most significant energy and environmental sustainability challenges.We are excited to share our Digital Technology team is expanding to meet growing business needs across the world. We re making a significant investment in our people and systems to strengthen our digital foundation, drive business optimization and enhance our customer experience. If you are passionate about achieving these goals as well, we would like you to consider joining our team!We have an immediate opening for a Data Engineer to help us build and maintain our Amazon Web Services (AWS) Data Lake. This position can be located at our global headquarters in Allentown, PA or can be available to those working remotely in the US. In the case of remote work, physical presence in the office may be required on occasion to engage in face-to-face collaboration among teammates.The Data Engineer is responsible for operationalizing data pipelines that support analytics initiatives for the company. The primary responsibilities include designing, building, managing, optimizing and documenting data flows from various sources into our enterprise data lake. Data ingestion timing can be near real-time or streaming. Delivery of high-quality data is a key item of focus.Essential skills needed include advanced AWS Redshift skills, AWS Glue/PySpark, AWS Athena, AWS Lambda/Python, AWS IAM. Advantageous skills would include EMR, & Big Data technologies (Hadoop/Spark).The data lake enables consumers such as data scientists and business and IT data analysts to complete advanced analytics projects as well as business reporting. The data engineer is expected to collaborate with data scientists, data analysts and other data consumers to productionize data models and algorithms developed by those users to improve the overall efficiency of advanced analysis projects. Additionally, the data engineer is responsible for ensuring data quality, governance and data security procedures are met while curating data for use in the Data Lake.What you ll do:CoreDesign and develop Glue ETL jobs that can accommodate diverse and complex data sources, highly complex transformations and merges.Advanced understanding of both SQL and NoSQL technologies, to preferably include AW Redshift.Design and develop Lambda and AWS Batch scripts in Python.Perform data replication with Qlik Replicate and maintain data marts with Qlik Compose.Leverage EMR and Hive to process change-data-capture records both in S3.Design and incorporate error handling & Data Quality processes into pipelines and processes.Design, implement, and analyze robust test plans and stress tests.End-to-end Implementation of Data PipelinesLead and/or work with cross-disciplinary teams to understand, document and analyze customer needs.Identify and present a range of potential solution options for any demand, informing stakeholders of advantages and disadvantages of each; assist them in arriving at an optimal solution strategy.Optimize flexibility, scalability, performance, reliability, and future-proof capacity of IT services, at an optimal cost.Implement chosen solutions, including infrastructure, scripts, database resources, permissions, source control.Contribute to the wider enterprise architecture and roadmap.PlanningConduct research into, test, and trial new technologies and approaches they could enhance our work.Educate and train yourself and others as you evangelize the merits of data and analytics.Document own, or existing projects, in a clear yet comprehensive format for a wide range of audiences.Contribute to enhancing the team's own internal processes of communications, documentation, workload planning.Work closely with management to prioritize business and information request backlogs.Ensures data governance and data security procedures are followed.What we re looking for:Bachelors degree in Information Technology field (related technical discipline preferred)3+ years as a Python, PySpark, SQL developer; building scalable ETL applications and data warehousesAdvanced understanding of both SQL and NoSQL technologies, to preferably include AW RedshiftAdvanced proficiency programming in PySpark and Python ETL modules is required.Experience in working with and processing large data sets in a time-sensitive environment while minimizing errorsHands-on experience working with big data technologies (Hadoop, Hive, Spark, Kafka)Proficient experience working within the AWS and AWS tools (S3, Glue, EMR, Athena, Redshift)Experienced in maintaining infrastructure as code using Terraform or cloud formationHands-on experience working with Qlik (Attunity) Replicate & ComposeSolid understanding of data warehouse design patterns and best practicesAbility to develop test plans and stress test platformsExperience with complex Job schedulingDemonstrated strength in process development, process adherence, and process improvementEffective analytical, conceptual, and problem-solving skillsMust be organized, disciplined, and task/goal orientedAble to prioritize and coordinate work through interpretation of high-level goals and strategyEffective team player with a positive attitudeStrong oral and written English language communications skillsAt Air Products, we work in an environment where diversity is essential, inclusion is our culture, and each person knows they belong and matter. To learn more, visit About Air Products.We offer a comprehensive benefits package including paid holidays/vacation, affordable medical, dental, life insurance and retirement plans.Air Products thanks all applicants in advance for their interest; however, only those applicants who are being considered for an interview, or are currently employed by Air Products, will be contacted.We are an Equal Opportunity Employer (U.S.). You will receive consideration for employment without regard to race, color, religion, national origin, age, citizenship, gender, marital status, pregnancy, sexual orientation, gender identity and expression, disability, or veteran status.Req No.38182BREmployment StatusFull TimeOrganizationCorporateBusiness Sector / DivisionInformation TechnologyRegionNorth AmericaCountryUnited States

Keywords: Air Products, Allentown , Data Engineer - AWS Data Lake Engineer, Engineering , Allentown, Pennsylvania