We are looking for a savvy Data Engineer to join our growing team of analytics experts. The hire will be responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow and collection for cross functional teams. The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the ground up. The Data Engineer will support our software developers, DBA’s, database architects, data analysts and data scientists on data initiatives and will ensure optimal data delivery architecture is consistent throughout ongoing projects. They must be self-directed and comfortable supporting the data needs of multiple teams, systems and products. The right candidate will be excited by the prospect of optimizing or even re-designing our company’s data architecture to support our next generation of products and data initiatives.
Knowledge, Skills, and Abilities:
- Develops and maintains scalable data pipelines and builds out new API integrations to support continuing increases in data volume and complexity.
- Collaborates with analytics and business teams to improve data models that feed business intelligence tools, increasing data accessibility and fostering data-driven decision making across the organization.
- Implements processes and systems to monitor data quality, ensuring production data is always accurate and available for key stakeholders and business processes that depend on it.
- Defines company data assets including the data models, data storage technologies (open source and/or cloud) and the jobs necessary to populate those models using various analytics focused tools such as spark, sparkSQL, NiFi, Kafka, Sqoop, hiveSQL, (or other similar tools).
- Designs and evaluates open source and vendor tools for data lineage and provenance.
- Works closely with a team of developers, business customers, and analysts to enable advanced analytics and a digital business platform.
- Acts at the primary data curator and the analytics data steward
Preferred Education and Experience:
- Knowledge of best practices and IT operations in an always-up, always-available service.
- Experience with or knowledge of Agile Software Development methodologies
- 4+ years experience with modern analytics data structures including open source (e.g., various Hadoop components), purpose-specific structures such as graph databases (e.g., Neo4J), document databases (e.g., MongoDB), relational structures (e.g., SQLServer, Postgres, Hive), NoSQL (e.g., Cassandra).
- 2+ Years experience with Cloud-based data technologies such as AWS Redshift, Snowflake, S3, Neptune, etc.
- Excellent problem solving and troubleshooting skills
- Excellent oral and written communication skills with a keen sense of customer service
- BS or MS degree in Computer Science or a related technical field
- 4+ years of Python or Java development experience
- 4+ years of SQL experience (No-SQL experience is a plus)
- 4+ years of experience with schema design and dimensional data modeling
- Ability in managing and communicating data warehouse, data lake, or other analytics data structure plans to internal clients
- Experience designing, building, and maintaining data processing systems
- Experience working with an on premise or cloud-based MPP system on any size/scale