Sr. Data Engineer
100% Remote Forever
The Sr. Data Engineer will partner with product owners and data consumers to meet analytics and client data needs. In this role, you will collaborate with a team of data engineers, many of whom possess a deep healthcare background. You will have expertise in SQL Server Integration Services (SSIS) & SQL programming (T-SQL/Spark SQL) with an aptitude for collaboration and open communication.
In addition, you will participate in building our modern data platform – which uses infrastructure and tools like Kafka, Spark, Databricks, Airflow, Jenkins, and Snowflake. You will be accountable for high quality deliveries as an individual contributor. This is a fantastic opportunity to engage in a positive, creative, work environment that balances traditional and innovative ETL methodologies while delivering data to health care professionals to improve patient lives.
- Collaborate with product owners, business analysts and infrastructure engineers to deliver against an agile roadmap of analytics initiatives
- Develop efficient, high performing ETL solutions using SSIS to facilitate data movement consistent with business and data requirements
- Write efficient, high performing advanced/complex T-SQL / Spark SQL
- Design, build and maintain data ingestion services (batch and streaming) and end-to-end ETL data pipelines from SQL Server to cloud infrastructure (Kafka/Databricks Lakehouse/Delta Lake/Azure/Snowflake)
- Work with Databricks Notebooks and other big data tools to create and operationalize data transformation pipelines that consolidate data into a single, unified source
- Consume data from a variety of sources (RDBMS, APIs, FTPs and other cloud storage) & formats (Excel, CSV, XML, JSON, Parquet, Unstructured)
- Identify ways to improve data reliability, efficiency, and quality in on-premises and cloud data pipelines
- Assist in continuous refinement of best practices for all ETL related activities, including data movement, data quality, and data cleansing
- Build skills in reliability engineering, deployment planning, fault tolerant architecture, and test automation frameworks
- Provide reliable estimation of work and commit best efforts to delivering work in alignment with those estimates
- Communicate status, dependencies, and issues to the Product/Project Manager
- Monitor ETL processes associated with QA, UAT, and Production domains – 24/7 job monitoring, issue identification, documentation, analysis, and resolution
- Build collegial and effective working relationships with coworkers
- Apply a product mindset and product design thinking to build data as products
You are an expert with expansive data engineering experience. While you have hands-on experience with common tools, frameworks, and paradigms used in data engineering, you can rapidly switch gears to use new architecture and tools for the job at hand. To the core, you are a key backend engineer and architect, who has had the opportunity to focus on managing the data related aspects of large-scale healthcare-oriented enterprise backend systems. You understand how diverse types of stakeholders work with data, from product engineering teams to business analysts, to data scientists.
- Bachelor’s degree in Computer Science, Information Management and/or equivalent work experience.
- 3+ years of T-SQL, writing complex queries, stored procedures and functions, query optimization, performance tuning, and index utilization
- 3+ years of ETL experience using SSIS (SQL Server Integration Services)
- 2+ years of working on large data warehouse/big data projects
- 2+ years of experience in Spark (SparkSQL, Pyspark) implementing Databricks Delta Lake and/or Lakehouse Platform
- Excellent interpersonal/communication skills (both oral/written) with the ability to communicate at various levels with clarity & precision
- Experience with Agile product development (using tools such as Rally/TFS/GitHub)
- Deep experience with healthcare data, its standards, and its complex data integrations
- Kafka exposure is a huge plus
- Experience with cloud data warehouse (preferably Snowflake)
- Experience with Enterprise scheduling tools like Airflow
- CI/CD experience, ideally Jenkins
- Knowledge of data modelling techniques including slowly changing dimensions, aggregation, partitioning
- Exposure to Kimball or Inmon data modeling methodologies
Brooksource provides equal employment opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, national origin, age, sex, citizenship, disability, genetic information, gender, sexual orientation, gender identity, marital status, amnesty or status as a covered veteran in accordance with applicable federal, state, and local laws.