Data Engineer- Azure Databricks- Financial Domain Exp Must
Backend Developer
Application Security
Azure
Azure Data Engineer
Azure Databricks
Azure Functions
Big Data
Bigdata
Cloud
Cloud Data Engineering
Cloud Data Platform
Cloud Data Warehouse
Cloud Data Warehouse
Cloud Engineer
Cloud Infrastructure
Cloud Platform
Cloud Platforms
Cloud Platforms Cloud Platforms
Cloud Technology
Data
Data Analysis
Data Analytics
Data Engineer
Data Engineering
Data Integration
Data Integration Tools
Data Lakehouse
Data Pipeline
Data Platform
Data Processing
Data Warehouse
Data Warehousing
Database
Databases
Databricks
Databricks Pyspark
Databricks Spark
DevOps
DevSecOps
Distributed Systems
Engineer
Engineering
ETL
GraphQL
Informatica
Information Technology (IT)
Infrastructure As Code
Microsoft Azure
Programming
Programming Language
Programming Languages
Security Automation
Security Devops
Snowflake
Software Architecture
Software Development
Software Engineering
Spark
SQL
Job Description
This Data Engineer role at Josh Pros LLC is an onsite position in Eden Prairie, MN, focused on Azure Databricks and hands-on delivery for financial-domain data pipelines. The position emphasizes scalable engineering, ETL/ELT development, advanced SQL performance, and support for AI/ML and stakeholder-facing outcomes.
Location and Schedule
- Work location: Eden Prairie, MN (onsite)
- Expected hours: 40.0 hours per week
Compensation
- Salary: USD 70.00 to 80.00 per hour
Core Responsibilities
- Develop services, controls, and reusable patterns that help the team deliver value safely, quickly, and sustainably in the public cloud.
- Build highly scalable applications for the Azure cloud using microservices and Azure Functions, incorporating patterns that support public cloud enablement and security.
- Support high-performance, collaborative technical work and help drive high-quality delivery.
- Stay current on product capabilities and translate them into functional changes that can be verified.
- Gather and analyze data to inform decision-making and provide detailed, realistic estimates.
- Work effectively with business stakeholders and third-party technical organizations.
- Design and develop ETL/ELT solutions on Azure Databricks using LakeBase and Spark.
- Design, build, optimize, and manage large-scale data pipelines to support analytics and machine learning feature generation, including insights generation and business intelligence dashboards.
- Write advanced and complex SQL and perform performance tuning and optimization.
- Build AI-based solutions for business process automation and improving operational efficiency.
- Identify opportunities to improve data reliability, data integrity, system efficiency, and overall data quality.
- Contribute to architectural evolution of data engineering patterns, frameworks, systems, and platforms, including defining best practices and standards.
- Mentor other data engineers by providing technical direction and teaching how to leverage cloud data platforms.
Required Experience
- Python and Scala: 7+ years of experience.
- Apache Spark: 5+ years of experience (PySpark and Spark SQL).
- DevOps automation: 3+ years of experience with Terraform.
- API design and lifecycle management: 3+ years (GraphQL, REST, etc.).
- Azure Databricks and cloud solutions: 3+ years building and deploying cloud-based solutions using Azure Databricks with UC, Snowflake, Azure Functions, and Service Bus.
- SQL: 5+ years, including designing complex data schemas and optimizing query performance.
- LLM integrations: Experience building LLM integrations for workflow automation and business needs.
- CI/CD and tools: Experience with CI/CD processes and tools including GitHub Actions, GIT, Artifactory, and Sonar.
Technology Stack
- Python, Scala
- Apache Spark, PySpark, Spark SQL
- Terraform
- GraphQL, REST
- Azure Databricks (including UC)
- Snowflake, Azure Functions, Service Bus
- SQL, microservices
- LLM integrations
- ETL/ELT, LakeBase
- AI/ML
- CI/CD: GitHub Actions, GIT, Artifactory, Sonar
Minimum Experience
- Minimum experience: 3 years