Lead PySpark / SQL Engineer — Threat Detection & Databricks

Job overview

The Lead PySpark / SQL Engineer — Threat Detection & Databricks role is onsite in San Francisco, CA, with compensation of $76.55–$129.16 per hour. The role supports the Threat Detection team’s migration of detection rules and content to Databricks.

What you'll do

• Design and build threat-detection pipelines using PySpark, SQL, and Databricks
• migrate detection rules and content from a legacy platform
• maintain log pipelines and rule configuration files
• write unit tests and data validation checks
• deploy pipelines and infrastructure using Terraform
• optimize workflows and queries
• and collaborate with cross-functional teams on data requirements and detection objectives.

What we're looking for

Skills & competencies

lead pyspark engineer

sql engineer

threat detection

databricks

san francisco ca

$76.55–$129.16/hr

data engineering

log analytics

terraform

pyspark

detection pipelines

unit testing

data validation

Work arrangement

• Weekend coverage required

Why this role

The posting seeks 5–10 years of PySpark/SQL engineering experience, threat-detection or log-analytics pipeline experience, and hands-on Terraform experience. AWS or Azure familiarity is preferred.

About the employer

Macpower Digital Assets Edge Private Limited is hiring for this role. Industry: Engineering Services. Sector: 54.