Software Engineer - Big Data (Compute/Data Distiller), Adobe Experience Platform
The Opportunity
Compute / Data Distiller
• Data Distiller
• Compute Gateway
- the multi-cloud, multi-engine compute layer that orchestrates data processing workloads with policy-based routing and support for long-running applications and jobs. This position is for a key contributor who will help design modern data architectures and highly distributed cloud solutions, and take them all the way to production.
What You'll Do
- Design, build and operate complex features in a large-scale, multi-cloud, microservices-based SaaS product.
- Own the outcome: lead application, framework and service design efforts, optimizing for quality, testability, scalability and performance.
- Work on query execution, Spark SQL and query optimization, and the orchestration of compute workloads across multiple engines and clouds.
- Prototype new approaches and deliver production solutions at scale; research new technologies and keep up with industry trends.
- Partner with Product Management, Design, Data Science and engineering teams across domains, including our US counterparts on a shared roadmap.
- Provide guidance in architecture, design and code reviews; mentor teammates and promote good engineering practices and continuous learning.
- Take part in on-call activities on a follow-the-sun rotation with the rest of the team.
- Contribute to open-source projects where relevant.
What You Need to Succeed
- B.S. in Computer Science or equivalent engineering degree, and/or 3+ years of relevant experience building software systems.
- Strong programming skills in Java and/or Scala, with a solid grasp of OOP, functional programming and design patterns.
- Proven expertise with distributed data processing: Apache Spark (including Spark SQL), Databricks, Delta Lake or equivalent Lakehouse technologies.
- Good understanding of SQL, query engines and query optimization, table formats and schema evolution.
- Hands-on experience with performance optimization, large-scale cluster tuning and efficient big data architectures.
- Experience building scalable, resilient backend services and REST APIs.
- Experience with cloud platforms (AWS and/or Azure).
- Demonstrated experience delivering and operating a high-quality SaaS solution: testing, CI/CD, monitoring, infrastructure as code.
- Experience implementing observability best practices (monitoring, logging, tracing) in distributed systems.
- Ability to quickly learn new technologies, with an emphasis on using AI coding agents and the agentic ecosystem to boost productivity across the software lifecycle.
- A problem-solving mindset, able to diagnose root causes, plus a sharp eye for security issues and performance bottlenecks.
- Commitment to quality in both code and testing.
- Very good written and verbal communication skills in English.
Nice to Have
- Experience with Kubernetes, Docker and infrastructure tooling (Terraform, ArgoCD).
- Experience with multiple compute engines or orchestration frameworks (e.g. Ray/Anyscale, Flink, YARN).
- Python, and experience building LLM-based agents or tools.
- Experience with Kafka, NoSQL databases (e.g. Cassandra, HBase) or Iceberg.
- Familiarity with PostgreSQL / JDBC protocols or BI tool integrations.
- Familiarity with monitoring tools such as Splunk, Prometheus, Grafana or New Relic.
- Experience with Agile / Scrum.
Why Join Us
- Solve hard distributed-systems problems at petabyte scale for some of the world's largest brands.
- Work with cutting-edge technology: Spark, Databricks, Delta Lake, Ray and AI agents.
- Direct impact on the whole Adobe ecosystem, since AEP powers Adobe's products.
- A growing team, a shared global roadmap and a follow-the-sun on-call model.
About Adobe
AI Use Guidelines for Interviews
Not included in the source posting: what you'll do, qualifications, benefits.
Skills
Who can apply
The employer didn't state any visa, work authorization, citizenship or clearance requirements in this posting. Confirm with the employer before applying.
Read automatically from the employer's posting text. Always confirm with the employer — requirements can change after a job is published.