SDE III · Walmart Global Tech · Bengaluru

Deepanshu Maheshwari
builds the pipes data runs through.

Data platform engineer building batch and streaming pipelines end to end: Spark/Scala and Python/Airflow jobs, Kafka streaming into BigQuery and Cassandra, and data-quality validation, plus the design docs, phased rollouts and incident response that keep them reliable.

topic: catalog-translations4 partitions · idempotent consumer
committed offset 12,79,400 replays deduplicated 0 consumer lag 14
~6×Kafka consumer throughput, 1.3K → 8K rec/s
200M+records per Cassandra publish run
1.28Mcatalog items translated over async Kafka
199,917publishable pages restored from a 4.1M-record pipeline
3 → 1market pipelines unified into one config-driven job
7services moved JDK 8 → 17, zero downtime
Systems log

Selected systems

offset 0011SDE III

Real-time streaming ingestion

An always-on Python + Kafka → BigQuery pipeline over two independent regional clusters. I owned its 16-entry architecture decision log covering delivery semantics, per-cluster consumers, cost-driven storage scope and privacy. At-least-once processing comes from ack-after-write offset commits and idempotent insert IDs; client IP and cookies are never persisted, dead letters included.

Consumer throughput up ~6× (1.3K → 8K rec/s) by tuning the Kafka TCP receive window.

PythonKafkaBigQueryStream processing
offset 0010SDE III

One pipeline for three markets

Canada, Mexico and the US each had their own alt-text code path. As sole owner of the epic, I wrote the design and success criteria, replaced all three with one config-driven Spark job, and rolled it out through dual-write, shadow validation, cutover and legacy decommission.

Adding a new market is now a configuration change, not new code.

SparkScalaConfig-driven design
offset 0009SDE III

Finding the missing topic pages

Topic-page volume collapsed in a 4.1M-record Spark pipeline. I instrumented a seven-stage validation funnel, which traced the loss to over-strict filters. I then added fail-fast checks that block runs on incomplete vendor deliveries, and wrote the pipeline's architecture doc and prioritized roadmap.

Output restored to 199,917 publishable pages; eligible keywords grew from 221,618 to 435,846.

SparkData qualityValidation
offset 0008SDE III

200M-record Cassandra publisher

Spark/Scala jobs on GCP Dataproc scaling to 50 executors, including a Cassandra publisher. A write bottleneck turned out to be client-side overhead, not cluster capacity: prepared-statement reuse, batch tuning and bounded in-flight writes raised sustained throughput while protecting read latency.

200M+ records per publish run.

ScalaSparkCassandraDataproc
offset 0007SDE III

AI content serving on a multi-tenant model

The production path delivering AI-generated image alt text to Canada item pages: Spark ingestion into Cassandra on a multi-tenant schema with composite partition keys, so every tenant or locale read is a single-partition lookup. It is served through federated GraphQL, with DGS BatchLoader providers that eliminate N+1 reads.

3.2M records live for Canada.

CassandraGraphQL FederationNetflix DGSSpark
offset 0006SDE III

Incident ownership and monitoring

A consumer crash loop traced to oversized BigQuery streaming inserts. I fixed it with byte-capped batching, wrote an incident review with measured cost, and added a dedup view. I also shipped isolated health probes and Cloud Monitoring alerts routed to email and Slack, then tuned the alert noise.

Crash loop resolved, with a written incident review and live alerting.

BigQueryCloud MonitoringIncident review
5 more systems
offset 0005SDE III

Event-driven translation service

A rate-limited synchronous REST integration was holding catalog translation back. I replaced it with asynchronous Kafka messaging, built in Java 17 / Spring Boot as three independently runnable phases: BigQuery ingest, async produce, and consume with a MERGE back. Bulk record locking, Secret Manager-backed mTLS, completion tracking and graceful shutdown keep each phase safe.

1.28M catalog items sent for English → French translation.

Java 17Spring BootKafkaBigQuery
offset 0004SDE III

Zero-downtime migrations

Led the JDK 8 → 17 migration of seven services, and moved a Kafka service from mTLS to SPIFFE workload identity.

Both completed with zero downtime.

Java 17KafkaSPIFFE
offset 0003SDE III

JVM and Kubernetes tuning

Production pods were OOMKilling. Heap-dump and GC analysis showed payload growth rather than a leak, so the fix was retuning heap, container limits and autoscaling.

CPU allocation scaled from 120m to 2300m as traffic ramped.

JVMKubernetesGC analysis
offset 0002SDE I–II

Advertising data platform

Built and productionized the DSP and EA advertising Python/Airflow pipelines with dynamic DAG generation, dynamic Dataproc sizing and execution timeouts. Refactored the Spark audience-segmentation service with retries, job metrics and Prometheus alerts to Slack.

62 PRs on the Airflow pipelines and 32 on audience segmentation.

AirflowPythonSparkDataproc
offset 0001SDE I–II

Retail analytics and GCP infrastructure as code

Integrated internal config and secrets services across UK grocery item, order and marketing Scala/Spark aggregations, and established GCP infrastructure as code for IAM, service accounts and VPC Service Controls.

120+ commits of infrastructure as code.

ScalaSparkGCPIaC
Career

~5 years at Walmart Global Tech, three levels

Software Engineer III · Walmart Global Tech

Aug 2024 – present

Owns batch and streaming pipelines end to end: Kafka streaming into BigQuery, config-driven Spark jobs, Cassandra publishing and GraphQL serving for AI-generated content, along with design docs, phased rollouts and incident response. Bravo Award, Q2 2025.

Software Engineer I & II · Walmart Global Tech

Jan 2022 – Jul 2024

Built the DSP and EA advertising Python/Airflow pipelines and refactored the Spark audience-segmentation service; integrated config and secrets services across UK grocery Scala/Spark aggregations; established GCP infrastructure as code.

B.Tech, Information Technology · IIIT Lucknow

Aug 2018 – Jul 2022 · GPA 8.0/10

Data structures and algorithms, distributed systems, operating systems, database management and machine learning.

Stack

Tools used in production

Languages

Python, Scala, Java, SQL, Bash

Data & streaming

Spark, Kafka, Airflow, BigQuery; stream and batch processing, data-quality validation, event-driven architecture

Databases & storage

Cassandra (composite-key schema design, LCS compaction), Hive, PostgreSQL, Trino, Azure Cosmos DB, GCS

Backend

Spring Boot, GraphQL (DGS, Federation), REST APIs, microservices, concurrent programming

Cloud & DevOps

GCP (Dataproc, GCE, GCS, BigQuery, IAM, VPC Service Controls, Secret Manager), Azure, Docker, Kubernetes, Istio, SPIFFE, IaC, CI/CD (GitHub Actions, Jenkins, Looper, Concord)

Observability

Cloud Monitoring, Prometheus, Grafana, Dynatrace, OpenObserve

AI/ML integration

LLM productionization (Claude, GPT-4o, Gemini, internal models), LLM cost evaluation

Contact

Open to backend and data engineering roles.

LinkedIn GitHub