AK Ajay Kumar Let's talk

Data Engineer · Pune, India

I turn complex data into reliable pipelines.

I build scalable batch and near-real-time data pipelines on GCP using Python, SQL, PySpark and BigQuery. My work focuses on reliable ingestion, data quality, workflow orchestration and delivering trusted data for reporting and analytics.

Explore my work Email me
PIPELINE / PRODUCTION LIVE
01REST APIsJira · Confluence
02Enterprise dataSharePoint · CodeBeamer
ingest + validate
AF
ORCHESTRATIONCloud ComposerRetries · SLA · Backfills
RUNNING
transform + model
DATA WAREHOUSEBigQueryPartitioned · Clustered · Tested
BI Power BI
SF Spotfire
SQL Analytics
PYTHONBIGQUERYPYSPARKGCPAIRFLOWSQL
01 Professional Summary

About Me

I’m a Data Engineer around 4 years of experience building and supporting data pipelines on Google Cloud Platform. I work mainly with Python, SQL, PySpark, BigQuery, Dataflow, Pub/Sub and Cloud Composer.

My experience includes integrating data from REST APIs, relational databases and enterprise platforms such as Jira, Confluence, SharePoint and CodeBeamer. I have also worked on incremental loading, BigQuery optimisation, data-quality checks and production monitoring.

I enjoy troubleshooting pipeline issues, improving data reliability and preparing datasets that engineering, analytics and reporting teams can confidently use.

4
Years of Experience
10+
Data Pipelines
4+
Enterprise Integrations
5+
Dashboards
View Experience View Projects
⌖
BASED INPune, India
02 Technical toolkit

Tools I use to move data from source to insight.

A production-focused toolkit spanning ingestion, processing, warehousing, orchestration, quality, and analytics.

01

Programming & processing

Python · SQL · PySpark · Pandas · NumPy · Spark SQL · CTEs · Window Functions

02

Google Cloud

BigQuery · Dataflow · Cloud Storage · Pub/Sub · Cloud Functions · IAM · Cloud Logging

03

Orchestration & integration

Cloud Composer · Apache Airflow · Control-M · REST APIs · Pagination · Incremental Loading

04

Big data & streaming

Apache Spark · Kafka · Batch Processing · Hadoop · Hive · Medallion Architecture

05

Modeling & performance

Star Schema · Snowflake Schema · SCD Type 2 · Partitioning · Clustering · Materialized Views

06

Quality & analytics

Pytest · Data Validation · Deduplication · Referential Integrity · Power BI · Spotfire · Git · CI/CD

03 Projects

Featured Data Engineering Projects

Representative solutions covering ingestion, orchestration, transformation, data quality and analytics delivery. Project details are intentionally anonymised.

01 / GCP PIPELINE Batch + near-real-time

GCP Batch and Near-Real-Time Data Pipeline

Designed a scalable GCP pipeline that ingests batch and near-real-time data, validates and transforms it, and loads analytics-ready datasets into BigQuery.

PythonSQLPySparkBigQueryDataflowPub/SubCloud Composer
View Details

Business problem

Reporting teams needed dependable access to data arriving through scheduled extracts, relational systems and event-driven sources without maintaining separate one-off workflows.

Architecture

  1. REST APIs / relational sources / Pub/Sub
  2. Python ingestion
  3. Cloud Storage
  4. Dataflow
  5. BigQuery
  6. Power BI / Spotfire

Key responsibilities

  • Developed reusable Python ingestion components with authentication, pagination and incremental extraction.
  • Created Cloud Composer DAGs for scheduling, dependencies, retries and failure notifications.
  • Used Dataflow and PySpark for transformation and applied BigQuery partitioning and clustering.
  • Added structured logging, data-quality validation, rejected-record handling and production monitoring.

Outcome

Improved availability of key reporting datasets by reducing processing time from around four hours to under three hours.

02 / ANALYTICS Automotive data engineering

Vehicle Quality Data Platform

Built scalable batch and incremental pipelines on GCP to transform vehicle quality and manufacturing data into analytics-ready datasets for quality reporting and decision-making.

PythonSQLPySparkBigQueryGCPAirflowREST APIsPower BI
View Details

Business problem

Vehicle quality and manufacturing data was distributed across operational systems and APIs, with duplicates, missing values, inconsistent formats and schema changes delaying reporting and increasing processing costs.

Project overview

Developed a cloud-based vehicle quality platform at KPIT Technologies for the automotive / Renault domain, integrating vehicle, manufacturing, component, plant, supplier, defect, warranty and corrective-action data for trusted quality analytics.

Architecture

  1. REST APIs / source systems
  2. Python extraction
  3. Google Cloud Storage
  4. BigQuery raw
  5. Standardized
  6. Transformation
  7. Curated
  8. Power BI / Spotfire

Orchestration: Cloud Composer / Apache Airflow.

Data pipeline

  1. Composer trigger
  2. REST API extraction
  3. Pagination
  4. GCS raw files
  5. BigQuery staging
  6. SQL / PySpark transformation
  7. Deduplication
  8. BigQuery MERGE
  9. Quality validation
  10. Curated analytics

Cloud Logging and Cloud Monitoring provide execution visibility and operational alerts.

Incremental processing

Implemented watermark-based ingestion using updated timestamps, modified dates, created dates or the maximum processed ID.

  1. Previous watermark
  2. Incremental API extraction
  3. GCS
  4. BigQuery staging
  5. Validation
  6. MERGE to target
  7. Update watermark

Data quality

  • Duplicate detection
  • Null validation
  • Data-type validation
  • Date-format validation
  • Primary-key uniqueness
  • Referential integrity
  • Accepted values
  • Source-to-target reconciliation
  • Freshness checks

BigQuery optimization

  • Partitioning and clustering
  • Incremental processing
  • Partition pruning
  • Optimized joins
  • Explicit column selection
  • Pre-aggregated datasets
  • Reduced table scans
  • MERGE-based loading

Production support

Monitored Airflow DAGs and investigated API, authentication, schema, BigQuery load, duplicate-key, SQL, source-data, timeout and resource failures. Used retries, selective reruns, backfills and detailed logging to recover workflows.

Key responsibilities

  • Developed Python REST API ingestion with authentication, pagination, retries, error handling and incremental extraction.
  • Built SQL and PySpark transformations across raw, standardized, transformation and curated BigQuery layers.
  • Implemented watermark logic and BigQuery MERGE for new and updated records.
  • Added automated quality checks for duplicates, nulls, schemas, integrity, freshness and reconciliation.
  • Orchestrated and monitored production pipelines with Cloud Composer, Airflow, Cloud Logging and Cloud Monitoring.

Outcome

Automated vehicle-quality processing, improved data reliability and availability, optimized BigQuery workloads, and enabled faster analysis of defects, component issues, manufacturing trends and quality KPIs.

  • Processed approximately 1–2M records and 100–200 GB of data daily.
  • Reduced processing cost by roughly 25–35%, improved execution time by around 30%, and cut manual reporting effort by approximately 50–60%.
03 / DATA QUALITY Reusable validation

Automated Data Quality Framework

Developed reusable validations that check data before and after pipeline processing, isolate invalid records and support controlled reprocessing.

PythonSQLPytestBigQueryCloud ComposerCloud LoggingGit
View Details

Business problem

Late discovery of incomplete or inconsistent data increased reconciliation effort and allowed avoidable issues to reach downstream reports.

Validation flow

  1. Source data
  2. Validation rules
  3. Valid / rejected records
  4. BigQuery
  5. Monitoring and reprocessing

Validation checks

  • Completeness and null values
  • Duplicate detection
  • Schema and data types
  • Referential integrity
  • Source-to-target reconciliation
  • Record-count validation

Key responsibilities

  • Developed reusable validation functions with Python and Pytest.
  • Stored failed records separately for investigation and controlled reprocessing.
  • Added structured error logs, validation summaries and checks to scheduled pipelines.
  • Made validation results available for production monitoring and support.

Outcome

Reduced manual reconciliation effort and helped identify data issues before they reached downstream reports.

04 Experience

Professional Experience

Production data engineering across pipeline development, enterprise integration, analytics delivery and data-quality operations.

OCT 2025PRESENT

ACCENTURE

Data Engineer

Current role

Pune, India

  • Design and maintain batch and near-real-time data pipelines on GCP using Python, BigQuery, Dataflow, Cloud Storage and Pub/Sub, processing approximately 80-150 GB of data daily.
  • Build reusable Python ingestion components for REST APIs and relational sources, covering authentication, pagination, incremental extraction, schema validation, retries and error logging.
  • Develop Cloud Composer DAGs for scheduling, dependency management, incremental loads, backfills, automated retries and failure notifications.
  • Optimise BigQuery transformations using CTEs, window functions, MERGE, partitioning and clustering, reducing processing time for key datasets from around four hours to under three hours.
  • Implement automated data-quality checks for completeness, duplicates, schema changes, referential integrity and source-to-target reconciliation.
  • Monitor production pipelines through Cloud Logging and Cloud Monitoring, investigate failures and automate recurring validation and monitoring activities.
PythonSQLPySparkGCPBigQueryDataflowPub/SubCloud StorageCloud ComposerAirflow
JAN 2023SEP 2025

KPIT TECHNOLOGIES

Data Engineer

Pune, India

  • Built and maintained 8-10 Python and SQL-based ETL pipelines for ingesting and transforming enterprise data in BigQuery.
  • Integrated approximately 1-2 million records daily from Jira, Confluence, SharePoint and CodeBeamer using Python and REST APIs.
  • Developed reusable extraction workflows supporting API authentication, pagination, incremental loading, schema mapping and error handling.
  • Optimised BigQuery queries using CTEs, window functions, partitioning and clustering, reducing refresh time for key reporting datasets from 6-8 hours to 4-5 hours.
  • Developed and supported 5-7 Power BI and Spotfire dashboards for project-health, quality and performance reporting.
  • Automated Jira extraction, validation and reporting processes using Python, reducing manual data preparation and reporting effort by around 50%.
PythonSQLGCPBigQueryREST APIsPandasPower BISpotfireJiraConfluenceSharePointCodeBeamer
JUL 2022DEC 2022

KPIT TECHNOLOGIES

Data Engineer Intern

Pune, India

  • Developed Python scripts to automate data cleansing, formatting and validation for datasets collected from internal repositories.
  • Created and optimised SQL queries for completeness, accuracy, consistency and duplicate-record checks used in engineering quality reporting.
  • Prepared, modelled and validated datasets for Power BI and Spotfire dashboards while supporting data-issue investigation and Git-based version control.
PythonSQLPandasPower BISpotfireGit
05 Credentials
EDUCATION

B.Tech in Computer Science & Engineering

Dr. A. P. J. Abdul Kalam Technical University, Lucknow · 2022

LANGUAGES

Hindi · English

Professional communication across engineering and business teams.

Have a data challenge?

Let's build something reliable.

I'm always open to discussing data engineering opportunities, scalable pipelines, and analytics platforms.

ajaykumar2dev@gmail.com +91 72660 09048