Data Engineer & Architect · New York, NY

Shubham Malpani

Open to Senior Data Engineer

Data and AI leader with five years turning data into a strategic asset. I build the governed platforms and the AI layers that sit on top of them, so everyone in an organization, whatever their technical skill, has what they need to do their work well and make more informed decisions.

Email me Download Resume

Python
PySpark
SQL
Microsoft Fabric
Azure
Terraform
Power BI
MCP servers
Generative AI
Agentic AI
Medallion architecture
Data modeling
CI/CD

The short version

For five years I've built the infrastructure underneath other people's dashboards: the warehouses and pipelines that determine whether a number can be trusted. I like owning the full data lifecycle: the modeling, the ingestion, the transformations, the warehouse, the dashboards, and the governance that holds it all together.

I grew up in Mumbai, India, studied data science at Syracuse, and have spent the years since building data platforms for professional services and nonprofit firms in New York.

When I'm not working I'm usually cooking, boxing, or finding excuses to eat out.

The work I'm proudest of

Two flagship projects. Pick one to explore.

01Data Platform

An entire data platform, from scratch.

The company had a new CTO with a vision: build an enterprise technology platform for fiscal sponsorship. The centerpiece was a data warehouse to consolidate 21 disconnected source systems. There was no warehouse, no pipelines, no governance. I evaluated Fabric, Snowflake, and Databricks. Fabric won because it gave us the entire stack in one platform (notebooks, pipelines, warehouse, Power BI) within Azure. For a team that would shrink to one or two people after the build phase, operational simplicity mattered more than raw power.

I hired a team of 8, structured for the full scope: data engineers, DevOps, Power BI, business analysts, and a scrum master. We built a medallion architecture across three business domains and stood up 125 pipelines across 3 environments. The platform now serves 400+ users and runs every morning before anyone logs in.

21+ systems consolidated

400+ users served

98.2% pipeline success rate

125 pipelines across 3 envs

Interviewed stakeholders from CEO to operations staff to understand how each business unit thinks about data. This had never been done before.
Modeled three domains: Revenue, Cost, and Capacity. Revenue was the most complex because fiscal sponsorship and consulting recognize revenue differently. Cost was the hardest to learn (new domain for me). Capacity was the most strategically important (and eventually the one with the biggest business impact).
Produced a Kimball-style dimensional model: 12 conformed dimensions, 10 fact tables across 4 domains.
The completed data model was the first deliverable. It gave leadership confidence that we understood the business deeply enough to build the platform.

Built a config-driven PySpark framework: every source follows the same architectural pattern. Adding a new source means one notebook and a config entry, not a new codebase.
Each source has its own extraction logic (OAuth, Basic Auth, REST pagination, SOQL cursors, batch parquet), but the framework standardizes how data lands: MD5 content hashing for change detection, bidirectional schema evolution, and a 10% soft-delete safety threshold that prevents API failures from mass-deleting real data.
Parallel fetching with sequential writes (Spark isn't thread-safe), circuit breakers on high-volume APIs, and exponential backoff with retry.
21 source systems consolidated into one lakehouse: Salesforce, NetSuite, Paycor, OpenAir, HubSpot, Ignite, Active Directory, QuickBooks, Airtable, Certinia, and more.

Silver applies business rules and schema contracts. Every notebook starts with a PySpark StructType definition so the shape of the data is explicit, not inferred. Rows that fail validation get marked inactive with a reason code (never deleted).
Gold is where conformed business entities live. Config-driven stored procedure generation: define the entity and its rules, the system generates the T-SQL. SCD Type 4 (current + history + audit tables) tracks every column-level change across 11 entities.
Cross-system entity resolution with 10+ ID columns per entity (Salesforce ID, NetSuite ID, OpenAir ID, etc.). Deterministic surrogate keys via SHA-256.
Post-write reconciliation on every notebook. If counts don't match, the pipeline stops. Dollar-tolerance gates on financial data ($1.00 threshold).

36 KPIs across 5 domains (Finance, Capacity, Projects, Pipeline, Workforce Planning). Pushed back when different business units wanted different formulas for the same metric and drove a single unified calculation.
15+ Power BI dashboards for C-suite and line-of-business leadership, plus a self-service training program that cut ad-hoc requests by 40% (details in "What else shipped" below).
Automated report builders (6,400+ lines of Python): utilization scorecards, gross profit reports, and revenue scorecards. Each has built-in reconciliation gates; reports won't publish if the numbers don't tie.
Event-driven financial reporting that detects closed accounting periods and triggers downstream automatically, cutting turnaround from 5-6 days to same-day (details below).

Bronze: raw, as it landed

source	client_name	hours	rate	_content_hash
salesforce	aCme corp	42.0	NULL	a3f9…
salesforce	Acme Corp dup	42.0	NULL	a3f9…
ramp	Meridian LLC	16	210	77c1…
paycor	northstar health	-8 flag	185	e04d…

145 tables land here exactly as the source systems sent them. Nothing is edited. Every row gets a content hash, so the next run only moves what actually changed.

145 tables · 21 sources · hash-based change detection

Silver: cleaned and conformed

source	client_name	hours	rate	_content_hash
salesforce	Acme Corp	42.0	165	a3f9…
ramp	Meridian LLC	16.0	210	77c1…

northstar health · quarantined · negative hours · rule 7 of 10

19 conformed tables with 10 data-quality rules on the way in. Bad rows get quarantined with a reason code instead of silently dropped, so nothing disappears without a trace.

19 tables · 10 data-quality rules · quarantine, not deletion

Sample data. The traced record is highlighted in gold.

02AI + Data

The AI analytics layer, and the $7.6M it found.

Once the warehouse was running, the next problem was access. Non-technical leaders (the CFO, line-of-business heads) still couldn't get answers without asking the data team. Dashboards handled standard questions, but anything ad-hoc required someone who knew SQL.

I built an AI analytics layer on top of the warehouse using MCP servers and Claude. One server per data mart, each with access to its domain's schema and business context. Users ask questions in natural language, the model writes the SQL, and the answer comes back with the data behind it. The system handles 13,000+ queries a week at a 94.5% success rate.

The $7.6M came from asking a question nobody had thought to ask: how much billable capacity was the firm leaving on the table? The number was high enough that I assumed the AI was wrong. I validated everything by hand in raw SQL. The numbers held. The org used that data to reallocate staff and secure about $3M in new contracts.

$7.6M unrealized capacity identified

$3M in new contracts secured

94.5% query success rate

65 tools across 7 MCP servers

ask-the-data

Pick a question above. The answer shows up here, the way it did for the people who used the real thing.

Sample data, running in your browser. The real version sat on the live warehouse, behind the company's own auth.

How it actually works

Architecture. One MCP server per data mart: utilization, revenue, HR, contracts, and so on. Each server knows its domain's schema and the business context behind the columns (what a line-of-business code maps to in the real org, which join keys to use, which names map to which entities). Claude connects to all of them and routes each question to the right server.

Two-tool pattern. Each server started with six tools. Usage logs showed that Claude almost always did two things: check the schema, then write SQL. The other four added token overhead without value. Cutting to two tools per server made the model more accurate, not less.

Security. Every query passes through 8 validation layers before it touches the warehouse. The model can read data, not write or delete it, and only within tables it has been given access to.

Observability. Three-tier logging: a markdown file for humans, structured JSONL for scripts, and a SQLite archive for historical analysis. Weekly automated scans check for schema drift and query patterns.

Other things I built

Revenue Reconciliation System $200K recovered Read more →

The company ran a fiscal sponsorship business where revenue recognition follows grant-based rules: when funds come in for a client project, the company takes a management percentage and recognizes it over the service period.
Built a reconciliation system on Power Apps, Dataverse, and Logic Apps to replace the manual Excel process.
Ran both systems in parallel for 60 days. Discrepancies surfaced around revenue reversals that hadn't been properly propagated through the old process.
Finance traced the discrepancies and recovered approximately $200K in unbilled amounts.
The new system also expanded access from finance-only to 200+ users by adding a budgeted view (budgeted revenue vs. actual, at multiple aggregation levels).

Automated Employee Provisioning 93% faster onboarding Read more →

Automated the flow from Paycor (HRIS) into Active Directory and downstream systems using Logic Apps in a dedicated Azure resource group.
Modular component architecture: individual components (extract from Paycor, ingest to AD, write back to downstream systems) compose into larger processes (onboarding, status change, offboarding). Changing one component doesn't require changing the entire chain.
Two-tier AD setup: on-prem Domain Controller syncs to Azure Active Directory.
Handles onboarding, status changes, and offboarding as compositions of the same underlying components.

Automated Month-End Reporting 300+ analyst hours/month saved Read more →

Previously, consultants manually pulled data from each client's accounting system (QuickBooks, Sage Intacct, etc.) into Excel, applied formulas, and formatted. Repeated every month for every client.
Built Python scripts using CData unified connectors to extract from client systems, dump into a SQL layer, then produce formatted output in exactly the same layout the clients were used to.
Started with one client as a proof of concept. Scaled to 15 by word of mouth as other teams saw the time savings.
This is what I presented at the CData conference: how unified connectors replaced dozens of individual API integrations and accelerated reporting delivery.

AI Entity Resolution 25% → 5% rejection rate Read more →

No iPaaS tool (Celigo, Boomi, etc.) existed to integrate the CRM (Salesforce) and ERP (NetSuite). Records were supposed to be linked manually via an intake form, but people often skipped the step.
Two-pass matching: deterministic lookup first (if a shared key exists, it's already matched), then AI matching on entity attributes for everything else (handling variations like "Inc" vs "Incorporated," "Corp" vs "Corporate").
Human-in-the-loop approval: AI sends potential matches via email to the relevant domain owner. They reply to confirm or reject. Approved matches feed into the next pipeline run automatically.
Sat with each reviewer one-on-one to clear the initial backlog together. Once they saw that the ongoing burden was 5 rows instead of 25, adoption stopped being a problem.
Extended the same pattern to HRIS-to-ERP employee matching and contractor-to-FTE conversion tracking.

CI/CD Pipeline from Scratch 6 pipelines, 3 environments Read more →

Azure DevOps with Git-integrated Fabric workspaces. Dev/Test/Prod branching strategy with approval gates at each promotion.
6 YAML pipelines: dev sync, environment promotion (test or prod), cross-tenant repo mirroring, SQL object migration, Terraform plan, Terraform apply.
Deployment engine (5,448 lines of Python): recursive activity tree traversal for nested pipeline activities, fuzzy notebook name matching, cross-workspace artifact ID resolution, two-phase deployment with automatic service principal permission grants.
The harder part was the team transition: shifting from doing everything myself to reviewing and approving other people's work. That was as much a leadership shift as a technical one.

Event-Driven Financial Reporting 5-6 days → same day Read more →

Old process had two bottlenecks: finance had to manually notify the data team that the month was closed (sometimes days late), then the team had to manually pull and combine reports.
Found NetSuite's accounting time period table, which tracks open vs. closed months. The pipeline queries it nightly. When it detects a new closed month, everything downstream triggers automatically.
The signal is completely reliable: if finance hasn't closed the month, the table doesn't change, and the pipeline doesn't fire. Zero misfires.
Revenue reporting layer triggers at 8am. Reports are ready before anyone's at their desk.

Infrastructure as Code (IaC) 84 .tf files, 3 environments Read more →

Modules cover: Fabric workspaces, warehouses (with binary collation for faster string comparisons), lakehouses, capacity units, Key Vault with 3 access policy variants, Azure AD service principals, storage accounts, user-managed identities, Purview, and Power BI workspace assignments.
4 service principals with distinct scopes following least-privilege: CI/CD (admin), BI refresh (member), data ingestion (blob access, prod only), metadata scanning (viewer only).
3 environments (Dev/Test/Prod) with separate backend configs. ~1,500 lines across 84 .tf files. Zero hardcoded credentials anywhere.
Originally built so the platform could be replicated for other client organizations. That replication vision is why IaC mattered from day one.

Conference Talk: Data Integration at Scale CData Conference, 60+ attendees Read more →

CData invited me to present because our use case was distinctive: nonprofits serving nonprofits, reporting across a dozen different client accounting platforms (QuickBooks, Sage Intacct, etc.).
The talk covered how their unified connector layer replaced dozens of individual API integrations and accelerated client onboarding and report delivery.
This talk was about data integration and operational efficiency, not AI or MCP servers.
Audience: enterprise data teams across 15+ client organizations.

Self-Service BI Training Program 20+ users, 40% fewer ad-hoc requests Read more →

8 workshops over 2 weeks: 4 sessions, 2 groups of ~10 people each.
Used live, in-progress dashboards with real data. People saw their own numbers for the first time, which generated genuine engagement.
The sessions doubled as feedback rounds on the dashboards themselves. Users told us what was wrong, what was confusing, and what they actually needed. We shipped fixes between sessions.
Ad-hoc data requests dropped 40% afterward. The goal wasn't fewer questions; it was people who could answer their own.

Things I built for myself

Humanizer web app

[one line: what it does]

Anthropic API[stack]

[url]

Crystalline Emergence

[one line]

p5.jsgenerative art

[url]

Apple Watch health dashboard

[one line]

[stack]

[url]

How I got here

2016

B.E. Electronics & Telecom

University of Mumbai
2020

Diagnosed ETL pipeline failures across production systems, wrote operational runbooks, and cut data load times by 40%.

SQLETLPython
2022

M.S. Information Management, Certificate in Data Science

Syracuse University
2022

One company through two acquisitions. Three roles, each building on the last:

2022Data AnalystKiwi Partners

First data hire. Defined how the company measured profitability and built the reporting systems that leadership relied on for financial decisions. Automated the manual workflows that were eating analyst time every month-end.

2023Associate Director, Data Engineering & AnalyticsArabella Advisors

Hired and led a team to build the company's data platform from the ground up. Chose the architecture, stood up engineering practices from version control to CI/CD, and delivered governed data access to the entire organization.

2025Data ArchitectVital Impact Group

Designed the AI analytics layer that let non-technical leaders query the warehouse directly. Built the security and governance framework underneath it, then trained the organization on self-service analytics.

Microsoft FabricPySparkMCPPower BITerraformLeadership
2025

M.S. AI in Business

Harrisburg University

My data engineering principles

Flag it, never delete it.

When a row fails validation, it gets marked inactive with a reason code. If an API returns partial data or a permission changes upstream, the warehouse doesn't lose records it can't get back. Deleting data you might need later is a one-way door.

Human in the loop, always.

When we started with AI analytics, we validated every significant finding by hand in raw SQL before trusting it. The query logs then showed us what people actually asked and where the model struggled. Trust got built incrementally, not assumed.

Move only what changed.

Every row gets a content hash. Only rows where the hash differs move to the next layer. A weekly full reload catches anything that slipped through. Full nightly reloads are a workaround for not knowing what changed.

Train people, don't be their bottleneck.

I ran workshops on the live data people would actually use. Ad-hoc requests dropped 40%. The goal isn't fewer questions; it's people who can answer their own.