Data Engineering

An AI Agent Turned a Net $784K Into $5.3M of Debt. One Diff Catches It.

We Specified an AI Agent in 2016. We Called It Analytics.

Re-reading my own IBM patent on real-time concert feedback as what it actually was: an agentic loop with multimodal perception, memory-grounded inference, and a human in the loop. What it got right, and what a decade of hindsight teaches agent builders today.

Building a Production-Grade Dimensional Model on Snowflake with Cortex Code using Skills

How I designed, audited, and expanded an e-commerce star schema using an AI-powered skill — and what I learned about letting the machine enforce Kimball principles.

Part XI — RDD vs DataFrame vs Dataset: Why Your “Type-Safe Refactor” Made Spark 3x Slower

Part XI — RDD vs DataFrame vs Dataset: Why Your “Type-Safe Refactor” Made Spark 3x Slower Three APIs, two execution worlds — how Catalyst and Tungsten treat structured plans, why typed …

NEAREST BY Join: Scaling Vector Search in Databricks Runtime

Vector search originated as a serving problem. The classical use case is a chatbot...

Data Architecture Masterclass For Beginners (By Ansh Lamba)

Data Architecture | Data Architecture For Beginners | Data Engineering What you will learn? This is a complete 3 hours long Data Architecture Masterclass which covers all the concepts including - Data Maturity Levels, Architecture Design Session, Data Architecture Evolution, Data Architecture…

Your AI Isn’t Hallucinating. Your Data Is Lying to It.

Most of what we call hallucination is a model faithfully repeating data we never checked

Meet Jev: The AI Model That Doesn’t Write, and Why Data Engineers Should Care

A plain-English guide to “System One” models, with patterns, code, and honest caveats

OneLake vs. Traditional Data Lakes: A Practical Migration Checklist

Microsoft OneLake is not simply another storage account replacing a traditional data lake. It is the unified logical data lake underlying Microsoft Fabric.

Scraper-CI: Building a Web Data Reliability & Crawl-Governance Control Plane With Bright Data

How I built Scraper-CI, a web data reliability control plane for profiling, governing, validating, diagnosing, and recovering web acquisition.

From Hours to Minutes: Automating Data Debug with Multi-agent Systems

From Hours to Minutes: Automating Data Debug with Multi-agent Systems The Data Debugging Grind My engineers typically spend 30% of their time debugging data. On some days, that number climbs to 50% …

I Compacted 1,000 Apache Iceberg Files Into 6. Here’s What Happened to Query Performance.

Benchmarking the impact of fewer, larger files across three SQL workloads The post I Compacted 1,000 Apache Iceberg Files Into 6. Here’s What Happened to Query Performance. appeared first on Towards Data Science.

Future of Data Engineers After GPT-6 Astra (True Talk By Ansh Lamba)

Future of Data Engineers | Data Engineering | GPT-6 Astra | AI Data Engineer What you will learn? This video talks about the key areas where Data Engineers need to focus on after GPT-6 Astra as this is a powerful model that can perform a lot of Data Engineering tasks.

50 Data Engineering Terms EXPLAINED For Your Interview

Data Engineering | Data Engineering Fundamentals | Data Engineer Interview | Data Engineering Terms For Beginners What you will learn? This video covers 50+ Data Engineering Terms that are essential for anyone in the Data Domain.

New Role For Data Domain - Forward Deployed Engineer (FDE)

Forward Deployed Engineer Role For Data Domain | Data Engineering | AI Engineering | Data Anlytics What you will learn? This video talks about the new role "Forward Deployed Engineer" for the data domain.

Data Domain Is NOT Cooked

Data Engineering | Data Analytics | Data Science | AI Engineering What you will learn? This video will help you know where the Data Domain is heading in the AI era and how Data Roles like Data Engineers, Data Analysts, Data Scientists and more can actually grow with the right direction.

Data Engineering 101 - The Beginner's Guide [2026]

Data Engineering | Data Modeling | Data Engineer | Databricks | Snowflake | Azure Data Engineer | PySpark | Apache Spark What you will learn? This Data Engineering Masterclass is designed to take you from the fundamentals to advanced concepts, making it ideal for both complete beginners and…

Escape The HERD MENTALITY In Data Domain

Data Engineering | Data Science | Data Analytics What you will learn? This video will give you the right direction and tell you what to do and what not to do in order to grow in the data domain.