Posts

Featured Post

All about SnowLens - Snowflake observability and anomaly detection tool

-
Snow Lens Free Snowflake Observability -- 9 Detectors, Zero Data Egress 100% Free Runs Inside Snowflake 9 Detection Rules No Outbound Calls If you are running Snowflake in production, you have probably asked yourself: Which queries are burning credits? Why did costs spike last Tuesday? Are any warehouses just sitting there idle? Most observability tools answer these questions by shipping your query logs to an external SaaS -- which means data egress, vendor lock-in, and yet another subscription. SnowLens takes a fundamentally different approach: it is a Streamlit app that installs directly into your Snowflake account and runs on your own compute. Your data never leaves your environment. And it is completely free -- no payment, no signup, no feature gating, no expiry. In This Post How SnowLens Works The 9 Detection Rules (with Logic) Privacy and Security Model Memory Safety for Large Accounts How to Inst...
Post a Comment
Read more

DATADLE: A Daily Word Game for Data Engineers

-
Post a Comment
Read more

Data Engineering Fundamentals, Part 5: Data Visualisation Fundamentals

-
Image
This is the final post in a 5-part series on data engineering fundamentals. Part 4 covered data warehousing with Snowflake , . If you've followed the whole series, you've now gone from raw SQL and Python, through ELT pipelines, into a properly modeled warehouse, and now to the layer where people actually see and use the data. Why visualization is the last mile that matters most You can build a technically flawless pipeline and a perfectly modeled warehouse, but if the person making a business decision can't see the answer clearly and quickly, none of that engineering work translates into value. Visualization is where data engineering meets decision-making. Core concepts that apply across every tool Dimensions vs measures. Dimensions are the categorical fields you slice and group by, region, product, date. Measures are the numeric fields you aggregate, sales, count, average order value. Every visualization tool is built around this same distinction, understand it once and it...
Post a Comment
Read more

Data Engineering Fundamentals, Part 4: Data Warehousing and Why Snowflake Stands Out

-
Image
  This is Part 4 of a 5-part series on data engineering fundamentals. Part 3 covered ETL/ELT concepts . Part 5, the final post, covers data visualisation tools. What a data warehouse actually is? A data warehouse is a system optimized for reading and analyzing large volumes of structured data, as opposed to an operational database (OLTP) optimized for fast, small, transactional reads and writes. Where OLTP systems power an app's checkout process one order at a time, an OLAP-style warehouse is built to scan millions of rows and answer "what were our total sales by region last quarter" quickly.OLTP system will be always write optimised whereas OLAP read optimised. Core data warehousing concepts OLAP vs OLTP. Online Analytical Processing systems are designed for complex queries across large datasets, online Transaction Processing systems are designed for fast, frequent, small transactions. Data engineers typically move data from OLTP sources into an OLAP-style warehouse. Sta...
Post a Comment
Read more

Data Engineering Fundamentals, Part 3: ETL/ELT Concepts Every Data Engineer Should Understand

-
Image
  This is Part 3 of a 5-part series on data engineering fundamentals. Part 2 covered Python basics , link at the top once published. Part 4 moves into data warehousing with a focus on Snowflake. ETL vs ELT, and why the difference matters ETL, Extract, Transform, Load, means data is transformed before it reaches the warehouse, usually on a separate processing server. ELT, Extract, Load, Transform, loads raw data into the warehouse first and transforms it there using the warehouse's own compute. Most modern stacks have shifted to ELT because cloud warehouses like Snowflake are powerful and cheap enough to handle transformation directly, and it keeps a copy of raw, untransformed data available if your transformation logic needs to change later. I'll use Matillion as the example tool throughout this post, since it's a widely used ELT platform, but these concepts apply across tools like Fivetran, Airbyte, or Azure Data Factory too. Core concepts that matter regardless of tool Ex...
Post a Comment
Read more

Data Engineering Fundamentals, Part 2: Python Basics Every Data Engineer Should Know

-
Image
  This is Part 2 of a 5-part series on data engineering fundamentals. Part 1 covered SQL , . Part 3 moves into ELT/ETL tools and core data engineering concepts. Why Python, on top of SQL SQL is excellent at querying and transforming structured data that already lives in a database. But it can't call an API, loop through files in a folder, handle complex conditional logic cleanly, or glue together multiple systems. That's where Python comes in. You don't need to become a software engineer, but you do need to be comfortable enough to automate the parts SQL can't reach. Core data structures you'll use constantly Lists and dictionaries. Most of the data you'll manipulate in Python outside of a dataframe will move through lists and dicts, records as dictionaries, collections of records as lists of dictionaries. Get comfortable with this pattern early. List and dict comprehensions. Instead of writing a for loop to filter or transform a list, a comprehension does it in...
Post a Comment
Read more

Data Engineering Fundamentals, Part 1: Why SQL Is Still Your Most Important Skill

-
Image
  This is the first post in a 5-part series on data engineering fundamentals. We'll go from SQL, to Python, to ELT/ETL tools, to data warehousing with Snowflake, and finally to visualisation. Part 2 covers Python basics, link at the bottom. Why SQL, even in 2026? Every modern data platform, Snowflake, Databricks, Fabric, Matillion, dbt, still expects you to think in SQL underneath the interface. Cloud platforms change every few years. SQL has stayed the primary way to query and reason about structured data for decades. If you only master one skill in data engineering, make it this one. Important concepts you actually need to know Joins, properly understood. Not just INNER vs LEFT, but what happens to row counts when a join key isn't unique on one side, and why an unexpected LEFT JOIN can silently multiply your row count. Window functions. ROW_NUMBER, RANK, DENSE_RANK, and aggregate functions used with OVER() let you calculate running totals, rankings, and comparisons across row...
Post a Comment
Read more