VizCanvaz

This is the final post in a 5-part series on data engineering fundamentals. Part 4 covered data warehousing with Snowflake , . If you've followed the whole series, you've now gone from raw SQL and Python, through ELT pipelines, into a properly modeled warehouse, and now to the layer where people actually see and use the data. Why visualization is the last mile that matters most You can build a technically flawless pipeline and a perfectly modeled warehouse, but if the person making a business decision can't see the answer clearly and quickly, none of that engineering work translates into value. Visualization is where data engineering meets decision-making. Core concepts that apply across every tool Dimensions vs measures. Dimensions are the categorical fields you slice and group by, region, product, date. Measures are the numeric fields you aggregate, sales, count, average order value. Every visualization tool is built around this same distinction, understand it once and it...

  This is Part 4 of a 5-part series on data engineering fundamentals. Part 3 covered ETL/ELT concepts . Part 5, the final post, covers data visualisation tools. What a data warehouse actually is? A data warehouse is a system optimized for reading and analyzing large volumes of structured data, as opposed to an operational database (OLTP) optimized for fast, small, transactional reads and writes. Where OLTP systems power an app's checkout process one order at a time, an OLAP-style warehouse is built to scan millions of rows and answer "what were our total sales by region last quarter" quickly.OLTP system will be always write optimised whereas OLAP read optimised. Core data warehousing concepts OLAP vs OLTP. Online Analytical Processing systems are designed for complex queries across large datasets, online Transaction Processing systems are designed for fast, frequent, small transactions. Data engineers typically move data from OLTP sources into an OLAP-style warehouse. Sta...

  This is Part 3 of a 5-part series on data engineering fundamentals. Part 2 covered Python basics , link at the top once published. Part 4 moves into data warehousing with a focus on Snowflake. ETL vs ELT, and why the difference matters ETL, Extract, Transform, Load, means data is transformed before it reaches the warehouse, usually on a separate processing server. ELT, Extract, Load, Transform, loads raw data into the warehouse first and transforms it there using the warehouse's own compute. Most modern stacks have shifted to ELT because cloud warehouses like Snowflake are powerful and cheap enough to handle transformation directly, and it keeps a copy of raw, untransformed data available if your transformation logic needs to change later. I'll use Matillion as the example tool throughout this post, since it's a widely used ELT platform, but these concepts apply across tools like Fivetran, Airbyte, or Azure Data Factory too. Core concepts that matter regardless of tool Ex...

  This is Part 2 of a 5-part series on data engineering fundamentals. Part 1 covered SQL , . Part 3 moves into ELT/ETL tools and core data engineering concepts. Why Python, on top of SQL SQL is excellent at querying and transforming structured data that already lives in a database. But it can't call an API, loop through files in a folder, handle complex conditional logic cleanly, or glue together multiple systems. That's where Python comes in. You don't need to become a software engineer, but you do need to be comfortable enough to automate the parts SQL can't reach. Core data structures you'll use constantly Lists and dictionaries. Most of the data you'll manipulate in Python outside of a dataframe will move through lists and dicts, records as dictionaries, collections of records as lists of dictionaries. Get comfortable with this pattern early. List and dict comprehensions. Instead of writing a for loop to filter or transform a list, a comprehension does it in...

  This is the first post in a 5-part series on data engineering fundamentals. We'll go from SQL, to Python, to ELT/ETL tools, to data warehousing with Snowflake, and finally to visualisation. Part 2 covers Python basics, link at the bottom. Why SQL, even in 2026? Every modern data platform, Snowflake, Databricks, Fabric, Matillion, dbt, still expects you to think in SQL underneath the interface. Cloud platforms change every few years. SQL has stayed the primary way to query and reason about structured data for decades. If you only master one skill in data engineering, make it this one. Important concepts you actually need to know Joins, properly understood. Not just INNER vs LEFT, but what happens to row counts when a join key isn't unique on one side, and why an unexpected LEFT JOIN can silently multiply your row count. Window functions. ROW_NUMBER, RANK, DENSE_RANK, and aggregate functions used with OVER() let you calculate running totals, rankings, and comparisons across row...

I get asked a lot by people looking to break into data engineering, or move up in it, what they should actually focus on. After 15 years in this space, from writing SAP BEx queries to architecting Snowflake and Matillion pipelines today, here's the roadmap I'd give my younger self, based on what's really changed and what hasn't. Start with SQL, not a fancy tool. This still surprises people, but SQL remains the single most important skill in data engineering, more than any cloud platform or orchestration tool. Get comfortable with joins, window functions, aggregations, and query optimization before you touch anything else. Every tool you'll use later, Snowflake, dbt, Matillion, Databricks, still expects you to think in SQL underneath the interface. Learn Python well enough to automate and transform data. You don't need to be a software engineer, but you do need enough Python to write scripts, work with APIs, and understand basic data structures. Most modern ELT t...