Posts

Featured Post

Stop Guessing Snowflake Warehouse Sizes: A Free Advisor That Reads Your Own Usage History

-
Stop Guessing Snowflake Warehouse Sizes A free advisor that reads your own usage history and tells you what to change 100% Free Runs Inside Snowflake Zero Data Egress Generates ALTER Statements Almost every Snowflake team sizes a warehouse the same way: start at Medium, see if anything breaks, and never think about it again. Six months later nobody remembers why ANALYTICS_WH is a Large, whether it needs to be, or what it would cost to find out. The information to answer that question is already sitting in your account. Snowflake records spill, queue time, bytes scanned, and concurrency for every query in ACCOUNT_USAGE . The problem is that turning those columns into a defensible sizing decision takes a fair amount of SQL and a mental model of how the pieces interact. So I built SnowLens Warehouse Sizing Advisor -- a free Streamlit app that installs into your own Snowflake account, reads those signals, and produces a per-warehouse recommendation with the reasoning a...
Post a Comment
Read more

All about SnowLens - Snowflake observability and anomaly detection tool

-
Snow Lens Free Snowflake Observability -- 9 Detectors, Zero Data Egress 100% Free Runs Inside Snowflake 9 Detection Rules No Outbound Calls If you are running Snowflake in production, you have probably asked yourself: Which queries are burning credits? Why did costs spike last Tuesday? Are any warehouses just sitting there idle? Most observability tools answer these questions by shipping your query logs to an external SaaS -- which means data egress, vendor lock-in, and yet another subscription. SnowLens takes a fundamentally different approach: it is a Streamlit app that installs directly into your Snowflake account and runs on your own compute. Your data never leaves your environment. And it is completely free -- no payment, no signup, no feature gating, no expiry. In This Post How SnowLens Works The 9 Detection Rules (with Logic) Privacy and Security Model Memory Safety for Large Accounts How to Inst...
Post a Comment
Read more

DATADLE: A Daily Word Game for Data Engineers

-
Post a Comment
Read more

Data Engineering Fundamentals, Part 5: Data Visualisation Fundamentals

-
Image
This is the final post in a 5-part series on data engineering fundamentals. Part 4 covered data warehousing with Snowflake , . If you've followed the whole series, you've now gone from raw SQL and Python, through ELT pipelines, into a properly modeled warehouse, and now to the layer where people actually see and use the data. Why visualization is the last mile that matters most You can build a technically flawless pipeline and a perfectly modeled warehouse, but if the person making a business decision can't see the answer clearly and quickly, none of that engineering work translates into value. Visualization is where data engineering meets decision-making. Core concepts that apply across every tool Dimensions vs measures. Dimensions are the categorical fields you slice and group by, region, product, date. Measures are the numeric fields you aggregate, sales, count, average order value. Every visualization tool is built around this same distinction, understand it once and it...
Post a Comment
Read more

Data Engineering Fundamentals, Part 4: Data Warehousing and Why Snowflake Stands Out

-
Image
  This is Part 4 of a 5-part series on data engineering fundamentals. Part 3 covered ETL/ELT concepts . Part 5, the final post, covers data visualisation tools. What a data warehouse actually is? A data warehouse is a system optimized for reading and analyzing large volumes of structured data, as opposed to an operational database (OLTP) optimized for fast, small, transactional reads and writes. Where OLTP systems power an app's checkout process one order at a time, an OLAP-style warehouse is built to scan millions of rows and answer "what were our total sales by region last quarter" quickly.OLTP system will be always write optimised whereas OLAP read optimised. Core data warehousing concepts OLAP vs OLTP. Online Analytical Processing systems are designed for complex queries across large datasets, online Transaction Processing systems are designed for fast, frequent, small transactions. Data engineers typically move data from OLTP sources into an OLAP-style warehouse. Sta...
Post a Comment
Read more

Data Engineering Fundamentals, Part 3: ETL/ELT Concepts Every Data Engineer Should Understand

-
Image
  This is Part 3 of a 5-part series on data engineering fundamentals. Part 2 covered Python basics , link at the top once published. Part 4 moves into data warehousing with a focus on Snowflake. ETL vs ELT, and why the difference matters ETL, Extract, Transform, Load, means data is transformed before it reaches the warehouse, usually on a separate processing server. ELT, Extract, Load, Transform, loads raw data into the warehouse first and transforms it there using the warehouse's own compute. Most modern stacks have shifted to ELT because cloud warehouses like Snowflake are powerful and cheap enough to handle transformation directly, and it keeps a copy of raw, untransformed data available if your transformation logic needs to change later. I'll use Matillion as the example tool throughout this post, since it's a widely used ELT platform, but these concepts apply across tools like Fivetran, Airbyte, or Azure Data Factory too. Core concepts that matter regardless of tool Ex...
Post a Comment
Read more

Data Engineering Fundamentals, Part 2: Python Basics Every Data Engineer Should Know

-
Image
  This is Part 2 of a 5-part series on data engineering fundamentals. Part 1 covered SQL , . Part 3 moves into ELT/ETL tools and core data engineering concepts. Why Python, on top of SQL SQL is excellent at querying and transforming structured data that already lives in a database. But it can't call an API, loop through files in a folder, handle complex conditional logic cleanly, or glue together multiple systems. That's where Python comes in. You don't need to become a software engineer, but you do need to be comfortable enough to automate the parts SQL can't reach. Core data structures you'll use constantly Lists and dictionaries. Most of the data you'll manipulate in Python outside of a dataframe will move through lists and dicts, records as dictionaries, collections of records as lists of dictionaries. Get comfortable with this pattern early. List and dict comprehensions. Instead of writing a for loop to filter or transform a list, a comprehension does it in...
Post a Comment
Read more