I’ve been writing software in one form or another since I was a kid. Starting out with BASIC and moving through various languages and frameworks has always been an interesting process of discovery due to the ideas in the new materials being in opposition to my preconceived notions, or simply due to a lack of experience with a particular concept. Such challenges are generally what I would consider to be immensely fun, so I tend to revel in them. And besides, when I get stuck, there are plenty of blog posts and StackOverflow answers to help me figure out most problems.
A few months back, I decided to take on a new side project: a weather forecasting website/app. Why? Well, there are a few reasons. First, because I don’t like any of the current popular weather destinations: they are full of ads, or are painful to use (sometimes with a UI reminiscent of the early 2000s Web), or have been greedily gobbled up and shuttered with the ostensible purposes of “help[ing] as many people as we can” — give me a break! Another reason is that I’ve always been curious about how weather information is distributed and what such data looks like. And finally, because I’ve been missing the world of .NET, while at the same time, I’ve wanted to explore the recent resurgence of server-side web rendering with progressive client-side enhancement with technologies like htmx. I’d also hoped to work with a couple people on this as a way to hang out and share knowledge (to me, this is much more fun than board games or video games), but thus far, scheduling has admittedly been a bit of a challenge.
Initial thoughts
I started by trying to imagine roughly what I would like in a weather website, and then figuring out what to research after that. The first thing I would want is a nice map that supports pan and zoom. But I didn’t want to use existing services for that, as I wanted to minimize reliance on third parties except where absolutely necessary. This took me down a very long rabbit hole of exploring the world of Geographic Information Systems, or GIS. Map projections, geographical and geopolitical categorizations, and various open source tools to convert between a multitude of formats as well as to filter data down to what I’d need for weather: these have all been swimming in my head.
In addition to mapping, and, arguably, the most important thing to figure out, is how and where to get the actual weather data! The obvious high-level answer is the National Oceanic and Atmospheric Administration (NOAA, pronounced like Noah). Within NOAA, however, are a multitude of agencies, divisions, and what-have-you. The agency of interest here is the National Weather Service (NWS), and the National Centers for Environmental Prediction (NCEP) is the part of NWS that is responsible for things like forecasts. As its name implies, NCEP has multiple centers with differing responsibilities. The Environmental Modeling Center (EMC) is of particular interest here. Rather, the data that is generated by this center and made freely available to the world is what is of interest. There are many gigabytes, maybe terabytes, of data being generated within NCEP every day. They have supercomputers crunching numbers non-stop. The resulting data has lots of different shapes and sizes, and sometimes it’s pretty difficult to find concrete definitions and explanations for what I’m looking at that are comprehensible to a non-meteorologist like myself.
Figuring things out, step by step
I’m still in the middle of all this, and, while I have learned a lot, there is plenty more that I’ll need to understand in order to make a useful weather site. I’ll delve deeper into what I’ve been able to gather, deduce, or just guess, in later posts.
It does often feel like I’m being overly presumptuous in my ability to do this project. GIS and meteorology are fields that people study for years, and I’m just stumbling through them. I know I’ll be making plenty of mistakes and erroneous assumptions along the way, but hopefully I’ll recognize and correct them later on.
The biggest challenge so far, as I’ve already alluded to, has been my knowledge gap. I can search the GIS StackExchange all I want, but a lot of the questions and answers I simply don’t understand. Additionally, I’ve encountered situations where people have (apparently) solved some rudimentary issues that I’d faced, but it would seem that nobody is talking about the problems or the solutions. It is, of course, possible that the terminology is just so foreign to me that I don’t know how to ask the questions. While I generally consider myself to be a fairly competent software developer, trying to work through some of these problems has certainly been a humbling experience. For as much as (I think) I know about software, there are entire universes of information and fields of study that are just completely orthogonal to everything I’ve learned so far, and starting out in them is eye-opening.
In upcoming posts, I’ll discuss some of the problems and solutions I’ve dealt with on this journey, many of which might seem relatively simple. Some of them are, while others have been deceptively difficult to figure out. Interestingly, just like with good code, the solutions to many problems are concise, while the process of discovering them has been anything but.
Until recently, most of this information has been, as Donald Rumsfeld would say, unknown unknowns for me. Now, it’s slowly getting into the known unknowns territory. Join me in unraveling these mysteries!