Hi, I'm Peter, the main developer of Hierarchy. For those that are curious, I thought I'd just talk to you about some of things I had to go through to create Hierarchy. Hope you find it interesting.


I think that most of the devs out there would agree that compilers are hard to make. Not only is it technically challenging, but also, just the endless amounts of work is simply overwhelming. I knew this going into starting Hierarchy, so I decided to give myself one year, full-time, dedicated to getting the meta-compiler done. I had enough in my savings to last around a year and didn't mind living cheaply for awhile. And one year seemed like ample time to get the core features done. I had even done a round of requirements and planning to make sure I could meet my self-imposed deadline - And, actually, during this phase, I cut out a good 70% of what I had planned to do.

When I got started, things proceeded a little slower than I thought at first, but it wasn't too bad. So after the first year of development, I had gotten a lot of the core features working and it seemed like finishing the meta-compiler was just around the corner. I remember feeling pretty good. But it was amazing, it's like I blinked and another half year blew by. I couldn't believe it. I remember being very productive during that period, but a lot of small things needed to be fixed and added. All these fixes needed to get done, but still, if felt like in an instant, I was already half a year past my deadline. I was definitely a little worried.

Looking back, I now realize now there's a huge difference between creating a business application where you can leave out some things and let some buggy features slide compared to creating a low-level compiler. Actually, in my previous jobs, I had made quite a few Java-libraries before, so I thought I knew the amount of rigor that was needed, but even compared to this, a compiler needs a very large jump up in terms of care and polish; the core set of features needs to be rock solid (developers won't stand for a compiler that gets things right "most of the time"). Then, there was just so much more that needed to be added in for this to be a viable tool. For instance, during my early test projects, I realized we needed features like path-aware Descriptor-variables and support for creating matrices and schemas that could be accessed from Jar files (for non-devs, a "Jar" is a library file for Java code). These just couldn't be left out.

It was around this point, near year two of working on Hierarchy, when I started questioning whether it was worth it to go on. It just seemed like whenever I got close to finishing, there was just one more thing to add. Then I find another thing, and another and another. I'd like to think of myself as being pretty good with stress. I usually don't worry too much about the problems we all typically have at work, and I think I've had my fair share of stressful situations (unavoidable as a software consultant who's expected on each project to hit the ground running). I had completely depleted my savings - well, more than that, taken on tens of thousands of dollars of debt - and my stress-level was just sky high. This was one of the only times in my life when I remember having so much anxiety, I actually had trouble breathing.

And, to make this worse, it was around this time I also realized the perfect feature to add to the meta-compiler, but it was a tough one, persistence. Adding persistence to hierarchical data seemed like the perfect fit, but I could tell it'd be so much work! I'm sure both the devs and non-devs reading this have all been through scope creep before, but this was a scope creep on a whole new plane of existence, of not just weeks or months, but on the scale of years. It seemed endless (and was endless), and I wanted to stop. I realized I just needed to step back and figure out what to do.

And actually, after I had thought it over, it really wasn't that hard of a decision. After so much of an investment of time and money, I had crossed the point of no return and would see this through to the end. There was no way I could throw all that work away. It just seemed like such a waste. So, I did what any adventurous American would do, I went out of my house and scrambled to find a way to take out more loans. Then, I got back to my computer, put my head down and kept pushing through - whatever happened, happened. But, I needed to see this thing done.

It was about year three when I finally got persistence and the last of the features for the meta-compiler finished. I can still remember night after night of working in my bedroom until two or three in the morning until my eyes stung from the glare of the screen and my lower back was pinched and all knotted up (I've had some odd back problems for years). But finally, no more features. That was it! The Beta was done! I'll never forget that night, or the night when I first got the meta-compiler working. Unforgettable.

I remember reading about a year ago an old interview of Linus Torvalds where he said that if he had known how much work creating Linux was going to be, he would have never done it. I'm definitely, in no way, comparing the amount of work of creating an operating system to that of creating a meta-compiler, a modern OS takes hundreds of developers, while a meta-compiler takes handfuls of them. But, most of the work was done by just me. And if I ask myself the same question, I'm not sure how I would answer, because in the end, I'm not sure I would have dedicated four years of my life to this. It was worth it and in many ways not. I've had to make a lot of sacrifices. It's a pretty close call.

I know most of you out there, both developers and non-developers alike, have also had to put in weeks of seventy to eighty hours, one after another, often for months. Also, that you too know how mentally and actually physically tough our jobs are. Don't get me wrong, the work I've been doing is fascinating (there's nothing closer to the feeling of being a true creator than the fast-paced building we do as developers, one of the ultimate acts of creation we humans do), but at the same time, you can't help asking yourself, "why am I doing this?" I think most people would need a really good reason to justify devoting this amount of energy, agony, and pretty much every waking hour of your life to just one thing for four years (I had completely demolished my social life, losing touch with and actually angering a good deal of my friends). Well, there's money, which, in the software-development world, is typically very good, but it's not really enough, is it? For most things, seems like we need to feel there exists some bigger end-goal behind the things we do.

So, for me, why did I do it? The answer is pretty straightfoward actually. I thought I saw an area where something new could be added to our discipline and decided it might be worth it to try. And, I think at an even simpler level, I thought I would really enjoy making it. At that time, my work had become pretty routine as creating database-driven web-applications, one after another, had become tedious and numbing. I needed a new challenge, and this type of work let me utilize all my problem-solving skills to the absolute fullest - there were tons of areas to use my creativity. But still... Is this a good enough reason? Hmmm...

...I guess all I can say at this point is that building Hierarchy was easily, by far, the most interesting and challenging dev work I've ever done, and I'd be surprised if anything in my life will be as technically, mentally and financially difficult as this time period was. It's probably safe to say creating Hierarchy is the hardest thing I have done and ever will do.