Hatchit "Postmortem"


Before I dig into this I wanted to address a few things. If you haven’t been reading my past couple blog posts Hatchit is a game engine developed by myself and 7 other people at RIT. My work on the project was mostly focused on developing the Vulkan based renderer. Development is sort of on hold for now it isn't dead or gone just yet. Because of that I don't really think that Postmortem is such a great name for this but that's the format I'm going to be following here. On top of that I just wanted to make it absolutely clear that Hatchit is not just my baby - a huge shout-out to everyone on the ThirdDegree team for their amazing work. There was no way I would have had time to do nearly anything that I managed to get done without their help. I will do my best to drop names where appropriate. Now on to the good stuff.

What went right

I learned a lot. We all learned a lot. This isn't something quantifiable, but I'm much more comfortable not just with Vulkan, but graphics programming in general. When I started on the Vulkan renderer for this engine I was a bit worried that I wasn't going to really be able to get much done. Everything in Vulkan appears very different at first from OpenGL. Not only that but my first few weeks working with Vulkan left me feeling like a huge moron. Image barriers were ruining my life and there weren't many in-depth resources to lean on. I guess I've always been a bit of a fan of the "baptism by fire" approach and it's served me pretty well here. After a while I got a handle on Vulkan's much more modern and sensible approach to everything. A couple weeks in and I started getting a good grasp on all the little layers (hah) that Vulkan gave me to rebuild the structures I was more familiar with in OpenGL. I'm no expert but I'm feeling pretty comfortable with Vulkan at this point.

We realized we were hitting a "moving target". I'm not talking about Vulkan as the moving target here but rather our understanding of it. It's true that new Vulkan SDKs and tools were coming out - but those were usually improvements for development. However, our understanding of how to use elements of Vulkan was constantly evolving. For example, our understanding of how to use Pipeline Layouts changed quite a few times. At first we were thinking that we wanted to have a bunch; one per Pipeline. Then after GDC we were hearing how other developers were just using ONE. At the time of writing we've settled on using one layout per set of "compatible" render passes. The takeaway here is that the slogan for this project was "we can always change it" and we did constantly. Nothing was static; things got rewritten as we found better ways to do them. No one on the team took it personally either, which made it easy. Matt and I rewrote each other's code all the time because we wanted the best project possible.

Speaking of Matt, we have an amazing build system for Hatchit. It's a series of CMake and Shell/Batch scripts that make building on Windows and Linux extremely easy. Matt took a chunk of time to build something flexible and readable that everyone on the team was able to pick apart and add to as needed. Third Party dependencies are usually a pain but Matt found an awesome solution. We submodule the code of all necessary outside libraries (that are compatible with Hatchit's LGPL3 license) - since they all use CMake as well (save for the Windows SDK), it's just a quick script to get them all built for Windows. On Linux we still rely on the package manager because that just makes sense. The system isn't perfect. We have two scripts to run that could be narrowed down to one, we expect you're running a Debian based distribution for Linux but overall it's very fluid and easy to use.

I'd also like to point out that almost everything in Hatchit is a resource. From render targets to pipelines, we described as much as we could in JSON files. This means that right now all you'd have to do to make a scene is write Scenes, Pipelines, Render Targets, Render Passes, Materials (all of which are JSON), Shaders and Models. It's very flexible and allows you to dig into just as much as you want to. Adding a post process effect is as simple as making a new Render Pass with the Pipeline using your shaders and making sure it takes in the output from a previous render pass. Then just draw a fullscreen triangle with that pass and you're good to go! It's more flexible than it is easy to use, but an editor could take a lot of the edge off of it all.

What went wrong

This isn't really our fault but we sort of ran out of time. I don't want to say we over-scoped the project because we didn't. However, there are a few features that are a bit half-baked. Sure, we have a Multithreaded renderer but it's hardly stable. Sure, we have Python scripting but it's not all bound properly. With only 15 weeks we did the best we could but I think instead of having the team work on "features they wanted" as we had planned initially we instead focused on "what we needed to have a cool demo". Our example scene isn't really much to write home about if you're just looking at it as an end user.

We mostly worked on Nvidia based desktops. This wouldn't have been as big a problem if all hardware worked the same way. Turns out that Nvidia's hardware was a bit more forgiving when it came to image layouts. (UPDATE: I previously claimed that the Nvidia driver ignores parts of the Vulkan spec which is not exactly true. Thanks to Pierre Bourdier at Nvidia for clearing that up for me.). AMD cards turned out to be very strict about image memory barriers while Nvidia cards can operate just fine with images simply in the General layout. This means that when I went to go make sure that I had something I could show off on my AMD based laptop for GDC, I ended up having to write up a stackoverlow post instead and didn't have anything worth a damn to show anyone that I ran into. This wasn't the only problem either. I ran into a driver crash on my AMD card that magically fixed itself after I wasted about two days trying to fix it, gave up, and came back to it a week later after working on the desktop. I wasn’t working in a place that really let me use my desktop and laptop at the same time comfortably. So I didn't get to spend as much time on my AMD machine as I would have liked.

I already mentioned it a bit but we ran out of time on our renderer. We totally underestimated how long it was going to take to rewrite our entire rendering system. Who would have guessed? The product we ended up with technically works but magically leaks memory, doesn't run at all on Linux and while it's well-written it just isn't as fast as we would want it to be. Granted it's hard to just jump into a multithreaded renderer for a graphics API that we've never used before but it's still frustrating. Vulkan BEGS to be multithreaded and we quickly realized that if we just used one thread we weren't really making the most of what we had.

While our resource system is pretty cool, the actual code is a bit of a mess. We have a thread responsible for loading resources. That way if we need to, we could load a resource on that thread asynchronously, use a default resource while it loads, and then use the requested resource when it's all loaded. Except the way we decided to reference count our resources makes that a bit of a headache. The reference counting is necessary but at the time of this writing really needs an overhaul. I don't think we even have access to the reference count from any place that we'd need it. We really need to get the resource reference counting and loading systems reworked... again.

What we would do differently next time

Hatchit also runs with DirectX 12 but we didn't really utilize it as well as we should have. We didn't have all the manpower to get that AND Vulkan done so we started focusing more on Vulkan while still leaving room for DirectX 12. I bring this up because originally we were working just in OpenGL and DirectX 11. Vulkan wasn't out when we started and we weren't thinking about DirectX 12 at the time because we were afraid that Vulkan might end up being very different. Turns out that Direct X 12 and Vulkan are VERY similar. The smart move would have been to just start in DirectX 12 to get familiar with the new paradigms and then add in Vulkan after it came out. It could have given us some precious time back.

I should have done more Multithreading beforehand. We really came into this without much multithreading experience from anyone on our team. I think for a while I kept saying "I'll multithread it later" because I was a bit worried that I'd end up breaking too much, too quickly, too early. I think everyone else also kind of had this attitude just due to a lack of experience. There was no way for us to read the future but I think the signs were there. We were reading "Vulkan enables multithreaded graphics" and instead of jumping into some asynchronous programming books and getting familiar we shied away from it.

I didn't always read the spec as thoroughly as I should have either. The Vulkan spec is dense and sort of a chore to read. I was hoping I could dive into just the parts I needed to but right now there aren't enough resources for that. It's not like OpenGL where you can just hit up http://docs.gl as needed. You really need to dive into the spec and digest it or else you'll just make a lot of sloppy mistakes. I also occasionally made some really silly assumptions. Particularly about immutable samplers which resulted in me looking like a bit of a doof on stackoverflow.

I didn't jump on SpirV-Cross when it could have saved us a lot of time. This is a really awesome tool for handling Spir-V shaders in Vulkan. Vulkan does not provide a way to reflect your shaders. You COULD parse the shaders manually as Spir-V is fairly simple. Or you could do what I did and just expect a bunch of usable meta-data in some JSON resources. Or you could do what I should have done and just link against SpirV-Cross. Right now we have a fairly obtuse RootLayout JSON object that could be completely eliminated. We struggled with the structure of that file for a while and it was just wasted time. SpirV-Cross (or Spir2Cross as it was originally named) was announced by ARM at GDC. I was there and I stupidly thought "no need for another dependency". I'm going to take the time to utilize this in Hatchit when (or if) I can find the time.

Major takeaways

Also known as the tl;dr

  • Get comfortable with Vulkan; it's here to stay
  • Make an awesome build system
  • Jump into Multithreading before Vulkan
  • Reiterate until you get it right
  • Test on every piece of hardware you can get your hands on
  • Read the spec and then read it again
  • Use SpirV-Cross if you need it; don't reinvent the wheel

I'm pretty happy with Hatchit. It's not a great engine but it's open and there for you to dissect. It's not over either. While I have accepted a job at Amazon doing graphics work on Lumberyard I hope to find time where I can work on Hatchit. More importantly I want to find time that won't violate my non-compete agreement. Other members of the team will be working on it and maybe after reading this, you will too! Every issue posted or comment made is highly appreciated.

You can find our open Gitter discussion page here if you have questions or concerns. If you missed it you can also find a link to the Github page here as well. As always don't be afraid to email me at amt3824@rit.edu or follow me on twitter

A big thank you to Chris Cascioli and Chris Egert - Two professors at RIT who have pushed me to do my best. Without those guys I don't think I would have gotten nearly as much out of college as I did.