So, the jam is in its voting phase and people seemed to like our submission! If you haven’t tried it yet, Chainboom is a boomer shooter where you shoot explosive zombies, which blow up other zombies and so on in order to create total mayham!

In this postmortem, I reflect a bit on what went better than for our last jam and what issues with our workflow we ran into. I won’t go into any detail on game design. This is mostly technical or workflow focused. I also won’t go into basic advantages of Bevy. I love using Bevy, I love the ECS, and I don’t want to preach too much to the choir :)

Note that all of this is mainly my perspective, but is informed by talking a lot with my teammates. I think I represent their experience fairly, but it’s always possible I misunderstood something. If so, I will go back and edit this post.

What went well

Last time, Joona and I created Crazy Bike, with music by Mafi. That was a great success: we placed 6th overall and 1st in graphics. But the gameplay came together in literally the last hour. Now, with Chainboom, we finished the gameplay 3 days in advance. Wow, that’s quite the difference! I want to explore the factors that I believe led us to be so much more productive, in no particular order.

Experience Judging Feature Complexity

Compared to last time around, I feel much more capable of guessing whether a feature will be simple or hard to implement. During the planning phase, we brainstormed the following ideas:

• Zap enemies with magic chain lightning that jumps from enemy to enemy
• Place beacons that generate lightning between them
• Shoot nails into enemies, then trigger an attack that chain-zaps all nearby enemies with metal in them
• Kill enemies to explode them, which will explode other things nearby as well
• Build Tower Defense towers that trigger other towers
• Place sticky bombs and detonate them by shooting them, resulting in chain explosions when the bombs are close enough

Maybe it seems obvious in hindsight and when viewed in a neat list that option 4 is the simplest one to implement. But that was not as easy to notice when in the middle of brainstorming. Compared to last time, I was much better able to imagine possible implementations and how long they would take / how many unknowns could crop up. As such, I argued for chain exploding enemies on death because of time constraints, and my team accepted my suggestion.

Lesson learned: Strip as much gameplay as possible while planning, until you’re left with the core fun mechanics. You can always add more later!

Starting with Foxtrot

Reduced Boilerplate

Crazy Bike started with bevy_new_2d, while Chainboom started with Foxtrot as a base template. I really really like bevy_new_2d, but there is a catch: it’s 2D. Shocking, I know.
For Crazy Bike, this meant we spent about 2 days setting up

• A first person camera with good constraints
• Avian integration
• Tweaking Tnua so that our character controller behaved properly
• Integrating an input manager
• Setting up a level design pipeline through Blender integration All of this is not specific to Crazy Bike, so arguably, we spent 20% of the jam with boilerplate. And we didn’t even have time for the entire boilerplate. The animations for NPCs never worked, and we also didn’t add any navigation AI for them. They’re essentially just fancy statues in Crazy Bike, which hurts the believability.

Since then, I had completely reworked Foxtrot to serve as a better starting point for the first person experiences that I am interested in. The old Foxtrot allowed multiple camera perspectives and had a ton of dependencies on things that ended up being unmaintained. The new Foxtrot is much more focused on providing mechanics you would find in an Im Sim. And it’s built on top of bevy_new_2d, so all of it’s goodies like great state management, preloading assets, neat organization and a minimal widget system are still in Foxtrot.

Teaching a New User

All of this comes at a price: Foxtrot is big. It’s not meant for newcomers, and maybe it’s not even meant for intermediate users. That is why I rebranded it from a template to a reference project. My team was fine with using it as a starting point, which I was a bit nervous about at first, since Mafi, one of our developers, was still new to Bevy. Since he is a long time friend of mine, I had a very good idea of which concepts he already knew and which he needed help with. To my surprise, Mafi was very quick in understanding Foxtrot and contributing new features to it. I attribute this to

• Mafi being just a pretty experienced developer in general
• Bevy having really good API
• Bevy allowing excellent separation of concerns through the ECS
• Foxtrot being (mostly) well-organized
• Mafi having access to me via phone, lol

Since our game is heavily split into individual plugins and files that stand on their own, Mafi never needed to understand how all pieces fit together. It was enough for him to add a new file and do his logic in there, then slap some components on entities in observers with Trigger<OnAdd, Player> or Trigger<OnAdd, Npc>.

My initial worry was that using Foxtrot might slow us down due it only being comprehensible to me, but it instead was surprisingly easy to teach. We ended up starting on Chainboom’s day one with the progress we had on Crazy Bike’s day three.

A Tangent on Dependencies

My journey with Bevy’s ecosystem looks like this:

• I started using only vanilla Bevy. No dependencies for me!
• Oops, Bevy doesn’t even have a physics engine. I guess I should accept some dependencies.
• Woah, the ecosystem is great! Let’s pull in more deps!
• This is great, more!
• Oh, a new Bevy version came out. Let’s upgrade. Oh heck, my dependencies are out of date
• I waited 3 months now, and my dependencies are still not up to date. Let’s see if I can send in a PR
• Wait, how does this crate work at all? What, it does a custom render pipeline? And the breaking changes there are not in the migration guide? I don’t know what I’m doing
• Let’s just remove the dependency
• Oh heck, my code was tightly coupled with it. Now I need to refactor it
• Oh goooood this is a meeeeeessss

Since then, I have learned to categorize my dependencies in the following safe to use groups:

• This is widely used and has a track record of being maintained
  • e.g. Avian, Hanabi
• This is a dependency that I already manage, so I know that I will continue maintaining it
  • e.g. Yarn Spinner, Avian Pickup
• This may be unmaintained at some point, but I know that I can easily update a fork to the newest Bevy version without understanding the crate
  • e.g. landmass, Tnua

I have banished all bevy_ dependencies that don’t fall into one of these three categories from Foxtrot, which has greatly improved my enjoyment when a new Bevy version drops

Lesson learned: Start with a 3D template you’ve made in advance. I like Foxtrot, but you, dear reader, may be better served by creating your own one.

WebGPU

Due to particles being integral to the experience, we needed to make sure they were available on web. Hanabi is only available on WebGPU, so we would have to rely on a CPU particle library if we wanted to support WebGL2. That would mean putting extra stress on the CPU, which is already being pushed quite a bit, and maintaining two code paths if we still want to allow Hanabi on native. Additionally, I have known from experience that WebGL2 often has very different interpretations of how some graphical effect should look like, which results in losing time with tinkering with the Wasm builds.

So, we made the jump to only supporting WebGPU, at the cost of some users not being able to play the game. To my surprise, WebGPU ran well for the vast majority of people. We had people using Firefox, Chrome, and even Safari, all saying it worked well for them. Well, besides lag on mid-tier machines, but that is not WegGPU’s fault. The WebGPU version even nearly fulfilled the dream of “write your Bevy code for native, and it automatically works the same on web!”. See our bugs section below for the one exception.

Lesson learned: use WebGPU!

Level Design with TrenchBroom

Last jam, I spent significant time designing the map in Blender. Maybe 3 days, give or take, just mapping. What that showed me was that Blender is not the right tool for what I like in a level. Don’t get me wrong, Blender is great for high-detail levels with lush caves, interesting terrain, custom buildings, hand-painted roughness maps for puddles of water, etc.
But that is not where my interests lie, and it’s certainly out of scope for a jam if we don’t happen to work with a Blender veteran. Since then my SO, the lovely Sara aka PlsGiveMango, and I developed an interest in TrenchBroom. It’s a level design software primarily intended for Quake mapping, but it has gathered a user base coming from various engines over the last years. bevy_trenchbroom is a good integration layer for it. It still has some rough edges in the API, but crucially, it is fairly non-intrusive to your Bevy code.

We quickly learned to appreciate TrenchBroom for its simplicity. Working with brushes on a grid is limiting, but in those limits lies incredible efficiency. I can say with great confidence that using TrenchBroom, I would have needed about half the time for the design of Crazy Bike’s map.

Sara joined our team as the dedicated level designer, and the fact we had her on board shows. We quickly had a minimal level to play around in, but then she expanded and expanded the level and added little details, nooks and crannies, alternative routes, etc. In the end, we had much more time for coding, as we didn’t need to split someones attention between coding and level design, and the resulting level was really good, since Sara had so much time to refine it.

What was also great was that Sara never needed to touch a single line of Bevy. Okay, not quite, she needed to tweak the ambient color resource. But other than that one constant, she was able to only work in TrenchBroom, using normal TrenchBroom workflows. And, since bevy_trenchbroom supports hot reloading, she was able to quickly see her changes to the map in-game by keeping the game open and reloading the map. Great iteration times!

Lesson learned: Use TrenchBroom for 3D levels. If you know someone who can do level design, tell them they’re great.

Hotpatching

It’s known that Bevy’s UI story is lacking. But since hotpatching became available, this has greatly improved. Hotpatching allows you to change a function’s code at runtime and see the effects of your new code live. This is cool for tweaking enemy stats and walking speeds, but the real efficiency comes from making bevy_ui usable. Currently, it requires quite a few hoops:

• You need to install a dedicated CLI
• Your project has certain restrictions, such as not using a lib.rs
• Some linkers behave weirdly
• You need to annotate functions with #[hot]
• Already spawned UIs need to be despawned in order to hotpatch their setup
• The bigger your project, the longer hotpatching takes

To circumvent the last point, I created an empty project just to develop my UI. I created a system that spawn I got some help from my lovely local coding wizard Tau with the UI, as they have a knack for making pretty things. Together, we were able to prototype the UI way more efficiently than last time, as the iteration time between writing code and seeing the effects on the UI was about 0.5 seconds. If we now also started using the BSN prototype, we could get some good UI development experience!

Lesson learned: Hotpatching is neat. Use it for UI and for tweaking gameplay consts.

Observers

For gameplay purposes, I heavily relied on chaining bits of logic through observers. This was very efficient for prototyping, as I didn’t need to care about scheduling, ordering, system sets, etc. except for the system that does the initial triggering. Simply trigger some gameplay event, and let observers all across the codebase react to it! This made adding features a breeze, since every feature can live in its own little file, with the input event being the only connection to the rest of the codebase. Also, the code is a bit smaller than for regular events. This is not a big change for a single system, but a huge change across an entire code base.

Lesson learned: Events are cool for libraries, but in gameplay code, observers are king

What could be better

Naturally, not everything went well. We didn’t really run into significant bugs in our own codebase or game design issues, but a lot of Bevy limitations. Some of the problems described here are already fixed on Bevy’s main branch, but are not available in 0.16.

Bevy Bugs and Limitations

This is the list of things that either required workaround code or third-party tools to address.

Bugs

• Forward decals are broken on Web
• Sometimes, lights have an effect through walls
• TAA behaves weird in multiple-camera setups
• HDR cameras “stack” their tonemapping
• HDR camera renders nothing in a standard multi-camera setup
• Reported mouse movement is massively slower on Wasm than on native when cursor is locked
• On wasm, window.cursor.grab_mode is not updated when cursor is unlocked
• embedded_asset! is broken on Windows Wasm builds
• Coordinate system mismatch between Bevy and glTF
• Directional lights ignore walls on WebGPU when the camera is too parallel to them
• Bevy caches invalid cursor settings

Limitations

• Make GlobalVolume change running audio
• Importing cube maps is hard
• Make AnimationPlayer a relationship
• All of bevy_audio on Wasm. It’s nearly impossible to get non-choppy audio. We try to free the main thread as much as possible on gameplay start by
  • waiting for all assets to load
  • precompiling shaders
  • waiting until the navmesh is generated

Luckily for us, the last point may be fixed now, as bevy_seedling has an experimental web backend that uses the JS API. I’m excited to try that out, but since it got released right during the jam, I did not feel like exploring it too much yet :D

As you can see, this is quite the list. Fortunately, I happened to stumble into most of these already while working on Foxtrot, so I usually already knew the workaround already. However, that is specific to our team. I’m not sure if a team that did not already know about these issue would have been able to create a 3D game with our scope in time. My big takeaway is that the combination of “multiple rendering features on at the same time” + “Wasm” is very volatile.

Performance

The biggest negative feedback we got is performance related. A quick flamegraph shows that the issues come from rendering. We already decimated most of our point lights and disabled shadows for most of them in order to get the game playable at all. A big lag spike comes from fading decals and gibs away after a wave. Turns out that setting the alpha mode to blend for 100+ entities at the same time is expensive! But why do we even need to fade the decals? Well, the builtin decals in Bevy don’t work on web, so we used a crate that implements decals by constructing a mesh with the decal texture on top of the surface that should receive the decal. Which is much more expensive, but AFAIK the only way we could get decals at all on web. Problem is that Bevy’s renderer really really does not like to have many unique meshes, especially on web. So we need to despawn the decals every now and then, which caused the new lag spike. Aaarggh.

Another issue is that our game becomes massively more expensive with higher resolution monitors. A user reported that it is completely unplayable on 4k monitors. Welp. I guess we will need to figure out how to force Bevy to use a lower resolution, though I thought that simply setting the resolution in the window plugin should already do that? Didn’t look much into it, but it seems like a footgun.

Other than that, the renderer still struggles with the scene for some reason. No idea where those draw calls come from, as the scene should not be that demanding. Note that we already

• made the textures and models low-res
• generate mip maps
• use compressed KTX2 for everything
• as noted above, removed a ton of lights and disabled shadows

Panic on Missing Resource

One of our systems is in a scheduling ambiguity with a system that inserts a resource. Meh, wish resources resulted in fallible systems + warning in release builds. Well, fortunately Bevy 0.17 has When<Res<T>>.

Despawns in Observers

One gotcha with the way we use triggers is that none of them are allowed to despawn an entity, as that would result in a panic in any other observer looking at an entity that was just despawned by another observer. We solve this by having a dedicated Despawn component that results in the entity being despawned in a safe schedule (either PreUpdate or PostUpdate but before transform propagation and visibility checks, I forgot). Simple and widespread solution, but a bit clunky.

Broken Navmeshes

This cost a lot of time. We are quite happy with the navigation library landmass, but it does not provide its own navmeshes. AFAIK there are two good libraries for 3D navmeshes:

• vleue_navigator / polyanya
• oxidized_navigation

The first one looks great, but requires you to manually place planes on your map to tell it where to generate the navmesh. That is simply not good for iteration times on levels featuring a lot of verticality. So, we are pretty much forced to use oxidized navigation. That one is a Rust port of the popular recast library. It works great: you simple tag your colliders with NavMeshAffector, tell oxidized navigation how big your agents are, how high they can step, etc. and it spits out a full navmesh that landmass can directly use. It’s almost magic how it does everything for you.

When it works.

The issue is that everything in oxidized navigation is nearly good. A simple room will work, but a multi-room setup will often create a navmesh that ignores a chunk of the room, or creates a weird convergence at a pillar where all agents will run towards, or a broken staircase, etc.
The result is that you can walk to many specific magic areas in which agents will completely lose track of you. You can play with the generation constants to improve the situation, but that will only bring you so far. Making the navmesh usable always includes a phase of placing random pillars in the room to force the navmesh to break up at that part, or shuffling around some crates to appease the algorithm. This is much more art than science, and often, we simply had to trial-and-error our way through. Change the map in TrenchBroom, reload the level, check the issues, change the map again, repeat. And every change has the possibility of fixing one thing and breaking another.

Maybe in time for the next jam we will setup some bindings to recast. I dislike C++ in my build pipeline as much as the next person, but if it results in clean navmeshes, we might save a lot of time and headache, and that would be worth it.

Taffy Inconsistencies With Web

No specifics or bug reports since this happened right in the middle of a prototyping session, sorry. But this is not the first time that Taffy does something subtly different when compared to how CSS works, which straddles the line between “bug” and “just a different implementation”. In any case, it’s annoying when you bring certain expectations from web and then need to figure out Taffy’s quirks.

Some notable instances were fitting a healthbar without rounded corners into a background with rounded corners by using overflow: clip and centering something vertically while keeping its overflow behavior intact from how it behaves when not centered. Maybe I’ll open a bug report with these if I can remember the exact code we used.

Bottom Line

Even with all the issues presented, Bevy is in a really really good place for 3D. Surprisingly, I am coming more and more to the conclusion that I don’t need a Bevy editor. Sure, Bevy should have an editor. But I personally don’t need it anymore. I’m very happy with TrenchBroom, and I think (proto) BSN + hotpatching make Bevy UI pretty usable. All we really desperately need in that regard are widgets, but that is being worked on.

Working on the jam and seeing just how fast our iteration times can be with the right setup has shown me yet again that Bevy is the tool I want to continue investing in. I’m extremely proud of what our team has managed to accomplish in this short amount of time and happy that I got to work with such talented people. Also extra special thanks to Joona, who spent the last few days polishing the game solo because the rest of the team was in the mountains without any devices beyond phones. Though I did manage to sneak in a few PRs typed on my phone, hehe.