South of Midnight's Foliage Wind and Interaction System

Sunflowers sway in the breeze in the flooded farmland level.

When we were considering a foliage wind and interaction solution, I was the only technical artist on the team. So we needed a solution that could be maintained by one person, that was easy for environment artists to work with, could withstand LOD switches, could represent wind ranges up to hurricane force winds, and could perform at 60 fps on Xbox Series S. Simple!

Researching the Problem

The big example that everyone knows about is the elephant plants in Uncharted 4. I read a few different papers on the subject but Naughty Dog's The Technical Art of Uncharted 4 (2016) was the most helpful in delineating the problem space, and suggesting the sufficiency of a material based solution for wind and interaction.

For our timeline, it seemed like an off the shelf solution would be helpful, at least to kick start us in the right direction. We found that Epic had supplied their own wind system called Pivot Painter in two versions. These node graphs were complicated but not too difficult to understand. There were also marketplace plug ins that expanded on the Pivot Painter concept. We ended up adopting Unified Interactive Physical Foliage, because it seemed like it supported wind, player interaction, and other forces we could utilize for magic spells.

Rigging the Foliage

A big velocity problem for projects with a lot of foliage is getting pivot information for branching tree structures. The Pivot Painter plug ins had semi-automated pivot painting work flows, but this was a big ask from such a small team. I suggested we see if we could partner with our foliage outsourcer to develop a Houdini based tool and deliver the trees pivot painted from their end. The Tech Director liked this idea, as software outsourcing is very common, and the outsourcers agreed.

In our tests, we found the pivot painted foliage to be extremely resilient to Simplygon's decimation, and optimizing and LODing worked without much fuss. The down-rezzed trees animated perfectly without any need for a second pass to fix broken animations. Adding geometry or editing uvs was not possible without rerunning the Houdini tool, however.

For more information about how we utilized Simplygon's tools, check out our Customer Story on the Simplygon blog.

Wind and Interaction System

UIPF has two main branches: the physical foliage branch which animates skeletal meshes based on physical interactions and collisions, and the pivot painter branch. We cut out the physical interactions since they seemed unnecessary for our project, and were potentially a performance liability.

Once we had rigged foliage, we could begin tuning the wind and interaction animations.  Since the number of number of parameters in the material was extraordinary, I viewed this as my responsibility, but I did have help from the outsourcing Tech Artist and the environment artists with the wind.  There were many plants. In the process, we found that certain features were unnecessary and others were inadequate, and the vertex shader was accessing too many textures over-all. In addition, the grass wind solution was using Unreal's Simple Grass Wind which was unfortunately not sufficient for the quality we had in mind for our project.

Material Hierarchy and Shader Permutations

Never let outsourcers name your parent materials.

The two competing priorities of limiting shader permutations but making the material hierarchy meaningful and easy to use was in all-out war here. This is the one system where we accepted more permutations to allow for easier management.

The hierarchy was based on the assumption that similar tree or plants would respond to wind and being pushed-through similarly. We had a branching structure from the master material that split between grass, plants, and trees. For the plants and trees, we had a master for each species which would contain most of the parameter overrides for the behaviour. Some of our species had a wide variety of morphologies, such as the oak trees, which required tuning per example.

Grass Wind

Both the prototype grass wind solution that existed in the project prior to my involvement, and the solution provided by Unified Interactive Physical Foliage used the stock Unreal Simple Grass Wind. If you're familiar with it, you know it has a very distinctive 'winding and unwinding' circular motion which feels very mechanical. What's more, the circular motion is only on one plane regardless of the wind vector, so the motion is very different depending on alignment of the wind vector, varying from a spiraling motion to a kind of 'wax on, wax off' wiping motion, if you get the reference.

I decided to attempt a vertical bending motion based on a sum-of-sines- a very, very simple application of Fourier Series. I thought it would match the bendy motion of the pivot painter wind better. The function packs seeds for 4 sine waves of different speeds, amplitudes, and wavelengths into a vector 4 and converts them into rotations based on the vertex xy position and the z position of the object world space location. I felt like this gave the best representation of individual blades of grass in a grass card. The grass assets were constructed with AO in a vertex colour channel. I found this served well as a vertical mask to modulate the wind intensity near the ground. Another option would have been to use vertical component of the object bounds as a linear gradient.

At this point, the wind resembled uniform sine waves marching across the level in lines from a distant source, whose location was defined by the wind direction vector. I needed to add some randomness to it, but the unreal machinery for PerInstanceRandom wouldn't work because this had to work with GrassTypes, FoliageTypes, as well as hand-placed actors. In order to get pseudo-random offsets per instance or per component, I remembered an old trick. I could use the world space transform of the piece of foliage as a source of randomness. Transforming 0,0,0 in local space to world space worked well to get the world space pivot location of the hierarchical instanced static mesh instance, or static mesh component of an actor. I used the fractional component of the world space locations as a source of 'random' offsets.

Optimizations

When we were optimizing the foliage systems at various times in the project, we found that there was considerable opportunity to lighten the interaction and wind computational load. The interaction system sampled a render target that kept track of the player position in the level. This render target needed to be sampled by the vertex shader in every pass- in the pre-pass, in the main pass, and in shadow depths.

The first thing we did was decide on a maximum size for foliage that Hazel could pass through. This was primarily design driven as it had huge implications for metrics and traversal. On the tech art side, we optioned to disable the interaction system on species of foliage that Hazel could not pass through based on their size. This included all trees, which reduced the computational load considerably. Similarly, there was a minimum size for grass where interaction didn't read at all. So, for very short grass patches, we disabled the interaction system in the material as well.

The next thing to consider was foliage that could be interacted with, but was sampling the render target unnecessarily. We reasoned that Hazel could not interact with foliage that was far away. And, over a certain distance, the foliage would have stopped swaying after being disturbed. We didn't want to attempt dynamic switching, but instead found another way in: the LOD system. The LOD system already keeps track of object distance, and different LODs can't be batched together. So, we decided to make a lighter material tree and assign these materials to interactable foliage meshes in LOD 2. I wrote a python script that analyzed the foliage folder structure, correlated foliage meshes and their materials, made an alternate material tree with interaction disabled, and assigned the lighter materials to all LODs after LOD1. This script would have to be rerun if the LODs of a piece of foliage were regenerated, but it seemed like a small inconvenience. Further enhancements to this process would be to have the Simplygon LOD system kick off the script after building LODs, but this was very late in the project and the number of times it would have to be run was very finite.

Unexpected Use Case

Pivot painter data enabled this material driven regrowth effect.

Later in the project, the cinematics called for a dead weeping willow tree to magically sprout fronds and come back to life. It was a big moment, and we needed a compelling way to add foliage back onto a dead tree. Connor, VFX Artist extraordinaire, was talking about some kind of unfurling motion- having the fronds grow and uncurl themselves. I realized right away that we had all the ingredients to make this happen right in the material, because we had the pivot points of all of the fronds stored for the wind and interaction data.

So, working with Connor, I set the basic mechanism for how the will fronds could grow and unfurl. He combined this technique with others to realize the needs of the cinematic without the need for complex rigging or a whole new system bespoke for that one use-case.

Below you can see Connor's wonderful work on the cinematic effects.