This is a system I wrote in unity that uses tessellation and compute shaders to simulate terrain deformation on a plane. There are two parts to this project; the compute shader that regulates the deformation of the plane, and the shader used to render it.
The way the compute works is that it is given a base heightmap that determines the overall shape of the terrain. On top of that is overlaid the layer of uniform height flat “soft terrain” illustrated by another heightmap handled by the compute shader. Whenever the compute shader detects an overlap with the terrain and the deformers, it writes the difference to the soft terrain heightmap. As it’s a computebuffer that gets written and read to by the compute shader every frame the read/write versions of the soft terrain heightmap must be swapped every frame by a script running on the cpu. These heightmaps are then passed to the tessellation shader to be rendered.
The model used in this demo is only a 10 quad by 10 quad plane, but with the magic of a tessellation shader, it’s possible to dynamically increase the level of detail on the fly by adding in more polygons during the geometry function of the shader. The advantage of this process is that it adds the extra polygons in on the GPU at a late stage in the rendering process, which allows for rapid changes in the number of extra polygons without suffering a performance drop. (you can see this effect at 0:44 seconds in to the youtube video). Right now, the algorithm I’m using is fairly naïve, tessellating uniformly based on distance from camera, however, I believe that with some restructuring of the algorithm and how it handles data, I could get it to tessellate based on the derivative of the slope, which would have the advantage of only putting the extra vertexes where they are really needed, further optimizing this process. Unfortunately there are a few issues at play that make that idea a lot more challenging to implement, but it’s something that I want to come back to in the future. The shader also is responsible for the change in color; brown if you’ve deformed the terrain a bunch, green if it’s unbroken, as well as some other minor visual tweaks to get it to a point where I thought it looked good.
An extreme example of the tesselation based on distance from the camera.
The biggest issue with this process was trying to work out collision; since the extra polygons were added in at a late stage in the rendering process; they didn’t actually “exist” outside of that process so it was impossible to use them as any basis for colliders. The heightmap data too never leaves the GPU, and to attempt to pull all that data from the GPU every frame could cause some major bottleneck performance issues. Even if we could get the heightmap data; the heightmap data is constantly changing so dynamically on such a fine and detailed level, so any attempt to construct a collision mesh from the data would cook most computers. The solution I came up with this was to merely pole the heightmap every frame; performing a sort of simple version of a raycast, to get the height of the deforming object. It works perfectly well for what it is, but isn’t very friendly to work with and probably wouldn’t work in an actual game scenario. The reason I went with this method over other, potentially simple solutions, is that it allowed you to sink slowly into the terrain, rather than sinking instantly.
Ultimately I don’t think this is too big of an issue though, as in an actual game, it’s unlikely that you would have deformable terrain that’s as large and deep as in my demo, so it would be a lot easier to get away with other more practical methods of collision which would have looked tacky in my demo.
You can find the source for this project here:
https://github.com/TarAlacrin/HeightmapOnTheGPU