// HACKER NEWS — CYBERSECURITY
Optimizing Anamorphic Sculptures
At my workplace there’s an anamorphic sculpture that reveals a recognizable image only when viewed from a specific vantage point. The work of Jonty Hurwitz
explores that technique with mirrors (catoptric anamorphosis) and with perspective (oblique anamorphosis). In Blue Singularity, Jonty’s own head sculpture is displayed in slices. Those slices compose the head when looking from a specific angle.
My office’s sculpture always gets me thinking about how I would optimize for that kind of result. To experiment with that, I wrote an application that, given a set of 3d meshes, arrange them in the 3d space so that a target silhouette is rendered from a specific camera angle.
Sculptures are generated by optimizing the position, size, and rotation of different meshes with gradient descent. Meshes are placed randomly in the scene, the scene silhouette is rendered. The render is then compared to a target image, a loss is computed, and errors are backpropagated to the input parameters (meshes positions, rotations, and scale). At each step meshes are moved and adjusted so the rendered silhouette comes closer to the target image.
Although the process has few steps, the implementation is not simple. There are multiple implementation details necessary to generate visually interesting images. First and most important, to be able to backpropagate errors from the loss to the input, the rendering step should be differentiable. The loss function should also be tuned to value details. In other words, just getting most of the image right is not enough to get a similar silhouette.
A rasterizer is a function that maps vertices to pixels. It is responsible for converting a list of triangles (3d mesh) into an image. For every pixel it asks whether that pixel is covered by the triangle. This is a binary operation: covered or not. Nudging a triangle slightly changes nothing until its edge crosses a pixel center, and then the pixel flips. This means that the derivative is zero almost everywhere and undefined on the boundary. To backpropagate errors in this setting, some constraints need to be relaxed. Luckily, Liu et al.
had this problem before and published a paper on soft rasterizers.
The main idea in SoftRas is to replace the boolean (pixel inside or outside triangle) with a sigmoid of the signed squared distance to the triangle’s boundary. For pixel \(i\) and triangle \(j\), let \(d_{ij}\) be the distance from the pixel center to the triangle’s outline, \(\delta_{ij}\) = +1 if inside, -1 if outside triangle, and \(\sigma\) a constant to control the transition. Then the “belonging” of a pixel in a triangle can be measured by \(p_{ij}\):
Move the pixel center across a triangle edge. A smaller \(\sigma\) makes the transition sharper; a larger \(\sigma\) lets a missed shape send a gradient farther. A hard rasterizer is the \(\sigma\) → 0 limit.
As you can see, different \(\sigma\) values have different trade-offs. With a small \(\sigma\), errors are only propagated close to the triangle edge. Meshes placed far away won’t be moved. Having a large \(\sigma\) will make the silhouette more diffuse. The adopted solution was to anneal \(\sigma\): start large and shrink it as the optimization progresses. A large \(\sigma\) is the renderer squinting. The silhouette is a blur, but a mesh far from where it should be still feels a pull. Shrinking \(\sigma\) opens the renderer’s eyes and sharpens the edges.
Another optimization was splitting the meshes in bins. Not every pixel should be checked against every triangle. Also recomputing gradients during backprop helps keeping the memory in check.
The naive approach is to compare the rendered silhouette with the target image and adjust the objects. The problem with that is that although the silhouette can have a lot of “mass” in the center, the edges are what make silhouettes distinguishable. On top of that, I wanted the generated sculptures to be manufacturable in the real world, which meant avoiding collisi