Traces (2021)

While I was at Google Creative Lab (2021-2022), one of the longest running projects I worked on was a partnership with Australia’s leading Indigenous publishing house, Magabala Books. Through the process we got to work with artist/poet Kirli Saunders and artist/illustrator Kamsani Bin Salleh to design and develop a project that applied digital technology in service of storytelling. The outcome of this was Traces; a voice-to-art experience, honouring 60,000+ years of Indigenous storytelling.

The wrap-film above gives a strong sense of the meaning and experience of the work in the words of the artists. Traces was primarily shaped by aspects of Kirli and Kam’s arts practices. The projections were generated in response to spoken word; both live in the space and from a poem written and recorded by Kirli, and the visual elements were drawn by Kam and recombined programmatically to form new compositions.

In the rest of this page I’ll go into some detail of my role in developing the technical side of the project.

The concept for the work that Kirli and Kam most graviated towards was of a system to illustrate speaking and yarning. Kam is a prolific artist and illustrator, and Kirli an equally prolific artist and poet, so the concept met at the intersection of their practices.

Kam provided an “alphabet” of marks and illustrations as a basis for the generative system. The initial task was to convert these provided drawings to a form that we could animate, and recombine for form new compositions.

Kam has developed a particular visual style of closely woven lines with hidden forms and figures. Our intention was to go some way in producing compositions in his style. To do this we needed to write some code to determine whether a collection of marks were too close (or indeed too far) from those adjacent.

With these elements, and with Kam’s alphabet of illustrations we were able to generatively produce new compositions.

We explored several ideas for the colour palette of the work, but what resonated most with Kirli and Kam was drawing on the colours of Country.

In an early experiment I sampled sattelite imagery in a line across Australia and used the imagery to generate colour palettes. This experiment turned into a more intentional sampling process to find beautiful palettes in the landscape from across the entire country.

Our goal was for the compositions, illustrations and so on to be generated in response to the spoken word. This in itself involved a long process of research to extract key features from audio that could be used to drive the audio in a way the felt natural and responsive. With some help from an audio expert, we extracted volume, pitch, intonation, pulse and pace from the audio waveform.

These features in the end were key to making illustrations which appeared to flow from the voice. With the exception of pace, all other sonic features were used to either shape the marks or to change the way the were drawn on the composition.

This brings us to motion. To make the illustrations feel responsive the needed flow out slowly for long vowels and hums and quickly for sharp quick sounds, not simply to appear out of nowhere. I developed a bunch of animation experiments to change the way that the marks appeared and make this feel fluid and alive.

These elements, illustrations, composition, palettes, audio etc to form the final version of the work. The version you can see below was how Traces was primarily developed. It was adapted for the specific requirements and three dimensionality of the Sydney Opera House’s Tide Room projection space for the final performances that can be seen in the wrap film at the top of this page, but that’s another story.