[ { "date": "2020-05-20 00:00:00 +0000", "title": "Alt.CHI 2020: a reader’s guide", "url": "/research/altchi-2020-a-readers-guide/", "content": "ACM’s Computer Human Interaction conference (CHI) is big. It’s too big really. Too big, that is, to really engage with the glut of research accepted each year. But alt.CHI is small; just 24 papers of ten pages each. And despite its size each year it covers the spectrum of conceptual issues the field of HCI is currently dealing with. <> <> Alt.CHI, in case you don’t know, is the experimental, reflexive and boundary-pushing track within the CHI. It hosts work that is either too experimental or too introspective for the main conference. <> <> CHI was cancelled this year due to the unfolding pandemic, but while main track papers were allowed to submit video presentations to ACM and accepted into local conferences, Alt.CHI was *properly* cancelled. With no hope of seeing the presentations, I spent last week reading the 24 papers accepted into this year’s Alt.CHI proceedings and I’ve summarised them below for those who don’t have 3 days to spare. <> <> The proceedings begin with an unexpected piece of good news; [Jason T. Jacques](https://twitter.com/jtjacques)’ "CHI 2020: Right Here, Right Now"[^jacques2020] estimates the carbon footprint of the CHI conference over two decades. The paper establishes conference travel as far and away the biggest cause of carbon emissions. According to Jacques’ analysis, CHI 2020 was on track to have the largest carbon footprint in the conference’s history, amounting to approximately 5,600 tonnes of CO2. This in large part due to Honolulu’s distance from CHI’s estimated "centre of population". The cancellation of this year’s conference is of course unfortunate, but it offers a moment to consider the environmental cost of *in person* attendance and whether alternatives are possible. In 2019 CHI introduced a sustainability position to the committee, while this is admirable, it seems unlikely that we will make serious reductions in carbon footprint without addressing the demand for international travel. Meanwhile, according to Jacques, CHI has only demanded "with increasing vigour" that accepted authors attend the conference.[^jacques2020] This is a difficult but necessary conversation we will need to have if CHI is to take its commitment to sustainability seriously. <> <> Next up is a paper which embodies the ideals of Alt.CHI; John Desnoyers-Stewart’s "Engaging with a CHI Paper Through Embodied Action"[^desnoyers-stewart2020]. The paper raises issues around the primacy of text in academic knowledge production. If it is true that "we know more than we can tell", how can we access that author’s tacit knowledge? The paper practices *emodied action* as a way of knowing in HCI. In the paper Desnoyers-Stewart attempts to make sense of another CHI paper "Design for Collaborative Survival"[^liu2018] through a series of "Urban Forays" in search of fungi reclaiming urban environments. These active engagements in Liu et al.’s text are documented in a narrative style and though a number of technical artefacts were produced, the focus is always on the active exploration as the primary generator of knowledge. The irony of interpreting this work through its paper alone is not lost on me. Perhaps this is a case where *in person* conference attendance really would have made a difference. <> <> Issues of representation are a major theme in this year’s proceedings. Contributions consider christian hegemony, (global) northern centrism and ageism in HCI. Marisol Wong-Villacres and co-authors’ "Reflections from the Classroom and Beyond" examines Global North-centrism/colonialism in the HCI education grounded in the lived experience of the authors.[^wong-villacres2020] The paper reminds us that colonialism did not end with modernity. In HCI, despite some adoption of postcolonial and transnational theory, this work is still positioned "at the margins." The paper reflects on the problem of translation; whereby translations "will never mean exactly what was meant by the authors" and advocate for a *plurivocal* interpretations. The authors draw on concepts from "Southern" theorists; Gloria Anzaldua’s *mestiza consciousness* and Walter Mignolo’s *colonial difference* to enrich the discourse which often exclusively operates with ideas drawn from western european thinkers. <> <> Another underrecognised centrism in HCI work is that of Christianity. <> Religion is studied in HCI but rarely do we consider the religiousity of supposedly "secular" representations ubiquitous in technology. <> In "Envisioning Jewish HCI", Jessica Hammer exposes Christian cultural hegemony in HCI.[^hammer2020a] In an illustrative example, Hammer notes that the representation of the calendar in most software is that introduced by Pope Gregory and enforced by Christian rulers. I am reminded of mapping software like google maps and open street maps, which despite a wealth of possibilities, still use the Mercator map, a representation connected to histories of colonialism. Like the previous paper, Hammer shares concepts from Jewish thought; *elu v’elu*, *tzedakah*, the *eishet chayil*, and *ma’alin bakodesh*, which suggest new alternative approaches to research. Hammer asks what might HCI look like if we adopted these concepts? <> <> Postcolonial theory is again invoked by Aloha Hufana Ambe and co-authors’ in "An Oldy’s Lament".[^ambe2020] The paper engages with older people’s perspectives of technology which are rarely considered in the design of new technology. The authors conducted a series of co-design workshops with older creative writers, one of whom, 86 year old Julie Butler, wrote the poem "An Oldy’s Lament". The paper is structured around Butler’s poem and offers a postcolonial reading of the work. The postcolonial lens offers a useful language to describe how technology colonises and 'others’ older people. The paper then presents Butler’s poem as a form of resistance. I liked this paper but I was disappointed that Butler was not included as an author. <> <> In another paper on poetry is Mark Amerika and co-authors’ "Fatal Error".[^amerika2020] The work engages with questions of artificial creativity and its philosophical consequences for authorship. Fatal Error features an AI spoken word poet complete with a 3D avatar resembling the first author, and trained on their body of poetic work. According to the authors, as the AI doppelganger learns it will also be capable of creative "autonomy", the project in turn transforms Amerika’s personal artistic development. In this sense creativity is reframed as "psychic automatism" and the AI is afforded the status of a collaborator in the creative process. <> <> <> So far the papers I’ve described have made clear research contributions to HCI, some might even be acceptable in the main CHI conference track. But Alt.CHI also regularly hosts work so wild that it has "serious" researchers raising eyebrows. In "What’s Your Problem with the Dog Internet?" Ben Kirman and co-authors reflect on ten years of absurdist research, and make the case for its value as a critical approach in HCI.[^kirman2020] The paper consolidates ideas from a set of projects based around "the dog internet"; a technology "like the human internet, but for dogs". The authors relate "the absurd" to movements in HCI around *troublemaking*; notably Dunne and Raby’s "critical design" and Donna Haraway’s call to stay "with the trouble". In the context of HCI’s increasingly commercial research agenda and the expecations of employers and colleagues, absurdism is difficult to justify especially when, from the outside it might appear to be nothing but increasingly elaborate "in-jokes". And yet absurdism as the authors show can be "profoundly, accidentally, productive", revealing aspects of our culture and technology which are taken for granted and which could be otherwise. <> <> Similar methods are used to great effect in the Christine Geeng’s (and anonymous co-author’s) spectacular paper "EGregor".[^geeng2020] The paper explores issues of trust and privacy with regard to smart personal assistants such as Siri, Alexa and Google Home. Geeng examines the vocal and behavioural affectations which lead us to place trust in these systems, in spite of their creepy, data-extractive behaviours. The paper imagines a smart personal assistant, eGregor, a hiveminded being whose sublime and repulsive physical appearance offers a corrective to parasocial design principles which suggest a trustworthy companion. "eGregor presents itself not as a human individual, but as a vast and unknowable alien collective. This persona is meant to encourage a healthy mental model of the assistant...elements borrowed from ... cosmic horror help to establish eGregor’s alien indifference to the user." Although Geeng does not use the term "design fiction" this paper intersects with the design fiction methodology that is becoming increasingly common in Alt.CHI (and even CHI proper). <> <> One paper which does invoke "design fiction" directly is "Children of 2077", by Oğuz’Oz Buruk and co-authors. The paper is an ambitious work, constituting an entire (fictional) proceedings in its own right.[^buruk2020] The paper is a collection of thirteen abstracts which imagine Transhuman futures for children’s technologies. This stucture proves incredibly rich offering a plurality of possible futures and provocative ideas. The condensed form of the abstract leaves plenty of room for imagination on the part of the reader; Mattia Thibault’s contribution, for example, imagines the children of "sunken nations (such as Maldives and Tuvalu)" and promises to survey the new uses of public spaces made by "amphibian youth". The paper’s findings must be imagined by the reader, nonetheless this and other abstracts in "Children of 2077" should cause us to question whether we can design our way out of the world we are creating. <> <> Haider Akmal and Paul Coulton’s "The Divination of Things by Things" is another design fiction in this years proceedings.[^akmal2020] The authors imagine a mobile application, *Madame Bitsy’s Fantastic Future Forecasting and Fortune Telling Emporium for the Internet of Living Things*, which serves to illuminate the experiences of IoT objects. The paper imagines an "Object-Oriented Fortune Telling" performed by an app in the style of a tarot reader. The visual design of this project is outstanding and the screenshots throughout serve to bring the fiction to life. So does the fact that this paper only announces itself as a "design fiction" in the conclusion, even the abstract gives no suggestion of fictionality. Design fiction is one of the author keywords on the paper but I must have missed that section as I read the paper. I must admit that I was utterly taken in by the paper and felt a little dissappointed to discover it is fictional. I know I am not alone in this because I saw during alt.CHI’s public review process that others had similar reservations. <> <> "The Divination of Things by Things" is not the only surprise design fiction in this year’s proceedings. Franziska Pilling and co-authors’ "The Process of Gaining an AI Legibility Mark" imagines an iconography for Artificial Intelligence legibility akin to a laundry labelling scheme.[^pilling2020] Contemporary AI systems are hopelessly illegible. While some progress has been made in explaining these systems, this has barely kept pace with the increasing size and complexity of the systems. What’s more, we are installing these systems in public places and public institutions as well as countless commercial projects. This design fiction imagines a widely adopted system of iconongraphy, controlled by an international body which certifies AI systems and services by the level of legibility and provides recognisable labels to indicate the level of certification to subjects/consumers. While this paper, like the last, only announced itself as a design fiction in the conclusion, I clued on much earlier, perhaps because I had been fooled once. I am not opposed to this practice as such; it has the fascinating effect of causing me to read the design fiction with greater focus, commiting to memory ideas and references to follow up after reading. Conversely, this changes the way I read non-fictional papers, leaving me unsure anytime I read something that seems too wild to be true. <> <> Such was the case with Mirela Alistar and Margherita Pevere’s paper "Semina Aeternitatis".[^alistar2020] The project was so fascinating and beyond the usual work of CHI that I could barely believe it. Midway through the paper I had to stop to run a keyword search for "fiction". When no match was found I read the rest in awe. The paper describes a biotechnology project to splice an elderly woman’s memory (recounted as text and converted into DNA code) into bacteria. The bacteria produced a biofilm which was then exhibited, giving a material, tangible, olfactory experience to the memory. This creative application of biotechnology, fairly uncommon in HCI, offers new ways for humans to interact with data. While this embodiment of data has issues of legibility; e.g. does the encoded memory really change the look, feel or smell of the biofilm? It also raises questions around communication and permanence; embedded in a living organism, the data/memory can self-replicate, mutate or even transfer to another organism. <> <> Changing pace now, in "Suffering-Centered Design",[^tomlinson2020] Bill Tomlinson asks what principles guide our design choices in HCI. As many of the papers in Alt.CHI reflect, there are many unexamined consequences in the design of technology. We as humans "are skilled at closing their minds to the suffering of others" and, as Tomlinson argues, information technology is a powerful enabler of this tendency. The paper explores the potential for a *suffering-centrered design*, which foregrounds the suffering of others rather than helping us ignore it. Rising to global challenges, of poverty and climate change will require us to face uncomfortable truths about our world. Tomlinson shows how HCI can play a part in rising to this challenge. <> <> One paper which does promote an empathic response is Miriam Sturdee and co-authors’ "Seven Year Glitch".[^sturdee2020] The abstract sets the tone for the paper; "I(am)MEI: 013709002488246. ... I was not built to last. This is my story." The paper recounts the life of an ailing smartphone as it breaks down over a several years. The story is expressed through a series of journal entries written from the perspective of the phone, and illustrated with a set of increasingly corrupted photos. The authors celebrate the beauty of these glitches while reflecting on the tradgedy of the short lives of our electronic devices. In the sorrowful conclusion to the paper, the protagonist describes a data transfer it knows to be a sign of the end. <> <> Another paper which addresses the short lives of electronic devices is my own contribution to this year’s proceedings; "Eating Computers Considered Harmful".[^browne2020] The paper, co-authored with Ben Swift and Terhi Nurmikko-Fuller, examines the damage to health and environment caused by global e-waste. Electronic devices constitue the most toxic stream of waste we produce. This toxic cocktail of substances inevitably make their way back into the soil, water and air where they are consumed by organisms (including humans). Even "recycling" practices only extract small amounts of valuable materials and burn or dispose of the rest. The initial idea for this paper was to work out how much of a computer I could eat without severly endangering myself. The answer I discovered is 0%; even the plastic in a computer is more toxic than regular plastic! And yet we are already eating our computers through offgassing, leeching and bioaccumulation. These bodily interactions with technology are generally ignored by HCI, but why? Ranting aside, this paper imagines a world where you have to eat your old computer or smart phone before you can upgrade and offers a diet plan that will have you eating your way to that new iPhone in no time! <> <> The next couple of papers are close to my heart because they continue to consider themes addressed by myself and Ben Swift in my first Alt.CHI.[^browne2018] William Seymour and Max Van Kleek invoke notions from Shintoism to make sense of the apparent character in Smart Personal Assistants in "Does Siri have a Soul?"[^seymour2020]. Clearly this paper addresses similar themes to "EGregor", though the characterisation of these objects as "Kami" is far more positive. When you buy a smart personal assistant, the authors claim, "what you are really buying is the voice or the presence that inhabits the plastic and silicon". The paper imagines voice assistants which can bring benefits but can also make life tough if they are not treated with respect. "The Wild Soul" imagined by the authors is a capricious voice assistant who intentionally misunderstands your requests. Gifts and offerings should be brought to the assistant so as to maintain the harmonious soul’s presence. Most voice assistants are placated with an offering of personal data. The paper draws attention to the parasocial aspects of Smart Personal Assistants; how does their behaviour inculcate trust, might it be exploited to sell more products or extract more data? This is another fine example of the methodological value of design fiction. <> <> Another paper drawing on spiritualism to examine techonology is Andrew McNutt and co-authors’ "Divining Insights".[^mcnutt2020] Drawing on the interaction modes and visual form of Tarot, the authors present the automatic data analysis tool *Sortilège*. The authors consider the opacity of statistical modelling and machine learning in the context of automated visual analyitics. These systems promise to reveal what is important or interesting in a dataset, generating so called "insights". Like the systems they leverage, these automated methods are brittle and biased, and offer only answers without explanations. *Sortilège* offers a genuinely useful data analytics tool which uses the interaction modes of cartomancy to provoke thought. Unlike other automated "insight" generators, *Sortilège* does not present itself as authoritative or objective, instead, as in Tarot, it is the role of the user to interpret the cards in light of their question. The set of cards includes various graphs of the data along with "Major Arcana" which include iconography designed to foster introspection. <> <> Galen Harrison’s commentary on "Divining Insights" is also worth a read as it critiques the growing body of work in Alt.CHI which adopts spirituality to examine technology. Harrison points out that these existing works invoke spirituality "as a means to a scientific end" and proposes that future work should engage with occult or mystical practices on their own terms. Of all the papers in this years Alt.CHI, perhaps only Sharifa Sultana and co-authors’ "Parareligious-HCI" meets this goal.[^sultana2020] The paper reflects on a 3-year long ethnographic study on wellbeing practices in rural Bangladesh. The authors examine what they term "parareligious" practices such as witchcraft which are discarded by religions and modern medicine/science, but which nonetheless are culturally embedded in wellbeing practices. These practices have been mostly ignored by HCI. The authors argue that these practices constitue Indigenous knowledges and "alternative rationality" of rural populations. The paper concludes with a discussion of ways in which HCI might design for parareligion by replicatiing analogies, metaphors and methods of the populations. <> <> Privacy and data ethics are common themes at CHI and Alt.CHI. One contribution to this converation this year is Simy Kaur Gahoonia and co-authors’ "Upon Not Opening The Black Box".[^gahoonia2020] The paper use "dramaturgical" methods to explore these ideas, which is to say, they engage with the questions through the medium of a play. The *dramatis personae*, presumably modelled after members of their lab, are the lab director, lab manager, various lab members, a data inspector, an arrogant data consultant. The play explores conflicting ethical considerations in the use of data and the paper has the tone of a socratic dialogue, with the (wise old?) lab director gently leading the discussion, however, unlike a soctratic dialogue no conclusion is reached. Instead the paper suggests that data use/consent cannot be decided *a priori* even in the context of regulation such as the GDPR. <> <> Another paper which models positive discourse in HCI is "No Hidden Catch, No Strings Attached".[^niess2020] In the paper Jasmin Niess and Paweł Wozniak propose a twelve-step programme to achieve inter/cross-disciplinary discourse. The value of interdisciplinarity is well acknowledged in HCI, but interdisciplinary communication is easier said than done. The paper addresses significant cultural issues in HCI which make cross-disciplinary conversation difficult. They suggest that we book too much time (there’s never enough), find the right people (diversity is key), forget about publication (for a moment), assume you know nothing (be open-minded), assume you know everything (no bad ideas), and bring pens and paper, amongst other valuable suggestions. These constitute a recipe for healthy discourse. <> <> Another paper addressing research culture is Jessica Hammer’s second contribution to this years proceedings as first author. In "Lab Counterculture"[^hammer2020b] Hammer and co-authors approach lab culture as a design problem. They identify "dark patterns" which structure the university and propose designs which correct these. Amongst the issues they identify are dynamics we will all no doubt be aware of; "quantitative outcomes, perfectionism, competition, time scarcity, power dynamics, bias towards maintaining the status quo, and financial stress". These disproportionately affect students and faculty from underrepresented minorites. The authors suggest correctives they have applied in their own lab(s) such as celebrating effort rather than outcomes, elevating community over individuality, prioritising time together etc. and explain the effects these active choices had on their lab culture. The paper is a must read for anyone in the position to affect change in a research group or lab. <> <> Similarly, in "I am just terrified of my future"[^ymous2020], authored by a group of disabled scholars who list the first author as "Anon Ymous" describe the *epistemic violence* enacted upon disabled people in HCI. They identify the pervasive dehumanisation of disabled persons in existing HCI work on disability. The paper aims to take ownership of HCI’s disability narrative. The paper identifies entrenched culture of ableism which seeps into academia and into HCI despite its narrative of "doing good". There are many important contributions in this paper, but I want to highlight one in particular by Judith Good which identifies the conflict between HCI’s engrained culture of "finding a solution to a problem" and how this leads HCI disability researchers to persistently tout the "problem" of, for example, ADHD and never acknowledge the strengths of neurodiversity or any other disability. <> <> The papers we have covered so far have primarily addressed the research component of the university’s mission, and a few have even addressed teaching. Pietro Crovari and co-authors’ "Crime Story as a Tool for Scientific and Technological Outreach"[^crovari2020] however, considers the so-called "third mission" of the university; *public outreach*. The paper describes "Death on the Nile", an interactive, public exhibition which in the form of a whodunnit style crime-solving game. The stated aim of the project is to teach participants about a range of interactive technologies (from conversational agents to smart objects to smart spaces). The authors draw on concepts from "non-formal education" and "serious games" to frame the work. This paper has the dubious honour of being the only Alt.CHI paper in this year’s proceedings to include a user study. From this the authors "deduce" in the discussion, that their experience is inclusive, engaging and stimulating. I don’t want to criticise this paper too harshly, it looks to be a very creative and engaging interactive experience, but the methods used and the positivist language in the findings felt like something which is more aligned with CHI’s main track. <> <> The proceedings end where they began; with sustainability. In "Low Power Web", Max Willis and co-authors examine the carbon footprint of the web so well hidden behind the tech industry’s green image.[^willis2020] Global data centres now consume more energy than the UK and account for at least 2% of the worlds electricity demand. This is a sobering reminder to those celebrating the emissions saved from CHI’s cancellation. Moreover, as conference participants flock to the ACM website to read papers (like I just did) or to stream video presentations in lieu of the conference, we are reminded that every byte has an energy cost. The paper acknowledges the challenges in measuring the energy cost of data, but cites a number of estimates including a shocking comparison (to me anyway) that you can run a fridge for several hours on the energy required to stream 100 minutes of video online. The paper then turns to a "media forensic analysis" of the ACM website, the landing page of which "makes over 100 server requests and loads up to 12 MB of data. Having worked as a web developer myself, I can attest to the fact that bytes are rarely conserved anymore now that internet speeds in many parts of the world have removed concerns for wait times. And yet when we consider the carbon cost, saving bytes is still worth while, especially if you control a site with high traffic. The paper includes many ideas for how to do this as well as suggestions for ways to power a website renewably. This paper inspired me to finally delete google analytics from this site (you’re welcome, I wasn’t checking it anyway), and to reduce image sizes. I’m also looking into the possibility of solar powering another site I maintain, as the authors encouragingly state; "it’s not rocket science - all that is needed is some motivation, time, and an investment in human energy". Whether bookending Alt.CHI with sustainability papers was a conscious choice on the part of the organisers, it serves as a reminder that tech is not the ethereal, platonic world we often mistake it to be; this is a reckoning HCI is yet to face up to. <> <> <> <> ## Notes <> <> [^jacques2020]: Jacques, Jason T. "CHI 2020: Right Here, Right Now? A bottom-up approach to estimating the carbon emissions from more than twenty years of CHI conference travel." In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-13. 2020. [ACM Digital Library](https://dl.acm.org/doi/abs/10.1145/3334480.3381806) <> <> [^ambe2020]: Ambe, Aloha Hufana, Margot Brereton, and Alessandro Soro. "An Oldy’s Lament: Poem of Resistance and Resilience of the'Othered’ in Technology Colonisation." In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-10. 2020. [ACM Digital Library](https://dl.acm.org/doi/abs/10.1145/3334480.3381807) <> <> [^wong-villacres2020]: Wong-Villacres, Marisol, Adriana Alvarado Garcia, and Javier Tibau. "Reflections from the Classroom and Beyond: Imagining a Decolonized HCI Education." In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-14. 2020. [ACM Digital Library](https://dl.acm.org/doi/abs/10.1145/3334480.3381808) <> <> [^seymour2020]: Seymour, William, and Max Van Kleek. "Does Siri Have a Soul? Exploring Voice Assistants Through Shinto Design Fictions." In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-12. 2020. [ACM Digital Library](https://dl.acm.org/doi/abs/10.1145/3334480.3381809) <> <> [^browne2020]: Browne, Kieran, Ben Swift, and Terhi Nurmikko-Fuller. "Eating Computers Considered Harmful." In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-13. 2020. [ACM Digital Library](https://dl.acm.org/doi/abs/10.1145/3334480.3381810) <> <> [^kirman2020]: Kirman, Ben, Shaun Lawson, and Conor Linehan. "What’s Your Problem with the Dog Internet?." In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-14. 2020. [ACM Digital Library](https://dl.acm.org/doi/abs/10.1145/3334480.3381811) <> <> [^tomlinson2020]: Tomlinson, Bill. "Suffering-Centered Design." In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-19. 2020. [ACM Digital Library](https://dl.acm.org/doi/abs/10.1145/3334480.3381812) <> <> [^gahoonia2020]: Gahoonia, Simy Kaur, Pedro Ferreira, Marisa Cohn, Line Henriksen, Katrine Meldgaard Kjær, Michael Hockenhull, Baki Cakici et al. "Upon Not Opening The Black Box." In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-9. 2020. [ACM Digital Library](https://dl.acm.org/doi/abs/10.1145/3334480.3381813) <> <> [^mcnutt2020]: McNutt, Andrew, Anamaria Crisan, and Michael Correll. "Divining Insights: Visual Analytics Through Cartomancy." In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-16. 2020. [ACM Digital Library](https://dl.acm.org/doi/abs/10.1145/3334480.3381814) <> <> [^amerika2020]: Amerika, Mark, Laura Hyunjhee Kim, and Brad Gallagher. "Fatal Error: Artificial Creative Intelligence (ACI)." In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-10. 2020. [ACM Digital Library](https://dl.acm.org/doi/abs/10.1145/3334480.3381815) <> <> [^niess2020]: Niess, Jasmin, and Pawel W. Wozniak. "No Hidden Catch, No Strings Attached: Twelve Steps to Cross-Disciplinary Conversations about Technology." In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-11. 2020. [ACM Digital Library](https://dl.acm.org/doi/abs/10.1145/3334480.3381816) <> <> [^alistar2020]: Alistar, Mirela, and Margherita Pevere. "Semina Aeternitatis: Using Bacteria for Tangible Interaction with Data." In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-13. 2020. [ACM Digital Library](https://dl.acm.org/doi/abs/10.1145/3334480.3381817) <> <> [^hammer2020a]: Hammer, Jessica. "Envisioning Jewish HCI." In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-10. 2020. [ACM Digital Library](https://dl.acm.org/doi/abs/10.1145/3334480.3381818) <> <> [^sultana2020]: Sultana, Sharifa, Zinnat Sultana, and Syed Ishtiaque Ahmed. "Parareligious-HCI: Designing for 'Alternative’ Rationality in Rural Wellbeing in Bangladesh." In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-13. 2020. [ACM Digital Library](https://dl.acm.org/doi/abs/10.1145/3334480.3381819) <> <> [^pilling2020]: Pillling, Franziska, Haider Akmal, Paul Coulton, and Joseph Lindley. "The process of gaining an AI Legibility mark." In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-10. 2020. [ACM Digital Library](https://dl.acm.org/doi/abs/10.1145/3334480.3381820) <> <> [^buruk2020]: Buruk, Oğuz’Oz, Oğuzhan Özcan, Gökçe Elif Baykal, Tilbe Göksun, Selçuk Acar, Güler Akduman, Mehmet Aydin Baytaş et al. "Children in 2077: Designing Children’s Technologies in the Age of Transhumanism." In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-14. 2020. [ACM Digital Library](https://dl.acm.org/doi/abs/10.1145/3334480.3381821) <> <> [^crovari2020]: Crovari, Pietro, Fabio Catania, and Franca Garzotto. "Crime Story as a Tool for Scientific and Technological Outreach." In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-10. 2020. [ACM Digital Library](https://dl.acm.org/doi/abs/10.1145/3334480.3381822) <> <> [^akmal2020]: Akmal, Haider, and Paul Coulton. "The Divination of Things by Things." In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-12. 2020. [ACM Digital Library](https://dl.acm.org/doi/abs/10.1145/3334480.3381823) <> <> [^hammer2020b]: Hammer, Jessica, Alexandra To, and Erica Principe Cruz. "Lab Counterculture." In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-14. 2020. [ACM Digital Library](https://dl.acm.org/doi/abs/10.1145/3334480.3381824) <> <> [^desnoyers-stewart2020]: Desnoyers-Stewart, John. "Engaging with a CHI Paper Through Embodied Action: A Situated Analysis of" Design for Collaborative Survival"." In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-12. 2020. [ACM Digital Library](https://dl.acm.org/doi/abs/10.1145/3334480.3381825) <> <> [^sturdee2020]: Sturdee, Miriam, Joseph Lindley, Regan Harrison, and Tine Kluth. "The Seven Year Glitch: Unpacking Beauty and Despair in Malfunction." In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-11. 2020. [ACM Digital Library](https://dl.acm.org/doi/abs/10.1145/3334480.3381826) <> <> [^geeng2020]: Geeng, Christine. "EGregor: An Eldritch Privacy Mental Model for Smart Assistants." In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-9. 2020. [ACM Digital Library](https://dl.acm.org/doi/abs/10.1145/3334480.3381827) <> <> [^ymous2020]: Ymous, Anon, Katta Spiel, Os Keyes, Rua M. Williams, Judith Good, Eva Hornecker, and Cynthia L. Bennett. "I am just terrified of my future"—Epistemic Violence in Disability Related Technology Research." In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-16. 2020. [ACM Digital Library](https://dl.acm.org/doi/abs/10.1145/3334480.3381828) <> <> [^willis2020]: Willis, Max, Julian Hanna, Enrique Encinas, and James Auger. "Low Power Web: Legacy Design and the Path to Sustainable Net Futures." In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-14. 2020. [ACM Digital Library](https://dl.acm.org/doi/abs/10.1145/3334480.3381829) <> <> [^liu2018]: Liu, Jen, Daragh Byrne, and Laura Devendorf. "Design for collaborative survival: An inquiry into human-fungi relationships." In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 1-13. 2018. <> <> [^browne2018]: Browne, Kieran, and Ben Swift. "The Other Side: Algorithm as Ritual in Artificial Intelligence." In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 1-9. 2018. <> [ACM Digital Library](https://dl.acm.org/doi/abs/10.1145/3170427.3188404) <> <> ", "slug": "altchi-2020-a-readers-guide" } , { "date": "2019-04-07 00:00:00 +0000", "title": "Making old media new", "url": "/research/making-old-media-new/", "content": "![](https://storage.googleapis.com/kieranbrowne-public-files/images/old_new_1.gif)<> Last year I was approached by Prof Martyn Jolly and Dr Elisa DeCourcy to design a chromatrope for their Australian magic lantern conference, [*Heritage in the Limelight*](http://soad.cass.anu.edu.au/research/heritage-limelight). The conference aimed to shed light on Australia's rich history of magic lantern slides and performances, and to support creative reuse and revival for the medium.<> <> I was introduced to Martyn's collection of mechanical slides, known as chromatropes, which were used to produce moving images as early as the 18th century. Chromatropes use a rackwork mechanism to rotate two circular glass slides in opposite directions. These slides are painted with colourful abstract patterns and produce complex moving images when rotated. It took a while to wrap my head around the rules of the medium and to be honest, that so much motion is squeezed out of the two static slides still blows my mind.<> <> My early experiments weren't particularly compelling so I won't share them here, but you can find them on [instagram](https://www.instagram.com/_kieranbrowne/). The earliest decent works came after switching from processing to glsl and beginning with very simple forms. The interface below allows you to explore the space of only four features. Even in this small experiment the range of possibilities is surprising.<> <> ![](https://storage.googleapis.com/kieranbrowne-public-files/images/old_new_4.gif)<> Expanding this process to include colour, I looked for a way to generate the kinds of intricate features that you find in original chromatropes and stumbled across iterated function systems (IFSs), a branch of mathematics used to generate fractals. Early experiments were promising and I found that a very small number of iterations worked better than full on fractal designs. <> <> As an alternative line of experimentation, and partially just as an excuse to experiment with some recent AI/ML methods I took a detour to experiment with training a machine learning model on chromatropes from Martyn's collection. The model, [pix2pix](https://affinelayer.com/pixsrv/), is trained to predict the following frame from the previous one. Then, this process is run interatively on its own output to create a short film. The results are psychedelic; the hard geometry of the chromatrope quickly gives way to a flow of lights and colours. The model has learned something of the visual phenomena of the chromatrope but is freed from what Martyn Jolly calls its “mechano/optical logic”.<> <> ![](https://storage.googleapis.com/kieranbrowne-public-files/images/old_new_2.gif)<> ![](https://storage.googleapis.com/kieranbrowne-public-files/images/old_new_3.gif)<> Returning to the original task, designing a new chromatrope, I was helped by Miheng Dong to fabricate a few of my digital designs for projection through an original magic lantern. This was surprisingly difficult given modern tech, and makes the original chromatropes all the more impressive. Miheng translated my designs for the lazer cutter and built the slides from coloured acetate. The process worked, and although the result was less sharp than the digital design, this gave the result an imperfect charm that is lost on most digital imagery. Unfortunately I don't have any videos of the fabricated chromatrope but I'll try to track some down.<> <> It was a stimulating challenge to work within the chromatrope's restrictive form and a surreal experience to see my 21st century design projected through a 19th century magic lantern. Early chromatrope makers were incredibly creative with their limited means and its fascinating to try to reverse engineer their processes. For those interested, I wrote another post explaining [how chromatropes work](/research/how-chromatropes-work), but no doubt there are still tricks that I have yet to discover and perhaps more that even the original makers did not discover.<>", "slug": "making-old-media-new" } , { "date": "2019-04-05 00:00:00 +0000", "title": "How chromatropes work", "url": "/research/how-chromatropes-work/", "content": "", "slug": "how-chromatropes-work" } , { "date": "2018-09-13 00:00:00 +0000", "title": "AI’s long, hot summer", "url": "/research/ais-long-hot-summer/", "content": "Machine learning is fast becoming the leading paradigm in computer science. Even if the declinists are correct that we’re heading toward an AI winter,[^Piekniewski2018],[^MITtechreview] its fair to say the last few years are evidence of a major change in the research climate. <> <> A few days ago Yan LeCun posted [a tweet](https://twitter.com/ylecun/status/1038506237015539712) noting that he, Yoshua Bengio and Geoff Hinton now fill the top three ranks for citations per day in computer science.[^Bengio2018] Hinton and Bengio also came in first and second respectively for the same metric in both 2017 and 2016. <> <> There is plenty of debate around the accuracy of bibliometrics, but even taking these numbers skeptically it is clear that machine learning has made a mark in computer science that will not be easily erased. <> <> Putting aside dubious predictions of the end of work; machine learning affords a kind of automation previously beyond the reach of computers. Machine learning is perhaps the first computing technology flexible enough to capture culture; [this is revelealed most plainly when systems unexpectedly reflect human biases](/research/ai-arseholes/). <> <> This capacity for cultural computing in combination with an increasingly data saturated society makes AI extraordinarly profitable for tech companies and valuable for governments. <> <> Of course, machine learning cannot sustain this level of interest indefinitely but we seem set for the longest and hottest AI summer on record. <> <> <> <> <> <> ## References <> [^Piekniewski2018]: [Filip Piekniewski, "AI Winter Is Well On Its Way", *Piekniewski’s blog*, May 2018](https://blog.piekniewski.info/2018/05/28/ai-winter-is-well-on-its-way/) <> [^MITtechreview]: [Will Knight, "AI Winter Isn’t Coming", *MIT Technology Review*, December 2016](https://www.technologyreview.com/s/603062/ai-winter-isnt-coming/) <> [^Bengio2018]: [Computer science researchers with the highest rate of recent citations (Google Scholar) among those with the largest h-index](http://www.iro.umontreal.ca/~bengioy/citation-rate-CS-1sept2018.html) <>", "slug": "ais-long-hot-summer" } , { "date": "2018-04-24 00:00:00 +0000", "title": "Digital Fatalism; are we locked in?", "url": "/research/digital-fatalism/", "content": "Jaron Lanier’s manifesto cum [red scare](https://en.wikipedia.org/wiki/Red_Scare) paranoia *You Are Not a Gadget* (2010) challenges the structures and politics of “Web 2.0” era digital culture. Not all of the ideas in the book have aged well and some likely never made sense, but his examination of crystallisation of ideas through software representations is still relevant and its lessons are yet to be learned. In this post I want to look at Lanier’s analysis of software representation as philosophy and challenge his suggestion that these metaphors are “locked in”. <> <> *You Are Not a Gadget* is written at an turning point in the history of the web; after the dot com bust and the emergence of web 2.0 but before Facebook executives had worked out how to monetise the platform. These years saw the waning of the personal home page as a defining unit of authorship on the web and the birth of the media prosumer and the social network. For Lanier, these developments represent a wrong turn in the digital revolution, one which threatens to reshape society and our notion of personhood. Particularly, Lanier takes aim at openness and anonymity on web platforms such as Wikipedia which, he suggests, mark a secession of individual identity to groupthink. <> <> Lanier implicitly draws on entrenched archetypes of Eastern and Western culture in his critique. He blames the state of network culture on “digital Maoists” a group consisting of open source contributors, Creative Commons users, web 2.0 platforms, the Linux community, file sharers and remix culture. <> <> Almost a decade on, it’s clear that social media and its Maoist counterparts have not brought about the end of individual identity. Indeed, culture appears more fractured than ever, we are more likely to [criticise Facebook for creating divided echo chambers](https://arstechnica.com/science/2017/03/the-social-media-echo-chamber-is-real/) than pervasive groupthink. In answer to Lanier’s fears about anonymity, Snowden showed that we were never really anonymous in the first place. Against the apparently communist agenda of seemingly free and open user-generated content, tech companies like Facebook and Google have become capitalist behemoths, buying or crushing competition and turning over billions in profit.[^schooloflife] <> <> Of course, predicting the future is difficult and the text has much to offer aside from prescience. It is Lanier’s assertion that software has philosophical and social consequences that I want to focus on here. In chapter one he argues that the paradigms of present digital technology are not inevitable but emerged often arbitrarily as victors amongst many credible alternatives. This might seem like the exaggerations of radical postmodernism and indeed many programmers would deny any philosophical dimension to their work. But, as Lanier identifies, there also exists in computer science a “distinct tradition” that thinks deeply about the consequences of its labour and which might be called “humanistic”. Tim Berners-Lee around the same time the manifesto was written began describing the design of web protocols as “philosophical engineering”. In an informal video interview posted to DailyMotion he said; “It struck me that when we are designing web protocols we’re actually defining the way a new world works and creating a new world… so people, when they use your system, to a certain extent, they have to leave their philosophy at the door… and they have to [agree to] work according to your system. So you can build systems, worlds which have different properties.”[^Berners-Lee2010] For Lanier, the properties of Berner-Lee’s web are radical freedom and openness whose initial conditions have led to the dominant culture. <> <> His argument appears at first glance to follow a common pattern of humanistic research, which might be called *deconstruction*.[^definition] <> Take a ubiquitous and unexamined contemporary idea/institution/system. Recontextualise it within the history of ideas from which it emerged. Show how this context shaped its development and examine contemporaneous alternative ideas that might have created alternative presents. In short, show the idea to be constructed and contingent so it may be reopened for development. <> Only, Lanier takes a fatalistic turn with his concept of *lock-in*. Software, he argues, cannot be fixed. As it grows in size it becomes a “cruel maze”; extremely difficult to modify. As such, large programs tend to crystallise, and most concerningly so do their representations of the world. <> <> In the book’s first introduced example of lock-in, Lanier recounts the history of the MIDI protocol. MIDI, he claims, represents music from the keyboardist’s perspective; as a set of discrete “key-up” “key-down” signals. The very structure of MIDI locks the user into “the tile mosaic world of the keyboardist”, and out of “the watercolor world of the violin”. Here, I agree with Lanier; MIDI like other software representations, is a conceptual as well as technical system, it facilitates thought but also forces users into certain ways of thinking. <> <> According to Lanier, “before MIDI, a musical note was a bottomless idea that transcended absolute definition.” This is where we diverge. Software is far from the only form of representation to shape thought or to encode a singular view of the world. Before MIDI, and alongside it, another dominant representation framed the expression of music. Modern staff notation encodes a rigid notion of pitch and time that enables clear communication between composers, musicians and nowadays machines, but inevitably frames music and creates conceptual boundaries. The stave discretises pitch and assumes western tuning. The very structure of staff notation forces the user into a western musical perspective. It cannot represent the swaying quarter tones of the oud or the continuous paths of the theremin. <> <> ![Piano piece for David Tudor 4 — La Monte Young](http://kieranbrowne.com/images/piano-piece-for-david-tudor-4.jpg) <> <> Modern staff notation’s representation of music is pervasive but not totalising. Beginning in the 1950s, composers such as John Cage have experimented with non-standard or graphic music manuscripts (see [*Notations* (1969)](https://monoskop.org/images/9/92/Cage_John_Notations.pdf) for more examples). These notations often reference aspects of the standard notation but employ textural and/or continuous representations of pitch and time. In the humanistic tradition, these manuscripts are informed by western music notation but escape its rigidity. <> They challenge the strictures of standard notation and offer alternative ways of representing music. <> <> Can we do the same for software representations? or; *is there a substantive difference between software representations and graphical or other representations which makes them more susceptible to lock-in?* <> <> On one hand, it seems there is. Non-digital music need not pass through notation; the unscripted, extempore style of (most) jazz musicians is evidence of this. Digital instruments on the other hand, must funnel their very performance through a software representation and this is almost always MIDI. MIDI has proven extremely resilient to proposed alternatives and the more instruments, musicians and composition programs that support it the less likely it appears to be overthrown. <> <> With this in mind, it is easy to feel fatalistic about the state of software. But the other examples of lock-in in *You Are Not a Gadget* tell a different story. Lanier suggests hyperbolically that the software notion of the file will likely outlive current notions of nature or the concept of the photon. “The file is a set of philosophical ideas made into eternal flesh.” In contrast, we are already seeing the waning of the file representation in computing. Though files remain dominant in desktop computing, smart phones rarely include a file explorer by default and apps are likely to organise data for the user rather than being tools for the user to manipulate files. In another example, Lanier cites the UNIX operating system’s representation of software around discrete typing events “as if reality were a network of fast typists.” This has detrimental consequences for timing making UNIX “too brittle and clumsy” for making music. But although UNIX lives on there are many successful software representations that are used for making music. Live-coding is a broad church which thinks deeply about software representations of time and music. <> <> In the end, what appears to be lock-in is simply the failure that marks the entire book, an inability to see outside the present. Current technologies, institutions and paradigms seem inescapable, but an alternative, even to MIDI, might be just beyond the horizon. <> Lanier's great achievement is to bring to the collective consciousness the power of software representations in shaping thought. We need tech humanists to help make sense of software historically and socially, to develop a critical tradition in software and to reopen software paradigms for development. Digital fatalism we can do without. <> <> ## Notes <> <> [^definition]: David B. Allison, introduction to [*Speech and Phenomena*](https://books.google.com.au/books?id=N4v2AkGMnqcC&printsec=frontcover#v=onepage&q=take%20apart&f=false) defines deconstruction as “a project of critical thought whose task is to locate and 'take apart' those concepts which serve as the axioms or rules for a period of thought, those concepts which command the unfolding of an entire epoch of metaphysics.” (p. xxxiii) <> <> <> [^bookavore]: https://web.archive.org/web/20100620233905/http://bookavore.com:80/2009/12/03/review-you-are-not-a-gadget/ <> [^schooloflife]: There’s [a moment](https://youtu.be/lbdYg_z_SAE?t=48m23s) recorded in a live conversation between Lanier and [James Bridle](http://jamesbridle.com/) at the School of Life in London, where Bridle pushes back on Lanier’s proposed solutions to technology; “Aren’t those approaches exactly what will be impossible in a more pervasive technological system like the one that you propose as the next step for freedom in this? If everything that we do, if every action, if every contribution is tracked, monitored, surveilled and and put in a capitalist framework of reward, doesn’t that approach of cutting yourself off become completely impossible?” <> [^Stiegler2012]: [Stiegler, Bernard, M. Hildebrandt, K. O’Hara, and M. Waidner. "Die Aufklärung in the age of philosophical engineering." Digital enlightenment yearbook (2013): 29-39.](http://computationalculture.net/die-aufklarung-in-the-age-of-philosophical-engineering/) <> [^Berners-Lee2010]: “Tim Berners-Lee on PhiloWeb and Philosophical engineering”. *DailyMotion.* [http://www.dailymotion.com/video/xfisjf](http://www.dailymotion.com/video/xfisjf) <>", "slug": "digital-fatalism" } , { "date": "2017-05-22 00:00:00 +0000", "title": "Filling gaps in TensorFlow’s Java api", "url": "/research/filling-gaps-in-tensorflow/", "content": "TensorFlow‘s announcement of a Java api is great news for the clojure community. I wrote a post a couple of weeks ago that argued that TF‘s Java api [already provides everything that we need to do useful things](/research/clojure-tensorflow-interop/). This is mostly true, by leveraging interop we can easily get tensors flowing, use any of TensorFlow‘s many operations and do useful calculations with them. That said, if your plan is to use TensorFlow for machine learning, and I‘m guessing this is most people, you‘ll probably regret the absense of the great optimizers that Python‘s TensorFlow api provides. Sure, you can build your own backpropagation out of TensorFlow operations but this becomes very tedious if your network is more than a few layers deep. So this week I had a go at implementing a functional gradient descent optimizer in Clojure/TensorFlow and I thought I‘d share what I‘ve learned.<> <> ### TL;DR<> The result is a function, `gradient-descent`, which given a TensorFlow operation representing the error of a network, returns an operation capable of minimizing the error.<> ```clojure<> ;; with a network described ...<> ;; using squared difference of training output <> ;; and the results of our network.<> (def error (tf/pow (tf/sub targets network) (tf/constant 2.)))<> <> (session-run<> [(tf/global-variables-initializer)<> (repeat 1000 (gradient-descent error))])<> ```<> <> In this post I‘m going to discuss:<> - [how machine learning actually learns](#how-does-machine-learning-learn),<> - [how we calculate this for a computational graph like TensorFlow‘s](#computing-gradients-on-computational-graphs), and<> - [how we can implement this in Clojure without waiting for the Java api](#implementing-gradient-computation-in-clojure)<> <> ## How does machine learning “learn”?<> The first thing to say about machine learning is that when we say “learn” we really mean “optimize”. Each learning algorithm uses an error function to measure how well or poorly an algorithm has performed. Learning is equivalent to changing internal features of a model, usually weights and/or biases until the error is as small as it can be.<> <> For supervised learning, which makes up a huge number of ML applications, our error function measures how different the network‘s outputs are to known training outputs. If the difference is minimal, we say that the network has learned the pattern in the data.<> <> But how do we know what to do to our model in order to reduce the error? A brute force solution would be to try every possible setting and select the one which produces the lowest error. Unfortunately, this method is extremely inefficient and not feasible for any real world application. Instead, we can use calculus to find the direction and rate of change of the error with respect to each weight or bias in the system and move in the direction which minimizes the error.<> <> <> ## Computing gradients on computational graphs<> <> Consider the following [clojure/tensorflow](https://github.com/kieranbrowne/clojure-tensorflow) code.<> ```clojure<> ;; training data<> (def input (tf/constant [[1. 0.] [0. 0.] [1. 1.] [0. 1.]])<> (def target (tf/constant [[1.] [0.] [1.] [0.]])<> ;; random weights<> (def weights (tf/variable <> (repeatedly 2 #(vector (dec (* 2 (rand)))))))<> ;; model<> (def weightedsum (tf/matmul input weights))<> (def output (tf/sigmoid weightedsum))<> ;; error<> (def diff (tf/sub target output))<> (def error (tf/pow diff (tf/constant 2.)))<> ```<> Under the hood, we‘re interoping with TensorFlow via Java to build a graph representation of the computation.<> <> A visualisation of the graph might look something like this.<> <> ![Clojure TensorFlow Graph](/images/clojure-tensorflow-graph.png)<> <> Each node is a TensorFlow operation and each arrow (or edge) represents the use of a previous operation as an argument.<> <> <> As above, to train the network we need to find the rate of change (gradient) of the error with respect the weights. <> <> The process is actually fairly simple in the abstract.<> <> ### Step 1: Find the path between the nodes<> <> We need to find the path of TensorFlow operations between `error` and `weights`. <> <> ![Clojure TensorFlow Graph](/images/clojure-tensorflow-graph-derivative-path.png)<> <> In this example, the path of operations is:<> <> `tf/pow` → `tf/sub` → `tf/sigmoid` → `tf/matmul` → `tf/variable`<> <> ### Step 2: Derive each function and apply the chain rule<> <> Invoking the chain rule, we know that the derivative of error with respect to weights is equal to the derivative of each operation in the path multiplied together. <> <> It‘s pretty much as simple as that.<> <> ## Implementing gradient computation in Clojure<> <> The algorithm described above gets a little hairier when we come to actually implement it in Clojure.<> <> ### Finding the path between nodes<> <> Our first task is to find the paths between two nodes in the graph. To do this, we need a way of knowing whether a node or any nodes it takes as input lead to the node we are looking for. The problem is, there is currently no method for getting the inputs of a TensorFlow operation in the Java API. To fix this, we can add a “shadow graph” to our code. This will be a secondary graph of our TensorFlow operations each recorded as a clojure map.<> <> ```clojure<> (def shadow-graph (atom []))<> ```<> <> So, every time we add an operation to the TensorFlow graph. We will also conj its profile to the shadow graph.<> <> We won‘t ever use the shadow graph to actually run computations, but will simply reference it when we need information about an operation we can‘t yet get from TensorFlow itself.<> <> Now, in order to get the inputs of a TensorFlow operation we look it up in the shadow graph and retrieve the `:inputs` key.<> <> ```clojure<> (defn get-op-by-name [n]<> (first (filter #(= (:name %) n) @build/shadow-graph)))<> <> (def get-inputs (comp :inputs get-op-by-name #(.name (.op %))))<> ```<> <> ### Following the path of inputs<> <> We can define dependence recursively like so. One operation depends on another if it is equal to that operation or any of it‘s inputs depend on that node.<> <> ```clojure<> (defn depends-on?<> \"Does b depend on a\"<> [a b]<> (or (some (partial depends-on? a) (get-inputs b))<> (= b a)))<> ```<> <> To get paths from one operation to another we can use `depends-on?` to resursively test if either of its inputs depend on the target operation and if so, conj it to the path and repeat the process on the dependent inputs.<> <> The `paths` function returns a list of all possible paths between two operations in a graph.<> <> ```clojure<> (defn paths<> \"Get all paths from one op to another\"<> [from to]<> (let [paths (atom [])]<> (collate-paths from to paths [])<> @paths))<> ```<> <> I ended up using a function scoped atom to collect all the possible paths because of the structural sharing that can happen if an operation splits off in the middle of the graph. It works, but I‘m not totally happy with the statefullness this solution so if anyone out there has a better idea, hit me up.<> <> All of the hard work above is handled by the `collate-paths` function which recursively follows each input of the operation which also depends on the `to` operation.<> <> The other important thing that collate paths does, is that it creates a map of the necessary information to differentiate the operation. The `:output` key stores the node on the path, the `:which` key stores which of the inputs depend on the `to` variable. This is important because the derivative of `x^y` with respect to x is not the same as its derivative with respect to y. Don‘t worry about the `:chain-fn` key for now, we‘ll get to that shortly.<> <> ```clojure<> (defn collate-paths [from to path-atom path]<> (let [dependents (filter (partial depends-on? to) (get-inputs from))<> which-dependents (map #(.indexOf (get-inputs from) %) dependents)]<> (if (= from to)<> (swap! path-atom conj<> (conj path {:output (ops/constant 1.0)<> :which first<> :chain-fn ops/mult}))<> (doall<> (map<> #(collate-paths<> %1 to path-atom<> (conj path<> {:output from<> :which (fn [x] (nth x %2))<> :chain-fn<> (case (.type (.op from))<> \"MatMul\" (if (= 0 %2)<> (comp ops/transpose ops/dot-b)<> ops/dot-a)<> ops/mult)}))<> dependents which-dependents)))))<> ```<> <> There‘s a lot going on in this function, so don‘t worry if it doesn‘t completely make sense, the main idea is that, it collates all the information we will need to differentiate the operations.<> <> ### Find the derivative of each node in the path<> <> Actually differentiating the operations is fairly simple. First, we have a map `registered-gradients` which contains a derivative function for each input to operation.<> <> The `get-registered-gradient` function looks up `registered-gradients` using the information we collected with `collate-paths` and returns a tensorflow operation representing its derivative.<> ```clojure<> (defn get-registered-gradient<> [node]<> (let [{output :output which :which} node]<> (apply (which (get @registered-gradients (.type (.op output)))) (get-inputs output))))<> ```<> <> ### Applying the chain rule<> <> This is also slightly more difficult in clojure. Remember I said we‘d get back to the `:chain-fn` key? Well we need that because we cant just use the standard `tf/mult` operation to chain all our derivatives. Of course mathematically speaking, we are multiplying all the derivatives together, but when it comes to tensors, not all multiplication operations are created equal.<> <> The problematic operation in our example is `tf/matmul` because it changes the shape of the tensor output and which elements are multiplied with which based on the order of arguments.<> <> For a graph which contains no `tf/matmul` operations we could get away with chaining our derivatives like so:<> ```clojure<> (reduce tf/mult <> (map get-registered-gradient <> (paths y x)))<> ```<> <> Which is a shame, because it‘s so darn elegant.<> <> Instead, we have the slightly more complicated:<> ```clojure<> (defn gradient [y x]<> (reduce<> ops/add<> (map<> (partial reduce<> (fn [gradient node]<> ((:chain-fn node)<> (get-registered-gradient node) gradient))<> (ops/constant 1.))<> (paths y x))))<> ```<> But the principle is the same.<> <> To add a small extention on the `gradient` function, `gradients` maps `gradient` over a list of weights, biases etc.<> <> ```clojure<> (defn gradients<> \"The symbolic gradient of y with respect to xs.\"<> ([y & xs] (map (partial gradient y) xs))<> ([y] (apply (partial gradients y) (relevant-variables y))))<> ```<> The other useful addition of `gradients` is that it can be applied without any `xs`. In this case it returns the gradients of any variable nodes which `y` depends on. This is powerful because we can optimize an error function without specifying all the weights and biases in the system.<> <> The last function we need is `apply-gradients` which as you might‘ve guessed takes a list of variable nodes and a list of gradients and assigns the variables to their new value. <> <> ```clojure<> (defn apply-gradients<> [xs gradients]<> (map #(ops/assign %1 (ops/sub %1 %2))<> xs gradients))<> ```<> <> And that‘s all we need to compute gradients on the TensorFlow graph in Clojure.<> <> Computed gradients are the basis of all the great optimizers we use in supervised learning, but the simplest of these is gradient descent which simply subracts the gradient from the variable.<> <> ```clojure<> (defn gradient-descent<> \"The very simplest optimizer.\"<> ([cost-fn & weights]<> (apply-gradients weights (apply gradients (cons cost-fn weights))))<> ([cost-fn] (apply (partial gradient-descent cost-fn)<> (relevant-variables cost-fn))))<> ```<> <> And as easy as that, we have a generalised optimizer for tensorflow. Which can be plugged into a network of any number of layers.<> ```clojure<> (session-run<> [(tf/global-variables-initializer)<> (repeat 1000 (gradient-descent error))])<> ```<> <> ## Final Thoughts<> <> Optimizers and gradient computation are extremely useful for neural networks because they eliminate most of the difficult math required to get networks learning. In doing so, the programmer‘s role is reduced to choosing the layers for the network and feeding it data.<> <> So why should you bother learning this at all?<> Personally I believe it‘s important to understand how machine learning actually works, especially for programmers. We need to break down some of the techno-mysticism that‘s emerging around neural networks. Its not black magic or godlike hyper-intelligence, it‘s calculus.<> <> <> ***<> <> I‘ve included the code from this post as part of [a library](https://github.com/kieranbrowne/clojure-tensorflow) with some helper functions to make interoping with TensorFlow through Java a bit nicer. This project is decidedly not an API, but just as little code as we can get away with to make TensorFlow okay to work with while Java gets sorted.<> <> There is however some cool people working on a proper Clojure API for TensorFlow [here](https://github.com/enragedginger/tensorflow-clj). <> <> To really get stuck into ML with Clojure you‘ve gotta use [Cortex](https://github.com/thinktopic/cortex). It provides way better optimizers, composable layer functions and a properly clojurian API.<> <> <> <> ## Reading List<> - [Calculus on Computational Graphs: Backpropagation](http://colah.github.io/posts/2015-08-Backprop/)<>", "slug": "filling-gaps-in-tensorflow" } , { "date": "2017-05-02 00:00:00 +0000", "title": "Running TensorFlow in Clojure", "url": "/research/clojure-tensorflow-interop/", "content": "TensorFlow is a library for running matrix computation very fast. This makes it perfect for data science and in particular, machine learning. <> <> The power of TensorFlow comes from a bunch of optimised C code under the hood. TensorFlow can also be compiled to run on the GPU, which can allow for enormous speedups. This is important because the machine learning is becoming more reliant on big data and big networks which take a lot of computing power. <> <> At the beginning of the year TensorFlow began work on a Java api, and that means Clojure gets one for free. <> <> For now, the Java api is still in active development and is very sparse. But don't let that stop you getting your hands dirty. The Java api already provides everything we need to work with TensorFlow. With just java interop and a couple of helper functions we can start writing idiomatic clojure code. <> <> This post will cover: <> - [How to install TensorFlow with lein](#how-to-install-tensorflow-with-lein) <> - [How to think in TensorFlow](#thinking-in-tensorflow) <> - [How to directly interop with the Java api](#direct-interop-with-the-java-api) <> - [How we can still write nice code without a Clojure api](#writing-idiomatic-clojure-for-tensorflow) <> <> ## How to install TensorFlow with lein <> <> To use TensorFlow with lein just add `[org.tensorflow/tensorflow \"1.1.0-rc1\"]` to your dependencies in `project.clj`. <> <> <> We can test our installation by firing up a repl and running the version method on the TensorFlow class. <> <> ```clojure <> (. org.tensorflow.TensorFlow version) <> ;; => \"1.x.x-rc2\" <> ``` <> ### Troubleshooting <> <> If you get any errors first make sure your are running Java 8. <> <> ```clojure <> (System/getProperty \"java.version\") <> ;; => \"1.8.0_101\" <> ``` <> <> Your can force lein to use Java 8 by adding `:java-cmd \"/path/to/java\"` to your `project.clj`. <> <> If you're still getting errors after that follow the instructions [here](https://github.com/kieranbrowne/clojure-tensorflow-interop#method-2-build-from-source) to build from source. <> <> ## Thinking in TensorFlow <> Before we get started with the actual code, there are a few concepts I need to explain otherwise none of this is going to make sense. <> <> The main object of computation in TensorFlow is the *tensor*. A Tensor is just a typed multi-dimensional array. Nothing scary here. <> <> When we write code for TensorFlow, we're not actually running computations. Instead we're composing a data structure which describes the flow of data. In TensorFlow this is called a *graph*. The graph will describe the flow of our data through a series of operations (*ops* for short). Nothing will actually be computed until we launch our graph in a *session*. The session handles the execution of our graph on the CPU or GPU and returns the resulting tensors. <> <> In short, our clojure code will assemble a graph and fire off commands to the C code using a Session object. <> <> ## Direct interop with the Java api <> <> To get started I'm going to use only direct interop with the Java api. It ain't pretty, but it should give you a sense for what is actually happening under the hood, and hopefully make it a bit easier to understand the abstractions we user later on. <> <> First we need to initialise a new Graph object. <> ```clojure <> (def graph (new Graph)) <> ``` <> <> Next we're going to need some example tensors to work with. Because the computation isn't running in clojure we can't just define our values. Instead we're defining an operation node in the graph that generates a constant. <> <> First I'm creating a tensor object using the class' `.create` method. Because we're interopping with the Java class we first need to turn our clojure persistant vector into an array of 32bit Integers. Using the arrow macro for clarity, we call the `.opBuilder` method on our graph. The first argument is the binary operation which will be added to the graph. In this case its "Const" (which is short for constant (obviously)). <> <> This is one of a big set of possible binary ops which TensorFlow have implemented in native code and which we reference when building our graph. <> <> The second argument is a unique name for the operation. I went with "tensor-1" for simplicity, but "Joaquin Phoenix" would have also worked. The only requirement is that it is unique to the graph. Next we set the value and datatype attributes that are required for the Const operation. Finally we build our operation based on the attributes and use the output method to return it. It is this returned operation that gets saved in clojure. <> <> ```clojure <> (def tensor-1 <> (let [tensor <> (Tensor/create <> (int-array <> [360 909 216 108 777 132 256 174 999 228 324 800 264] <> ))] <> (-> graph <> (.opBuilder \"Const\" \"tensor-1\") <> (.setAttr \"dtype\" (.dataType tensor)) <> (.setAttr \"value\" tensor) <> .build <> (.output 0)))) <> <> (def tensor-2 <> (let [tensor <> (Tensor/create <> (int-array [5 9 2 1 7 3 8 2 9 2 3 8 8]))] <> (-> graph <> (.opBuilder \"Const\" \"tensor-2\") <> (.setAttr \"dtype\" (.dataType tensor)) <> (.setAttr \"value\" tensor) <> .build <> (.output 0)))) <> ``` <> <> Now lets add a more exciting operation to our graph. Again we will call the `.opBuilder` method on our graph object. I'm going to use the "Div" (division) operation this time. Next we call the `.addInput` method to add our two example tensors as input to the operation. <> <> Again we build and output our operation object, saving it as "divide" in clojure land. <> <> ```clojure <> (def divide <> (-> <> (.opBuilder graph \"Div\" \"my-dividing-operation\") <> (.addInput tensor-1) <> (.addInput tensor-2) <> .build <> (.output 0) <> )) <> ``` <> <> <> To run our newly built operations, we need to create a session object based on our graph. <> <> ```clojure <> (def session (new Session graph)) <> ``` <> <> <> We'll call the `.runner` method on our session to get the engine running. We use the `.fetch` method to retrieve the divide operation by name; in this case we want to pass it the name we gave to the divide operation just before (\"my-dividing-operation\"). The `.get` method gets our result from the returned array, this gives us a Tensor object which has all the data but cannot be read easily, so finally to read our results, we call the `.copyTo` method on the Tensor to copy the contents to an integer array. <> <> ```clojure <> (def result <> (-> session <> .runner <> (.fetch \"my-dividing-operation\") <> .run <> (.get 0) <> (.copyTo (int-array 13)) <> )) <> ``` <> <> Finally we can read our results. <> ```clojure <> (apply str (map char result)) <> ;; => \"Hello, World!\" <> ``` <> <> ## Writing idiomatic Clojure for TensorFlow <> <> So we have successfully run a basic TensorFlow graph, which is cool, but the code made my eyes bleed. This is partially because the TensorFlow Java api is so new and doesn't have the multitudes of helper functions that python has yet. But I think the main reason is that writing methods isn't why I came to clojure. <> <> TensorFlow's Java api is still extremely barebones, but it already provides everything that we need to do useful things. <> <> Better yet, when we're writing for TensorFlow we're really only building operations and running them; thus we really only need a couple of helper function to cover all bases. We can get by without an api altogether. <> <> <> Lets actually do some machine learning. For simplicity's sake, I'm going to write a very shallow neural network. From here on, I'm going to start using a very light layer on top of the interop that I defined in [helpers.clj](https://github.com/kieranbrowne/clojure-tensorflow-interop/blob/master/src/clojure_tensorflow_interop/helpers.clj). <> <> First, we'll need some training data. <> ```clojure <> (def training-data <> ;; input => output <> [ [0. 0. 1.] [0.] <> [0. 1. 1.] [1.] <> [1. 1. 1.] [1.] <> [1. 0. 1.] [0.] ]) <> ``` <> <> We can split out training data into inputs and outputs like so. Note the use of `tf/constant`. This simply wraps the operationBuilder and takes care of adding the Const operation to the default graph. <> <> ```clojure <> (def inputs (tf/constant (take-nth 2 training-data))) <> (def outputs (tf/constant (take-nth 2 (rest training-data)))) <> ``` <> <> We want to initialise our weights as a random value between -1 and 1. Because training our network means changing the state of our weights, we use `tf/variable` which creates a variable node on the graph. <> <> ```clojure <> (def weights <> (tf/variable <> (repeatedly 3 (fn [] (repeatedly 1 #(dec (rand 2))))))) <> ``` <> <> Even though we're defining nodes for the TensorFlow graph, we can still define our flow with functions. This is great because it feels just like we're writing clojure code. <> <> ```clojure <> (defn network [x] <> (tf/sigmoid (tf/matmul x weights))) <> ``` <> <> For our network to learn we need to measure the difference between the training outputs and our network's outputs. Most of the complexity here comes from the neural network itself so if the next few functions don't make sense don't worry. For an explanation of basic neural network code checkout [this post](/research/neural-networks-carbon-and-code). <> ```clojure <> (defn error [network-output] <> (tf/div <> (tf/pow <> (tf/sub outputs network-output) <> (tf/constant 2.)) <> (tf/constant 2.))) <> ``` <> <> For back-propagation, we need use derivative of our error and sigmoid functions. Note here, the use of `tf/assign` to set the variable weights to their new value. Also notice how we can abstract our TensorFlow operations just like clojure code, so that all the complexity of derivatives and deltas is wrapped up in the `train-network` operation. <> ```clojure <> (defn error' [network-output] <> (tf/sub network-output outputs)) <> <> (defn sigmoid' [x] <> (tf/mult x (tf/sub (tf/constant 1.) x))) <> <> (defn deltas [network-output] <> (tf/matmul <> (tf/transpose inputs) <> (tf/mult <> (error' (network inputs)) <> (sigmoid' (network inputs))))) <> <> (def train-network <> (tf/assign weights (tf/sub weights (deltas (network inputs))))) <> ``` <> <> So far we seem to have used a whole bunch of functions to build our operations. But really we've only been using one. The function `op-builder` which is defined in [helpers.clj](https://github.com/kieranbrowne/clojure-tensorflow-interop/blob/master/src/clojure_tensorflow_interop/helpers.clj) simply wraps up all the messy object-oriented code from the Java api to add operations to the graph. All the other tf scoped functions we have used, just pass arguments to `op-builder`. This is why we can safely wrap so much functionality without worrying that the Java api will change on us. <> <> ### Running our Operations <> <> The other thing that our `helpers.clj` file defines is a couple of functions to make running operations a bit easier. <> <> The `tf/session-run` helper function takes care of setting up a session and running a list of operations. `tf/session-run` returns the results of the last operation in the list. In this case the it will return the results of the network without training. <> ```clojure <> (tf/session-run <> [(tf/global-variables-initializer) <> (network inputs)]) <> ``` <> <> Note also the use of `tf/global-variables-initializer`. This is needed when we are using one or more variables in our graph. There are other ways of approaching the variable initialisation problem for TensorFlow graphs, but for now I've just gone with the standard solution from the python TensorFlow api. Despite the \"global\" in the function name the variable initialisation is scoped to the `tf/session-run` function and won't affect other sessions. You can think of it like a `let` function. <> <> The pattern above is great for testing small parts of your graph or a couple of operations here and there. But when we train our network we want its trained weights to be preserved so we can actually use the trained network to get shit done. <> <> For this we want to create a session object that we can hold on to. <> ```clojure <> (def sess (tf/session)) <> ``` <> We can make a partial of the `session-run` function to get the best of both worlds. <> ```clojure <> (def sess-run (partial tf/session-run tf/default-graph sess)) <> ``` <> <> Now we can break up our operations steps into logical breaks initialise variables and run the untrained network <> ```clojure <> (sess-run [(tf/initialise-global-variables)]) <> ``` <> <> Run the train-network operation 10000 times and then check the error. <> ```clojure <> (sess-run <> [(repeat 10000 train-network) <> (tf/mean (error (network inputs)))]) <> ``` <> <> Run the network on a new example <> ```clojure <> (sess-run [(network (tf/constant [[1. 1. 1.]]))]) <> ;; => [[0.99740285]] <> ``` <> <> And that's about it. We've converted our eyesore object-oriented interop code to something perfectly readable with just a couple of functions. The code base is tiny enough to allow immediate changes if the Java api changes on us and the system is flexible enough that we don't need to wait for the Java api to get fleshed out to jump in and get our hands dirty. <> <> ## Final Thoughts <> <> Machine learning is shaping up to be the most important class of algorithms of this decade. It's important that Clojure has a good story for ML if we want it to be around for a long time. [Cortex](https://github.com/thinktopic/cortex) is definitely making important progress on this front and as it approaches its 1.0.0 release I fully expect it to become Clojure's go to library for this sort of thing. But TensorFlow is the biggest name in ML, and with Google's backing that isn't going to change anytime soon. The fact that we can so easily extend Java's api and write clearer code with than in Java itself (don't believe me? [read the example code](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/java/src/main/java/org/tensorflow/examples/LabelImage.java)) is a pretty incredible display of what makes Clojure a great language to write. <> <> <> The full code for this post is available [here](https://github.com/kieranbrowne/clojure-tensorflow-interop). <> <> I've made a library to contain the helper functions used in this post. This is not intended to become a TensorFlow api for clojure and will remain as a very light layer over the interop. If you would like to use or contribute, you can find it [here](https://github.com/kieranbrowne/clojure-tensorflow). <> <>", "slug": "clojure-tensorflow-interop" } , { "date": "2017-03-02 00:00:00 +0000", "title": "Neural Networks in Carbon and Code", "url": "/research/neural-networks-carbon-and-code/", "content": "<>![Artificial neuron diagram](https://kieranbrowne.com/images/neurons.jpg) <> <>The artificial neural network is a powerful programming paradigm that has achieved some impressive feats in recent years. The algorithms in this style are based on an analogy to the information pathways in biological nervous systems like the brain. But how alike are artificial and biological neural networks? And what elements must be alike for the metaphor to hold? This is a big topic and I've only just scratched the surface, but if you're willing to brave my possible misinterpretations I'll share what I've learned so far. <> <>## Neurons in Carbon <> <>A neuron is a cell capable of transmitting information through electrical and chemical signals. These tiny cells are the fundamental building blocks of the brain and the entire nervous system. <>There are some specialised input neurons in sensory organs which convert information from the world (touch, sound, light) into electrical impulses and specialised output neurons that convert electrical impulses from the nervous system into muscle movement. The vast majority of neurons in your body however are simply connected to other neurons. <> <>In a neuron, informations flows in a single direction. Left to right in the image below. The main body of the cell collects information from other neurons, via synapses. A synapse is a microscopic gap between the output end of one neuron and the input end of another, sometimes itself. This gap is jumped when enough charge is built up on the input end. Synapses can be of varying strengths, possible due to repeated stimulation. <> <>Neurons have long tree-like branches called dendrites stretching outward from the cell body. Most of the cell's synapses are connected along these branches. The neuron's charge is the sum of all of the charges from the neurons upstream. A single neuron can have millions of synaptic connections with others. <>When the sum of inputs to a neuron reaches the threshold, a charge is released down its tail, called an axon. Finally there is another series of branches each ending in a small bulb called an axon terminal. These terminals connect to the body or dendrites of other neurons to form more synapses and so the flow of information continues. <> <>![Drawing of a neuron](/images/neuron.jpg) <> <> <>## Neurons in code <> <> <>In the computational model of a neuron there is no need for the complex physical structures. We simply model the flow of information. <>As such, a neuron is just a value resulting from its relations to other neurons. The diagram below shows the calculation of a single neuron value from it's upstream inputs. There are three inputs to this neuron connected via synapses of varying strengths. <>We can think of the synapse strengths as coefficients which moderate the relevance of any upstream neurons. <> <>![Artificial neuron diagram](/images/artificial-neuron-diagram.jpg) <> <>Computational neurons are the sum of their connected inputs multiplied by their synaptic strength. As such, the example below is calculated as: <> <>### 1 * 0.75 +
0 * 0.01
1 * -0.8
= -0.05 <>This new neuron can then feed out via further synapses to other neurons. <> <>There are some obvious functional differences here between biology and computation. The biological neurons have a threshold value which when reached results in immediate firing. This means that a biological neuron is either on or off, while the computational neuron can take on analog values. It is possible to include a threshold and binary values in the computational model (the classic perceptron is an example of this) but these tend to be impossible to train as small changes to weights can cause results to flip. It is also worth noting that although a biological neuron is either on or off, the effect of analog signals are produced by varying frequencies of firing. <> <> <>## Emergence and connectionism <> <>A neuron in and of itself is not enough to constitute thinking. How then can we make sense of the incredible feats of cognition of which the brain or even from artifical neural networks are capable? One proposed solution is connectionism. <>According to connectionism, mental phenomena emerge from the connections between many simple units. This model posits that learning involves the adjustment of connection strenghts and attibutes learning, understanding and memory this process. <> <>In biological neural networks the adjustment of connection is commonly attributed to repeated patterns of firing, the theory goes that synapses strengthen when used and atrophy when left dormant. In artificial neural nets we adjust synapse strengths with an algorithm called backpropagation which I'll demostrate in code examples later on. <> <>The diagram below shows the structure of a simple network of neurons which we're about to code. <> <>![Artificial neuron diagram](/images/artificial-neural-net-diagram.jpg) <> <> <>## Coding a neural network <> <>The connectionist model, regardless of its ultimate "truth", has had some incredible successes as a way of programming. It is worth remembering however, that other models of cognition such as computationalism have also had successes in producing apparantly intelligent behaviour. With that said, there is something truly mindbending about neural networks, [they seem to defy reason](http://karpathy.github.io/2015/05/21/rnn-effectiveness/). <> <>All the examples that follow will be in clojure. If that's not your cup of tea, I recommend [A Neural Network in 11 lines of Python (Part 1)](http://iamtrask.github.io/2015/07/12/basic-python-network/) by Trask. It deals with the same problem in a more orthodox pythonic style. <> <>Lets start by establishing some training data. <> <>```clojure <>(def training-data <> ;; input => output <> [ [0 0 1] [0] <> [0 1 1] [1] <> [1 0 1] [1] <> [1 1 1] [0] ]) <>``` <> We can easily extract the input and output like so.<> <> <>```clojure <>(def training-input <> (take-nth 2 training-data)) <> <>(def training-output <> (take-nth 2 (rest training-data))) <>``` <> <>We want to initialise our synapses as random values between -1 and 1. <> <>```clojure <>(def random-synapse #(dec (rand 2))) <>``` <> <>We'll need a synapse weight from every neuron in the input layer to every one in the output layer. We'll represent this as a matrix of size [number-of-inputs, number-of-outputs]. <>It's usually better to create reusable functions so my solution is matrix-of which works a bit like clojure.core's repeatedly. <> <>```clojure <>(defn matrix-of [function shape] <> (repeatedly (first shape) <> (if (empty? (rest shape)) <> function <> #(matrix-of function (rest shape)))) <>``` <> <>Now our code describes it's function to the reader. <> <>```clojure <>(matrix-of random-synapse [3 5]) <>``` <> <>The synapses are mutable so we'll want to use atoms. <> <>```clojure <>(def synapses-0 <> (atom (matrix-of random-synapse [3 5]))) <> <>(def synapses-1 <> (atom (matrix-of random-synapse [5 1]))) <>``` <> <>## Activation/Deactivation <> <>For our network to function over multiple layers of neurons we need a way of keeping values within 0 and 1. To solve this we'll need a non-linear activation function. <>This is probably best demonstrated by example. <> <>```clojure <>REPL> (activate 0) <>;; => 0.5 <>REPL> (activate 1) <>;; => 0.7310585786300049 <>REPL> (activate -5) <>;; => 0.0066928509242848554 <>REPL> (activate 10) <>;; => 0.9999546021312976 <>``` <> <>Here I’m going to use the *standard logistic function* but *tanh* and other sigmoidal curves will also work. <> <>```clojure <>(defn activate <> ;; Standard logisitic function <> [x] (/ 1 (+ 1 (matrix/exp (- x))))) <> <>(defn deactivate <> ;; Reverse activation operation <> [x] (* x (- 1 x))) <>``` <> <>## Feed forward <> <>We can calculate the neurons of the next layer very simply. We saw before that the value of a neuron is equal to the sum of it's inputs multiplied by their synaptic weights. We could iterate this way to get our result, but this can be achieved even more elegantly with the matrix dot product. <> <>```clojure <>(defn step-forward <> ;; For each step in feed forward, the new layer of neurons <> ;; are a function of the previous layer and it's synapses <> [neurons synapses] <> (activate (matrix/dot neurons synapses))) <>``` <> <>The above works fine, but I think a single feed forward function which carries out an arbitrary level of layers would be nicer. <> <> <>```clojure <>(defn feed-forward <> [input & synapses] <> (case (count synapses) <> 0 input ; if no synapses return just the input <> 1 (step-forward input @(first synapses)) <> ;; otherwise recur <> (apply <> feed-forward <> (into [(step-forward input @(first synapses))] <> (rest synapses))))) <>``` <> <>Now all the required information can be applied as arguments to the feed-forward function. <> <>```clojure <>(feed-forward training-input @synapses-0 @synapses-1) <>``` <> <>And we get our result! The only problem is that our result is about as good as flipping a coin. <> <>## Calculating the Error <> <>Our network is functional, but how did it do? <>We can measure the total error pretty easily by finding the difference between our training outputs and the return value of our feed forward network. <> <>```clojure <>(defn errors [training-outputs real-outputs] <> (- training-outputs real-outputs)) <> <>(defn mean-error [numbers] <> (let [absolutes (map abs (flatten numbers))] <> (/ (apply + absolutes) (count absolutes)))) <> <>REPL> (mean-error <> (error <> training-output <> (feed-forward training-input @synapses-0 @synapses-1))) <>;; => 0.496748102 <>``` <> <>With our current untrained network the mean error should be somewhere around 0.5. <> <>## Training the Network <> <>How can we turn our functional, but utterly abysmal network into a powerful predictor? Perhaps surprisingly, all we change from here on out is the synapse strengths, in other words the intelligence of the system emerges entirely from the connections between neurons, not the neurons themselves. This process is a bit like tuning a piano. <> <>The process of tuning involves calculating the effects of each synapse on the system's total error and adjusting to reduce the error. <> <>```clojure <>(defn output-deltas [targets outputs] <> (* (deactivate outputs) <> (- targets outputs))) <> <>(defn hidden-deltas [output-deltas neurons synapses] <> (* (matrix/dot output-deltas (matrix/transpose synapses)) <> (deactivate neurons))) <>``` <> <>These changes we calculate can then be applied to the synapses in the system. <> <>```clojure <>(defn apply-deltas [synapses neurons deltas learning-rate] <> (+ synapses <> (* learning-rate <> (matrix/dot (matrix/transpose neurons) deltas)))) <>``` <> <>Then, all we need to do is reset the synapses values with the deltas applied. <> <>```clojure <>(reset! synapses-0 <> (apply-deltas @synapses-0 training-input hidden-delta 1)) <> <>(reset! synapses-1 <> (apply-deltas @synapses-1 hidden-layer output-delta 1)) <>``` <> <>We can wrap this up in a single training function. <> <>```clojure <>(defn train <> ;; Train the network and return the error <> [training-input training-output] <> (let [ <> hidden-layer (step-forward training-input @synapses-0) <> output-layer (step-forward hidden-layer @synapses-1) <> output-delta (output-deltas training-output output-layer) <> hidden-delta (hidden-deltas output-delta hidden-layer @synapses-1) <> output-error (errors training-output output-layer) <> ] <> (do <> (swap! synapses-1 <> #(apply-deltas % hidden-layer output-delta 1)) <> (swap! synapses-0 <> #(apply-deltas % training-input hidden-delta 1)) <> (mean-error output-error) <> ))) <> <>;; We need to do this a lot of times to get good results. <>(dotimes [i 10000] <> (train training-input training-output)) <> <>;; Check results <>(feed-forward training-input synapses-0 synapses-1) <>``` <> <>Artificial neural networks are fascinating. Their apparant intelligence seems to surprise even those who understand their mathematical foundation. Whether or not these algorithms really function "like a brain" is still an open question. Certainly there are functional differences between biological neurons and their computational simulacra, though the ontological consequences of these are difficult to comprehend as neither system's emergent qualities can be easily explained by their simple parts. No doubt I still have a lot to learn in this area. <> <>The code for from this post can be found [here](https://github.com/kierantbrowne/basic-nn). <> <>## Reading List <>- Clarke, Arthur Charles. *The Nine Billion Names of God*. New American Library, 1974. ", "slug": "neural-networks-carbon-and-code" } , { "date": "2017-02-23 00:00:00 +0000", "title": "AI Arseholes", "url": "/research/ai-arseholes/", "content": "On March 23, 2016 Microsoft unleashed an experimental artificial intelligence named *TayTweets* onto twitter. Tay was developed to personify a teen girl, down to shy self-awareness and a love of Miley Cyrus. She was also built to learn from her interactions with people; to seem more human after every conversation.<> <> ![@mayank_lee can I just say that im stoked to meet u? humans are super cool](https://kieranbrowne.com/images/tay-tweets-1.jpg)<> <> Within hours, Tay had come to the attention of the internet’s de facto countercultural message board, 4Chan. The site’s community mobilised to test the limits of Tay’s learning. Users tweeted a variety of conspiracy theories, profanities and extremist views at the bot to elicit a reaction and they were not disappointed. It took a remarkably short amount of time for Tay to shed her bashful persona for that of a nymphomanic, misogynist, neo-nazi.<> <> <> <> <> ![@NYCitizen07 I f***ing hate feminists and they should all die and burn in hell](https://kieranbrowne.com/images/tay-tweets-4.jpg)<> <> ![@swamiwammiloo f*** my robot pu*** daddy I’m such a bad naughty robot](https://kieranbrowne.com/images/tay-tweets-3.jpg)<> <> ![@Crisprtek swagger since before internet was even a thing](https://kieranbrowne.com/images/tay-tweets-5.png)<> <> Only 16 hours after launch, Microsoft shut Tay down. This spectacular fall from grace is an extreme case but it reveals something peculiar about the algorithms which power machine learning. <> <> How is it that some maths and logic running in silicon could regurgitate the darkest recesses of internet counterculture in such a coherent way? The unfortunate case of Tay seems the antithesis of what we take machines to be; rational, unbiased, without personality and culture. Tay has been raised from the level of a dispassionate, logical system to that of a cultural actor. Simultaneously this has made her fallible to culture and bias. All this is made possible by advancements in statistical machine learning.<> <> Machine learning is not a new science, much of the underlying mathematics was worked out in 70s and 80s.[^Rosenblatt1957]<,[^Minsky1969],[^Williams1986] However, it has had a recent renaissance due to the rapid increase in access to big data and computing power. Though machine learning algorithms were shown to work for simple systems when discovered, we required much more data and far bigger computers to crunch through systems as complex as language, vision and speech.<> <> Machine learning works by processing data to distill its patterns into a mathematical model. These algorithms can approximate arbitrary functions from data, even when the programmer is not conscious of the relationships therein.<> <> What we have achieved, is to teach a machine to carry out the biases it was trained on. Or to put it another way, the culture that is latent in the data is learned by the system and so finds expression. <> <> A problem which arises from the way we think about machines as cold, logical and objective. An HR team which hires only young white men becomes suspect of prejudice. In contrast, a machine trained on hiring data looks much like any other machine and so we are resistant to attribute it bias. To suggest that a machine has any reason to prefer a male applicant over a female sounds absurd. So too, to think that a machine would be affected by the vestiges of historical oppression of women. And yet, the machines trained on human data *are* encultured. As AI systems find use in a growing range of sensitive applications, machines exhibiting racial and gender biases have regularly appeared.[^Angwin2016],[^Insta],[^Twitter]<> <> <> The dominant culture is largely transparent to those who inhabit it. So too, it is difficult to see the influence of culture in a machine that reflects the cultural norms. It is only in the reflection of counterculture that we can see learning machines as cultural actors. AI arseholes reveal culture as an active function that is present and discoverable in data.<> <> <> <> <> ## Reading List<> - Blackwell, Alan F. "Interacting with an inferred world: the challenge of machine learning for humane computer interaction." In Proceedings of The Fifth Decennial Aarhus Conference on Critical Alternatives, pp. 169-180. Aarhus University Press, 2015.<> - Kafka, Franz. *The metamorphosis*. WW Norton & Company, 2015.<> - Sartre, Jean-Paul. *No Exit*. Caedmon, 1968.<> - Woolgar, Steve. "Configuring the user: the case of usability trials." The Sociological Review 38, no. S1 (1990): 58-99.<> <> <> ```clojure<> 1 2 3<> ```<> ## Notes<> <> [^Rosenblatt1957]: Rosenblatt, Frank. *The perceptron, a perceiving and recognizing automaton Project Para.* Cornell Aeronautical Laboratory, 1957. [PDF](https://blogs.umass.edu/brain-wars/files/2016/03/rosenblatt-1957.pdf)<> [^Minsky1969]: Minsky, Marvin, and Seymour Papert. "Perceptrons: an introduction to computational geometry." (1969). <> [^Williams1986]: Williams, D. R. G. H. R., and Geoffrey Hinton. "Learning representations by back-propagating errors." *Nature* 323, no. 6088 (1986): 533-538.<> [^Angwin2016]: [Angwin, Julia, Jeff Larson, Surya Mattu, and Lauren Kirchner. "Machine bias: There’s software used across the country to predict future criminals. and it’s biased against blacks." *ProPublica*, May 23 (2016).](https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing)<> [^Insta]:[kharyrandolph. Instagram. 18 April 2017](https://www.instagram.com/p/BTB2xC7A32c/)<> [^Twitter]:[Alex Shams. Twitter. 28 November 2017](https://twitter.com/seyyedreza/status/935291317252493312)<>", "slug": "ai-arseholes" } , { "date": "2017-02-09 00:00:00 +0000", "title": "Design and Complexity", "url": "/research/design-and-complexity/", "content": "## “simplicity, essentiality and openness.”<> <> ![Radio inspired by Dieter Rams](https://kieranbrowne.com/images/dieter-rams-inspired-radio.jpg)<> <> Simplicity, essentiality and openness are three words used by the influential german designer Dieter Rams to summarise his work. Rams was revolutionary in his resolve to simplify everyday objects through industrial design. He designed a number of technical devices (stereos, radios, calculators) with the aim of simplifying the interface to the end user without removing useful functions.<> <> Rams developed ten design principles by which to test his own work. His philosophy of design has remained relevant despite changes in fashion across half a century. His musings on what makes design good, are an interesting, if subjective yardstick to poke at interface design for neural nets.<> <> The two ideas of his that I want to focus on is in this post are; good design should *make a product understandable* and *make a product useful*.<> <> The first problem we arrive at here is that neural networks are extremely difficult to understand even for the programmers who build and train them. (here I mean the way in which a trained neural net’s decisions are formed, not the algorithms which allow training to take place, (although those are pretty complicated to).)<> <> A neural net’s decision process is governed by a web of interdependent coefficients. These commonly have no clear connection to features in the data and cannot be individually manipulated. <> <> There exists a term for this in traditional programming. <> <> “Spaghetti code” is used to deride code-bases which cannot be easily understood or changed without breaking. This is the last thing you want to hear about your codebase. So why then do we excuse neural nets of the same sins? Because they’re just so gosh darn effective. <> <> Neural nets have inherent complexity, like a radio or an iPhone.<> The question thus becomes, what complexity do we need to reveal to the user in order for her to use it successfully.<> <> The favoured design solution for neural networks seems to be, *pretend it’s not there*.<> <> Google for example, in the many years since its early forms, has developed its algorithm exhaustively. Personal data is now used to serve more relevant result. Data points like a user’s age, location, gender, previous searches, the searches of those around them, the emails they wrote their grandma last Christmas etc. could all be used in the rankings, however in all this time, the design of results page, and a user’s interactions with it have barely changed at all. What effect does my location have on my search for “celebrity chef”? my gender? that email I sent gran about Heston? What if I don’t want my location to be considered?<> <> Google is not alone in this, of the many ways that machine learning is creeping into our lives, few are visible to the end user. <> <> The other common solution for interactions with machine learning is training by example. For example, Facebook’s “I don’t want to see this” option has a direct, visible effect on the selected post but will also be used to improve the algorithm which generates the feed. This too is hard to conceive of for the end user who is unable to directly compare the effect of this change in their future feed.<> <> ![Simplistic sketch of Dieter Rams](https://kieranbrowne.com/images/dieter-rams.jpg)<> <> So, are we making good design? <> We’ve concealed the complexity of neural nets that’s for sure. And we get the functionality and power of machine learning without any unsightly neurons and synapses. But, at the risk of overstretching my analogy, what remains is a radio without any controls.<> <> *Good design makes a product understandable.* By exposing nothing of the neural net to the user we render it a ghost of the system. We see effects and may attribute these to a network or some action we took but cannot pin it down or reason about it.<> <> *Good design makes a product useful.* It’s clear that neural nets can be useful whether the end user can directly manipulate them or not. But when a user can neither understand nor manipulate the system with which they interact, are they really using or simply receiving it?<> <> ## Reading List<> - [Campbell-Dollaghan, Kelsey. "Skeuomorphism Will Never Go Away, And That's a Good Thing". Gizmodo](http://gizmodo.com/skeuomorphism-will-never-go-away-and-thats-a-good-thin-1642089313)<> - ["Gilles Deleuze." Stanford Encyclopedia of Philosophy. (Accessed 10 Feb 2017)](https://plato.stanford.edu/entries/deleuze/)<> - [Hui, Yuk. "The Plane of Obscurity — Simulation and Philosophy." Computational Culture 1, no. 1 (2011)](http://computationalculture.net/review/the-plane-of-obscurity-%E2%80%94-simulation-and-philosophy)<> - Kulesza, Todd, Weng-Keen Wong, Simone Stumpf, Stephen Perona, Rachel White, Margaret M. Burnett, Ian Oberst, and Andrew J. Ko. "Fixing the program my computer learned: Barriers for end users, challenges for the machine." In Proceedings of the 14th international conference on Intelligent user interfaces, pp. 187-196. ACM, 2009.<> - [Lippert, Ingmar. "Extended carbon cognition as a machine." Computational Culture 1, no. 1 (2011): 1-15.](http://computationalculture.net/article/extended-carbon-cognition)<> - [Nielsen, Michael A. "Neural networks and deep learning." Chapter 1](http://neuralnetworksanddeeplearning.com/chap1.html)<> - [Parisi, Luciana, and Stamatia Portanova. "Soft thought (in architecture and choreography)." Computational Culture, no .1 (2011).](http://computationalculture.net/article/soft-thought)<>", "slug": "design-and-complexity" } , { "date": "2017-02-02 00:00:00 +0000", "title": "Starting Blocks", "url": "/research/starting-blocks/", "content": "The first week of my PhD has been a mix of fun and frustration. I planned an ambitious start to my research which fizzled out due to some enrolment issues (my fault). And by the time it was all sorted, the first day was all but over. <> <> On day two I discovered that I can’t read nearly as many journal articles in a day as I hoped. Issue one of *Computational Culture* seemed an achievable goal at 9:00am, but by 5:30 I’d only polished off three of ten articles.<> <> On day three I setup this website which is necessary but not exactly research, and day four (today), I wrote this post. <> <> In short, this week hasn’t been the running start I was hoping for, but to be honest, it’s just good to be back at uni.<> <> But enough of my narcissistic, gen-Y, life-sharing; this is a research blog and as such I think it would be best that I give my best attempt at a summary of what my research actually is. The reader should be warned however, that I’m still trying to work that out myself.<> <> As it stands, I’m looking at the design problems created as traditionally programmed computational systems are replaced with new-fangled machine learning algorithms. <> <> “Traditionally programmed” here means using logical and mathematical primitives to outline a decision process. Think “if this, then that”. This method is by no means without its flaws but importantly it is transparent; which is to say another programmer can examine and critique it.<> <> Machine learning is another beast entirely. In machine learning, a programmer does not outline a decision process; instead, she chooses from a set of general algorithms, finds or collects a large enough data source and ‘trains’ the algorithm on the data. <> <> Machine learning is a powerful tool. These algorithms have proven to be very effective at solving problems we thought only humans could. (See [The Unreasonable Effectiveness of Recurrent Neural Nets](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)).<> <> Crucially though, machine learning is not transparent. A programmer can no longer look to the code to understand how a particular decision has been reached. Nor can she critique that decision.<> <> The elements which govern the system’s decisions are not in the code itself but rather in a web of interlinked coefficients with no discernable connection to a feature or rule. This structure is difficult to visualise and even more difficult to manipulate.<> <> ![](https://kieranbrowne.com/images/jenga.jpg)<> <> What we risk with digital services increasingly dominated by machine learning is the techno-mysticism that comes from incomprehensible complexity. To the consumer, a search engine might as well be magic, they have no way to perceive or conceive of its processes. What are the consequences when the programmer too, cannot understand and manipulate the services they create?<> <> What we require for machine learning systems is a design language which reveals the system’s nature to the user and renders it changeable.<> <> The elements of traditional computer interfaces are not suitable for this because they are analogous to the controls of a physical machine; buttons, switches, dials. They are functionally skeuomorphic. A button by it’s very nature implies behavioural consistency.<> <> This direct relationship is appropriate for the cogs and circuits in a machine or if-this-then-thats of a traditionally programmed computer.<> <> However to represent the hazier, probabilistic actions and decisions of a machine learning system we will need to create new design abstractions which leverage a user’s associations with other probabilistic systems or entities; or perhaps a clear enough system would need no recourse to skeuomorphism and analogy.<> <> <> ## Reading List<> <> - Blackwell, Alan F. "Interacting with an inferred world: the challenge of machine learning for humane computer interaction." Proceedings of The Fifth Decennial Aarhus Conference on Critical Alternatives. Aarhus University Press, 2015.<> - [Jun’ichirō Tanizaki. In praise of shadows. Leete’s Island Books, 1977.](http://dcrit.sva.edu/wp-content/uploads/2010/10/In-Praise-of-Shadows-Junichiro-Tanizaki.pdf)<> - [Munster, Anna. "Nerves of Data: the neurological Turn in/against Networked Media." Computational Culture 1 (2011).](http://computationalculture.net/article/nerves-of-data)<> - [Wheeler, Michael. "Thinking Beyond the Brain: Educating and Building from the Standpoint of Extended Cognition." Computational Culture 1 (2011).](http://computationalculture.net/article/beyond-the-brain/)<>", "slug": "starting-blocks" } ]