INTRODUCTION: London-based George Simms, a Master of Arts student at Goldsmiths University studying Computational Arts, reflects on his speculative practice, exploring 3D and virtual worlds through different perceptions. His work has reimagined the futures of touch interfaces as well as creating generative render tools with machine learning.
INDEX:
(LINK) A Speculative Hello . . . Intro to me and to my practice.
(LINK) Virtual Worlds . . . . . . . . How I have imagined virtual environments and worlds.
(LINK) Ways Of Touching . . . . Research project into haptics, with speculative animations.
(LINK) Mesh2Matter . . . . . . . . . New materiality, Machine learning generative rock and mineral textures.
(LINK) Collapse . . . . . . . . . . . . . . . Collaborative AV animation exploring forces in virtual worlds.
(LINK) Advice . . . . . . . . . . . . . . . . . Some bits of advice for building a strong craft and perception.
A SPECULATIVE HELLO: Hi. My name is George Simms, I am a speculative digital artist, building digital tools and augmenting perceptions around them. Much of my work is rooted in the ideas of speculative realism, a genre of theory that conceptualises a world outside of the human gaze, one outside of human structures. This position see’s the worlds we create through science and society as speculative possibilities of what the universe and cosmos could be. Freeing us to question and imagine a world more fluid and complex than the structures we could impose on it.
When approaching this article I was looking back at some older works, re-reading some of my thoughts from years ago. I realised how far I felt I had come and yet how close my inspirations still are, looking through my notes one concept still remains strong today:
"You can’t understand it until you know how it works."
In my earlier film, net art and installation work this was very much the ethos. It focused on exploring how our worlds are extended through digital portals, and how these structures perform a subjective world into set narratives. As a rough example mainstream films and media construct a memory, a perception and an experience, but if you know they do this and you understand how they do it, then in some ways you acknowledge or counter this bias or performance of the world.
“It matters what matters we use to think other matters with; it matters what stories we tell to tell other stories with; it matters what knots knot knots, what thoughts think thoughts, what descriptions describe descriptions, what ties tie ties. It matters what stories make worlds, what worlds make stories.”
― Donna J. Haraway, Staying with the Trouble: Making Kin in the Chthulucene
I am currently culminating my studies for a Master of Arts in Computational Arts at Goldsmiths in London, having the honor and privilege to study under Helen Pritchard, exploring computational theory and the concepts of feminist technoscience. The quote above is a beautiful snippet from Donna Haraway, which in some ways represents my transition of perception during this study, extending my earlier mentioned need to deconstruct to understand and counter to a need to deconstruct to reform. It’s a very complex quote but for me it starts to display and unravel the intricacies and interdependent nature of the world, how mechanisms of representation and simulation not only mimic the world but actually shape and construct our worlds.
This time studying made me reconsider my work and the effect I want to be having on the world around me. Forming from this new approach I have a need for my work to be useful, or as the Haraway quote beautifully put it even changing the perceptions around existing tools. Looking to explore this in a direction that works with other people as there is no world without others and a reality between us is hopefully more vivid and true.
Still from Houdini world building workshop
VIRTUAL WORLDS:
3D software and other digital virtual environments provide me with a testing ground outside our own worlds; a fertile environment where many if not all of the laws constructed in the space are created by us. These apparatuses provide strength to our weakness, a way to imagine complex and intricate interactions throughout a dynamic window, giving us great insight into emergent worlds. Digital spaces are not without their limitations of course, but they do provide amazing liberties which our imagination can run with.
Through these dynamic systems of mathematics we can create many representations of natural emergence, fluids like liquid and gas, rigid and soft bodies as well as all manner of growth algorithms and more. These create beautiful representations of natural phenomena, in ways helping us understand the workings of the world around us, and amplifying the fictions we build with them. Realism is a well trodden path though and I really want to ask more of these softwares, they are so capable and powerful.
Teaming these mechanisms up with our imagination, how far can we go together? What fantasies can we fiction into reality? And how can these help us to understand and change the world around us?
To explore these concepts I’m going to run through a few different projects that orientate 3D animation, virtual tools/worlds and their speculative uses. Thinking about them as a tool that not only constructs but also creates space to think through structures of exchanges.
WAYS OF TOUCHING - TEMPORAL MATERIALITY: This animation was produced as part of a research project exploring the imagined futures of haptic interfaces, interfaces which produce the sense of touch. Carried out by Julian Mercier, Johnny Fowler and myself. We focused on a case study of Ultra-haptics, a start-up based in Bristol which produces revolutionary haptic interfaces, creating the sense of touch in mid air. It works through an array of ultrasonic-transducers modulated to produce pockets of low density air, when in contact with skin this pressure difference creates the sense of haptic touch. After testing out their interfaces and looking deeper into the psychology and cognitive understanding of touch, we saw a promising but misrepresented future, with haptics only grazing the surface of what touch really is. The sense of touch entangled between a number of different sensory modes and combinations, far beyond haptic.
This led us to build a speculative future of touch, one away from its cultural-imaginaries (Term from Lucy Suchman to describe futures of technologies that are imagined through culture, film and media.), the dreamed haptic interface built up in mainstream media and science fiction, seen in ‘minority report’ and other sci-fi. We thought about the revolutionary qualities of the Ultra-Haptics interface, how it provided a new temporal materiality to interface with computers, whilst still contemplating the limiting outlook the current development of haptics has in terms of the grand scheme of touch. We imagined unlocking the complexities of touch, the sensorial entanglement, composing new languages through this fluid medium. Creating ways to connect to one another beyond our current limitations, hacking into our sensorial languages and transforming the ways we can connect through touching.
To represent this I produced this short animation in Houdini to speculatively probe the future of touch. The animation is a sphere transitioning through different material shaders, moving between the structural materials of our modern world (glass, steel, cloth), the materiality of the medium (sound, vibration) and the materiality of touch itself (skin). Using a set of attribute transfers to control a tree of shader mixers, this simple setup gave really beautiful results. The shader mixers create points of convergence between the materials, promising a touch beyond the ones we know so far. Computer shaders materially are in many ways very similar to the temporal existence of these haptic interfaces, presenting skins to abstract forms, ever shifting and fluid.
Attribute transfer controlling shader change.
Segment of shader VOP that is repeated to control the shader layers, using Shader_Controller controlling a layer mix node to blend between shaders.
This was actually the first job I rendered on GridMarkets, their student deal made it affordable for me to produce it for the project and the quick turn around meant we had the piece in time for the deadline.
MESS2MATTER: This project is another speculative exploration into digital and virtual materiality. Looking into the rendering and texturing of materials, I started by deconstructing how these apparatus figure the world, or at least the virtual worlds they produce. In particular, focusing on ray tracing and its creation of a projected reality, a structure formed from abstracted and inverted understandings of light and its experience through vision/cameras. In this setup the virtual camera fires out hundreds of thousands of rays instead of receiving an insurmountable sum, like a real world camera or your eye would. [For reference the eye has over 100 million sensors.]
Looking to machine learning as another way of understanding the rendering process, allowing a generator model to be trained to imagine this translation in new ways. Through high dimensional nets these virtual meshes are translated into rendered images. Cascading through convolutions and deconvolutions these forms are interwoven out of layers or segmentations the model understands are there and imposes the imagined relationship these endorse.
Generated crystal produced through the trained U-Net.
In actual fact the complexity of creating a render engine from Machine Learning is immense, and that's an understatement! So I focused on the feasible element of texture generation, thinking about bringing some of the materiality of the world into the virtual by creating textures from memories or understandings, instead of blind calculations and static textures. For the texture generation I used a modified Pix2Pix setup, trained on a dataset of crystals/minerals and their photogrammetry mesh partners. When trained, this produced a generator which outputs beautiful and complex generative textures to use for rendering. The setup is built in Python, Tensorflow and Blender, mainly because of the speed of rendering and compatibility, but the trained U-Net generator could be loaded into any python, c++ or java environment and integrated into the render pipeline.
This has quite a big pipeline, covering batch photogrammetry, datasets creation, training a Pix2Px model and using it in a render pipeline. This is all in python as well so I’m not going to explore the technical side in this article but if you want to know more check out this reasonably well documented Github repo.
“The Transform Node of Houdini”
I concentrated the generator on minerals for a number of reasons, as an object of the world they are very representative of the base materiality, fractals of pure molecular building blocks, producing stunning and sometimes alien forms.
In terms of computation they are key to the materiality of the hardware, forming the very components the digital worlds are built from. Minerals also hold abstract and essential roles such as keeping time and memories. Finally from a practical and functional level for the Pix2Pix apparatus it provides a subject matter which is regular enough to comprehend but complex enough to imagine into.
Machine Learning Generated texture rendered on the original mesh in Cycles.
At the moment this project is in its early stages with a successful test run, but I think it will be exciting to move it on and see what other dimensions we can weave into its process of imaging and rendering materiality. Working with its generated formation to think deeper about what is necessary in emerging forms in our real world, and how we can translate, test or challenge our conceptions of them through new technologies like machine learning.
If you want to read more about it, check out this essay I wrote on it with more depth into the pipeline and training of the ML model HERE.
I also used the trained models from this project to help control the genetic selection of a cubic crystal growth algorithm. I produced It’s a bit of a crude exploration into machine learning understanding and making decisions on aesthetics, but check it out HERE.
COLLAPSE: Collapse is a short film produced using a Gridmarkets Covid19 grant, made with long-term collaborator and Composer Izzy Nahkla. It was also conceived during my time on the In-grid.io residency at Arebyte gallery, and was partially fictioned through a workshop exploring 3D animating and its possible narratives, addressing aesthetics and relations of virtual spaces, their journeys and motions.
Produced in Houdini, this short film is a journey through structural collapse and reformation, using constraints and forces present in the Houdini mechanism to imply different relational and contextual changes within the scene. Seeing the virtual space as a place free from real world laws and constraints, taking immovable and unknown forces and structures which construct our world, such as gravity, and fluctuating them. Exploring how relationships in other worlds could be enacted, as well as how dynamic systems can be drastically altered through the slightest changes.
Unnatural forms peeking through, behind the columns.
The film starts by creating the space, reflecting a future through the wall, this reflection acting as the single animated element bringing depth to a dormant space. Opening up to an other worldly form, a grid of titanium rectangular cells held in a stiff structure by laser blue forces, unnaturally floating above the cold ground. As we explore the form we see it start to weaken, what were once strong bonds break. The cells drop, colliding into one another, creating an emergent collapse. When the structure finally gives in, the whole form drops and gives into the “natural” forces of this world, crashing into the floor, flattening the previous hierarchies and constructs.
I created this VOP inside the glue constraint to control a natural looking breaking of the glue constraint. I used a ramp
param to control the collapse over the frame number. When Cd is over a set number the constraint is deleted.
Close up and scaling of the forms, from collective structure to interdependent individuals.
After the dust settles and the gravity and friction take hold, a new force starts to come through. Instead of a grid of constraints it is affected by a field of velocity, a movement that overcomes its frictional bonds, coming from within or intra-acting* through another plane. The individuals unified in a new direction, out of an anarchic yet fluid motion rises a new form, kinetic, dynamic and emergent. As you see the mouth eclipse the animation cuts, leaving you to fill the vacuum, potential formations and exchanges are left for the viewer to imagine.
* Intra-action is a term used by Karen Barad to talk about working within an exchange, intra - within, action - change/movement.
Thinking about agency (the power) subjects have in exchanges, extending this beyond humans, into the natural and external worlds.
A base form rotating into itself
That Rotation translated into a point cloud velocity field
Nodes to create the velocity field
I haven’t worked on this part of my personal practice for a while and producing even a short AV piece for myself was really rewarding. It was great to revisit film and animation techniques and concepts after spending some time working in code and studying computational theory. GridMarkets as a platform provided a super quick and easy way to develop the project, letting me go way beyond my hardware restraints and push out tests quickly for a fast project development. The focus remains on the content and creation instead of the rendering limitations, allowing the close collaboration of Izzy and myself on this project, using this real time dynamic to simultaneously develop the animation and sound together. Providing fertile ground for each to feed off of the other, the soundscape coming from the test renders, the camera angles and shots imagined in listenings of the early compositions. I really enjoyed producing this abstract animated experience and I am looking forward to pushing out some more collaborative works in the future.
ADVICE: I thought it might be helpful to conclude this article with a few bits of advice I would give myself a few years ago, or to anyone really. A few may sound a little cliche or easier said than done but understanding these ways of building a craft have been really helpful.
Time, things take ages - Learning and understanding a topic takes a long time but we can learn a lot, it just takes time. I remember my drum teacher told me at a young age that it takes eight years to master something, unfortunately that something wasn’t drumming but the idea stuck. I have heard it measured in hours and other metrics but it's just a long time and a lot of practice. It just takes a little bit of time often, and it normally gets easier the more you do it!
Second Languages, it’s hard to translate - I started thinking about this when I was in Taiwan talking to a friend I met out there, he was semi-fluent and learning Mandarin. We talked about the point where you stop having to translate the words from your first tong and just started to know how to speak in that language. The point you subconsciously start understanding the meanings and structure of this interchange. I started to take this concept and approach to non lingual situations, try to understand things as second languages, getting to that level where the exchange is fluent, allowing you to think and understand the craft in ways you may not be able to put into words.
Skill Building, day by day - Not only is having a range of skills always a plus practically but it also helps you to understand different ways of approaching a project, software or situation. By skills this doesn’t mean just technical skills or computer languages but also means many other skills. One of the most beneficial skills I have been working on recently is teaching or holding workshops. Having to explain what you normally do without thinking to someone who has little or no previous understanding can actually allow you to see it from a completely different way. I also find allowing yourself to concentrate or work through an idea in a totally different context or medium can be an amazing way to subconsciously process a task or problem. Hobbies and skills I find super helpful for thinking things through are doodling/sketching, writing and making music. I recommend things that are quite meditative and produce new rhythms and dimensions to your process.
Question Everything, say’s Don Ihde - There's a great essay by Don Ihde called ‘Why Not Science Criticism?’. He talks about how science and structures of knowledge sometimes ignore or create friction for prioritized lines of thought. Ihde encourages us to criticize these traditions and at the end of the text calls for new technoscience critics. People who can sit on the fence, with a deep love for the area and reasonable understanding, but are still concluded as an outsider. Someone who is not experienced enough to ignore the black box questions or “false alarms” as he puts it, and someone who does not necessarily follow the trodden frameworks to form their conclusions.
Taking this into our practice doesn’t mean that we get blinded by the complexity of the system, but we should check what's going on under some of the panels and behind some of the facades. Not taking down the entire structure, but just being inquisitive, questioning and rearranging, learning what foundations you are working from and testing them. I always imagine it as kayaking on a deep sea, floating above an endless unknown chasm, with each paddle pushing us forward whilst dipping a little deeper into the ocean below.
George Simms 2020
By: Patricia Cornet
GridMarkets marketing