During my undergraduate studies, I majored in fine art photography, focusing on creating images through interdisciplinary fine art techniques, such as analogue photographic works. I admired Jo Ann Callis, a pioneering female fine art photographer who teaches at the California Institute of the Arts (CalArts). Therefore, I always regarded CalArts as my dream place for graduate school. In my senior year, I applied for the graduate program in photo/media at CalArts and was unexpectedly accepted into the Art and Technology program, where I began exploring interactive media and generative art.
I have been drawing since I was a child, but painting is not my primary medium. I do not enjoy the process of creating something from scratch. Instead, I like the process of forming new things through different arrangements and combinations of existing objects, which is why photography offers me the possibility of reconstructing reality. After studying art history, I discovered that this approach aligns with the assemblage art techniques and creative methods popularized in the 20th century. In my process of making art, I first create the system and then collaborate with it to generate new forms, materials, compositions and aesthetics. In other words, my art is created by raising conceptual questions, designing different critical machines for generating a “ready-made” and then further processing the readymades with artistic decisions. This process shares many similarities with 20th-century movements like Abstract Expressionism, Dadaism and Futurism. For example, the creation process emphasizes chance and choice operations to offer new artistic possibilities.
Cangjie’s Poetry
Cangjie’s Poetry is an intelligent multimodal system designed as a conceptual response to the future of semantic human-machine interaction. I created this work with computer scientist Donghao Ren. This artwork draws inspiration from Cangjie, an ancient Chinese legendary historian from around 2650 BCE, who is credited with inventing Chinese characters by observing the characteristics of everything on Earth. We trained a neural network called Cangjie to understand the constructions and principles of over 9,000 Chinese characters. After successful training, Cangjie can interpret images through the lens of these characters and produce new symbols constructed from Chinese strokes. Additionally, we implemented a pre-trained model to generate localized descriptive sentences of images in natural language, thus creating meaning for this symbolic system.
In the art installation, the Cangjie system uses a camera to capture its surroundings, transforming the real-world stream into a cluster of ever-changing new symbols in real time. These novel symbols, made from Chinese strokes and intertwined with the captured imagery, are visualized algorithmically as an abstract, pixelated landscape. This landscape dynamically moves, evolves and writes poetry based on the live-stream image data. We project this visualization of the semantic landscape onto a wall in the exhibition space as the first projection. Simultaneously, Cangjie generates descriptive sentences about the surroundings based on its interpretation. These sentences, designed as flowing poetry written in ink, are assembled with fragments of real-time captured imagery and projected as the second display in the space.
This artwork transforms the machine’s interpretation of its surroundings into an interactive semantic experience, constructing a conceptual and futuristic mode of communication between humans and machines. This work addresses the tension and fragility inherent in our coexistence with machines and serves as a conceptual expression of a future language that reflects on ancient truths. It aims to evoke a sense of enchantment in this era of artificial intelligence. Much like the legendary Cangjie did nearly 5,000 years ago, our Cangjie system continuously writes poetry collaboratively with humans as long as the real world exists.
We also created a special edition of Cangjie’s Poetry as an artistic response to the COVID-19 pandemic and social distancing regulations. In this version, instead of using real-time live streaming of surroundings in the art installation as a video feed for the Cangjie system, we collected and curated submissions of daily footage worldwide. We edited the submitted footage from Canada, China, the United Kingdom, the United States, Japan, Korea and Egypt. Cangjie interpreted the edited footage and generated the visualization based on its interpretation. We rendered the animation generated by Cangjie and designed the sound using the granular synth technique. This special edition of visualization is a pre-rendered audiovisual experience of collective memories. It travels and extends Cangjie’s vision to different places by collecting footage from around the world.
Cangjie’s Poetry has garnered global recognition, having been awarded Honorary Mention at the prestigious Prix Ars Electronica, the world’s most notable electronic arts award, as well as the Red Dot Design Award and the Best in Show Award from Siggraph.
ReCollection
ReCollection is an interactive AI art installation in collaboration with AI scientist and researcher Rodger Luo from 2022 to 2024. This artwork was inspired by witnessing my grandmother’s memory regression due to dementia, where her cherished stories dissolved into fragmented words. Dr. Mary Steedly once described memories as a “densely layered, sometimes conflictual negotiation with the passage of time.” In 2020, over 55 million people faced the painful reality of memory loss due to Alzheimer’s and related dementias. Yet, amidst this poignant backdrop, the emergence of text-to-image AI systems in 2022 offered a glimmer of a new perspective, harnessing the power of language to imagine and reassemble fragmented memories, potentially weaving together what time and disease had stolen.
This work starts with a list of conceptual design questions: when we coexist with machines, will we accumulate synthetic recollections of collective symbiotic imagination? Is language capable of re-weaving and synthesizing memories? How does our collective memory inspire new visual forms and alternative narratives?
ReCollection is an assemblage of intimate human-machine artefacts that emphasizes contributions from three sources: artists, machines and participants. This customized AI application employs multiple AI techniques, including speech recognition, text auto-completion, and text-to-image, to convert language input into image sequences of new memories. As an interactive experience, participants whisper their personal memories in fragmented sentences, and our system automatically fills in the details, creating touching new visual memories.
We developed our customized AI system by fine-tuning a pre-trained transformer-based AI model to learn from the visual memories and the descriptions of Alzheimer’s patients’ documentaries. The system imagines new memories of “love” and “loss” by interpreting real-time narratives from participants in the installation. Our system serves as a vibrant and inclusive conversation starter, transcending boundaries with support for over 89 different languages and embracing diverse cultural artefacts.
In the art installation, we chose not to showcase the direct visual output generated by our AI system. Instead, we drew inspiration from fine art practices such as the Monotype, a printmaking technique dating back to the 1640s, and slit-scan photography, known for capturing sequential slices of a subject over time. We aimed to present ReCollection by combining generative methodologies with fine art practices, investigating new aesthetics that explore fleeting visual imagery undergoing dissolution, tilting, printing and reprinting over time.
By providing a conceptual framework for non-linear narratives that constitute symbiotic imaginations and future scenarios of memories, culture production and reproductions, this artwork may inspire the cure for memory regression. It offers a future scenario, a thought experiment, and an intimate recollection of symbiosis between beings and apparatuses. It raises awareness of future memory preservation and empathy for the dementia community through a personalized aesthetic experience. It presents an artistic approach and future prototype for cultural heritage reproduction and reimagination, exploring the tensions between visual representations, language and narratives.
Astro
In 1970, Sister Mary Jucunda, a nun based in Zambia, wrote to Dr. Ernst Stuhlinger, the associate director of science at NASA. She questioned how he could justify spending billions of dollars on projects exploring Mars while countless children were suffering from poverty. Dr. Stuhlinger responded with an iconic image of Earth taken in 1968 from the moon. He explained that Earth is a beautiful but vulnerable island in an infinite void. In his letter, he stated, “Although our space program seems to lead us away from our Earth and out toward the moon, the sun, the planets, and the stars, I believe that none of these celestial objects will find as much attention and study by space scientists as our Earth. It will become a better Earth, not only because of all the new technological and scientific knowledge which we will apply to the betterment of life, but also because we are developing a far deeper appreciation of our Earth, of life, and of man.”
In 2021, Astro was invited to participate in the Metaxis festival to celebrate the 60th anniversary of the first manned flight into space. Inspired by Dr. Stuhlinger’s letter, Astro is an audiovisual artwork that explores the question of ‘why explore space’ from an artistic and imaginative perspective.
Astro is an immersive audiovisual experience performed in a full-dome format. In Astro, Earth –the only known astronomical object that harbours life – is revealed through the lens of an intelligent being in outer space. As the lens zooms in and out, the journey of observation brings multi-scale discoveries of ecological changes and machinic visions with artistic imagination. The audience witnesses a vast forest engulfed in flames, melting icebergs revealing hidden information, ancient pseudoscientific diagrams, AI-generated latent walks, data-driven landscapes and algorithmic generative visuals. This work poses a profound question: why explore space in the context of the known and unknown, folding and unfolding?
Astro has delivered an immersive and thought-provoking audiovisual experience to audiences across continents. It has been showcased in major full-dome venues and performed in countries such as Germany, the UK, Canada, China and the US. International visitors have had the opportunity to witness Astro’s exploration of ecological changes and machinic visions. The artwork received the Prix Trames Sonores Award from the Society for Arts and Technology (SAT) in Montreal for its innovative fusion of audio and visual elements. In 2021, the performance was particularly noteworthy for taking place at the historic and renowned Zeiss Planetarium in Germany.
Through its full-dome format, Astro offers an imaginative and engaging way for audiences to explore complex ecological themes through the perspective of an intelligent being in outer space. It provides a fresh viewpoint on our planet’s challenges and encourages viewers to consider their own place within the broader ecological system.
Recommended citation: ZHANG, Weidi. A speculative assemblage. Mosaic [online], October 2024, no. 201. ISSN: 1696-3296. DOI: https://doi.org/10.7238/m.n201.2403
Deja un comentario