A sign is just a metaphor. Interview with Andreas Lutz
In situ at OSA Festival, Sopot / Poland, 2023 (Photos by Helena Majewska and Konrad Gustawski)
I didn’t know why I was so moved by your performance during Open Source Art Festival 2023, but then I started reading about your project, Abstract Language Model, and – as I’m a natural language processing engineer – I realized that it didn’t happen by coincidence. It seems that I deciphered your message, subconsciously read what you did there. Then I read about the whole creation process, and I thought: “That’s a sign.” But maybe let’s go back for a minute to your performance during Open Source Art Festival, because from what I found, it was your first show in Poland.
Yes, actually it was really my first performance in Poland; I think I’ve also never had an exhibition in Poland.
And how did you find the audience? Was there anything that surprised you?
No, to be honest. When people come to this kind of festival, to this kind of new media exhibition or performance, it’s the same kind of minded people, the audiences all over the world are pretty similar. I was there only for one day, so there was not so much time to talk with many other people than the organizers, but I felt it was a well organized festival with a selected audience. I don’t know how many people were there, maybe 150 or so, but it felt very nice actually.
What I noticed was that during your performance everyone fell silent. During other performances people were talking a little bit, they were moving their heads, their hands, and I got the impression that during your performance everyone felt completely silent. Maybe this is because a lot of people don’t know exactly what to expect during performances because the artists invited to Open Source Art Festival are completely different. I guess people were just swept off their feet by both the visuals and music.
Yes, on Friday and on Sunday the focus was mostly on sound. I think my performance can be perceived as challenging because it’s both visual and sonic. At the moment of the performance I don’t realize anything about the audience, so I don’t notice if somebody’s talking or how it’s going because I try to make a good show. Sometimes it’s hard for me to see if people enjoyed it or what went on during the performance… but it’s good that you observed that people seemed to like it.
I rarely see people so focused on something, so that struck me… There was this important visual part of your performance and in one interview I found information that you stated that you’re a visual artist who works primarily in kinetic sculptures and digital art, and that made me wonder where this puts music in your activity. Are all those fields for you inseparable or do you perceive yourself first as a visual artist then as a musician or the other way around?
I think you can’t separate them in a way. So you have an idea, and my artistic idea came about that it’s about language, about the language between humans and machines. That’s a really broad aspect. But then how do you express this idea? Can it be in an installation or in a performance, or in music? And to be honest, I can’t really explain why sometimes it’s a sound, why sometimes it’s in a kinetic sculpture or something like an audio-visual installation, because for me they’re not separable. Music is what I started out with, so it was the first thing I had done before I even started with installation work. And this is what I always get back to. I guess most of the artists in the new media field you would ask nowadays would say everything is connected anyway, so you don’t distinguish there.
Abstract Language Model, Trained artificial neural network, custom software, 2023
Four-channel synchronized video, 12:05 minutes, 4x 2650 x 1440 px
But you’re specific in a way that you create, you conceive, your own language for your art; it’s not the metaphor, I mean the exact process of creating a language, and I was wondering what the process looks like, and whether the language you create is disposable or if you have any plans to use it later on.
I’ve always worked with this idea of an assumed language system; it was always a theory, a theoretical concept, but I’ve used it in sound or installation work. Now, with AI and machine learning, it is possible to make a practical concept of the whole idea. I was always interested in the foundation of communication – I talk German, you talk Polish, so our connection is English – also between machines and humans there needs to be this kind of connection. Somehow, everything starts from the language; how you feel, how you express yourself – it’s all based on this foundation. I was just thinking – it’s still a theoretical concept – about making it possible that we, as human beings, don’t have to learn to adapt to the machines, and then the machines naturally understand us. Also, right now we have to make a stretch to understand binary expressions for example, and machines try to understand us in this analog way, basically speaking, so it’s not a finished process. That’s something I will continue to work on, but for Abstract Language Model it was just the first time, because with the help of machine learning, I actually could create a model which can be practically applied. I’m still going to work on the topic.
You focus quite a lot on the relationship between human beings and technology, and for Abstract Language Model you created something you call a “trans-human/trans-machine language.” In the description of the whole creation process you wrote something like “what if there is an inner monologue by a machine before communicating to the outside world” – somehow it seems you personalize the machines, and I was wondering how you perceive the relationship between human beings and machines. Do you think it’s a kind of a power struggle, or do you think that we operate with the machines in a peaceful way?
The relationship between machines and humans was always described in a way that machines should be our servants. It comes from a human mind that somehow machines should serve us or they’re used for making our life easier. But now there’s a shift, in the last few years or so, in this view. For me, it was always more interesting to assume that machines somehow are self aware, and they have consciousness, so it’s not a question so much of how this practically could look like. I’m not talking about how you can realize something like a robot or like a droid; I’m more interested in the philosophical side. Assuming a machine can have a language, so it’s self aware, and then for me it has always been the most important or interesting question – what happens then? Is it possible that machines, between themselves, also develop what we already have as humans: culture, music, sound, or religion, all these kinds of metaphysical things? And that’s why you can always find in my work descriptions of things like, for example, a “monologue,” which comes really only from humans because only we know this concept and we are self-aware that not always what we say is what we really think. I always challenge these ideas – basically, what we already know from us, human history, but apply them also to abstract machines; that’s really interesting.
Some people are pessimists; they think that machines would start taking their revenge on us, because we treated them as servants. And it’s somewhere between sci-fi and philosophical discussion. You gave them their own voice by your work. You gave them language, you gave them, I would say, even personality. And this is something really fascinating.
But that’s my point, because I think I didn’t give it to them. I’m just thinking about it from a human mind perspective, from my perception, what it could be. But I just make assumptions. I don’t want to continue this process that the human being says “machine is what we think it should be” so I say “machines should have the freedom to be whatever they want.” I’m just interested in what that could be.
In situ at Sónar+D, Barcelona / Spain, June 2023 (1-channel version)
I guess at this point we can only wonder, because nobody knows!
Yes, and isn’t it a nice thing? You then just realize we’re not in control; maybe we, as humanity, have to accept something else, something that doesn’t depend on us so much, and that’s a process I find very liberating.
Also, some people focus on the moral side, the moral aspect of machine learning, and in your case it was semiotics, the sign; you said in the process of creating an abstract language model that between two existing signs there is an infinite number of potential signs. That brings me to the next question – how do you know that the process is finished if there is an infinite number of signs? How did you know that it was done and you could stop at this point instead of looking for more signs?
Well, I don’t know, and that’s the point; I’m going to make more progress about that, but, as I said, it’s not finished. It’s funny that you mention this, because my artistic research shows that between two letters an infinite number of signs can exist, and this idea alone says we only have a limited view based on our language model. If you really think about it and extrapolate, there are so many hidden signs and “infinity” in even only one word, so it’s never going to be over for sure. When you learn something new and you have to write about it, you’re aware you’re never gonna know enough because the research will always go on; it’s the same principle.
So maybe it’s more of a philosophical question – what is a sign for you? Do you have your own definition of a sign?
Well, a sign can be a lot. A sign, how we, humans, define it, is just a metaphor for a lot of things. So a sign itself can be just pure information, but it also can have a lot of meaning and history in it, for example. If you just take any letter, in a lot of different cultures it can have a lot of different meanings. Just the sign itself means in a way nothing; it’s how you fill it up with information and with meaning. I think the same is true for communication and information itself – it’s how you use it and how it’s used by people or maybe machines who are using it as well. I would say it’s not a strict concept, it’s always changing, exactly like language itself has always changed.
In situ at Sónar Istanbul / Turkey, April 2024 (Photos by Mete Kaan Özdilek, Sónar Istanbul and Zorlu PSM)
So it’s something more liquid…
Yeah, it’s a fluid thing, and it always changes, like the society always changes and also like the world changes through the times.
But when I looked at your album Abstract Language Model and at the titles of the particular pieces there, my first idea was that they seem to refer to the phases of a machine learning process. But the result, even if I think only about music without audiovisuals… It seems very spacious and just simply beautiful. I was wondering if the beauty that came out of here was the result of the technological process, or was it your aim itself just to create something of aesthetic value.
It’s both. Some parts of the music are also generated from interpolation through the models, so what you could see in the visual way was also sometimes reflected in the sonic way. But here again – the same is with installation or sculpture. Obviously, I have my own aesthetic vision of what something should look like or how something should sound, and the same goes for the album or for the installation. If you listened to all my history of audio works, I did a lot of research and analysis on how abstract soundscape sounds, so now I came back to making it more understandable for an audience or for human ears, so that it’s aesthetically appealing in a way. Now I would say it’s more a mix of my aesthetics, but also it has this abstract communication in a sonic way in it.
Did you train the artificial neural network for Abstract Language Model yourself? I’ve seen some screens in the document that you’ve recorded from Visual Studio Code or something; a lot of coding, but I was wondering if you did it yourself and if yes, did you learn anything new from it?
Abstract language model or the machine learning part was always meant to be used as a visual experiment, because that’s just the first part of this research. It was more about the analysis of the letters to find out all these infinite “letters” or “signs” which are hidden, so the next step will be to turn it also into a more practical and usable model… But this first step was more to discover and research about the visual side of the thing that you have. All the human made signs are arranged in a machine readable format or model, and you can make interpolations between all the characters that ever existed.
Yes, but still all the hard programming work was yours…
Well, I have some friends whom I can always ask for the super heavy programming stuff, mostly that’s part of it, but the rest I try to do myself. For building the neural net and training all Unicode characters there was a lot of Python programming involved, but usually there are already some people who have discovered something which one can use, and then you put everything together. That’s usually the process or how inspiration works generally, I think – you see what other people did and then you put it together for your own idea, for your own concept and it becomes something new, your creation.
Did you find it interesting as well, or was it the most tedious part of the process that you needed to see sit down with your code and work on it?
No, no, I enjoyed it; this is also where I come from. A lot of my pieces are really technical and have a lot to do with programming. I enjoy the research part and putting something together, because this is the part you have to think about. Once you do it and it has a fixed structure, it’s over because then you put this in a form which you can present to an audience. The most interesting part for me is always the research, finding new things and creating something. That involves all parts: programming, maybe doing something with your hands, creating sound… The aspect of doing is always the thing I enjoy the most.
…and this is what I loved about your work – that the beauty that stems from it is not the result of pure coincidence, but the result of hard work and a well organized process.
That’s what you could also say about learning. Most people today, when they read about ChatGPT and all the things connected with AI, think “okay, so AI really can do everything”, but what most of them forget is why AI can do this: it saw it somewhere, and this somewhere came from the human brain. ChatGPT can always write a nice poem because it read one million lines of nice poems by really good writers. It is the same with everything we discussed before: if there’s a fight between machines and humans – that’s all like a sci-fi thing. I think what is a nice look into the future is that we can coexist; we can just take the best of machines and humans, as machines need us to be able to create something or to understand something about beauty and aesthetics, and we can cooperate with machines to make it easier for us or maybe get results which would take much longer for us because our processing speed is super limited. That’s an exciting way of looking forward to finding a good way in which we can coexist.
In situ at OSA Festival, Sopot / Poland, 2023 (Photos by Helena Majewska and Konrad Gustawski)
Well, that was such a nice, I would say, closing remark, because I’ve got the last question. When I came home from your performance, I also listened to the album and I noticed something. I had the opportunity to see the whole performance with audiovisuals and the music, and I have the impression that listening to the album itself gives a completely different feeling – that it carries a different meaning – and I was wondering if it was on purpose to make it as something completely different when listened to without the audiovisuals, or do you intend it to be the whole only to be listened to together with audiovisuals?
No, it was not intended, but what you’re saying is just something I experienced – or is what you generally experience as an artist: a lot of people see your work completely differently; they see something in it or think about something you’ve never intended. I’ve never thought that you would experience it differently if you just listen to the music or you see the performance where it belongs together. For you seeing first the performance, then listening to it makes you think about it in a different way – that’s what for me has always been the beauty of art, that it activates your thinking in general, seeing something just randomly or listening to something that inspires you to do or see something in a totally different way.
I don’t even know which one was more powerful for me – whether the whole performance, the whole show, or listening to the album itself, let’s say, at home when your mind is wandering somewhere. For me those are completely different experiences, and, to be honest, if I hadn’t known it was your album, I would have even thought that those are completely different music pieces…
The album has seven parts, but the performance is a whole; it has more rhythm to it, because there’s also a visual side. There are accents which represent the synchronized moments in sound, and the performance is for sure more… maybe not powerful but, as it uses both senses, it has more impact on you. And you’re also in a space which it’s super loud, so it’s just a different experience than when you listen to it at home with your headphones on.
The performance was so strong that even when I listen to the album at home I have this flashback with your visuals there, but still everyone, I think, while listening to the music have different images in their head – something that connects with their experience.
That’s exactly what we said before about science and language. How you grew up shapes you to have certain flashbacks caused by a certain kind of melody or a certain sound, and to another person it can be totally different.
There is this discussion going on whether it is the language that we grew up in that shapes our ourselves or whether we shape the language we speak in. I’ve noticed something, especially with people who know more than one or two languages, that they have different personalities depending on the language they speak in which is really fascinating.
Well, I think it’s true, as it’s both – the language shapes us, because when you’re a baby and you just hear something, you can’t do anything about it, so the language shapes your way how to think about the world; but once you’re old enough and you learn a different language or you develop your own idea about something, then you can force yourself or shift your brain to think about it in a different way. Language and thinking are always fluid processes – the word from your childhood can have a different meaning now that it had back then; it changed in between 10 years. Maybe it can have a totally different meaning when you’re 60 or so.
… and the amount of vocabulary available in a particular language changes; we add some new words, we forget about some other words, so it’s fluid again…
… but I’m not sure if – maybe with some people it’s the case – you become a different person when you talk in another language. I believe you still stay the same, the inner voice I think stays the same, and what is the language of your inner voice is for sure not fixed to letters or to signs. That’s something really individualistic. Maybe that’s what people call a soul. It’s something we all have and we all know we have, but we can’t really talk about it because we don’t have words for it.
Well, I have a lot to think about after our conversation because you’ve put so many interesting ideas in my head that I need to digest. Talking about language with you was like, wow, especially that I’ve never thought you can express music and create sheer beauty out of technology. It was also something really amazing for me.
It’s interesting, because you come from the field of linguistics, so you know a lot about it, and I don’t, at all; I’m an engineer and come from the field of technology – that’s what I studied. But that’s also what I found out in all the years I’ve been making art – that you somehow find yourself being busy with the topic just because you’re so interested in something, and then you delve so deep in the topic you were never intended to be occupied with. You just land there out of curiosity. There’s a work from me called Soft Takeover from 3, 4 years ago. It’s also an abstract machine, but verbalizing in a physical way. It turns out that there’s an Italian artist in the 60. who did exactly the same aesthetics. His name is Enrico Castellani. 40-50 years ago he worked on sculptures that looked exactly like mine, but his work was not about language at all. It’s funny that sometimes you arrive at exactly the same or similar physical form without knowing about the predecessor; also, with a totally different theoretical background you arrive at the exact same aesthetic idea.
Soft Takeover, 2019-2022 — Kinetic sculpture, dim. 75 x 100cm
Trained artificial neural network (Abstract Language Model), actuators, stretch fabric, canvas frame
That’s not pure coincidence; this is the way the human brain works. This is the only conclusion I can draw from it because I don’t think that’s a coincidence… really, that wouldn’t be possible.
Do you think this is hard-wired in our brains that somehow, when we think about certain stuff, it leads to a similar kind of expression?
That’s my initial idea. I mean, let’s say your brain was triggered by something and it moved in a different way towards the same point. And his brain was also triggered by something else, but in a different meander he also arrived at the same point. We will never know perhaps…
Exactly, but that’s maybe also the beauty of it…
The interview was held in October 2023.