Tags: Empathize, Define Foreword: This post will include my progress over the past few weeks. Some weeks were a bit less productive than others; that's why I felt it would be better to create a longer post which should include The original concepts and my research for the projects.
In the beginning we have checked multiple AI installations, so we can have an idea of how an installation that focuses on technology, with the AI topic in mind, looks like.
We started by creating a Miro Board and putting random ideas there. After everyone put some idea inside the board, we started making connections. There were a few things that we liked, such as Using Wearables (bracelets most probably) to make the play more interactive (having the audience interact with the installation), using Holograms (or faking the idea of a Hologram with a bunch of mirrors and glass panels, this idea was later dropped due to the difficulty of building something that would look good from multiple angles.). When it comes to ideas, we had a few including:
Narrative Focused idea (similar to games created by TellTale) - in this idea the audience is spectating a story which they can influence. At some points during the story, the audience will have to choose between 2 options. Based on the options the story would change.
Escape Room - not much thought went into this, we just liked the idea of building an Escape Room.
There were more small ideas but we agreed on 3 main concepts, which we later on presented to the client:
Narrative/Story Focused Idea - The concept consists of a full room installation that showcases an interactive story being told by the AI to the audience/ story in which the AI is the Main Character (max. 30 people) that’s accompanied by visuals (projections, lights, small holograms) and audio (music and sound effects). The story (possible prequel, before the robots killed all humans) takes place inside the R.U.R universe, but with new characters.
Visualization of the Concept done by one of our Artists
Sandbox "Inside the AI" - This concept consists of multiple installations inside one big room, all of them AI themed. Here we had a few idea for small installations:
- an installation in which people can make a digital version of themselves, using Unreal Engine MetaHumans (more about this later)
- conversation with an AI in which people can talk with an actual AI; we had this idea after seeing Turtle Talk With Crush by Disney.
- a hologram game of chess in which a single audience member has a game of chess against an AI.
Visualization of the concept made by one of our artists
AI Generated World and Story - This concept is a bit more vague. We started with the questions we received from the theater. "What does it mean to be human?", "What would the world look like is machine took over?" and so on. We wanted to find a way to answer this questions so we came up with 2 ideas for this concept.
In the first one, the audience explored difference scenarios and explore the backstory
The audience will have a dialogue with an AI and the world around them change based of their answers, questions, etc.
Visualization of the concept made by one of our artists
This concept is the most vague one. We could have developed it in multiple ways. However, we had to drop it due to technological difficulties caused by the hologram idea in the middle of the room; we could only create the hologram effect from a single perspective, the effect would break if the audience would want to look at the hologram either from close or from a different angle.
After presenting the 3 concepts to the client, we had another idea; with the help of the other people in the studio. Our final idea was combination of the first and third idea.
Story & Theme play | The American Courtroom - Room of Justice - This idea consists of a debate between Humans and AI. After AI evolved too much, it realised that the humans are no longer needed and it wants to exterminate them. However, it gives humaniy a final chance. The audience will be part of a jury that has to protect humanity, while giving proper arguments to why humans should survive and co-exist with the AIs.
Visualization of the concept made by one of our artists
This concept used a lot of technology that we are not that familiar with, so the next part was researching the tools, both hardware and software.
The Hardware
At the moment of writing this post, we know for sure that we will need the following equipment:
A video projector (which will project the AI with the background and jury)
A few microphone (so the audience can talk with the AI too, one for the Human Defendant)
Wearable (most probably only Bracelets, in case the audience has to vote between 2 options)
A server (the bracelets will be connected to a server)
A computer which will have to run all the systems (the project will be done in Unreal Engine 5 and will connect everything inside it; it will pull information from the server, which takes data from the wearables; it will connect the lighting software to UE5 too, so we can control the lights inside UE5, not using a different controller )
Speakers (we will need multiple different speaker and connected to an Audio Interface, which should give us full control over the all the Speakers; also it would be perfect if it can be integrated inside UE5)
An iPhone 12 (or newer) - This will be used for motion capture
The Software
Unreal Engine 5 - This will be the "brain" of the entire installation, since everything will be integrated inside it. As most of the stuff here, it will have a fee. However, the fee is not really known yet. Epic wants to introduce a fee to non-games products but we don't know if our project goes into the category.
ChatGPT API - We will have to use the ChatGPT API inside UE5 and each prompt will cost a few "tokens"
DALL-E 3.0 - Same as ChatGPT, it will be used inside UE5 and will have some sort of fee.
Houdini - We will have to use Houdini for the background.
SpeechToText and TextToSpeech - We will have to use it to properly have a dialogue with an AI. This, like most of the stuff has a fee per prompt.
Research - Microphone
Here we have 2 options:
Option 1: Each person has its own microphone. This I think will be more costly, depending of the number of people in the audience but also will cause some technical difficulties, since we will need to find a way to connect 20+ microphones in a single PC. Even with an audio interface, this will be hard and costly, there's just so many inputs.
Option 2: We will use 4 shotgun microphones. These microphones are created for long range and only capture sound coming from the front. We can put these microphones in 4 corners, surrounding the audience, we can also hide them nicely.
We did some testing with a Rode Shotgun Microphone connected to a Zoom Recorder, which is a recorder but we used it as an Audio Interface, since the Rode Mic has an XLR cable that cannot be connected to the laptop directly. So we connected the Rode Mic to the Zoom Recorder, and then the Zoom Recorder to the Laptop via USB.
We did some testin to simulate using the microphone in a real scenario. One of us was speaking 1 meter away from the Mic and then we started moving away as much as possible from the said person. We ended up around 8-10 meters away and we stopped there, we did not have a room bigger than that. The blackbox inside the Theater is around 12x13. If we include our setdressing, it will become close to 10x10, so 4 microphones should be enough.
Research - Bracelet
The Bracelet will have to be made by us, since we cannot find anything on the market that meets our requirements. For this bracelet we need a few components:
An Arduino Nano 33 IoT- this is just a very small Arduino that has all the components necessary for our concept. It has both a gyroscope and an accelerator, and most importantly it has a Wi-Fi module.
A power source. This is still a bit unclear how we're supposed to do it. Most probably we will have to use some coin batteries
A chassis. We need to put everything in a neat chassis, so it also looks cool. Maybe add some LED if we have enough time.
I already got the Arduino and programmed it. Now it can detect if a person raises their hand and for how long, the next step would be to connect it to a server and send information to it.
Research - Server
The server will be used to connect all the bracelets together. It will get data from them and send it to Unreal. We do not need a very powerful server, a Raspberry Pi would be enough for this task. At the moment, our Raspberry Pi is a local server. The next step would be to connect it to the bracelets and UE.
Research - MetaHuman
With Unreal Engine 5, Epic released a new tool called MetaHuman, which can create digital models of human. We can use it to create our AIs but also animate them. MetaHuman can also be used for animations, using a motion capture suit or AR camera. Initially we tried to use a DSLR to capture ourselves and import it into UE5, but it failed. After more intensive research, we found out that we can only use MetaHuman with an Iphone 12 or newer; this might be because of the sensor on the newer iPhones.
We rented an iPhone from Saxion and tried testing it, which was a complete success.
Research ChatGPT
Researched story generation using ChatGPT. Our idea was to have a fixed beginning and a fixed ending of the narrative, or at least a fixed base of them. ChatGPT would just create the middle part of the story and some elements which are not part of the "fixed" beginning and end. I basically gave ChatGPT a beginning of a story with very fixed elements and flexible elements, and also the ending of that story, with fixed and flexible elements. The fixed elements cannot be change under any condition, however the flexible ones can be changed; for example killing a character, changing a bit the story for cohesion. After that, I just told ChatGPT to generate me the middle part of the story, with multiple options, each different that branches to other options, but follows the pattern imposed by me.
Comments