My concepts
Empathize, Define
We spent a lot of time brainstorming possible concepts as a team, but some individual research as well. For me, that included technical research about machine & deep learning. The research is still in progress and more questions will be added depending on what concept we end up working on.
For now, I worked on 3 research questions which helped me learn more about how the concepts we have come up with could be implemented. As part of this research, I tried initializing a Unity project with Unity's Machine Learning package, but so far I haven't succeeded in installing the prerequisites on my laptop since Python is giving me some trouble. However, with more research and possibly some backtracking/trying on my PC at home, I'm quite confident I can get it working. After that's been accomplished, I will start working on prototyping and deepening my knowledge on the field of machine learning.
After the meeting with our clients last week on Wednesday the 27th of September, I looked at ways to implement the ChatGPT API into Unity, as that might be part of our concept if it works well.
If I remember it correctly (I forgot to add the concepts during the relevant weeks), we had a set of "base concepts/keywords", and we all iterated on that and created our own interpretation. So, the following list are my interpretations of the concepts.
Escape room
Taken prisoner by an AI. Convince the AI (so develops a personality) to release you, the humans. Lights & sounds for immersion. Audience members can pick up/touch objects in the room, possibly as part of a puzzle. => _Wearables
Physical imprisonment? Or only “guide the way”? Hologram installation for the AI to feel like you are actually talking to someone instead of at a screen. Structure inside structure, like “layers” in a prison.
For choosing an option: have at least 2 “sides” the audience can stand. If everyone is wearing a bracelet with NFC, every “option” side should have an interactable object that can scan the NFC tag so it can count how many people and who chose the option. So, by scanning the wearables we don’t need live tracking of people/wearables. Since that wearable probably wouldn’t have a battery so it can’t be “running” a program that allows for live tracking. So, scanning the NFC tag allows for choosing options anyway.
X = people standing in an area where they scan their NFC tag to show their choice.
Fully agree | Somewhat agree | neutral | Somewhat disagree | Fully disagree |
xx | x | xx |
| xxxxx |
"Telltale"-style for the play
AI personality & storyline shaped by audience choices. Several possible endings.
Following an R.U.R extended character in the play. Audience makes choices
-
Audience selection
Randomly select someone in the audience, through app (spotlight on audience/phone/wearable lighting up. selected person makes a choice/asks question.
Going through all users during a selected period of time and gradually slowing down, until the selection lands on one person is a great way to build suspense and increase immersiveness. However, I’m not sure how this concept would work on its own. Rather, it could be a subconcept, as a part of something larger.
Deepfake(s)
Several installations placed around the room. Free roaming.
Installation description:
A large (60-100inch) TV placed on a platform.
A construction of plexiglass. Either a sphere (if possible to bend the plastic that way) or in a pyramid form. This allows for a holographic display. Many particles?
Possibly combine this with deepfakes? Scan the face (and body?) of user, use that data to construct a 3D mesh which is manipulated by a deepfake algorithm. User can become the visualization of the AI (idk how possible that is)
Combine with “AI personality shaped by audience choices”?
Wearables
Everyone gets a wearable (bracelet?). Interact with locations/objects around the room (NFC tag). Combined with other concepts, this concept is not standalone. Wearables are more like “part of” a bigger idea. Bracelet could have some light, vibration or sound element (should we consider accessibility for blind/deaf people?). If those elements are used, it could have some kind of connection to the AI (so the computer on which the AI is running).
MetaHumans
Prototype, Ideate
We also made a test-MetaHuman animated video of one of our team members. We're quite happy with how it looks, and with more post-recording polishing and better lighting in the source video, it can look quite realistic.
In the first video below, you can see the source video file as well as a default face model being animated, which follows the facial movements detected in the source file.
In the next video, the face motions are transferred to the actual mesh. The most work for this was waiting for the tool to do its job (training the machine-learning algorithm of MetaHumans to learn how to do the animations), and waiting for the animations to be transferred to the model.
Machine & Deep Learning Research
Empathize
I didn't answer all the questions, since brainstorming was a priority at the time and after that, executing on the decided concept.
Technical research questions
What’s the difference between deep learning and machine learning?
How can neural networks be implemented into Unity?
How can AI be combined with advanced technology such as facial recognition (or at least detection)?
How can AI be combined with technology such as audio input, and possibly translating the given input into a specific language?
What are the technological limitations of introducting AI into our society?
How can a prompt be summarized, retaining all the important information but removing all unnecessary characters?
What’s the difference between machine learning and deep learning?
First, the meaning of “machine learning” as well as the term “neural networks” should be defined before moving on to “deep learning”.
Machine learning
A part of the Artificial Intelligence field. It concentrates on using so-called “big data”, which is used to train an algorithm so that it can mimic human behaviour to a certain degree of perfection. Machine learning can be seen of the parent of all the things that are to be discussed next. Everything derives from machine learning.
Neural networks
In Figure 1 “Neural network depiction”, you see several layers, which are categorized as Input, Hidden or Output. Directly or indirectly, all nodes in every layer are connected to each other. This means that small changes in one input node can have an enormous impact on the final output of the neural network.
The specific neural network shown in Figure 1 “Neural network depiction” is called a “Feed forward neural network” (as found on (Turing, n.d.)), as the layers through which the input data is passed is a one-way street. This means that once the input layer stops receiving input from its source (could be a sensor, a human giving commands, etc), the neural network will stop working after the stream of data has stopped.
However, every neural network can be turned into a looping, self-sustaining network by implementing backpropagation. This means that the output data is fed back into the input layer.
The more hidden layers a neural network has, the more complex it becomes and the more accurate the output data will be. Every subsequent hidden layer will improve the output accuracy of the previous layer, whether that is a categorization, prediction or some other type of decision.
Every layer has a “weight” assigned to it, which is multiplied with the data received on that layer. This way, some circumstances are prioritized over others.
Deep learning
The only difference from machine learning is the way each network learns, the amount of layers the network has, and how much data they use. Therefore, it makes sense to say that neural networks that can be classified as a “Deep Learning Neural Network” (DNN) are also smarter than a “Machine Learning Neural Network” (MNN), simply because the data of a DNN is more complex than that of an MNN.
How can neural networks be implemented into Unity?
There are not a lot of official resources from Unity about neural networks/machine learning, besides the package they released (Unity, n.d.). However, there are a lot of blog posts, scientific papers and tutorials breaking down how neural networks “think” and how they are implemented.
Some software/libraries/API:
PyTorch
TensorFlow
What are the common machine learning algorithms to train an AI?
As you can see in Figure 2 “Types of machine learning” (Bhatt, 2018), there are 3 “main categories”.
REWRITE LATER
Many different algorithms in the 3 categories
Supervised learning is often used for classification of data.
Linear & Logistic Regression, Support Vector Machines, K Nearest Neighbours, Decision Tree
Unsupervised learning
K Means Clustering
Hierarchical Clustering
DBSCAN
Principal Component Analysis
Reinforcement learning
Q-Learning
SARSA | State Action Reward State Action
Monte Carlo
Deep Q Network
PPO | Proximal Policy Optimization
How are neural networks taught to “understand” human language?
References
Banoula, M. (2023, 02 22). What Is Q-Learning: The Best Guide To Understand Q-Learning. Retrieved from www.simplilearn.com: https://www.simplilearn.com/tutorials/machine-learning-tutorial/what-is-q-learning#what_is_qlearning
Bhatt, S. (2018, 03 19). Reinforcement Learning 101. Retrieved from towardsdatascience.com: https://towardsdatascience.com/reinforcement-learning-101-e24b50e1d292
IBM Data and AI Team . (2023, 07 6). AI vs. Machine Learning vs. Deep Learning vs. Neural Networks: What’s the difference? Retrieved from www.ibm.com: https://www.ibm.com/blog/ai-vs-machine-learning-vs-deep-learning-vs-neural-networks/
IBM Data and AI Team. (n.d.). What is a neural network? Retrieved from www.ibm.com: https://www.ibm.com/topics/neural-networks
IBM Data and AI Team. (n.d.). What is deep learning? Retrieved from www.ibm.com: https://www.ibm.com/topics/deep-learning
IBM Data and AI Team. (n.d.). What is machine learning? Retrieved from www.ibm.com: https://www.ibm.com/topics/machine-learning
Lague, S. (2022, 08 12). How to Create a Neural Network (and Train it to Identify Doodles). Retrieved from www.youtube.com: https://www.youtube.com/watch?v=hfMk-kjRv4c&t=14s
Nielsen, M. A. (2019). Neural Networks and Deep Learning. Determination Press.
Turing. (n.d.). Understanding Feed Forward Neural Networks With Maths and Statistics. Retrieved from www.turing.com: https://www.turing.com/kb/mathematical-formulation-of-feed-forward-neural-network#why-are-neural-networks-used?
Unity. (n.d.). UNITY MACHINE LEARNING AGENTS. Retrieved from unity.com: https://unity.com/products/machine-learning-agents
Comments