To get started on our wearable prototype, we borrowed a Raspberry Pi 2 from one of our CMGT teachers. This mini-computer will be used as a server, to which the data from all wearable devices is sent. The built Unreal Engine application will then communicate with the server to determine the proper actions to take in the application.
The networking side
Prototype
Setting up a server using Raspberry Pi falls under my learning goal that targets input/output devices.
It was my first time working with a Raspberry Pi, and getting involved with networking as well. So a few days were spent on getting familiar with the Raspberry Pi OS (based on Debian Linux) terminal commands, the terminal itself, and the essentials of networking.
This included solving a DNS (= Domain Name System) connection issue first, otherwise the concept could of course not be executed. After research, I found that any network settings should be configured through the DHCP (Dynamic Host Configuration Protocol) settings.
I was able to fix all network issues by setting a static IP address, rather than a dynamic one and using an external WiFi dongle. After this, we could get started on converting the Raspberry Pi to function as a server. While I was doing that setup, Daniel was prototyping an Arduino gyroscope & accelerometer.
How will the Unreal Engine - RPi - Wearable communication work?
The Unreal Engine application will communicate with the Raspberry Pi, which will exact as the server. The server receives signals from all available wearables (so all the Arduino's that are in WiFi range). These signals wil represent a raised wearable. If no signal was received, the wearable was not raised and should not be sending a signal.
Depending on the battery capacity, which will probably not be a lot since the wearable is quite small, we can decide how many times per second an update loop is performed on the Arduino wearable. Reducing the "framerate" should cause the battery usage to decrease as well. Only sending out a signal when the wearable is raised is another reason: unnecessarily sending a signal will only drain the battery too fast. The wearables should last long enough for at least one full interactive experience duration, around 45 minutes.
Re-learning Unreal Engine 5
Prototype
The last time I touched Unreal Engine 5 was during a Personal Portfolio course during the 2nd year., which is almost 2 years ago. That means most of the skills I learned have dimished. I have been trying to get more comfortable in Unreal Engine again these past weeks. Since my workflow is entirely based on how Unity works, it takes a lot of effort to understand how, even the simplest things, are done in Unreal Engine.
Prototyping ChatGPT & DALL-E
Prototype, Ideate
For our concept, it is necessary to use the LLM (Large Language Model) ChatGPT, and the image generation model DALL-E (preferably DALL-E 3). It is possible to use their functionality in Unreal Engine, as there are a few plugins available made specifically for the engine. However, there are some issues with that. None of the plugins are made by OpenAI itself, so it is always a 3rd party that the application will be dependant on. Often, these plugins have not been updated in months or even years so if OpenAI changes the endpoints of their API, there is a chance the plugin stops working and break the application. That is however a risk we might have to take to get an MVP (Minimum Viable Product) off the ground. I have compared several Unreal Engine plugins to determine which fits our project best.
The plugin I have tried first works as it should. However, when it comes to generating images it seems DALL-E 2 is used rather than DALL-E 3. When it comes to the conversational capacities of the LLM, it is also not as fast as it should be for our use case. However, we might be able to make that work, as we can explain it like "the AI is thinking carefully about what to answer" so it fits the concept a bit more.
The output images are of lesser quality than those being generated in browser-versions of DALL-E 3, with the same prompts. The generation speed of the images is also questionable.
The following 2 images are made with the exact same prompt:
An icy landscape under a starlit sky, where a magnificent frozen waterfall flows over a cliff. In the center of the scene, a fire burns bright, its flames seemingly frozen in place, casting a shimmering glow on the surrounding ice and snow.
The prompt was taken from OpenAI's research paper (https://cdn.openai.com/papers/dall-e-3.pdf) about DALL-E 3, to ensure the quality of the prompt is high, and with that the quality of the output image. The first image is generated by the mentioned plugin in Unreal Engine, and the second is made by Bing Creator.
If you compare the 2 images, you can notice a significant difference in quality. The first image looks more pixel-y, much less detailed and a bit more abstract. DALL-E 2 being used is not considered a dealbreaker to me, since we can still generate visuals. But we should still consider other options.
The next plugin I tested was this one, which is the plugin that was referred to by the documentation of OpenAI. This one has not been updated in well over 5 months. At the time of writing, there have been created some threads on Github, asking when the plugin will be updated for Unreal Engine 5.3. Hopefully, that includes an endpoint that targets DALL-E 3.
As you can see, the images do not match the given prompt at all. Again, this is probably an issue caused by the plugin using DALL-E 2, as I suspect. My current hope is set on the plugin being updated any time now, as soon as possible. This would hopefully fix this issue. I have yet to figure out how to implement a conversation tool with the plugin. The available documentation is pretty barebones. It only consists of some blueprint screenshots and a tutorial video, which use nodes that are not available in the plugin version I am using (5.1.1), and the screenshots don't show the entire editor setup. So it is often guessing work what the developer did to get something working.
If none of these plugins work properly, it could be worth considering making our own implemetation of the API in Unreal Engine. But also for that, there are dangers to consider. As I mentioned, I don't have a lot of experience in Unreal Engine, so developing a plugin might be too much for 1 developer, especially since I am also responsible for setting up the server and get it to communicate with at least the Unreal Engine side. A possible solution for this would be to try a course on how to create that specific plugin, using this resource.
Secondly, all documentation provided by OpenAI is written in Python, or Node.js. I am not sure how the conversion from either of those languages to Unreal Engine C++ would work.
And finally, at the moment we are already in week 8. The deadline for the final prototype might be approaching too fast to develop our own plugin.
Comments