unity.hfbk.net

99. Unity & Arduino Interaction

A interesting way to use Unity is to make it interact with physical Hardware. This could of course mean existing solutions like:

Hacking together something from these things can already get you very far, but sometimes this might not be enough – e.g. if you want to control physical things from within Unity. If this is the case, using a microcontroller like a Arduino, Teensy, ESP32 (for wireless) or similar might be a good approach.

Any physical interaction can be seen as either Input or Output. Sensors, Buttons, Dials, Sliders, Encoders, Switches, Microphones, etc could become Inputs, while Relais, Transistors, Servos, LEDs Motors, Buzzers, Screens, etc could become Outputs.

Serial Communication

This means we have to have some way to send messages from Arduino to Unity and vice versa. The way this is usually done is so called Serial Communication. Serial means, the messages are sent one after another (As opposed to say in parallel). What is being sent is usually just ASCII-Text.

Communication between Unity and Arduino

This means for example that we could periodically get sensor data from the Arduino to control stuff inside Unity, while simultaneously having our Arduino react to messages that are sent from Unity. The messages Unity receives can be used to control stuff in your scene (e.g. you could use a light sensor to change the light inside the scene). The messages Arduino receives can be used to switch on/off things, control motors, blink LEDs, show stuff on tiny displays etc.

Physical interfacing is quite intersting from an art perspective, because it allows you to influence as which medial dispositif the viewers will perceive themselves: is it passive television watching, listening to a radio or a record, is it a computer game, a arcarde game or rather just magic?

Setup and Test Arduino Software

  1. Install the Arduino IDE
  2. Connect your Arduino to Computer via USB-Cable to your computer
  3. Check if the Arduino works with the blink example:
    1. Open File > Examples > 01.Basics > Blink
    2. Try to flash it onto the Arduino using the round Arrow button
    3. If it doesn't work, you probably need to change the Arduino Model or the COM-Port. If you use cheap Arduino Nanos from China, you might also have to set Tools > Processor [...] to Atmega 328P (Old Bootloader)
  4. Copy the code below into the Arduino Editor and flash it onto the Arduino

// A variable to keep track of the last time
unsigned long last_time = 0;

void setup() {
    // Set the built in LED pin to Output Mode
    pinMode(LED_BUILTIN, OUTPUT);

    // Start serial communication with a BAUD-Rate of 9600
    Serial.begin(9600);
}

void loop() {
    // Print a heartbeat every 2000 seconds (by comparing if 2000ms
    // have passed since the value stored in the last_time variable)
    if (millis() > last_time + 2000)
    {
        // Send a message to Unity. If you want to use this message
        // to control things you will have to change the Unity Script
        // (e.g. SerialController.cs) If you are reading out a sensor
        // and want Unity to react to it, consider sending a message from
        // Unity and sending your data in a response instead
        Serial.println("Arduino is alive!!");

        // Store the current time in the variable
        last_time = millis();
    }

    // Send some message when the given letter is received
    switch (Serial.read())
    {
        case 'A':
            // Send a response to Unity
            Serial.println("Arduino received letter A: switch on LED.");
            // Switch the LED
            digitalWrite(LED_BUILTIN, HIGH);
            break;
        case 'Z':
            // Send a response to Unity
            Serial.println("Arduino received letter Z: switch off LED.");
            // Switch the LED
            digitalWrite(LED_BUILTIN, LOW);
            break;
    }
}

Setup Unity

  1. Install the Ardity Addon from the Asset-Store

  2. Make sure that the Option API Compability Layer under Edit > Project Settings... > Player > Other Settings is not set to .NET Standard 2.0. At the point of writing the correct Option was .NET 4.x but this could change with new versions of Unity:

Set API Compability Layer to .NET 4.x

In your Project Window check out Assets > Ardity > Scenes to see how the thing is set up. You could use the Prefabs in Assets > Ardity > Prefabs to allow your Unity Scene to interact with one or more Arduino/Teensy/etc via USB connection. A very similar approach would potentially also work wireless with a ESP32.

PLEASE NOTE: make sure that the Object with the Serial Controller-Script on it (e.g. the Serial Controller-Prefab) has the right Port Name selected. If you are unsure where to get this value, it is the one listed in the Arduino Editor under Tools > Port ...

These Ports are named as follows (replace * with a number):

Example Project

If you are curious how to achieve bidirectional communication between Arduino and Unity have a look at this Example Scene