TouchDesigner & Unreal: Interactive Controllers

Course details

As brands, studios, and performers look to make a statement in a media-saturated world, a range of multi-faceted design firms are creating fascinating work beyond the constraints of traditional screens and linear timelines. In this course, learn how to fuse multiple disciplines using real-time graphics software to create engaging content for deployment beyond the traditional screen.

This course focuses on using interactive hardware controllers to drive real-time 3D scenes in TouchDesigner and Unreal Engine. In TouchDesigner, look at a MIDI controller with sliders that you can use to drive the creation of effects in a real-time rendering system. Next, learn about using TouchOSC to build a custom UI that you can transfer to an iPad that sends OSC messages to TouchDesigner. Then, learn how to use those messages to move particles and objects around on screen. To wrap up, learn how to get data off of a Kinect sensor into TouchDesigner so you can control that same particle and object movement. The course also covers MIDI, OSC, and Kinect as they apply to Unreal Engine.

Learning objectives
Using TouchOSC to build a custom UI
Transferring a custom UI to an iPad
Controlling objects with OSC
Setting up Kinect
Getting data off of a Kinect sensor and into TouchDesigner
Using OSC messages to move particles and objects around
Installing VS and downloading the plug-in
Creating a C++ Unreal project and compiling a plugin
Installing the Kinect 4 Unreal plugin

Concepts

0. Introduction

01 - Welcome
02 - How to use the exercise files

1. TouchDesigner - MIDI

03 - MIDI device mapper
04 - MIDI input component
05 - Build effects UI
06 - Connect MIDI to effects UI
07 - Effects 1 - Blur
08 - Effects 2 - RGB
09 - Effects 3 - Feedback, part 1
10 - Effects 3 - Feedback, part 2
11 - Effects 3 - Feedback, part 3

2. TouchDesigner - OSC

12 - OSC overview
13 - TouchOSC layout
14 - OSC input component
15 - Controlling objects with OSC
16 - Particles - Instancing, part 1
17 - Particles - Instancing, part 2
18 - Particles - Instancing, part 3
19 - Particles - Instancing, part 4
20 - Particles - GPU, part 1
21 - Particles - GPU, part 2
22 - Particle type switching with OSC

3. TouchDesigner - Kinect

23 - Kinect setup
24 - Kinect input component
25 - Connecting Kinect data to network, part 1
26 - Connect Kinect data to network, part 2

4. Unreal Engine - MIDI

27 - Blueprint 1 - Listing MIDI devices
28 - Blueprint 2 - Printing out incoming data
29 - Post-process material
30 - Blueprint 3 - Data to parameter 1
31 - Blueprint 4 - Data to parameters 2 & 3

5. Unreal Engine - OSC

32 - Install VS and download the plugin
33 - Create a C++ Unreal projects and compile a plugin
34 - TouchOSC - UI Layout
35 - Blueprint 1 - OSC input
36 - Blueprint 2 - Moving geometry
37 - Blueprint 3 - Switching geometry
38 - Blueprint 4 - Rotating geometry
39 - Blueprint 5 - Moving particles

6. Unreal Engine - Kinect

40 - Installing the Kinect 4 Unreal plug-in
41 - Blueprint 1 - Kinect input
42 - Blueprint 2 - Object movement
43 - Blueprint 3 - Particle movement
44 - Blueprint 4 - Adding in right hand

Conclusion

45 - Next steps