Empat Blok is an interactive audio-visual installation which explores issues of wealth inequality in cities. The installation examines not only the divide itself but also how we perceive inequality and how governments attempt to obscure the problem to outsiders and to their own residents. Division between socio-economic groups is a problem which exists universally, and is a particularly prevalent occurrence for Malaysia. When in Kuala Lumpur it is often possible to see upmarket developments contrasted against low-income neighbourhoods, the close proximity between different socioeconomic groups demonstrating the severity of inequality. Using a motion sensing camera the viewer is shown depictions from different perspectives dependent on how close they are to the tower and which side they stand on.
The focus point of the installation is a sculpture of a skyscraper. The sculpture consists of four stacked white cubes onto which depictions of different socioeconomic groups are projection mapped. The different levels of the tower represent the different groups of people, whilst the different sides of the tower explore differences in perception and reality for each level. The content of the visuals is based around several representative themes which vary with different levels; these themes include food, transport, living conditions and lighting. The levels are represented visually through the content of the projection but also through visual design and effects such as by utilising clear and bright colours at the top of the tower and darker, muted tones as well as distorted and faded imagery at the bottom.
The installation incorporates interactivity through the use of a motion sensor camera which controls what audio and vision is presented to the viewer based on their position in the room. Standing against the back wall at the furthest point from the tower the viewer can see only the windows of the highest level, demonstrating the perspective often presented by media and by governments. As the viewer moves closer to the tower the windows of lower levels sequentially appear to the viewer, with the bottom level only appearing when the viewer is at the closest point to the tower. The two different sides of the tower are revealed as the viewer moves left or right across the field as well as affecting the projected film to represent differences in perception and reality. The audio also represents the subject interactively, with the audio changing based on the movements and position of the viewer. The sound design constitutes one piece which can represent the spectrum of wealth distribution, changing mood and timbre as the viewer moves around the space. In this sense it could be considered that the sound design is a performance of viewer, and the music is therefore an indeterminate piece.
In our goal to build metaphor though distance we believed a motion sensor or ambient triggers would be required in the installation in order to bring the users perspective into the feedback loop of interaction as they move around the tower and throughout the space. The Microsoft Kinect, a depth sensing camera was chosen as it’s been a proven tool in comparative interactive installations. To output user’s positions from the Kinect as a max-min float value, Cycling 74’s Max was used to grab skeleton position data and send this to both Ableton and Touch Designer over Open Sound Control, a network protocol allowing multiple computers to work with the same data set and in theory distributing computing resources for intensive projects. In this case a Mac was used for visuals and a Windows machine was required to interface with the Kinect SDK.
The interaction loop being initiated by the users position in the scene, and furthered by the audio and visual responses. Audio being built in Ableton Live as 4 channels looping for the durations of the installation, each channel being panned or changing output levels in relation to the user’s position in the space. The sound channel change represent the user’s observations of class hierarchies in relationship to the tower. The stereo speakers set beside the sides of the tower are panned from between themselves to follow the user throughout the space.
As for the visuals we collected film throughout Kuala Lumpur and edited them into to animated loops representing each level on the tower. Traveling from lower income areas then later into high end shopping malls capturing objects as they might be composited into these films. These were edited into 4 sequences, each exemplifying elements of perception in the class hierarchy. Using a 1:1 aspect ratio at 12 frames per second, each window was rendered in Premiere Pro producing an aesthetic based on glitch and composure each concentrated on either axis.
These films were composed in Touchdesigner animated and changed based on the distance data sent from MAX. Values from 0 to 1 were re-clamped into 0 to 40, where math functions define 4 zones that change the brightness of these window animations in relation to the user’s distance from the tower. The highest window continually illuminated as a call to action in the space. The background of the tower are randomly deforming textures and footage of vegetation composited into the tower, subtly controlled by the user’s motions.
Beyond this though researching projection mapping techniques, Katan mapper that is built into Touchdesigner was used to deform the 2D video into an frame mapped for projection onto the towers two illuminated sides. Using polygon deformation clean lines were mapped onto the tower structure, with contrast and brightness settings on the projector stopping light leaking onto walls and other’s projects in the gallery space. After some troubleshooting with Katan mapping in Touchdesigner it became flexible to setup and respond to changes.
Team Members and Skills
Trent, Mike, Joe, and I were blessed with a near to perfect balance regarding skills that we were able to use to contribute to the project. The team was split into two groups, audio (Joe and I) and visual (Mike and Trent), and then under these two broad headings, jobs were further subdivided. Whilst we all contributed and helped out each other in some ways, and made big decisions as an entire group this way of dividing the work made us much more productive as a team. From the moment we formed the group we were all aware of each other strengths and we planned out the idea for our piece in such a way that it took advantage of all the skills we have.
Mike and Trent worked together to do the visuals in the piece. This included location filming, video editing, and projection mapping. The local filming was done in part by all of us. Mike and Trent did the majority of the actual camera work, but the four of us spent the whole day going to spots all over K.L looking for footage to include and all of us contributed ideas of what footage we should be trying to get and why it was relevant. We first met Mike in Sabung, where he lives and got footage all around his area, which was a good spot to collect footage for our middle two floors. Next we took the train to K.L city centre, and got footage for the top and bottom floors respectively. Petaling St was a good area to show examples of poverty, and a large, high-end shopping centre near the K.L towers gave us plenty to shoot for the top floors.
After the day of filming, Mike began to edit the clips into four short loops for the respective floors. He used techniques like double exposure to transition between each clip within the loop, giving a more dream-like, abstract style as opposed to using hard cuts. His use of filters on the footage also helped style the loops in this way. Once the clips were ready Mike sent them over to Trent, and he dropped them into touch design so he could begin creating effects for the model tower, and start to work out the placement and the projection mapping. Trent used footage from the location recording to create a surface pattern for the tower, and then dropped the loops onto their respective floors and sides.
After he had put together the basic touch design file, we were ready to put the tower together, which was a simple as piling up a few blocks that were left over from the old exhibition in the E-Gallery, and positioning the projector correctly. We ended up using a tripod, since it allowed the projector to tilt sideways, covering the full height of the tower. Next came a few days of troubleshooting, and getting the Kinect to interact with the visuals, which meant sending messages over a network, from Joe’s computer, to Trent’s. Ironing out a few issues took time and was a collective effort from Joe and Trent mostly, with me being the test dummy for the Kinect. Once this was running as smoothly as we could get it, the visual side of the work was done.
Audio for the project was left to Joe and I. We both have a decent knowledge of Ableton, so it was the obvious choice of DAW. Also its easy interface into MAX and MAX for Live was what would allow us too hook up the Kinect. Joe has a windows computer, which was much easier to interface with the Kinect, since it is a Microsoft piece of hardware, so we decided it would be Joe’s laptop that we would run the session on. Once we decided this, I began composing on my version of Ableton Live. We wanted a piece that would be able to evolve and change as the user walked through the space, so this meant it would work much better with an ambient style soundtrack, rather than say a traditional musical composition. I used a few audio files, but mostly used Midi, because it was much easier to control the parameters, by Midi Mapping (a process of assigning parameters to read incoming Midi signal, usually from a hardware pot or fader). I designed most of the sounds using a popular software synth, called ‘Massive’ which can be used as a third party plugin inside an Ableton session. While I was composing, Joe hard the nasty task of searching the web for programs that would allow the Kinect to send data to his Ableton project. This was proving difficult and after a couple hours Trent and I also joined in researching how to do this. After hours of tutorials and failed programs we finally had a working piece. Once this was up and running, Joe and I both worked on composition together, and ended up with a roughly 15 minute long loop, with different track volumes and parameters mapped to the signals coming from the Kinect, which changed the mood of the piece as the viewer moved closer to the piece.
The design decisions were made to serve the concept of the piece. Most often a balance was struck between how closely we could convey our concepts and the technical limitations we faced, whilst also taking into account aesthetic considerations. We also took into account the pieces ability to engage the viewer, as this is equally as crucial to having the desired impact.
The first major design decision was what we were to projection map onto. Early ideas included projecting onto a wall or corner, however we soon settled on a tower. There were many reasons behind choosing the tower; firstly it fulfilled the relation to the city which was the topic of the project, whilst also representing the physical place most affected by wealth inequality. Aesthetically, the tower is commanding of attention much more so than projecting onto a wall for example, this is necessary in order to engage the viewer and hold their attention better. More technical reasons for choosing the tower were that it is particularly effective to projection map onto; this creates better clarity to express ideas to the viewer. Lastly the design of the tower was influenced by the fortuitous presence of many large white cubes in our immediate vicinity.
The content of the visuals was another design consideration we faced. We thought for this consideration that the clarity with which the ideas were expressed was crucial to the impact of the installation. We therefore spent as much time as we could filming material in order to demonstrate the range of wealth inequality. When editing this material, we chose to use a rotoscope effect to show different images at the same time whilst preserving intelligibility. The other main consideration for the content of the visuals was aesthetic choices; making sure we felt that what we were projecting would be visually interesting to the audience.
The design of the interactivity was a main focus of the project. It was important to us that the interactivity was closely tied to the concept of the piece; we therefore designed it to be embedded in the piece, which followed that it was included in the earliest prototypes. Another main reason behind incorporating the interactivity was to engage the audience and create a piece which people would be drawn to. It is only once the audience is engaged that they will begin to consider the concept, it is therefore an important aspect to the installation. The motion sensor served the dual purpose of being incorporated into the concept as well as capturing the attention of the audience.
The sound design was made based on a range of reasons. Firstly, the sound is tied to the concept by being representative of whatever is being depicted at a particular time, for example when the viewer is far away from the tower and the top levels are being projected, the sound is calm and pleasant. Another consideration when designing the sound was how it would react to the movement of viewer; the difficulty in this was that it had to cover the entire range of wealth inequality whilst still sounding like one consistent piece. If the sound were to change too quickly it would lose aesthetic value, therefore we reduced the audio into four quadrants of sound which covered the concept whilst still being aesthetically pleasing and consistent when played over top of each other.
In our earliest stages of production we envisioned the generative audio and visuals being split between computers as this would allow space to work and to reconvene these aspects during installation. Early into production MMU provided us with the tools we needed such as a 4K video camera for gathering content, stereo speakers, Microsoft Kinect and later a projector and tripod.
From our research Dp.Kinect was acquired, an extension to Cycling 74’s Max that processed Kinect data into values for defining movement and position in the space. Dp.Kinect included many examples such as using the Microphones, Skeletal data and RGB and depth cameras, all useful to dissect and reference for our needs. Our first proof of concept was changing an oscillator’s pitch in Max based on the height of a user’s hand, which sounded like a finger running up and down a piano.
With this data streaming from the Kinect in MAX a multitude of attempts with different middleware to convert these values into MIDI data for Ableton Live’s panning were made. While these values could be converted into MIDI notes it was difficult to set up and see changes when sending them into Live within the same computer. This was remedied by using OSC messages sent from MAX, these messages would be received by Live and Touchdesigner.
Audio was built gradually over time as ambient loops, editable MIDI clips in Ableton Live. The audio channels that would blend into each other based on user location in the scene. These loops would follow the same scene tempo and transition smoothly as the user move throughout the space. The master project file was developed on one computer and was deployed on the Windows machine working with the kinect data. In Live’s sessions view clips were triggered and set to loop for 15 minutes, with the multiple channels representing the axis in the scene the user could move.
Early tests of the visuals were more psychedelic and distorted using feedback loops to distort the windows based on scale and position using noise. In consultation with Takashi Aiman, class mentor and audiovisual artist based in Kuala Lumpur we decided the focus should be located in the windows and representation within not the effects surrounding. This helped refocus the key visuals in the scene, being the rendered film loops and compose them more naturally into the shape of the projected building. Before installation Katan mapper was tested in the hotel rooms, mapping into a corner of the room, in inverse of the final install.
While setting up our gallery space earlier we found difficulties in the placement of the projector, needing it to be aligned vertically to the stacked boxes. This was solved with a tripod mounted projector allowing the horizontal width of the projector to run the height of the installation and producing almost twice as much resolution in pixels than if it was mounted horizontally. The installation area was cordoned with the Kinect shooting in from the base of the tower and a singular box used in the centre of the space to house the projector shooting out to the tower and holding the Mac running the visuals.
During our first installation before the exhibition tape was marked on the floor to define the area the user might move throughout, visuals were connected to the networked OSC data from the Kinect for the first time. Using a phone’s hotspot a network was established to allowing OSC messages to pass. The last hurdle to meet our design goal was to connect this positional data to the brightness of the windows on the tower as the user moves throughout the space. This was accomplished after several attempts use to math functions in Touchdesigner converting a 0-1 float into an 0-40 integer, the difficulty being in the remapping of the brightness as defined by 4 zones from the base of the tower, rather than a constant slow change over the length of the installation area.
With this completed and audio and visuals linked, our concept installation was ready for exhibition in the E-gallery. The morning setup was swift plugging in laptops and setting the installation to run throughout the day with minor hiccups we didn’t encounter the day before but might have expected. These being the phone’s hotspot dropping out and the laptop running visuals being unplugged at a certain points forcing further dropouts. In spite of this and with 98 percent uptime the installation was a success technically and conceptually.
At the end of our two weeks at MMU, the entire group was satisfied with the project as a whole. We had accomplished nearly everything we had hoped too, which we were proud of, given the amount of time we had been given to finish the project. Being given the chance to work on what we thought was a truly interesting project, and meeting new friends and learning from new people overseas was a great experience. The project received mostly positive feedback from our teachers and our peers, and we as a group look forward to further developing the project in Melbourne, and continuing to collaborate with Mike using the internet. We are also looking into other possibly options, and venues in which to show the work in the future, including a possible show in Shanghai, a competition for emerging artists in Melbourne, and even returning to Malaysia for another installation. None of these ideas are a sure thing as of yet, but have been mentioned and we intend to follow them up. Despite a small number of minor delays, the process of putting the installation together basically ran like clockwork. We had a well rounded, and skilled group that allowed for an easy division of labour, and MMU gave us everything we asked for in terms of equipment. The study tour to Malaysia was a great experience and I believe everyone involved would recommend it highly to others.