HEARTS OF METROPOLITAN
Project Concept & Design Response
Over the two weeks of exchange at MultiMedia University, we collaborated with a group of MMU students to create a digital storytelling work surrounding the theme of “city”. The brief for the for the project was simple and allowed for a vast variety of options to be explored. The teams were required to deliver a project with the emphasis on “City”, which can be expressed through linear or non-linear storytelling.
The theme allowed for much creative adaptation and experimentation, and as we explored different concepts in our team, several keywords and themes began to surface. We wanted to create a visual representation of both Kuala Lumpur and Melbourne combined. Focusing on the specific cities which are connected by the tour, we decided to present an interactive exploration of both cultures, new and old. As we brainstormed ideas, we explored different ways of representing them visually and aurally. As we looked at the similarities and differences between both cities, we realise that although the culture is very different, the essence of the city is the same. Plus our approach is a celebration of the similarities and qualities that make the cities alive and human. For example, both Kuala Lumpur and Melbourne are extremely multicultural, have a blend of International qualities. Diving into details, the magnification of the cities allow us to explore details that make each city unique. For example, the familiar sound of the traffic light is not something you would hear in Kuala Lumpur. Likewise, the track sounds of the monorail is also something you can’t hear in Melbourne.
Our project consists of visuals projections of both city’s distinctive locations and views. This is juxtaposed by generative images which were also culturally distinctive, such as footage of the Golden Wattle and the Lotus Flower. This imagery is supported by a mix of diegetic and non-diegetic sounds, such as field recordings mixed with musical harmony and elements that communicates emotions and fragments of each city. Lastly, there is the element of interactivity, as no city is alive without human interaction. The interaction is accomplished by allowing the audience to remix the sounds, videos and generated effects through a MIDI soundboard controller. This elevates the project to a different level, as it is no longer just a video but an interactive piece designed for an installation exhibit.
The softwares used was also a major part of the project, as it allowed the ideas which we brainstormed to be become a reality. For the sound aspect of the installation we used Ableton Live 10. This program is industry standard and allows for live loop triggers, which are an essential part of the project. For the visual side of the installation we employed the use of Resolume Arena 6. This software also allows for live visual triggers that can be looped or overlayed depending on the user input. With the integration of both elements, we were able to map visual and auditory loops to MIDI triggers, creating an immersive experience.
Bao-Linh: Skill & Contributions
Everyone in our team has exceptionally different and vast skills. My contribution to the team was through my newly found interest in the world of audiovisual – creating generative visuals and effects, which I had learnt of a workshop taught at the Multimedia University in Malaysia. We were taught by Takashi Aiman, the founder of The Tomoe, an organisation termed as “an underground creative solution organization + multimedia visual collective established in 2012”. I was in awe of what a VJ (visual jockey) produced, and how they are able to create visuals in sync with different sounds. Choosing to delve into generative visual effects was a big risk as it was outside of my comfort zone and I’ve always usually developed my projects in the direction of virtual reality. However, seeing how excited Takashi was to share his works really inspired me to try something new as well.
To achieve my interest in audiovisual and generative visual effects, I needed to learn and use After Effects within the two weeks and unfortunately, I cannot apply my expertise in Unity from previous virtual reality projects. To create proper visuals that would relate to our project, I needed to research and learn to use a plugin called Trapcode Suite by Red Giant. However, the program itself was very costly, which had affected my choice of wanting to commit to purchasing it, as I have never attempted audiovisual before. Nevertheless, I still persevered and was able to find an alternative and it allowed me to create visuals for our project within the short timeframe. Through a substantial amount of research and self-teaching via online tutorials, I was able to produce visuals that exceeded my capabilities and allowed me to develop new skills. We hope to expand our creative abilities and create better visuals for our upcoming project.
VFX & Animation Lead
In terms of design considerations, production of visuals that correlate to both of the countries and cultures had to be made. This involved the use of symbolism including things such as flag symbols from Malaysia and Australia, outlines of both countries, national flowers – the golden wattle and the hibiscus – city outlines that showcase each city’s main attractions, and a lotus flower to represent Kuala Lumpur.
As the theme for the brief revolved around “city”, it was important to showcase this in the visuals, as culture is the main focus being conveyed to the audience. To present both the contrasts and similarities of cultures held by both individual cities in the connection of human nature, the piece is titled Hearts of Metropolitan, representing the essence and heart of both cities through the installation collaboration. Another design consideration had been the choice colour. It was important to make sure that the colours used would able to stand out from the background. As the background video is already saturated and busy with details, the visual effects had to be bright and colourful, ensuring that the audience would be able to distinguish what is being presented in the foreground from the background. Making sure the visuals are clear and concise enables the audience to observe which symbols belong to which country.
During the design iteration process, the production of the visuals was made through a combination of research and online tutorials. From researching, the program Trapcode Suite was acquired, allowing the commencement of the production of visual effects. Trying to figure out the plugin was a definite challenge but a reward in itself when successful. The visuals themselves were kept simple because of the unfamiliarity of the plugin and time constraint, however, were not impossible to do due to a lot of trial and error, patience and practice. Additionally, the visuals themselves could not be too distracting or too large because it could deter the audience from concentrating on the background video and sounds/music. The visuals were there to add another element of texture to the whole experience, so it was important not to have them override the over elements of the installation.
The exportation of files had come to be difficult as files were exported to be used in Resolume Arena 6 – a VJ program where a visual jockey can create and control visuals so that they are with sounds/ music – however, the only way files were able to be opened in that program was dxv. To get those files to dxv, a dxv converter had to be downloaded from the website, where it was then converted to MPEG Streamclip. Nevertheless, the conversion of the files was futile as it still did not work. This error of file type had been fixed, as opening it as mov had resolved the problem. This, however, was not the only issue as once opened in Resolume Arena 6, the background was not transparent. The resolution to this obstacle was by the turning on of the transparent grid in After Effects, and then toggle the noisy filter in Resolume Arena 6 until the visuals were clear again.
Vinicius: Skill & Contributions
My personal expertise lies in video shooting, editing and foley production and I contributed through a variety of roles throughout the project. At the beginning of the two weeks, footage was needed to be produced rapidly to allow all team members to visualise their work, hence I was involved in creating short two-minute clips of videos shot in Malaysia. Using Adobe Premiere Pro for this task, I was involved in cutting, arranging and enhancing the footage into a cohesive piece of visual storytelling.
I was also involved in the production of sounds and music for the installation. This involved the use of Adobe Audition to remove background noise and using pass-gates to highlight the necessary sounds. The musical side of the project was done in the Ableton Live 10, which involved a simple 808 drum beat to represent the heartbeat of the cities. This was paired with the high-hat like the sound of pedestrian crossings from Melbourne produced by another team member. There was also a simple sparse xylophone melody backed by a choir.
With the help of another team member, I designed the poster for the installation. The concept is to use the imagery of the ocean and boat to symbolise the great distances between Melbourne and Kuala Lumpur. The paper boat expresses the will of the people to be together despite great distances.
Vinicius Riederer Branco
Sound Design & Post-Production
Lastly, for the poster, we wanted to create a visual interpretation of the great distances between the two cities. This representation of distance is a difficult concept to demonstrate without time and space, therefore I altered the concept slightly to display the feeling of isolation instead, which is an easier concept to demonstrate visually. Although isolation is different to the initial idea of separation, it still conveys similar emotional chord to the audience. This isolation echoes through themes such as physical barriers and human interaction.
During the video editing process, there were many stages of iteration and development. Initially, I worked with multiple long loops of footage – up to five minutes in length. However, we eventually chose to only use a single loop of shorter duration, which was about three minutes long. This loop formed a base for the whole project and plays in the background when no user interaction is detected. Furthermore, it forms a connecting thread that unifies the motion graphics and short loops that can be layered on top. The choice to supplement a longer video with very short loops of thirty seconds made it easier to transition smoothly between loops in accordance with user input.
Looking back at the sound design workflow, the original idea was to use a mixture of aboriginal music mixed with traditional Malaysian music. This is a celebration of both cultures, but the current political sensitivity of the topic was a deterring factor. Therefore, we chose to construct a soundscape with focus on modern cultures, with inspiration coming from trap music. This was then transformed into a more original interpretation of culture by adding the soundscape of everyday life within the city to create an interactive percussive composition.
Lastly, the poster originally showed a map of the oceanic region, displaying both countries and in part, both cities. This was then transformed into the current poster due to the tackiness of the original design. It simply conveys the physical aspect of the area and nothing more, with the second iteration demonstrating more, in a subtler way. The last iteration also has graphics and colours which are aesthetically similar to that demonstrated in the video, especially the main loop.
Jolly Nga Wan: Skill & Contributions
My design expertise lies in VR and interactive content, which was the direction we initially thought about pursuing. This is due to the fact that several of our other members have backgrounds in similar skills and this would allow us to most effectively make use of our team’s expertise. However, after brainstorming the idea and putting our team members’ skill-sets into consideration, we realised it would be impractical to use VR and it is not the ideal way to portray the ideas we have. Other ideas included using augmented reality or projection mapping and they were rejected as well due to time and skill constraints.
During the two weeks, I volunteered to be the creative visual lead, which involved working with the sound creative lead and the rest of the team to create a cohesive project. It was challenging as we had less than two weeks to complete, and it was my responsibility to oversee and lead the project to a successful conclusion. Drawing from a diverse group of skills and backgrounds, we worked together in an agile team structure which allowed us to be more efficient and work dynamically.
As we planned the project, we worked with our teams’ capabilities, and what skills we can realistically acquire within the short time period. We brainstormed all sorts of ideas and narrowed the focus down to an idea that was challenging yet achievable. This included mapping out both visual and sound workflow diagrams and allocating each team member to tasks that make the best use of their skillset and would allow them to expand and develop further skills.
Personally, I was able to develop and improve my skills during the few weeks. The experiences leadership role as creative lead allowed me to improve my project planning, time management, and communication skills. I was also able to further improve my technical skills in Adobe Premiere Pro and After Effects. During the two weeks, I researched and learnt how to apply colour correction in various circumstances to allow a cohesive visual direction for the final outcome. I also experimented with visual effects and applying alpha channel masking to create overlays and double exposure effects.
Jolly Nga Wan Yau
Creative Visual Lead, Cinematography & Post-Production
During the initial brainstorm process, our goal was to think of as many ideas as possible that represents the theme of “cities”. Through the process of elimination and consideration of the project deadline, we were able to scale down the overly ambitious ideas and think realistically about what we can achieve in two weeks.
As we brainstormed ideas to on how to visually symbolising the similarities and differences of both cities by contrasting and combining visual representations, we chose to use overlay and juxtaposition techniques. As we discussed ways to incorporate interaction into the project, we discovered a powerful software called Resolume Arena 6, which allowed us to fully extend and create a non-linear interactive experience depending on how the user interacts with the installation. We wanted the user to create their own narrative, as the human side of the cities required human interaction. Through the non-linear storytelling, each user experience will be inherently different as they are in control of the story.
As I was responsible for the colour correcting and post-production process, I began experimenting with different hues and curves on the Lumetri Colour panel in After Effects. While researching symbolism of both Kuala Lumpur and Melbourne, I realised that both the national flag colours have repeated colours of blue, red and white. When mixed together, it creates a beautiful purple haze that also ties the compositions together in a visually cohesive way.
There needs to be a central theme or visual identity that connects the installation together. As I looked through the footage from Melbourne, I discovered several timelapses of sunsets and sunrises. This created an interesting concept to portray a “day in the life” of each city, telling the story of a day in the life of each city, both cities waking up from the quiet night to a buzzing morning and calming afternoon, before returning to the vibrant night of each city and repeating the day again. A central theme that threads the whole project together.
One of the limitations we worked around was the lack of versatility in capturing specific footage in Melbourne, as we weren’t able to fly back to Melbourne to capture what we needed. However, we were able to overcome this obstacle by using previously recorded footage from each of our four team members. Using iconic places and sounds of the cities, we delved into exploring what exactly makes each city feel like home to you?
There were many iterations throughout the visual creative process. We were able to use the workflow diagram and guide us through the entire process, but there were also exciting unexpected findings during the design iterations.
As we began with an ambitious idea, we began revising the idea as soon as we initialised the project. Starting from an idea that incorporated a 360-degree video projection with augmented reality integration and surround-sound, there were many areas that needed to be refined and condensed. This idea was out of our reach due to limited skills and limited time, but I believe we may be able to achieve a portion of the desired effect through an alternative approach.
For the projection, we started with four projections, one on each side to simulate a 360-degree video. This was cut down to two projectors and finally only using one for the installation set-up. The way we built upon the complexity of the idea is to allow user interaction of the video and projection through a MIDI soundboard set-up at the exhibition. There was also an abundance of footage and film to be used in the background loop and overlay of the videos. Abstract, simple footage tended to be better suited for overlaying and achieving the effect of juxtaposing on top of the background video.
Akash: Skill & Contributions
My design expertise lies in VR, sound design, user experience and combined with work experience as a multimedia designer in an advertising studio back in Mauritius. As an avid self-learner and always keen to explore new technologies, I volunteered to research on a production workflow for Hearts of Metropolitan. In case any programming would be needed, I would get on board to help as well.
I have always been intrigued by sound and music, and also how they complement visual experiences such as movies and art installations. In the beginning, I wanted to apply my 360/VR sound skills to this project. As the project evolved we decided to not use VR technologies and instead to create an experience where the audience is the one who remixes the live visuals and audio. My suggestion was to use organic elements to recreate visuals and audio.
After a few brainstorming sessions, I went on a journey of self-learning with Ableton Live. I have always excluded Ableton Live as part of my sound design and production workflow. It was a discovery for me as I implemented Ableton Live in the workflow. Everything was very simple, fast and intuitive. My sampling process was very fluid and so was the same with creating a drum kit instrument using Malaysian drums and bells. Resolume Arena and live visual performances also caught my attention during this project. I had the chance to explore the software and understand more about the VJ scene.
Prior to the deadline, I was also involved with injecting some leadership at some stage in our process. I took responsibility to bring the team together when I found out that we all lacked clarity and direction in what we were doing. I wanted my teammates to see themselves as leaders in their own areas of expertise. We discussed each other’s’ views on the project and the direction we want to walk in. This indeed helped us in pushing boundaries and producing great results as a team together with meeting deadlines.
Sound Design Lead & VJ
Throughout the five full days of intensive project work at the Multimedia University, we encountered and embraced a few challenges along the way. There was a variety of pros and cons examined before projecting ourselves into the idea execution phase. We accustomed our design decisions after assessing a list of factors: setup space, materials, equipment, team skills, interactive elements, physical location, environment and time allocated to us.
Being physically present in Malaysia, we were very restricted in capturing footage and sound that would signify the city of Melbourne. We tried finding footage and sounds from the internet, unfortunately, the quality was poor. Our team member, Vini was able to find some good quality footage and audio he recorded in Melbourne some time ago
The software involved in our workflow was also highly regarded as an integral part of the process. For creating a rich audio-visual experience that the audience can easily interact with, Ableton Live was the best fit. We also consulted other sound production experts, who as well confirmed that Ableton Live would be very easy to use for production and the live set for the audience. The next step was to control visuals with Ableton Live. The easiest way suggested by mentors was to use VDMX. However, we discovered Resolume Arena during a workshop by Takashi Aiman. It was the game-changer in our workflow. I was advised to map and use my Korg Nanopad(midi controller) with Ableton Live and Resolume Arena at the same time, which made it very easy for the live set.
As time was also very limited, we were advised by mentors to drop the quantity and only produce materials for half of the number of keypads on the controller, which was eight instead of sixteen.
Our concept evolved during the design and production stage. Our biggest question was practicality over complexity. The time factor was crucial and we wanted to have a working prototype ready on Thursday afternoon. We were advised to drop the number of keypads so that we could have a working version of the installation rather than a product that is not complete.
We could not use VDMX in the first place, however, we found Resolume Arena which suited the workflow perfectly in terms of ease of use and linking with Ableton Live. Both of the software applications were very intuitive to use and easy to map with the midi controller I brought in my bag from Melbourne. The Korg Nanopad (midi controller) proved to be a very great asset to our project. It was easy to implement and has large buttons which are good for different finger sizes. Also, being a music enthusiast, I would also add that our project workflow was very seamless with midi controllers on Mac operating system. The plug and play feature saved us time with setting up drivers and creating midi connections.
On Thursday when Linh and I made a mock setup of the Live experience, we had to modify the look and keep things very simple. We were not able to conceal the projectors while using the wooden blocks supplied. Yet, the projector and sound system supplied to us worked really great. While setting up, Linh and I decided to drop the idea of an illustrated table top panel that would cover the midi controller. Instead, we used yellow fluorescent stickers with hand-drawn icons. After setting up, we tested the triggers, visuals and audio. Linh’s visuals were not overlaying as transparent layers on top of the footage. However, I managed to modify some layer values on Resolume Arena and it all worked fine with the footage that Jolly and the others created.
FINAL OUTPUT & CONCLUSION
Final Output of Work
After two weeks of the study tour, the installation was finally completed between the two universities. A recap of what and how each team member contributed is summarised below:
- Jolly: Creative lead, creating video loop, editing video overlay, colour correction;
- Akash: Sound design lead, Resolume Arena 6 master;
- Vinicius: Recording video footage, sound design, video editing;
- Bao-Linh: Generative visual effects lead, animation, creating overlays;
- Harith: Recording video and sound, audio clean up;
- Daniel: Editing video footage, editing background loop;
- Naily: Editing video footage, poem writing, voice-over recording.
As time constraint was the dominant issue present, limitations arose. Initially, the plan of using two projections had been scaled back to only one projection. Not all sounds nor footage were incorporated into the final project, and some background loops did not appear on the MIDI controller, hence only one constant loop was displayed. Predominantly however, most of what was planned have been ideated and executed successfully. This includes the use of footage from both cities and arranging them into a loop to be used in the background. We were also successful in blending the foley and music symbolic to each city and creating visual effects to be played over the video.
Overall, satisfaction had been met with the installation collaboration with the team members of Multimedia University. As a team made up of RMIT and MMU students, communication and work was well delivered and conveyed. The final output of the shared work had met the team’s standards and was exceptionally done well, considering the pressure of the time constraint.
Hearts of Metropolitan has helped us push boundaries in two weeks within a different study environment and with limited equipment. We have made discoveries, acquired new skills, strengthened our current ones, learnt more about each other and created connections with a different culture.
The two weeks we spent in Malaysia was an immersion into the culture and lifestyle of the country. We also embraced a design process in which we had the creative freedom to tell a story about two cities. Along the way, we discovered iconic places of Kuala Lumpur and included them in our project. Some of us also found new areas of interest to specialise in.
Our creative team is comprised of a colourful cultural mix and we all believe cross-cultural collaboration is an extension beyond what we are bound to see as an individual. As the four of us have bonded well since the study, we are looking forward to expanding the concept and make Hearts of Metropolitan a successful project which we can pitch for exhibitions.
Since we got back to Melbourne, we already met a few times for brainstorming sessions. We intend to add more complexity in the visuals with projection mapping and elaborate more on the sound design for a fluid mix. In conjunction to cross-collaboration, we also wish to involve our friends in Malaysia for more creative input. We have already started seeking inspirations from art galleries and other visual artists we found on the internet. We aim to create an interactive physical live set which will immerse the audience in the story of two connected cities.