DOI: 10.24308/iass-2014-036

Faculdade Cásper Líbero, São Paulo, Brazil

felipegeg@gmail.com

Faculdade Cásper Líbero, São Paulo, Brazil

digo.morais@gmail.com

Abstract

The contiguity existent among intermediatic immediate objects becomes clearer in contemporaneity, showing the importance to study these signs of the time today. Several technological artifacts that own different functions and dynamizations are available, for example, the interactivity enabled by the relation between videogames and Kinect (first named as Project Natal): a motion sensor of great importance at some science areas for bringing complexities never known before.

Therefore, this paper aims to explore the universe of the immediate object taken at the spheres of the technological esthetics given by the relation of Kinect + videogame, demonstrating part of the perceptive and cognitive process in the intermedia relation observed. Then, also showing how a part of the perceptive process contains temporary events incorporated in given spaces by technological artifacts. This analysis will be based on the deployments made by Lúcia Santaella, specifically in the books Applied Semiotics (2002) and Liquid Languages at the Mobile Age (2007) that show with emphasis a detailed critical analysis about the themes to be explored.

1. Introduction

The Kinect, for sure, revolutionized the technological sphere and the interaction between the users and the games. With an unique composition, beyond breaking the sedentary paradigms and ignore the joystick use in their games, it brought benefits not only for the games scope – area which was designated – but also for medicine, with treatments that goes from physiotherapy to stroke patients, for the robotics revolution, for the pedagogical scope and even for the archeological researches.

This accessory that was created by the Brazilian Alex Kipman with the goal of helping the users of Xbox platform was nominated by Guinnes World Records the electronic artifact with the fastest selling of the history, impressing the Microsoft’s executives.

This article will analyze the semiotic composition inside the games universe which uses this mentioned technology, in regarding of the interpretation questions and the perception of the signs. The analysis is going to approach the subject based on the phenomenological theory of Charles Sanders Peirce and also based on the dissertation and statements of Lúcia Santaella.

In this way, the immediate object’s universe will be explored inside the technological spheres given by the relation of Kinnect and videogame, approaching the perceptive and cognitive process in the relation of the intermedia observed, since the process is continuous and the semiotic creates a need of reading about itself.

2. Technological evolution

The coming of Kinect wasn’t a random situation, but the result of a technological advance allowed through the years. The technology that we know and was possibly unimagined a few years ago, is given by the technological revolution of the twentieth century (XX). Beside and entailed by it, a vast control over the image. “The unquestionable domain of the images is due to the fact that the technological revolution has put an enormous apparatus to the vision service that you can’t deny that the twentieth century (XX) was the century of the technovision triumph.” (Santaella, 2007:259)

Then, we should understand the evolutionary character of the technologies, mainly those ones which influenced the studying and the confection of current ones, basing on the evolutionary line showed by Santaella in the “Liquid Languages at the Mobile Age” book, due to the Weibel theory:

(a) The invention of photography with automation that it brought to the production of the image was contemporaneous transmitting pictures and words in long distancevia scanning (breaking of two-dimensional image into a linear sequence of points in time) with the help of machines. The telegraph, the telephone, the fax, the forerunner of Nipkow television in 1884, and electronic telescope, all contributed to the development of technological systems for the transmission of sound, fixed images and moving images. With this, the imaging was immediately followed by transmission. At the same time, Weibel says, about electromagnetic waves’ discovery (Maxwell em 1873, Hertz em 1887) meant the birth of new visual worlds, foreshadowing the advent of telematics culture.

(b) After a short time the spatial form of the image, as in the painting and the photo, was followed by a temporal form in the movie. From a space environment, the imageturned into a temporal environment.

(c) The discovery of the electron and the cathode ray tube, both in 1879, established the basic conditions for the production and transfer of electronic images, which led to television.

(d) The magnetic recording of visual signals in 1951, which had been anticipated by recording beeps at the beginning of the twentieth century, combined the film, radioand television in a new channel, the video. The technology of transistors, integrated circuits, chips and semiconductor revolutionized the data process technology in the mid-twentieth century, leading to the production of images completely generated on computers. Later, the multimedia computers not only united all the historicalpossibilities of production and transmission of images generated computationally, but also opened up perspectives for interactive visual worlds controlled by the machine.These worlds brought unexpected news: virtuality, variability and viability thanks to the interaction between the information and the user.

(Weibel apud Santaella, 2007:259)

With this panorama that is described, we can understand the historical and cultural context of the emerging technologies that messes with the modern world.

3. Kinect

The object of our study, the Kinect (= kinetics), is an accessory for the Xbox video game platform created by the Brazilian Alex Kipman. Primarily known as Project Natal, the device enables the interaction of people with games without them necessarily need to push buttons or use joysticks, then requiring body movements. Soon, we can define the Kinect as a peripheral for body movements and voice capturing that was created with the purpose of enabling the player-game interaction and even player-videogame-player2.

The accessory goes beyond its original meaning. In other words, beyond fulfilling the purpose of helping and entertaining along with the games, the Kinect is also usedfor body and voice recognition, allowing it to have functions outside the universe of games. For example: to connect the Xbox platform, the individual can access the menu“Applications” by voice control without using any joystick. The same happens when the body recognition is activated on your device’s menu through the continuous movementof the hands, “waving” for the platform. All this can be used for a specific purpose that does not necessarily need to be the act of playing itself, allowing since the act of doing a simple research, till accessing an application with just commands that require specific sensory activities.

With the wide option of interaction that this peripheral allows, people started using it in unusual ways and it “escapes” from the original purpose, but they don’t stop representing a social-technological revolution, which will be explored later in this article as Kinect Effect.

Therefore, we have the interactive system given by the relation Kinect-user, how this interaction takes place and the whole human sensory system that is necessary for the process to be completed, are included in the scenario of virtual reality.

The virtual reality creates multisensory synthetic scenes, simulated in numerical language configured as data or datascapes landscapes, fully structured by abstract language, which respond to the actions of those who experience them. This produces intense effects on human perception due to the feeling of being truly immersed in asynthetic world. Differently from the e scenes that record the visible reality through devices based on “Visionica” (Virilio 1988), such as cameras, telescopes, satellites,virtual world, according Couchot (2003: 175), is entirely computational and involves completely the viewer’s body. This can change your point of view and move your body under its own command. The immersion happens because, on one hand, the body is equipped with sensory or interfaces devices that extend the sensory field, on the other hand, the environments, which the bodies are connected, are built of three-dimensional computer language qualities, composing landscapes with proprioceptivequalities that replicate spatial situations experienced when we move our bodies in the physical world (Domingues 2004: 37). (Santaella, 2007:274)

3.1. The sensor’s features

We can say that Kinect is a highly innovative combination of cameras, microphones and software that transforms the player's body into a video game controller.

There is an innovative tripod that works with the Kinect sensor: video camera, depth sensor and microphone.

Fig. 1: Kinect configuration and composition (Microsoft).

Source: http://msdn.microsoft.com/en-us/library/jj131033.aspx.

Color Video Camera VGA (Color Sensor) – This camcorder is for facial recognition and the detection of other features, when detection occurs in three component colors:red, green and blue. This makes capturing of a color image possible.

(IR Depth Sensor and IR Emitter) – The emitter emits infrared light beams and the depth sensor reads the IR beams reflected back to the sensor. The reflected beams are converted into depth information measuring the distance between an object and the sensor. This makes the capturing of a depth image possible.

Voice Detector (Microphone Array) – This is the series of four microphones that can isolate the voices of the players from the noise of the room. It allows players to be a few meters away from the microphone and still use voice controls.

Motorized base (Tilt Motor) – The tilt motor lets you change the slope to + or – 27 degrees, which effectively doubles the volume of the captured field of view. The slope is related to the gravity, rather than being connected to the base of the sensor. An elevation angle of zero indicates that Kinect points perpendicular to gravity.

The detection system of the skeleton is able to “see” up to six bodies – of which only two are active and possible of interaction – and also has the ability to map 48 points of the human body. For everything to work naturally, the player must stand at a distance from 1 to 4 feet of the sensor without obstacles.

3.2.1. Kinect effect

The Kinect was developed by Microsoft to revolutionize the way we play videogame and its entertainment experience, as already discussed in this document.

However, people started using the device in ways never imagined before. The use of this technology that is been studied is plural and diverse, with records from the educational context, passing through assistance to archaeological researches and going to the operating room, taking Kinect beyond games.

That is what has been termed by Microsoft as “Kinect Effect”: the diversity in the use of peripheral, used in many different ways, bringing new experiences to the world and in a revolutionary way.

That’s why we bring a significant contribution to the understanding of multi plurality view about the usability of the sensor[1].

3.2.2. In archaeology

Students of the California University at San Diego were part of a futuristic archaeological dig in Jordan: a three-dimensional mapping of the soil which helps in accurate location details. Instead of using an expensive and complex system such as LIDAR, the team used Kinect.

Fig. 2: Archeological excavation

Source: http://www.fastcompany.com/1771025/kinect-hacked-3-d-scanning-archaeology-site.

After capturing the images, models can be made in 3D and project them in an immersive virtual reality environment. In addition to capturing the images, researchers can also add information about each part of the space.

3.2.3. In the student scope

The interactive learning is fundamental to engage and motivate students in the technologically advanced world of today.

Given by the nature of the digital lives of the students and having in mind their great interest in video games, educators are increasingly using immersive approaches based on games, helping students to develop their essential skills.

In classrooms across the United States, teachers who integrate Kinect into their classes reported tangible improvements in student engagement and learning.

The technology is being used as part of pilot programs underway at the district of Los Angeles Unified School (California), Chicago Public Schools (Illinois), Houston Independent School District (Texas), Scottsdale Unified School District (Arizona), Flagstaff Unified School District (Arizona), Fairfax County Public Schools (Virginia) and Loudoun County Public Schools (Virginia).

3.2.4. In Medicine

A fact to be explored with regard to the use of Kinect in medicine is the case that happened in Brazil in 2012, the Evangelical Hospital in Londrina, located in Paraná.

Fig. 3: Doctor using the Kinect system instead of using negatoscope at a hospital in Londrina, Paraná (Brazil).

Source: http://revistadonna.clicrbs.com.br/saude/hospital-desenvolve-tecnologia-com-ferramenta-de-videogame/.

This was the first Brazilian medical center to develop a method that uses Kinect in operating rooms. The intention was to end with the use of film sheets in the negatoscope, a device with special lighting that assists and enables observation of radiographs. An examination sheet measures approximately 35 cm wide by 45 cm high, and a panel typically hold a maximum of three sheets. In more complex surgeries, it is common to perform the exchange of images numerous times. As the medical team can not touch anything, to avoid contamination, the display of tests was handled by nurses or assistants.

During a conversation with the IT staff of the hospital the context of using the Kinect came out. The professionals started to develop a software that uses the peripheral connected to a PC. With adjustments and repairs over the months, the INTERA emerged – Interaction between man and machine – that allowed safety, agility and dynamism at the access and movement of patient’s radiological images, with the doctor dressed in a sterile manner, without the need to contact the equipment.

4. The Perception by Pierce

After passing through the features, unique composition and usage of our object of study, we can now enter the semiotic question.

Order to understand the process in which the interaction takes place with the Kinect player when referring to the perception of it, we must first understand the theory of the perception of Charles S. Peirce, which is grounded in phenomenology, “a quasi-science that investigates the ways we apprehend anything that appears to our mind” (Santaella, 2007:02).

Then, the development and analysis of the perceptual and cognitive question given by the relation Kinect user (or gamer), first we must understand the inference relation, reference and interpretation made possible by the sign, as shown in Santaella:

[...] in Charles Sanders Peirce’s definition, the sign is a triadic nature, in other words, it can be analyzed:

· in itself, in its internal properties, in other words, in its power to signify;

· in its reference to what it indicates, refers or represents; and

· in the types of effects that is able to produce in their receptors, that is, the types of interpretation that it has the potential to awaken in their users.

Therefore, the semiotic theory allows us to penetrate the inner movement of the messages itself, in how they are engendered, procedures and resources it used. (Santaella, 2007: 5)

With this, knowing that “sign is anything of any kind that represents something else” (Santaella, 2012:8), now, we can analyze here in semiotic terms as the process of signification is given in the relation user = Kinect.

5. Percept, Percipuum and Proprioception

Then appears to us the following question: how a perception that contextualizes the process of human interaction within systems that make up the universe of virtual reality is given? In other words, there is no interaction without the possibility of knowledge of interactive human being for their overall self-representation in body terms.

For this, we must begin the research process based on the Peirce’s perception theory.

The theoretician identifies and characterizes two former terms of the perceptual process that is distinguished by its characteristics: the percept and the percipuum.

To understand the percept, we must resort to what is external to us and presented through the perceptual process, it can only be understood through a judgment of perception, and for this last one, we won’t be attentive to a robust analysis. Soon, percipuum can be seen as the percept within the boundaries that “our sensors impose” (Santaella, 1998, p. 52). Briefly, the percipuum is to the immediate object and the percept is to the dynamic object. Rodrigo Morais gives us a good overview of these terms:

The percept remains independent and external to the mind, while the percipuum is found at the judgment of perception, In other words, the percept can be taken as any stimulus that hasn’t had contact with any sensory receptor, while percipuum is the percept at the metaphysical moment when processed by a body. (Morais, 2013:32)

For example, the percipuum is in the immediate object, in here and now. Is in the representation of something through a space, the sign itself. Therefore, the percept, which is connected directly to the dynamic object, presents itself to perception at a referential, metaphysical moment, the fugacity of the perceptual process.

6. Proprioceptive Question

As already discussed in this article, there is a possibility of interaction between the player and the videogame – that uses Kinect – without having, first, a self-recognitionas a whole in tangible and spatial terms. The player acknowledges himself in that realities composition through his own knowledge in relation to his own image.

[...] opening up the understanding field of human recognition through the senses, it is evident that this temporal and spatial self-determination inherent to the construction of reality of each individual is directly connected to the sef-image, not only in visuality terms, but understanding the imagery contemplation from synesthesia, from all sensory possibilities concerning the human. (Morais, 2013:25)

This process is enabled by the existence of a proprioceptive system, that promotes precisely this spatial recognition of the body, making its existence be one of the leading factors to the interactive cycle that Kinect searches be completed.

7. Proprioceptive equivalence in Kinect

When facing a complex system that simulates the moves of a body, the person who is interacting, instead of what you may think, doesn’t find parallel realities that are separated , in other words, there is a composition of reality that makes this person perceives and interact with the technological composition, having, this complex composition all the immediate object.

With this, the self-perception, the external perception which is given by the interactive moment and the whole context that encompasses the act of playing stands to capture the immediate object.

The biological body that generates the avatar used in the Kinect interactivity can be understood as an equivalent body to the sensory representation given in virtual platforms taken by the technical apparatus analyzed here. That is, given the facts cited about the proprioception, the spatiotemporal self-recognition, it can be seen that the human being has access to itself through representations given by its perceptual possibilities, just as also happens with the body understanding generated from Kinect and projected in other technological advent.

Finally, we understand that the Kinect and all the technological advents today that permeate the contemporary life, open the doors to new studies that show a possibility in multiplicity of identity in the understanding of the human being as its own object of study.

References

SANTAELLA, Lucia. 2007. Linguagens Líquidas na Era da Mobilidade [Liquid Languages at the Mobile Age]. São Paulo: Paulus.

SANTAELLA, Lucia. 2002. Semiótica Aplicada [Applied Semiotics]. São Paulo: Cengage Learning.

EATON, Kit. 2011. Kinect hacked for 3-D scanning of archaeology site. http://www.fastcompany.com/1771025/kinect-hacked-3-d-scanning-archaeology-site (last accessed: 31 December 2014)

TABORDA, Cauã. 2012. Hospital brasileiro usa Kinect para fazer cirurgias [Brazilian hospital uses Kinect to do surgery].http://exame.abril.com.br/tecnologia/noticias/hospital-brasileiro-usa-Kinect-para-fazer-cirurgias?page=1 (last accessed: 31 December 2014)

REDMOND, Wash. 2012. Teachers Are Using Kinect for Xbox 360 to Engage Students and Bring Learning to Life. http://news.microsoft.com/2012/03/06/teachers-are-using-kinect-for-xbox-360-to-engage-students-and-bring-learning-to-life/ (last accessed: 31 December 2014)