style: markdown formatting

submissions/438/presentation/A handful of pixels of blood - Transcript.md

@@ -2,7 +2,9 @@
 created: 2024-09-06T11:02
 updated: 2024-09-09T13:04
 ---
+
 # A handful of pixels of blood
+
 ## A Historical and Technological Perspective on Understanding Video Game Graphics
 
 Good afternoon, colleagues. Today, I'd like to share with you parts of my research on video game programming practices of the 1980s and 1990s, with a particular focus on graphics. This work is an integral part of my dissertation, where I'm exploring the technological foundations of video games as a popular medium.
@@ -17,13 +19,13 @@ To conduct this research, I'm working with two primary datasets. The first is ca
 
 The second dataset is much larger and more comprehensive, the Video Game History Screenshot Dataset, or VHS-Dataset for short. It contains approximately 113,000 images from around 4,300 games, encompassing nearly all pre-1990s games available on the video games information platform MobyGames. I created this dataset to provide a map or a distant view of video game graphics history and its development.
 
-Further, I'm using the Framework for the Analysis of Video game Representation (FAVR), which was introduced in a seminal 2015 paper. I find this framework particularly useful because it emerged from a fundamental critique of existing methods for analyzing video game images. The authors argued, and I agree, that vocabulary borrowed from art history and film studies was insufficient for capturing the unique aspects of video games. 
+Further, I'm using the Framework for the Analysis of Video game Representation (FAVR), which was introduced in a seminal 2015 paper. I find this framework particularly useful because it emerged from a fundamental critique of existing methods for analyzing video game images. The authors argued, and I agree, that vocabulary borrowed from art history and film studies was insufficient for capturing the unique aspects of video games.
 
-FAVR is built around five core elements: modes, composition, construction, spaces, and planes. Modes refer to the overall structure of what's visible on screen and its specific function, like title screens or gameplay screens. Composition and construction represent different levels and aspects of visual analysis. Composition focuses on the overall arrangement and organization of spaces on the screen surface, providing a broad view of how visual elements are laid out, and foundational understanding of the screen's visual organization. 
+FAVR is built around five core elements: modes, composition, construction, spaces, and planes. Modes refer to the overall structure of what's visible on screen and its specific function, like title screens or gameplay screens. Composition and construction represent different levels and aspects of visual analysis. Composition focuses on the overall arrangement and organization of spaces on the screen surface, providing a broad view of how visual elements are laid out, and foundational understanding of the screen's visual organization.
 
-In contrast construction delves deeper into the layered nature of video games images, recognizing that the visual representation is composed of multiple conceptual planes. These can be agents, in-game environment, off-game environment, and interfaces. They take formal aspects into account, such as graphic materials and projection methods. These attributes help reading into technical aspects of what we see on screen, a bit like art historians that know what kind of pigmets were used by looking at a painting's colors in conjunction with when and where it was painted. Planes offer a more granular analysis of the technical and conceptual elements that make up the game's images. 
+In contrast construction delves deeper into the layered nature of video games images, recognizing that the visual representation is composed of multiple conceptual planes. These can be agents, in-game environment, off-game environment, and interfaces. They take formal aspects into account, such as graphic materials and projection methods. These attributes help reading into technical aspects of what we see on screen, a bit like art historians that know what kind of pigmets were used by looking at a painting's colors in conjunction with when and where it was painted. Planes offer a more granular analysis of the technical and conceptual elements that make up the game's images.
 
-As part of my research, I translated the framework into an ontology, based on CIDOC CRM, and created Tropy templates to be able to properly annotate images. I've found it also necessary to expand FAVR by another element. I've incorporated the concept of ludemes as another type of plane. A ludeme is the minimal unit of play or the smallest element that can be grasped by a player, in terms of game rules or game mechanics. Think of the +4 card in Uno. They're essential for gameplay and can take many forms, such as these switch-orbs-and-barrier-puzzles in The Legend of Zelda: A Link to the Past. Ludemes combines both the structural aspect of game design and the experiential aspect of player engagement, and can be composed of graphics, sounds, and game mechanics. What I'm particularly interested in is their service as an interface between playing and creating games. They reflect the dual materiality of video games: playable forms and underlying code. I see ludemes as the meeting point between player agency and the game's visual representation of its internal states. 
+As part of my research, I translated the framework into an ontology, based on CIDOC CRM, and created Tropy templates to be able to properly annotate images. I've found it also necessary to expand FAVR by another element. I've incorporated the concept of ludemes as another type of plane. A ludeme is the minimal unit of play or the smallest element that can be grasped by a player, in terms of game rules or game mechanics. Think of the +4 card in Uno. They're essential for gameplay and can take many forms, such as these switch-orbs-and-barrier-puzzles in The Legend of Zelda: A Link to the Past. Ludemes combines both the structural aspect of game design and the experiential aspect of player engagement, and can be composed of graphics, sounds, and game mechanics. What I'm particularly interested in is their service as an interface between playing and creating games. They reflect the dual materiality of video games: playable forms and underlying code. I see ludemes as the meeting point between player agency and the game's visual representation of its internal states.
 
 ## Findings
 
@@ -33,7 +35,7 @@ Second, FAVR has been crucial in my conceptualization of ludemes. It's helped me
 
 And now for a brief detour. Working on the VHS-Dataset has provided valuable insights. So let me briefly talk about that, since it is shining another light on the same issue. I used the self-supervised transformer model DINOv2 and UMAP for dimensionality reduction, followed by k-means clustering. Interestingly, I found that clusters primarily formed around either formal or semantic aspects, or in some cases bringing these two together to build genre clusters.
 
-An example of a cluster around formal aspects is the isometric projection cluster. This cluster picked up on the specific formal element of isometric or oblique projection, which is a way of constructing a 3-dimensional representation without perspective. Regarding semantics we can pull up this nsfw-ish one. The ability to include images, graphics and illustrations, of course, introduced visual material to video games, that is deemed not safe for work. This includes pornographic images, display of violence and gore, and material that shows ideologically problematic material, like war and fascist symbolic. Lastly and is an example of a clusters around a video game genre. The *Boxing* cluster is an example of a clustering, in which formal and semantic aspects mingle and evolve in the direction of a genre. Despite being of fundamental different game play and mechanics, and aesthetics, the cluster is held together by a few semantic-formal elements.
+An example of a cluster around formal aspects is the isometric projection cluster. This cluster picked up on the specific formal element of isometric or oblique projection, which is a way of constructing a 3-dimensional representation without perspective. Regarding semantics we can pull up this nsfw-ish one. The ability to include images, graphics and illustrations, of course, introduced visual material to video games, that is deemed not safe for work. This includes pornographic images, display of violence and gore, and material that shows ideologically problematic material, like war and fascist symbolic. Lastly and is an example of a clusters around a video game genre. The _Boxing_ cluster is an example of a clustering, in which formal and semantic aspects mingle and evolve in the direction of a genre. Despite being of fundamental different game play and mechanics, and aesthetics, the cluster is held together by a few semantic-formal elements.
 
 I'm at the moment in the process of compiling the research on the VHS-dataset into a paper, so I'm not going into any more details for now. The important takeaway for this presentation is that DINOv2 can identify modes. Many of the clusters that were produced, for example only contain start screens or in-game screens with levels with an isometric projection. But, the model struggles with ludemes. Those typically manifest as repeated graphical elements within a game, but were not recognized as an important visual or semantic unit within the clusterings. Ludemes are most often intuitively comprehended by players, if the game developers and designers did their job. But for now they pose a semantic category, that is beyond what the computer vision model can see. Beyond epistemological questions, this also raises the need to continue the work on the FAVR ontology, in order to teach the computer to see these elements.
 
@@ -47,4 +49,4 @@ Of course, this research comes with its challenges and limitations. The scarcity
 
 As of now, the two datasets, as well as the formalized FAVR ontology stand and are published on Zenodo. I wish I could have already done concrete case studies and ask questions to the datasets, through the application of FAVR. But for now the creation and maintaining of the datasets as well the construction of the ontology used far more time than I anticipated. After finishing the VHS-dataset paper I will continue with this work here and start to annotate specific examples from the datasets. If you have some free time on your hand, I'd be more than welcoming if you want to join this endeavour. And if not, I would be eager to hear your thoughts and be open to any questions or discussions about this work.
 
-Thank you for your attention. 
+Thank you for your attention.