1. Summary

The paper presents how digital photographs of street art and graffiti writing
[1]

are analyzed with computational methods by the Computer Vision group of Heidelberg University, where an interdisciplinary collaboration between art history and computer vision is embedded since 2009. The project on urban art started in November 2017 and has the following aims: It studies the effect of digital possibilities on street art and graffiti writing regarding access, dissemination and mobility. Per definition urban art is strongly attached to a street environment, which is canvas and frame at the same time. This resulting immobility of urban art is in contrast with traditional art, where the materiality simplifies a display at alternating locations. Eventually, the paper highlights why urban art can only endure within a digital context. An example of Bristol-born artist Banksy (*1974) illustrates this: In 2015, he put up a stencil on a wall in Calais, depicting
The Raft of the Medusa

(Fig.1); only two years later, workers painted over the wall and covered Banksy’s work (Samuel, 2017). The project also establishes a data collection of urban art, consisting of reproductions from
Google Arts and Culture
, other image archives and a private collection by art historian and street art scholar Ulrich Blanché. Lastly, it demonstrates how computer-based tools are used to study images with regards to form and content. In this way, patterns over time and space or artistic networks are revealed and relations between artwork and urban environment can be evaluated. Therefore, the project team utilizes an interface, which was developed within the group and allows for a visual search based on multiple image regions in large image sets.

2. Evaluating street art and graffiti writing

In 2009, a collaboration between art history and computer vision was established within the Computer Vision group. Thus building a bridge between the two disciplines, which resulted in the realization of works, including the creation of an interface (Bell et al., 2014), reconstruction of drawing processes (Monroy et al., 2011) or the detection and analysis of gestures in medieval manuscripts (Yarlagadda et al., 2013), (Schlecht et al., 2011), (Yarlagadda et al., 2010). The group uses deep learning algorithms and unsupervised approaches to study visual similarities on image level (Bautista et al., 2017), (Bautista et al., 2016) and whole sequences (Milbich et al., 2017). The current project utilizes existing methods to study urban art. The presence of digital image collections of urban art and computational approaches enable both large-scale evaluations and detailed studies, which has not been done by scholars so far. Previous work mainly concentrated on terminology (Blanché, 2015), social aspects (Ross, 2016) or individual artists (Blanché, 2012), (Blanché, 2010), highlighted its mediality (Glaser, 2017) and generally justified its study in art history.

The presentation highlights the influence of digitization on urban art, describes the building of a suitable dataset and its evaluation through computational methods. (1) Digital possibilities have influenced all of humanities; for urban art, however, the effect is even more profound. Most traditional artworks are mobile; artists paint on canvas or paper, which allows for easy transportation and public display at various places. In this way, styles, content, or individual motifs spread and art reveals itself to be less bound to a specific place. On the contrary,

urban art is per definition tied to the street; its meaning only fully unfolds on site. The street not only provides a canvas, but also imposes form and additional meaning. As a result, urban art is greatly ephemeral: Works are being over-painted by authorities and artists (Samuel, 2017) – as the example of Banksy showed – or buildings are torn down. In reaction, works are increasingly documented and made available online.

Since its start in November 2017, the project has studied the presence of urban art on the
Internet. Its visibility on different websites has impacted the community and visuality of urban art: Communication between artists and fans has increased and is simpler, motifs are disseminated faster and wider, breaking national borders and indicating a tight network.

It is only through digital possibilities that urban art can be preserved and disseminated – this distinguishes it from traditional art.

(2) In order to study form and content of images with computer-based tools, the project gathers a dataset of urban art, providing metadata if available. Images are taken from
Google Arts and Culture
or
Facebook’s Global Street Art
. However, the project team also received a comprehensive set of photographs, capturing urban art in various cities worldwide between 2007 and 2017. All images were taken by art historian Ulrich Blanché; this unique data enables to address new questions regarding the capturing process: How did the photographic perspective and thematic focus change over time? Does it vary for different locations? Is there a correlation between alternating perspectives and Blanché’s social role? Eventually, he captured urban art first as a simple admirer, then as a student and finally as a scholar – although the first role persisted throughout time. The final image collection, including metadata, will be published and can be used by other scholars. A large number of images contain large context regions and objects, including buildings or cars. To improve performance and detection, the data was pre-processed: Around 200 images from the Blanché-dataset were annotated with bounding boxes marking artwork or context.

(3) The project studies the visuality of urban art on the basis of this image collection using computer-based methods. It aims to find recurrences and variances of a motif, ultimately not only pointing to the same but different artists. On a smaller scale, the example of Cologne street artist ‘kurznachzehn’ illustrates this: She uses old family photographs to create paste-ups, which she attaches to walls in various German cities. Her most recognized motif is a young girl – the artist’s mother as a child. The girl appears throughout her oeuvre in a similar pose but in varying scenarios: while picking up a dandelion (Fig.2), painting or feeding a little bird (Glaser, 2017), (‘kurznachzehn’, 2017). In order to study image collections, the project team utilizes unsupervised methods, which have been successfully applied on other tasks and do not rely on labeled data. This is valuable, since digital reproductions of urban art rarely have information regarding artist, title or creation date – this is mainly due to the anonymity of artists and legal reasons. Reproductions are evaluated on an interface, which not only allows to search for individual but also multiple regions and thus to consider geometrical relations between artworks and urban environment. The example of the dandelion-picking girl (Fig.2) by ‘kurznachzehn’ illustrates this: True to the nature of her gesture, she always appears close to the ground. Underlying algorithms use a SVM-classifier trained with one positive against many negative examples. While other retrieval systems require manual tagging, the algorithm purely operates on visual qualities. Currently state-of-the-art methods are being implemented, using CNN instead of HOG-features to train the classifier. Eventually, the interface not only detects identical motifs but also variations. First tests showed promising results; the project team studied images of artworks by Brazilian street artists OsGemeos. The user was interested in a figure seen from behind and a text region to its right; (Fig.3) shows the search results for the given queries after the second training round; the bottom row includes all correctly retrieved images as selected by the user. Results can now be analyzed regarding formal and semantic similarities or variances; also, it allows to evaluate the position of the motif in relation to the urban context. Future work should study the motif of the figure seen from behind also in the context of its general appearance in art history.

Applying computational methods to urban art data has emphasized chances and benefits, not only for art history but also for computer vision. Existing algorithms have been tested on challenging data and proofed their efficiency. However, working with urban art data has also highlighted some challenges: Collections are biased towards certain time periods, nationalities and dominated by works of popular artists. The new dataset, established within the project, is therefore extremely valuable. First tests, although overall successful, showed that algorithms are challenged by a dominating background, imaging mode (perspective) and the size of artworks. To remedy the latter, the project team decided to annotate part of the Blanché dataset with bounding boxes, which improved detection.

3. Conclusion

The presentation consists of two parts: A theoretical basis will be established in the first, discussing the influence of digital possibilities on aspects, such as mobility, access or dissemination, while the dataset will be introduced in the second half, which also includes presentations of search results on the interface. The project team aims to further establish urban art as a profound research topic in academia, point to new research questions and possible challenges when working with urban art data. Most importantly, the presentation emphasizes the chances offered by computer-based methods to study urban art in detail and on large-scale. (Words: 1485)

4. List of Illustrations

Fig.1: Banksy,
The Raft of the Medusa
, Calais, 2015

Fig.2: ‘kurznachzehn’,
Girl picking Dandelion
, Dusseldorf, 2013

Fig.3: Search results for figure and text to its right on interface; image collection of Brazilian street artists OsGemeos