1.
Image Processing Pipeline

1.1.
Photo/Cardboard Extraction

The records in the Cini Foundation consist of a photographic reproduction mounted on a cardboard card onto which metadata information is recorded. The initial scan of these records is a 300 dpi picture produced on a scanning table, and includes the digitized cardboard and color balance markers. The first task consists in separating the cardboard backing and the photographic reproduction from the raw scanned image.

Despite the apparent simplicity of such a task, it proved challenging on account of the multiple layouts of the metadata information on the cardboard cards, and the variations in the sizes and positions of the attached images. In the end, what proved most effective in the extraction of the image was a Convolutionnal Neural Network (CNN) architecture designed for semantic segmentation (Ronneberger, O. et al 2015). For this, an accurate model was trained on scans which had been annotated in the course of 2 hours. The details oft he approach are part of another study (Ares Oliveira, S. and Seguin, B. 2018).

Figure 1. Figure Left: original scan with the extracted areas highlighted with red and blue rectangles.

Right: the prediction mask generated by the neural network.

1.2.
Text Extraction

The second part of the pipeline consists of extracting and reading the metadata. For this task, the open-source Tesseract toolkit and the commercial Google Vision API were tested, with the latter having better performance.

The OCR system provided a list of words and their positions, which were then clustered into blocks of text representing the different metadata fields (authorship, title of painting, location etc.). A layout model was used to represent the expected positions of these different fields. This allowed the assignment of each block of text to its corresponding metadata field.

A precise analysis of the performance of this step is presented in another publication (Seguin, B. 2018).

Figure 2. Figure Illustration of the OCR process. The extracted words (top-left) are clustered into blocks of metadata (top-right) and then assigned to their corresponding label (bottom).

2.
Automatic Alignment of Artist Names

In order to leverage the extracted metadata to get insights into a collection, it is important to link them to a knowledge database. This can allow, for example, city names to be placed geographically on a map. Here, we focus on aligning artist names with a knowledge database: the Union List of Artist Names (ULAN), managed by the Getty. This opens up a wealth of new information for the contextual understanding of the artwork’s creation.

The alignment process is depicted on Figure 3, it is a complex two-pass process that integrates automatic matching with collection specific knowledge in an efficient manner. The first pass tries to perform an exact match with a large name dictionary. For the second pass, a list of candidates are generated from the correctly matched elements of the first pass, and approximate matching is used to correct small OCR errors.

Figure 3. Figure Alignment process. The parts in color correspond to collection-specific knowledge.

There are three challenges that needed to be tackled during this alignment process :

• Names variation : one major issue that arises is that a given artist may be called by different names, depending on regional variations and pseudonyms. Many variations are recorded in ULAN (i.e “
  Tiepolo Gianbattista” and “
  Tiepolo Giovanni Battista” both corresponding to the same artist), although some have to be added to the name dictionary. Furthermore, the naming conventions for elements whose dating or provenance is known but not authorship, which may be specific to a collection, can be added to the dictionary.
• Implicit knowledge : one related challenge is linked with the pragmatics of the annotation process. Understanding that if one archivist writes “
  Leonardo” on a file, he or she is referring to
  Leonardo da Vinci implies modeling a series of implicit assumptions which are changing depending on the evolution of local cataloging practices and that of the art historical field itself. In our case, we tackle this by disambiguating unclear names. For instance “
  Tiziano Vecellio” could technically refer to the well-known “
  Tiziano”, or his relative “
  Tizianello”, but the first is much more prominent than the second.
• Compositional structure : the last challenge is linked with the practice of archivists to describe particular unknown authors using specific syntactic process like (“
  Tiziano (bottega di-)”, “
  Tintoretto (Maestro di)” or “
  Michelangelo (copia da-)”), referring to workshop productions or copies. Understanding and modeling this “grammar” permits to generate, in a compositional manner, potential matching strings to be considered when looking for possible alignments. Such strings do not only give a link to an artist but also qualify relationships (how strongly an artist was involved in the creation process of a painting, whether the piece is an original or a copy, etc.).

3.
Results

Figure 4. Figure Left : Distribution of number of artworks assigned for each artist.

Figure 4. Right : Proportion of images assigned with respect to the most common artists. The 200 most represented artists represent 43% of the collection.

Of the 330,078 scans composing the corpus of study, 14.6% had an empty author field, mostly because the photographs represented architecture or aerial city views. Out of the remaining 85.4% with an authorship field, 73.8% were automatically matched to an author (61.6% after the first pass), with an additional 1.4% representing ambiguous situations which could be resolved. This accounts for 208’510 elements automatically matched. At the end of pre-processing, the potential author names can be divided into three categories :

• (A) Author names which have been matched with a reference record of another database
• (B) Author names which may have been matched if the algorithm were to be improved (e.g. in terms of author name variation or possible compositional structure)
• (C) Authors undocumented in standard databases of artists.

Figure 5 shows the global matching results for category A. The geographical composition of aligned authors is dominated by Venetian artists (Tiepolo, Tintoretto, Palladio, Tiziano, Veronese, etc.) showing the rationale behind the creation of the collection. In terms of chronology, the collection is focused on the sixteenth century, as shown by the distribution of year of death of the aligned artists. This is in line with the period referred to as the “Venetian Golden Age”. Figure 4 shows the very uneven representation of artists, with only 346 having more than 100 images, representing more than 50% of the whole collection.

Figure 5. Figure Spatial (right) and temporal (left) distribution of the 1’746 artists with at least 10 images assigned.

Category B is predominant in the elements that were not matched. Apart from OCR errors, the most typical unmatched string corresponds to collective works in which several authors are named. For instance, the string “
Bassano Jacopo e Francesco” (his son) corresponds to 134 records. Adding additional parsing capabilities to the system could enable the resolution of such cases in the future.

Names in category C, which were not matched with ULAN, are in fact not a product of misalignment but represent new discoveries in the collection. In the present study, a number of artists who do not feature in ULAN were uncovered in the Cini archive. These include, Augusto Caratti, a minor artist from nineteenth-century Padua, who is represented by 65 images in the Cini collection, and Natale Melchiori an early eighteenth-century painter from Castelfranco, Veneto, represented by 39 images. Another artist who does not feature in the ULAN database but nevertheless has a significant presence in the Cini archive with 106 drawing, is Antonio Contestabile, an eighteenth-century draftsman from Piacenza.

4.
Conclusion

These early results show the potential of the systematic processing of a large number of art historical records, leading to the mapping of unknown collections, and to new discoveries. It also highlights for the first time the challenges inherent in the process. Such challenges, it is important to note, are not purely technical but rather linked with the complexity of modeling local archiving traditions and the historical practices of art history.