by Alexander Shvets, researcher at Pompeu Fabra University
The recent advances in 3D-reconstruction of buildings are difficult to overestimate: a decent model can be created out of a bunch of 2D images with a detailed object segmentation and preserved aesthetics and styles. Even more impressive the way streaming technologies work as V4Design platform does: it continuously crawls the Web and generates in an autonomous way new models of different buildings that are ready to be ported as an asset into CAD libraries and VR applications. This creates a unique opportunity for designers to observe architectural solutions from various angles and perspectives that empowers their research capabilities and boosts the creative process. The novel problem comes when being overwhelmed by the number of models the user needs a description of what actually was modeled:
- What are the key components of an asset to pay attention to?
- What are the distinctive aesthetics of a particular building?
- Which variations of a particular object exist and how to find the examples in the library when an object of interest is just a small part of a building?
Here the place where textual analysis gives a hand to a designer. Specifically, the crucial role is played by the concept extraction techniques.
Formally, the task of concept extraction consists in the detection of signifiers – minimal indivisible units inside the text that correspond to real-world entities. Knowing the exact span of a signifier allows distinguishing words that are used right to define a concept from words-modifiers that are used to describe features of a concrete real-world instance of a concept (for example, “twin D-shaped” is a modifier for the concept “gatehouse keep” highlighted in a fragment of text below[1]). These words-modifiers are key to detect the architectural characteristics of objects.
The example above was not selected at random: a collection of castles is one of the main collections in V4Design. These buildings are especially complex to work with as they have many distinct elements that vary in many features so that it is often tremendously difficult to capture commonalities or dissimilarities just visually.
The texts about castles are not simpler either as they contain specific terminology and some words are used in unusual for a language way (e.g., “keep” atypically frequent appears as a noun) that may confuse general-purpose natural language processing engines.
UPF tackles the challenge of text understanding and develops a deep learning-based model especially for the task of concept extraction in several languages, namely, English, German, Spanish, and Greek among others.
In what follows, we show how applying the universal model trained on a large open-domain dataset helps to carry out a study of individual assets and an explorative analysis of the overall set of castles from the architectural point of view.
To help study individual objects, we start from captions of the 2D images used for the reconstruction. Detection of concepts in captions with a lack of linguistic information is a challenging task on its own. Once concepts have been extracted, the corresponding Wikipedia pages are sought for. Further processing of found texts allows us to provide to a user additional information about the object.
To allow for the explorative analysis, we pre-processed 17K Wikipedia pages with the word “Castle” and a few other related keywords in the title. We applied asentence-wise topic modeling[2], firstly, to discard pages with irrelevant content (e.g., “HMS Pretoria Castle (F61)” about escort aircraft carrier of the Royal Navy),secondly, to separate sentences describing historical events from sentences with architectural characteristics. The latter topic formed a single cluster of 8K sentences with 23K concepts extracted automatically. The top-50 frequent single-word and multiple-word signifiers are shown in the following table.
| Language | Concepts |
| English | keep, tower, style, walls, building, moat, outer, chapel, bailey, gatehouse, house, architect, stone, Renaissance, curtain wall, round, design, Gothic, motte, donjon, site, courtyard, side, windows, gate, ditch, construction, entrance, structure, enceinte, drawbridge, tower house, inner bailey, park, interior, palace, rooms, Romanesque, front, roof, work, Baroque, medieval, storeys, parts, area, corner, gardens, hall, 16th century, fortifications, great hall, top, 13th century, ruins, ward |
How to make use of the obtained list of concepts? Let us conduct a small research on towers that are typical elements of a castle.
To find different types/classes of towers we observe multiple-word concepts with the word “tower” as a head. They might be separated into two groups: a) typified by the purpose (staircase tower, gate tower, defensive tower, clock tower, watch tower, mural tower, flanking tower, bell tower, guard tower, fortified tower, drum towers, artillery towers, observation tower, fighting tower), b) typified by the position in a castle that, by the way, also defines their purpose to some extent (entrance tower, main tower, central tower, corner tower, side tower).
Along with the concepts, we can extract modifiers that usually stand in front of the corresponding signifiers in a text: they define the shape (conical, circular, round, square, octagonal, rectangular, cylindrical), materials (stone, wooden), and relative location (south, east, south-east, north-west, southwest, south-western).
Now we can find different castles with a specific type of a tower with desired aesthetics. For example, going back to the pages where we detected a staircase tower of octagonal shape will give us various examples of how this object can be designed. As it is seen from the pictures, the castles located in different countries might look substantially dissimilar even though they share the common set of components and have a similar history: both buildings were built in the 14th century and remodeled in the 16th century.
It is worth mentioning that the traditional textual search would not allow us to obtain such results: the fuzzy search returns among others castles with staircase tower and some other octagonal tower while the exact search for “octagonal staircase tower” misses the instances with additional modifiers as in “octagonal spiral staircase tower”. Furthermore, it is unlikely that the image similarity search will put the considered images nearby as very particular details are concerned. Object localization approaches also have a limited chance to perform well as the staircase tower is hardly seen on the right picture and its shape even less recognizable.
Let us consider a few more cases of concept-based search. The pages with signifiers of a concept “mural tower” provide information about castles that are currently in a state of ruins as this type of towers mostly refers to the 13th century.
Survey of castles with so-called drum towers that are of the round shape and have a diameter of a comparable (often longer) length to the height gains examples of how these towers might be positioned and decorated. They might be entrance towers or side towers (the top images in the following figure). They may have different shapes even within one ensemble as shown on a bottom-left image. They are often coupled closeby although they might be located quite far from each other and be situated on a very complex landscape despite their width as on a bottom-right figure. No doubt, this gives a solid ground for experimentations with the design!
One may argue that some of these groups of castles might be found on a dedicated page with a list of links to the corresponding Wikipedia articles. This is indeed the case sometimes. However, even when this is the case, the list is not exhaustively full. For example, the list of castles with battery towers[3] misses some castles, namely, for example, Steckelberg Castle and Grenzau Castle
Finally, it is worth mentioning that the developed by UPF model for concept extraction covers all the four languages considered in V4Design – this is beneficial for the description of reconstructed 3D objects as Wikipedia pages in a language native for a place where the object is physically located often provide more additional information to study all the possible limitations that opens even more potential for the creativeness.
Why not to choose the next location for filming Game of Thrones using the V4Design platform?
To know more: alexander.shvets@upf.edu, montserrat.marimon@upf.edu, leo.wanner@upf.edu
[1] The fragment was taken from the page https://en.wikipedia.org/wiki/Aberystwyth_Castle. The highlighted concepts have corresponding Wikipedia pages
[2] We used a deep learning model, which returns interpretable and diverse topics https://www.aclweb.org/anthology/P17-1036/