Mind the gaps: cognitive-inspired AI for high-level visual sensemaking. Towards abstract concept image classification

The abundance of visual data and the push for robust AI are driving the need for automated visual sensemaking. Computer Vision (CV) faces growing demand for models that can discern not only what images "represent," but also what they "evoke." This is a demand for tools mimicking human perception at a high semantic level, categorizing images based on concepts like freedom, danger, or safety. However, automating this process is challenging due to entropy, scarcity, subjectivity, and ethical considerations. These challenges not only impact performance but also underscore the critical need for interoperability. This dissertation focuses on abstract concept-based (AC) image classification, guided by three technical principles: situated grounding, performance enhancement, and interpretability. We introduce ART-stract, a novel dataset of cultural images annotated with ACs, serving as the foundation for a series of experiments across four key domains: assessing the effectiveness of the end-to-end DL paradigm, exploring cognitive-inspired semantic intermediaries, incorporating cultural and commonsense aspects, and neuro-symbolic integration of sensory-perceptual data with cognitive-based knowledge. Our results demonstrate that integrating CV approaches with semantic technologies yields methods that surpass the current state of the art in AC image classification, outperforming the end-to-end deep vision paradigm. The results emphasize the role semantic technologies can play in developing both effective and interpretable systems, through the capturing, situating, and reasoning over knowledge related to visual data. Furthermore, this dissertation explores the complex interplay between technical and socio-technical factors. By merging technical expertise with an understanding of human and societal aspects, we advocate for responsible labeling and training practices in visual media. These insights and techniques not only advance efforts in CV and explainable artificial intelligence but also propel us toward an era of AI development that harmonizes technical prowess with deep awareness of its human and societal implications.

A Semantic-based Framework for Digital Survey of Architectural Heritage

The research aims at developing a framework for semantic-based digital survey of architectural heritage. Rooted in knowledge-based modeling which extracts mathematical constraints of geometry from architectural treatises, as-built information of architecture obtained from image-based modeling is integrated with the ideal model in BIM platform. The knowledge-based modeling transforms the geometry and parametric relation of architectural components from 2D printings to 3D digital models, and create large amount variations based on shape grammar in real time thanks to parametric modeling. It also provides prior knowledge for semantically segmenting unorganized survey data. The emergence of SfM (Structure from Motion) provides access to reconstruct large complex architectural scenes with high flexibility, low cost and full automation, but low reliability of metric accuracy. We solve this problem by combing photogrammetric approaches which consists of camera configuration, image enhancement, and bundle adjustment, etc. Experiments show the accuracy of image-based modeling following our workflow is comparable to that from range-based modeling. We also demonstrate positive results of our optimized approach in digital reconstruction of portico where low-texture-vault and dramatical transition of illumination bring huge difficulties in the workflow without optimization. Once the as-built model is obtained, it is integrated with the ideal model in BIM platform which allows multiple data enrichment. In spite of its promising prospect in AEC industry, BIM is developed with limited consideration of reverse-engineering from survey data. Besides representing the architectural heritage in parallel ways (ideal model and as-built model) and comparing their difference, we concern how to create as-built model in BIM software which is still an open area to be addressed. The research is supposed to be fundamental for research of architectural history, documentation and conservation of architectural heritage, and renovation of existing buildings.

Métamorphoses de l'Image des Tartares dans la littérature européenne du XXème siècle

Ce travail doctoral analyse le changement de l’image des Tartares dans la littérature européenne en langue allemande, anglaise, française et italienne du XXe siècle par l’étude de trois figures : la horde mongole, Gengis-khan et Khoubilaï-khan. Il soutient la thèse que, grâce à quelques facteurs historico-culturels comme la remise en question du concept de barbarie, l’essor des totalitarismes, l’ouverture de la Mongolie vers l’Occident, la redécouverte de l’Histoire secrète des Mongols et la fortune de Le divisament dou monde, au cours du XXe siècle, l’image littéraire des gengiskhanides de négative devient positive. Cette étude se compose d’une introduction, de trois chapitres et d’une conclusion. Dans l’introduction, on analyse la formation de l’image des Tartares et son évolution jusqu’à la fin du XIXe siècle, on retrace les facteurs historico-culturels qui la remettent au goût du jour et en provoquent le changement au XXe siècle et on présente le travail. Dans le premier chapitre, on se penche sur la prosopographie des Tartares dans les textes littéraires du XXe siècle, en la confrontant avec leur représentation dans l’art contemporain. Dans le deuxième chapitre, on étudie la façon des Tartares de se rapporter aux autres au sein de la société dans les textes littéraires du XXe siècle. Dans le troisième chapitre, on examine les lieux des gengiskhanides dans les textes littéraires du XXe siècle. Enfin, dans la conclusion, les données acquises au moyen de l’analyse conduite sont confrontées et interprétées. Le changement de l’image des Tartares va de pair avec une Europe qui, après avoir fait l’expérience de deux guerres mondiales, avoir assisté aux revendications de la décolonisation et avoir introjecté la thèse freudienne du « malaise dans la civilisation », remet en discussion sa façon de concevoir la barbarie et l’Altérité.

Searching and retrieving in content-based repositories of formal mathematical knowledge

In this thesis, the author presents a query language for an RDF (Resource Description Framework) database and discusses its applications in the context of the HELM project (the Hypertextual Electronic Library of Mathematics). This language aims at meeting the main requirements coming from the RDF community. in particular it includes: a human readable textual syntax and a machine-processable XML (Extensible Markup Language) syntax both for queries and for query results, a rigorously exposed formal semantics, a graph-oriented RDF data access model capable of exploring an entire RDF graph (including both RDF Models and RDF Schemata), a full set of Boolean operators to compose the query constraints, fully customizable and highly structured query results having a 4-dimensional geometry, some constructions taken from ordinary programming languages that simplify the formulation of complex queries. The HELM project aims at integrating the modern tools for the automation of formal reasoning with the most recent electronic publishing technologies, in order create and maintain a hypertextual, distributed virtual library of formal mathematical knowledge. In the spirit of the Semantic Web, the documents of this library include RDF metadata describing their structure and content in a machine-understandable form. Using the author's query engine, HELM exploits this information to implement some functionalities allowing the interactive and automatic retrieval of documents on the basis of content-aware requests that take into account the mathematical nature of these documents.

Ricerche sul lessico della provvidenza in Agostino

Questa ricerca è un’indagine semasiologica del lessico agostiniano della provvidenza divina, costituito dalle parole-chiave prouidentia, prouideo, prouidens, prouidus, prouisio, prouisor, prouisus, e dai lessemi in relazione logico-sintattica diretta con esse. La prospettiva è sia sincronica (si considerano tutte le attestazioni delle parole-chiave presenti nel corpus agostiniano), sia diacronica: si soppesano di volta in volta analogie e differenze agostiniane rispetto agli antecedenti, nell’intento di arricchire il panorama dei possibili modelli lessicali latini (pagani, biblici, patristici) di Agostino. I dati lessicali sono stati raccolti in una banca dati appositamente costituita, selezionati secondo i criteri di frequenza e pregnanza semantica, e analizzati per nuclei tematici, coincidenti in parte con i capitoli della tesi. Si studiano dapprima i lessemi che esprimono il governo della provvidenza (le famiglie lessicali di administro, guberno e rego, e altri lessemi che designano l’azione della provvidenza); sono poi analizzati lessemi e iuncturae in cui prevale l’idea del mistero della provvidenza. Gli ultimi due capitoli sono dedicati al tema della cura divina, e a quello della cosiddetta “pedagogia divina”: attraverso i segni esteriori, la provvidenza ‘richiama’ l’uomo a rientrare in se stesso. Un’appendice approfondisce infine l’uso agostiniano di Sap 6,16 e Sap 8,1. L’apporto di Agostino al lessico filosofico latino va individuato a livello semantico più che nell’innovazione lessicale. Accanto a suffissazione, composizione, calco, la metafora svolge un ruolo essenziale nella formazione del lessico dell’Ipponate, e proviene spesso da altre lingue tecniche oppure è radicata nel patrimonio di immagini tradizionali della religione pagana. Il debito di Agostino è indubbiamente verso Cicerone, ma anche verso Seneca, per l’uso in ambito esistenziale-biografico di alcuni lessemi. Agostino li trasferisce però dal piano umano a quello divino, come nel caso del concetto di admonitio: parte integrante del programma filosofico senecano; ‘richiamo’ della provvidenza per Agostino, concetto che risente anche dell’apporto di retorica ed esegesi.

Tensor-Train decomposition for image classification problems

In these last years a great effort has been put in the development of new techniques for automatic object classification, also due to the consequences in many applications such as medical imaging or driverless cars. To this end, several mathematical models have been developed from logistic regression to neural networks. A crucial aspect of these so called classification algorithms is the use of algebraic tools to represent and approximate the input data. In this thesis, we examine two different models for image classification based on a particular tensor decomposition named Tensor-Train (TT) decomposition. The use of tensor approaches preserves the multidimensional structure of the data and the neighboring relations among pixels. Furthermore the Tensor-Train, differently from other tensor decompositions, does not suffer from the curse of dimensionality making it an extremely powerful strategy when dealing with high-dimensional data. It also allows data compression when combined with truncation strategies that reduce memory requirements without spoiling classification performance. The first model we propose is based on a direct decomposition of the database by means of the TT decomposition to find basis vectors used to classify a new object. The second model is a tensor dictionary learning model, based on the TT decomposition where the terms of the decomposition are estimated using a proximal alternating linearized minimization algorithm with a spectral stepsize.

Understanding 3D structures from sketches

Sketches are a unique way to communicate: drawing a simple sketch does not require any training, sketches convey information that is hard to describe with words, they are powerful enough to represent almost any concept, and nowadays, it is possible to draw directly from mobile devices. Motivated from the unique characteristics of sketches and fascinated by the human ability to imagine 3D objects from drawings, this thesis focuses on automatically associating geometric information to sketches. The main research directions of the thesis can be summarized as obtaining geometric information from freehand scene sketches to improve 2D sketch-based tasks and investigating Vision-Language models to overcome 3D sketch-based tasks limitations. The first part of the thesis concerns geometric information prediction from scene sketches improving scene sketch to image generation and unlocking new creativity effects. The thesis proceeds showing a study conducted on the Vision-Language models embedding space considering sketches, line renderings and RGB renderings of 3D shape to overcome the use of supervised datasets for 3D sketch-based tasks, that are limited and hard to acquire. Following the obtained observations and results, Vision-Language models are applied to Sketch Based Shape Retrieval without the need of training on supervised datasets. We then analyze the use of Vision-Language models for sketch based 3D reconstruction in an unsupervised manner. In the final chapter we report the results obtained in an additional project carried during the PhD, which has lead to the development of a framework to learn an embedding space of neural networks that can be navigated to get ready-to-use models with desired characteristics.

Connecting works of art within the semantic web of symbolic meanings

My doctoral research is about the modelling of symbolism in the cultural heritage domain, and on connecting artworks based on their symbolism through knowledge extraction and representation techniques. In particular, I participated in the design of two ontologies: one models the relationships between a symbol, its symbolic meaning, and the cultural context in which the symbol symbolizes the symbolic meaning; the second models artistic interpretations of a cultural heritage object from an iconographic and iconological (thus also symbolic) perspective. I also converted several sources of unstructured data, a dictionary of symbols and an encyclopaedia of symbolism, and semi-structured data, DBpedia and WordNet, to create HyperReal, the first knowledge graph dedicated to conventional cultural symbolism. By making use of HyperReal's content, I showed how linked open data about cultural symbolism could be utilized to initiate a series of quantitative studies that analyse (i) similarities between cultural contexts based on their symbologies, (ii) broad symbolic associations, (iii) specific case studies of symbolism such as the relationship between symbols, their colours, and their symbolic meanings. Moreover, I developed a system that can infer symbolic, cultural context-dependent interpretations from artworks according to what they depict, envisioning potential use cases for museum curation. I have then re-engineered the iconographic and iconological statements of Wikidata, a widely used general-domain knowledge base, creating ICONdata: an iconographic and iconological knowledge graph. ICONdata was then enriched with automatic symbolic interpretations. Subsequently, I demonstrated the significance of enhancing artwork information through alignment with linked open data related to symbolism, resulting in the discovery of novel connections between artworks. Finally, I contributed to the creation of a software application. This application leverages established connections, allowing users to investigate the symbolic expression of a concept across different cultural contexts through the generation of a three-dimensional exhibition of artefacts symbolising the chosen concept.