Seminars

Forests cover 31% of the Earth's surface and play a vital role in the sustaining life on the planet. Remote sensing is used to efficiently monitor forest health at scale. Many estimated forest parameters such as biomass can be more accurately estimated when species are known or individual tree crowns can be counted. Therefore, crown delineation and species classification are important to remote sensing forest parameter estimation. However, creating robust machine learning classifiers to delineate crowns and classify species is a difficult task. In this talk we discuss how neuro-symbolics can be leveraged to improve performance of remote sensing based tree species classifiers.

In-database analytics has become one of the most studied topics in data science community, because of its significance in reducing the gap between the management and the analytics of data, which can save much time on exchanging data between databases and external analytic tools. But implementing in-database algorithms inside mainstream databases without pre-verification of the ideas is risky and may result in a significant waste of time. In this talk we present a testbed, DBSim, which simulates a relational database and allows users to easily extend it based on their needs, such that users can fast prototype their ideas and estimate the performance before diving into the large codebase of real RDBMS.

Continuation of neuro-symbolics applications.

Similarity queries are becoming increasingly important in the age of big data where the data objects do not possess any natural order. Examples include large collections of images, text and multimodal data objects. Embeddings are widely used in processing of similarity queries for measuring semantic similarity of data. However, current database systems cannot adequately support embedding based similarity query processing. Furthermore, multiple types of embeddings may be constructed over data objects that contain multiple modalities, such as text and image, where similarity-based join queries are required but the traditional join operators do not work well on these queries.
So we propose SystemV, a generic platform on top of Spark, providing scalable end-to-end solutions for efficient embedding based similarity query processing, including search on single embedding space and join on multiple embedding spaces. The goal of SystemV is to democratize AI by facilitating the use of pre-trained embeddings (e.g., learned from large amounts of data such as Wikipedia and ImageNet). Using such pre-trained models, users can focus more on their core application rather than designing the retrieval models.

Motivated by human's ability of learning visual concepts by jointly understanding vision and language, a recent work proposes the Neuro-Symbolic Concept Learner (NS-CL), a model that learns visual concepts, words, and semantic parsing of sentences without explicit supervision on any of them; instead, our model learns by simply looking at images and reading paired questions and answers. This work has some similarity with previous works from our lab focusing on information extraction with multimodal approaches. In this talk I will include these works and discuss the similarity and distinction between them for possible future research.

Electronic Health Record (EHR) is an electronic version of a patient’s medical history. The state-of-the-art model- ‘Text to SQL Generation for Question Answering on Electronic Medical Records’ has some major shortcomings. The accuracy of this question answering system on MIMIC III database is very poor. It is very challenging to obtain a good accuracy in multi-relational health records. In this project, Relational aware schema encoding and Linking for Text to SQL (RAT-SQL model) has been used for question answering over structured medical records. The talk would also cover the annotation of the drug dataset for question answering over structured and unstructured clinical records. Furthermore, this QA system could aid in applications like Sepsis phenotype detection and treatment prediction.

In this talk, we will talk about two of the main reasoning methods on knowledge graphs, rule learning, and embedding models. We will briefly cover how KGs are constructed and why reasoning on KGs is crucial for both completing the KG and using them in applications. We will then discuss differentiable rule mining and the benefits of rules as an interpretable way of reasoning. Finally, we will discuss embedding based methods on KGs and dive deeper into recent advances of these latent models in temporal KG completion.

Knowledge bases (KBs) present large amounts of data in a structured manner. Because of the amount of information in them, there has been much work on querying these structured repositories of data using natural language. This work extends to complex question-answering (complexQA) where the complexity of question increases by mentioning multiple entities, relations, constraints, or all three.

In this talk, we discuss the problems with current complexQA systems over knowledge graphs and motivate our work in KB-QA narration. Namely, we differentiate our problem from conventional KB-QA by observing questions that cannot be solved well given the current approaches. We first introduce EventKG in order to motivate our work by giving examples of the complexQA narrative task over this dataset. To solve this new problem, we review some work in both complexQA and graph-to-text and show how our task is a combination of these two. Next, we discuss our baseline approach in generating answer narratives. Finally, we present our approach to building a novel dataset for the complex question-narratives task and discuss possible models that generate narratives from questions and KB components. This talk will serve as an introduction to works in both the KB-QA and graph-to-text translation tasks and as our roadmap to solving the new problem of KB-QA narration.

The widespread adoption of electronic health records (EHRs) has enabled the secondary use of EHR data for clinical research and healthcare delivery. An Electronic Health Record (EHR) is an electronic version of a patient’s medical history and may include demographics, progress notes
problems, medications, vital signs, past medical history, and laboratory data. Natural language
processing (NLP) techniques have been used over the years to develop question answering
systems over structured as well as unstructured electronic health records. The following two
papers will be presented:
The paper - ‘A Translate-Edit Model for Natural Language Question to SQL Query Generation
on Multi-relational Healthcare Data’ develops a deep learning-based approach that can translate
a natural language question on multi-relational EHR data into its corresponding SQL query,
which is referred to as a Question-to-SQL generation task. To address the challenge of
generating queries on multi-relational databases from natural language questions, TRanslate-Edit
model for Question-to-SQL query (TREQS) is proposed.
The paper- ‘CREATE: Cohort Retrieval Enhanced by Analysis of Text from Electronic Health
Records using the OMOP Common Data Model’ presents a cohort retrieval system that can execute
textual cohort selection queries on both structured and unstructured EHR data. CREATE is a
proof-of-concept system that leverages a combination of structured queries and IR techniques on
NLP results to improve cohort retrieval performance while adopting the Observational Medical
Outcomes Partnership (OMOP) Common Data Model (CDM) to enhance model portability.

Innovative approaches to Information Extraction(IE) have enabled the creation of large Knowledge Graphs (KGs) (e.g., YAGO, NELL, DBPedia, Wikidata) and Dynamic Knowledge Graphs (e.g., ICEWS, GDELT). These knowledge graphs have become an increasingly popular domain knowledge representation used in semantic search, recommendation systems, question-answering, natural language processing, etc.

Despite the increased efforts, KGs are still predominantly incomplete and contain a high degree of uncertainty. In this dissertation, we study two approaches to fill in the gaps in existing knowledge graphs

1) Reasoning over facts from different knowledge graphs and/or extractors to increase coverage and automatically evaluate the correctness in the aggregate, this task is known as Knowledge Fusion (KF). We propose CM Fusion, an ensemble approach that combines supervised learning and unsupervised consensus. We also propose SigmaKB, a query engine that uses CM Fusion and user feedback to integrate and improve query results compared to single KGs.

2) Reasoning over existing facts to determine possibly missing ones, moreover we propose to do this over dynamic knowledge graphs. We propose to learn observed feature models over dynamic knowledge graphs in the form of sequential rules and use them to infer missing facts and forecast events in a streaming fashion.

CM fusion uses the Consensus Maximization algorithm to ensemble supervised classifiers learned using information extractors that can be assessed a priori and generates consensus with black box extractors that can not be assessed beforehand. Consensus Maximization Fusion is able to promote high-quality facts and eliminate incorrect ones. We demonstrate the effectiveness of our system on the NIST Slot Filler Validation evaluation, which seeks to evaluate and aggregate multiple independent information extractors. Our system achieved the highest F1 score relative to other system submissions.

We mine temporally constrained first-order inference rules over dynamic knowledge graphs. The learned rules are the first set of temporal rules mined over dynamic knowledge graphs. The algorithm we propose uses adjusted definitions of support, confidence and head coverage metrics that consider minimal occurrences and time windowing constrain. We ground the learned sequential rules over dynamic knowledge graphs in two tasks: temporal link prediction and streaming link prediction. Our experiments show that inference using sequential rules can outperform representation learning approaches while at the same time yield interpretable patterns. These patterns can be further refined by experts or used by analysts in QA tasks. Finally, we show that rules and embeddings complement each other and propose to cast the ensemble as a ranking aggregation problem. We Reciprocal Rank Fusion, an unsupervised rank aggregation model.

In this talk, we will discuss StarSpace, a method that can be used to embed multiple types of objects and solve a variety of tasks such as labeling tasks, ranking tasks, collaborative filtering-based or content-based recommendation, embedding of multi-relational graphs, and learning word, sentence or document level embeddings.

The following two papers will be presented- ‘Automated template generation for question answering over knowledge graphs’ and ‘Never-Ending Learning for Open-Domain Question Answering over Knowledge Bases’. They both emphasize on different template-based approaches to handle complex question answering systems in knowledge graphs.

Templates are an important asset for question answering over knowledge graphs, simplifying the semantic parsing of input utterances and generating structured queries for interpretable answers. The paper ‘‘Automated template generation for question answering over knowledge graphs” presents QUINT, a system that automatically learns utterance-query templates solely from user questions paired with their answers. Additionally, QUINT uses basic templates to answer structurally complex compositional questions without observing such questions during training.

The paper - “Never-Ending Learning for Open-Domain Question Answering over Knowledge Bases” presents NEQA, a continuous learning paradigm for KB-QA. Translating natural language questions to semantic representations such as SPARQL is a core challenge in open-domain question answering over knowledge bases. The existing methods require access to a large annotated training set that is not always readily available and fail on questions from before-unseen domains. In this paper NEQA when offline, automatically learns templates mapping syntactic structures to semantic ones from a small number of training question-answer pairs. If a new question cannot be satisfactorily answered via templates, user feedback is used on the output of a semantic similarity function to learn a new template based on the new question.

We are going to present two papers: CognitiveDB and RankSQL. They are relevant to supporting AI in DBMS and efficient execution of top-k queries in DBMS. These are two popular topics currently in the database community.

CognitiveDB is an approach for transparently enabling Artificial Intelligence (AI) capabilities in relational databases. A novel aspect of their design is to first view the structured data source as meaningful unstructured text, and then use the text to build an unsupervised neural network model using a Natural Language Processing (NLP) technique called word embedding. This model captures the hidden inter-/intra-column relationships between database tokens of different types. They seamlessly integrate the word embedding model into existing SQL query infrastructure and use it to enable a new class of SQL-based analytics queries called cognitive intelligence (CI) queries. CI queries use the model vectors to enable complex queries such as semantic matching, inductive reasoning queries such as analogies, predictive queries using entities not present in a database, and, more generally, using knowledge from external sources. This system exemplifies using AI functionality to endow relational databases with capabilities that were previously very hard to realize in practice.

RankSQL is a system that provides a systematic and principled framework to support efficient evaluations of ranking (top-k) queries in relational database systems (RDBMS), by extending relational algebra and query optimization. They aim to support ranking as a first-class database construct. So they extend relational algebra by proposing a rank-relational model to capture the ranking property and introducing new and extended operators to support ranking as a first-class construct. Enabled by the extended algebra, they present a pipelined and incremental execution model of ranking query plans (that cannot be expressed traditionally) based on a fundamental ranking principle. Their approach can significantly reduce the cost of ranking operation in DBMS.

Users often converse with chatbots in order to ask task-specific questions. Much of the information is only accessible in databases which requires one to know a query language. Thus, when building a conversational question answering system it is crucial to converting natural language questions into query language, in order for users to obtain answers to their questions. We approach this problem in two different domains and utilize two types of query languages: SQL and SPARQL. Furthermore, we develop a novel data collection methodology in order to generate synthetic data for which to use on the natural language to query language tasks. With this methodology, one can quickly and efficiently generate data to use on their models for their specific domain needs. We show promising results for our synthetically generated datasets and present further steps needed for future work.

Question and answering systems remain relatively unexplored in clinical domains. This paper proposes a novel methodology to generate domain-specific large-scale question answering datasets and demonstrates an instance of this methodology in creating a large-scale QA dataset for electronic medical records. The method creates questions and logical form templates obtained through expert annotations and with existing annotations in clinical notes it generates questions, logical form, and answers. The dataset’s learning potential is explored by training baseline models for question to logical form and question to answer mapping.

Research efforts in reasoning over large scale Knowledge Bases (KBs) such as Freebase, YAGO or NELL has largely focused on static representations of knowledge. The recent availability of time annotated KB facts in YAGO and Wikidata and timed stamped Event Knowledge Bases (EKBs) GDELT and ICEWS has ignited efforts in developing models that also reason over the temporal dimension. The majority of work in this research area is centered in representation learning for the link prediction task. In contrast, in this paper, we study the problem of learning first-order inference rules where the rule atoms occur in sequential order. In particular, we propose an algorithm to mine sequence rules from temporal knowledge bases and interestingness metrics that take into account time windowing constraints. Our experiments show that interpretable patterns can be mined from ICEWS and further used by human analysts and improved by experts.

There has been a lot of work on simple question-answering over structured knowledge bases. That is, answering questions which contain only one relation between two entities. Though there has been a lot of development in simple question-answering over structured data by using templates, syntactic parsers, or language models. In this talk, we present our progress towards implementing the approach used by Question Answering Over Knowledge Graphs: Question Understanding Via Template Decomposition. We show experimental results thus far on the WebQuestions dataset (simple questions) and describe some of the challenges encountered during the process. We also discuss future steps and some statistics of the natural language pattern question templates.

Question and answering systems remain relatively unexplored in clinical domains. This paper proposes a novel methodology to generate domain-specific large-scale question answering datasets and demonstrates an instance of this methodology in creating a large-scale QA dataset for electronic medical records. The method utilizes existing expert annotations on clinical notes for various NLP tasks from i2b2 datasets. The corpus contains question templates and logical forms and question-answer pairs. The dataset’s learning potential is explored by training baseline models for question to logical form and question to answer mapping.

Logical rules play an important role in capturing interpretable patterns present in knowledge bases. An extensive body of research has focused on automatically discovering first-order logical rules by exploring a discrete search space for the rule structure and the applying statistical measures to assess the correctness of each rule. In this talk, we consider a new parametric approach for mining logical rules from knowledge graphs. We proof the limitations of the state-of-the-art differentiable technique for mining logical rules and propose a method that (1) Has no theoretical restrictions on the structure of meaningful logic rules, (2) Adaptive to the data, and (3) Avoids over-parametrization while finding meaningful signals in the data. We show that our method outperforms existing rule mining methods in the quality of the mined rules over benchmark datasets as well as in link prediction task.

In this talk, I will go over my work on a question answering problem, involving tutorial videos. Previous works have focused on generating short responses or factoids as answers to a user’s question. Here, the answer is in the form of a span (from a video segment) where the length of the answers are usually multiple sentences long. To model and accomplish this task we use a dataset, containing the video transcript, question, and answer. To conclude, we apply some baseline models on our task and present some future directions for the task.

This talk is going to focus on my previous work on Inconsistency Detection for the M9 evaluation of the AIDA project. After outlining the general AIDA project scenario and motivating the hypothesis generation assumption, I will give a brief introduction on what should be considered as an inconsistency, and which metrics have been already developed in current literature to summarise such inconsistency information. I also present the difference between the standard definition of MultiValued Dependencies with "equality" and their generalisation with "relatedness" and "is-a" relationships. I provide the benchmarks and the quality measures for both approaches and outline some future work directions.

There has been a lot of work on simple question-answering over structured knowledge bases. That is, questions which contain only one relation between two entities. Though there has been a lot of development in simple question-answering over structured data--by using templates, syntactic parsers, or language models--solving complex questions is still an ongoing issue. Recent work has delved into complex question-answering where there can be multiple relationships between more than two entities. For example, the question "Where was the wife of the US president born?" is a complex question that can be divided into multiple simple questions using multiple relations. In this talk, we look into recent research which has dealt with complex questions (those with more than one relationship) by using template based methods in order to formulate a formal query from a natural language utterance. Another method which will examine is that of state-transition approach. This translates a natural question into a semantic query graph to find answers in a KB. We will also briefly go over simple question cases in order to build a basis from which the complex questions are formulated into formal queries. This talk serves as a survey of the work for complex question answering over KBs and will motivate future work in improving the current state-of-the-art and developing new methods.

In the last decades ecology has increasingly become a data-intensive discipline, whose challenges inherently overlap with data science problems in a real world complexity scenario. For example, in order to forecast how tree species distribution and productivity change in uncertain future, we need to develop generalized methods to extract information from big data, account for the uncertainty in the data source, integrate different sources into cross-scale models, and formally link biogeochemical knowledge to observed patterns from small unbalanced training sets. These real world challenges represent cutting edge fundamental problems that the data science community has started recognizing. In this seminar we will address what our interdisciplinary group at UF has done to address such methodological and biological issues. First, we built a Data Science Evaluation Series aiming to predict species labels for each individual tree at scale of thousands of hectares from remote sensing data. Second, we built a fully bayesian hierarchical model (GJAM) on a dataset of millions of individual trees sampled from a coarse grid across the country, to identify rules for how those species are distributed. The final goal is to merge these products to understand how those rules change with scale.

We provide an overview of HypoGator, our hypotheses generation system. HypoGator relies on the KB constructed from combining and aligning of the document level knowledge elements. Using multiple features and inconsistency detection methods, it extracts multiple coherent and consistent hypotheses. It finally returns a sorted list of the alternative hypotheses relevant to the query. In this talk, we describe the major components of our system and present our experiments and results. Initial analysis of our results provides insight into some of the things that can be done by TA2-TA1 to help improve the generation of the hypotheses.

Present day web-scale knowledge bases (KBs) incorporate a substantial amount of information in a structured format. Availability of this readily machine-digestible data has made KBs a desirable resource for other applications. This has motivated many to explore learning from KBs. Embedding methods and learning inference rules are examples of such methods. Rules provide great inference power and are also easily understandable. Most recent work focuses only on normal rules (where all the predicates only support variables). We explore conditional inference rules, a class of logical rules which allow predicates with constants and have more expressive power. We show their effectiveness in knowledge expansion by comparing to normal rules’ number of predictions and precision. However, due to the larger search space, mining conditional rules is much more time-consuming compared to mining normal rules. Current state-of-the-art rule mining methods adapted to mine conditional rules, are infeasibly slow on medium/large KBs. To aid with this shortcoming, we introduce a scalable conditional rule mining algorithm. Our algorithm makes it possible to mine conditional rules from web-scale KBs.

Narrating structured data with a paragraph of text remains a challenging problem. In this presentation, we examine recent efforts to tackle the challenge of natural language generation from Wikipedia tables, focusing on Wang et al.'s paper, Narrating a Knowledge Base. We begin with a brief review of seq2seq neural networks, next investigating how Wang et al. successfully applied multiple types of self-attention to increase the length and completeness of their Wikipedia summaries. Finally, we assess the successes and failures of their method and propose future research directions.

Current literature faces three main problems with inconsistency detection. First, theoretical approaches treat all the entities that are not "syntactically" the same as being not equal and perform query answering with the repair-then-query approach. These constraints are too naïve for real-world data: data may contain different descriptions at different abstraction levels, and biased data sources will not allow us to discriminate which is the actual correct answer. Second, traditional FOL system cannot cope with inconsistencies in the reasoning process, because the principle of explosion allows drawing any possible conclusion from inconsistent facts. Third, traditional SRL models using FOL logic may be affected by the same problem, thus allowing to infer implausible hypotheses with near to zero scores. This presentation will focus on solving the first problem by using hierarchies to define inequality, and on how to detect inconsistencies in data using external validation. For the second problem, we will briefly introduce paraconsistent logics that refute the contradiction principle and can be exploited to reason with inconsistent data. We will leave the third problem for future work on SRL paraconsistent models.

In this talk, we study the integral projection model (IPM) for analyzing size demography of ecological systems. First, a basic version of IPM is introduced to model ecological dynamics. Further, the IPM is extended to include climate factors, such that it can be scaled to broader geographic areas. Finally, the effectiveness of IPM in investigated upon the FIA dataset with focus on two example species from the year 2015 through 2010.

Knowledge bases are very useful to store complex structured and unstructured information. The rise of the internet has given rise to huge knowledge bases. However, with the humongous amount of information we have, it creates a need to generate meaningful insights to understand connections between entities. In this work, the ICEWS data was modeled as a knowledge graph. The knowledge graph was further enhanced with entities from Wikidata. Since the two data sources have been curated differently, one of the major problems was aligning the entities to have a connected graph. The alignment was done by calculating similarity scores using n-grams and using rules mined from the data. Using the knowledge graph, a hypothesis has been considered as a weighted path between two entities. Here, the approach to creating the knowledge graph has been presented, along with a comparison of different entity alignment techniques. Using the knowledge graph, hypotheses were generated and ranked for multiple scenarios. The observations and future work for the same has been shared.

We are faced with an explosion of information (and misinformation) published through different mediums like blogs, youtube videos, newspapers, audio podcasts, etc. Knowledge bases have proven to be a great way to store complex information in a semi-structured way.
One can process and convert all the information from the mentioned mediums into a single KB. The big question now is how to generate “good” hypothesis about different ongoing scenarios from this KB. To answer this, one must first define what a “good” hypothesis is? In this presentation, we will assume that a KB is built from the mentioned different mediums and that there exists a system that generates hypothesis from the KB in the form of subgraphs. We will give a definition of a good hypothesis based on Grice’s maxims and propose different ways of evaluating systems that mine hypothesis from KBs.

An insider threat is a malicious threat to an organization that comes from people within the organization, such as employees, former employees, contractors or business associates, who have inside information concerning the organization's security practices, data and computer systems. Because insiders may attempt to steal property or information for personal gain, or to benefit another organization or country, the insiders committing threats may be related with each in some ways such as coming from the same alien country, working in the same team, or having the same ex-employer or previously serving in the same organization. Being able to utilize this kind of additional relational information makes it an interesting research topic for more precise insider threat detection.

In this project, we focus on the CERT dataset by building upon existing attribute-based threat detection, applying graph-based models to improve our detection, and ultimately combining both for the most reliable anomaly detection. We will talk about our progress so far.

In this presentation, I will talk about my thesis work progress through the winter break. Electronic Health Records (EHR) are a great source for answering questions related to patient data. The main focus of my thesis work is to convert the patient questions into logical forms using questions and their corresponding answers from EHR. These logical forms are transformed to Fast Healthcare Interoperability Resources (FHIR) query for retrieving the answer(s) from EHR. I will briefly talk about Semantic Parsing and some related works in this domain of Patient Question Answering using Semantic Parsing. Then, I will explain the various steps of my thesis work and talk about my progress during the winter break.

This talk will cover our plans for AIDA TA3 and summary of the kickoff meeting.