Refine
Document Type
- Doctoral Thesis (18) (remove)
Keywords
Institute
- Institut für Computervisualistik (18) (remove)
This thesis addresses the automated identification and localization of a time-varying number of objects in a stream of sensor data. The problem is challenging due to its combinatorial nature: If the number of objects is unknown, the number of possible object trajectories grows exponentially with the number of observations. Random finite sets are a relatively new theory that has been developed to derive at principled and efficient approximations. It is based around set-valued random variables that contain an unknown number of elements which appear in arbitrary order and are themselves random. While extensively studied in theory, random finite sets have not yet become a leading paradigm in practical computer vision and robotics applications. This thesis explores random finite sets in visual tracking applications. The first method developed in this thesis combines set-valued recursive filtering with global optimization. The problem is approached in a min-cost flow network formulation, which has become a standard inference framework for multiple object tracking due to its efficiency and optimality. A main limitation of this formulation is a restriction to unary and pairwise cost terms. This circumstance makes integration of higher-order motion models challenging. The method developed in this thesis approaches this limitation by application of a Probability Hypothesis Density filter. The Probability Hypothesis Density filter was the first practically implemented state estimator based on random finite sets. It circumvents the combinatorial nature of data association itself by propagation of an object density measure that can be computed efficiently, without maintaining explicit trajectory hypotheses. In this work, the filter recursion is used to augment measurements with an additional hidden kinematic state to be used for construction of more informed flow network cost terms, e.g., based on linear motion models. The method is evaluated on public benchmarks where a considerate improvement is achieved compared to network flow formulations that are based on static features alone, such as distance between detections and appearance similarity. A second part of this thesis focuses on the related task of detecting and tracking a single robot operator in crowded environments. Different from the conventional multiple object tracking scenario, the tracked individual can leave the scene and later reappear after a longer period of absence. Therefore, a re-identification component is required that picks up the track on reentrance. Based on random finite sets, the Bernoulli filter is an optimal Bayes filter that provides a natural representation for this type of problem. In this work, it is shown how the Bernoulli filter can be combined with a Probability Hypothesis Density filter to track operator and non-operators simultaneously. The method is evaluated on a publicly available multiple object tracking dataset as well as on custom sequences that are specific to the targeted application. Experiments show reliable tracking in crowded scenes and robust re-identification after long term occlusion. Finally, a third part of this thesis focuses on appearance modeling as an essential aspect of any method that is applied to visual object tracking scenarios. Therefore, a feature representation that is robust to pose variations and changing lighting conditions is learned offline, before the actual tracking application. This thesis proposes a joint classification and metric learning objective where a deep convolutional neural network is trained to identify the individuals in the training set. At test time, the final classification layer can be stripped from the network and appearance similarity can be queried using cosine distance in representation space. This framework represents an alternative to direct metric learning objectives that have required sophisticated pair or triplet sampling strategies in the past. The method is evaluated on two large scale person re-identification datasets where competitive results are achieved overall. In particular, the proposed method better generalizes to the test set compared to a network trained with the well-established triplet loss.
Der Wettbewerb um die besten Technologien zur Realisierung des autonomen Fahrens ist weltweit in vollem Gange.
Trotz großer Anstrengungen ist jedoch die autonome Navigation in strukturierter und vor allem unstrukturierter Umgebung bisher nicht gelöst.
Ein entscheidender Baustein in diesem Themenkomplex ist die Umgebungswahrnehmung und Analyse durch passende Sensorik und entsprechende Sensordatenauswertung.
Insbesondere bildgebende Verfahren im Bereich des für den Menschen sichtbaren Spektrums finden sowohl in der Praxis als auch in der Forschung breite Anwendung.
Dadurch wird jedoch nur ein Bruchteil des elektromagnetischen Spektrums genutzt und folglich ein großer Teil der verfügbaren Informationen zur Umgebungswahrnehmung ignoriert.
Um das vorhandene Spektrum besser zu nutzen, werden in anderen Forschungsbereichen schon seit Jahrzehnten \sog spektrale Sensoren eingesetzt, welche das elektromagnetische Spektrum wesentlich feiner und in einem größeren Bereich im Vergleich zu klassischen Farbkameras analysieren. Jedoch können diese Systeme aufgrund technischer Limitationen nur statische Szenen aufnehmen. Neueste Entwicklungen der Sensortechnik ermöglichen nun dank der \sog Snapshot-Mosaik-Filter-Technik die spektrale Abtastung dynamischer Szenen.
In dieser Dissertation wird der Einsatz und die Eignung der Snapshot-Mosaik-Technik zur Umgebungswahrnehmung und Szenenanalyse im Bereich der autonomen Navigation in strukturierten und unstrukturierten Umgebungen untersucht. Dazu wird erforscht, ob die aufgenommen spektralen Daten einen Vorteil gegenüber klassischen RGB- \bzw Grauwertdaten hinsichtlich der semantischen Szenenanalyse und Klassifikation bieten.
Zunächst wird eine geeignete Vorverarbeitung entwickelt, welche aus den Rohdaten der Sensorik spektrale Werte berechnet. Anschließend wird der Aufbau von neuartigen Datensätzen mit spektralen Daten erläutert. Diese Datensätze dienen als Basis zur Evaluation von verschiedenen Klassifikatoren aus dem Bereich des klassischen maschinellen Lernens.
Darauf aufbauend werden Methoden und Architekturen aus dem Bereich des Deep-Learnings vorgestellt. Anhand ausgewählter Architekturen wird untersucht, ob diese auch mit spektralen Daten trainiert werden können. Weiterhin wird die Verwendung von Deep-Learning-Methoden zur Datenkompression thematisiert. In einem nächsten Schritt werden die komprimierten Daten genutzt, um damit Netzarchitekturen zu trainieren, welche bisher nur mit RGB-Daten kompatibel sind. Abschließend wird analysiert, ob die hochdimensionalen spektralen Daten bei der Szenenanalyse Vorteile gegenüber RGB-Daten bieten
In this work has been examined, how the existing model of the simulation of cables and hoses can be advanced. Therefore an investigation has been made on the main influences to the shape simulation and the factors of constraints and side conditions were analyzed. For the validation of the accuracy, the simulation has to be compared to real specimen behavior. To obtain a very precise digitalization of the shape, the choice was made to use a laser scanner that converts the pointcloud into a .vrml file which can be imported into the simulation environment. The assumption was that the simulation method itself has the highest impact to the simulated shape. This is why the capabilities of the most sophisticated methods have been analyzed. The main criterion for the success of a simulation approach proved not to be accuracy, as expected. Process integration and usability showed to be of higher interest for the efficient exertion. Other factors like the pricing, the functionality and the real-time capability were assayed as well. The analyzed methods are based on the solution of the equations of elasticity with different ways of discetization, finite-elements and a spring-impulse-system. Since the finite-element-system takes several minutes for the computation of the shape and the spring-impulse-system reacts retarded on user manipulation, the competitiveness of these approaches is low. The other methods distinguish more in real-time performance, data interfaces and functionality than in accuracy. For the accuracy of a system, the consideration of other factors proved to be very important. As one of these main factors, the accurate assignment of the material properties was indicated. Until the start of this work, only the finite-element-approach dealt with this factor, but no documentation or validation is provided. In the publications of the other methods, the material properties are estimated to obtain a plausible simulation shape. Therefore the specific material values of real specimen have been measured and assigned to the simulation. With the comparison to the real shape it has been proven that the accuracy is very high with the measured properties. Since these measurements are very costly and time consuming, an investigation on a faster and cheaper way to obtain these values has been made. It has been assumed that with the knowledge of the cross-section it should be possible to compute the specimen behavior. Since the braid distribution changes individually from specimen to specimen, a more general way to obtain the values needed to be found. The program composer has been developed, where only the number of the different braids and the taping is entered. It computes with very high precision the stiffness, the density and the final diameter of the bundle. With the measured values and the fitting to the real shape it has been proven that the simulation approach reflects the precise behavior of cables and hoses. Since the stiffness of the single braids is wasteful to measure, a measurement setup was created where the stiffness has a large impact to the shape. With known density, the stiffness of the specimen can be reconstructed precisely. Thus a fast and beneficial way of obtaining the stiffness of a cable has been invented. The poissons ratio of cables and bundles cannot be measured with a tensile test, since the inner structure is very complex. For hoses, the variation of the inner diameter has been measured during the tensile test as well. The resulting values were reasonable, but their accuracy could not be proven. For cables and hoses, it has been tried to obtain the poissons ratio via the computation of the cross section, but the influence of individual changes and the crosstalk of the braids is very high. Therefore a setup was constructed where the torsion stiffness can be measured. For cables and hoses, the individual cross-sections and taping lead to varying results. For hoses, expected and repeatable good values for the poissons ratio were obtained. The low influence of the poisons ratio in the range between 0 and 0.5 has been proven. Therefore we decided to follow the advice of [Old06] and our own experiences to set the poisons ratio for cables and bundles to 0.25. With the knowledge of the measurability and the capabilities of the developed program composer, a procedure to obtain material properties for bundles has been designed. 1. Measurement of the braid density with via pyknometer or mass, length and diameter. 2. Empirical reconstruction of the stiffness with the designed setup. 3. Composing the bundle with the program composer. 4. Adding a factor for the taping and transfer the values to the simulation. The model of the cable simulation has been improved as follows: The main influences in the simulation of cables and hoses are the simulation method, the material properties and the geometric constraints. To obtain higher accuracy, an investigation on the correct material properties is indispensable. The scientific determination of material properties for the simulation of cables, bundles and hoses has been performed for the first time. The influence of geometrical constraints has been analyzed and documented. The next steps are the analysis of pre-deformation and further investigations to the determination of the poisons ratio with a more precise torsion test. All analysis were made with the simulation approach fleXengine. A comparison to other simulation methods would be of high interest.
Leichte Sprache (LS, easy-to-read German) is a simplified variety of German. It is used to provide barrier-free texts for a broad spectrum of people, including lowliterate individuals with learning difficulties, intellectual or developmental disabilities (IDD) and/or complex communication needs (CCN). In general, LS authors are proficient in standard German and do not belong to the aforementioned group of people. Our goal is to empower the latter to participate in written discourse themselves. This requires a special writing system whose linguistic support and ergonomic software design meet the target group’s specific needs. We present EasyTalk a system profoundly based on natural language processing (NLP) for assistive writing in an extended variant of LS (ELS). EasyTalk provides users with a personal vocabulary underpinned with customizable communication symbols and supports in writing at their individual level of proficiency through interactive user guidance. The system minimizes the grammatical knowledge needed to produce correct and coherent complex contents by intuitively formulating linguistic decisions. It provides easy dialogs for selecting options from a natural-language paraphrase generator, which provides context-sensitive suggestions for sentence components and correctly inflected word forms. In addition, EasyTalk reminds users to add text elements that enhance text comprehensibility in terms of audience design (e.g., time and place of an event) and improve text coherence (e.g., explicit connectors to express discourse-relations). To tailor the system to the needs of the target group, the development of EasyTalk followed the principles of human-centered design (HCD). Accordingly, we matured the system in iterative development cycles, combined with purposeful evaluations of specific aspects conducted with expert groups from the fields of CCN, LS, and IT, as well as L2 learners of the German language. In a final case study, members of the target audience tested the system in free writing sessions. The study confirmed that adults with IDD and/or CCN who have low reading, writing, and computer skills can write their own personal texts in ELS using EasyTalk. The positive feedback from all tests inspires future long-term studies with EasyTalk and further development of this prototypical system, such as the implementation of a so-called Schreibwerkstatt (writing workshop)
In the context of augmented reality we define tracking as a collection of methods to obtain the position and orientation (pose) of a user. By means of various displaying techniques, this ensures a correct visual overlay of graphical information onto the reality perceived. Precise results for calculation of the camera pose are gained by methods of image processing, usually analyzing the pixels of an image and extracing features, which can be recognized over the image sequence. However, these methods do not regard the process of image synthesis or at least in a very simplyfied way. In contrast, the class of model-based methods assumes a given 3D model of the observed scene. Based on the model data features can be identified to establish correspondences in the camera image. From these feature correspondences the camera pose is calculated. An interesting approach is the strategy of analysis-by-synthesis, regarding the computer graphics rendering process for extending the knowledge about the model by information from image synthesis and other environment variables.
In this thesis the components of a tracking system are identified and further it is analyzed, to what extend information about the model, the rendering process and the environment can contribute to the components for improvement of the tracking process using analysis-by-synthesis. In particular, by using knowledge as topological information, lighting or perspective, the feature synthesis and correspondence finding should lead to visually unambiguous features that can be predicted and evaluated to be suitable for stable tracking of the camera pose.
This thesis focuses on the utilization of modern graphics hardware (GPU) for visualization and computation purposes, especially of volumetric data from medical imaging. The considerable increase in raw computing power in recent years has turned commodity systems into high-performance workstations. In combination with the direct rendering capabilities of graphics hardware, "visual computing" and "computational steering" approaches on large data sets have become feasible. In this regard several example applications and concepts such as the "ray textures" have been developed and are discussed in detail. As the amount of data to be processed and visualized is steadily increasing, memory and bandwidth limitations require compact representations of the data. While the compression of image data has been investigated extensively in the past, the thesis addresses possibilities of performing computations directly on the compressed data. Therefore, different categories of algorithms are identified and represented in the wavelet domain. By using special variants of the compressed format, efficient implementations of essential image processing algorithms are possible and demonstrate the potential of the approach. From the technical perspective, the GPU-based framework "Cascada" has been developed in the course of this thesis. The introduction of object-oriented concepts to shader programming, as well as a hierarchical representation of computation and/or visualization procedures led to a simplified utilization of graphics hardware while maintaining competitive performance. This is shown with different implementations throughout the contributions, as well as two clinical projects in the field of diagnosis assistance. On the one hand the semi-automatic segmentation of low-resolution MRI data sets of the human liver is evaluated. On the other hand different possibilities in assessing abdominal aortic aneurysms are discussed; both projects make use of graphics hardware. In addition, "Cascada" provides extensions towards recent general-purpose programming architectures and a modular design for future developments.
On the recognition of human activities and the evaluation of its imitation by robotic systems
(2023)
This thesis addresses the problem of action recognition through the analysis of human motion and the benchmarking of its imitation by robotic systems.
For our action recognition related approaches, we focus on presenting approaches that generalize well across different sensor modalities. We transform multivariate signal streams from various sensors to a common image representation. The action recognition problem on sequential multivariate signal streams can then be reduced to an image classification task for which we utilize recent advances in machine learning. We demonstrate the broad applicability of our approaches formulated as a supervised classification task for action recognition, a semi-supervised classification task for one-shot action recognition, modality fusion and temporal action segmentation.
For action classification, we use an EfficientNet Convolutional Neural Network (CNN) model to classify the image representations of various data modalities. Further, we present approaches for filtering and the fusion of various modalities on a representation level. We extend the approach to be applicable for semi-supervised classification and train a metric-learning model that encodes action similarity. During training, the encoder optimizes the distances in embedding space for self-, positive- and negative-pair similarities. The resulting encoder allows estimating action similarity by calculating distances in embedding space. At training time, no action classes from the test set are used.
Graph Convolutional Network (GCN) generalized the concept of CNNs to non-Euclidean data structures and showed great success for action recognition directly operating on spatio-temporal sequences like skeleton sequences. GCNs have recently shown state-of-the-art performance for skeleton-based action recognition but are currently widely neglected as the foundation for the fusion of various sensor modalities. We propose incorporating additional modalities, like inertial measurements or RGB features, into a skeleton-graph, by proposing fusion on two different dimensionality levels. On a channel dimension, modalities are fused by introducing additional node attributes. On a spatial dimension, additional nodes are incorporated into the skeleton-graph.
Transformer models showed excellent performance in the analysis of sequential data. We formulate the temporal action segmentation task as an object detection task and use a detection transformer model on our proposed motion image representations. Experiments for our action recognition related approaches are executed on large-scale publicly available datasets. Our approaches for action recognition for various modalities, action recognition by fusion of various modalities, and one-shot action recognition demonstrate state-of-the-art results on some datasets.
Finally, we present a hybrid imitation learning benchmark. The benchmark consists of a dataset, metrics, and a simulator integration. The dataset contains RGB-D image sequences of humans performing movements and executing manipulation tasks, as well as the corresponding ground truth. The RGB-D camera is calibrated against a motion-capturing system, and the resulting sequences serve as input for imitation learning approaches. The resulting policy is then executed in the simulated environment on different robots. We propose two metrics to assess the quality of the imitation. The trajectory metric gives insights into how close the execution was to the demonstration. The effect metric describes how close the final state was reached according to the demonstration. The Simitate benchmark can improve the comparability of imitation learning approaches.
Bio-medical data comes in various shapes and with different representations.
Domain experts use such data for analysis or diagnosis,
during research or clinical applications. As the opportunities to obtain
or to simulate bio-medical data become more complex and productive,
the experts face the problem of data overflow. Providing a
reduced, uncluttered representation of data, that maintains the data’s
features of interest falls into the area of Data Abstraction. Via abstraction,
undesired features are filtered out to give space - concerning the
cognitive and visual load of the viewer - to more interesting features,
which are therefore accentuated. To address this challenge, the dissertation
at hand will investigate methods that deal with Data Abstraction
in the fields of liver vasculature, molecular and cardiac visualization.
Advanced visualization techniques will be applied for this purpose.
This usually requires some pre-processing of the data, which will also
be covered by this work. Data Abstraction itself can be implemented
in various ways. The morphology of a surface may be maintained,
while abstracting its visual cues. Alternatively, the morphology may
be changed to a more comprehensive and tangible representation.
Further, spatial or temporal dimensions of a complex data set may
be projected to a lower space in order to facilitate processing of the
data. This thesis will tackle these challenges and therefore provide an
overview of Data Abstraction in the bio-medical field, and associated
challenges, opportunities and solutions.
The goal of this thesis is the development of methods for augmented image synthesis using 3D photo collections. 3D photo collections are representations of real scenes automatically generated from single photos and describe a scene as a set of images with known camera poses as well as a sparse point-based model of the scene geometry. The main goal is to perform a photo-realistic augmented image synthesis of real and virtual parts, where the real scene is provided as a 3D photo collection. Therefore, three main problems are addressed.
Since the photos may be represented in different device-specific RGB color spaces, a color characterization of the 3D photo collections is necessary to gain correct color information that is consistent with human perception. The proposed novel method automatically transforms all images into a common RGB color space and thereby simplifies color characterization of 3D photo collections.
As a main problem for augmented image synthesis, all environmental lighting has to be known in order to apply illumination to virtual parts that is consistent with the real portions shown in the photos. To solve this problem, two novel methods were developed to reconstruct the lighting from 3D photo collections.
In order to perform image synthesis for arbitrary views on the scene, an image-based approach was developed that generates new views in 3D photo collections making direct use of its point cloud. The novel method creates new views in real-time and allows free-navigation.
In conclusion, the proposed novel methods show that 3D photo collections are a useful representation for real scenes in Augmented Reality and they can be used to perform a realistic image synthesis of real and virtual portions.
The cytological examination of bone marrow serves as clarification of variations in blood smears. It is also used for the clarification of anemia, as exclusion of bone marrow affection at lymphoma and at suspicion of leukemia. The morphological evaluation of hematopoietic cells is the basis for the creation of the diagnosis and for decision support for further diagnostics. Even for experienced hematologists the manual classification of hematopoietic cells is time-consuming, error-prone and subjective. For this reason new methods in the field of image processing and pattern recognition for the automatic classification including preprocessing steps are developed for a computer-assisted microscopy system. These methods are evaluated by means of a huge reference database. The proposed image analysis procedures comprise methods for the automated detection of smears, for the determination of relevant regions, for the localization and segmentation of single hematopoietic cells as well as for the feature extraction and classification task. These methods provide the basis for the first system for the automated, morphological analysis of bone marrow aspirates for leukemia diagnosis and are therefore a major contribution for a better and more efficient patient care in the future.