Reinforcement Lernen mit Regularisierungsnetzwerken

Die Arbeit behandelt das Problem der Skalierbarkeit von Reinforcement Lernen auf hochdimensionale und komplexe Aufgabenstellungen. Unter Reinforcement Lernen versteht man dabei eine auf approximativem Dynamischen Programmieren basierende Klasse von Lernverfahren, die speziell Anwendung in der Künstlichen Intelligenz findet und zur autonomen Steuerung simulierter Agenten oder realer Hardwareroboter in dynamischen und unwägbaren Umwelten genutzt werden kann. Dazu wird mittels Regression aus Stichproben eine Funktion bestimmt, die die Lösung einer "Optimalitätsgleichung" (Bellman) ist und aus der sich näherungsweise optimale Entscheidungen ableiten lassen. Eine große Hürde stellt dabei die Dimensionalität des Zustandsraums dar, die häufig hoch und daher traditionellen gitterbasierten Approximationsverfahren wenig zugänglich ist. Das Ziel dieser Arbeit ist es, Reinforcement Lernen durch nichtparametrisierte Funktionsapproximation (genauer, Regularisierungsnetze) auf -- im Prinzip beliebig -- hochdimensionale Probleme anwendbar zu machen. Regularisierungsnetze sind eine Verallgemeinerung von gewöhnlichen Basisfunktionsnetzen, die die gesuchte Lösung durch die Daten parametrisieren, wodurch die explizite Wahl von Knoten/Basisfunktionen entfällt und so bei hochdimensionalen Eingaben der "Fluch der Dimension" umgangen werden kann. Gleichzeitig sind Regularisierungsnetze aber auch lineare Approximatoren, die technisch einfach handhabbar sind und für die die bestehenden Konvergenzaussagen von Reinforcement Lernen Gültigkeit behalten (anders als etwa bei Feed-Forward Neuronalen Netzen). Allen diesen theoretischen Vorteilen gegenüber steht allerdings ein sehr praktisches Problem: der Rechenaufwand bei der Verwendung von Regularisierungsnetzen skaliert von Natur aus wie O(n**3), wobei n die Anzahl der Daten ist. Das ist besonders deswegen problematisch, weil bei Reinforcement Lernen der Lernprozeß online erfolgt -- die Stichproben werden von einem Agenten/Roboter erzeugt, während er mit der Umwelt interagiert. Anpassungen an der Lösung müssen daher sofort und mit wenig Rechenaufwand vorgenommen werden. Der Beitrag dieser Arbeit gliedert sich daher in zwei Teile: Im ersten Teil der Arbeit formulieren wir für Regularisierungsnetze einen effizienten Lernalgorithmus zum Lösen allgemeiner Regressionsaufgaben, der speziell auf die Anforderungen von Online-Lernen zugeschnitten ist. Unser Ansatz basiert auf der Vorgehensweise von Recursive Least-Squares, kann aber mit konstantem Zeitaufwand nicht nur neue Daten sondern auch neue Basisfunktionen in das bestehende Modell einfügen. Ermöglicht wird das durch die "Subset of Regressors" Approximation, wodurch der Kern durch eine stark reduzierte Auswahl von Trainingsdaten approximiert wird, und einer gierigen Auswahlwahlprozedur, die diese Basiselemente direkt aus dem Datenstrom zur Laufzeit selektiert. Im zweiten Teil übertragen wir diesen Algorithmus auf approximative Politik-Evaluation mittels Least-Squares basiertem Temporal-Difference Lernen, und integrieren diesen Baustein in ein Gesamtsystem zum autonomen Lernen von optimalem Verhalten. Insgesamt entwickeln wir ein in hohem Maße dateneffizientes Verfahren, das insbesondere für Lernprobleme aus der Robotik mit kontinuierlichen und hochdimensionalen Zustandsräumen sowie stochastischen Zustandsübergängen geeignet ist. Dabei sind wir nicht auf ein Modell der Umwelt angewiesen, arbeiten weitestgehend unabhängig von der Dimension des Zustandsraums, erzielen Konvergenz bereits mit relativ wenigen Agent-Umwelt Interaktionen, und können dank des effizienten Online-Algorithmus auch im Kontext zeitkritischer Echtzeitanwendungen operieren. Wir demonstrieren die Leistungsfähigkeit unseres Ansatzes anhand von zwei realistischen und komplexen Anwendungsbeispielen: dem Problem RoboCup-Keepaway, sowie der Steuerung eines (simulierten) Oktopus-Tentakels.

Life cycle assessment in construction field: A life cycle cost analysis of reinforcement concrete bridge

The present work is included in the context of the assessment of sustainability in the construction field and is aimed at estimating and analyzing life cycle cost of the existing reinforced concrete bridge “Viadotto delle Capre” during its entire life. This was accomplished by a comprehensive data collection and results evaluation. In detail, the economic analysis of the project is performed. The work has investigated possible design alternatives for maintenance/rehabilitation and end-of-life operations, when structural, functional, economic and also environmental requirements have to be fulfilled. In detail, the economic impact of different design options for the given reinforced concrete bridge have been assessed, whereupon the most economically, structurally and environmentally efficient scenario was chosen. The Integrated Life-Cycle Analysis procedure and Environmental Impact Assessment were also discussed in this work. The scope of this thesis is to illustrate that Life Cycle Cost analysis as part of Life Cycle Assessment approach could be effectively used to drive the design and management strategy of new and existing structures. The final objective of this contribution is to show how an economic analysis can influence decision-making in the definition of the most sustainable design alternatives. The designers can monitor the economic impact of different design strategies in order to identify the most appropriate option.

Frequency-dependent response of snow in uniaxial compression and comparison with numerical simulation

La determinazione del modulo di Young è fondamentale nello studio della propagazione di fratture prima del rilascio di una valanga e per lo sviluppo di affidabili modelli di stabilità della neve. Il confronto tra simulazioni numeriche del modulo di Young e i valori sperimentali mostra che questi ultimi sono tre ordini di grandezza inferiori a quelli simulati (Reuter et al. 2013)⁠. Lo scopo di questo lavoro è stimare il modulo di elasticità studiando la dipendenza dalla frequenza della risposta di diversi tipi di neve a bassa densità, 140-280 kg m-3. Ciò è stato fatto applicando una compressione dinamica uniassiale a -15°C nel range 1-250 Hz utilizzando il Young's modulus device (YMD), prototipo di cycling loading device progettato all'Istituto per lo studio della neve e delle valanghe (SLF). Una risposta viscoelastica della neve è stata identificata a tutte le frequenze considerate, la teoria della viscoelasticità è stata applicata assumendo valida l'ipotesi di risposta lineare della neve. Il valore dello storage modulus, E', a 100 Hz è stato identificato come ragionevolmente rappresentativo del modulo di Young di ciascun campione neve. Il comportamento viscoso è stato valutato considerando la loss tangent e la viscosità ricavata dai modelli di Voigt e Maxwell. Il passaggio da un comportamento più viscoso ad uno più elastico è stato trovato a 40 Hz (~1.1•10-2 s-1). Il maggior contributo alla dissipazione è nel range 1-10 Hz. Infine, le simulazioni numeriche del modulo di Young sono state ottenute nello stesso modo di Reuter et al.. La differenza tra le simulazioni ed i valori sperimentali di E' sono, al massimo, di un fattore 5; invece, in Reuter et al.⁠, era di 3 ordini di grandezza. Pertanto, i nostri valori sperimentali e numerici corrispondono meglio, indicando che il metodo qui utilizzato ha portato ad un miglioramento significativo.

Finite element modeling of flow/compression-induced deformation of alginate scaffolds for bone tissue engineering

Trauma or degenerative diseases such as osteonecrosis may determine bone loss whose recover is promised by a "tissue engineering“ approach. This strategy involves the use of stem cells, grown onboard of adequate biocompatible/bioreabsorbable hosting templates (usually defined as scaffolds) and cultured in specific dynamic environments afforded by differentiation-inducing actuators (usually defined as bioreactors) to produce implantable tissue constructs. The purpose of this thesis is to evaluate, by finite element modeling of flow/compression-induced deformation, alginate scaffolds intended for bone tissue engineering. This work was conducted at the Biomechanics Laboratory of the Institute of Biomedical and Neural Engineering of the Reykjavik University of Iceland. In this respect, Comsol Multiphysics 5.1 simulations were carried out to approximate the loads over alginate 3D matrices under perfusion, compression and perfusion+compression, when varyingalginate pore size and flow/compression regimen. The results of the simulations show that the shear forces in the matrix of the scaffold increase coherently with the increase in flow and load, and decrease with the increase of the pore size. Flow and load rates suggested for proper osteogenic cell differentiation are reported.

Ease of Application of Medical Compression-stocking Systems for the Treatment of Venous Ulcers

OBJECTIVE: To evaluate the ease of application of two-piece, graduated, compression systems for the treatment of venous ulcers. METHODS: Four kits used to provide limb compression in the management of venous ulcers were evaluated. These have been proven to be non-inferior to various types of bandages in clinical trials. The interface pressure exerted above the ankle by the under-stocking and the complete compression system and the force required to pull the over-stocking off were assessed in vitro. Ease of application of the four kits was evaluated in four sessions by five nurses who put stockings on their own legs in a blinded manner. They expressed their assessment of the stockings using a series of visual analogue scales (VASs). RESULTS: The Sigvaris Ulcer X((R)) kit provided a mean interface pressure of 46 mmHg and required a force in the range of 60-90 N to remove it. The Mediven((R)) ulcer kit exerted the same pressure but required force in the range of 150-190 N to remove it. Two kits (SurePress((R)) Comfort and VenoTrain((R)) Ulcertec) exerted a mean pressure of only 25 mmHg and needed a force in the range of 100-160 N to remove them. Nurses judged the Ulcer X and SurePress kits easiest to apply. Application of the VenoTrain kit was found slightly more difficult. The Mediven kit was judged to be difficult to use. CONCLUSIONS: Comparison of ease of application of compression-stocking kits in normal legs revealed marked differences between them. Only one system exerted a high pressure and was easy to apply. Direct comparison of these compression kits in leg-ulcer patients is required to assess whether our laboratory findings correlate with patient compliance and ulcer healing.

Comparison of low-strength compression stockings with bandages for the treatment of recalcitrant venous ulcers

OBJECTIVE: To compare the proportion and rate of healing, pain, and quality of life of low-strength medical compression stockings (MCS) with traditional bandages applied for the treatment of recalcitrant venous leg ulcers. METHODS: A single-center, randomized, open-label study was performed with consecutive patients. Sigvaris prototype MCS providing 15 mm Hg-25 mm Hg at the ankle were compared with multi-layer short-stretch bandages. In both groups, pads were placed above incompetent perforating veins in the ulcer area. The initial static pressure between the dressing-covered ulcer and the pad was 29 mm Hg and 49 mm Hg with MCS and bandages, respectively. Dynamic pressure measurements showed no difference. Compression was maintained day and night and changed every week. The primary endpoint was healing within 90 days. Secondary endpoints were healing within 180 days, time to healing, pain (weekly Likert scales), and monthly quality of life (ChronIc Venous Insufficiency Quality of Life [CIVIQ] questionnaire). RESULTS: Of 74 patients screened, 60 fulfilled the selection criteria and 55 completed the study; 28 in the MCS and 27 in the bandage group. Ulcers were recurrent (48%), long lasting (mean, 27 months), and large (mean, 13 cm2). All but one patient had deep venous reflux and/or incompetent perforating veins in addition to trunk varices. Characteristics of patients and ulcers were evenly distributed (exception: more edema in the MCS group; P = .019). Healing within 90 days was observed in 36% with MCS and in 48% with bandages (P = .350). Healing within 180 days was documented in 50% with MCS and in 67% with bandages (P = .210). Time to healing was identical. Pain scored 44 and 46 initially (on a scale in which 100 referred to maximum and 0 to no pain) and decreased within the first week to 20 and 28 in the MCS and bandage groups, respectively (P < .001 vs .010). Quality of life showed no difference between the treatment groups. In both groups, pain at 90 days had decreased by half, independent of completion of healing. Physical, social, and psychic impairment improved significantly in patients with healed ulcers only. CONCLUSION: Our study illustrates the difficulty of bringing large and long-standing venous ulcers to heal. The effect of compression with MCS was not different from that of compression with bandages. Both treatments alleviated pain promptly. Quality of life was improved only in patients whose ulcers had healed.

The in vivo effects of unloading and compression on T1-Gd (dGEMRIC) relaxation times in healthy articular knee cartilage at 3.0 Tesla

The purpose was to investigate the in vivo effects of unloading and compression on T1-Gd relaxation times in healthy articular knee cartilage.

Reinforcement learning in dendritic structures

The discovery of binary dendritic events such as local NMDA spikes in dendritic subbranches led to the suggestion that dendritic trees could be computationally equivalent to a 2-layer network of point neurons, with a single output unit represented by the soma, and input units represented by the dendritic branches. Although this interpretation endows a neuron with a high computational power, it is functionally not clear why nature would have preferred the dendritic solution with a single but complex neuron, as opposed to the network solution with many but simple units. We show that the dendritic solution has a distinguished advantage over the network solution when considering different learning tasks. Its key property is that the dendritic branches receive an immediate feedback from the somatic output spike, while in the corresponding network architecture the feedback would require additional backpropagating connections to the input units. Assuming a reinforcement learning scenario we formally derive a learning rule for the synaptic contacts on the individual dendritic trees which depends on the presynaptic activity, the local NMDA spikes, the somatic action potential, and a delayed reinforcement signal. We test the model for two scenarios: the learning of binary classifications and of precise spike timings. We show that the immediate feedback represented by the backpropagating action potential supplies the individual dendritic branches with enough information to efficiently adapt their synapses and to speed up the learning process.

Reinforcement learning in dendritic structures

The discovery of binary dendritic events such as local NMDA spikes in dendritic subbranches led to the suggestion that dendritic trees could be computationally equivalent to a 2-layer network of point neurons, with a single output unit represented by the soma, and input units represented by the dendritic branches. Although this interpretation endows a neuron with a high computational power, it is functionally not clear why nature would have preferred the dendritic solution with a single but complex neuron, as opposed to the network solution with many but simple units. We show that the dendritic solution has a distinguished advantage over the network solution when considering different learning tasks. Its key property is that the dendritic branches receive an immediate feedback from the somatic output spike, while in the corresponding network architecture the feedback would require additional backpropagating connections to the input units. Assuming a reinforcement learning scenario we formally derive a learning rule for the synaptic contacts on the individual dendritic trees which depends on the presynaptic activity, the local NMDA spikes, the somatic action potential, and a delayed reinforcement signal. We test the model for two scenarios: the learning of binary classifications and of precise spike timings. We show that the immediate feedback represented by the backpropagating action potential supplies the individual dendritic branches with enough information to efficiently adapt their synapses and to speed up the learning process.

An Evaluation of Compression Schemes for Wireless Networks

Data gathering, either for event recognition or for monitoring applications is the primary intention for sensor network deployments. In many cases, data is acquired periodically and autonomously, and simply logged onto secondary storage (e.g. flash memory) either for delayed offline analysis or for on demand burst transfer. Moreover, operational data such as connectivity information, node and network state is typically kept as well. Naturally, measurement and/or connectivity logging comes at a cost. Space for doing so is limited. Finding a good representative model for the data and providing clever coding of information, thus data compression, may be a means to use the available space to its best. In this paper, we explore the design space for data compression for wireless sensor and mesh networks by profiling common, publicly available algorithms. Several goals such as a low overhead in terms of utilized memory and compression time as well as a decent compression ratio have to be well balanced in order to find a simple, yet effective compression scheme.

Compression hosiery for occupational leg symptoms and leg volume: a randomized crossover trial in a cohort of hairdressers

OBJECTIVE: Occupational leg symptoms are highly prevalent in the general population and impair the psychic state of health. We investigated hairdressers, a cohort exposed to prolonged standing during work, in a randomized crossover trial. We hypothesized that hairdressers wearing low-strength compression hosiery would benefit from less leg volume increase and discomfort. METHODS: One hundred and eight hairdressers were randomized to wear medical compression stockings (MCS; 15-20 mmHg) in a crossover study. The effect of MCS on symptoms and on lower leg volume was compared with no compression treatment. Symptoms were assessed with a comprehensive questionnaire, categorized using factor analysis with varimax rotation and correlated with leg volume changes. RESULTS: Wearing MCS reduced the symptom score for pain and feelings of swelling (range 0-4) by an average of 0.22 (12%, P < 0.001). Sleep disturbance, feeling of unattractive legs and depressiveness improved with MCS compared with no MCS. Subjects initially obliged to refrain from wearing stockings showed a significant decrease of pain and feelings of swelling as well (by 0.10 [6%], P = 0.015). Wearing MCS was associated with a decrease of lower leg volume by an average of 19 mL (P < 0.001), with preference in older hairdressers (P < 0.001). The effects of wearing MCS on symptoms and on leg volume were not correlated with each other. CONCLUSIONS: Individuals working in a standing profession experience leg pain, feelings of swelling, heaviness and various other disturbing feelings. These symptoms can be alleviated by wearing low-strength MCS.

High-speed Imaging of a Bulk Metallic Glass During Uniaxial Compression

High-speed imaging directly correlates the propagation of a particular shear band with mechanical measurements during uniaxial compression of a bulk metallic glass. Imaging shows shear occurs simultaneously over the entire shear plane, and load data, synced and time-stamped to the same clock as the camera, reveal that shear sliding is coincident with the load drop of each serration. Digital image correlation agrees with these results. These data demonstrate that shear band sliding occurs with velocities on the order of millimeters per second. Fracture occurs much more rapidly than the shear banding events, thereby readily leading to melting on fracture surfaces.