Reinforcement Lernen mit Regularisierungsnetzwerken

Die Arbeit behandelt das Problem der Skalierbarkeit von Reinforcement Lernen auf hochdimensionale und komplexe Aufgabenstellungen. Unter Reinforcement Lernen versteht man dabei eine auf approximativem Dynamischen Programmieren basierende Klasse von Lernverfahren, die speziell Anwendung in der Künstlichen Intelligenz findet und zur autonomen Steuerung simulierter Agenten oder realer Hardwareroboter in dynamischen und unwägbaren Umwelten genutzt werden kann. Dazu wird mittels Regression aus Stichproben eine Funktion bestimmt, die die Lösung einer "Optimalitätsgleichung" (Bellman) ist und aus der sich näherungsweise optimale Entscheidungen ableiten lassen. Eine große Hürde stellt dabei die Dimensionalität des Zustandsraums dar, die häufig hoch und daher traditionellen gitterbasierten Approximationsverfahren wenig zugänglich ist. Das Ziel dieser Arbeit ist es, Reinforcement Lernen durch nichtparametrisierte Funktionsapproximation (genauer, Regularisierungsnetze) auf -- im Prinzip beliebig -- hochdimensionale Probleme anwendbar zu machen. Regularisierungsnetze sind eine Verallgemeinerung von gewöhnlichen Basisfunktionsnetzen, die die gesuchte Lösung durch die Daten parametrisieren, wodurch die explizite Wahl von Knoten/Basisfunktionen entfällt und so bei hochdimensionalen Eingaben der "Fluch der Dimension" umgangen werden kann. Gleichzeitig sind Regularisierungsnetze aber auch lineare Approximatoren, die technisch einfach handhabbar sind und für die die bestehenden Konvergenzaussagen von Reinforcement Lernen Gültigkeit behalten (anders als etwa bei Feed-Forward Neuronalen Netzen). Allen diesen theoretischen Vorteilen gegenüber steht allerdings ein sehr praktisches Problem: der Rechenaufwand bei der Verwendung von Regularisierungsnetzen skaliert von Natur aus wie O(n**3), wobei n die Anzahl der Daten ist. Das ist besonders deswegen problematisch, weil bei Reinforcement Lernen der Lernprozeß online erfolgt -- die Stichproben werden von einem Agenten/Roboter erzeugt, während er mit der Umwelt interagiert. Anpassungen an der Lösung müssen daher sofort und mit wenig Rechenaufwand vorgenommen werden. Der Beitrag dieser Arbeit gliedert sich daher in zwei Teile: Im ersten Teil der Arbeit formulieren wir für Regularisierungsnetze einen effizienten Lernalgorithmus zum Lösen allgemeiner Regressionsaufgaben, der speziell auf die Anforderungen von Online-Lernen zugeschnitten ist. Unser Ansatz basiert auf der Vorgehensweise von Recursive Least-Squares, kann aber mit konstantem Zeitaufwand nicht nur neue Daten sondern auch neue Basisfunktionen in das bestehende Modell einfügen. Ermöglicht wird das durch die "Subset of Regressors" Approximation, wodurch der Kern durch eine stark reduzierte Auswahl von Trainingsdaten approximiert wird, und einer gierigen Auswahlwahlprozedur, die diese Basiselemente direkt aus dem Datenstrom zur Laufzeit selektiert. Im zweiten Teil übertragen wir diesen Algorithmus auf approximative Politik-Evaluation mittels Least-Squares basiertem Temporal-Difference Lernen, und integrieren diesen Baustein in ein Gesamtsystem zum autonomen Lernen von optimalem Verhalten. Insgesamt entwickeln wir ein in hohem Maße dateneffizientes Verfahren, das insbesondere für Lernprobleme aus der Robotik mit kontinuierlichen und hochdimensionalen Zustandsräumen sowie stochastischen Zustandsübergängen geeignet ist. Dabei sind wir nicht auf ein Modell der Umwelt angewiesen, arbeiten weitestgehend unabhängig von der Dimension des Zustandsraums, erzielen Konvergenz bereits mit relativ wenigen Agent-Umwelt Interaktionen, und können dank des effizienten Online-Algorithmus auch im Kontext zeitkritischer Echtzeitanwendungen operieren. Wir demonstrieren die Leistungsfähigkeit unseres Ansatzes anhand von zwei realistischen und komplexen Anwendungsbeispielen: dem Problem RoboCup-Keepaway, sowie der Steuerung eines (simulierten) Oktopus-Tentakels.

Influence of chain stiffness on structure and dynamics of polymers in the melt

Die vorliegende Arbeit beschaeftigt sich mit der Untersuchung vonPolymeren mit intrinsischer Steifigkeit. Es werden vor allem lokale statische unddynamische Eigenschaften anhand zweier verschiedener Simulationsmodellebetrachtet: Ein generisches Polymermodell, bei dem nur dieSteifigkeit als ein das spezifische Polymer charakterisierenden Parametereingeht und ein atomistisches Modell fuer trans-Polyisopren. Mit Hilfe des ersten Modells koennen Statik und Dynamik wurmartiger Kettenbeobachtet werden. Das Blob-Konzept ist eine angemessene statischeBeschreibung. Lokale Orientierungen haengen schwach von derSteifigkeit ab. Das Reptationsmodell kann die beobachtete Dynamik fuer lange Kettennicht mehr angemessen beschreiben. Lange Ketten bewegen sich, als obsie in Roehren gezwaengt waeren; jedoch ist die Bewegung starkabhaengig von der Steifigkeit. Fuer Ketten dieser Art konntequalitativ das Verhalten reproduziert werden, das in NMR-Experimentenbeobachtet wird. Eine Verhakungslaenge laesst sich fuer solche Kettenkaum mehr definieren. Dynamische Strukturfunktionen und insbesonderedie direkte Visualisierung der Ketten verdeutlichen die effektiv aufeine Roehre beschraenkte Bewegung. Das atomistische Polyisoprenmodell wurde mit verschiedenen Experimenten,verglichen. In den Simulationen bei konnten qualitativ undsemiquantitativ experimentelle Ergebnisse reproduziert werden. Zuletzt wurden die Laengen- und Zeitskalen der beiden Modelleerfolgreich aufeinander abgebildet.

Strukturbildung in dünnen Filmen aus Mischungen statistischer Copolymere

Dünne Polymerfilme besitzen einen weiten Anwendungsbereich in vielen High-Tech Applikationen. All diese Anwendungen erfordern ein bestimmtes Anwendungsprofil des dünnen Films. Diese Anforderungen umschließen sowohl die physikalischen Eigenschaften des Films als auch seine Struktur. Um sie zu realisieren, werden oftmals Mischungsfilme aus verschiedenen Polymeren verwendet. Diese neigen jedoch in vielen Fällen zur bereits während der Präparation zu Phasenseparation.Vor diesem Hintergrund wurde untersucht welchen Einfluss die Verträglichkeit der gemischten Polymere auf die Strukturbildung des dünnen Films ausüben. Als Modellsystem hierfür dienten Mischungen statistischer Poly-styrol-stat-para brom-styrol Copolymere.Die Oberflächenstrukturen, die sich währen der Präparation der Mischungsfilme einstellten, wurden mit Rasterkraftmikroskopie untersucht. wobei die Topologie einer statistischen Analyse unterzogen wurde. Zum einen wurde hierzu die spektrale Leistungsdichte der Oberflächenkontour zum anderen die zugehörigen Minkowski-Funktionale berechnet.Neben Oberflächenstrukturen bilden sich während der Präparation auch Entmischungsstrukturen im inneren des Filmes. Zur Charakterisierung dieser Strukturen wurden die Filme durch Streuung unter streifendem Einfall untersucht. Durch eine modellfreie Interpretation der Streuexperimente gelang der Nachweis der inneren StrukturenFür nur schwach unverträglich Filme konnte auf Basis der Streuexperimente eine Replikation der Oberflächenstruktur des Substrates auf die Filmoberflächen nachgewiesen werden. Diese Replikation wurde für verschieden raue Substrate und bezueglich der Kinetik ihrer Abnahme beim Quellen der Filme untersucht.

Pinning-Effekte kolumnarer Schwerionenspuren in Kuprat-Supraleiter-Dünnschichten

In this work the flux line dynamics in High-Temperature Superconductor (HTSC) thin films in the presence of columnar defects was studied using electronic transport measurements. The columnar defects which are correlated pinning centers for vortices were generated by irradiation with swift heavy ions at the Gesellschaft für Schwerionenforschung (GSI) in Darmstadt. In the first part, the vortex dynamics is discussed within the framework of the Bose-glass model. This approach describes the continuous transition from a vortex liquid to a Bose-glass phase which is characterized by the localization of the flux lines at the columnar defects. The critical behavior of the characteristic length and time scales for temperatures in the vicinity of this phase transition were probed by scaling properties of experimentally obtained current-voltage characteristics. In contrast to the predicted universal properties of the critical behavior the scaling analysis shows a strong dependence of the dynamic critical exponent on the experimentally accessible electric field range. In addition, the predicted divergence of the activation energy in the limit of low current densities was experimentally not confirmed.The dynamic behavior of flux lines in spatially resolved irradiation geometries is reported in the second part. Weak pinning channels with widths between 10 µm and 100 µm were generated in a strong pinning environment with the use of metal masks and the GSI microprobe, respectively. Measurements of the anisotropic transport properties of these structures show a striking resemblance to the results in YBCO single crystals with unidirected twin boundaries which were interpreted as a guided vortex motion effect. The use of two additional test bridges allowed to determine in parallel the resistivities of the irradiated and unirradiated parts as well as the respective current-voltage characteristics. These measurements provided the input parameters for a numerical simulation of the potential distribution in the spatially resolved irradiation geometry. The results are interpreted within a model that describes the hydrodynamic interaction between a Bose-glass phase and a vortex liquid. The interface between weakly pinned flux lines in the unirradiated channels and strongly pinned vortices leads to a nonuniform vortex velocity profile and therefore a variation of the local electric field. The length scale of these interactions was estimated for the first time in measuring the local variation of the electric field profile in a Bose-glass contact.Finally, a method for the determination of the true temperature in HTSC thin films at high dissipation levels is described. In this regime of electronic transport the occurrence of a flux flow instability is accompanied by heating effects in the vortex system. The heat propagation properties of the film/substrate system are deduced from the time dependent voltage response to a short high current density pulse of rectangular shape. The influence of heavy ion irradiation on the heat resistance at the film/substrate interface is studied.

A photophysical study of excited state dynamics in donor-acceptor copolymer photovoltaic blends

This thesis deals with the investigation of charge generation and recombination processes in three different polymer:fullerene photovoltaic blends by means of ultrafast time-resolved optical spectroscopy. The first donor polymer, namely poly[N-11"-henicosanyl-2,7-carbazole-alt-5,5-(4',7'-di-2-thienyl-2',1',3'-benzothiadiazole)] (PCDTBT), is a mid-bandgap polymer, the other two materials are the low-bandgap donor polymers poly[2,6-(4,4-bis-(2-ethylhexyl)-4H-cyclopenta[2,1-b;3,4-b']-dithiophene)-alt-4,7-(2,1,3-benzothiadiazole) (PCPDTBT) and poly[(4,4'-bis(2-ethylhexyl)dithieno[3,2-b:2',3'-d]silole)-2,6-diyl-alt-(2,1,3-benzothiadiazole)-4,7-diyl] (PSBTBT). Despite their broader absorption, the low-bandgap polymers do not show enhanced photovoltaic efficiencies compared to the mid-bandgap system.rnrnTransient absorption spectroscopy revealed that energetic disorder plays an important role in the photophysics of PCDTBT, and that in a blend with PCBM geminate losses are small. The photophysics of the low-bandgap system PCPDTBT were strongly altered by adding a high boiling point cosolvent to the polymer:fullerene blend due to a partial demixing of the materials. We observed an increase in device performance together with a reduction of geminate recombination upon addition of the cosolvent. By applying model-free multi-variate curve resolution to the spectroscopic data, we found that fast non-geminate recombination due to polymer triplet state formation is a limiting loss channel in the low-bandgap material system PCPDTBT, whereas in PSBTBT triplet formation has a smaller impact on device performance, and thus higher efficiencies are obtained.rn

Dispersion approach to real and virtual Compton scattering

In recent years, new precision experiments have become possible withthe high luminosity accelerator facilities at MAMIand JLab, supplyingphysicists with precision data sets for different hadronic reactions inthe intermediate energy region, such as pion photo- andelectroproduction and real and virtual Compton scattering.By means of the low energy theorem (LET), the global properties of thenucleon (its mass, charge, and magnetic moment) can be separated fromthe effects of the internal structure of the nucleon, which areeffectively described by polarizabilities. Thepolarizabilities quantify the deformation of the charge andmagnetization densities inside the nucleon in an applied quasistaticelectromagnetic field. The present work is dedicated to develop atool for theextraction of the polarizabilities from these precise Compton data withminimum model dependence, making use of the detailed knowledge of pionphotoproduction by means of dispersion relations (DR). Due to thepresence of t-channel poles, the dispersion integrals for two ofthe six Compton amplitudes diverge. Therefore, we have suggested to subtract the s-channel dispersion integrals at zero photon energy($nu=0$). The subtraction functions at $nu=0$ are calculated through DRin the momentum transfer t at fixed $nu=0$, subtracted at t=0. For this calculation, we use the information about the t-channel process, $gammagammatopipito Nbar{N}$. In this way, four of thepolarizabilities can be predicted using the unsubtracted DR in the $s$-channel. The other two, $alpha-beta$ and $gamma_pi$, are free parameters in ourformalism and can be obtained from a fit to the Compton data.We present the results for unpolarized and polarized RCS observables,%in the kinematics of the most recent experiments, and indicate anenhanced sensitivity to the nucleon polarizabilities in theenergy range between pion production threshold and the $Delta(1232)$-resonance.newlineindentFurthermore,we extend the DR formalism to virtual Compton scattering (radiativeelectron scattering off the nucleon), in which the concept of thepolarizabilities is generalized to the case of avirtual initial photon by introducing six generalizedpolarizabilities (GPs). Our formalism provides predictions for the fourspin GPs, while the two scalar GPs $alpha(Q^2)$ and $beta(Q^2)$ have to befitted to the experimental data at each value of $Q^2$.We show that at energies betweenpion threshold and the $Delta(1232)$-resonance position, thesensitivity to the GPs can be increased significantly, as compared tolow energies, where the LEX is applicable. Our DR formalism can be used for analysing VCS experiments over a widerange of energy and virtuality $Q^2$, which allows one to extract theGPs from VCS data in different kinematics with a minimum of model dependence.

The Dynamics of a Small Model Glass Former as Viewed from Its Potential Energy Landscape

Despite intensive research during the last decades, thetheoreticalunderstanding of supercooled liquids and the glasstransition is stillfar from being complete. Besides analytical investigations,theso-called energy-landscape approach has turned out to beveryfruitful. In the literature, many numerical studies havedemonstratedthat, at sufficiently low temperatures, all thermodynamicquantities can be predicted with the help of the propertiesof localminima in the potential-energy-landscape (PEL). The main purpose of this thesis is to strive for anunderstanding ofdynamics in terms of the potential energy landscape. Incontrast to the study of static quantities, this requirestheknowledge of barriers separating the minima.Up to now, it has been the general viewpoint that thermallyactivatedprocesses ('hopping') determine the dynamics only belowTc(the critical temperature of mode-coupling theory), in thesense that relaxation rates follow from local energybarriers.As we show here, this viewpoint should be revisedsince the temperature dependence of dynamics is governed byhoppingprocesses already below 1.5Tc.At the example of a binary mixture of Lennard-Jonesparticles (BMLJ),we establish a quantitative link from the diffusioncoefficient,D(T), to the PEL topology. This is achieved in three steps:First, we show that it is essential to consider wholesuperstructuresof many PEL minima, called metabasins, rather than singleminima. Thisis a consequence of strong correlations within groups of PELminima.Second, we show that D(T) is inversely proportional to theaverageresidence time in these metabasins. Third, the temperaturedependenceof the residence times is related to the depths of themetabasins, asgiven by the surrounding energy barriers. We further discuss that the study of small (but not toosmall) systemsis essential, in that one deals with a less complex energylandscapethan in large systems. In a detailed analysis of differentsystemsizes, we show that the small BMLJ system consideredthroughout thethesis is free of major finite-size-related artifacts.

A molecular approach towards tethered bilayer lipid membranes: synthesis and characterization of novel anchor lipids

Tethered bilayer lipid membranes (tBLMs) are a promising model system for the natural cell membrane. They consist of a lipid bilayer that is covalently coupled to a solid support via a spacer group. In this study, we developed a suitable approach to increase the submembrane space in tBLMs. The challenge is to create a membrane with a lower lipid density in order to increase the membrane fluidity, but to avoid defects that might appear due to an increase in the lateral space within the tethered monolayers. Therefore, various synthetic strategies and different monolayer preparation techniques were examined. Synthetical attempts to achieve a large ion reservoir were made in two directions: increasing the spacer length of the tether lipids and increasing the lateral distribution of the lipids in the monolayer. The first resulted in the synthesis of a small library of tether lipids (DPTT, DPHT and DPOT) characterized by 1H and 13C NMR, FD-MS, ATR, DSC and TGA. The synthetic strategy for their preparation includes synthesis of precursor with a double bond anchor that can be easily modified for different substrates (e.g. metal and metaloxide). Here, the double bond was modified into a thiol group suitable for gold surface. Another approach towards the preparation of homogeneous monolayers with decreased two-dimensional packing density was the synthesis of two novel anchor lipids: DPHDL and DDPTT. DPHDL is “self-diluted” tether lipid containing two lipoic anchor moieties. DDPTT has an extended lipophylic part that should lead to the preparation of diluted, leakage free proximal layers that will facilitate the completion of the bilayer. Our tool-box of tether lipids was completed with two fluorescent labeled lipid precursors with respectively one and two phytanyl chains in the hydrophobic region and a dansyl group as a fluorophore. The use of such fluorescently marked lipids is supposed to give additional information for the lipid distribution on the air-water interface. The Langmuir film balance was used to investigate the monolayer properties of four of the synthesized thiolated anchor lipids. The packing density and mixing behaviour were examined. The results have shown that mixing anchor with free lipids can homogeneously dilute the anchor lipid monolayers. Moreover, an increase in the hydrophylicity (PEG chain length) of the anchor lipids leads to a higher packing density. A decrease in the temperature results in a similar trend. However, increasing the number of phytanyl chains per lipid molecule is shown to decrease the packing density. LB-monolayers based on pure and mixed lipids in different ratio and transfer pressure were tested to form tBLMs with diluted inner layers. A combination of the LB-monolayer transfer with the solvent exchange method accomplished successfully the formation of tBLMs based on pure DPOT. Some preliminary investigations of the electrical sealing properties and protein incorporation of self-assembled DPOT and DDPTT-based tBLMs were conducted. The bilayer formation performed by solvent exchange resulted in membranes with high resistances and low capacitances. The appearance of space beneath the membrane is clearly visible in the impedance spectra expressed by a second RC element. The latter brings the conclusion that the longer spacer in DPOT and the bigger lateral space between the DDPTT molecules in the investigated systems essentially influence the electrical parameters of the membrane. Finally, we could show the functional incorporation of the small ion carrier valinomycin in both types of membranes.

The electron antineutrino angular correlation coefficient a in free neutron decay: Testing the standard model with the aSPECT-spectrometer

The beta-decay of free neutrons is a strongly over-determined process in the Standard Model (SM) of Particle Physics and is described by a multitude of observables. Some of those observables are sensitive to physics beyond the SM. For example, the correlation coefficients of the involved particles belong to them. The spectrometer aSPECT was designed to measure precisely the shape of the proton energy spectrum and to extract from it the electron anti-neutrino angular correlation coefficient "a". A first test period (2005/ 2006) showed the “proof-of-principles”. The limiting influence of uncontrollable background conditions in the spectrometer made it impossible to extract a reliable value for the coefficient "a" (publication: Baessler et al., 2008, Europhys. Journ. A, 38, p.17-26). A second measurement cycle (2007/ 2008) aimed to under-run the relative accuracy of previous experiments (Stratowa et al. (1978), Byrne et al. (2002)) da/a =5%. I performed the analysis of the data taken there which is the emphasis of this doctoral thesis. A central point are background studies. The systematic impact of background on a was reduced to da/a(syst.)=0.61 %. The statistical accuracy of the analyzed measurements is da/a(stat.)=1.4 %. Besides, saturation effects of the detector electronics were investigated which were initially observed. These turned out not to be correctable on a sufficient level. An applicable idea how to avoid the saturation effects will be discussed in the last chapter.

Microemulsion: prediction of the phase diagram with a modified Helfrich free energy

Microemulsions are thermodynamically stable, macroscopically homogeneous but microscopically heterogeneous, mixtures of water and oil stabilised by surfactant molecules. They have unique properties like ultralow interfacial tension, large interfacial area and the ability to solubilise other immiscible liquids. Depending on the temperature and concentration, non-ionic surfactants self assemble to micelles, flat lamellar, hexagonal and sponge like bicontinuous morphologies. Microemulsions have three different macroscopic phases (a) 1phase- microemulsion (isotropic), (b) 2phase-microemulsion coexisting with either expelled water or oil and (c) 3phase- microemulsion coexisting with expelled water and oil.rnrnOne of the most important fundamental questions in this field is the relation between the properties of the surfactant monolayer at water-oil interface and those of microemulsion. This monolayer forms an extended interface whose local curvature determines the structure of the microemulsion. The main part of my thesis deals with the quantitative measurements of the temperature induced phase transitions of water-oil-nonionic microemulsions and their interpretation using the temperature dependent spontaneous curvature [c0(T)] of the surfactant monolayer. In a 1phase- region, conservation of the components determines the droplet (domain) size (R) whereas in 2phase-region, it is determined by the temperature dependence of c0(T). The Helfrich bending free energy density includes the dependence of the droplet size on c0(T) as