Studying Recommender Systems to Enhance Distributed Computing Schedulers

Distributed Computing frameworks belong to a class of programming models that allow developers to

launch workloads on large clusters of machines. Due to the dramatic increase in the volume of

data gathered by ubiquitous computing devices, data analytic workloads have become a common

case among distributed computing applications, making Data Science an entire field of

Computer Science. We argue that Data Scientist's concern lays in three main components: a dataset,

a sequence of operations they wish to apply on this dataset, and some constraint they may have

related to their work (performances, QoS, budget, etc). However, it is actually extremely

difficult, without domain expertise, to perform data science. One need to select the right amount

and type of resources, pick up a framework, and configure it. Also, users are often running their

application in shared environments, ruled by schedulers expecting them to specify precisely their resource

needs. Inherent to the distributed and concurrent nature of the cited frameworks, monitoring and

profiling are hard, high dimensional problems that block users from making the right

configuration choices and determining the right amount of resources they need. Paradoxically, the

system is gathering a large amount of monitoring data at runtime, which remains unused.

In the ideal abstraction we envision for data scientists, the system is adaptive, able to exploit

monitoring data to learn about workloads, and process user requests into a tailored execution

context. In this work, we study different techniques that have been used to make steps toward

such system awareness, and explore a new way to do so by implementing machine learning

techniques to recommend a specific subset of system configurations for Apache Spark applications.

Furthermore, we present an in depth study of Apache Spark executors configuration, which highlight

the complexity in choosing the best one for a given workload.

Common Bound: The Small Groups of Methodism

The system of small groups John Wesley established to promote a proper life of discipleship in early Methodist converts was, in many respects, the strength of the Methodist movement. Those who responded to Wesley’s initial invitation to “flee the wrath to come” were organized into large gatherings called “societies,” which were then subdivided into smaller bands, class meetings, select societies, and penitent bands. The smaller groups gave Wesley the opportunity, through a system of appointed leaders, to keep track of the spiritual progress of every member in his movement, which grew to tens of thousands by the time of his death in 1791. As Methodism shifted from renewal movement to institutional church in the nineteenth century, however, growth slowed, and participation in such groups declined rapidly. By the early twentieth century, classes and bands were virtually extinct in every sector of Methodism save the African-American tradition. In recent years, scholars in various sectors of the Wesleyan tradition, particularly David Lowes Watson and Kevin Watson, have called for a recovery of these small groups for purposes of renewal in the church. There is no consensus, however, concerning what exactly contributed to the vitality of these groups during Wesley’s ministry.

Over the last century, sociological studies of group dynamics have revealed three common traits that are crucial to highly functioning groups: interdependence created by the existence of a common goal, interaction among group members that is “promotive” or cooperative in nature, and high levels of feedback associated with personal responsibility and individual accountability. All three of these were prevalent in the early Methodist groups. Interdependence existed around a shared goal, which for Wesley and the Methodists was holiness. That interdependence was cooperative in nature; individuals experienced the empowering grace of God as they each pursued the goal in the company of fellow pilgrims. Finally, the groups existed for purposes of feedback and accountability as individuals took responsibility both for themselves and others as they progressed together toward the goal of holy living. Wesley seemed to instinctively understand the essential nature of each of these characteristics in maintaining the vitality of the movement when he spoke of the importance of preserving the “doctrine, spirit and discipline” of early Methodism. Analysis of some of the present-day attempts to restore Wesley’s groups reveals frequent neglect to one or more of these three components. Perhaps most critical to recovering the vitality of the early Methodist groups will be reclaiming the goal of sanctification and coming to a consensus on what its pursuit means in the present day.