Determining the number of clusters in categorical data

Cluster analysis for categorical data has been an active area of research. A well-known problem in this area is the determination of the number of clusters, which is unknown and must be inferred from the data. In order to estimate the number of clusters, one often resorts to information criteria, such as BIC (Bayesian information criterion), MML (minimum message length, proposed by Wallace and Boulton, 1968), and ICL (integrated classification likelihood). In this work, we adopt the approach developed by Figueiredo and Jain (2002) for clustering continuous data. They use an MML criterion to select the number of clusters and a variant of the EM algorithm to estimate the model parameters. This EM variant seamlessly integrates model estimation and selection in a single algorithm. For clustering categorical data, we assume a finite mixture of multinomial distributions and implement a new EM algorithm, following a previous version (Silvestre et al., 2008). Results obtained with synthetic datasets are encouraging. The main advantage of the proposed approach, when compared to the above referred criteria, is the speed of execution, which is especially relevant when dealing with large data sets.

The tricks of the trade (un)exposed

One of the filmic trends which has been neglected by the Academy Awards is the metacinema, which for practical purposes I will consider to be a cross between the complexities of the self-reflexive cinema (highly connoted with modernism) and the Hollywood Film (the classical films about the urge to ‘make it’ in Hollywood). Indeed, these films have always existed and some, as Sunset Boulevard (Billy Wilder, 1950, USA) and Mulholland Dr. (David Lynch, 2001, FRA/USA), have even made it to the ceremony, but were, predictably, defeated in the main categories, by other more ‘serious’ or less self-reflexive products. The United States has always insisted on not revealing the tricks of the trade while, ironically, generating films that deal with this theme, in order to cater to the curiosity of the metacinema-inclined spectator. For this reason such films are usually about the universe of cinema but not its medium, at least not in a way that discloses the operations of the technical apparatus. Why are these films not viewed as serious enough and artistic enough to be awarded Oscars by the Academy in the categories of Best Picture, Best Director, Best Screenplay, and Best Cinematography? Are they being discarded for the same reasons that comedy and musicals usually are? Or are they being punished for being too unveiling? Or is the industry going for commercial products that can easily be pushed on a global scale and make a profit?