Learning meaningful, low-dimensional representations of data is a challenging problem. Particularly for an autonomously learning system, representations learned from observations can play a crucial role. Consider for example a system that received many images of faces and is capable of finding out that there are common factors explaining most of the characteristics, such as gender or hair color. Variational Autoencoders can do this to an astonishing extend, but it was unclear why this is actually happening. We found that the reason is a side-product of a simplifying assumption made to make the objective tractable and thereby the method suitable for application . Interestingly, this insight allowed us to connect the method to classical principle component analysis. For the future we hope to utilize this knowledge for further advances in general data-analysis.