Jascha Sohl-Dickstein

I am a graduate student in the Redwood Center for Theoretical Neuroscience, at University of California, Berkeley. I am a member of Bruno Olshausen's lab, and the Biophysics Graduate Group. My email address is jascha@berkeley.edu.

I am interested in how we learn to perceive the world. There is evidence that much of our representation of the world is learned during development rather than being pre-programmed - everything from the way light intensity is correlated on adjacent patches of the retina, all the way up to the behavior (and existence!) of objects. We seem to infer most of human scale physics from examples of sensory input.

How this unsupervised learning problem is solved - how we learn the structure inherent in the world just by experiencing examples of it - is not well understood. This is the problem I am interested in tackling.

There are two large (known) parts to this problem. The first is the design of models which are flexible enough to describe *anything* without being unwieldy (greater flexibility frequently comes at the cost of an explosion in the number of parameters and/or computationally costly implementation). The second is training these models once they've been designed (almost any complex model you can write down is impractical to exactly evaluate - a factor called the "partition function", coming from the constraint that the probabilities of all states must sum to 1, is intractable to compute. This inability to exactly evaluate the model makes training exceedingly difficult.). I am attempting to work on both parts of this problem, though I've had more success so far with the second.

Current projects

I am working with Peter Battaglino and Michael DeWeese on a technique for parameter estimation in probabilistic models with intractable partition functions, involving minimization of probability flows. See the arXiv pre/e-print. Matlab code implementing Minimum Probability Flow learning for the Ising model and RBM cases, and for comparing performance to other techniques under the RBM case, is available on my public github repository.

I am working with Jimmy Wang and Bruno Olshausen to build a Lie algebraic model of the transformations which occur in natural video. See an arXiv pre/e-print, a poster pdf, or Jimmy's web page.

I am working with Jack Culpepper and Bruno Olshausen on novel uses of sampling algorithms in learning. Specifically, efficient ways to maintain the full posterior during EM, and ways to exactly calculate the log likelihood and partition function for distributions by treating the sampling chain as an alternative analytic form for the distribution.

I am experimenting with techniques for online Hessian-aware learning. More on this soon...

I am working to develop deep architectures for unsupervised learning based on deterministic, recurrent, networks.

I am working with Nicol Harper and Chris Rodgers to build a device enabling human echolocation.

Notes

• Sampling the Connectivity Pattern in Minimum Probability Flow Learning - Describes how the connectivity pattern between states in MPF can be described using a proposal distribution, rather than a deterministic rule.

• Entropy of Generic Distributions - Calculates the entropy that can be expected for a distribution drawn at random from the simplex of all possible distributions (John Schulman points out that ET Jaynes deals with similar questions in chapter 11 of "Probability Theory: The Logic Of Science")

The following are titles for informal notes I intend to write, but haven't gotten to/finished yet. If any of the following sound interesting to you, pester me and they will appear more quickly.

• Natural gradients explained via an analogy to signal whitening
• A log bound on the growth of intelligence with system size
• The field of experts model learns Gabor-like receptive fields when trained via minimum probability flow or score matching
• For small time bins, generalized linear models and causal Boltzmann machines become equivalent
• How to construct phase space volume preserving recurrent networks
• Maximum likelihood learning as constraint satisfaction
• A spatial derivation of score matching

Code

Code related to my research is available on my public github repository. The code there includes:

• MPF_ising/ - Matlab code for parameter estimation in the Ising model via Minimum Probability Flow learning.
• MPF_RBM_compare_log_likelihood/ - Matlab code for parameter estimation in Restricted Boltzmann Machines via Minimum Probability Flow learning, pseudolikelihood, and Contrastive Divergence. Also code comparing the log likelihood of data under the estimated RBMs for all 3 techniques.

Publications

J Sohl-Dickstein, BJ Culpepper. Hamiltonian annealed importance sampling for partition function estimation. Under review. http://marswatch.astro.cornell.edu/jascha/pdfs/HAIS.pdf

J Sohl-Dickstein, P Battaglino, M DeWeese. Minimum Probability Flow Learning. Under review. http://marswatch.astro.cornell.edu/jascha/pdfs/MPF.pdf

A Hayes, J Grotzinger, L Edgar, SW Squyres, W Watters, J Sohl-Dickstein. Reconstruction of Eolian Bed Forms and Paleocurrents from Cross-Bedded Strata at Victoria Crater, Meridiani Planum, Mars, Journal of Geophysical Research (2011)

CM Wang, J Sohl-Dickstein, I Tosik. Lie Group Transformation Models for Predictive Video Coding. Proceedings of the Data Compression Conference (2011) http://marswatch.astro.cornell.edu/jascha/pdfs/PID1615931.pdf

J Sohl-Dickstein, CM Wang, BA Olshausen. An Unsupervised Algorithm For Learning Lie Group Transformations. (2009) http://arxiv.org/abs/1001.1027

J Sohl-Dickstein, P Battaglino, M DeWeese. Minimum probability flow learning. (2009) http://arxiv.org/abs/0906.4779

C Abbey, J Sohl-Dickstein, BA Olshausen. Higher-order scene statistics of breast images. Proceedings of SPIE (2009) http://link.aip.org/link/?PSISDG/7263/726317/1

K Kinch, J Sohl-Dickstein, J Bell III, JR Johnson, W Goetz, GA Landis. Dust deposition on the Mars Exploration Rover Panoramic Camera (Pancam) calibration targets. Journal of Geophysical Research-Planets (2007) http://www.agu.org/pubs/crossref/2007/2006JE002807.shtml

POSTER - J Sohl-Dickstein, BA Olshausen. Learning in energy based models via score matching. Cosyne (2007) - this (dense!) poster introduces a spatial derivation of score matching, applies it to learning in a Field of Experts model, and then extends Field of Experts to work with heterogeneous experts (to form a "tapestry of experts"). I'm including it as it hasn't been written up elsewhere. download poster

JR Johnson, J Sohl-Dickstein, WM Grundy, RE Arvidson, J Bell III, P Christensen, T Graff, EA Guinness, K Kinch, R Morris, MK Shepard. Radiative transfer modeling of dust-coated Pancam calibration target materials: Laboratory visible/near-infrared spectrogoniometry. Journal of Geophysical Research (2006) http://www.agu.org/pubs/crossref/2006/2005JE002658.shtml

J Bell III, J Joseph, J Sohl-Dickstein, H Arneson, M Johnson, M Lemmon, D Savransky In-flight calibration and performance of the Mars Exploration Rover Panoramic Camera (Pancam) instruments. Journal of Geophysical Research (2006) http://www.agu.org/pubs/crossref/2006/2005JE002444.shtml

Parker et al. Stratigraphy and sedimentology of a dry to wet eolian depositional system, Burns formation, Meridiani Planum, Mars. Earth and Planetary Science Letters (2005)

Soderblom et al. Pancam multispectral imaging results from the Opportunity rover at Meridiani Planum. Science (2004) http://www.sciencemag.org/content/306/5702/1703

Soderblom et al. Pancam multispectral imaging results from the Spirit rover at Gusev crater. Science (2004) http://www.sciencemag.org/content/305/5685/800

Smith et al. Athena microscopic imager investigation. Journal of Geophysical Research-Planets (2003)

Bell et al. Hubble Space Telescope Imaging and Spectroscopy of Mars During 2001. American Geophysical Union (2001)