Opening the Black Box

As early as his bachelor’s studies, Dr. Tom Tirer fell in love with the field of signal processing. “It’s an elegant mathematical field that gets into algorithmics, and I loved it,” he shares. “My master’s thesis focused on parameter estimation, and in my doctoral thesis I researched methods of signal restoration—particularly images—from partial and/or degraded measurements using mathematical modeling. But for many real-world signals—natural images, for example—mathematical modeling is difficult, and that’s where it is advantageous to use machine learning, particularly learning based on deep neural networks.”

Tirer (38) earned his bachelor's degree at Ben Gurion University, and his master’s and PhD degrees at Tel Aviv University. He conducted his postdoctoral research at Tel Aviv University and at the New York University Center for Data Science. “A large part of my research focuses on signal reconstruction: say we have a noisy image from which we want to extract a clean image; or a blurry, or low-resolution image, and we want to restore the original image, clean and in high resolution. These tasks are the first stage in many systems, and from there we can move on to object detection or classification,” explains Tirer. He claims that as his research progressed, he found himself integrating more and more deep learning tools. “In the field of signal processing, having an algorithm that’s based on a mathematical model has been considered an advantage, because it allows you to perform a mathematical analysis, establish theory that predicts the algorithm’s performance, or bound its errors. The problem starts when you want to recover signals you can’t properly model mathematically: like natural images that we take with a camera, which can be landscapes, soccer games or selfies. These signals are hard to model mathematically.”

And here’s where deep learning techniques come in handy. “In these methods, you train deep neural networks using a vast number of examples, which saves the need for mathematical modeling of complex data,” explains Tirer. “Take for example the super-resolution task: turning a low-resolution image into a good, high-resolution image. The typical machine learning solution would be to first collect plenty of high resolution images and turn each of them into its low resolution version using a known physical observation model; the next step would be to train a neural network to learn the inverse mapping—from the low resolution version to the high resolution version (which is known in the training phase). In this manner we skip the need to model the images themselves. But this method has two key shortcomings: the first is that while previously we could develop a theory and understand what happens throughout the algorithm, the networks we’re dealing with here are like a black box—we can’t know exactly what happens in each phase in the network, and for the most part we can’t perform a theoretical mathematical analysis. The second issue is that when we trained a network for a specific task, we used the relationship between the original image and the observations; but once something deviates from what we assumed in the training phase, the method won’t function properly. The traditional algorithms are more flexible to the observation model.”

For this reason, a considerable part of Tirer’s PhD research revolved around combining the two stances: using deep neural networks to circumvent the mathematical modeling of complex signals—within an algorithmic framework that allows for flexibility. “Furthermore,” he elaborates, “I want the trained network to provide me prior information about the signal and allow me to solve very different problems without retraining the network, because I’m integrating it as a sort of building block within a more generalized traditional scheme. The goal is to train a network to capture knowledge on the signal that is independent of the observation model, for example using generative models or noise reduction models. In order to combine the two methods—those based on machine learning and those based on more traditional signal processing—I use advanced optimization tools.”

Today, Tirer’s research tackles two main fronts. One is primarily applicative: solving problems from the realm of image processing by using signal processing, machine learning and optimization tools. “My methods are usually applicable to general signals, but I like to demonstrate them on images, because acquiring prior information on images is both challenging and necessary, and the scale of the problems is huge,” he explains. “Even if we want to improve the resolution of a tiny 250x250 pixel image by 2, there are 500² unknown parameters that we have to estimate. We need to use good,  computationally efficient optimization methods and to utilize patterns that recur within the image and across different images.” The second front is more theoretical—analyzing algorithms and deep learning theory. “During my postdoc I worked a lot on analyzing phenomena that come up in deep learning: before we can train a neural network, we have to determine its architecture, the links between the different layers of the network, and I study how different architectures affect the network’s behavior—whether the function that is learned by the network is smoother, or if the optimization throughout the training becomes easier. I also study the evolution of features, namely, the representations that the network learns in its different layers.”

Tirer joined the Faculty of Engineering this past October, as part of the data science track. Next semester he will be teaching a course on optimization and later some courses on advanced topics in image and signal processing. For now, he is forming his research group: “I’m looking for master’s and doctoral students who are curious and self-motivated, to join me in exploring this spectrum that ranges from signal processing to machine learning—facing theoretical and applicative research questions. My overarching goal is to improve our understanding of methods that use deep learning, whether as building blocks in general algorithmic frameworks, or as a complete solution, so that we can employ them effectively and reliably, rather than treat them as a black box.

Sounds interesting? Contact Dr. Tom Tirer for more information and applications: tirer.tom@biu.ac.il