Deep Learning Iteration

Fast iteration is vital to software and company development. This is pretty widely recognized:

This is a problem in the computation-heavy field of machine learning, particularly deep learning.

A particular example of this becoming important in deep learning is the ability to recognize correlations. On a particular project I had a very long iteration loop. It was about half an hour. There was so much time between a change I’d make and receiving a data point on progress that it was difficult to notice correlations. I had built an adversarial net that was supposed to achieve a certain goal, and I went through days of iteration before I was able to discover that this adversarial net had no hand whatsoever in producing the effect I was seeing. The effect I thought it was producing was actually just a byproduct of the primary net being too small and not capturing enough of the input data’s structure. This changed without my noticing because when I first added the adversarial net, the primary net was large enough for the adversarial net to cause the effect I wanted. Through later iterations I had decreased the size of the primary net until the adversarial net simply didn’t matter even though it appeared to produce the same effect. If running a test took under a second, then something like this could easily be a unit test. Alas, a half hour + half hour test is not a viable unit test.

Ultimately, identifying correlations and testing small changes to discover causations is how software development progresses. Human memory is imperfect and our desire to pattern match where there is no pattern means that the more infrequent data points occur, the harder it is to correct our memories and pattern matchers.

What can we do in deep learning?

Related posts:

I had a dream last night about the Cyrus Cylinder which isn’t something I am consciously aware of knowing about and so I found it delightfully reaffirming this morning when my research revealed that…

Terraform Cloud is a powerful infrastructure automation platform that offers numerous benefits for managing and deploying infrastructure as code. However, like any tool, it also has its drawbacks. In…

Imagine for a moment your brain as a bustling, intricate city. Its buildings are the neurons and the roads, the neural pathways connecting these structures. Every time you learn something new, it’s…