My Attempt at Outperforming Deepmind’s Atari Results – UPDATE 1

Hello again!

Time for an update!

The first thing I did after my original post was try to optimize HTMRL in order to get more gradient descent updates for the actor/critic portion. I started by experimenting with various numbers of HTM layers and sizes such that the last layer (the input to the actor/critic) was of an appropriate size that was small enough to run fast and large enough to convey enough information.

Secondly, I stopped using vanilla stochastic gradient descent for the actor/critic and started using RMSProp, just like in the original DeepMind paper. It was a simple change but quickly gave me better performance.

Thirdly, I started experimenting with ways to convert the binary information outputted by the last HTM region for the actor/critic into a floating-point representation that preserves information but does not hurt generalization at the same time. Originally I did a straight up binary to floating point value conversion, but this makes similar HTM configurations result in vastly different outputs, which greatly hurts generalization. So, instead I then opted for the lossy but generalizing approach: to sum the inputs and divide by the maximum count. This doesn’t take all the positional information of the input into account, but with a small enough condensing radius it can provide a decent compression/loss tradeoff.

Fourth, I played with some advantage learning replacements for standard Q learning. Advantage learning functions better then standard Q learning in continuous environments, since it amplifies the differences in state-action value between the successive timesteps more. This makes it less susceptible to errors in the function approximation as well as noise.

Finally, I received help from the SFML community in making the code work on more platforms. They even added CMake support for me! Many thanks!

That’s it for this update, time to get back to coding!

My Attempt at Outperforming Deepmind’s Atari Results


I am a reinforcement learning hobbyist, and I made myself a challenge: To outperform DeepMind’s reinforcement learning agent in the Arcade Learning Environment!

I will be posting updates here as I go!

The codebase I am using (currently separate from the ALE for easier experimentation) is available here:

It contains a large number of agents, the one I am currently working on is called HTMRL (hierarchical temporal memory reinforcement learning).

What do I do differently?

First off, I do not have a fixed time window of previous inputs to “solve” the hidden state problem. Rather, I am using HTM (hierarchical temporal memory) to form a context for the input automatically, as well as compress the input down to a manageable number of features.

From there, I use a simple feed-forward neural network(s) to be the actor/critic (I have two versions, one with an actor for continuous actions, not necessary for the ALE). These take the output of the last HTM region as input.

The critic-only version (discrete action) updates using standard Q learning updates plus eligibility traces.

The actor-critic version maintains a Q function in the critic and uses a form of policy gradient to optimize the actor on the Q values.

Right now I am working on getting it to function flawlessly on the pole balancing task before scaling up to the ALE.

Here is an image of HTMRL performing pole balancing. The top right shows the highest-level HTM region.

More coming soon!