How to run learning agents against PyGame

1. An array of all the pixels on the screen when ever it is updated.
2. The ability to output a set of key presses back into the game.
3. A feedback value to tell it when it is doing well/badly.

 screen = pygame.surfarray.array3d(pygame.display.get_surface())  

We could write our learning agent by hacking the game file and inserting this line into the main loop after the screen has been updated. But a better way to do it(and a way that allows us to have zero touches of the game file) is to intercept any calls to the pygame.display.update method and then grab the buffer then, like so:

 import pygame  
 import pygame.surfarray  

 # function that we can give two functions to and will return us a new function that calls both
 def function_combine(screen_update_func, our_intercepting_func):  
   def wrap(*args, **kwargs):  
     screen_update_func(*args,  
               **kwargs) # call the screen update func we intercepted so the screen buffer is updated  
     our_intercepting_func() # call our own function to get the screen buffer  
   return wrap  

 def on_screen_update():  
   surface_array = pygame.surfarray.array3d(pygame.display.get_surface())  
   print("We got the screen array")  
   print(surface_array)  

 # set our on_screen_update function to always get called whenever the screen updated  
 pygame.display.update = function_combine(pygame.display.update, on_screen_update)  
 # FYI the screen can also be modified via flip, so this might be needed for some games  
 pygame.display.flip = function_combine(pygame.display.flip, on_screen_update)  

You can try this out by inserting this code before you start your game and it will print out the screen buffer as it comes in.

Intercepting key presses

The normal method in PyGame detecting key presses is via this method:

 events = pygame.event.get()  

So we can intercept it and have it return our learning agents key presses:

 import pygame  
 from pygame.constants import K_DOWN, KEYDOWN  

 def function_intercept(intercepted_func, intercepting_func):  
   def wrap(*args, **kwargs):  
     # call the function we are intercepting and get it's result  
     real_results = intercepted_func(*args, **kwargs)
     # call our own function and return our new results  
     new_results = intercepting_func(real_results, *args, **kwargs)  
     return new_results  
   return wrap  

 def just_press_down_key(actual_events, *args, **kwargs):  
   return [pygame.event.Event(KEYDOWN, {"key": K_DOWN})]  

 pygame.event.get = function_intercept(pygame.event.get, just_press_down_key)  

I should also warn you that the pygame.event.get method can be called with args to filter out which events are needed. If your running a game that uses these you will either need to handle them or just use my complete implementation here.

Getting feedback to the player

The final piece of the puzzle is handling the feedback/reward from the game. Unfortunately there is no standard way of doing scoring in PyGame so this will always require some amount of going through the game code, but it can still be done with zero touches.

For the pong game the scores are stored in two global variables bar1_score and bar2_score, which can be imported. Our reward is when the score changes in our favor.

 last_bar1_score = last_bar2_score = 0  

 def get_feedback():  
   global last_bar1_score, last_bar2_score  

   # import must be done inside the method because otherwise importing would cause the game to start playing  
   from games.pong import bar1_score, bar2_score  
   # get the difference in score between this and the last run  
   score_change = (bar1_score - last_bar1_score) - (bar2_score - last_bar2_score)  
   last_bar1_score = bar1_score  
   last_bar2_score = bar2_score  
   return score_change  

But for other games, such as Tetris there may not be a globally scoped score variable we can grab. But there may be a method or a set of that methods that we know are good/bad. Such as a player_takes_damage, level_up or kill_enemy. We can use our function_intercept code from before to grab these. Here is an example in Tetris using the result of removeCompleteLines to reward our agent:

 import tetris  

 new_reward = 0.0
 def add_removed_lines_to_reward(lines_removed, *args, **kwargs):  
   global new_reward  
   new_reward += lines_removed  
   return lines_removed  

 tetris.removeCompleteLines = function_intercept(tetris.removeCompleteLines,  
                            add_removed_lines_to_reward)  

 def get_reward():  
   global new_reward  
   temp = new_reward  
   new_reward = 0.0  
   return temp  

Dealing with frame rates

Pong in PyGamePlayer

Now all that remains is too stitch all those parts together and plug in the learning agent. In my project I’ve chosen to do this in a class, but it would be fine as a script.
Below is an example of the full thing set up to learn against Pong using the PyGamePlayer module. The PongPlayer simply needs to inherit from the PyGamePlayer class and implement the get_keys_pressed and get_feeback methods, the framework handles everything else.

 from pygame.constants import K_DOWN  
 from pygame_player import PyGamePlayer  

 class PongPlayer(PyGamePlayer):  
   def __init__(self):  
     """  
     Example class for playing Pong  
     """  
     super(PongPlayer, self).__init__(force_game_fps=10) # we want to run at 10 frames per second
     self.last_bar1_score = 0.0  
     self.last_bar2_score = 0.0  

   def get_keys_pressed(self, screen_array, feedback):  
     # The code for running the actual learning agent would go here with the screen_array and feeback as the inputs
     # and an output for each key in the game, activating them if they are over some threshold.
     return [K_DOWN]  

   def get_feedback(self):  
     # import must be done here because otherwise importing would cause the game to start playing  
     from games.pong import bar1_score, bar2_score  

     # get the difference in score between this and the last run  
     score_change = (bar1_score - self.last_bar1_score) - (bar2_score - self.last_bar2_score)  
     self.last_bar1_score = bar1_score  
     self.last_bar2_score = bar2_score  
     return score_change

 if __name__ == '__main__':  
   player = PongPlayer()  
   player.start()
   # importing pong will start the game playing  
   import games.pong  

So hazar! We now have the worlds worst Pong AI.

In my next post I’ll go through writing a good reinforcement learning agent for this.
If you have any questions/correction please don’t hesitate to contact me.

Full source code here.