In TFA it says that he was training a supervised learner to predict the control state from the screen state. So yes, "if the screen looks like this, apply these controls", and that can play Mario Kart 64.