Reinforcement Learning for tabular Data
Reinforcement learning for tabular data is an exciting topic, especially in the context of machine learning. RL is a type of machine learning in which an agent learns decisions by interacting with its environment. Normally it is used for games, but within the realm of possibility I tried it out for the SpaceTitanic data set.
The basic functions are
1. Presentation of the condition:
I define the states based on the tabular data. Each line or combination of lines represents a state in my environment.
2. Action space:
I identify the possible actions I can take as an agent. These actions refer to changes to the tabular data, such as updating specific values or rows.
3. Design of the reward:
I establish a reward structure that should lead me as an agent to desirable outcomes. Rewards are based on changes to the tabular data and its consistency with my goals.
4. Q Learning:
I implement algorithms like Q-Learning to update the Q-values (action values) based on my experiences. These algorithms are designed for discrete state and action spaces.
5. Exploration-Exploitation:
I consider strategies so that I can effectively explore the state action space as I learn and make decisions.
6. Training:
I train my RL agent on the tabular data with the selected algorithm. I monitor my learning progress and adjust the parameters if necessary.
Get Tom Kaminski’s stories in your inbox
Join Medium for free to get updates from this writer.
7. Evaluation:
I’m evaluating the performance of my trained RL agent on new tabular data to ensure that it generalizes well.
Let’s code
At first we have to import necessary library's.
"""
Reinforcement Learning for Tabular Data using Stable Baselines3
This code implements a simple reinforcement learning setup for tabular data using the Stable Baselines3 library.
The environment is a basic custom environment created with OpenAI Gym.
Dependencies:
- numpy
- pandas
- matplotlib
- seaborn
- time
- gym
- stable-baselines3
- tensorflow
Note: Make sure to install the required libraries before running the code.
"""
# Import necessary libraries
import numpy as np
import pandas as pd
import time
import gym
from gym import spaces
import random
import numpy as np
import tensorflow as tf
from stable_baselines3.dqn import MlpPolicy
from stable_baselines3.common.vec_env import DummyVecEnv
from stable_baselines3 import PPO
from stable_baselines3.common import monitor
from stable_baselines3.common import logger
# Define and document the purpose of the code
"""
This code sets up a reinforcement learning environment for tabular data using the Stable Baselines3 library.
The environment is defined using OpenAI Gym, and the agent is trained using both Proximal Policy Optimization (PPO) algorithms from Stable Baselines3.
"""
# Define the custom environment
"""
The custom environment is defined using the OpenAI Gym framework. The state space and action space are tailored
to handle tabular data. The environment is expected to have a discrete state and action space.
"""
# Create the RL agent using PPO
"""
The Proximal Policy Optimization (PPO) algorithm from Stable Baselines3 is used to train the RL agent.
The MlpPolicy is chosen for this particular task.
"""
# Perform training and evaluation
"""
The RL agent is trained using the specified algorithm and then evaluated on new tabular data.
Training progress and evaluation results are monitored and logged for analysis.
"""
# Display results
"""
Matplotlib and Seaborn are used to visualize and display the training and evaluation results.
"""Feature Engineering Function Documentation
This Python function, named FeatureEngineering, is designed to perform feature engineering on a given DataFrame (df). The function extracts and transforms information from specific columns in the DataFrame, creating new features to enhance the dataset for machine learning applications.
def FeatureEngineering(df):
# 1. Cabin Splitting:
# - Splits 'Cabin' into 'Deck', 'Number', and 'Side' columns.
df[['Deck','Number','Side']] = df.Cabin.str.split('/',expand=True)
# 2. PassengerId Splitting:
# - Splits 'PassengerId' into 'gggg' and 'pp' columns.
df[['FirstName','LastName']] = df.Name.str.split(' ',expand=True)
df[['gggg','pp']] = df.PassengerId.str.split('_',expand=True)
# 4. RowSum Calculation:
# - Creates 'RowSum' representing the sum of certain columns.
df['RowSum'] = df[['RoomService','FoodCourt', 'ShoppingMall', 'Spa', 'VRDeck']].sum(axis=1)
# 5. Conversion to Categorical Values:
# - Converts selected categorical columns to category dtype.
cat_cols = ['HomePlanet', 'CryoSleep', 'Destination', 'VIP', 'Deck','Number', 'Side', 'FirstName', 'LastName', 'gggg', 'pp']
for cat in cat_cols:
df[cat] = df[cat].astype('category')
# 6. Category Code Transformation:
# - Encodes categorical columns to numeric codes.
hp = df['HomePlanet'].cat.codes.values
cs = df['CryoSleep'].cat.codes.values
dest = df['Destination'].cat.codes.values
vip = df['VIP'].cat.codes.values
deck = df['Deck'].cat.codes.values
number = df['Number'].cat.codes.values
side = df['Side'].cat.codes.values
fn = df['FirstName'].cat.codes.values
ln = df['LastName'].cat.codes.values
ggg = df['gggg'].cat.codes.values
pp = df['pp'].cat.codes.values
categories = ['HomePlanet', 'CryoSleep', 'Destination', 'VIP', 'Deck', 'Number', 'Side', 'FirstName', 'LastName', 'gggg', 'person']
cats = np.stack([hp, cs, dest, vip, deck, number, side, fn, ln, ggg, pp], 1)
df_to_concat = pd.DataFrame(cats, columns=categories)
# 7. DataFrame Concatenation and Cleanup:
# - Concatenates encoded categorical columns, drops unnecessary columns.
df.drop(['PassengerId','Name','Cabin','HomePlanet', 'CryoSleep', 'Destination', 'VIP', 'Deck',
'Number', 'Side', 'FirstName', 'LastName', 'gggg', 'pp'], axis=1, inplace=True)
# Concatenate DataFrames
df = pd.concat([df, df_to_concat], axis=1, sort=False)
# Fill NaN values with zeros
df = df.fillna(0)
return dfAt next loading and Feature Engineering for Training Data:
- You read the training data from a CSV file (
../Data/train.csv). - Applied feature engineering using the
FeatureEngineeringfunction. - Converted the ‘Transported’ column to integer type.
- Prepared the features (
df_train_X) and labels (df_train_y) for training.
df_train = pd.read_csv('../Data/train.csv')
df_train = FeatureEnginiering(df_train)
df_train['Transported'] = df_train['Transported'].astype(int)
df_train_X = np.array(df_train.drop('Transported', axis=1))
df_train_y = np.array(df_train['Transported']).astype(np.float32)
df_train_X = np.expand_dims(df_train_X, 1)
df_train_y = np.expand_dims(df_train_y, 1)The Model
Description:
- The environment is designed for spaceship control, where actions are discrete (0 or 1).
- Observations include 18 features represented as a Box space.
- Data can be either randomly selected or follow a sequential order from the provided dataset.
class SpaceShipEnv(gym.Env):
"""
Action Space:
- Discrete space with two actions (0 or 1). For Classification
Observation Space:
- Box space with shape (1, 18) and dtype float32, representing a set of features for the space ship control task.
Methods:
- step(action): Takes an action and returns the next observation, reward, done flag, and additional info.
- reset(): Resets the environment to the initial state and returns the initial observation.
- _next_obs(): Returns the next observation based on the current dataset and mode.
Attributes:
- action_space: Discrete space with two actions (0 or 1).
- observation_space: Box space with shape (1, 18) and dtype float32.
- row_per_episode (int): Number of rows per episode.
- step_count (int): Counter for the number of steps within the current episode.
- x, y: Features and labels from the dataset.
- random (bool): If True, observations are selected randomly from the dataset; otherwise, follows a sequential order.
- dataset_idx (int): Index to keep track of the current observation in sequential mode.
- expected_action (int): Expected action based on the current observation.
"""
def __init__(self, row_per_episode=1, dataset=(df_train_X, df_train_y), random=True):
super().__init__()
# Define action space
self.action_space = gym.spaces.Discrete(2)
# Define observation space
obersevation = np.array([[np.finfo('float32').max] * 18], dtype=np.float32)
self.observation_space = spaces.Box(-obersevation, obersevation, shape=(1, 18), dtype=np.float32)
# Initialize parameters
self.row_per_episode = row_per_episode
self.step_count = 0
self.x, self.y = dataset
self.random = random
self.dataset_idx = 0
def step(self, action):
"""
Takes an action and returns the next observation, reward, done flag, and additional info.
Parameters:
- action (int): The action taken by the agent.
Returns:
- obs (numpy array): The next observation.
- reward (int): The reward obtained based on the action.
- done (bool): Flag indicating whether the episode is done.
- info (dict): Additional information.
"""
done = False
reward = int(action == self.expected_action)
obs = self._next_obs()
self.step_count += 1
if self.step_count >= self.row_per_episode:
done = True
return obs, reward, done, {}
def reset(self):
"""
Resets the environment to the initial state and returns the initial observation.
Returns:
- obs (numpy array): The initial observation.
"""
self.step_count = 0
obs = self._next_obs()
return obs
def _next_obs(self):
"""
Returns the next observation based on the current dataset and mode.
Returns:
- obs (numpy array): The next observation.
"""
if self.random:
next_obs_idx = random.randint(0, len(self.x) - 1)
self.expected_action = int(self.y[next_obs_idx])
obs = self.x[next_obs_idx]
else:
obs = self.x[self.dataset_idx]
self.expected_action = int(self.y[self.dataset_idx])
self.dataset_idx += 1
if self.dataset_idx >= len(self.x):
self.dataset_idx = 0
return obsLets Train the model
In this code section, I first configure a logger to store logs in the ‘../logs/mnist_ppo’ directory and capture both standard output and TensorBoard logs. Then I initialize an environment (SpaceShipEnv) with certain parameters like row_per_episode=1 and random=False. This environment is wrapped using DummyVecEnv and monitored for logging purposes. I then initialize a PPO model (‘MlpPolicy’) with the specified environment. The model is trained for a total of 12,000 time steps using the model.learn function. Finally, I measure the training time using the time module and output it, noting that the time measurement is not correct because start_time is not initialized properly.
#1. Import necessary libraries and modules:
%%capture
logger.configure('../logs/mnist_ppo', ['stdout', 'tensorboard'])
env = SpaceShipEnv(row_per_episode=1, random = False)
env = DummyVecEnv([lambda: monitor.Monitor(SpaceShipEnv(row_per_episode=1, random = False),
logger.Logger('./logs/mnist_ppo',['stdout', 'tensorboard']).get_dir())])
#2. Initialize a PPO model:
model = PPO(
'MlpPolicy', env,
verbose=1)
#3. Train the PPO model:
model.learn(total_timesteps=int(1.2e4))
#4. Measure and print the training time:
start_time = time.time()
print("PPO Training Time:", time.time() - start_time)
To save the model
#model.save('ppo_TitanicSpaceShip.pkl')Evaluation
In these steps, I loop 12,000 time steps to run the trained PPO model on the environment and measure the accuracy. In this loop, actions are repeatedly predicted by the trained model and executed on the environment. Accuracy is calculated by counting the number of times the reward is positive (i.e. the action was correct) and the result is reported as a percentage. It was noted that the observation status is reset after each action.
attempts, correct = 0,0
obs = env.reset()
for i in range(12000):
action, done = model.predict(obs)
#print(action)
obs, rewards, dones, info = env.step(action)
attempts += 1
if rewards > 0:
correct += 1
obs = env.reset()
print('Accuracy: {0}%'.format((float(correct) / attempts) * 100))Accuracy: 81.0167%
For the prediction
action, done = model.predict(df_test_X)
predictions = np.expand_dims(action, 1)It’s fun to experiment with reinforcement models
