This study aims to analyze the behavior of retail traders in capital markets and propose an intelligent framework based on reinforcement learning and high-frequency limit order book data. In recent years, the increasing complexity of financial markets and the growing availability of high-frequency data have highlighted the need for advanced computational models to understand and predict market behavior. In this context, the capital market is modeled as a complex adaptive system, where retail traders’ behavior is inferred from limit order book data using reinforcement learning algorithms.
Three main algorithms, including Q-learning and Deep Q-Network (DQN), Actor-Critic, and Soft Actor-Critic (SAC), are employed to model and predict market behavior. The dataset consists of features extracted from the high-frequency limit order book, including order imbalance, market depth, order flow, and short-term price dynamics. The results indicate that reinforcement learning models, particularly the Soft Actor-Critic approach, outperform other methods in terms of prediction accuracy, cumulative returns, and risk management. Furthermore, order imbalance is identified as the most influential feature in short-term market direction prediction.
The findings suggest that integrating reinforcement learning with micro-level market data enables the extraction of hidden behavioral patterns of retail traders and significantly improves decision-making in dynamic financial environments. This framework can contribute to the development of intelligent trading systems and enhance the understanding of market behavior at the microstructure level.