This study aims to analyze the behavior of retail traders in capital markets and propose an intelligent framework based on reinforcement learning and high-frequency limit order book data. In recent years, the increasing complexity of financial markets and the growing availability of high-frequency data have highlighted the need for advanced computational models to understand and predict market behavior. In this context, the capital market is modeled as a complex adaptive system, where retail traders’ behavior is inferred from limit order book data using reinforcement learning algorithms. Three main algorithms, including Q-learning and Deep Q-Network (DQN), Actor-Critic, and Soft Actor-Critic (SAC), are employed to model and predict market behavior. The dataset consists of features extracted from the high-frequency limit order book, including order imbalance, market depth, order flow, and short-term price dynamics. The results indicate that reinforcement learning models, particularly the Soft Actor-Critic approach, outperform other methods in terms of prediction accuracy, cumulative returns, and risk management. Furthermore, order imbalance is identified as the most influential feature in short-term market direction prediction. The findings suggest that integrating reinforcement learning with micro-level market data enables the extraction of hidden behavioral patterns of retail traders and significantly improves decision-making in dynamic financial environments. This framework can contribute to the development of intelligent trading systems and enhance the understanding of market behavior at the microstructure level.
Eslami,N . (2025). Modeling Retail Traders’ Behavior in Capital Markets Using Reinforcement Learning and High-Frequency Order Book Data. Journal of Intelligent Financial Management, 1(2), 71-86. doi: 10.25843/JIFM.2025.8563.23450
MLA
Eslami,N . "Modeling Retail Traders’ Behavior in Capital Markets Using Reinforcement Learning and High-Frequency Order Book Data", Journal of Intelligent Financial Management, 1, 2, 2025, 71-86. doi: 10.25843/JIFM.2025.8563.23450
HARVARD
Eslami N. (2025). 'Modeling Retail Traders’ Behavior in Capital Markets Using Reinforcement Learning and High-Frequency Order Book Data', Journal of Intelligent Financial Management, 1(2), pp. 71-86. doi: 10.25843/JIFM.2025.8563.23450
CHICAGO
N Eslami, "Modeling Retail Traders’ Behavior in Capital Markets Using Reinforcement Learning and High-Frequency Order Book Data," Journal of Intelligent Financial Management, 1 2 (2025): 71-86, doi: 10.25843/JIFM.2025.8563.23450
VANCOUVER
Eslami N. Modeling Retail Traders’ Behavior in Capital Markets Using Reinforcement Learning and High-Frequency Order Book Data. JIFM. 2025;1(2):71-86 (In Persian). doi: 10.25843/JIFM.2025.8563.23450
