A Deep Reinforcement Learning Approach for Automated Cryptocurrency Trading by Giorgio Lucarelli, Matteo Borrotti

1 Introduction
Nowadays, Artificial Intelligence (AI) is reshaping our daily life. AI is the study
and design of intelligent agents where an agent is a system that perceives its
environment and takes actions in order to maximize its chances of success. AI
excels at interpreting signals and real-time analytic which underpin many different

applications. For instance, AI is changing the way medical science was
perceived just few years ago. Autonomous machines play an increasingly important

role in surgery, improving patient outcomes and reducing expensive hospital
stay time. Elsewhere, computer vision are improving diagnostic technologies and
making them more accessible, while predictive algorithms are facilitating more
rapid drug discovery. A less noble application is related to the financial sector,
where AI is used to build automatic trading systems which are poised to foster a
new financial technology transformation. Furthermore, the arrival of cryptocurrency

has given new interest in the application of AI techniques for predicting
the future price of a financial asset (i.e. Bitcoin).
In this context, Reinforcement Learning (RL) has demonstrated the
potential to transform how classical trading systems work. RL is an autonomous,

self-teaching system that essentially learns by trial and error. It performs actions
with the aim to maximize rewards and achieve the best outcomes.
In this work, we investigate the performance of two different trading systems based

on deep RL approaches: Double Deep Q-Network (D-DQN) and
Dueling Double Deep Q-Network (DD-DQN). The two trading systems are
compared with a Deep Q-Network (DQN).
The article is structured as follows: Section 2 provides a definition of cryptocurrency

and bitcoin. Section 3 gives a short description of Reinforcement
Learning. Section 4 introduces and describes the proposed Q-learning trading
system. Main results are reported in Section 5 and Section 6 concludes the work.
1.1 Related Work
Deep Learning (DL) and Reinforcement Learning (RL) are viable approaches
for market making. In recent years, the use of DL and RL is increased a lot
demonstrating the powerful of these techniques.
McNally, S. et al. (2018) applied different Machine Learning (ML) techniques

on bitcoin cryptocurrency. More precisely, they compared Recurrent Neural Network

(RNN) and Long Short Term Memory (LSTM) network against
a more classical approach such as AutoRegressive Integrated Moving Average
(ARIMA) model. RNN and LSTM outperformed ARIMA in a traditional classification setting.
Patel, Y. (2018) proposed a multi-agent approach that operates at two
different levels: (i) minute level (macro-agent) and (ii) order book level (microagent).

The macro-agent is based on a Double Q-learning network composed by
a Multi-Layer Perceptron (MLP) and the micro-agent is realized with a Dueling
Double Q-learning network with reward function based on volume weighted average

bitcoin price. The multi-agent did not outperfom the simple macro-agent
but it obtained better results with respect to a uniform Buy and Hold and Momentum

Investing techniques in terms of cumulative profits.
Previous works were applied only to bitcoin movements, Bu, S.-J. et al (2018)
[5] tested a hybrid approach (Boltzmann machine and Double Q-learning network)

against LSTM, MLP, Convolutional Neural Network (CNN) over eigth
cryptocurrencies. They used the ratio between total value after investement and
initial value as evaluation score. The hybrid approach demonstrated to be more
profitable than competitors but more risky and unstable.
Alessandretti, L. and coauthors (2018) and Jiang, Z. et al. (2017)
applied Artificial Intellingent (AI) approaches on portfolio management. In
the authors applied a gradient boosting decision tree (i.e. XGBoost) and LSTM
network on a cryptocurrency portfolio. Performance were evaluated considering
Sharpe ratio [20] and geometric mean return. All proposed strategies produced
profit over the entire test period. Jiang, Z. et al. (2017) applied a deterministic

policy gradient using a direct reward function (average logarithmic return)
for solving the portfolio management problem. The approach demonstrated to
outperfom classical management techniques except against a Passive Aggressive
Mean Reversion technique in terms of cumulative return.

A Deep Reinforcement Learning Approach for
Automated Cryptocurrency Trading
Giorgio Lucarelli, Matteo Borrotti