Welcome to Download Crypto Currencies Data project¶
This is the documentation of dccd
package, package to download crypto-currencies data from Binance, Bitfinex, Bitmex, GDax (Coinbase), Kraken and Poloniex.
Installation¶
- From pip:
- $ pip install dccd
- From source:
$ git clone https://github.com/ArthurBernard/Download_Crypto_Currencies_Data.git
$ cd Download_Crypto_Currencies_Data
$ python setup.py install –user
Presentation¶
The dccd
package allow you two main methods to download data. The first one is recommended to download data at high frequency (minutely or tick by tick), and the second one is recommended to download data at a lower frequency (hourly or daily):
- Continuous Downloader
dccd.continuous_dl
: - Download and update continuously data (orderbook, trades tick by tick, ohlc, etc) and save it in a database. Currently only support Bitfinex and Bitmex exchanges.
- Continuous Downloader
- Historical Downloader
dccd.histo_dl
: - Download historical data (ohlc, trades, etc.) and save it. Currently only support Binance, GDax, Kraken and Poloniex exchanges.
- Historical Downloader
Contents¶
Continuous Downloader (dccd.continuous_dl
)¶
Module to download continuously data.
Module to download continuously data (orderbook, trades, etc.) and update automatically the database. Currently only supports Bitfinex and Bitmex exchanges.
Bitfinex Continuous Downloader (dccd.continuous_dl.bitfinex
)¶
Objects and functions to download data from Bitfinex exchange.
These functions and objects allow you to continuously download data and update your database.
High level API¶
-
dccd.continuous_dl.bitfinex.
get_data_bitfinex
(channel, process_func, process_params={}, save_method='dataframe', io_params={}, time_step=60, until=None, path=None, **kwargs)¶ Download data from Bitfinex exchange and update the database.
Parameters: - channel : str, {‘book’, ‘book_raw’, ‘trades’, ‘trades_raw’}
Websocket channel to get data, by default data will be aggregated (OHLC for ‘trades’ and reconstructed orderbook for ‘book’), add ‘_raw’ to the channel to get raw data (trade tick by tick or each orders).
- process_func : callable
Function to process and clean data before to be saved. Must take data in arguments and can take any optional keywords arguments, cf function exemples in
dccd.process_data
.- process_params : dict, optional
Dictionary of the keyword arguments available to process_func, cf documentation into
dccd.process_data
.- save_method : {‘DataFrame’, ‘SQLite’, ‘CSV’, ‘Excel’, ‘PostgreSQL’, ‘Oracle’, ‘MSSQL’, ‘MySQL’},
It will create an IODataBase object to save/update the database in the specified format save_method, default is ‘DataFrame’ it save as binary pd.DataFrame object. More informations are available into
dccd.tools.io
.- io_params : dict, optional
Dictionary of the keyword arguments available to the
dccd.tools.io.IODataBase
callable method. Note: With SQL format some parameters are compulsory, see details intodccd.tools.io
.- time_step : int, optional
Number of second between two snapshots of data, default 60 (1 minute).
- until : int, optional
Number of seconds before stoping to download and update, default is None. If until equal 0 or None it means it never stop.
- path : str, optional
Path to save/update the database, default is None. If path is None, database is saved at the relative path ‘./database/bitfinex/channel’.
- **kwargs
Any revelevant keyword arguments will be passed to the websocket connector, see Bitfinex API documentation [2] for more details.
Warning
‘book_raw’ and ‘trades_raw’ can be very memory expensive.
See also
process_data
- function to process/clean data (set_marketdepth, set_ohlc, set_orders, set_marketdepth).
tools.io.IODataBase
- object to save/update the database with respect to specified format.
References
[2] https://docs.bitfinex.com/v2/docs/ws-public
-
dccd.continuous_dl.bitfinex.
get_orderbook_bitfinex
(symbol, precision='P0', frequency='F0', lenght='25', time_step=60, until=None, path=None, save_method='dataframe', io_params={})¶ Download orderbook from Bitfinex exchange.
-
dccd.continuous_dl.bitfinex.
get_trades_bitfinex
(symbol, time_step=60, until=None, path=None, save_method='dataframe', io_params={})¶ Download trades tick by tick from Bitfinex exchange.
Low level API¶
-
class
dccd.continuous_dl.bitfinex.
DownloadBitfinexData
(time_step=60, until=3600)¶ Bases:
dccd.continuous_dl.exchange.ContinuousDownloader
Basis object to download data from a stream websocket client API.
Parameters: - time_step : int, optional
Number of seconds between two snapshots of data, minimum is 1, default is 60 (one minute). Each time_step data will be processed and updated to the database.
- until : int, optional
Number of seconds before stoping or timestamp of when stoping, default is 3600 (one hour).
Attributes: - host : str
Adress of host to connect.
- conn_par : dict
Parameters of websocket connection.
- ws : websockets.client.WebSocketClientProtocol
Connection with the websocket client.
- is_connect : bool
True if is connected, False otherwise.
- ts : int
Number of second between two snapshots of data.
- t : int
Current timestamp but rounded by ts.
- until : int
Timestamp to stop to download data.
Methods
set_process_data
(self, func, \*\*kwargs)Set processing function. set_saver
(self, call, \*\*kwargs)Set saver object to save data or update a database. __call__
(self, channel, \*\*kwargs)Open a websocket connection and save/update the database. -
__call__
(self, channel, **kwargs)¶ Open a websocket connection and save/update the database.
Run asynchronously two loops to get data from bitfinex websocket and save/update the database.
Parameters: - channel : {‘book’, ‘book_raw’, ‘trades’, ‘trades_raw’}
Channel to get data, by default data will be aggregated (OHLC for ‘trades’ and reconstructed orderbook for ‘book’), add ‘_raw’ to the channel to get raw data (trade tick by tick or each orders).
- **kwargs
Any revelevant keyword arguments will be passed to the websocket connector, see API documentation [1] for more details.
Warning
‘book_raw’ and ‘trades_raw’ can be very memory expensive.
References
[1] https://docs.bitfinex.com/v2/docs/ws-public
-
get_parser
(self, key)¶ Get allowed data parser.
Parameters: - key : str
Name code of data to parse. If key is not allowed then return a debug_parser which will display data structure.
Returns: - function
The allowed function to parse this kind of data.
-
on_close
(self)¶ On websocket close print and fire event.
-
on_error
(self, error, *args)¶ On websocket error print and fire event.
-
on_message
(self, data)¶ Set data to order book.
-
on_open
(self, **kwargs)¶ On websocket open.
Parameters: - **kwargs
Any relevant keyword arguments to set connection.
-
parser_book
(self, data)¶ Parse market depth of order book.
Parameters: - data : list
Order data.
-
parser_raw_book
(self, data)¶ Parse raw order book, each timestep set in a list all orders.
Parameters: - data : list
Order data.
-
parser_raw_trades
(self, data)¶ Parse trade data.
Parameters: - data : list
Trade data.
-
parser_trades
(self, data)¶ Parse OHLC data.
Parameters: - data : list
Trade data.
-
set_process_data
(self, func, **kwargs)¶ Set processing function.
Parameters: - func : callable
Function to process and clean data before to be saved. Must take data in arguments and can take any optional keywords arguments, cf exemples in
dccd.process_data
.- **kwargs
Any keyword arguments to be passed to func.
-
set_saver
(self, call, **kwargs)¶ Set saver object to save data or update a database.
Parameters: - call : callable
Callable object to save data or update a database. Must take data in arguments and can take any optional keywords arguments, cf exemples in
dccd.io_tools
.- **kwargs
Any keyword arguments to be passed to call.
-
wait_that
(self, is_true)¶ Wait before running.
Bitmex Continuous Downloader (dccd.continuous_dl.bitmex
)¶
Objects and functions to download data from Bitmex exchange.
These functions and objects allow you to continuously download data and update your database.
High level API¶
-
dccd.continuous_dl.bitmex.
get_data_bitmex
(process_func, *args, time_step=60, until=None, path=None, save_method='dataframe', io_params={}, **kwargs)¶ Download data from Bitmex exchange and update the database.
Parameters: - channel : str, {‘book’, ‘book_raw’, ‘trades’, ‘trades_raw’}
Websocket channel to get data, by default data will be aggregated (OHLC for ‘trades’ and reconstructed orderbook for ‘book’), add ‘_raw’ to the channel to get raw data (trade tick by tick or each orders).
- process_func : callable
Function to process and clean data before to be saved. Must take data in arguments and can take any optional keywords arguments, cf function exemples in
dccd.process_data
.- process_params : dict, optional
Dictionary of the keyword arguments available to process_func, cf documentation into
dccd.process_data
.- save_method : {‘DataFrame’, ‘SQLite’, ‘CSV’, ‘Excel’, ‘PostgreSQL’, ‘Oracle’, ‘MSSQL’, ‘MySQL’},
It will create an IODataBase object to save/update the database in the specified format save_method, default is ‘DataFrame’ it save as binary pd.DataFrame object. More informations are available into
dccd.tools.io
.- io_params : dict, optional
Dictionary of the keyword arguments available to the
dccd.tools.io.IODataBase
callable method. Note: With SQL format some parameters are compulsory, seed details intodccd.tools.io
.- time_step : int, optional
Number of second between two snapshots of data, default 60 (1 minute).
- until : int, optional
Number of seconds before stoping to download and update, default is None. If until equal 0 or None it means it never stop.
- path : str, optional
Path to save/update the database, default is None. If path is None, database is saved at the relative path ‘./database/bitmex/channel’.
- **kwargs
Any revelevant keyword arguments will be passed to the websocket connector, see Bitmex API documentation [2] for more details.
Warning
‘_raw’ option not yet working for bitmex.
See also
process_data
- function to process/clean data (set_marketdepth, set_ohlc, set_orders, set_marketdepth).
tools.io.IODataBase
- object to save/update the database with respect to specified format.
References
[2] https://www.bitmex.com/api/
-
dccd.continuous_dl.bitmex.
get_orderbook_bitmex
(*args, time_step=60, until=None, path=None, save_method='dataframe', io_params={})¶ Download orderbook from Bitmex exchange.
-
dccd.continuous_dl.bitmex.
get_trades_bitmex
(*args, time_step=60, until=None, path=None, save_method='dataframe', io_params={})¶ Download trades tick by tick from Bitmex exchange.
Low level API¶
-
class
dccd.continuous_dl.bitmex.
DownloadBitmexData
(time_step=60, until=3600)¶ Bases:
dccd.continuous_dl.exchange.ContinuousDownloader
Basis object to download data from a stream websocket client API.
Parameters: - time_step : int, optional
Number of seconds between two snapshots of data, minimum is 1, default is 60 (one minute). Each time_step data will be processed and updated to the database.
- until : int, optional
Number of seconds before stoping or timestamp of when stoping, default is 3600 (one hour).
Attributes: - host : str
Adress of host to connect.
- conn_par : dict
Parameters of websocket connection.
- ws : websockets.client.WebSocketClientProtocol
Connection with the websocket client.
- is_connect : bool
True if is connected, False otherwise.
- ts : int
Number of second between two snapshots of data.
- t : int
Current timestamp but rounded by ts.
- until : int
Timestamp to stop to download data.
Methods
set_process_data
(self, func, \*\*kwargs)Set processing function. set_saver
(self, call, \*\*kwargs)Set saver object to save data or update a database. __call__
(self, \*args)Open a websocket connection and save/update the database. -
__call__
(self, *args)¶ Open a websocket connection and save/update the database.
Run asynchronously two loops to get data from Bitmex websocket and save/update the database.
Parameters: - channel : {‘orderBookL2_25’, ‘trade’}
Channel to get data, by default data will be aggregated (OHLC for ‘trades’ and reconstructed orderbook for ‘orderBookL2_25’), add ‘_raw’ to the channel to get raw data (trade tick by tick or each orders).
- **kwargs
Any revelevant keyword arguments will be passed to the websocket connector, see API documentation [1] for more details.
Warning
‘_raw’ option not yet working for Bitmex.
References
[1] https://www.bitmex.com/api/
-
get_parser
(self, key)¶ Get allowed data parser.
Parameters: - key : str
Name code of data to parse. If key is not allowed then return a debug_parser which will display data structure.
Returns: - function
The allowed function to parse this kind of data.
-
on_close
(self)¶ On websocket close print and fire event.
-
on_error
(self, error, *args)¶ On websocket error print and fire event.
-
on_message
(self, data)¶ Set data to order book.
-
on_open
(self, **kwargs)¶ On websocket open.
Parameters: - **kwargs
Any relevant keyword arguments to set connection.
-
parser_book
(self, data)¶ Parse data of order book.
Parameters: - data : dict
Order data from the ws API.
-
parser_trades
(self, data)¶ Parse data of trades.
Parameters: - data : dict
Order data from the ws API.
-
set_process_data
(self, func, **kwargs)¶ Set processing function.
Parameters: - func : callable
Function to process and clean data before to be saved. Must take data in arguments and can take any optional keywords arguments, cf exemples in
dccd.process_data
.- **kwargs
Any keyword arguments to be passed to func.
-
set_saver
(self, call, **kwargs)¶ Set saver object to save data or update a database.
Parameters: - call : callable
Callable object to save data or update a database. Must take data in arguments and can take any optional keywords arguments, cf exemples in
dccd.io_tools
.- **kwargs
Any keyword arguments to be passed to call.
-
wait_that
(self, is_true)¶ Wait before running.
High level API¶
bitfinex.get_data_bitfinex (channel, process_func) |
Download data from Bitfinex exchange and update the database. |
bitfinex.get_orderbook_bitfinex (symbol[, …]) |
Download orderbook from Bitfinex exchange. |
bitfinex.get_trades_bitfinex (symbol[, …]) |
Download trades tick by tick from Bitfinex exchange. |
bitmex.get_data_bitmex (process_func, \*args) |
Download data from Bitmex exchange and update the database. |
bitmex.get_orderbook_bitmex (\*args[, …]) |
Download orderbook from Bitmex exchange. |
bitmex.get_trades_bitmex (\*args[, …]) |
Download trades tick by tick from Bitmex exchange. |
Low level API¶
bitfinex.DownloadBitfinexData ([time_step, until]) |
Basis object to download data from a stream websocket client API. |
bitmex.DownloadBitmexData ([time_step, until]) |
Basis object to download data from a stream websocket client API. |
Historical Downloader (dccd.histo_dl
)¶
Module to download historical data.
Module to download historical data (ohlc, trades, etc.) and automatically update the database. Currently only supports Binance, GDax, Kraken and Poloniex exchanges.
Only Poloniex allow you to download old historical data.
The ‘histo_dl’ module contains a main class and four classes to download and update data for each exchange.
The four classes to download data are FromBinance
, FromGDax
,
FromKraken
and FromPoloniex
. All have the same methods and almost
the same parameters:
- __init__(path, crypto, span, fiat(optional), form(optional)):
- Initialisation with path is the path where save the data (string), crypto is a crypto currency (string) and span is the interval time between each observation in seconds (integer) or can be a string as ‘hourly’, ‘daily’, etc. (see details on the doc string). The optional parameters are fiat the second currency (default is ‘USD’ and ‘USDT’ for poloniex and binance) and form the format to save the data (default is ‘xlsx’).
- import_data(start, end):
- Download data with start and end the timestamp (integer) or the date and time (string as ‘yyyy-mm-dd hh:mm:ss’), respectively of the first observation and the last observation (default are special parameters start=’last’ allow the last data saved and end=’now’ allow the last observation available). Exclusion: Kraken don’t allow the end parameter and provide only the thousand last observations.
- save(form(optional), by(optional)):
- Save the data with form the format of the saved data (default is ‘xlsx’) and by is the “size” of each saved file (default is ‘Y’ as an entire year). Exclusion: This optional parameters are in progress, let the default parameter for the moment, other are not allow.
- get_data():
- returns the data frame without any parameter.
Method chaining is available for these classes.
Binance Historical Downloader (dccd.histo_dl.binance
)¶
Objects to download historical data from Binance exchange.
-
class
dccd.histo_dl.binance.
FromBinance
(path, crypto, span, fiat='USD', form='xlsx')¶ Bases:
dccd.histo_dl.exchange.ImportDataCryptoCurrencies
Class to import crypto-currencies data from the Binance exchange.
Parameters: - path : str
The path where data will be save.
- crypto : str
The abreviation of the crypto-currency.
- span : {int, ‘weekly’, ‘daily’, ‘hourly’}
- If str, periodicity of observation.
- If int, number of the seconds between each observation, minimal span is 60 seconds.
- fiat : str
A fiat currency or a crypto-currency. Binance don’t allow fiat currencies, but USD theter.
- form : {‘xlsx’, ‘csv’}
Your favorit format. Only ‘xlsx’ and ‘csv’ for the moment.
See also
FromGDax
,FromKraken
,FromPoloniex
Notes
See Binance API documentation [Rbab00e26f0b6-1] for more details on parameters.
References
[Rbab00e26f0b6-1] https://github.com/binance-exchange/binance-official-api-docs Attributes: - pair : str
Pair symbol, crypto + fiat.
- start, end : int
Timestamp to starting and ending download data.
- span : int
Number of seconds between observations.
- full_path : str
Path to save data.
- form : str
Format to save data.
Methods
import_data
(self[, start, end])Download data from Binance for specific time interval. save
(self[, form, by_period])Save data by period (default is year) in the corresponding format and file. get_data
(self)Print the dataframe. -
get_data
(self)¶ Print the dataframe.
Returns: - Data : pd.DataFrame
Current data.
-
import_data
(self, start='last', end='now')¶ Download data from Binance for specific time interval.
Parameters: - start : int or str
Timestamp of the first observation of you want as int or date format ‘yyyy-mm-dd hh:mm:ss’ as string.
- end : int or str
Timestamp of the last observation of you want as int or date format ‘yyyy-mm-dd hh:mm:ss’ as string.
Returns: - data : pd.DataFrame
Data sorted and cleaned in a data frame.
-
save
(self, form='xlsx', by_period='Y')¶ Save data by period (default is year) in the corresponding format and file.
TODO : to finish
Parameters: - form : {‘xlsx’, ‘csv’}
Format to save data.
- by_period : {‘Y’, ‘M’, ‘D’}
- If ‘Y’ group data by year.
- If ‘M’ group data by month.
- If ‘D’ group data by day.
-
set_hierarchy
(self, liste)¶ Set the specific hierarchy of the files where will save your data.
TODO : to finish
GDax Historical Downloader (dccd.histo_dl.gdax
)¶
Objects to download historical data from GDax exchange.
-
class
dccd.histo_dl.gdax.
FromGDax
(path, crypto, span, fiat='USD', form='xlsx')¶ Bases:
dccd.histo_dl.exchange.ImportDataCryptoCurrencies
Class to import crypto-currencies data from the GDax exchange.
Parameters: - path : str
The path where data will be save.
- crypto : str
The abreviation of the crypto-currency.
- span : {int, ‘weekly’, ‘daily’, ‘hourly’}
- If str, periodicity of observation.
- If int, number of the seconds between each observation, minimal span is 60 seconds.
- fiat : str
A fiat currency or a crypto-currency.
- form : {‘xlsx’, ‘csv’}
Your favorit format. Only ‘xlsx’ and ‘csv’ for the moment.
See also
FromBinance
,FromKraken
,FromPoloniex
Notes
See GDax API documentation [Rfd0c943f5f8a-1] for more details on parameters.
References
[Rfd0c943f5f8a-1] https://docs.pro.coinbase.com/ Attributes: - pair : str
Pair symbol, crypto + fiat.
- start, end : int
Timestamp to starting and ending download data.
- span : int
Number of seconds between observations.
- full_path : str
Path to save data.
- form : str
Format to save data.
Methods
import_data
(self[, start, end])Download data from GDax for specific time interval. save
(self[, form, by_period])Save data by period (default is year) in the corresponding format and file. get_data
(self)Print the dataframe. -
get_data
(self)¶ Print the dataframe.
Returns: - Data : pd.DataFrame
Current data.
-
import_data
(self, start='last', end='now')¶ Download data from GDax for specific time interval.
Parameters: - start : int or str
Timestamp of the first observation of you want as int or date format ‘yyyy-mm-dd hh:mm:ss’ as string.
- end : int or str
Timestamp of the last observation of you want as int or date format ‘yyyy-mm-dd hh:mm:ss’ as string.
Returns: - data : pd.DataFrame
Data sorted and cleaned in a data frame.
-
save
(self, form='xlsx', by_period='Y')¶ Save data by period (default is year) in the corresponding format and file.
TODO : to finish
Parameters: - form : {‘xlsx’, ‘csv’}
Format to save data.
- by_period : {‘Y’, ‘M’, ‘D’}
- If ‘Y’ group data by year.
- If ‘M’ group data by month.
- If ‘D’ group data by day.
-
set_hierarchy
(self, liste)¶ Set the specific hierarchy of the files where will save your data.
TODO : to finish
Kraken Historical Downloader (dccd.histo_dl.kraken
)¶
Objects to download historical data from Kraken exchange.
-
class
dccd.histo_dl.kraken.
FromKraken
(path, crypto, span, fiat='USD', form='xlsx')¶ Bases:
dccd.histo_dl.exchange.ImportDataCryptoCurrencies
Class to import crypto-currencies data from the Kraken exchange.
Parameters: - path : str
The path where data will be save.
- crypto : str
The abreviation of the crypto-currency.
- span : {int, ‘weekly’, ‘daily’, ‘hourly’}
- If str, periodicity of observation.
- If int, number of the seconds between each observation, minimal span is 60 seconds.
- fiat : str
A fiat currency or a crypto-currency.
- form : {‘xlsx’, ‘csv’}
Your favorit format. Only ‘xlsx’ and ‘csv’ for the moment.
See also
FromBinance
,FromGDax
,FromPoloniex
Notes
See Kraken API documentation [R1ab1d422d7d0-1] for more details on parameters.
References
[R1ab1d422d7d0-1] https://www.kraken.com/features/api Attributes: - pair : str
Pair symbol, crypto + fiat.
- start, end : int
Timestamp to starting and ending download data.
- span : int
Number of seconds between observations.
- full_path : str
Path to save data.
- form : str
Format to save data.
Methods
import_data
(self[, start, end])Download data from Kraken since a specific time until now. save
(self[, form, by_period])Save data by period (default is year) in the corresponding format and file. get_data
(self)Print the dataframe. -
get_data
(self)¶ Print the dataframe.
Returns: - Data : pd.DataFrame
Current data.
-
import_data
(self, start='last', end=None)¶ Download data from Kraken since a specific time until now.
Parameters: - start : int or str
Timestamp of the first observation of you want as int or date format ‘yyyy-mm-dd hh:mm:ss’ as string.
Returns: - data : pd.DataFrame
Data sorted and cleaned in a data frame.
-
save
(self, form='xlsx', by_period='Y')¶ Save data by period (default is year) in the corresponding format and file.
TODO : to finish
Parameters: - form : {‘xlsx’, ‘csv’}
Format to save data.
- by_period : {‘Y’, ‘M’, ‘D’}
- If ‘Y’ group data by year.
- If ‘M’ group data by month.
- If ‘D’ group data by day.
-
set_hierarchy
(self, liste)¶ Set the specific hierarchy of the files where will save your data.
TODO : to finish
Poloniex Historical Downloader (dccd.histo_dl.poloniex
)¶
Objects to download historical data from Poloniex exchange.
-
class
dccd.histo_dl.poloniex.
FromPoloniex
(path, crypto, span, fiat='USD', form='xlsx')¶ Bases:
dccd.histo_dl.exchange.ImportDataCryptoCurrencies
Class to import crypto-currencies data from the Poloniex exchange.
Parameters: - path : str
The path where data will be save.
- crypto : str
The abreviation of the crypto-currency.
- span : {int, ‘weekly’, ‘daily’, ‘hourly’}
- If str, periodicity of observation.
- If int, number of the seconds between each observation, minimal span is 300 seconds.
- fiat : str
A fiat currency or a crypto-currency. Poloniex don’t allow fiat currencies, but USD theter.
- form : {‘xlsx’, ‘csv’}
Your favorit format. Only ‘xlsx’ and ‘csv’ for the moment.
See also
FromBinance
,FromGDax
,FromKraken
Notes
See Poloniex API documentation [R5770b2672718-1] for more details on parameters.
References
[R5770b2672718-1] https://docs.poloniex.com/#introduction Attributes: - pair : str
Pair symbol, crypto + fiat.
- start, end : int
Timestamp to starting and ending download data.
- span : int
Number of seconds between observations.
- full_path : str
Path to save data.
- form : str
Format to save data.
Methods
import_data
(self[, start, end])Download data from Poloniex for specific time interval. save
(self[, form, by_period])Save data by period (default is year) in the corresponding format and file. get_data
(self)Print the dataframe. -
get_data
(self)¶ Print the dataframe.
Returns: - Data : pd.DataFrame
Current data.
-
import_data
(self, start='last', end='now')¶ Download data from Poloniex for specific time interval.
Parameters: - start : int or str
Timestamp of the first observation of you want as int or date format ‘yyyy-mm-dd hh:mm:ss’ as string.
- end : int or str
Timestamp of the last observation of you want as int or date format ‘yyyy-mm-dd hh:mm:ss’ as string.
Returns: - data : pd.DataFrame
Data sorted and cleaned in a data frame.
-
save
(self, form='xlsx', by_period='Y')¶ Save data by period (default is year) in the corresponding format and file.
TODO : to finish
Parameters: - form : {‘xlsx’, ‘csv’}
Format to save data.
- by_period : {‘Y’, ‘M’, ‘D’}
- If ‘Y’ group data by year.
- If ‘M’ group data by month.
- If ‘D’ group data by day.
-
set_hierarchy
(self, liste)¶ Set the specific hierarchy of the files where will save your data.
TODO : to finish
Processing Data (dccd.process_data
)¶
Functions to clean, sort and other process data.
-
dccd.process_data.
set_marketdepth
(book, t=None)¶ Set a market depth dataframe with list of order books.
Parameters: - book : dict
Orderbook as dict, where keys is the price and value is the amount.
Returns: - pd.DataFrame
Order book as dataframe.
-
dccd.process_data.
set_ohlc
(trades, ts=60)¶ Aggregate and set a dataframe with list of trades.
Parameters: - trades : list
Historical trades tick by tick as list.
- ts : int, optional
Timestep in seconds to aggregate data, default is 60.
Returns: - pd.DataFrame
Aggregated trades as OHLC, dataframe is indexed by timestamp and columns contains ‘open’, ‘high’, ‘low’, ‘close’, and ‘volume’.
-
dccd.process_data.
set_orders
(orders, t=None)¶ Set a dataframe with list of each order.
Parameters: - orders : list
Each orders in a list.
Returns: - pd.DataFrame
List of orders as dataframe.
-
dccd.process_data.
set_trades
(trades)¶ Set a dataframe with list of trades.
Parameters: - trades : list
Historical trades tick by tick as list.
Returns: - pd.DataFrame
Historical trades tick by tick as dataframe.
Tools (dccd.tools
)¶
Module with I/O, date/time and websocket tools.
Time and date tools (dccd.tools.date_time
)¶
Tools to manage some time functions with respect to specific API exchanges.
-
dccd.tools.date_time.
TS_to_date
(TS, form='%Y-%m-%d %H:%M:%S', local=True)¶ Convert timestamp to date in specified format.
Parameters: - TS : int
A timestamp to convert.
- form : str (default ‘%Y-%m-%d %H:%M:%S’)
Time format.
- local : bool (default is True)
Local time is used if true else return UTC time.
Returns: - date : str
Date as specified format.
Examples
>>> TS_to_date(1548432099, form='%y-%m-%d %H:%M:%S', local=False) '19-01-25 16:01:39'
-
dccd.tools.date_time.
date_to_TS
(date, form='%Y-%m-%d %H:%M:%S')¶ Use your local time-zone to convert date in specific format to timestamp.
Parameters: - date : str
A date to convert.
- form : str (default ‘%Y-%m-%d %H:%M:%S’)
Time format.
Returns: - TS : int
Timestamp of specified date.
Examples
# >>> date_to_TS(‘19-01-25 16:01:39’, form=’%y-%m-%d %H:%M:%S’) # 1548428499
-
dccd.tools.date_time.
str_to_span
(string)¶ Return the equivalent interval time in seconds.
Parameters: - string : str
Time periodicity
Returns: - span : int
Number of seconds in time interval.
Examples
>>> str_to_span('minutely') 60
-
dccd.tools.date_time.
span_to_str
(span)¶ Return the time periodicity.
Parameters: - span : int
Time interval in second.
Returns: - date : str
Time periodicity.
Examples
>>> span_to_str(3600) 'Hourly'
-
dccd.tools.date_time.
binance_interval
(interval)¶ Return the time interval in the specific format allowed by Binance.
Parameters: - interval : int
Must be in seconds as 60, 180, 300, 900, 1800, 3600, 7200, 14400, 21600, 28800, 43200, 86400, 259200, 604800, 2592000.
Returns: - form : str
Specific format allowed by Binance.
Examples
>>> binance_interval(7200) '2h'
Input/Output tools (dccd.tools.io
)¶
Tools and object to load, append and save differnet kind of database.
-
class
dccd.tools.io.
IODataBase
(path='./', method='csv')¶ Object to save a pd.DataFrame into different kind/format of database.
Parameters: - path : str, optional
Path of the database, default is ‘./’ (current directory).
- method : {‘DataFrame’, ‘SQLite’, ‘CSV’, ‘Excel’, ‘PostgreSQL’, ‘Oracle’, ‘MSSQL’, ‘MySQL’}
Format of database, default is CSV.
Attributes: - path : str
Path of the database.
- method : {‘DataFrame’, ‘SQLite’, ‘CSV’, ‘Excel’, ‘PostgreSQL’, ‘Oracle’, ‘MSSQL’, ‘MySQL’}
Kind/format of the database.
- parser : dict
Values are function to corresponding to method.
Methods
save_as_dataframe
(self, new_data[, name, ext])Append and save new_data as pd.DataFrame binary object. save_as_sql
(self, new_data[, table, name, …])Append and save new_data in SQL database. save_as_sqlite
(self, new_data[, table, …])Append and save new_data in SQLite database. save_as_csv
(self, new_data[, name, ext, …])Append and save new_data in database as CSV format. save_as_excel
(self, new_data[, name, …])Append and save new_data in database as Excel format. __call__
(self, new_data, \*\*kwargs)Append and save new_data in database as method format. -
get_from_dataframe
(self, name, ext='.dat')¶ Get data from pd.DataFrame binary object.
With pickle get as binary pd.DataFrame object.
Parameters: - name : str
Name of the database.
- ext : str, optional
Extension of the database, default is ‘.dat’.
-
get_from_sqlite
(self, name, table='main_table', ext='.db')¶ Get data from SQLite database.
Parameters: - name : str, optional
Name of the database, default is the current year.
- table : str, optional
Name of the table, default is ‘main_table’.
- ext : str, optional
Extension of the database, default is ‘.db’.
- index : bool, optional
Write pd.DataFrame index as a column. Uses index_label as the column name in the table. Default is True.
- index_label : string or sequence, optional
Column label for index column(s). If None is given (default) and index is True, then the index names are used. A sequence should be given if the pd.DataFrame uses pd.MultiIndex.
-
save_as_csv
(self, new_data, name=None, ext='.csv', index=True, index_label=None)¶ Append and save new_data in database as CSV format.
With pickle save as binary pd.DataFrame object, if name database already exists append new_data, otherwise create a new file.
Parameters: - new_data : pd.DataFrame
Data to append to the database.
- name : str, optional
Name of the database, default is the current year.
- ext : str, optional
Extension of the database, default is ‘.csv’.
- index : bool, optional
Write row names (index), default is True.
- index_label : str or sequence, optional
Column label for index column(s) if desired. If not specified (default is None), and index are True, then the index names are used. A sequence should be given if the DataFrame uses MultiIndex. If False do not print fields for index names. Use index_label=False for easier importing in R.
-
save_as_dataframe
(self, new_data, name=None, ext='.dat')¶ Append and save new_data as pd.DataFrame binary object.
With pickle save as binary pd.DataFrame object, if name database already exists, load it, append new_data and save it, else create a new database.
Parameters: - new_data : pd.DataFrame
Data to append to the database.
- name : str, optional
Name of the database, default is the current date.
- ext : str, optional
Extension of the database, default is ‘.dat’.
-
save_as_excel
(self, new_data, name=None, sheet_name='Sheet1', ext='.xlsx', index=True, index_label=None)¶ Append and save new_data in database as Excel format.
With pickle save as binary pd.DataFrame object, if name database already exists append new_data, else create a new file.
Parameters: - new_data : pd.DataFrame
Data to append to the database.
- name : str, optional
Name of the database, default is the current date.
- sheet_name : str, optional
Name of sheet which will contain new_data, default is ‘Sheet1’.
- ext : str, optional
Extension of the database, default is ‘.xlsx’.
- index : bool, optional
Write row names (index), default is True.
- index_label : str or sequence, optional
Column label for index column(s) if desired. If not specified (default is None), and index are True, then the index names are used. A sequence should be given if the DataFrame uses MultiIndex.
Warning
Slow method, not recommanded for large database.
-
save_as_sql
(self, new_data, table='main_table', name=None, ext='', index=True, index_label=None, driver=None, username=None, password=None, host=None, port=None, **kwargs)¶ Append and save new_data in SQL database.
SQL database as method={‘PostgreSQL’, ‘Oracle’, ‘MSSQL’, ‘MySQL’}. If name already exists append new_data, else create a new database. See SQLAlchemy documentation for more details [1].
Parameters: - new_data : pd.DataFrame
Data to append to the database.
- table : str, optional
Name of the table, default is ‘main_table’.
- name : str, optional
Name of the database, default is the current year.
- ext : str, optional
Extension of the database, default is ‘.db’.
- index : bool, optional
Write pd.DataFrame index as a column. Uses index_label as the column name in the table. Default is True.
- index_label : string or sequence, optional
Column label for index column(s). If None is given (default) and index is True, then the index names are used. A sequence should be given if the pd.DataFrame uses pd.MultiIndex.
- driver : {‘psycopg2’, ‘pg8000’, ‘mysqlclient’, pymysql’, ‘cx_oracle’, ‘pyodbc’, ‘pymssql’}
The name of the DBAPI to be used to connect to the database using all lowercase letters. If not specified, a default DBAPI will be imported if available - this default is typically the most widely known driver available for that backend.
- username, password : str
Username and password to connect to the SQL database.
- host : str, optional
Host to connect, default is ‘localhost’.
- port : str, optional
The port number, default is None.
- kwargs : dict, optional
A dictionary of options to be passed to the dialect and/or the DBAPI upon connect.
References
[1] https://docs.sqlalchemy.org/en/13/core/engines.html
-
save_as_sqlite
(self, new_data, table='main_table', name=None, ext='.db', index=True, index_label=None)¶ Append and save new_data in SQLite database.
With sqlite, if name database already exists append new_data, else create a new data base.
Parameters: - new_data : pd.DataFrame
Data to append to the database.
- table : str, optional
Name of the table, default is ‘main_table’.
- name : str, optional
Name of the database, default is the current year.
- ext : str, optional
Extension of the database, default is ‘.db’.
- index : bool, optional
Write pd.DataFrame index as a column. Uses index_label as the column name in the table. Default is True.
- index_label : string or sequence, optional
Column label for index column(s). If None is given (default) and index is True, then the index names are used. A sequence should be given if the pd.DataFrame uses pd.MultiIndex.
-
dccd.tools.io.
get_df
(path, name, ext='')¶ Load a dataframe as binnary file.
Parameters: - path, name, ext : str
Path to the file, name of the file and the extension of the file.
Returns: - pandas.DataFrame
A dataframe, if file not find return an empty dataframe.
-
dccd.tools.io.
save_df
(df, path, name, ext='')¶ Save a dataframe as a binnary file.
Parameters: - df : pandas.DataFrame
A dataframe to save as binnary file.
- path, name, ext : str
Path to the file, name of the file and the extension of the file.
Websocket Tools (dccd.tools.websocket
)¶
Connector objects to WebSockets API client to download data.
-
class
dccd.tools.websocket.
BasisWebSocket
(host, conn={}, subs={})¶ Basis object to connect at a specified stream to websocket client API.
Parameters: - host : str
Adress of host to connect.
- conn : dict
Parameters to connection setting.
- subs : dict
Data to subscribe to a stream.
Attributes: - host : str
Adress of host to connect.
- conn_para : dict
Parameters of websocket connection.
- ws : websockets.client.WebSocketClientProtocol
Connection with the websocket client.
- is_connect : bool
- True if connected.
- False`otherwise.
Methods
on_open
(self, \*\*kwargs)On websocket open. -
on_close
(self)¶ On websocket close print and fire event.
-
on_error
(self, error, *args)¶ On websocket error print and fire event.
-
on_message
(self, message)¶ On websocket display message.
-
on_open
(self, **kwargs)¶ On websocket open.
Parameters: - **kwargs
Any relevant keyword arguments to set connection.
-
wait_that
(self, is_true)¶ Wait before running.