History database

All posts which do not fit into any other sections of the forum.

Moderator: admin

History database

Postby Bebbspoke » Fri Sep 16, 2016 12:42 pm

Hi Folks; -

I find it most frustrating that the FXCM Trading Station does not give access to adequate historic data.

Sure - you can download from Marketscope - but that is only available at a maximum of 15k rows which is certainly insufficient for long term analysis... and it comes in a much oversized .xml format; - why not straight to .xlsx or .csv?

A few years back, the member Gidien posted a privately written downloader aka Publisher HISTORY which covered the requirement as and when the old Order2Go API was in use - I doubt I need comment on the reliability of that API and the Publisher HISTORY app has long since been unmanaged... well; - why should a private investor maintain his goodwill service?

If FXCM expects traders to risk their equity on the Forex Markets then surely FXCM should have the courtesy of allowing traders to easily observe and process accurate history.

I am well aware that "past performance cannot be relied upon to predict future events"... but

PLEASE, PLEASE, PLEASE will someone at FXCM have the consideration to rewrite the Publisher HISTORY app to function reliably with the FXConnect API... Thank you.

Sincerely, Bebbspoke
User avatar
Bebbspoke
FXCodeBase: Confirmed User
 
Posts: 170
Joined: Tue Aug 16, 2011 4:04 am
Location: Walton on the Naze UK

Re: History database

Postby Alexey.Pechurin » Mon Sep 19, 2016 1:26 am

Hello Bebbspoke,

FXCM provides special API for loading and managing price data - Quotes Manager and Price History API. Look here for more details - http://fxcodebase.com/wiki/index.php/Ca ... istory_API

This API has command line GetHistPrice sample that downloads price data for any time period that FXCM provides. It's very easy to modify it to any simple text format for example csv. If you wish we can modify this sample and build it for you. This sample also requires ForexConnect.

Also note that the main goal of this API is to provide price data for FXTS and 3rd party programs. It provides functions to load historical data. Also it caches it locally which improves performance in further calls of these data. FXTS uses this API to load and manage historical prices.

It's a recommended way to quickly load FXCM prices.

Alexey
Alexey.Pechurin
 
Posts: 98
Joined: Sun Jul 22, 2012 11:26 pm

Re: History database

Postby Alexey.Pechurin » Wed Sep 21, 2016 6:17 am

GetHistPricesCSV utility is a modified version of the standard QuotesManager API sample. It loads prices of any timeframe for any time period and saves them to a CSV file which can be opened in MS Excel. It's a Windows command line utility.

How to run GetHistPricesCSV utility:
1) Extract GetHistPricesCSV.zip archive to some folder. It contains GetHistPricesCSV folder inside.
2) Start command prompt. In Windows 7 you can type cmd is Start menu and run the found program.
3) Change the current drive if required. For example, if the current drive is C:\ and GetHistPricesCSV was extracted to D:\ then type D:
3) Move to a folder where the archive was extracted to: type cd <folderPath>, where <folderPath> is a full path to GetHistPricesCSV folder.
4) Run GetHistPricesCSV utility with required arguments.

GetHistPricesCSV utility takes the following arguments:

/login | --login | /l | -l
Your user name.

/password | --password | /p | -p
Your password.

/url | --url | /u | -u
The server URL. For example, http://www.fxcorporate.com/Hosts.jsp.

/connection | --connection | /c | -c
The connection name. For example, "Demo" or "Real".

/sessionid | --sessionid
The database name. This argument is optional.
Required only for users who have accounts in more than one database.

/pin | --pin
Your pin code. This argument is optional. Required only for users who have a pin.

/instrument | --instrument | /i | -i
The instrument that you want to use in the sample. For example, "EUR/USD".

/timeframe | --timeframe
The time period that forms a single candle. For example, m1 - for 1 minute, H1 - for 1 hour.
Custom timeframes (e.g. m2) are also supported.

/datefrom | --datefrom
The date and time starting from which you want to receive historical prices. This argument is optional. If you leave this argument as is, a default value will be used. The format is "m.d.Y H:M:S". The time is in UTC timezone.

/dateto | --dateto
The date and time until which you want to receive historical prices. This argument is optional.
If you leave this argument as is, the data will be received up to now. The format is "m.d.Y H:M:S".
The time is in UTC timezone.

/count | --count
The number of historical prices you want to receive. This argument is optional.
If you leave this argument as is, a default value will be used or the argument will be ignored if 'datefrom' is specified.

/output | --output
The output file name.

Examples:

GetHistPricesCSV /login {LOGIN} /password {PASSWORD} /connection Demo /instrument EUR/USD /timeframe m1 /url http://www.fxcorporate.com/Hosts.jsp /datefrom "06.27.2014 20:50:00" /dateto "06.29.2014 22:05:00" /output result.csv
Loads and saves to result.csv file m1 EUR/USD bars in a specified dates range.

GetHistPricesCSV /login {LOGIN} /password {PASSWORD} /connection Demo /instrument EUR/USD /timeframe H1 /url http://www.fxcorporate.com/Hosts.jsp /count 1000 /dateto "06.29.2014 22:05:00" /output result.csv
Loads and saves to result.csv file 1000 H1 EUR/USD bars up to a specified time.

GetHistPricesCSV /login {LOGIN} /password {PASSWORD} /connection Demo /instrument EUR/USD /timeframe m2 /url http://www.fxcorporate.com/Hosts.jsp /datefrom "06.27.2016 20:50:00" /output result.csv
Loads and saves to result.csv file m2 EUR/USD bars starting from a specified time up to now.

Additional notes:
1) Argument date/times and date/times in a resulting CSV file are in UTC timezone.
2) Prices are received using option Weekend data = Show. So a resulting CSV file contains Weekend bars.
3) Prices are received using option Price Gaps = Hide. So Open = Close of the previous bar.
4) It's not recommended to use it with ticks timeframe - QuotesManager does not support caching of ticks and receiving operation may take a very long time.

To built the utility from the sources you need to install the last versions of ForexConnect and PriceHistory APIs and to extract sources to cpp samples folder of PriceHistory API.
Attachments
GetHistPricesCSV.zip
The archive with built executable.
(1.86 MiB) Downloaded 574 times
GetHistPricesCSV_source.zip
The archive with sources.
(78.9 KiB) Downloaded 572 times
Alexey.Pechurin
 
Posts: 98
Joined: Sun Jul 22, 2012 11:26 pm

Re: History database

Postby Alexey.Pechurin » Thu Sep 22, 2016 5:28 am

To simplify running you can use the attached script. Just extract RunGetHistPricesCSV.bat from the attached archive to GetHistPricesCSV folder, edit it using any text editor - set your login/password and required DATE_FROM, DATE_TO and COUNT parameters - and run this script from Windows Explorer.
Attachments
RunGetHistPricesCSV.bat.zip
(638 Bytes) Downloaded 534 times
Alexey.Pechurin
 
Posts: 98
Joined: Sun Jul 22, 2012 11:26 pm

Re: History database

Postby Bebbspoke » Thu Sep 22, 2016 8:21 am

Alexey,

Thank you for your message and instruction for the RGHPCSV; -

I downloaded and extracted it to C:\Program Files\Candleworks\GHPCSV folder

Within that folder is now the GHPCSV App, RGHPCSV Win Batch File, a folder "RGHPCSV.bat~", the RGHPCSV Win ZIP & several (x15) .dll App Exts.

Clicking on any relevant file procures only a momentary flash of the Command Prompt - I still have NO access to enable the download of Historic data.

It is problematic enough to process historic data to achieve one's aims... I should not be nigh impossible to acquire the data to start with.

You imply that a user friendly GUI should be available next week..
Please let's have the option of download in .xlsx format; - .xlm is far too bulky (> x10 larger than necessary) and .csv requires additional post processing.

I am currently in communications with Nikolay regarding the private build of a tradebot... should the build proceed; - it better function considerably better than the present farcical GetHistPrices system...

It would appear that FXCM have exceedingly little consideration for the traders who are effectively FXCM's source of income!

Sincerely,
Bebbspoke
User avatar
Bebbspoke
FXCodeBase: Confirmed User
 
Posts: 170
Joined: Tue Aug 16, 2011 4:04 am
Location: Walton on the Naze UK

Re: History database

Postby Alexey.Pechurin » Wed Sep 28, 2016 7:56 am

Hello,

In attach you can find the GUI utility that allows to download historical prices and save them to CSV file. There are an installation package and an archive with sources. After installation the utility may be started from Start menu (GetHistPricesGUI->Get History Prices). Note that the utility requires .NET Framework 3.5. If it's not installed on your computer you can download and install it manually using one of the following links:
https://www.microsoft.com/en-us/downloa ... x?id=25150
https://www.microsoft.com/en-us/downloa ... aspx?id=21

How to use the utility:
1) Enter your login and password and click Login.
2) After login completed enter required symbol, period and dates range or count (any combination is supported).
3) Click Get History and enter the output filename.
4) Get History may be clicked several times with different parameters in the same session.

Input dates and output dates are in UTC timezone.
Attachments
GetHistPricesGUI-1.0.0-win32.exe
The installation package.
(1.53 MiB) Downloaded 612 times
GetHistPricesGUI_source.zip
The archive with sources.
(69.1 KiB) Downloaded 597 times
Alexey.Pechurin
 
Posts: 98
Joined: Sun Jul 22, 2012 11:26 pm

Re: History database

Postby Bebbspoke » Thu Sep 29, 2016 2:05 am

Hi Alexey; -

Seems to be a bug in the GUI: -

All examples Start at 01/Aug/2015 18:07 & End at 28/Sep/2016 18:07

On some Instruments a "Warning widow" appears: -

Get History Prices
HTTP receive response failed
object=’/pricearchive/AUDJPY_Default/2015/45.7z’ errorCode= 12030

HTTP receive response failed
object=’/pricearchive/AUDUSD_Default/2016/16.7z’ errorCode= 12030

HTTP receive response failed
object=’/pricearchive/CADJPY_Default/2016/12.7z’ errorCode= 12030

HTTP receive response failed
object=’/pricearchive/NZDUSD_Default/2016/17.7z’ errorCode= 12030

also fails with some Indices

HTTP receive response failed
object=’/pricearchive/JPN225_Default/2015/50.7z’ errorCode= 12030

HTTP receive response failed
object=’/pricearchive/NAS100_Default/2016/3.7z’ errorCode= 12030

Strangely - on closing the Warning window and re-clicking the "Get History" button...
the relevant data is downloaded in a few seconds - it would seem that a cache is being loaded but for some reason it is NOT being released on the first attempt.

Cheers, Bebbspoke
User avatar
Bebbspoke
FXCodeBase: Confirmed User
 
Posts: 170
Joined: Tue Aug 16, 2011 4:04 am
Location: Walton on the Naze UK

Re: History database

Postby Alexey.Pechurin » Fri Sep 30, 2016 2:20 am

Hello,

12030 is a code of a Connection error ("The connection with the server has been reset or terminated"). Probably there were some problems on a server side, or some local problems with your Internet connection. It seems that the problems were temporary because you could download these prices later.

Alexey
Alexey.Pechurin
 
Posts: 98
Joined: Sun Jul 22, 2012 11:26 pm

Re: History database

Postby Bebbspoke » Fri Sep 30, 2016 5:58 am

OK, thanks
User avatar
Bebbspoke
FXCodeBase: Confirmed User
 
Posts: 170
Joined: Tue Aug 16, 2011 4:04 am
Location: Walton on the Naze UK

Re: History database

Postby Alexey.Pechurin » Tue Oct 25, 2016 6:31 am

Update:

If you receive "Wrong user name or password" error with correct credentials then use this version of GetHistPricesGUI utility. It contains a workaround of this problem.

Alexey
Attachments
GetHistPricesGUI-1.0.0-win32_WithLoginErrorFix.exe
(1.53 MiB) Downloaded 582 times
Alexey.Pechurin
 
Posts: 98
Joined: Sun Jul 22, 2012 11:26 pm

Next

Return to General Discussions

Who is online

Users browsing this forum: Google [Bot] and 3 guests