Python: TestGenerator: unmatched data error (low value X at Y is not reached from the least timeframe, low price Z mismatches)

[kenorb]

After fixing #85, this error start happening at M30 timeframe when backtesting:

TestGenerator: unmatched data error (low value 1.08294 at 2015.07.20 09:15 is not reached from the least timeframe, low price 1.08391 mismatches)

Similar error at TestPeriod M5 (plus volume error, similar as per: #70):

TestGenerator: unmatched data error (low value 1.08391 at 2015.07.20 09:15 is not reached from the least timeframe, low price 1.08474 mismatches)
TestGenerator: unmatched data error (volume limit 590 at 2015.07.20 09:30 exceeded)

And M15:

TestGenerator: unmatched data error (low value 1.08294 at 2015.07.20 09:15 is not reached from the least timeframe, low price 1.08391 mismatches)

---

This can be tested using Docker container, e.g.

docker run ea31337/ea-tester run_backtest -v -t -T M30 -e MACD -y 2015 -m 7 -D5 -b DS -M 4.0.0.1010

• Try testing also with M5 and M15 (-T). And all months: -m 1-12. Remove -b and -M params to avoid re-downloading the data when running it again.
• You can login to Docker container by: docker run -it ea31337/ea-tester bash.
• Add -x for debug.

After fix in convert_csv_to_mt.py, clone and convert the CSV data e.g. via (see: Makefile as example):

find . -name '*.csv' -print0 | sort -z | xargs -r0 cat | tee all.csv > /dev/null
./convert_csv_to_mt.py -v -i all.csv -f hst4 -t M1,M5,M15,M30,H1,H4,D1,W1,MN

and move generated files and replace with the existing HST files in ~/.wine/drive_c/Program Files/MetaTrader 4/history/default and re-run the test.

To read the converted HST files, you can use convert_mt_to_csv.py tool, e.g.

./convert_mt_to_csv.py -i EURUSD15.hst -f hst4

---

The problem is with low value mismatch as per error. The fix shouldn't break the fixed logic in dbb096e, which wasn't working before as per #85. The solution should include the proper calculation of OLHCV values (open/low/high/close/volume) across the different timeframes to avoid any unmatched data errors.

See also:

• Generated FXT/HST files.
• Failing CI tests.

Requirements

Basically these commands needs to work with no data error after complete testing:

• docker run ea31337/ea-tester run_backtest -v -t -M4.0.0.1010 -d 1000 -p EURUSD -m 1-12 -y 2015 -s 10 -b DS -D5 -e TestFXTHeader (see: Build #1087).
• docker run ea31337/ea-tester run_backtest -T M1 -v -t -e MA -y 2015 -m 7 (this works)
• docker run ea31337/ea-tester run_backtest -T M5 -v -t -e MA -y 2015 -m 7
• docker run ea31337/ea-tester run_backtest -T M15 -v -t -e MA -y 2015 -m 7
• docker run ea31337/ea-tester run_backtest -T M30 -v -t -e MA -y 2015 -m 7
• docker run ea31337/ea-tester run_backtest -T M30 -v -t -e MA -y 2015 -m 1-12 (test all months just in case)

Resources

• THE ALGORITHM OF TICKS' GENERATION

--

Est. 10h

[paulohrpinheiro]

I'm very sorry, and I want to apologize, but I'm not able to create the environment to run the tests. I would take a long time to get set up, and a whole day is gone, so I'm giving up the task. Once again, I'm sorry, thank you.

[kenorb]

I'm sorry about it. If you're still interested, I can give you the access to the remote Linux where the issue can be easily reproduced. Or this can be tested using Travis CI.