PDA

View Full Version : Holdem manager won't import my 5,000,000 hand histories



ptartaglio
09-01-2009, 05:02 PM
I have 5,000,000 hand histories that I need to import into my database. Whenever I begin the import process into holdem manager it says it will take 1 day, hour. As it continues to import it gets slower and slower until it just isn’t importing nothing. A day has passed both times I tried and I have got 10% the way through.

How can I import these hand histories and how can I do so in a reasonable amount of time. There is 5,000 files in the folder I am importing and importing 1 by 1 is not really an opton.

Thanks,
Paul

Patvs
09-01-2009, 06:31 PM
Only 5000 files, so the average file size is really big? Like 10 MB?

(importing all 5000 x 10 MB files at once won't work)
You have three options:
-1 Create 10 x 10 MB files folders and import one folder at a time. (Holdem Manager should create some sort of batch import for such a task)
-2 Cut all the 10 MB files into 20 x 512Kb files (I'm not sure if there is FREE software that can to this to .txt files, because it's too much work to do this manually) And import 500 x 512Kb at a time.
-3 Have someone who has a "super computer" do the import for you and have that person sent you the compressed backup.

All options are very time consuming. What is your current hardware setup: CPU, RAM, HARDDISK?

I did a import test two weeks ago (I have Intel i7 920, 6GB DDR3-1600, 2x OCZ Vertex in RAID0) Results:

Bulk import speed test:
Test 1: 3x 40 MB txt files (total 84.000 hands). (the 3 largest files from a once "export hands" database, each 40 MB text file contains 28.000 hands)
This took 14 minutes to import, 90 hands /s. (it took about 30 seconds of analysing the files before the actual importing began)

Test 2: 1000 handhistory files in 25 subfolders (total 120 MB, each file ranging from 1Kb-500Kb: average filesize 120Kb) (50.000 total number hands)
This took 18 minutes (half of the time spend on the last 250 files!), 45 hands /s (it was importing with 70 hands /s for the first 750 files, but after that the importing would often freeze for a minute and the "files imported 750 of 1000" would suddenly JUMP to "770 of 1000" instead of to 751. After each "freeze" the hands xx / s would drop.