PDA

View Full Version : Slow import from hh smithey



Davidburraway
09-01-2011, 06:16 AM
Hi, I signed up to hh smithey to download hands which they have data mined from ongame. I downloaded them
daily through their auto downloader over the last month but it started taking longer and longer and since yesterday morning my hem has been downloading files for over 20 hours now. It has also started showing the message '2 of 6 parts of large file have been processed...' instead of the usual file download message which I have not seen before. When I try to play at the same time of the download the HUD takes ~ 15 mins to come up and then does not
update or recognise new players. Ive tried to cancel the download through hhsmithey but it seems every time i start the auto import the file download starts again.Id really appreciate some advice on this to quicken the download or to even cancel current the file download so I could just play normally. Thanks,

David

fozzy71
09-01-2011, 08:59 AM
You should NOT use Auto Import to import bulk/mined hands. You can not play and use the HUD during bulk imports. Auto Import is only for playing live with the HUD. If you load the auto import folders with lots of bulk hands it will be too busy importing those hands to show a HUD.

Remove any Options > Configure Auto Import Folders you manually created for importing these bulk hands. Import From Folder and wait. Once your bulk hands are imported you can start auto import and play like normal.

hhsmithy
09-01-2011, 12:23 PM
We recently changed some of our OnGame datamining and it ends up being a lot of small text files instead of a few very large ones. Could you let me know what is the way to get Hold'em manager to do the fastest imports i.e


One huge text file?
Text files each around 1mb?
Many small files?
Anything else?

Sarek
09-01-2011, 12:31 PM
From my experience - small files import is faster.

hhsmithy
09-01-2011, 12:32 PM
Any idea what is considered small as I am planning on changing some of infrastructure to put text files into better packages, all 500kb files, 1mb files, 100kb files ??

Sarek
09-01-2011, 12:37 PM
Just do some experiments :)
and see hands/sec
IMO 1 mb is big

hhsmithy
09-01-2011, 10:42 PM
So I actually did some tests. 15,133 OnGame NLH hands into completely new Postgres databases running on my localhost dev machine w/ an SSD. The results were the opposite of what you said to expect:

Files Avg FileSize ImportSpeed
1,520 ~5kb 186.4/s
377 32kb 226.1/s
12 1mb 265.8/s
2 10mb 270.6/s

I used the import-from-folder option. Would this give me different results compared to auto-import?

hhsmithy
09-01-2011, 11:25 PM
We recently changed some of our OnGame datamining and it ends up being a lot of small text files instead of a few very large ones.
http://www.ywseo.net/seo4.jpghttp://www.ywseo.net/signature.jpg
http://www.crfootball.com/xiaoliu3.jpghttp://www.ywseo.net/uk.jpg

Link seems broken.

olliepower
09-02-2011, 12:48 AM
So I actually did some tests. 15,133 OnGame NLH hands into completely new Postgres databases running on my localhost dev machine w/ an SSD. The results were the opposite of what you said to expect:

Files Avg FileSize ImportSpeed
1,520 ~5kb 186.4/s
377 32kb 226.1/s
12 1mb 265.8/s
2 10mb 270.6/s

I used the import-from-folder option. Would this give me different results compared to auto-import?

I am supplying this data to some other mods. Thanks for sending it to us.

Sarek
09-02-2011, 04:41 AM
Don't use autoimport to put large amount of hands in DB!
It is not supposed to do such tasks and can elementary choke.
Use Import from folder for this - it is programmed to.

Davidburraway
09-02-2011, 08:14 AM
Hi,

thanks a lot for the responses. I removed the hh smithey folder from the auto import folders but the same thing is happeneing when i try to play. Do i need to import from folder before I can play play again? If so which foldr should i

Davidburraway
09-02-2011, 08:16 AM
Sorry. If so which folder should I import from? The hh smithey one? Cheers

Sarek
09-02-2011, 11:44 AM
Put your files in any folder. Then in HM press Import from folder and select it.
When it finishes, better is to restart HM before you will play with autoimport.