Import disaster

[1geoff99]

HM1 is intelligent enough to know which files were previously imported and does not attempt to re-import previously imported files. For HM2, at every import, it always tries to import every single file in the specified folder again, regardless of whether or not files have already been imported. And if you interrupt an import, it always starts from the beginning again.

I choose to keep a large import folder and don't do archiving. With HM1, it always takes only a few seconds to find new hands in my import folder. With HM2, it takes about 7 minutes!

I've only tried HM2 for a few minutes so far, but this is already a showstopper for me when it comes possible upgrading. Even in HM2, it is a valid option to decide not to do archiving. If that that option is there, then you should not be penalised with useless performance.

[Patvs]

Move the hands to the archive folder and use auto-archiving.

[1geoff99]

Move the hands to the archive folder and use auto-archiving.

I realise I can do that, but I prefer the way HM1 works.

I don't understand the reason for the difference in behaviour.

[fozzy71]

...
I choose to keep a large import folder and don't do archiving. ....

Why exactly do you do this? Why not let HM1/2 import from the client folder and archive/move them to the folder you want them in? That would solve all of your problems and would really accomplish the same thing from what I understand of the problem.

[Azalin]

HM1 is intelligent enough to know which files were previously imported and does not attempt to re-import previously imported files. For HM2, at every import, it always tries to import every single file in the specified folder again, regardless of whether or not files have already been imported. And if you interrupt an import, it always starts from the beginning again.

I choose to keep a large import folder and don't do archiving. With HM1, it always takes only a few seconds to find new hands in my import folder. With HM2, it takes about 7 minutes!

I've only tried HM2 for a few minutes so far, but this is already a showstopper for me when it comes possible upgrading. Even in HM2, it is a valid option to decide not to do archiving. If that that option is there, then you should not be penalised with useless performance.

Oh boy.. I was importing hands from a big folder (13million hands) and I needed to restart my computer so I stopped the import after 3-4 hours.. what happened next? Well when I restarted HEM2 it need to go over all hands and thats not all. Originally I was getting import speeds up to 1500hands/sec but after stopping the import and restarting HEM2 I was getting like 400-600hands/sec cause it was checking for duplicates (which there were a lot). In any way I calculated that it would be much faster to delete my entire database and start from the beginning cause simply even though I had like 10 million hands imported the 3million hands would take a LOT more time cause the 400-600h/sec speed was for checking ALL hands from the beginning..

To be fair, the import speeds of HEM2 are excellent and way way ahead of PT3/4.. Maximum 5 hours for 13mil hands is a dream come true and to tell the truth I was not bothered much. Of course notecaddy took like 30 hours so I was left with a bitter taste :/

[Patvs]

Normally duplicate hands import a lot faster than new hands.
Once it's detected as being duplicate--> it's skipped. (skipping should be faster than regular import of new hands)

Your super-computer setup could have reached a magic threshold of the max speed for detecting and skipping duplicate hands.
Where new hands actually import faster.

The issue of importing 13 million hands--> where you want it to continue importing where it left off, will be fixed eventually.
As a temporary solution, break up the folder into 4 folders of 3 million hands each.

[Azalin]

Normally duplicate hands import a lot faster than new hands.
Once it's detected as being duplicate--> it's skipped. (skipping should be faster than regular import of new hands)

Your super-computer setup could have reached a magic threshold of the max speed for detecting and skipping duplicate hands.
Where new hands actually import faster.

The issue of importing 13 million hands--> where you want it to continue importing where it left off, will be fixed eventually.
As a temporary solution, break up the folder into 4 folders of 3 million hands each.

Thanks for the reply Patvs.

It also looked strange to me that skipping hands was slower than importing the new ones but as I said before thats no biggie. Continue giving me 1000+ h/sec import speed in a 10mil+ database instead of a "remember where it left off" HEM1/PT3/PT4 speeds and I'll take that all day...

[Patvs]

Out of curiosity: what are you average import speeds, importing 1+ million hands into the latest PT4 beta?

[Azalin]

Just reinstalled PT4 and started importing in a new database all 13 mil in order to check. I get 206 h/sec but it seems to be increasing slowly but still.. 206

edit: up to 230h/s

[mphh]

Oh boy.. I was importing hands from a big folder (13million hands) and I needed to restart my computer so I stopped the import after 3-4 hours.. what happened next? Well when I restarted HEM2 it need to go over all hands and thats not all. Originally I was getting import speeds up to 1500hands/sec but after stopping the import and restarting HEM2 I was getting like 400-600hands/sec cause it was checking for duplicates (which there were a lot). In any way I calculated that it would be much faster to delete my entire database and start from the beginning cause simply even though I had like 10 million hands imported the 3million hands would take a LOT more time cause the 400-600h/sec speed was for checking ALL hands from the beginning..

To be fair, the import speeds of HEM2 are excellent and way way ahead of PT3/4.. Maximum 5 hours for 13mil hands is a dream come true and to tell the truth I was not bothered much. Of course notecaddy took like 30 hours so I was left with a bitter taste :/

Would You mind sharing Your computer spec.?

Im thinking about building a new one just for poker and Your performance sounds awesome.