PDA

View Full Version : are observed hands placed in HMArchive too?



Daniel
01-10-2011, 03:40 PM
I've started a new database and am now reimporting my hands from HMarchive and wondering if observed hands are saved there too.

If not, is there a way to export these hands? Or do I have to export all hands (including a duplicate 500k hands from HMArchive) and reimported them all? I actually started with exporting all hands, but the window freezed.

Patvs
01-11-2011, 01:49 AM
HM doesn't datamine the observed hands, unless you have very old (<2008) HM hands..... or you just want stats for the pre-fetched hands.

To be able to reimport those hands, it's best to EXPORT all your hands--> reimport them.

HM will skip all the duplicate hands, so it should import 500k hands really fast.

Daniel
01-11-2011, 07:28 PM
I am importing all the hands I've exported. In the import screen it says x of y parts of large files have been processed. When a file was processed, the number of players didn't increase. Everything going OK here?

Patvs
01-11-2011, 08:15 PM
After such a (large) file, does it state it imported x out of x hands, or imported duplicate hands?

You already imported your old archive folder right? So 99.9% of the hands from your exported database will be duplicate hands.

Daniel
01-12-2011, 03:32 AM
"After such a (large) file, does it state it imported x out of x hands, or imported duplicate hands?"

The first import session it were new hands, later on some duplicate. But it's going really fast. 75 large files are processed within half an hour.

"You already imported your old archive folder right? So 99.9% of the hands from your exported database will be duplicate hands. "

Well, to import datamined hands I had to export all hands anyway, so I didn't finish importing all hands from HMArchive. These datamined hands are quite large in volume. 2 months of all hands played on Ipoker and Stars for .5/1 and 1/2 NL.

Patvs
01-12-2011, 07:45 PM
It's normal it imports large files as part y out of x.
When it was finished with ALL files (exported database + archive)
does the new database appear to have all your hands?

Daniel
01-13-2011, 04:47 AM
I stopped the import, because it was taking too long. The number of players in my DB is up to 100K now and it was 200K. So it seems to go OK. Thanx.

Daniel
01-14-2011, 04:32 AM
Just started a bulk import last night. I selected 8 files of approx 150MB, earlier exported with HM. This morning, 8 hours later it was still busy. I don't know how many hands were imported, because these files are split (of y parts of large files...). This morning, only a few thousand players were added to my DB. I'm pretty sure these hands are not imported yet, because it takes long (more than an hour per large file).

My database consisted originally of 210000 players. Exporting all hands resulted in about 400 files, of which 60 are large (>100MB).

Importing in this pace would take more than 60 hours for the large files alone it seems.

Still going strong, or reasons for concern?

netsrak
01-14-2011, 04:45 AM
Does it display the import speed?
What type of computer do you use?
Is the harddisk very full?

Daniel
01-14-2011, 06:29 AM
I think it does display import speeds, but only after one large file. Since I selected more than 1 file, and I am not watching te process, I'm not sure if it does and what the import speed is (I'm now at work elsewhere)

Computer is a desktop, 2.3 GHz double core, 2 GB RAM, Gforce 9200, Vista and postgres 8.3.

The harddisk partition is 150GB, with something like 20 GB free space. I still have to remove the old database btw.

netsrak
01-14-2011, 10:38 AM
Hard to say - the import becomes slower the more hands you have in your database.
Sometimes a vacuum/full and reindex will speed it up again.

What happens if you try to import only 1 file? How long does it take and how many hands?

Daniel
01-14-2011, 10:45 AM
1 large file was about 95 parts and was about 100000 duplicate hands, finished in 10-15 minutes. Don't know how long it would take if these were new hands.

netsrak
01-15-2011, 06:46 AM
much longer...

Daniel
01-16-2011, 04:51 PM
Yep, much longer indeed! I'm still busy here (computer must have been running 40 hours or so in 5 sessions), and I only used datamined hands for 2 months, including dubplicate hands.

For the future, I hope it's possible to purge some hands to decrease import time. Because of my experiences, this is not something I want to do again. I was searching for purging hands completely (dataminded hands) and found a lot of topics. It seems that the stats won't go away because there is no cache rebuilder. The advice was to purge hands and start a new DB. Is it possible to completely remove (datamined) hands including stats? If not, this is definetely a thing to work on to keep a DB manageable. What is the status on the cache rebuilder?