PDA

View Full Version : lagging Folder Import?



malfaire
08-11-2008, 01:10 PM
i'm trying to do a mass import across my entire c:\ drive to get all old hands into HEM. sadly, i think this entails about 3mil hands, so that means i'm looking at doing this over the next day.

the last 3 times i've gone to do this, it'll get to about 100k hands and then stall. the green status bar will still be "shimmering", however no new hands are being imported. i've tried to export from PT3 to bucket hands into a more modular import process (e.g. split up into 10 mass imports into folders instead of everything at once), but the damn thing doesnt want to export either!

help! i really want to get up and running with HEM and my full database but seem to be stuck. suggestions?

fabio
08-11-2008, 01:37 PM
Try http://www.rvgsoftware.com/hm/hmupdate1.06.01t.exe

There were a lot of improvements regarding imports in the last couple of releases

malfaire
08-11-2008, 02:17 PM
I'll give it a shot, thanks

malfaire
08-11-2008, 09:05 PM
seems that i still have the issue.

my workaround -- i've split all of these exported files into folds as i got pt3 to mass export all 8mil hand (yes my estimate was way off). seems that i dont get a 'hangup' if i limit each import to ~400 files.

PROBLEM: about 2/3 of the files i'm importing have errors on them. i've left a message with PT3's admin to see what the problem could be (i'm assuming PT3's hh export would be a "universal" text which all db's can import...).

has this happened to anyone before?

fabio
08-12-2008, 06:49 AM
All your hands you want to import are PT3 exported ones? For which site?

malfaire
08-12-2008, 05:08 PM
The majority of them are Pokerstars, which are the only ones which give me errors.

I set another "full import" before I went to bed last night, supposedly 12 hours left. Here's what I find when I wake up. Note the fact that all buttons are "active", which doesn't happen during normal import:

http://www.djdishevel.com/misc/stuck.jpg

another interesting point. i discussed the other method of importing (by way of pt3 export) with PT3's admins. to paraphrase their comments, they said pt3 exports in the same format in which it was imported. however, in notepad the raw txt looks "mashed" together and essentially has no formatting (e.g. no line breaks, etc). BUT, when i upload this info to my website and view as txt there, it looks fine. to see the change, save it to your desktop as a text file and open in notepad. this is how exported pt3 txts are looking to me now, which is, i'd think, incorrect.

http://www.djdishevel.com/misc/PS_HandsExport_200808110833_cash_15859.txt

malfaire
08-12-2008, 05:48 PM
Also, this same "smashed" file output is what I see when I find a hand in my Processed folder fairly often but not all of the time. Which would lead me to believe that though PT3 is capable of importing / exporting this "smashed" file output, HEM is not.

fabio
08-12-2008, 06:04 PM
It's just a formatting issue, the editor cant "read" the line breaks.

malfaire
08-12-2008, 07:00 PM
Okay, then what would be the root cause of the errors?

Rvg72
08-12-2008, 09:21 PM
Okay, then what would be the root cause of the errors?

Malfaire, this particular issue is probably something else. Basically some type of error that is kicking the import out completely. I found a couple of bugs in earlier versions of 1.06.01 that did this but they should have been fixed. Version "u" is coming out tomorrow, could you please try that one since many more importing issues have been fixed since then as well.

Roy

malfaire
08-12-2008, 11:14 PM
/gets on knees
//prays

malfaire
08-13-2008, 07:14 AM
Hm. Well, I did find another issue -- the hands from this month which I imported with the "mass import" method don't have any hole cards (mine simply not shown in database, nor will the actual HH pull when I right click and select View Selected Hand Histories).

However, I did play a few sessions on HEM with Auto-Import and everything seems to be working fine there.

morny
08-13-2008, 12:18 PM
Im unsure from your post whether youve tried version U yet or this new problem is still using T, Either way if there is corrupt files already in there updating will only fix new problems and not corrupt ones previously imported. Youll need to use the Purge Option http://208.109.95.123/faq/?f=152 and reimport them. One thing to note when you purge and reimport its best to put the files in a new folder to force HM to recognise they are not duplicate files as occassionaly happens.

If you reimport and you still have the problems then can you please export some of those HH where the hole card dosent show to morny@holdemmanager.net and ill check them out in my database

malfaire
08-13-2008, 12:49 PM
This is all on version T; I didn't see any new updates / links for version U in the revision forum, forum header or main page. Cool, figured you might mention purge / reimport for the newer version; I'll do this a bit later and get back to you. Thanks again for your help.

morny
08-13-2008, 01:55 PM
Better off waiting for version U before purging and reimporting, should be released pretty soon.

malfaire
08-13-2008, 07:07 PM
Tracked down the U version and installed. Mixed news.

Bad: Went to first folder of already imported hands (this was a PT3 export) -- click to import; 0 files found. Hmm.

Went to another folder of hands which weren't imported yet --

Files Imported: 180
Hands Imported: 27600
Hands/s: 59.7/s
Duplicate Hands: 0
Import Errors: 62000

PT3 batches its exported files 500 hands per document. I did note that either the import file would result in 500 hands imported or 500 hands being listed as Error, no mix of errors between the two. Would it be of any help if I send you one of the folders which is importing with errors?

Good: I decided to try the full scan of C:\ again. It's chugging along (and quickly, too!). No stalls and it's further than it's ever gotten. It's picking up duplicates, but that's to be expected.

Rvg72
08-13-2008, 08:09 PM
Tracked down the U version and installed. Mixed news.

Bad: Went to first folder of already imported hands (this was a PT3 export) -- click to import; 0 files found. Hmm.

Went to another folder of hands which weren't imported yet --

Files Imported: 180
Hands Imported: 27600
Hands/s: 59.7/s
Duplicate Hands: 0
Import Errors: 62000

PT3 batches its exported files 500 hands per document. I did note that either the import file would result in 500 hands imported or 500 hands being listed as Error, no mix of errors between the two. Would it be of any help if I send you one of the folders which is importing with errors?

Good: I decided to try the full scan of C:\ again. It's chugging along (and quickly, too!). No stalls and it's further than it's ever gotten. It's picking up duplicates, but that's to be expected.

1) You need to rename the folder to trick HM to read already processed Files
2) Can you send me some of the files that register all errors along with 1 that registers no errors
3) Are you really scanning the "c:\" + subfolders??? That is really bad! Stop the import please! It will move all sorts of files and scan ever single text file on your computer.

Roy

malfaire
08-13-2008, 08:20 PM
1) Okay, I'll give that a shot
2) Yup.
3) Oops.