PDA

View Full Version : Import from folder - large .txt files problem



Sugar Nut
07-13-2011, 09:58 AM
Hi,

after my problems with HEM which I have reported in this thread (http://forums.holdemmanager.com/manager-general/14425-all-hands-wont-show-reports.html) I decided to format my hard drive and re-install windows (which I had wanted to do for quite some time and now seemed to be as good a time as any).

I requested my entire playing history from pokerstars and they sent some 30+ text files ranging from a few KB to 90MB in size. The largest of these files has ~80k hands inside of it. On top of the fact that all files have the same name (Hands.txt) and I can't just unzip them into one folder and import (which is obviously not HEM's fault), there seem to be HEM related issues with larger .txt files.

Starting at the ~15MB/file mark HEM seems to have problems importing them.

First of all it takes an unusually long time, and I get lots of messages like:

1 of 9 parts of a large file has been processed
2 of 4 parts of a large file has been processed
etc. (all while importing ONE unique Hands.txt file)

Secondly I get an extraordinarily high number of duplicates. Very frequently HEM would show something like:

Imported hands: 23,876

Duplicates: 589,743

Now after importing a lot of these files all one by one I decided to make a folder and inside of it three subfolders. In each folder I dumped one of the hands.txt files and started importing. After about two hours HEM import crashed.

How can I get HEM to import these hands properly?

Patvs
07-13-2011, 10:44 AM
It's normal large .txt files take a much longer time to import.

1 of 9 parts of a large file has been processed
2 of 4 parts of a large file has been processed

Duplicate issue: are you importing them into a new database?
Are you running a import from folder on the files (good), or did you drop the .txt files into one of your auto import imports (bad)?



Read: (the section on assigning more RAM to PostgreSQL by editing postgresql.config) @
http://forums.holdemmanager.com/manager-general/11194-top-10-hem-performance-increase-tips.html

Sugar Nut
07-13-2011, 11:19 AM
It's kind of a new DB. After re-installing windows (XP btw) I had to create everything from scratch. I used the HEM/PostgreSQL combo installer to install HEM and PostgreSQL (great feature btw as installing PostgreSQL gives me headaches). Then I created a new empty DB and imported ~150k hands from other sites that I had managed to get exported from my old DB before formatting everything. There had been ZERO Stars hands in the DB previous to the issue described in the OP.

Yes, like I said in the title of this thread, I used the "Import from folder" option.

I'll read the link now and see if it helps.

Sugar Nut
07-13-2011, 12:04 PM
PostgreSQL:

-7 Edit your PosgreSQL.conf file in the SQL 8.x/data folder.
Do not use the Tuning Wizard. On some systems it will make changes to the postgresql.conf it shouldn't, resulting in being unable to connect to the database!
If you experience this problem--> simply go to the 8.4/data folder, undo the changes you made and reboot.
Recommended changes:
shared_buffers = <number> MB (set this to 20% of your system memory, but not more than 1000 MB.... round it off to 128 MB, 256 MB, 512 MB or 768 MB)
effective_cache_size = <number> MB (set this to 2x the size of the shared_buffers)

I have 2046MB of RAM (according to dxdiag), so now I have set the values in my postgresql.conf to the following values:

shared_buffers = 256MB (2046 * 0.2 = 409.2 rounded off to 256)
effective_cache_size = 512MB (256 * 2)

Is that correct? I'm not exactly computer illiterate, but I wouldn't consider myself a scholar either. That's why I'm asking.

Patvs
07-13-2011, 10:30 PM
Yes that's fine.
Reboot your computer for the changes to have any effect.

Also consider updating PostgreSQL to 8.4.8 with the installer from PostgreSQL: The world's most advanced open source database (http://www.postgresql.org)

Sugar Nut
07-14-2011, 04:37 AM
I don't know whether the actual changes I made to the postgresql.conf were the reason or not. Fact is everything worked fine.

I created a new DB and imported my Stars hands first. Not a single duplicate this time whereas before it would produce about 3 to 4 times as many duplicates as actual hands imported.

Anyways, thanks for the help.

One last question: How do I delete the first DB I created? In the DB management menu there doesn't seem to be an option to do so.

Patvs
07-14-2011, 05:12 AM
Delete it in pgadminIII: http://faq.holdemmanager.com/questions/75/Create%7B47%7DDeleteDatabase

Sugar Nut
07-14-2011, 07:23 AM
Thanks a lot for your help!