PDA

View Full Version : Appropriate rate: Import From Folder



LowWater
06-07-2009, 04:05 PM
Following a hardware failure during a previous import which irrevocably corrupted my database, I'm re-importing all hands into a new default database using HEM's Import/Import From Folder function. The files are processing without error and the Import Progress rate is reporting 22 Hands/second. Is this an appropriate rate for my XP SP3 system described below? It feels sluggish.

With no other applications running, I have these relevant processes using memory at the following approximate rates:

HMImport.exe 70,000 K
HoldemManager.exe 56,000 K
postgres.exe 44,000 K
postgres.exe 40,000 K
postgres.exe 16,000 K

There are 5 additional iterations of postgres.exe using under 10,000 K/process. Is this to be expected while importing from a folder?

Thanks

HARDWARE
CoolerMaster WaveMaster TAC-T01 Mid-Tower, Intel D975XBX2KR Bad Axe 2 Mobo, Intel® Core™2 Quad 2.4GHz Q6600 CPU, Corsair 4GB [2(2 x 1GB kit)] 5300 DDR2 SDRAM, eVGA e-GeForce 8800GT AKIMBO 512MB SC, Hauppauge HVR-1600, PreSonus Firebox Firewire audio interface/sound card, 4 Seagate 320GB 7200.10 16MB cache HDDs. Seagate FreeAgent XTreme 1TB External Backup.

morny
06-07-2009, 06:14 PM
This can vary alot depending on sites, whther its HU or FR etc try defragging using www.defraggler.com but go to Start > All Programs > PostgreSQL > Stop Service before doing it and then to Start > All Programs > PostgreSQL > Start Service to restart it when finished, that should help increase the speed.

LowWater
06-07-2009, 08:14 PM
Sorry, I'm using PostgreSQL 8.3 and HEM 1.08.04 and I'm only talking about hand imports from folder. I don't have a client site open and this has nothing to do with "Auto-Import". I apologize if I was unclear before.

As mentioned, I had to establish a second "default" database as the first was corrupted and I couldn't figure out how to dump it or uninstall the db in its entirety. I'm reinstalling over a million hand histories di novo.

The other question regarding the load on my system: it's really dragging it down. For a quad-core w/ 4GB RAM on a Bad Axe 2 as clean as my mama's kitchen floor w/ < 0.1% fragmentation, it is a bit unnerving. It's caching pretty hard. I can barely multi-task. Is it normal to have all those processes running while "Importing From Folder" as I mentioned above? This thing's chomping a lot of RAM. I just want to alleviate my suspicion that the db is still messed up and/or the front end software (HEM) is possibly messed up.

Oh, the recuva product - do you suggest using it to defrag the db specifically and exclusively?

The Minder
06-08-2009, 10:25 AM
I have 2 computers. A laptop with a 2.8 P4, 512Mb mem and Windoze XP. Bulk imports on this system typically run about 55ish hands per second. However, occassionally, when playing tournies/games, the system freezes for a couple of seconds... very unnerving.

My second puter is a simple 2.8 core 2 duo, 4Gb with an old 5600rpm 80Gb IDE HD. This is setup with Linux as the primary OS and has postgress installed. I also have Win XP shoved into a Vitualbox guest and that's where my poker clients and HEM sit. When HEM triggers the data feed lines the request is pumped out an ODBC link to the Linux side where postgress does it's thing. On this system I get about 70-75ish HH per sec and HEM never freezes the Windoze guest.

So, I'm guessing you've got some issues with your puter that may not be attributable to HEM and/or postgress as I feel with your hardware setup you should be getting 70+ HH per sec... imo.

morny
06-08-2009, 11:49 AM
One other things that can slow it down are antiviruses, sometimes they run a realtime scan on files being imported and they start trying to access those files your importing which can bog things down. Disabling them wont work so try completely uninstalling it and seeing if it makes a difference.

For the defraggler you can run a system scan or if your short on time maybe just the database files

Its normal for PostgreSQL to have alot of processes running in the taskbar

LowWater
06-10-2009, 05:39 AM
The Minder and Morny, thank you so much for your replies. Any other rates from our community "Importing From Folder" that I could use to compare against my system rates?

LowWater
06-10-2009, 06:56 AM
The Minder, what would you think of a simple dual boot configuration with the same XP SP3 OS but sans the connectivity and subsequent security utilities. What about slimming down Windows ala BlackViper (http://www.blackviper.com/WinXP/servicecfg.htm) in this specific boot?

Everyone, my bigger question is this: In a normal online desktop config, defragged and clean w/ XP's firewall, Prevx3 and NOD32 protecting it in real-time, do my "import from folder" rates approximating 25h/s approaching the norm?