HEM Import

[Josey Wales]

I have huge issues with the HEM import process. That code is so sloppily written it boggles my mind. For years now I've been forced to import large quantities of hands trough no fault of my own and every time it's been a horror experience.

My last adventure began when I (stupidly) decided to update windows, thus rendering HEM unusable. So I got the latest HEM update and ran it. After it finished, a box popped up saying HEM needs to make some alterations to the database. OK, fine, go ahead. Them alterations were never made. Every time I've tried to open HEM, the alteration process began and was never finished due to some error.

Ran pgAdmin, vacuumed, verified the integrity of the DB, there were no problems. So the only thing I could do was export all hands, reinstall HEM and postgres and re-import them. Knowing full well I'm in for a hell of a ride, I hit up a friend of mine who built me a desktop for this very task. The specs are

- Intel Quad Core @ 4 GHz with water cooling
- 8 GB DDR3 RAM @ 1.3 GHz
- 2x Intel SSD drives in RAID 0
- etc
- Win7 x64, tuned postgres, no swap file, temp file in RAM

Now this rig is easily three times faster than my notebook that runs my poker setup just fine, so I expected it to handle the task at hand in a reasonable amount of time.

Yeah, right.

All four processor cores idle at around 0 to 3 %. RAM is hardly used and the HD activity light flashes rarely. WTF is HEM doing?

So I thought maybe reading .txt files from the SSD array and writing to the database to the same array at the same time slows things down. OK, I set up a RAM drive where I copied the hands. So now HEM reads .txt files from a blazing fast RAM drive, processes them with the fastest CPU available and writes them to the fastest storage option available. And it still runs as fast (lol, "fast"!) as my three year old laptop? Like, where's the bottleneck here?

The progress indicator needs some serious work imo. Sometimes it just sits there with no indication if it's doing anything (see pic). The estimated time remaining thing is a joke, please get rid of it. I have "7 minutes remaining" after the first file and the same "7 minutes" with two files remaining after 75 minutes.

20111

I am now in the 8th day of losing time and money. I know there is nothing I can do at this point but pretty please, with sugar on top, design a new, multithreaded import module with detailed notifications for HEM2 from scratch. I'd gladly pay for it just for that feature alone.

[Patvs]

HM2 has fixed multithreaded import. And will use up ALL your RAM, and fully utilize the SSD. (I've seen 600 hands/s import speeds)

On your computer, the import speeds shouldn't be so low. What type of games are you playing? Are you mostly playing satellite STTs?

Read:
http://forums.holdemmanager.com/manager-general/11194-top-10-hem-performance-increase-tips.html

(if you also take a look at the last page of that thread, you'll also see other power computer + SSD users will manage to get 300 hands/s import speeds in HM1)

[Josey Wales]

I'm glad that's sorted. Unfortunately HEM2 crashes all the time on all the PC's I've tried, so I'll have to wait until it becomes more stable.

Games are MTT SnG's and MTT's, no cash hands at all. Average .txt file size is around 4 MB and there are about 900 of them. I'm importing them in batches of 30-50 at a time.

[Patvs]

Such low import speeds are normal if the .txt files are >1MB.

I'm not sure how HM2 handles >4MB files. I'll test that later today.

But for HM1, ideally you should only import small files <300kb, which will already happen during a live import playing a session. HM1 is a disaster if you want to import an exported database (with many 20 MB files), or ALL-YOUR-requested hands from PokerStars (can be one big 150 MB file).

If you purchase handhistories (from PTR) they're usually in files of 1 MB which should import a lot faster.