high performance HEM

[almaplayera]

i want to be able to import 5 million HH in a couple hours, as opposed to a couple days / weeks / months.

i have HEM running on a windows server 2008 box with 4GB RAM and 15Krpm SA-SCSI SAN storage, and importing large #s of hands is just taking foreverrrrrrrr.

do i need a supercomputer to be able to work productively with tens of millions of HH in HEM / postgresql? what are minimum system requirements for my purposes?

[fozzy71]

What kind of speeds are you getting currently? What processor in the server? AFAIK the best way to get optimal speed is to use SSD's in RAID0 for your database.

[Patvs]

i want to be able to import 5 million HH in a couple hours, as opposed to a couple days / weeks / months.

i have HEM running on a windows server 2008 box with 4GB RAM and 15Krpm SA-SCSI SAN storage, and importing large #s of hands is just taking foreverrrrrrrr.

do i need a supercomputer to be able to work productively with tens of millions of HH in HEM / postgresql? what are minimum system requirements for my purposes?

You don't need a supercomputer to WORK productively with a 10+ million database. However you DO need one to IMPORT 5 million hands at once.
I have 2x Solid State Disks in RAID0 and the import starts just fine, but the import speed drops significantly over time.

Tip: don't import them all at once. Try importing 500.000 at a time.

[almaplayera]

What kind of speeds are you getting currently? What processor in the server? AFAIK the best way to get optimal speed is to use SSD's in RAID0 for your database.

getting 20-30 hands/s. know of any hosting company that offers ssd raid 0 storage?


You don't need a supercomputer to WORK productively with a 10+ million database. However you DO need one to IMPORT 5 million hands at once.
I have 2x Solid State Disks in RAID0 and the import starts just fine, but the import speed drops significantly over time.

Tip: don't import them all at once. Try importing 500.000 at a time.

i find that i'm constantly wanting to import / export / delete from large databases. but as they've gotten into the 5-10 million+ record range, as opposed to the 500k-1 million record range, it's just not feasible anymore. i'd spend all my time waiting.

i really haven't seen much difference in total import time between importing 10 different folders with 500K hands in each folder (in 1MB files) vs importing 5 million hands in a single folder (still in 1MB files).

[fozzy71]

Yes, smaller groups definitely help. Vacuum and Defrag could also help. make sure you stop your postgresql service before you defrag.

[fozzy71]

getting 20-30 hands/s. know of any hosting company that offers ssd raid 0 storage?.......

I get 80 - 100 h/s on my new Vista 64 Ci7 920 w/ 9gb RAM, standard OEM 7200 RPM Drive. My 3 year old XP AMD 4200+ would only get about 40 h/s.

[almaplayera]

I get 80 - 100 h/s on my new Vista 64 Ci7 920 w/ 9gb RAM, standard OEM 7200 RPM Drive. My 3 year old XP AMD 4200+ would only get about 40 h/s.

you get 80-100 h/s importing into databases already containing 1 million+ hands? is 9GB RAM vs 4GB RAM going to make a difference when 15K RAID SA-SCSI should perform significantly better than 7200RPM SATA? when HEM is importing my 4GB RAM server is only using 20% of RAM...

[almaplayera]

you get 80-100 h/s importing into databases already containing 1 million+ hands? is 9GB RAM vs 4GB RAM going to make a difference when 15K RAID SA-SCSI should perform significantly better than 7200RPM SATA? when HEM is importing my 4GB RAM server is only using 20% of RAM...

also, if so, have you made any changes to your postgresql.conf that could possibly have any effect (yes i've disabled logging)?

[Patvs]

What are the 4K read/write speeds of your 15K RAID SA-SCSI (you can test that with CrystalDiskMark (http://release.crystaldew.info/CrystalDiskMarkSetup))

Most fast "normal" harddisks will only get about 0.5 MB/s. A fast SSD (Intel X25-M) gets 30-60 MB/s which will really noticable increase performance. (a lot more than adding more RAM will)


http://images.anandtech.com/reviews/storage/Intel/TRIM/randomread.png


Also check out Top 10: HEM performance increase tips (http://forums.holdemmanager.com/showthread.php?t=11194) for more performance tips.

[almaplayera]

What are the 4K read/write speeds of your 15K RAID SA-SCSI (you can test that with CrystalDiskMark (http://release.crystaldew.info/CrystalDiskMarkSetup))

Most fast "normal" harddisks will only get about 0.5 MB/s. A fast SSD (Intel X25-M) gets 30-60 MB/s which will really noticable increase performance. (a lot more than adding more RAM will)


http://images.anandtech.com/reviews/storage/Intel/TRIM/randomread.png


Also check out Top 10: HEM performance increase tips (http://forums.holdemmanager.com/showthread.php?t=11194) for more performance tips.

i'm at 26MB read / 16MB write. i've seen that post and done everything that looked applicable.

[fozzy71]

you get 80-100 h/s importing into databases already containing 1 million+ hands? is 9GB RAM vs 4GB RAM going to make a difference when 15K RAID SA-SCSI should perform significantly better than 7200RPM SATA? when HEM is importing my 4GB RAM server is only using 20% of RAM...

My DB is admittedly smaller. I dont have enough hands to test that, unless I were to buy some hands for testing.


also, if so, have you made any changes to your postgresql.conf that could possibly have any effect (yes i've disabled logging)?

No. I used to use the enterprise tuning wizard until it started causing DB connection issues and became a registration required addon.

Have you tried editing your config file? http://forumserver.twoplustwo.com/28/internet-poker/postgresql-optimization-tips-speed-up-yo-slow-ass-computer-362568/

[justdoit]

how many hands/sec do you import with ssd?

im not sure if i should buy one or not. but as it seems the import is way faster with ssd than a common 7.2k upm disk?

[guitarizt]

how many hands/sec do you import with ssd?

im not sure if i should buy one or not. but as it seems the import is way faster with ssd than a common 7.2k upm disk?

dying to know this ^

And I hate to hijack but anyone know what the min quadcore speed I can get away with for 40 tabling on stars with a hud would be? The video card shouldn't matter right as long as it's not onboard (nvidia 8600gt)?

[justdoit]

any quad + 8gb ram i guess?

[Patvs]

I did a import test. (I have Intel i7 920, 6GB DDR3-1600, 2x OCZ 30 GB Vertex in RAID0) Results:

Bulk import speed test:
Test 1: 3x 40 MB txt files (total 84.000 hands). (the 3 largest files from a once "export hands" database, each 40 MB text file contains 28.000 hands)
This took 14 minutes to import, 90 hands /s. (it took about 30 seconds of analysing the files before the actual importing began)

Test 2: 1000 handhistory files in 25 subfolders (total 120 MB, each file ranging from 1Kb-500Kb: average filesize 120Kb) (50.000 total number hands)
This took 18 minutes (half of the time spend on the last 250 files!), 45 hands /s (it was importing with 70 hands /s for the first 750 files, but after that the importing would often freeze for a minute and the "files imported 750 of 1000" would suddenly JUMP to "770 of 1000" instead of to 751. After each "freeze" the hands xx / s would drop.

(importing very small batches of only a few files, I always have 100+ hands/s)


----

When buying a SSD, make sure of two things:
-1 it supports TRIM
-2 its speed is actually A LOT faster than a normal 7200 RPM harddrive. (which is NOT the case with the cheapest/smallest SSDs/older models)

Tip: Best one is the 80+ GB Intel X25-M G2 Postville (so DO NOT buy the 40 GB Intel X25-X, or X25-M G1..... and DO buy the Kingston M-series.... however DO NOT buy the Kingston V-series...... OCZ Vertex is best alternative to the X25-M G2.)

But again.... don't buy an SSD just for improving import speeds. (you can put two cheap 7200 RPMs drives in RAID if you ONLY want to improve that) Its real strength is in:
-2x faster boot time / shutdown time
-5x faster startup time (of any program)
-20x faster VACUUM/ANALYSE time
-20x faster PURGE HANDS time
-10x faster RESTORE/BACKUP time
-NO NEED TO DEFRAG THE DRIVE
-SILENT/LOW POWER CONSUMPTION (perfect for laptops)/GENERATES LITTLE HEAT
-NO RISK OF MECHANICAL FAILURE
-ABILITY TO ENDURE EXTREME SHOCKS/VIBRATIONS/TEMPERATURES
-LOWER CHANCE OF IRRECOVERABLE DATA DAMAGE

[fozzy71]

.....
But again.... don't buy an SSD just for improving import speeds. (you can put two cheap 7200 RPMs drives in RAID if you ONLY want to improve that) Its real strength is in:
-2x faster boot time / shutdown time
-5x faster startup time (of any program).....

I really wish I could have waited to afford a SSD system. :mad:

[justdoit]

I did a import test. (I have Intel i7 920, 6GB DDR3-1600, 2x OCZ 30 GB Vertex in RAID0) Results:

Bulk import speed test:
Test 1: 3x 40 MB txt files (total 84.000 hands). (the 3 largest files from a once "export hands" database, each 40 MB text file contains 28.000 hands)
This took 14 minutes to import, 90 hands /s. (it took about 30 seconds of analysing the files before the actual importing began)

Test 2: 1000 handhistory files in 25 subfolders (total 120 MB, each file ranging from 1Kb-500Kb: average filesize 120Kb) (50.000 total number hands)
This took 18 minutes (half of the time spend on the last 250 files!), 45 hands /s (it was importing with 70 hands /s for the first 750 files, but after that the importing would often freeze for a minute and the "files imported 750 of 1000" would suddenly JUMP to "770 of 1000" instead of to 751. After each "freeze" the hands xx / s would drop.

(importing very small batches of only a few files, I always have 100+ hands/s)


----

When buying a SSD, make sure of two things:
-1 it supports TRIM
-2 its speed is actually A LOT faster than a normal 7200 RPM harddrive. (which is NOT the case with the cheapest/smallest SSDs/older models)

Tip: Best one is the 80+ GB Intel X25-M G2 Postville (so DO NOT buy the 40 GB Intel X25-X, or X25-M G1..... and DO buy the Kingston M-series.... however DO NOT buy the Kingston V-series...... OCZ Vertex is best alternative to the X25-M G2.)

But again.... don't buy an SSD just for improving import speeds. (you can put two cheap 7200 RPMs drives in RAID if you ONLY want to improve that) Its real strength is in:
-2x faster boot time / shutdown time
-5x faster startup time (of any program)
-20x faster VACUUM/ANALYSE time
-20x faster PURGE HANDS time
-10x faster RESTORE/BACKUP time
-NO NEED TO DEFRAG THE DRIVE
-SILENT/LOW POWER CONSUMPTION (perfect for laptops)/GENERATES LITTLE HEAT
-NO RISK OF MECHANICAL FAILURE
-ABILITY TO ENDURE EXTREME SHOCKS/VIBRATIONS/TEMPERATURES
-LOWER CHANCE OF IRRECOVERABLE DATA DAMAGE

havent you said last time smaller files are better for ssd than bigger ones?
however it has proven now otherwise.
for me ssd look pretty nice even tho a raid is way cheaper... but why giving away all the benefits of an ssd like u mentioned :)

[Patvs]

Test 1: 3x 40 MB txt files
Test 2: 1000 x 120Kb files.

Test 1 was so fast because it were only 3 files.
Importing 100+ x 1 MB files is a lot slower than 1000 x 120 Kb files on my system.

[dlip]

Just an fyi in my experience the import speed hasn't been directly linked to disk speed. I think it has more to do with HM processing speed. When I do large imports I use multiple machines with HM installed and all connected to the postgresql server on my network. I find that doing this the import speeds scales linearly with the number of machines (i.e. with 2 machines it's almost twice as fast and with three it's almost 3 times as fast). I've only done this using three machines (since I only have once license - used a trial on the other) but had planned to try more if needed.

[reemas]

Test 1: 3x 40 MB txt files
Test 2: 1000 x 120Kb files.

Test 1 was so fast because it were only 3 files.
Importing 100+ x 1 MB files is a lot slower than 1000 x 120 Kb files on my system.

can you guys make it so when i "Export Hands" using HEM they export to files that are 100KB in size?

that way when we backup and reimport hands, we can do it a lot faster

[schmunzler]

when considering to buy a ssd disk what values should you consider regarding the best possible performance for holdemmanager.

I read some benchmarking tests and the intel x25-m g2 seems to have the best price-performance ratio. However it is pretty bad at sequential writes.

From my understanding the most important benchmark considering holdemmanager should be the I/O Performance? Can anyone confirm this?

Charts, benchmarks SSD-Charts 2009, IOMeter 2006.07.27 (http://www.tomshardware.com/de/charts/ssd-charts-2009/IOMeter-2006.07.27,930.html)

Thanks!

[Patvs]

Yep... it's I/O Performance.
Also access times, read speeds, and 4K read and write speeds of small files.
(but those will ALL be reflected in the I/O performance)

Sequential write speeds are only useful for copying large files / video editing.
Many 'tweakers' now put 2x 40 GB Intel X25-X (which only has 40 MB write speeds) in RAID0, which gives one of the best price/performances.