PDA

View Full Version : How many hands can HEM take?



baggin
10-08-2008, 11:21 AM
Hi,

I am considering importing a huge chunk of hands (10-20 mill) and then keep on adding a lot of hands through datamining etc.

Is there any limit for how many hands HEM can take eventually? Can it handle 100 mill ++ hands? By handling I mean not becoming unusable in terms of speed and usability.

pellshek
10-08-2008, 01:50 PM
Can I add a question here as it's kinda related:

Why is it that HEM can handle a very large numbers of hands whereas my old PT2 (Access) db used to get slower and slower as more data was added? Is it something to do with the way a SSQL db works or something?

Thanks.

morny
10-08-2008, 04:41 PM
I know Roy has a database around the 15-20million mark IIRC and runs without problems, ill ask him about what to expect beyond 20 million

Access is more for home/office and smaller projects when it comes to larger databases SQL is much more reliable and faster

baggin
10-08-2008, 05:28 PM
Ok, thanks. If Roy nows, it would be nice to hear.

My database is going to exceed 15-20 mill by quit a lot. And keep growing.

Rvg72
10-11-2008, 02:02 AM
Hi,

I am considering importing a huge chunk of hands (10-20 mill) and then keep on adding a lot of hands through datamining etc.

Is there any limit for how many hands HEM can take eventually? Can it handle 100 mill ++ hands? By handling I mean not becoming unusable in terms of speed and usability.

It should be fine - I know of people who are running those numbers. The issue you will hit will be disk space so just make sure you have a big, fast drive and it will be fine. You should also do frequent backups because if you do have drive issues then bigger DB = more likilihood of problems and it also means it will take much longer to reimport if you had to.

Postgresql is an enterprise DB so it can handle it and the HM database was built with huge quantities of hands in mind.

baggin
10-11-2008, 12:35 PM
Ok, thanks. Sounds good.

pellshek
10-11-2008, 02:45 PM
Postgresql is an enterprise DB so it can handle it and the HM database was built with huge quantities of hands in mind.


Thanks Roy. Just to ask another quick question (I know you're busy!) - does this kind of db "clog up" a computer in the way an Access db does? Or is the data somehow "lighter"?

Tx.

Rvg72
10-12-2008, 12:29 PM
Postgres should not cause any issues like this. When it is not in use it is still running in the background but when idling it uses up very little computer power

SootedNinjas
11-10-2008, 05:45 AM
I am looking at around apx 3M hands/month (100K/day) from importing datamined hands.

With this in mind I need to determine how old should I keep the hands in the DB.

What is the upper threshold for the # of hands that the HEM DB can handle ?

If I have an accurate estimate of the threshold then I could determine how long I can keep the hands before deleting it out of the DB.

e.g. if the upper threshold is between 18M to 36M hands, with 3M hands a month then I should be trimming my DB of any hands that is older between 6-12 months ?

Any suggestions....

morny
11-10-2008, 01:20 PM
I guess it hasnt really been tested beyond the 20 million mark so its unknown territory, id keep going until you notice a deteriation and then use that mark, even if we did know the amount it would depend alot on your specs and what kind of HH i,e HU 6max, FR

SootedNinjas
11-10-2008, 01:47 PM
I guess it hasnt really been tested beyond the 20 million mark so its unknown territory, id keep going until you notice a deteriation and then use that mark, even if we did know the amount it would depend alot on your specs and what kind of HH i,e HU 6max, FR


With 3M hands a month then I should be trimming my DB of any hands that is older between 6 to 12 months ?

Any suggestions....

baggin
11-10-2008, 04:31 PM
I'll shoot in a couple of questions here.

1.) What is your Hands/s when importing such a large number of hands?
I am currently importing somewhere around 400k hands (not sure how many it is, but less than 500k and more than 300k).

It is currently running on 12.9/s which makes this take a loooong time.

I usually import smaller samples and the import will usually be somewhere around 30 to 40 hands/s. Is this normal?

Is there any way to speed it up? What is your speed SootedNinjas?

My database is starting to grow pretty big. Not sure of how many hands right now, but 110k+ number og players.

2.) How do you trim the database of old hands, as talked about?

SootedNinjas
11-10-2008, 06:18 PM
I'll shoot in a couple of questions here.

1.) What is your Hands/s when importing such a large number of hands?
I am currently importing somewhere around 400k hands (not sure how many it is, but less than 500k and more than 300k).

It is currently running on 12.9/s which makes this take a loooong time.

I usually import smaller samples and the import will usually be somewhere around 30 to 40 hands/s. Is this normal?

Is there any way to speed it up? What is your speed SootedNinjas?

My database is starting to grow pretty big. Not sure of how many hands right now, but 110k+ number og players.

2.) How do you trim the database of old hands, as talked about?


1) When I first importing datamined hands it was around 50 hands/ seconds and @ 100K hands per day it is taking about 30 minutes. My laptop is running Dual Core (NOT CORE2).

Now that my DB is starting to bloat the hand/s is about 20 hands/s and taking about 1 hour or so to complete.

Now thinking about it, in the error log the other day I saw a message stating that there is frequent clustering happening. Need to research this a little bit more.

Can not remember the exact message and I'm at work so I can't look up the error log right now. Will keep the posted later about this.

Anyway, couple of things I need to try and to see if I can bring the hands/s again back to 50 or more. A) Vacuum maintenance b) Move the Postsgres and the DB out of laptop and into a seperate drive on my desktop (running CORE2 2.1GHZ and SATA Drives) and my laptop HEM accessing the DB thru the LAN or external drive whichever method that's faster.

Having a faster CPU and the DB out of the C: drive will help speed up the importing process. Also, if you are importing datamined hands and playing at the same time then it will definitely slow you down. I usually import my datamined hands just before bedtime. A days worth of hands will NOT necessarily have a HUGE effect on your opponent stats.

If you have the bankroll and want super super fast there is a way. I have done this particular method at work for a project that I am doing.

Get yourself a Solid State Drive (SSD) which is a bit pricey but if you are crushing the 100NL then your bankroll can afford to buy this drive.

I install Windows/XP into the SSD and was able to boot XP from the power-on to the welcome/log in screen IN 7 SECONDS.

Another way of using the SSD is I would keep XP, HEM and other programms in the C: drive but put Postsgress and DB, the windows page file, temp files and internet files into the SSD.

2) Trimming of hands. Read the main FAQ page of HEM and haf-way thru there is a link to a FAQ on purging/trimming hands. Just make sure to backup before deleting hh so if something goes bad you can roll back the delete hands.

http://208.109.95.123/faq/afmviewfaq.aspx?faqid=107

Regards.....

morny
11-10-2008, 08:34 PM
Defragging will have a better impact than vacuuming on importing speed, vacumming is more helpful in running queries more efficently.

Id certainly trim hands older than 6 months with the amount of hand histories you have and there not really useful after that period. After 12 months id delete them completely. Its really a matter of keeping an eye on perfromance and if its droppping increase the trimming and deleting

SootedNinjas
11-10-2008, 10:37 PM
Defragging will have a better impact than vacuuming on importing speed, vacumming is more helpful in running queries more efficently.

Id certainly trim hands older than 6 months with the amount of hand histories you have and there not really useful after that period. After 12 months id delete them completely. Its really a matter of keeping an eye on perfromance and if its droppping increase the trimming and deleting

Performance issue aside.

In regards to profiling opponents, any data that is 6 months old you would consider, for the lack of a better word, "useless" ?

Regards ..........

morny
11-10-2008, 10:42 PM
Youll still have their stats if you just trim it so the only thing you wont have access to is the HH so from a reviewing HH point of view i would doubt id use them after 6 months, depends whether you just use the mining to have stats on players or whether you study players in detail, in the first case stats older than 6 months become unreliable and in the latter case older hands can be useful for profiling but again peoples games can adapt alot over the space of 1-3 months and be unrecogniseable after 6 in some cases. I guess useless was a bit strong:D

SootedNinjas
11-10-2008, 10:46 PM
Thanks for the informative response.

Northernwind
11-11-2008, 11:01 AM
Get yourself a Solid State Drive (SSD) which is a bit pricey but if you are crushing the 100NL then your bankroll can afford to buy this drive.

I install Windows/XP into the SSD and was able to boot XP from the power-on to the welcome/log in screen IN 7 SECONDS.

Regards.....
How fast the import run on SSD drive compared to SATA HDD? How many hands you database have?

SootedNinjas
11-11-2008, 03:24 PM
How fast the import run on SSD drive compared to SATA HDD? How many hands you database have?

Right now I've got 3M+ hands in my DB with an average of 100k hands per day being imported.

The SSD trial I did was for work. But I do plan to use SSD in the future once my BR can subsidize the cost.

baggin
11-13-2008, 04:52 PM
SootedNinjas:
Thanks for the tips. I am running on a pretty top spec pc. Core2Duo E8500, 8 GB RAM etc. At this time I don't really want to buy a SSD. But we'll see.

Trimming hands is something I have to look into.

Morny:
By defragging, do you mean the windows defragging procedure of hard disk??

To Morny/HEM support:
What do you think is acceptable import speed?, when doing this kind of import of datamined hands? Is there a speed that would have to be considered as "the best I could expect"?

SootedNinjas
11-13-2008, 05:01 PM
SootedNinjas:
Thanks for the tips. I am running on a pretty top spec pc. Core2Duo E8500, 8 GB RAM etc. At this time I don't really want to buy a SSD. But we'll see.

Trimming hands is something I have to look into.

Morny:
By defragging, do you mean the windows defragging procedure of hard disk??

To Morny/HEM support:
What do you think is acceptable import speed?, when doing this kind of import of datamined hands? Is there a speed that would have to be considered as "the best I could expect"?

Wow. for specs like that 20 hands/s is not acceptable. Something is slowwing down your import.

And yes defrag is by windows but if your going to defrag the C: drive, well .....

What I would do first is move your HM db & Postsres out of th C: drive and install into a fast SATA external drive. If you have a the BR then spend a little bit on this by getting 10k to 15k RPM drives. e.g. Seagate Barracuda

There is a FAQ in the forum on how to do this procedure

baggin
11-16-2008, 09:06 AM
Wasn't defragging C: what he meant?

It's kind of strange. The import speeds differ a lot....on some imports it can be like 20/s, while other times it is 40-50/s. Maybe it has something to do with the amount that is imported...that it would be smart to import in smaller portions.

Buying a fast SATA external drive could be a option later on. But my db is so big now (78GB), I guess such a hdd would be pretty expensive. I would probably need to buy one close to 200GB.

Well...I'll try some things out and we'll see.

What is the speed everyone else here is getting when importing?

cokedrinker
11-17-2008, 04:29 AM
i get ~120h/s on a 2.5Ghz Penryn, but my database is only 4Gb at the moment - still i would like to hope it doesnt slow to 20h/s when my database gets larger :/
Is there a way of checking how many hands are in the database?

fabio
11-17-2008, 11:54 AM
http://208.109.95.123/faq/afmviewfaq.aspx?faqid=19

Rhibee
11-18-2008, 06:46 AM
Windows has had trouble defragging my PostgreSQL folder so I use a free program called Power Defragmenter (http://www.softpedia.com/progDownload/Power-Defragmenter-Download-20185.html). It allows you to choose a specific file or folder to defragment. I always run it before a big import (I use datamined hands) and it makes a substantial difference to the hands per second.

craiggerz
11-18-2008, 09:57 AM
Id certainly trim hands older than 6 months with the amount of hand histories you have and there not really useful after that period. After 12 months id delete them completely. Its really a matter of keeping an eye on perfromance and if its droppping increase the trimming and deleting

What's the difference between trimming and deleting? And which of them is equal to purging?

I know how to purge, which I believe is deleting, but I do not know how to trim... can you explain this?

morny
11-18-2008, 10:12 AM
Trimming:
http://www.holdemmanager.net/faq/afmviewfaq.aspx?faqid=107

Deleting:
http://www.holdemmanager.net/faq/afmviewfaq.aspx?faqid=152

craiggerz
11-18-2008, 10:33 AM
Trimming:
http://www.holdemmanager.net/faq/afmviewfaq.aspx?faqid=107

Deleting:
http://www.holdemmanager.net/faq/afmviewfaq.aspx?faqid=152

TY sir.