PDA

View Full Version : How Big can a Database be?



DrAction
01-16-2009, 04:50 AM
How many Hands/Players can a DB contain if it will get purged and vacuumed?

Thanks.

AxelFoley
01-16-2009, 04:55 AM
Hi.

On the mainpage => keyfeatures you can see, that you have a unlimited database-size.

DrAction
01-16-2009, 05:12 AM
but how does it apply in reality if it gets really huge.
lets assume someone has a db with 100million hands and it is purged, vacuumed etc. does it get noticeable slower than someone with 1mio or are there no effects?

Bloged
01-16-2009, 10:02 AM
It will get slower as more data needs to be searched before the right one is found. But everything is depended on the hardware en the kind of data in the database.

Grtz, Bloged

knicknut
01-24-2009, 04:04 PM
I guess as a related question, then: is it better to keep a number of smaller databases than one big database?

For example, which would be faster, 10 databases of 1 million hands each (say, one per month of datamining), or 1 database with 10 million hands?

morny
01-24-2009, 04:18 PM
We dont support multiple databasese however we will in future but it will be slower to use multiple database when we do than it will to just use one.

Leif
01-28-2009, 03:39 PM
Yesterday i imported 140.000 hands @ 18,6 hands/second. (app. 2 hours)

Today i imported 140.000 hands @ 13,1 hands pr. second. (app. 3 hours)


I am afraid that the speed will quickly deteriorate, making it impossible to continue this.

Would it help if i did a VACUUM, and should it be: FULL, FREEZE or ANALYSE ?

Does computer power, RAM etc, have big influence on the import speed (hands/seconds), since i have a kind of shitty computer.

PCP Poker
01-28-2009, 04:10 PM
I have a rather large db (around 20 million hands). Yes the import speed does slow down as the db gets larger. However, this is manageable with regular maintainence. After every large import that I do, I run a full vaccuum and then defrag my hard drive. I use a program called Auslogics Disk Defrag (http://www.auslogics.com/disk-defrag). The first time you run this program, it will take a rather long time to run. However, it does a much better job than the default windows defragmenter and each subsequent time you run the utility it will run much faster.

Also, when your db gets exceedingly large, you can run the reindex application from the HEM Database Management tool. You can get there by going to Options>Database Management>Connect to DB, then click Fix Indexes. This can also be done from PG Admin in the same manner that you run the Vaccuum.

DrAction
01-28-2009, 06:08 PM
had this message after a full vacuum.
has anyone an idea how to do it and what affects this has?



DETAIL: A total of 204800 page slots are in use (including overhead).
2216560 page slots are required to track all free space.
Current limits are: 204800 page slots, 1000 relations, using 1269 kB.NOTICE: number of page slots needed (2216560) exceeds max_fsm_pages (204800)
HINT: Consider increasing the configuration parameter "max_fsm_pages" to a value over 2216560.

morny
01-28-2009, 07:35 PM
Yesterday i imported 140.000 hands @ 18,6 hands/second. (app. 2 hours)

Today i imported 140.000 hands @ 13,1 hands pr. second. (app. 3 hours)


I am afraid that the speed will quickly deteriorate, making it impossible to continue this.

Would it help if i did a VACUUM, and should it be: FULL, FREEZE or ANALYSE ?

Does computer power, RAM etc, have big influence on the import speed (hands/seconds), since i have a kind of shitty computer.

Not sure on this tbh, maybe ask some of the postgreSQL people on 2+2 or ask at the PostgreSQL forums they might have an idea

Bloged
01-28-2009, 07:50 PM
What I wonder if import speed goes down when the database gets larger what is HEM storing when importing? Is it calculating/storing calculated data like VPIP every hand?

Grtz, Bloged

simpletttte
02-12-2009, 05:29 AM
I have a rather large db (around 20 million hands). Yes the import speed does slow down as the db gets larger. However, this is manageable with regular maintainence. After every large import that I do, I run a full vaccuum and then defrag my hard drive. I use a program called Auslogics Disk Defrag (http://www.auslogics.com/disk-defrag). The first time you run this program, it will take a rather long time to run. However, it does a much better job than the default windows defragmenter and each subsequent time you run the utility it will run much faster.

Also, when your db gets exceedingly large, you can run the reindex application from the HEM Database Management tool. You can get there by going to Options>Database Management>Connect to DB, then click Fix Indexes. This can also be done from PG Admin in the same manner that you run the Vaccuum.

just a question : what is the size of your database (how many giga for around 20 million hands ?)

PCP Poker
02-13-2009, 02:18 AM
just a question : what is the size of your database (how many giga for around 20 million hands ?)

Mine is ~ 21 gb, but i have some custom tables added, so I'm not sure exactly how big it would be. I think from some of my observations in the past showed about 750 megs per million hands when the db has been fully vaccuumed.

As an aside, you can really shrink the size of your db by doing a backup and restore, which I have not done on this db. You can often get it down to about 30% of it's original size, if disk space is a concern for you.