PDA

View Full Version : Purge not purging ???



nachtwacht
01-06-2010, 10:17 AM
Hello,

Running HEM 1.10 Beta 6 Postgrep 8.3

I followed the following steps today.

1) Went to options, observed hand history cofig, purge hand histories.

I made some selections and let HEM work it out. When finished it gave me a notice that roughly 1.5M hands were marked for deletion.

2) I got the option to do a vacuum analyse from there but it did not work. I had to do it manualy.

3) Went o Postgre, selected Vacuum, full and analyse and let it run for a long time.

4) When finished, I did a hand count (had just done one for a reply in the HEM forums) and now it shows that zero hands actualy have been deleted. My free disk size also is still the same meaning my database still consumes just as much space.

This is the second time I notice this, first time I just thought it was my mind playing tricks on me... turns out, it's not my mind :)

anyone any idea whats going wrong ?

fozzy71
01-07-2010, 01:49 AM
I have duplicated this and am forwarding it to be duplicated again by another tech. I will post my step by step screenshots a bit later.

nachtwacht
01-13-2010, 04:50 AM
Funny thing...

Database shows the hands are in the database (when doing a hand count) but I am looking at some player now for wich I have some hand history but have not played that hand myself and I can not get the HH in the replayer.

Also, in the replayer "stats up to day of hand" does not work anymore. (in hindsight, this will be normal when deleting a lot of old hands but for analysis purpose it might be very usefull if you would have a table in the database wich would have the stats up till that day)

Anyway, my SQL database shows that the hands are still in the database, my HEM does however not show them.

something funky going on there :)

netsrak
01-13-2010, 05:29 AM
How do you do the handcount?

nachtwacht
01-13-2010, 05:37 AM
How do you do the handcount?

I followed these steps:

http://www.holdemmanager.net/faq/afm....aspx?faqid=19

The count stayed exactly the same before and after the purge. (as did the size on the harddisk)

netsrak
01-13-2010, 10:02 AM
This command should display a difference after a successful purge. Are you sure you checked the correct database or did you purge tourney hands.

When you get the display of 1.5 M hands purged did you choose ok? Because otherwise the command will be rolled back.

For getting more free harddisk space you'll need to run a vacuum/full analyse after the successfull purge but when no hands were deleted it will not help

nachtwacht
01-13-2010, 10:52 AM
This command should display a difference after a successful purge. Are you sure you checked the correct database or did you purge tourney hands.

Yes. I only have 1 database with so many hands, all other databases are empty (I test sometimes with databases and also have PT3 and PT2 so thats why I have 4 or 5 databases) The vacuum/full analyse took about 2 hours so I can only imagine the correct database was selected.

Also, Fozy clearly states "I have duplicated this and am forwarding it to be duplicated again by another tech" so he apperantly could duplicate it....


When you get the display of 1.5 M hands purged did you choose ok? Because otherwise the command will be rolled back.

I have no idea what the other options available would have been but I think usualy after I do something like that OK is my default choice :)


For getting more free harddisk space you'll need to run a vacuum/full analyse after the successfull purge but when no hands were deleted it will not help

I will need to do that anyway because it failed to do it from inside HEM. I got the message "do not close DOS window" after wich it said, failed, please do it manualy, wich I did.

But again, Fozy could duplicate it. My understanding is then that he could duplicate my problem. I think it's probably better for you to ask him what he is doing because you will trust him more than you would trust your average user who you ask if he is realy pressing the ok button :)

If my understanding is wrong that he could not duplicate it, then please let me know and I will do everything again making screenshots etc.

fozzy71
01-13-2010, 01:29 PM
Ooops, I should have posted these sooner. Hopefully I remember them all. :o

Before purging:

http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-sqlb4purge.png


Before Purging but after Full/Vacuum/Analyze

http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-sqlb4purge-aftervac-1.png


Purging:

http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-sql-purge1a.png

http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-sql-purge1b.png

http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-sql-purge1c-WTF.png


Before reboot and vacuum:

http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-sql-b4rebootNvac.png


After reboot and before vacuum

http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-sql-aftRebootB4vac.png


Vacuum details:

http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-sql-vacwindow.png


After Full/Vac/Analyze:

http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-sql-aftFullVacAnalyze.png


After Reindex:

http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-sql-aftReindex.png

nachtwacht
01-13-2010, 02:37 PM
2 intresting things....

1) you seem to change your database directory all the time ? I see 3 directories, 8.4 and 8.4.2.1 and postgresql without any number.

I guess you backed up several times or something....

2) if I only look at the 8.4.2.1 directory I see your database grow :)

using the created date, xxxx minutes ago, we see the oldest screenshot being the one from 1 minute ago. (the longer the directory exists, the higher the xx minutes ago will be ofcourse :) )

The size there is 1.727.000.0000 bytes

Then one from 1 hour ago size 1.880.000.000 bytes

then one with 2 hours ago size 1.939.000.000 bytes

and last one from 3 hours ago size 1.947.000.000 bytes

(its ofcourse not exactly 000.000 but I did not want to type all those numbers :) )

So your directory size is actualy growing while you are doing things like vacuum.

I am also very curious, how many hands do you have in your database anyway ? You have a 1.6GB directory and you are deleting 190.834 hands. I would say that you are roughly deleting all hands in your database (not bad for test by the way)

That should have the directory on the harddisk shrinking considerably ofcourse....

So I am a little curious.... are you posting these images for your technical support to show something is wrong ? Because this is exactly indeed what I discribed, after doing a full vacuum analyze the disk usage does not shrink.

fozzy71
01-14-2010, 03:10 AM
2 intresting things....

1) you seem to change your database directory all the time ? I see 3 directories, 8.4 and 8.4.2.1 and postgresql without any number.

I guess you backed up several times or something....

Looks like I took a couple extra screen shots. The only ones that matter really are the 8.4.2.1, that I took the most pics of.


2) if I only look at the 8.4.2.1 directory I see your database grow :)

using the created date, xxxx minutes ago, we see the oldest screenshot being the one from 1 minute ago. (the longer the directory exists, the higher the xx minutes ago will be ofcourse :) )

The size there is 1.727.000.0000 bytes

Then one from 1 hour ago size 1.880.000.000 bytes

then one with 2 hours ago size 1.939.000.000 bytes

and last one from 3 hours ago size 1.947.000.000 bytes

(its ofcourse not exactly 000.000 but I did not want to type all those numbers :) )

So your directory size is actualy growing while you are doing things like vacuum.

Yes, my DB grew during this.


I am also very curious, how many hands do you have in your database anyway ? You have a 1.6GB directory and you are deleting 190.834 hands. I would say that you are roughly deleting all hands in your database (not bad for test by the way)

If I run a SQL query I think it is about 235k hands.


That should have the directory on the harddisk shrinking considerably ofcourse....

The Options > Purge Hands method removes more actual data (hand histories)


So I am a little curious.... are you posting these images for your technical support to show something is wrong ? Because this is exactly indeed what I discribed, after doing a full vacuum analyze the disk usage does not shrink.

Yes, I was posting to confirm that when using the Observed Hand History Configuration > Purge Hands, some things aren't working properly.


I am doing some further testing with the Options > Purge Hands method.

nachtwacht
01-14-2010, 03:59 AM
I am doing some further testing with the Options > Purge Hands method.

Probably also intresting is to check if you can still acces some of the hands that should have been deleted but are not available anymore. They still show up in the grid but when you want to play them in the replayer just nothing happens.

That is what is also happening with me. Allthough I clearly did not delete any hands from my database as far as disksize and handcount shows, I can not replay the hands anymore that would have been deleted with this purge hands method.

ps: I see you are using 8.4 I was using 8.3 and was a little afraid that it might be my postgresql messing with me

fozzy71
01-14-2010, 01:31 PM
That purging option worked as expected. I am going to try and duplicate my original results once more. Note, this SQL installation contains 2 databases. My HM DB and the same hands imported to PT3.


http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-purge-2a.png

http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-purge-2b.png

http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-purge-2c.png

http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-purge-2d.png

http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-purge-2e.png

http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-purge-2f.png

nachtwacht
01-24-2010, 05:21 AM
any followup on this ? Before we forget it :)

fozzy71
01-25-2010, 03:43 PM
I just tried this again, on a smaller DB.

The Purge was instantaneous again, leading me to believe nothing was actually purged. But, this time my DB did seem to get a little smaller, unlike last time when it actually grew larger. :confused:

*I forgot to run a PGAdmin query before and after for a total hasnd count on the DB :(

*I will try and reproduce it again and count the total hands in the DB.

Before Purge:
http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-purge-4a.png


http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-purge-4b.png


http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-purge-4c.png


After Purge:
http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-purge-4d.png


After Full/Vacuum/Analyze:
http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-purge-4e.png


After Reindex:
http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-purge-4f.png

nachtwacht
01-25-2010, 04:16 PM
Lets hope we find out whats happening.... but so far it looks like this specific deletion of hands does not seem to work for some reason.

Just my observation though :)

fozzy71
01-26-2010, 12:13 AM
Lets hope we find out whats happening.... but so far it looks like this specific deletion of hands does not seem to work for some reason.

Just my observation though :)

I agree with you. It is the \data folder issues that are confusing me now. It doesn't seem to be consistent at all.

nachtwacht
01-26-2010, 03:25 AM
I agree with you. It is the \data folder issues that are confusing me now. It doesn't seem to be consistent at all.

Yes. The small shrinkage you get this time might be due to postgre just cleaning up some unused data but why the database grew the first time is a mistery to me...

It does not do any indexing while doing the vacuum I think ? That would be the only reason I can think of to why a database would grow without importing any data...

Don't you have an SQL expert somewhere ? If not, maybe I know someone that could help you out in the future.

fozzy71
01-26-2010, 10:28 PM
Yes. The small shrinkage you get this time might be due to postgre just cleaning up some unused data but why the database grew the first time is a mistery to me...

It does not do any indexing while doing the vacuum I think ? That would be the only reason I can think of to why a database would grow without importing any data...

Right, I probably should have vacuumed before the purge, and vacuumed after the purge again. But that isn't the main concern here. The hands are definitely not being purged. The observed hands also are not purged during this process. I ran both SQL queries before and after the purge and there was no change.



Don't you have an SQL expert somewhere ? If not, maybe I know someone that could help you out in the future.

The developer is the sql expert around here usually, and we try and duplicate/triplicate the bug before we bug him with it. Feel free to PM me the details about anyone you know that is an SQL power user. We may be looking to fill that position sometime in the future, to help reduce the developer's workload further.


Here is my latest run through this process.

http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-purge5a.png
http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-purge5b.png
http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-purge5c.png
http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-purge5j.png
http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-purge5k.png
http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-purge5l.png
http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-purge5m.png
http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-purge5n.png
http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-purge5o.png
http://i207.photobucket.com/albums/bb176/fozzy71/HEM/support/hm-purge5p.png

nachtwacht
01-27-2010, 02:03 PM
ok, shooting from the hip here (since we don't know anything about your database design) but I think we (me and my friend that has some database knowledge) have found where the problem comes from.

I think it's probably best if your developer contacts me through skype or something or you send me an email message where I can send some detailed information. I don't mind putting it all in here but not realy usefull.

I will send you a PM.

fozzy71
01-27-2010, 06:21 PM
Can you please email me the details to fozzy@holdemmanager.net so they are saved in our email management system. Then I can forward it to the developer and other tech members.