PDA

View Full Version : Export -> Import generates new hands



batmanhm
05-10-2011, 10:12 AM
I have HM on 2 computers, playing alternatively. I transfer my hand history almost daily, using the "Export all hands to hard drive" function in Sessions tab.

Bug no 1: When you export RUSH hands to disk and import it in a new DB, they are not seen as RUSH hands, they are seen as normal hands.

Bug no 2: When you check for duplicate hands, you are checking for the hand history to match rather than checking for identical hand id, which results is hands which are actually duplicated to be imported serveral times. Even when the only difference between the HH text is just a 'new line' character.

Bug no 3: Sometimes, somewhere along the line of action - import from Full Tilt hand history file, Export to Disk, Import from Folder - your software is adding a new line character to a hand history text(or maybe at the end of a hand history file which it exports, I don't know).

But 1 is easy to reproduce. Bug 2 as well(you can just export a hand, add a new line at the end, and import it using import from folder function). I don't know how you could reprocude Bug 3, since it seems to be very rare. I moved over about 30k hands this time and I got him only in 4 of them.

Below is how bugs 2 and 3 were proven:

After a while, I started to notice that the number of hands in the 2 db is not matching. My investigation was just pointless, so I just decided to transfer all from one side to another and then vice versa.

After I exported the hands from Machine 1, I decided to reimport them just to make sure that all of them were exported. I was expecting HM to see all hands which I was trying to import as duplicates. However, I was surprised to see that 4 of the exported hands(from the same DB!) were actually seen as new hands.

I am no SQL expert, but I started to look a bit into it. All the hands which I have in the DB are from Full Tilt Poker, so it was that easy:

select count(cast(SUBSTR(handhistory,23,11) as bigint)) from handhistories;

Result: 168987


select count(distinct cast(SUBSTR(handhistory,23,11) as bigint)) from handhistories;

Result: 168983


select count(t2) as t3, t1 from (select pokerhand_id as t2, cast(SUBSTR(handhistory,23,11) as bigint) as t1 from handhistories) as my_fake_table group by t1 order by t3 desc;

Result: the duplicate hand ids as the first records.


One of the duplicate hands is:

First record, imported directly from Full Tilt HH file:


Full Tilt Poker Game #30291165132: Table Mercury - $0.25/$0.50 - No Limit Hold'em - 08:18:10 ET - 2011/05/04
Seat 1: batilt ($21.50)
Seat 2: Boztastic ($56.20)
Seat 3: KoReaNcHoBO ($50)
Seat 4: larsrit ($88.10)
Seat 5: rinarina1 ($39.50)
Seat 6: rtutn ($69.25)
Seat 7: KrzysiekJ ($59.30)
Seat 8: xntriq ($69.55)
Seat 9: FullTiltPoker81 ($48.70)
KrzysiekJ posts the small blind of $0.25
xntriq posts the big blind of $0.50
The button is in seat #6
*** HOLE CARDS ***
Dealt to batilt [7s 6d]
FullTiltPoker81 folds
batilt folds
Boztastic folds
KoReaNcHoBO folds
larsrit has 8 seconds left to act
larsrit raises to $1.40
rinarina1 folds
rtutn folds
KrzysiekJ folds
xntriq calls $0.90
*** FLOP *** [3d 7c Jh]
xntriq checks
larsrit bets $2
xntriq folds
Uncalled bet of $2 returned to larsrit
larsrit mucks
larsrit wins the pot ($2.90)
*** SUMMARY ***
Total pot $3.05 | Rake $0.15
Board: [3d 7c Jh]
Seat 1: batilt didn't bet (folded)
Seat 2: Boztastic didn't bet (folded)
Seat 3: KoReaNcHoBO didn't bet (folded)
Seat 4: larsrit collected ($2.90), mucked
Seat 5: rinarina1 didn't bet (folded)
Seat 6: rtutn (button) didn't bet (folded)
Seat 7: KrzysiekJ (small blind) folded before the Flop
Seat 8: xntriq (big blind) folded on the Flop
Seat 9: FullTiltPoker81 didn't bet (folded)






Second record, imported from the file resulted from the export from HM:

Full Tilt Poker Game #30291165132: Table Mercury - $0.25/$0.50 - No Limit Hold'em - 08:18:10 ET - 2011/05/04
Seat 1: batilt ($21.50)
Seat 2: Boztastic ($56.20)
Seat 3: KoReaNcHoBO ($50)
Seat 4: larsrit ($88.10)
Seat 5: rinarina1 ($39.50)
Seat 6: rtutn ($69.25)
Seat 7: KrzysiekJ ($59.30)
Seat 8: xntriq ($69.55)
Seat 9: FullTiltPoker81 ($48.70)
KrzysiekJ posts the small blind of $0.25
xntriq posts the big blind of $0.50
The button is in seat #6
*** HOLE CARDS ***
Dealt to batilt [7s 6d]
FullTiltPoker81 folds
batilt folds
Boztastic folds
KoReaNcHoBO folds
larsrit has 8 seconds left to act
larsrit raises to $1.40
rinarina1 folds
rtutn folds
KrzysiekJ folds
xntriq calls $0.90
*** FLOP *** [3d 7c Jh]
xntriq checks
larsrit bets $2
xntriq folds
Uncalled bet of $2 returned to larsrit
larsrit mucks
larsrit wins the pot ($2.90)
*** SUMMARY ***
Total pot $3.05 | Rake $0.15
Board: [3d 7c Jh]
Seat 1: batilt didn't bet (folded)
Seat 2: Boztastic didn't bet (folded)
Seat 3: KoReaNcHoBO didn't bet (folded)
Seat 4: larsrit collected ($2.90), mucked
Seat 5: rinarina1 didn't bet (folded)
Seat 6: rtutn (button) didn't bet (folded)
Seat 7: KrzysiekJ (small blind) folded before the Flop
Seat 8: xntriq (big blind) folded on the Flop
Seat 9: FullTiltPoker81 didn't bet (folded)







The only difference I notice between them is the number of lines at the end of the text(4 in the first one, 5 in the second one).

Patvs
05-10-2011, 09:14 PM
Bug no 1: When you export RUSH hands to disk and import it in a new DB, they are not seen as RUSH hands, they are seen as normal hands.


Correct. Never export RUSH hands. Just import the original handhistories from the archive folder C:/HMArchive. When you export them the RUSH characteristics will get lots. This is not something that will be fixed, as the RUSH hands if you only look at the handhistory (how they're stored in the handhistory index in the PostgreSQL database) look identical to non-RUSH hands.

Only during the import of the original hands (where HoldemManager can see by the name of the handhistory file: entry1, entry2, etc.) will it know it's a RUSH hands.

(note: HoldemManager doesn't add the name of the handhistory file to every hand, otherwise it could make the database even larger. It's now already at 1 million hands = 10 GB)





Bug no 2: When you check for duplicate hands, you are checking for the hand history to match rather than checking for identical hand id, which results is hands which are actually duplicated to be imported serveral times. Even when the only difference between the HH text is just a 'new line' character.


HoldemManager only checks for the hand ID.
If two hands with the same hand ID show up in the database, it's a sign of a corrupt database.



Bug no 3: Sometimes, somewhere along the line of action - import from Full Tilt hand history file, Export to Disk, Import from Folder - your software is adding a new line character to a hand history text(or maybe at the end of a hand history file which it exports, I don't know).


Don't export handhistories. Just always save the hands in their original formatting by saving the archive folder.
The "export" should only be used: if you completely lost your archive folder,... or if you want to export a single hand so you can post it on 2+2 or a poker forum/blog to comment on it.

batmanhm
05-11-2011, 05:47 AM
the RUSH hands if you only look at the handhistory (how they're stored in the handhistory index in the PostgreSQL database) look identical to non-RUSH hands.

Only during the import of the original hands (where HoldemManager can see by the name of the handhistory file: entry1, entry2, etc.) will it know it's a RUSH hands.


Actually, they do not look identical. In the handhistory text column the table name is a characteristic of rush hands. There is no normal table named as a RUSH table.


(note: HoldemManager doesn't add the name of the handhistory file to every hand, otherwise it could make the database even larger. It's now already at 1 million hands = 10 GB)

That's beyond the point, but it actually does.
select filename from handhistories order by filename limit 1;
returns a filename, which means that all my 150k+ hands have a filename associated to them.


HoldemManager only checks for the hand ID.
If two hands with the same hand ID show up in the database, it's a sign of a corrupt database.

Sorry, but this is not true. If you would actually have a check on id then I could not have the 2 hands with the same ID on the database. You could maybe name it corrupt if there would be a 'hand id' column, other then the sequential PK, which would have a unique restriction. But such a column is not there. When you don't have a check on a data structure which logically you'd wish to have, and when that nonexistent check is not met, you don't get to call the data which entered there corrupt. Especially not in this case, where the data is easily fixable by an update.


Don't export handhistories. Just always save the hands in their original formatting by saving the archive folder.
The "export" should only be used: if you completely lost your archive folder,... or if you want to export a single hand so you can post it on 2+2 or a poker forum/blog to comment on it.

Please take no offence, but I can not take such advice. The entire point of databases is to avoid having a structured file system for your data. Why would you even use postgres sql if all the data which enters the database must still be kept in its original format? Just use another 30-40 files for the rest of the tables and get rid of the DB server, if you don't intend to use it for its purpose.

Export/Import issues are easily fixable, both from Postgre side and from GUI side(I assume it is java). Here are the java side solutions:

Bug 1: Tweak the filename value which you send to DB at the import time. On each import, check the table name. If it is one of the RUSH table names, just add Entry 1 to the filename value for that hand. Probably about 5 to 10 lines of code.
Bug 2: instead of string1.trimI().equals(string2.trim()), write
string1.replace("\n"," ").trim().equals(string1.replace("\n"," ").trim()); (or something like that, you get the point)
But 3: Who cares about it when you have Bug 2 fixed?

Patvs
05-11-2011, 12:52 PM
Please email us two handhistories with the same ID, that will both show up in HoldemManager when you import them to support@holdemmanager.com




Why would you even use postgres sql if all the data which enters the database must still be kept in its original format?


Example: we're adding 80 new stats in the next update. The original hands need to be reimported, in order to update the database tables.
Ideally, we would have a "fix database" option... we used to have a "fix currency" button... and "fix EV" but we prefer if users just reimport ALL their original handhistories into a NEW database.

I'll forward your comments on the exporting RUSH hands to a programmer.