Auto importer 50% slower than before

[LeCopain]

I've been using 18a + table scanner add-on since last week.

I noticed my import speed has halved, eventhough I didn't import a significant amount of new hands.

I can't remember anything else than import speeds around 50-58 hands / second.
Today I am surprised to see the speed at 22 - 24 hands / seconds.

I am using Vista 64, not downloading / using other applications.

Has anything changed?

[fozzy71]

I would suggest you do the majority of the things in this thread: http://208.109.95.123/forum/showthread.php?t=11194

Some of the most important things, imo:

- Turning off logging and deleting existing logs
- Vacuum/Full/Analyze, then Reindex
- SQL Tuning Wizard
- Defrag last and make sure you stop the postgresql service before you defrag. I personally use Defraggler.
- Do a Cold Boot (Shut Down > Start) before you use HM again

[LeCopain]

tx for the response, will look into it.

But my main concern is, speed halved within a week. That doesn't look like normal 'pollution'.

[netsrak]

You don't give any details so you can't expect more than general hints.

Which pokersite?
How many hands do you have in your database?
How many hands are you importing - how many files?
Which import type?

Btw. for me 20h/sec are normal for a database with millions of hands.

[LeCopain]

Well I didn't expand my DB more than normal over last week, so a drop from 58h/s to 23h/s does worry me.

Sites:
iPoker & FTP
hands: plenty (130k known players)
import: 130 files, 40k hands (regular pack of datamined hands)
auto import.

[netsrak]

Do the tuning from Fozzys links but i think the speed is ok.

[LeCopain]

I got this message after a a few hours vacuum / analyzing:

Current limits are: 204800 page slots, 1000 relations, using 1269 kB.NOTICE: number of page slots needed (319568) exceeds max_fsm_pages (204800)
HINT: Consider increasing the configuration parameter "max_fsm_pages" to a value over 319568.

any clues where I can do this?

[netsrak]

postgresql.conf

[LeCopain]

I am fairly confident to say something has changed in the software.

I did a lot of tuning, purged 12 million hands, defragged and vacuum analyzed afterwards, added files to exception list of defender and firewall, updated to 1.09 beta 20. editted the max_fsm_pages

Unfortunately my importing speed is still below the usual rates.
As mentioned before until this week I've been getting import speeds around 51-58 hands /s, yesterday it dropped to 22, today after tuning and stuff i got to 31 hands / s.

I am currently importing 390 files, with approximately 400 000 hands. iPoker and FTP 6max hands. Very dissappointing to see it takes several hours.

Did anything change in the beta's after 1.18 which could have caused this drop in speed?

[netsrak]

I didn't notice such a change when updating from 1.08 to 1.09x but i will ask Roy for a statement.

[Rvg72]

I am fairly confident to say something has changed in the software.

I did a lot of tuning, purged 12 million hands, defragged and vacuum analyzed afterwards, added files to exception list of defender and firewall, updated to 1.09 beta 20. editted the max_fsm_pages

Unfortunately my importing speed is still below the usual rates.
As mentioned before until this week I've been getting import speeds around 51-58 hands /s, yesterday it dropped to 22, today after tuning and stuff i got to 31 hands / s.

I am currently importing 390 files, with approximately 400 000 hands. iPoker and FTP 6max hands. Very dissappointing to see it takes several hours.

Did anything change in the beta's after 1.18 which could have caused this drop in speed?

Are you using auto import or import file/folder? Due to FTP changes I had to make HM do more work during auto import for FTP specifically but it shouldn't affect anything when using import file / folder.

Try importing just ipoker and see if that works better.

Nothing in the 1.09 HM should be causing this. You could also try creating a new DB and importing these new hands and see what happens

Roy

[brbrbr]

Perhaps you turned on "Insert Standardized Hand Format" in the Options - Observed Hand History Configuration. This could slow down things.

[LeCopain]

I did not have the insert standardizez HH format turned on.

@ Roy,

Yes I did use the auto import instead of 'import from folder'. Good to know it makes a difference. Will try later to see if anything improved.
thanks