[Bizgres-general] Performance - WAL bypass + parse

[Luke Lonergan]

I can save you some time - I think you are seeing expected behavior.

Your bottleneck on your BSD system is probably in the disk access, not in
the CPU processing.  You need to check how fast your disk subsystem is with
a simple test like:
  time dd if=/dev/zero of=bigfile bs=1024k count=4000

This times the write of 4GB of data to a file using a 1MB blocksize.  If you
then divide 4000/time you will get the MB/s number that represents your
sequential filesystem write speed (with minimal buffering because you used
4GB or roughly 2x memory).  Do the same for read:
  time dd if=bigfile of=/dev/null bs=1024k

My guess is that you'll find your IDE RAID 0 is slow for some reason.

The fast parsing is effective when your underlying disk speed is much faster
than the existing parse logic can handle.  As previously posted, we measured
the existing (unpatched) parse logic at 17MB/s max speed on a 3.2GHz Xeon
processor.  So, on your P-III, which is faster clock for clock than a P4
Xeon, the existing parser speed is likely about 6MB/s, which I'm guessing to
be faster than your disk.

The speed increase in the new parser code is 550%, but you'll only see it if
the other two stages of loading (attribute conversion, data insertion) are
the bottleneck.  The WAL bypass is designed to speed up the data insertion
stage, but I'm not sure we have a working WAL bypass for COPY that you've
seen.  We've tested one and it did perform, but it wasn't ready for release
yet.

- Luke 


On 6/7/05 4:19 AM, "Mark Kirkwood" <markir at paradise.net.nz> wrote:

> Mark Kirkwood wrote:
>> Mark Kirkwood wrote:
>> 
>>> This is about a 14% improvement - (snippage). Hmmm, will get the
>>> profiler out an' see...
>>> 
>> 
>> FreeBSD 5.4's gcc/gprof generates output with all zeros in the '%time'
>> column - urg - when I get that sorted I will continue with this....
>> 
> 
> Arrg - I am picking up the wrong postmaster.gmon, or else some
> short-lived postmaster's .gmon is overwriting my 'interesting' one....
> I'll see if I can get some proper profiles tomorrow!
> 
> 
> Cheers
> 
> mark
> 
>