[Bizgres-general] Ketl functionality in bizgres

[Alon Goldshuv]

Nick,

Thanks for the details.

> 3 Layers to this.
>   1. The patch, exposes something called the copy manager via the
> copyAPI metod. This allows copy commands to be send to the db via JDBC.
>   2. PGCopyWriter, this sits on top of the copy manager to give it a
> prepared statement feel. It deals with producing the command, escape,
> data formats etc.. All you do is prepare, setParameters for each tuple,
> submit it to the batch and then execute the batch.
>   3. PGBulkWriter, this sits on top of the PGCopyWriter to expose the
> the copy functionality t KETL. So by simply switching the class names in
> the KETL xml, you start bulk loading using the copy command.

Got it. Nice.
 
> Note. The PGBulkWriter doesn't perform multiple inserts at once, using
> threads, we have another component that does this but it's not ready for
> release. This one works in a similar way to a ring bus.

The java loader already has functionality to run multiple threads feeding
data into COPY commands in round robin. Integrating PGCopyWriter into it as
is will do the job.

> I would like to see this patch be part of the standard driver, as people
> do ask for it every once in a while.

I can't understand why COPY is not part of PG-JDBC yet, given a patch is out
there for a while. Do some people object to it?

> If you want to integrate into the
> loader feel free, I can talk you through how it works, its really
> simple. Give me a call if you want to talk more - Luke has my number.

Great. I do want to use it with the loader eventually, mainly to reduce
complexity and improve code maintenance. I think this won't be done in the
next few months though, as there are more pressing issues. I'll log it in
JIRA and will come back to it a bit later.

Thanks,
Alon.