[Bizgres-general] Question: Is 64-bit cache support valuable?

[Simon Riggs]

On Fri, 2005-07-22 at 21:07 +1000, Neil Conway wrote:
> Simon Riggs wrote:
> > Most important thing is that the memory management needs to be a Top-
> > down Memory Allocation scheme. So, all work_mem requests get allocated
> > from a single top-level pool. This is defined as total_work_mem which is
> > a SUSET parameter allowing the administrator to change it while the
> > server is running.
> 
> A "single top-level pool" would suggest it is allocated in shared 
> memory, right? If so it won't be possible to change on the fly.

No. I'm suggesting just a maximum total limit. It would still be private
local memory to palloc'd to each backend. So just a thin layer to ensure
that the total of all allocated work_mem related memory does not exceed
a defined total which could then lead to swapping.

> I don't think it would be a good idea to actually allocate work_mem in 
> shared memory, anyway -- ISTM you can get the same behavior by just 
> tracking allocation *requests* globally, but still doing the allocations 
> themselves via palloc().
> 
> > Limit action
> > What should we do if the requested size is unavailable?
> > - Overallocate (current action)
> > - Trimming - Give back the maximum available, however much that is...
> > - Step down - Return the next smaller unit of size, in a variable scheme
> > - Block on a Wait loop until limit request succeeds... but might wait a
> > long time depending upon workload
> > - Throw error and allow user to resubmit later - if we did that it would
> > be important to throw and error quickly, rather than execute for an
> > hour, hit a limit and then abort
> 
> None of these seem very appealing :( "Trimming" is right out, as it 
> would only work for the first such request (once you've handed over the 
> remaining total_work_mem, there's nothing available to satisfy another 
> request). Similar for "step down" -- if you make planning decisions with 
> the "guarantee" that a certain amount of work_mem will be available, 
> this would violate that. Blocking or throwing an error would be 
> acceptable in some circumstances, but not all.

I agree, but then neither is overallocation, which is the behaviour all
of this is designed to avoid. I would suggest that it would be settable
to cater for a range of situations. 

I was going to take Josh's suggestion and take a much simpler design
over to -hackers, though not for a few weeks yet (only because I'm
finishing off other things).

Best Regards, Simon Riggs