Modify

Ticket #4960 (closed Bugs: fixed)

Opened 3 years ago

Last modified 3 years ago

boost::pool_allocator for vector of vectors exhausts memory

Reported by: mattiasg Owned by: cnewbold
Milestone: To Be Determined Component: pool
Version: Boost Development Trunk Severity: Problem
Keywords: Cc: boostpool@…

Description

glr9940 reported in #386 that using boost::pool_allocator for a vector of vectors caused problems. I have now found a slight extension to the example glr9940 reported which does not crash, but exhausts memory even for a tiny vector of vectors.

I have used the development trunk code and gcc 4.1.2.

#include <boost/pool/pool_alloc.hpp>
#include <vector>
#include <iostream>

typedef std::vector<int, boost::pool_allocator<int> > EventVector;
typedef std::vector<EventVector> IndexVector;

int main() 
{
  IndexVector iv;
  int limit = 100;
  for (int i = 0; i < limit; ++i) 
    iv.push_back(EventVector());

  std::cout << "it works\n";
  return 0;
}

I suspect that the critical value for 'limit' depends on the machine. On my machine a limit of 20 seems to work fine, a limit of 24 uses a few Gb of memory, while a limit of 30 stalls the process, 'top' showing that the memory footprint doubles rapidly until memory is exhausted. I've noticed for other examples of this issue that when I replace the vector push_back loop with a rolled out version just creating a number of pooled vectors, it works fine.

Have I made some noob mistake, or is this a bug?

Attachments

singleton_pool.hpp.patch Download (1.9 KB) - added by steven_watanabe 3 years ago.
Patch adding instrumentation to singleton_pool
poolalloctest2_rhel5 Download (119.8 KB) - added by mattiasg 3 years ago.
Output of the modified test program with the patched singleton_pool.hpp
rhel5_out2 Download (2.5 KB) - added by mattiasg 3 years ago.
Proper output file

Change History

comment:1 Changed 3 years ago by steven_watanabe

I can't reproduce the problem with gcc 4.4.1, or with MSVC 10. I'd guess that it has something to do with the standard library implementation of vector, though.

Can you try applying the patch to singleton_pool which I'm about to attach, run the following variation and save the output? It should produce code that you can compile which demonstrates the problem. Check that it actually triggers the problem, and attach it.

#include <boost/pool/pool_alloc.hpp>
#include <vector>
#include <iostream>

typedef std::vector<int, boost::pool_allocator<int> > EventVector;
typedef std::vector<EventVector> IndexVector;

int main() 
{
  std::cout << "#include <boost/pool/pool_alloc.hpp>\n";
  std::cout << "int main() {\n";
  std::cout << "void* tmp;\n";
  {
    IndexVector iv;
    int limit = 30;
    for (int i = 0; i < limit; ++i) 
      iv.push_back(EventVector());
  }
  std::cout << "}\n";
  return 0;
}

Changed 3 years ago by steven_watanabe

Patch adding instrumentation to singleton_pool

Changed 3 years ago by mattiasg

Output of the modified test program with the patched singleton_pool.hpp

comment:2 Changed 3 years ago by mattiasg

On a Suse 11 machine with gcc 4.3.4 I cannot reproduce the problem, but on a RHEL5 machine with gcc 4.1.2 a run with your patched singleton_pool.hpp produces the output which I've attached to this issue. To avoid crashing the server I ran the test on I had to abort the program around the time it used half (~16Gb) of the available memory.

comment:3 Changed 3 years ago by steven_watanabe

Sorry for the delay. The file you attached is an executable. I don't think it's the output.

Changed 3 years ago by mattiasg

Proper output file

comment:4 Changed 3 years ago by mattiasg

Ahhh, crap. Attached the output file now, sorry.

comment:5 Changed 3 years ago by johnmaddock

  • Cc boostpool@… added

comment:6 Changed 3 years ago by johnmaddock

I can reproduce this with gcc-4.0.4 as well. Strange bug!

comment:7 Changed 3 years ago by johnmaddock

Update:

This is a result of two bugs:

1) std::vector invokes undefined behaviour by calling Allocator::allocate(0). This has been fixed in more recent GCC versions. 2) Boost.Pool, when it tries to allocate 0 chunks thinks allocation has failed (a bug), so allocates a whole new block, *and* ups the size of the next block to allocate by a factor of 2. So just a few allocation requests for 0 chunks blows up the heap.

This will be fixed in the sandbox version of Boost.Pool soon.

comment:8 Changed 3 years ago by johnmaddock

(In [68920]) Add test case for bug #4960. Add conditional instrumentation code to headers so we can see what's being allocated when. Fix issue #4960 by allowing pool to allocate 0 chunks. Refs #4960.

comment:9 Changed 3 years ago by johnmaddock

  • Status changed from new to closed
  • Resolution set to fixed

(In [73495]) Merge updated Pool lib from trunk. Fixes #1252. Fixes #2696. Fixes #4960. Fixes #5526. Fixes #5568. Fixes #5700.

View

Add a comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
The resolution will be deleted. Next status will be 'reopened'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.