Opened 11 years ago

Closed 10 years ago

#1975 closed Bugs (wontfix)

windows_shared_memory implicitly commits virtual memory

Reported by: Jason Sachs <jsachs@…> Owned by: Ion Gaztañaga
Milestone: Boost 1.36.0 Component: interprocess
Version: Boost 1.35.0 Severity: Problem
Keywords: Cc:

Description

http://svn.boost.org/svn/boost/trunk/boost/interprocess/windows_shared_memory.hpp windows_shared_memory::priv_open_or_create() is the function that calls CreateFileMapping?() to reserve a segment of virtual memory. It does not specify either SEC_RESERVE or SEC_COMMIT, and the docs for CreateFileMapping?() say that SEC_COMMIT is assumed.

Either this is a small bug and SEC_COMMIT should be specified explicitly (w/ no change in behavior or other code needed), or there is a larger problem and SEC_RESERVE should be used instead.

The problem with SEC_COMMIT is as follows. Suppose you have a N processes where each process i=0,1,...N-1 is going to need a pool of related memory with a maximum usage of sz[i] bytes, where sz[i] is not known beforehand but is guaranteed less than some maximum M, and has a mean expected value of m where m is much smaller than M. From a programmer's standpoint, the best way to handle this would be to reserve a single shared memory segment and ask Boost::interprocess to make the segment size equal to M.

But if I use the existing boost libs to do this, then because SEC_COMMIT is used, my total system resource usage for virtual memory is N*M (instead of sum(sz[i]) which is likely to be much smaller).

SEC_RESERVE should be used instead. This is more complicated because then you need to do VirtualAlloc? or something later to actually commit the memory on-demand (in fact I'm not quite sure how this would have to work esp. since I'm unfamiliar w/ the inner workings of Boost::interprocess).

as an aside, it would be helpful for me to find out sooner rather than later whether this is considered a valid bug that can be addressed in the coming months; otherwise I would have to change my architecture + use multiple shared memory segments in order to make more efficient use of the limited total system resource of virtual memory.

Change History (4)

comment:1 Changed 11 years ago by Ion Gaztañaga

Resolution: invalid
Status: newclosed

I don't consider this a bug, but a feature. Managing pages on-demand is not a feature currently provided by Interprocess and I find difficult this to be managed by Interprocess.

As mentioned, an alternative would be to use several managed segments created on demand by the user.

comment:2 in reply to:  1 Changed 11 years ago by Jason Sachs <jsachs@…>

Replying to igaztanaga:

I don't consider this a bug, but a feature. Managing pages on-demand is not a feature currently provided by Interprocess and I find difficult this to be managed by Interprocess.

ok but it would help to explicitly specify SEC_COMMIT in the code + to mention this in the documentation, so that when deciding the memory segment size, one is aware of the resource costs.

comment:3 in reply to:  1 Changed 11 years ago by Jason Sachs <jsachs@…>

Resolution: invalid
Status: closedreopened

Replying to igaztanaga:

I don't consider this a bug, but a feature. Managing pages on-demand is not a feature currently provided by Interprocess and I find difficult this to be managed by Interprocess.

As mentioned, an alternative would be to use several managed segments created on demand by the user.

I hope you won't take offense if I reopen this. I do agree that managing arbitrary pages on demand is something unreasonable. However, the problem I'm facing, and the only thing I'm looking for a resolution, is the implications of (a) having to specify a fixed maximum size for the segment, despite possible/probable usage that is much smaller and (b) the Boost library's only option is to commit all that virtual memory at once in Windows.

I've looked around a little bit at the source files for boost::interprocess and it looks like there some facilities in http://svn.boost.org/svn/boost/trunk/boost/interprocess/detail/managed_memory_impl.hpp to grow() file sizes. The memory reserved by CreateFileMapping/MapViewOfFile? can't grow, but what if there were a separate "size" and "max_size" in http://svn.boost.org/svn/boost/trunk/boost/interprocess/mapped_region.hpp ? (which appears to be the class that calls MapViewOfFile? via map_view_of_file_ex()) "max_size" would represent the argument passed to CreateFileMapping? (with the SEC_RESERVE flag) / MapViewOfFile?, and "size" representing the portion of that memory allocated; calls to grow() would call VirtualAlloc?, not in a haphazard random-access per-page way, but merely to increase the size used by the shared memory segment, in a sequential, linear (or exponential if that makes sense) fashion.

comment:4 Changed 10 years ago by anonymous

Resolution: wontfix
Status: reopenedclosed

This feature won't be implemented because of complexity and time constraints.

Note: See TracTickets for help on using tickets.