Modify

Opened 10 years ago

Closed 9 years ago

#1975 closed Bugs (wontfix)

windows_shared_memory implicitly commits virtual memory

Reported by: Jason Sachs <jsachs@…> Owned by: Ion Gaztañaga
Milestone: Boost 1.36.0 Component: interprocess
Version: Boost 1.35.0 Severity: Problem
Keywords: Cc:

Description

http://svn.boost.org/svn/boost/trunk/boost/interprocess/windows_shared_memory.hpp windows_shared_memory::priv_open_or_create() is the function that calls CreateFileMapping?() to reserve a segment of virtual memory. It does not specify either SEC_RESERVE or SEC_COMMIT, and the docs for CreateFileMapping?() say that SEC_COMMIT is assumed.

Either this is a small bug and SEC_COMMIT should be specified explicitly (w/ no change in behavior or other code needed), or there is a larger problem and SEC_RESERVE should be used instead.

The problem with SEC_COMMIT is as follows. Suppose you have a N processes where each process i=0,1,...N-1 is going to need a pool of related memory with a maximum usage of sz[i] bytes, where sz[i] is not known beforehand but is guaranteed less than some maximum M, and has a mean expected value of m where m is much smaller than M. From a programmer's standpoint, the best way to handle this would be to reserve a single shared memory segment and ask Boost::interprocess to make the segment size equal to M.

But if I use the existing boost libs to do this, then because SEC_COMMIT is used, my total system resource usage for virtual memory is N*M (instead of sum(sz[i]) which is likely to be much smaller).

SEC_RESERVE should be used instead. This is more complicated because then you need to do VirtualAlloc? or something later to actually commit the memory on-demand (in fact I'm not quite sure how this would have to work esp. since I'm unfamiliar w/ the inner workings of Boost::interprocess).

as an aside, it would be helpful for me to find out sooner rather than later whether this is considered a valid bug that can be addressed in the coming months; otherwise I would have to change my architecture + use multiple shared memory segments in order to make more efficient use of the limited total system resource of virtual memory.

Attachments (0)

Change History (4)

comment:1 Changed 10 years ago by Ion Gaztañaga

Resolution: invalid
Status: newclosed

I don't consider this a bug, but a feature. Managing pages on-demand is not a feature currently provided by Interprocess and I find difficult this to be managed by Interprocess.

As mentioned, an alternative would be to use several managed segments created on demand by the user.

comment:2 in reply to:  1 Changed 10 years ago by Jason Sachs <jsachs@…>

Replying to igaztanaga:

I don't consider this a bug, but a feature. Managing pages on-demand is not a feature currently provided by Interprocess and I find difficult this to be managed by Interprocess.

ok but it would help to explicitly specify SEC_COMMIT in the code + to mention this in the documentation, so that when deciding the memory segment size, one is aware of the resource costs.

comment:3 in reply to:  1 Changed 10 years ago by Jason Sachs <jsachs@…>

Resolution: invalid
Status: closedreopened

Replying to igaztanaga:

I don't consider this a bug, but a feature. Managing pages on-demand is not a feature currently provided by Interprocess and I find difficult this to be managed by Interprocess.

As mentioned, an alternative would be to use several managed segments created on demand by the user.

I hope you won't take offense if I reopen this. I do agree that managing arbitrary pages on demand is something unreasonable. However, the problem I'm facing, and the only thing I'm looking for a resolution, is the implications of (a) having to specify a fixed maximum size for the segment, despite possible/probable usage that is much smaller and (b) the Boost library's only option is to commit all that virtual memory at once in Windows.

I've looked around a little bit at the source files for boost::interprocess and it looks like there some facilities in http://svn.boost.org/svn/boost/trunk/boost/interprocess/detail/managed_memory_impl.hpp to grow() file sizes. The memory reserved by CreateFileMapping/MapViewOfFile? can't grow, but what if there were a separate "size" and "max_size" in http://svn.boost.org/svn/boost/trunk/boost/interprocess/mapped_region.hpp ? (which appears to be the class that calls MapViewOfFile? via map_view_of_file_ex()) "max_size" would represent the argument passed to CreateFileMapping? (with the SEC_RESERVE flag) / MapViewOfFile?, and "size" representing the portion of that memory allocated; calls to grow() would call VirtualAlloc?, not in a haphazard random-access per-page way, but merely to increase the size used by the shared memory segment, in a sequential, linear (or exponential if that makes sense) fashion.

comment:4 Changed 9 years ago by anonymous

Resolution: wontfix
Status: reopenedclosed

This feature won't be implemented because of complexity and time constraints.

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain Ion Gaztañaga.
The resolution will be deleted.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.