Modify

Ticket #5230 (closed Feature Requests: fixed)

Opened 3 years ago

Last modified 2 years ago

Interprocess shared memory between 32 and 64 bit processes

Reported by: tgermer@… Owned by: igaztanaga
Milestone: To Be Determined Component: interprocess
Version: Boost Development Trunk Severity: Problem
Keywords: shared memory 32 64 bit Cc: aschoedl@…

Description

For sharing memory between 32 and 64 bit processes, all structures in shared memory must have the same binary layout with matching size_type and difference_type. In addition, offset_ptr must be 64bit in this case.

This is a patch that parameterizes offset_ptr to explicitly accept its difference_type along with its internal offset_type. This difference_type (and the according size_type) is propagated to all other affected structures (memory algorithms, allocators, containers, ...). In 32/64 bit shared environments, difference_type should be set to int and offset_type to unsigned __int64. The default values are ptrdiff_t and size_t.

Note: rbtree_best_fit also uses offset_type to store its internal block sizes, because these need to wrap-around correctly on 64bit machines.

Attachments

boost_interprocess_32_64bit_patch.zip Download (51.2 KB) - added by tgermer@… 3 years ago.
boost_r70306_aligned_storage.zip Download (507 bytes) - added by tgermer@… 3 years ago.
aligned-storage-patch

Change History

Changed 3 years ago by tgermer@…

comment:1 Changed 3 years ago by igaztanaga

Thanks, I've merged the patch with my latest local changes and I'm reviewing it. Regarding rbtree_best_fit's usage of offset_type, I think I will remove it. That might cause a binary incompatibility with earlier versions, but wrap-around logic is not needed to allocate memory correctly and increases bookeeping overhead for all allocations. I think I can easily remove this need in a day or two.

I hope I'll review the patch and push it into trunk this week. Best.

comment:2 Changed 3 years ago by tgermer@…

I'm sorry, I found a bug in my patch at the selector that gets the offset_type for rbtree_best_fit. (I forgot a template parameter.) This should read:

   template<class S, class T> 
   struct ptr_size_type_selector<boost::interprocess::offset_ptr<void, S, T> >
   {
	   typedef typename boost::interprocess::offset_ptr<void, S, T>::offset_type type;
   };

But this will get removed anyway, once we don't rely on wrap-around logic anymore...

Best, Tobias

comment:3 Changed 3 years ago by igaztanaga

Please check latest SVN trunk code. Download both Move and Interprocess (new Intrusive is optional). There is one thing left, though: I can't make boost::aligned_storage work with offset_ptr and boost 1.47 in msvc (some namespace clashing when compiling). I hope you can find a workaround faster than me. Let me know if this version makes 32-64 bit process communication possible.

comment:4 Changed 3 years ago by tgermer@…

Thank you for integrating the patch.

I also encountered this namespace conflict. It results from the ambiguity between "boost::detail" and "boost::interprocess::detail" when referring to "detail::" at the local namespace. I resolved this by declaring the absolute namespace path in aligned_storage.hpp. So every "detail::" becomes "::boost::detail::", which removes this ambiguity. This was also included in the patch. Is this change not feasible?

We need some time to integrate the new version into our framework and test it. I'll post if it worked.

comment:5 follow-up: ↓ 12 Changed 3 years ago by igaztanaga

Modifying aligned_storage is not in my hand, I think you should post this patch in the mailing list. If no one replies or has any solid arguments against the patch I will apply it.

comment:6 follow-up: ↓ 7 Changed 3 years ago by tgermer@…

OK, I just posted the patch to the mailing list (waiting for moderation).

I also updated our boost libraries and tested the new version in our 32-64 bit mixed environment, and it works like a charm :) Thanks again for your effort!

comment:7 in reply to: ↑ 6 ; follow-up: ↓ 8 Changed 3 years ago by xijing dai <dxj19831029@…>

Replying to tgermer@…:

OK, I just posted the patch to the mailing list (waiting for moderation).

I also updated our boost libraries and tested the new version in our 32-64 bit mixed environment, and it works like a charm :) Thanks again for your effort!

hey,

I am currently on version 1.46.1 I would like to get this patch and have a go, is that possible? where can i get the patch? do i need to upgrade all my boost libs to trunk head in order to use it?

Cheers

comment:8 in reply to: ↑ 7 ; follow-up: ↓ 9 Changed 3 years ago by tgermer@…

Replying to xijing dai <dxj19831029@…>:

I am currently on version 1.46.1 I would like to get this patch and have a go, is that possible? where can i get the patch? do i need to upgrade all my boost libs to trunk head in order to use it?

I'm also on 1.46.1. You just have to update boost/interprocess, boost/intrusive and boost/move (which is new) to the trunk version in order to use it. No need to update anything else yet. Currently there is one open issue to make it work: the namespace conflict. If you want to resolve it manually, you need to remove the comment from "union internal_type" in boost/interprocess/offset_ptr.hpp. Then you have to replace every "detail::" with "::boost::detail::" in aligned_storage.hpp (the patch that I suggested in comment 4 and also on the mailing list).

comment:9 in reply to: ↑ 8 ; follow-up: ↓ 10 Changed 3 years ago by xijing dai <dxj19831029@…>

Replying to tgermer@…:

Replying to xijing dai <dxj19831029@…>:

I am currently on version 1.46.1 I would like to get this patch and have a go, is that possible? where can i get the patch? do i need to upgrade all my boost libs to trunk head in order to use it?

I'm also on 1.46.1. You just have to update boost/interprocess, boost/intrusive and boost/move (which is new) to the trunk version in order to use it. No need to update anything else yet. Currently there is one open issue to make it work: the namespace conflict. If you want to resolve it manually, you need to remove the comment from "union internal_type" in boost/interprocess/offset_ptr.hpp. Then you have to replace every "detail::" with "::boost::detail::" in aligned_storage.hpp (the patch that I suggested in comment 4 and also on the mailing list).

First all, thanks to this patch, whoever did it, I really appreciate their work, it helps me a lot.

I tested in my application, it works great. testing includes:

  • windows_shared_memory
  • message queue
  • managed_windows_shared_memory

create named item raw allocate

By the way, I didn't change anything relating to the namespace issue, it passed compiling and worked great.

comment:10 in reply to: ↑ 9 ; follow-up: ↓ 11 Changed 3 years ago by xijing dai <dxj19831029@…>

Replying to xijing dai <dxj19831029@…>:

Replying to tgermer@…:

Replying to xijing dai <dxj19831029@…>:

I am currently on version 1.46.1 I would like to get this patch and have a go, is that possible? where can i get the patch? do i need to upgrade all my boost libs to trunk head in order to use it?

I'm also on 1.46.1. You just have to update boost/interprocess, boost/intrusive and boost/move (which is new) to the trunk version in order to use it. No need to update anything else yet. Currently there is one open issue to make it work: the namespace conflict. If you want to resolve it manually, you need to remove the comment from "union internal_type" in boost/interprocess/offset_ptr.hpp. Then you have to replace every "detail::" with "::boost::detail::" in aligned_storage.hpp (the patch that I suggested in comment 4 and also on the mailing list).

First all, thanks to this patch, whoever did it, I really appreciate their work, it helps me a lot.

I tested in my application, it works great. testing includes:

  • windows_shared_memory
  • message queue
  • managed_windows_shared_memory

create named item raw allocate

By the way, I didn't change anything relating to the namespace issue, it passed compiling and worked great.

Interesting, I am not sure if I had wrong testing last week, or I did something wrong this week.

This week, the program doesn't work properly between 32/64 managed_windows_shared_memory

I do a little trace, when i allocate 1000000 bytes, it said free memory is: 999912, in 32bit. in 64 bit, it is: 999848.

For me, it looks like something wrong in: rbtree_best_fit, since when i trace from get_free_memory method, it gives different values, so I assume there is some setting wrong for that class.

by the way, I do uncomment union internal_type section, and I didn't change "detail::" stuff inside aligned_storage.hpp, is that a problem?

Do you have any idea what's going on here?

Cheers

comment:11 in reply to: ↑ 10 Changed 3 years ago by tgermer@…

Replying to xijing dai <dxj19831029@…>:

by the way, I do uncomment union internal_type section, and I didn't change "detail::" stuff inside aligned_storage.hpp, is that a problem?

No, if your compiler doesn't produce an error, then this should be OK.

Do you have any idea what's going on here?

Not really. Certainly Ion knows the details better and can tell what's going on. You could also look into raw memory and watch for inconsistencies across 32/64-bit builds.

Changed 3 years ago by tgermer@…

aligned-storage-patch

comment:12 in reply to: ↑ 5 Changed 3 years ago by tgermer@…

Replying to igaztanaga:

Modifying aligned_storage is not in my hand, I think you should post this patch in the mailing list. If no one replies or has any solid arguments against the patch I will apply it.

@Ion: The guys on the mailing list just found an error with a digraph that I mistankingly had in the patch. In the last couple of days, there were no further replies. I think you could apply the patch.

I added the patch as an attachment to this ticket. Interestingly, trac doesn't accept the patch as plain text and thinks it is spam :)

comment:13 follow-ups: ↓ 14 ↓ 15 Changed 3 years ago by xijing dai <dxj19831029@…>

here is the code:

this->memorySegment = new boost::interprocess::managed_windows_shared_memory( boost::interprocess::open_or_create, name, size);

int sizeB = this->memorySegment->get_free_memory();

i allocate 1000000 bytes, it said free memory is: 999912, in 32bit. in 64 bit, it is: 999848.

Just wonder, do I need to use different namespace or class name in order for managed shared memory to be compatible for 32 and 64 system?

emm...by the way, how can i reach Ion?

Cheers

comment:14 in reply to: ↑ 13 Changed 3 years ago by tgermer@…

Replying to xijing dai <dxj19831029@…>:

Just wonder, do I need to use different namespace or class name in order for managed shared memory to be compatible for 32 and 64 system?

No. In our case, the code for the 32 and 64 bit processes is actually identical.

emm...by the way, how can i reach Ion?

Well, I think he follows this conversation :)

comment:15 in reply to: ↑ 13 ; follow-ups: ↓ 16 ↓ 17 Changed 3 years ago by igaztanaga

Replying to xijing dai <dxj19831029@…>:

here is the code:

The patch is not about making all interprocess classes workable between 32 and 64 bit applications. Is about customizing offset_ptr and propagate its attributes (size_type, difference_type, etc..) so that that this *could* be possible, of course depending on the compiler ABI of 32/64 bit applications. The patch is experimental, so it won't be documented and made official until it gets some real-application experience.

Regarding message_queue, it is a bug that it does not work between 32 and 64 bit applications. I should change queue internals to make internal types fixed length. It's in my to-do list but i haven't found time to implement it.

comment:16 in reply to: ↑ 15 Changed 3 years ago by xijing dai <dxj19831029@…>

Replying to igaztanaga:

Replying to xijing dai <dxj19831029@…>:

here is the code:

The patch is not about making all interprocess classes workable between 32 and 64 bit applications. Is about customizing offset_ptr and propagate its attributes (size_type, difference_type, etc..) so that that this *could* be possible, of course depending on the compiler ABI of 32/64 bit applications. The patch is experimental, so it won't be documented and made official until it gets some real-application experience.

Regarding message_queue, it is a bug that it does not work between 32 and 64 bit applications. I should change queue internals to make internal types fixed length. It's in my to-do list but i haven't found time to implement it.

got u!

which class have u tested? let me check if I can replace with them.

otherwise, I am not sure if we could wait for next boost release, I need to think some backup plans. :)

comment:17 in reply to: ↑ 15 Changed 3 years ago by xijing dai <dxj19831029@…>

Replying to igaztanaga:

Replying to xijing dai <dxj19831029@…>:

here is the code:

The patch is not about making all interprocess classes workable between 32 and 64 bit applications. Is about customizing offset_ptr and propagate its attributes (size_type, difference_type, etc..) so that that this *could* be possible, of course depending on the compiler ABI of 32/64 bit applications. The patch is experimental, so it won't be documented and made official until it gets some real-application experience.

Regarding message_queue, it is a bug that it does not work between 32 and 64 bit applications. I should change queue internals to make internal types fixed length. It's in my to-do list but i haven't found time to implement it.

boost::interprocess::offset_ptr<void> is 4 bytes long, is that normal for 32 program? should it be 8 bytes?

comment:18 Changed 3 years ago by anonymous

offset_ptr uses its template parameter OffsetType for its internal representation. That means it depends on you how large it is. Per default, OffsetType=size_t, which is 4 bytes for 32 bit programs. In 32/64-bit mixed applications, you should indeed take an 8-byte offset_ptr, for example: offset_ptr<void, int, unsigned long long>

comment:19 follow-up: ↓ 20 Changed 3 years ago by igaztanaga

  • Status changed from new to closed
  • Resolution set to fixed

Fixed for Boost 1.47

comment:20 in reply to: ↑ 19 Changed 2 years ago by anonymous

Replying to igaztanaga:

Fixed for Boost 1.47

According to the 1.48.0 release notes, this bug fix went into 1.48.0, not 1.47. (See http://www.boost.org/users/history/version_1_48_0.html )

View

Add a comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
The resolution will be deleted. Next status will be 'reopened'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.