Ticket #9235 (closed Feature Requests: fixed)

Opened 3 years ago

Last modified 13 months ago

Construction from binary data

Reported by: psiha Owned by: johnmaddock
Milestone: To Be Determined Component: multiprecision
Version: Boost 1.54.0 Severity: Optimization
Keywords: Cc:

Description (last modified by psiha) (diff)

I didn't find a way to directly construct fixed sized cpp_int from existing binary data (a static array, for example an RSA key hardcoded into a .cpp after being generated with OpenSSL)... I had to resort to either using the from-string constructor or inserting the data byte by byte with 8 bit shifts and or-ing in a loop both of which are ugly and inefficient: ...

Ideally it would be possible to create static fixed size numbers as one creates PODs, without a dynamic initializer being created by the compiler..


Change History

comment:1 Changed 3 years ago by johnmaddock

This is supported.... but only in C++11 compliant compilers (ie not VC++ I'm afraid at present). See:

When supported though, the integer is compile-time constructed from the data with no runtime overhead at all.

Otherwise I'd strongly recommend you use the "construct from hexadecimal string" method, as that runtime code is well optimized (I hope!).

comment:2 Changed 3 years ago by johnmaddock

  • Status changed from new to closed
  • Resolution set to wontfix

comment:3 Changed 3 years ago by psiha

  • Description modified (diff)

comment:4 Changed 3 years ago by psiha

  • Status changed from closed to reopened
  • Resolution wontfix deleted

1) you somewhat contradict yourself here WRT to where you refer to contruction from a string as something non trivial

2) regardless, closing this as wontfix (instead of at least postponing until you or someone else finds the time) is lame IMO because this is certainly doable...'of the top of my head': you'd 'just' need to define an underlying POD which could be filled through a macro that'd:

  • fill the raw array that represents the number
  • set any aditional/auxiliary members you might have in number<>
  • define a fully typeed number<> & bound to the helper underlying POD instance...
  • "I know" it is doable because in my code (which I don't have with me now as I'm not at work) I had to do the opposite: retrieve the raw binary data from a cpp_int in order to compare it to an expected signature which I did by reinterpret_casting your number<> to a boost::array<> and calling std::reverse() on this is one more function missing from the interface (retreiving a numbers raw byte representation as opposed to a slow EH littered dynamically allocated std::string conversion)

3) the ticket isn't (only) about static strings but creation from binary data in general (which one may receive or generate at runtime, for example from a different big-int library) so a function that does this (what I currently do with the mentioned loop in the other ticket) is certainly needed

4 when one uses your lib for cryptography or hash/licence verification storing (private) keys as plain text hex strings is not an option

comment:5 follow-up: ↓ 6 Changed 3 years ago by johnmaddock

  • Type changed from Bugs to Feature Requests

It's not quite as trivial as you think: the internal data layout is highly platform/compiler dependent. For example a 128-bit integer is represented as 4 32 bit words on MSVC, but as a single 128-bit value on GCC-x64 (which has __int128).

Re (1), conversion from a hex string is fast, but decimal string conversion is heavyweight: of course they use the same routine though. Your example(4) is solved by user-defined literals I believe - it's the optimally efficient solution as it's all done at compile-time. For (3) would get_byte/set_byte non-member functions (similar to the existing bit twiddling functions) do the job? Note that these would only be efficient for unsigned types: signed types would require a check for negative values and the associated error handling - and yes that means exceptions.

comment:6 in reply to: ↑ 5 Changed 13 months ago by psiha

Replying to johnmaddock:

I've just noticed you've added import/export_bits() :) ...interface-wise this would be enough to close this ticket however I'm not fully happy with the implementation so let's not yet ;-D

It's not quite as trivial as you think: the internal data layout is highly platform/compiler dependent. For example a 128-bit integer is represented as 4 32 bit words on MSVC, but as a single 128-bit value on GCC-x64 (which has __int128).

This does/should not matter if the endianess of the machine and your internal implementation match, the byte layout will be the same. Except if the 'raw'/binary data is in a different endianess than your internal representation (modeled by the msv_first import/export_bits() parameter) but even then it is a matter of a simple byte reversal. Tried it with both importing and exporting with Boost 1.60 and 'it just works':

    template <class Key>
    void LE_NOTHROW load( Key & key, std::array<std::uint8_t, Constants::rsaKeySizeInBytes> const & keyBytes )
    #if 0 
        BOOST_VERIFY( &import_bits( key, keyBytes.begin(), keyBytes.end() ) == &key );
        key.backend().resize( key.backend().internal_limb_count, 0 );
        std::reverse_copy( keyBytes.begin(), keyBytes.end(), reinterpret_cast<std::uint8_t *>( key.backend().limbs() ) );

and looks trivial in the disassembly window ;)

ps. first I tried to force key (a fixed unchecked uint) limb count to max by constructing it as Key key( -1 ) but that didn't quite give a picturesque codegen ;D

Last edited 13 months ago by psiha (previous) (diff)

comment:7 Changed 13 months ago by johnmaddock

  • Status changed from reopened to closed
  • Resolution set to fixed

Please see

Note this is import only, which I think fixes your use case. Export via memcpy to an output iterator is unsafe and not implemented. I could provide a separate interface for that, but I'd like to see a genuine need for high-speed-export first.

In any case, use of user-defined-literals (which is all constexpr) is the better solution for your use case, although I accept that's not supported on msvc yet :(


Add a comment

Modify Ticket

Change Properties
<Author field>
as closed
The resolution will be deleted. Next status will be 'reopened'

E-mail address and user name can be saved in the Preferences.

Note: See TracTickets for help on using tickets.