Modify

Opened 6 years ago

Closed 7 months ago

Last modified 6 months ago

#5908 closed Bugs (fixed)

iostreams gzip fails to handle optional extra fields in gzip header

Reported by: Travis Abbott <typedef.struct@…> Owned by: Jonathan Turkanis
Milestone: To Be Determined Component: iostreams
Version: Boost 1.47.0 Severity: Problem
Keywords: gzip Cc:

Description

When bit 2 (FEXTRA) is set in a gzip header's flags, the iostreams code fails to read the XLEN field before starting to read the extra comment. The code is actually there to do it, but it gets skipped. This means the code goes directly a loop like: while (--xlen != 0) with xlen still set to 0. This results in the rest of the file being slurped in by this comment reading code (or at least until xlen wraps around to 0 again, which could take awhile). I ran into this because many popular file formats in bioinformatics (BAM, tabix) are gzipped and include extra optional fields in their headers.

I've attached an example gzipped file with an optional header, a test program that should demonstrate the problem (against 1.47 and latest svn), as well as a patch that fixes it.

Attachments (4)

sample.txt.gz (68 bytes) - added by Travis Abbott <typedef.struct@…> 6 years ago.
gzip file with extra comment in header
example.cpp (733 bytes) - added by Travis Abbott <typedef.struct@…> 6 years ago.
test case that demonstrates the problem (run ./example sample.txt.gz)
iostreams-gzip.patch (493 bytes) - added by Travis Abbott <typedef.struct@…> 6 years ago.
patch that fixes the issue (patch -p0 < iostreams-gzip.patch from top level of the repo)
iostreams-gzip_hdr_test.patch (2.4 KB) - added by typedef.struct@… 6 years ago.
second patch adding unit test to prevent future regressions.

Download all attachments as: .zip

Change History (12)

Changed 6 years ago by Travis Abbott <typedef.struct@…>

Attachment: sample.txt.gz added

gzip file with extra comment in header

Changed 6 years ago by Travis Abbott <typedef.struct@…>

Attachment: example.cpp added

test case that demonstrates the problem (run ./example sample.txt.gz)

Changed 6 years ago by Travis Abbott <typedef.struct@…>

Attachment: iostreams-gzip.patch added

patch that fixes the issue (patch -p0 < iostreams-gzip.patch from top level of the repo)

comment:1 Changed 6 years ago by typedef.struct@…

Seems like this is still present in 1.48 and current SVN. To be clear, the sample file is not empty. The desired output would be for the test program to read "hello there", but it gets nothing. You can compare to gzip -dc. The supplied patch still works with current SVN.

comment:2 Changed 6 years ago by typedef.struct@…

Also worth mentioning is that this worked in 1.40, but when the ability to support multiple compressed objects in 1 gzipped stream was introduced, this bug appeared.

Changed 6 years ago by typedef.struct@…

second patch adding unit test to prevent future regressions.

comment:3 Changed 6 years ago by typedef.struct@…

The test attached in the second patch (iostreams-gzip_hdr_test.patch) will fail with the current code, demonstrating the inability to parse certain types of RFC 1952 compliant gzip headers. Application of the first patch will cause the test to pass.

comment:4 Changed 6 years ago by Jonathan Turkanis

Status: newassigned

I have applied the patches to trunk.

comment:5 Changed 5 years ago by Daniel James

Resolution: fixed
Status: assignedclosed

(In [77368]) Iostreams: Merge from trunk.

comment:6 Changed 7 months ago by Smithc195

Component: iostreamsxpressive
Milestone: To Be DeterminedWebsite 1.X
Resolution: fixed
Severity: ProblemNot Applicable
Status: closedreopened
Type: BugsLibrary Submissions
Version: Boost 1.47.0Boost.Build-M3

Hi. Only wanted to ask a quick issue. Now i am eaadkebdkckdgfgd

comment:7 Changed 7 months ago by Daniel James

Resolution: fixed
Status: reopenedclosed

comment:8 Changed 6 months ago by Kohei Takahashi

Component: xpressiveiostreams
Milestone: Website 1.XTo Be Determined
Severity: Not ApplicableProblem
Type: Library SubmissionsBugs
Version: Boost.Build-M3Boost 1.47.0

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain Jonathan Turkanis.
The resolution will be deleted.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.