Skip Menu |
 

This queue is for tickets about the Bio-SamTools CPAN distribution.

Report information
The Basics
Id: 116683
Status: new
Priority: 0/
Queue: Bio-SamTools

People
Owner: Nobody in particular
Requestors: kevin.buckley.ecs.vuw.ac.nz [...] gmail.com
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)



Subject: Bio-SamTools-1.43 is confused about what various SAMtools build dirs supply where - and so am I
Date: Wed, 3 Aug 2016 17:39:59 +1200
To: bug-Bio-SamTools [...] rt.cpan.org
From: Kevin Buckley <kevin.buckley.ecs.vuw.ac.nz [...] gmail.com>
Download (untitled) / with headers
text/plain 2.8k
Came to add Bio-SamTools-1.43 into a BioPerl installtion cpan[1]> install LDS/Bio-SamTools-1.43.tar.gz I get told that Show quoted text
> This module requires samtools 0.1.10 or higher (samtools.sourceforge.net). > Please enter the location of the bam.h and compiled libbam.a files:
so I supply the path to the directory in which I built SAMtools 1.3 path/to/my/samtools-1.3/ The Bio-SamTools-1.43 build then fails telling me that In file included from c_bin/bam2bedgraph.c:2: /path/to/my/samtools-1.3/sam.h:28:24: \ error: htslib/sam.h: No such file or directory and that's right, because samtools-1.3/ has an htslib-1.3/ subdirectory and not a plain, unversioned htslib/ subdirectory . Furthermore, inside the htslib-1.3 subdirectory there is no file sam.h but there is a file named sam.h in the top-level samtools-1.3 directory. I therefore create a link htslib -> htslib-1.3/ and within htslib-1.3, another link sam.h -> ../sam.h and the build gets further, before failing with this one /path/to/my/samtools-1.3/bam.h:48:25: error: htslib/bgzf.h: No such file or directory and that's where I run out of ideas, because that header (* see below) doesn't exist anywhere in samtools-1.3, let alone where Bio-SamTools seems to think it should be. I also tried looking in SAMtools 0.1.19 (after all there are a multitude of SAM/BAM/THANKYOU/MAM/tools out there in BioInfo-land) but that doesn't have an htslib subdirectory of any kind. * But read on I become aware that although there is no header file, bgzf.h, in samtools 1.3 there is a bgzf.c inside the htslib-1.3/ subdirectory and I see that that #includes htslib/bgzf.h at which point I realise that the htslib-1.3/ subdirectory has it's own (unversioned) htslib subdirectory (or subsubdirectory of samtools 1.3 if you prefer) and inside there I find all of these fine fellows bgzf.h hts.h klist.h regisubdirectory (or subsubdirectory dx.h vcf_sweep.h cram.h kbitset.h knetfile.h sam.h vcfutils.h faidx.h kfunc.h kseq.h synced_bcf_reader.h hfile.h khash.h ksort.h tbx.h hts_defs.h khash_str2int.h kstring.h vcf.h so now I am assuming that subdirectory. or subsubdirectory, depending where you start from is what Bio-SamTools is probably looking for, so I undo all my previous attempts to trick Bio-SamTools into finding what it wants to find by creating a link from the top-level samtools-1.3/ directory, where you'll recall, libbam.a is, as htslib -> htslib-1.3/htslib/ And, after all that, Bio-SamTools tells me lib/Bio/DB/Sam.xs:29:19: error: khash.h: No such file or directory but, as we saw above khash.h: is there for all to see. I know I must have missed something obvious: but what ? Any clues welcome. Kevin M. Buckley eScience Consultant School of Engineering and Computer Science Victoria University of Wellington New Zealand
Subject: Re: [rt.cpan.org #116683] Bio-SamTools-1.43 is confused about what various SAMtools build dirs supply where - and so am I
Date: Thu, 4 Aug 2016 11:47:27 +1200
To: bug-Bio-SamTools [...] rt.cpan.org
From: Kevin Buckley <kevin.buckley.ecs.vuw.ac.nz [...] gmail.com>
Got a bit further If I link three flles from within the htslib-1.3/htslib/ directory up into the top build directpory for samtools-1.3 $ ln -s htslib-1.3/htslib/khash.h . $ ln -s htslib-1.3/htslib/faidx.h . $ ln -s htslib-1.3/htslib/hts_defs.h . then I get to defeat the fact that the way the #includes are defined differently within lib/Bio/DB/Sam.xs #include "bam.h" #include "khash.h" #include "faidx.h" and in the samtools-1.3/sam.h #include "htslib/sam.h" and i ovecome the fact that the Bio-SamTools Build.PL doesn't generate a correct set of -I/path/to/include/dirs. However, I now see that there's probably some even greater mis-match between what Bio-SamTools-1.43 and samtools-1.3 think about how the internal structures are defined: lib/Bio/DB/Sam.xs: In function 'XS_Bio__DB__Bam__Alignment_l_aux': lib/Bio/DB/Sam.xs:575: error: 'bam1_t' has no member named 'l_aux' lib/Bio/DB/Sam.xs:576: error: 'bam1_t' has no member named 'l_aux' Has anyone out there compiled Bio-SamTools-1.43 against a SMAtools version and, if so, which version ?
Subject: [rt.cpan.org #116683] Bio-SamTools-1.43 is confused about what various SAMtools build dirs supply where - and so am I
Date: Thu, 4 Aug 2016 14:40:06 +1200
To: bug-Bio-SamTools [...] rt.cpan.org
From: Kevin Buckley <kevin.buckley.ecs.vuw.ac.nz [...] gmail.com>
Download (untitled) / with headers
text/plain 1.1k
It appears I have posted this reply with out the RT bit in the subect line and started another issue - apologies. To save anyone merging the other strand, here you go again Bit more info gleaned about this In the bam.h file that's in the top build directory for samtools-1.3, I read ---8<---------8<---------8<---------8<---------8<---------8<---------8<---------8<---------8<---- @discussion Notes: 1. qname is zero tailing and core.l_qname includes the tailing '\0'. 2. l_qseq is calculated from the total length of an alignment block on reading or from CIGAR. 3. cigar data is encoded 4 bytes per CIGAR operation. 4. seq is nybble-encoded according to bam_nt16_table. */ // typedef struct { ... } bam1_t; // NOTE htslib version doesn't have l_aux; use bam_get_l_aux(b) instead #ifndef SAMTOOLS_HTSLIB_SUPPRESS_HACKS // NOTE htslib also renames data_len to l_data; this macro may help or hinder ---8<---------8<---------8<---------8<---------8<---------8<---------8<---------8<---------8<---- so the missing member really isn't there by design. So which version of SAMtools should Bio-SamTools-1.43 be compiled against ?
Subject: [rt.cpan.org #116683] Bio-SamTools-1.43 is confused about what various SAMtools build dirs supply where - and so am I
Date: Thu, 11 Aug 2016 15:37:30 +1200
To: bug-Bio-SamTools [...] rt.cpan.org
From: Kevin Buckley <kevin.buckley.ecs.vuw.ac.nz [...] gmail.com>
Download (untitled) / with headers
text/plain 6.2k
I do hope that someone is using this software to study worms, as this issue seems to have been one massive can of them, however I think I have accounted for all of them now. Based on what I'd seen so far, I decided to go looking in the past versions of SAMtools, in case it was possible to determine which one of them might match the perculiar layout of header files that Bio-SamTools clearly expected to be there in the SAMtools it was pointed to. Here's what I found, amongst the ones I had lying around from previous encounters into the SAMtools universe SAMtools 1.2 * doesn't have a plain htslib subdir but htslib-1.2.1 * bam.h has the note about htslib version doesn't have l_aux SAMtools 1.1 (Github doesn't bundle htslib) * doesn't have an htslib subdir AT ALL * bam.h has the note about htslib version doesn't have l_aux * Doesn't build: Makefile:86: ../htslib/htslib.mk: No such file or directory SAMtools 1.1 (Sourceforge) * doesn't have a plain htslib subdir but htslib-1.1 * bam.h has the note about htslib version doesn't have l_aux SAMtools 1.0 (Sourceforge) * doesn't have a plain htslib subdir but htslib-1.0 * bam.h has the note about htslib version doesn't have l_aux SAMtools 0.1.19 (Sourceforge) * doesn't have an htslib subdir AT ALL * bam.h DOES HAVE a bam1_t type which has an l_aux * Might have all the required header files So now I go and build a SAMtools 0.1.19 and point Bio-SamTools at that and get a bit further, but it fails trying to do this gcc -shared -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions \ -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic \ -o blib/arch/auto/Bio/DB/Sam/Sam.so lib/Bio/DB/Sam.o c_bin/bam2bedgraph.o \ -L/path/to/samtools-0.1.19 -lbam \ -lpthread -lz /usr/bin/ld: /path/to/samtools-0.1.19/libbam.a(bgzf.o): \ relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making \ a shared object; recompile with -fPIC /path/to/samtools-0.1.19/libbam.a: could not read symbols: Bad value However, that -lbam in the compilation looked to me as though it was trying to link in a dynamic library and not the static one it was pointed to as part of the configuration - surely not ? So I thought to go and create a libbam.so dynamic library, using all of the object files that SAMtools 0.1.19 produces in creating the static library. After the main build doing its: ar -csru libbam.a bgzf.o kstring.o bam_aux.o bam.o bam_import.o sam.o \ bam_index.o bam_pileup.o bam_lpileup.o bam_md.o razf.o faidx.o bedidx.o \ knetfile.o bam_sort.o sam_header.o bam_reheader.o kprobaln.o bam_cat.o I go and do a gcc -shared -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions \ -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic \ -o libbam.so \ bgzf.o kstring.o bam_aux.o bam.o bam_import.o sam.o \ bam_index.o bam_pileup.o bam_lpileup.o bam_md.o razf.o faidx.o bedidx.o \ knetfile.o bam_sort.o sam_header.o bam_reheader.o kprobaln.o bam_cat.o and I find I can get the Bio-SamTools module to build, however it fails to run any of its tests, notionally because the libbam.so isn't available to it at run time. I say "notionally" because it was that point, having seemingly built the Bio-SamTools module but failed the tests, that I decided to take a look at the Bio-SamTools test source lying around within the Perl build directorires, and what do you think I saw ... There's a README File and it says, right at the top, ---8<------8<------8<------8<------8<------8<------8<------8<------8<--- This is a Perl interface to the SAMtools sequence alignment interface. It ONLY works on versions of Samtools up to 0.1.17. It does not work on version 1.0 or higher due to major changes in the library structure. ---8<------8<------8<------8<------8<------8<------8<------8<------8<--- so there's the answer to my question from the previous posts: I should be building against 0.1.17. Seems a shame that no-one could be bothered to tell me that but hey-ho, no matter. Armed with that knowledge and the knowledge that Bio-SamTools is going to try and link against a non-existant dynamic libray that I'll need to create for myself, I added, to my list of potential SAMtools versions SAMtools 0.1.17 (Sourceforge) * doesn't have an htslib subdir AT ALL * bam.h DOES HAVE a bam1_t type which has an l_aux * Might have all the required header files and proceed as follows, noting that I'll still need to create a link to get around the way that the htslib headers are referred to in the various parts of Bio-SamTools tar xf amtools-0.1.17.tar.bz2 cd samtools-0.1.17 cp Makefile Makefile.orig vim Makefile make ln -s . htslib where the Makefile gets altered as follows 2c2 < CFLAGS= -g -Wall -O2 #-m64 #-arch ppc --- Show quoted text
> CFLAGS= -g -Wall -O2 -fPIC #-m64 #-arch ppc
I then build the shared library as shown above, adding in the -fPIC, and make it available, to the yet to be built Bio-SamTools module, by poking it into LD_LIBRARY_PATH export LD_LIBRARY_PATH=/path/to/samtools-0.1.17 The build of Bio-SamTools module no runs through and we see the following files being installed created into the Perl heirachy (paths truncated to avoid wrapping) Installing .../lib/perl5/x86_64-linux-thread-multi/auto/Bio/DB/Sam/Sam.so Installing .../lib/perl5/x86_64-linux-thread-multi/auto/Bio/DB/Sam/Sam.bs Installing .../bin/bam2bedgraph Installing .../lib/perl5/x86_64-linux-thread-multi/Bio/DB/Sam.pm Installing .../lib/perl5/x86_64-linux-thread-multi/Bio/DB/Sam/Segment.pm Installing .../lib/perl5/x86_64-linux-thread-multi/Bio/DB/Sam/Constants.pm This (OK, make that "All this") seems to have done the trick. I hope documenting this might be useful to someone, although, given the lack of feedback, it's quite possible that no-one, bar the user I was looking to install it for on my resources, is even using it anymore? Footnote: I suppose it's possible that, even though the README says that it only works with 0.1.17, there's clearly enough similarity between 0.1.17 and 0.1.19 failing to build that Bio-SamTools-1.43 might even compile aganst that later version, as long as the hoop-jumping above id performed, however I'm quitting whilst I am ahead, and will leave that as an exercise for the reader, should this thread ever have one, I guess.


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.