Bug 15327 - liblzma-dev is required for --enable-R-shlib
Summary: liblzma-dev is required for --enable-R-shlib
Alias: None
Product: R
Classification: Unclassified
Component: Installation (show other bugs)
Version: R 3.0.1
Hardware: x86_64/x64/amd64 (64-bit) Linux
: P5 enhancement
Assignee: R-core
Depends on:
Reported: 2013-05-29 23:07 UTC by Sebastian Meyer
Modified: 2017-03-16 23:25 UTC (History)
4 users (show)

See Also:

Patch to fix the loading issue (1.73 KB, patch)
2013-10-14 10:21 UTC, gordon.m.lack
Details | Diff
adding missing lzma dependency to link (702 bytes, patch)
2014-06-12 14:16 UTC, Marco Atzeri
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Sebastian Meyer 2013-05-29 23:07:59 UTC
I'd like to apologize right at the beginning that I don't know if there is actually room for improvement in the following, but I found it slightly inconvenient so I just report what I observed:

I downloaded the sources of R-3.0.1 and ran
./configure --disable-nls --enable-R-shlib
(the crucial option is enable-R-shlib here)
The configure script went through until the end (showing the summary) and thus I continued with make. But "make" stopped with error:

make[4]: Entering directory `/opt/R/R-3.0.1/src/library/stats'
byte-compiling package 'stats'
Error in dyn.load(file, DLLpath = DLLpath, ...) : 
  unable to load shared object '/opt/R/R-3.0.1/library/utils/libs/utils.so':
  /opt/R/R-3.0.1/library/utils/libs/utils.so: undefined symbol: lzma_crc64
Calls: <Anonymous> ... asNamespace -> loadNamespace -> library.dynam -> dyn.load
Execution halted

It turned out that I was simply missing the "liblzma-dev" package on my system. With that installed everything worked fine.
However, note that with ./configure --disable-nls, i.e. if I did not require R shared libraries, "make" worked also without installed liblzma-dev.

I wonder if already the configure script could have visibly told me that I was missing this package on my system by stopping with error when checking for it.
Actually there was some error with lzma I saw later in config.log, but I didn't recognize during runtime of configure (lines pass very fast, and it exited normally).

(So that's it and I'm sorry if what I reported is useless.)
Comment 1 Brian Ripley 2013-06-03 18:03:20 UTC
I really don't understand this, and cannot reproduce it.  I have my Fedora boxes set up that way, and configure correctly uses the internal liblzma in the R sources.  utils.so has an undefined symbol lzma_crc64, is linked against libR.so and that provides the symbol.

There are far too many possibilities as to what is wrong on your system, but it is not that configure is not telling you that you are missing a package which you should not need.  (It is an Ubuntu-ism that it is called liblzma-dev: the recommended software is part of xz.)

If the internal xz is not being used, you can force it by

Comment 2 Sebastian Meyer 2013-06-04 00:09:05 UTC
I reproduced the reported make failure on another machine running Ubuntu 13.04.

./configure --disable-nls --enable-R-shlib --with-system-xz=no
'make' led to the same error as before.

Again, both not requesting R shared libraries or the installation of the liblzma-dev package (in addition to the already installed liblzma5 package and using the default --with-system-xz=yes) solved the problem.
Note that only after installation of liblzma-dev (and default --with-system-xz), "lzma" is listed under "External libraries" in the configuration summary.

Unfortunately I have no idea how I could help here other than by reporting or running given tests. Maybe it's really an ubuntu-specific issue.
Comment 3 gordon.m.lack 2013-10-10 13:21:00 UTC
I'm reopen this, rather than starting a new report, to keep the info together,.

I *can* reproduce this (hence the WORKSFORME status), and this is on a RedHat5 system (so is not specific to Ubuntu).

[Also I have the problem at 3.0.2, which isn't a version option on the form.]

And I can explain what is happening.

I can't supply a fix, though.

The problem occurs when you specify:
   --enable-R-shlib --with-system-xz=no
in the options.  (These are not default settings).

This is asking that the xz code internal to R (in src/extra) is used.  That is compiled and linked in to libR.so

But *ALL* of the bits under src/extra are *compiled* with an option of -fvisibility=hidden, so none of the symbols in that code will be visible outside of libR.so.

The "utils" package calls lzma_crc64() [utils.c:65], but that symbol is now hidden within libR.so and so it can't find it.

If the R executable is built without the shared library then the symbol will be within the R executable, and hence visible to R.

===== Path to Solutions....

It seems clear, from the fact that all of the src/extra code is compiled with hidden symbols, that the intention is to keep these internal to the core R code.
So the utils package should not be calling an entry in liblzma directly.
It's noticeable that unzip is handled by a call to Runzip.  Perhaps something similar needs to be done here?

HOWEVER, I also see that the call to lzma_crc64() is in a function called crc64().
It turns out that this code is a DIRECT COPY(!!) of the do_crc64() function from src/main./util.c (i.e. part of core R).  And that function is specifically marked with attribute_hidden.

So is this (possibly?, hopefully?) a case of the utils package defining (historically?) something which is actually already (now?) in cor R, and that *removing* it from the utils package would fix things?
Comment 4 gordon.m.lack 2013-10-10 13:33:04 UTC
>> So is this (possibly?, hopefully?) a case of the utils package defining
>> (historically?) something which is actually already (now?) in cor R, and that
>> *removing* it from the utils package would fix things?

Actually, it looks more as though the utils package is trying to provide a crc64 command and, having found the entry in the core util code hidden, took a copy of it realizing the lzma_crc64 symbols would also be hidden with a shared libR.

So, what about:
Not hiding do_crc64() in main/util.c and changing the crc64() code in the utils package code (in utils.c) to call do_crc64?
Comment 5 gordon.m.lack 2013-10-14 10:21:17 UTC
Created attachment 1493 [details]
Patch to fix the loading issue

I created an account an hour or so ago to update this ticket.  Then the account disappeared, and so did all of the ticket updates.
So I started again.  But with more info anyway.
And then, over the weekend, the original posts re-appeared and the updated info(and patch) disappeared.
So I'm adding the patch again.

I can reproduce this on a Redhat5 system (so it's nothing to do with Ubuntu) for R-3.0.2 (a version that is not a choice).

It is caused by using the options:
   --enable-R-shlib --with-system-xz=no
when configuring.
This causes the xz (lzma) code under src/extra to be used but *ALL* code under src/extra is compiled with the -fvisibility=hidden option, so any symbols defined there are not visible outside of the object they are linked into.
With a non-shared lib, all of the code gets loaded into the R executable, so the symbols are visible to the shared libs of extensions but when you use a shared libR then the src/extra parts get linked into libR.so, not R, so are *not* visible to other dynamically loaded libraries (such as the utils.so which reports this error).  This causes the undefined symbol message.

Looking at the code I see that the call to lzma_crc64() is in a crc64() function in library/utils/src/utils.c. The code for this is almost identical to that inthe do_crc64() function defined in main/util.c (the calling parameters differ).HOWEVER - that function is *also* defined as hidden, so it looks like someone has copied the code into the utils package to overcome this "hiddenness" without realizing that there would be a similar issue if the internal xz code were used in a shared libR.

Looking at how the unzip code is handled nearby I reckon what is needed is an unhidden entry in main/util.c that can be called from library/utils/src/utils.c, so that it never needs to find lzma_crc64.

So I've written a patch (against 3.0.2) which does:

   Add a new (non-hidden) function do_R_crc64(SEXP in) which contains most of the code
   Changes do_crc64 (the hidden function) to handle the old parameter list and get the needed "SEXP in", which it then passes on to do_R_crc64()

   Change the crc64() function (and lzma.h include) to just be a call to do_r_crc64().

The resulting patched code compiles and links OK with a shared libR and internal xz.
It passes "make check", although I have no idea whether any of the check would actually test anything that this code change actually touches.

If anyone has an example of some (simple) code that can be run to check this I can test it out.
Comment 6 Daniel Macks 2014-04-29 17:13:40 UTC
This sounds similar to Bug #15415; in both cases, the utils module is using lzma directly but not directly linking against it. If I understand, the assumption is that the module links libR, which links/contains other code, so the symbols are resolved indirectly? Or else the module is loaded by a library/executable that links/contains other code, so the loaded module can see the symbols in the parent's binary image? But now there are two cases where that fails for various reasons. There is a configure.ac determination of $dylib_undefined_allowed, which is used to control in various Makefile.in whether libintl is linked directly. It sounds like liblzma would be another case of this same concept. What are the advantages of *not* linking directly to libraries that supply directly-used symbols?

FWIW, I can reproduce the situation on darwin with external liblzma by removing "-undefined dynamic_lookup" from $shlib_ldflags in configure.ac. In concept, that flag contradicts the dylib_undefined_allowed=no setting that configure.ac uses for this same platform.
Comment 7 Brian Ripley 2014-05-27 09:55:08 UTC
I have finally managed to reproduce this.  The problem was that some OSes (but not x86_64 Linux for me, in none of Fedora 17, 18, 20) were not observing the visibility override in LZMA_API.  That has been removed and replaced by a more explicit one which we know to work elsewhere.
Comment 8 Marco Atzeri 2014-06-12 14:16:08 UTC
Created attachment 1609 [details]
adding missing lzma dependency to link
Comment 9 Marco Atzeri 2014-06-12 14:18:54 UTC
(In reply to Daniel Macks from comment #6)
> This sounds similar to Bug #15415; in both cases, the utils module is 
> using lzma directly but not directly linking against it.

attached the workaround patch used to directly link with system lzma/xz on 

Likely there are other ways, but this seems the less intrusive as patch.
Comment 10 Sebastian Meyer 2014-06-12 22:30:52 UTC
This is just to confirm that the initially described bug no longer occurs on my system with a current R-devel (2014-06-11 r65921).
Note that for R-3.1.0 the bug is still reproducible (Brian Ripley's fix came afterwards).
Comment 11 Erik Bray 2017-03-16 23:25:55 UTC
I know Cygwin isn't supported, but for what it's worth the patch suggested by Marco works for me to get past this issue when building on Cygwin.  I don't see what harm it is to include all required -l flags from $(LIBS) when linking.