[polyml] LInkage and the Foreign structure

Discussion:

Mark Clements

2018-09-09 20:15:26 UTC

I am enjoying using polyml's Foreign structure.

However, in using the GNU Scientific Library (GSL), I have run into a
problem with linkage. The GSL library (libgsl.so) calls standard cBLAS
matrix functions with unknown symbols to allow the user to link with
different cBLAS implementations (NB: GSL provides an unoptimised cBLAS
implementation named libgslcblas.so). When I call a GSL function that
depends on the cBLAS functions, I get a symbol not defined error.

I have tried unsuccessfully (a) loading the libgslcblas library either
before or after loading libgsl, (b) changing the loadLibrary
implementation to use dlopen with RTLD_LAZY | RTLD_GLOBAL, and (c)
compiling the SML code to an object file and then linking against the
libraries.

How could this be made to work?

Sincerely, Mark.

PS test code for (a) is:

open Foreign;
val gsl = loadLibrary "/usr/lib/x86_64-linux-gnu/libgsl.so";
val gslcblas = loadLibrary "/usr/lib/x86_64-linux-gnu/libgslcblas.so";
type gsl_matrix_t = Memory.voidStar; (* for demonstration only *)
local
val call1 = buildCall2((getSymbol gsl "gsl_matrix_calloc"),
(cInt,cInt), cPointer)
val call2 = buildCall4((getSymbol gsl "gsl_matrix_set"),
(cPointer,cInt,cInt,cDouble), cVoid)
val call3 = buildCall3((getSymbol gsl
"gsl_linalg_exponential_ss"), (cPointer, cPointer, cDouble), cVoid)
in
fun gsl_matrix_alloc (size1,size2) = call1(size1,size2)
fun gsl_matrix_set (ptr : gsl_matrix_t, i, j, x) = call2(ptr, i, j, x)
fun gsl_matrix (arr : real Array2.array) =
let
val size1 = Array2.nRows arr
val size2 = Array2.nCols arr
val ptr = gsl_matrix_alloc(size1, size2)
val _ = Array2.tabulate Array2.RowMajor (size1, size2, fn (i,j) =>
gsl_matrix_set(ptr, i, j, Array2.sub(arr, i, j)))
in
ptr
end
fun gsl_mexp (arr : real Array2.array) =
let
val size1 = Array2.nRows arr
val size2 = Array2.nCols arr
val ptr = gsl_matrix arr
val ptr2 = gsl_matrix_alloc(size1, size2)
val _ = call3(ptr, ptr2, 0.01)
in
ptr2
end
end;
fun main() =
let
val a = Array2.fromList [[1.0,2.0],[3.0,4.0]];
val ptr = gsl_matrix(a);
val m = gsl_mexp(a) (* fails *)
in
()
end;
(* symbol lookup error: /usr/lib/x86_64-linux-gnu/libgsl.so: undefined
symbol: cblas_dgemm *)

For (c):

polyc -c test.sml
g++ test.o -L/usr/lib/x86_64-linux-gnu -lgsl -lgslcblas `pkg-config
--libs polyml`
./a.out
# same error

Phil Clayton

2018-09-10 14:29:53 UTC

Permalink

Hi Mark,

Post by Mark Clements
(* symbol lookup error: /usr/lib/x86_64-linux-gnu/libgsl.so: undefined
symbol: cblas_dgemm *)

To avoid looking for the symbol "cblas_dgemm" in libgsl (which doesn't
define it), you could let the dynamic linker look for it by using an
empty string for the library which has the effect of passing
RTLD_DEFAULT to dlsym.

val gsl = loadLibrary "";

To make sure these libraries are searched, you would need to start your
poly session with something like

LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libgsl.so":"/usr/lib/x86_64-linux-gnu/libgslcblas.so
\
poly

which could be put into a wrapper script. Or you could build an
executable (which could be a poly REPL) and link to them.

Another advantage of this approach is portability. The SML code is more
portable because shared object names and their paths vary across
systems. Linking also avoids the problem of specifying the library
names because you use flags such as -lgsl and the linker works it out.
Linking can be done in an even more portable way using a utility like
pkg-config to generate the linker arguments:

pkg-config --libs gsl

(This usual requires the -devel or -dev package for the library.)

Regards,
Phil

Post by Mark Clements
I am enjoying using polyml's Foreign structure.
However, in using the GNU Scientific Library (GSL), I have run into a
problem with linkage. The GSL library (libgsl.so) calls standard cBLAS
matrix functions with unknown symbols to allow the user to link with
different cBLAS implementations (NB: GSL provides an unoptimised cBLAS
implementation named libgslcblas.so). When I call a GSL function that
depends on the cBLAS functions, I get a symbol not defined error.
I have tried unsuccessfully (a) loading the libgslcblas library either
before or after loading libgsl, (b) changing the loadLibrary
implementation to use dlopen with RTLD_LAZY | RTLD_GLOBAL, and (c)
compiling the SML code to an object file and then linking against the
libraries.
How could this be made to work?
Sincerely, Mark.
open Foreign;
val gsl = loadLibrary "/usr/lib/x86_64-linux-gnu/libgsl.so";
val gslcblas = loadLibrary "/usr/lib/x86_64-linux-gnu/libgslcblas.so";
type gsl_matrix_t = Memory.voidStar; (* for demonstration only *)
local
val call1 = buildCall2((getSymbol gsl "gsl_matrix_calloc"),
(cInt,cInt), cPointer)
val call2 = buildCall4((getSymbol gsl "gsl_matrix_set"),
(cPointer,cInt,cInt,cDouble), cVoid)
val call3 = buildCall3((getSymbol gsl
"gsl_linalg_exponential_ss"), (cPointer, cPointer, cDouble), cVoid)
in
fun gsl_matrix_alloc (size1,size2) = call1(size1,size2)
fun gsl_matrix_set (ptr : gsl_matrix_t, i, j, x) = call2(ptr, i, j, x)
fun gsl_matrix (arr : real Array2.array) =
let
val size1 = Array2.nRows arr
val size2 = Array2.nCols arr
val ptr = gsl_matrix_alloc(size1, size2)
val _ = Array2.tabulate Array2.RowMajor (size1, size2, fn (i,j) =>
gsl_matrix_set(ptr, i, j, Array2.sub(arr, i, j)))
in
ptr
end
fun gsl_mexp (arr : real Array2.array) =
let
val size1 = Array2.nRows arr
val size2 = Array2.nCols arr
val ptr = gsl_matrix arr
val ptr2 = gsl_matrix_alloc(size1, size2)
val _ = call3(ptr, ptr2, 0.01)
in
ptr2
end
end;
fun main() =
let
val a = Array2.fromList [[1.0,2.0],[3.0,4.0]];
val ptr = gsl_matrix(a);
val m = gsl_mexp(a) (* fails *)
in
()
end;
(* symbol lookup error: /usr/lib/x86_64-linux-gnu/libgsl.so: undefined
symbol: cblas_dgemm *)
polyc -c test.sml
g++ test.o -L/usr/lib/x86_64-linux-gnu -lgsl -lgslcblas `pkg-config
--libs polyml`
./a.out
# same error
_______________________________________________
polyml mailing list
http://lists.inf.ed.ac.uk/mailman/listinfo/polyml

Phil Clayton

2018-09-10 14:53:52 UTC

Permalink

Slight correction: using an empty string for the library has the effect
of passing NULL to dlopen which means the handle passed to dlsym is the
current program.

Post by Phil Clayton
Hi Mark,

Post by Mark Clements
(* symbol lookup error: /usr/lib/x86_64-linux-gnu/libgsl.so: undefined
symbol: cblas_dgemm *)

To avoid looking for the symbol "cblas_dgemm" in libgsl (which doesn't
define it), you could let the dynamic linker look for it by using an
empty string for the library which has the effect of passing
RTLD_DEFAULT to dlsym.
val gsl = loadLibrary "";
To make sure these libraries are searched, you would need to start your
poly session with something like
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libgsl.so":"/usr/lib/x86_64-linux-gnu/libgslcblas.so
\
poly
which could be put into a wrapper script. Or you could build an
executable (which could be a poly REPL) and link to them.
Another advantage of this approach is portability. The SML code is more
portable because shared object names and their paths vary across
systems. Linking also avoids the problem of specifying the library
names because you use flags such as -lgsl and the linker works it out.
Linking can be done in an even more portable way using a utility like
pkg-config --libs gsl
(This usual requires the -devel or -dev package for the library.)
Regards,
Phil

Post by Mark Clements
I am enjoying using polyml's Foreign structure.
However, in using the GNU Scientific Library (GSL), I have run into a
problem with linkage. The GSL library (libgsl.so) calls standard cBLAS
matrix functions with unknown symbols to allow the user to link with
different cBLAS implementations (NB: GSL provides an unoptimised cBLAS
implementation named libgslcblas.so). When I call a GSL function that
depends on the cBLAS functions, I get a symbol not defined error.
I have tried unsuccessfully (a) loading the libgslcblas library either
before or after loading libgsl, (b) changing the loadLibrary
implementation to use dlopen with RTLD_LAZY | RTLD_GLOBAL, and (c)
compiling the SML code to an object file and then linking against the
libraries.
How could this be made to work?
Sincerely, Mark.
open Foreign;
val gsl = loadLibrary "/usr/lib/x86_64-linux-gnu/libgsl.so";
val gslcblas = loadLibrary "/usr/lib/x86_64-linux-gnu/libgslcblas.so";
type gsl_matrix_t = Memory.voidStar; (* for demonstration only *)
local
     val call1 =    buildCall2((getSymbol gsl "gsl_matrix_calloc"),
(cInt,cInt), cPointer)
     val call2 =    buildCall4((getSymbol gsl "gsl_matrix_set"),
(cPointer,cInt,cInt,cDouble), cVoid)
     val call3 =    buildCall3((getSymbol gsl
"gsl_linalg_exponential_ss"), (cPointer, cPointer, cDouble), cVoid)
in
fun gsl_matrix_alloc (size1,size2) = call1(size1,size2)
fun gsl_matrix_set (ptr : gsl_matrix_t, i, j, x) = call2(ptr, i, j, x)
fun gsl_matrix (arr : real Array2.array) =
     let
     val size1 = Array2.nRows arr
     val size2 = Array2.nCols arr
     val ptr = gsl_matrix_alloc(size1, size2)
     val _ = Array2.tabulate Array2.RowMajor (size1, size2, fn (i,j) =>
gsl_matrix_set(ptr, i, j, Array2.sub(arr, i, j)))
     in
     ptr
     end
fun gsl_mexp (arr : real Array2.array) =
     let
     val size1 = Array2.nRows arr
     val size2 = Array2.nCols arr
     val ptr = gsl_matrix arr
     val ptr2 = gsl_matrix_alloc(size1, size2)
     val _ = call3(ptr, ptr2, 0.01)
     in
     ptr2
     end
end;
fun main() =
     let
     val a = Array2.fromList [[1.0,2.0],[3.0,4.0]];
     val ptr = gsl_matrix(a);
     val m = gsl_mexp(a) (* fails *)
     in
     ()
     end;
(* symbol lookup error: /usr/lib/x86_64-linux-gnu/libgsl.so: undefined
symbol: cblas_dgemm *)
polyc -c test.sml
g++ test.o -L/usr/lib/x86_64-linux-gnu -lgsl -lgslcblas `pkg-config
--libs polyml`
./a.out
# same error
_______________________________________________
polyml mailing list
http://lists.inf.ed.ac.uk/mailman/listinfo/polyml

_______________________________________________
polyml mailing list
http://lists.inf.ed.ac.uk/mailman/listinfo/polyml

Kostirya

2018-09-11 03:15:58 UTC

Permalink

Hello.
What is the difference between (LoadLibrary "") and (loadExecutable ()) calls?

Slight correction: using an empty string for the library has the effect of
passing NULL to dlopen which means the handle passed to dlsym is the current
program.

Post by Phil Clayton
Hi Mark,

Post by Mark Clements
(* symbol lookup error: /usr/lib/x86_64-linux-gnu/libgsl.so: undefined
symbol: cblas_dgemm *)

To avoid looking for the symbol "cblas_dgemm" in libgsl (which doesn't
define it), you could let the dynamic linker look for it by using an empty
string for the library which has the effect of passing RTLD_DEFAULT to
dlsym.
val gsl = loadLibrary "";
To make sure these libraries are searched, you would need to start your
poly session with something like
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libgsl.so":"/usr/lib/x86_64-linux-gnu/libgslcblas.so
\
poly
which could be put into a wrapper script. Or you could build an
executable (which could be a poly REPL) and link to them.
Another advantage of this approach is portability. The SML code is more
portable because shared object names and their paths vary across systems.
Linking also avoids the problem of specifying the library names because you
use flags such as -lgsl and the linker works it out. Linking can be done in
an even more portable way using a utility like pkg-config to generate the
pkg-config --libs gsl
(This usual requires the -devel or -dev package for the library.)
Regards,
Phil

_______________________________________________
polyml mailing list
http://lists.inf.ed.ac.uk/mailman/listinfo/polyml

Phil Clayton

2018-09-11 20:50:14 UTC

Permalink

Looking at polyffi.cpp it actually looks like `loadLibrary ""` ends up
calling
dlopen ("", ...)
whereas `loadExecutable ()` ends up calling
dlopen (NULL, ...)
A couple of things I have read suggest that these often give the same
result, if not always. So for non-Windows platforms `loadLibrary ""`
and `loadExecutable ()` look similar. However, the dlopen man page
doesn't mention the empty string case, so it seems preferable to use
`loadExecutable ()` over `loadLibrary ""`.

For Windows, I don't know what LoadLibrary("") does. So again it seems
preferable to use `loadExecutable ()` over `loadLibrary ""`.

Phil

Post by Kostirya
Hello.
What is the difference between (LoadLibrary "") and (loadExecutable ()) calls?

Slight correction: using an empty string for the library has the effect of
passing NULL to dlopen which means the handle passed to dlsym is the current
program.

Post by Phil Clayton
Hi Mark,

Post by Mark Clements
(* symbol lookup error: /usr/lib/x86_64-linux-gnu/libgsl.so: undefined
symbol: cblas_dgemm *)

To avoid looking for the symbol "cblas_dgemm" in libgsl (which doesn't
define it), you could let the dynamic linker look for it by using an empty
string for the library which has the effect of passing RTLD_DEFAULT to
dlsym.
val gsl = loadLibrary "";
To make sure these libraries are searched, you would need to start your
poly session with something like
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libgsl.so":"/usr/lib/x86_64-linux-gnu/libgslcblas.so
\
poly
which could be put into a wrapper script. Or you could build an
executable (which could be a poly REPL) and link to them.
Another advantage of this approach is portability. The SML code is more
portable because shared object names and their paths vary across systems.
Linking also avoids the problem of specifying the library names because you
use flags such as -lgsl and the linker works it out. Linking can be done in
an even more portable way using a utility like pkg-config to generate the
pkg-config --libs gsl
(This usual requires the -devel or -dev package for the library.)
Regards,
Phil

_______________________________________________
polyml mailing list
http://lists.inf.ed.ac.uk/mailman/listinfo/polyml

Phil Clayton

2018-10-14 14:07:42 UTC

Permalink

In fact `loadLibrary ""` doesn't work on Darwin, `loadExecutable ()` is
required.

Phil

Post by Phil Clayton
Looking at polyffi.cpp it actually looks like `loadLibrary ""` ends up
calling
dlopen ("", ...)
whereas `loadExecutable ()` ends up calling
dlopen (NULL, ...)
A couple of things I have read suggest that these often give the same
result, if not always. So for non-Windows platforms `loadLibrary ""`
and `loadExecutable ()` look similar. However, the dlopen man page
doesn't mention the empty string case, so it seems preferable to use
`loadExecutable ()` over `loadLibrary ""`.
For Windows, I don't know what LoadLibrary("") does. So again it seems
preferable to use `loadExecutable ()` over `loadLibrary ""`.
Phil

Post by Kostirya
Hello.
What is the difference between (LoadLibrary "") and (loadExecutable ()) calls?

Slight correction: using an empty string for the library has the effect of
passing NULL to dlopen which means the handle passed to dlsym is the current
program.

Post by Phil Clayton
Hi Mark,
undefined
> symbol: cblas_dgemm *)
To avoid looking for the symbol "cblas_dgemm" in libgsl (which doesn't
define it), you could let the dynamic linker look for it by using an empty
string for the library which has the effect of passing RTLD_DEFAULT to
dlsym.
    val gsl = loadLibrary "";
To make sure these libraries are searched, you would need to start your
poly session with something like
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libgsl.so":"/usr/lib/x86_64-linux-gnu/libgslcblas.so
\
    poly
which could be put into a wrapper script. Or you could build an
executable (which could be a poly REPL) and link to them.
Another advantage of this approach is portability. The SML code is more
portable because shared object names and their paths vary across systems.
Linking also avoids the problem of specifying the library names because you
use flags such as -lgsl and the linker works it out. Linking can be done in
an even more portable way using a utility like pkg-config to generate the
    pkg-config --libs gsl
(This usual requires the -devel or -dev package for the library.)
Regards,
Phil

Post by Mark Clements
I am enjoying using polyml's Foreign structure.
However, in using the GNU Scientific Library (GSL), I have run into a
problem with linkage. The GSL library (libgsl.so) calls standard cBLAS
matrix functions with unknown symbols to allow the user to link with
different cBLAS implementations (NB: GSL provides an unoptimised cBLAS
implementation named libgslcblas.so). When I call a GSL function that
depends on the cBLAS functions, I get a symbol not defined error.
I have tried unsuccessfully (a) loading the libgslcblas library either
before or after loading libgsl, (b) changing the loadLibrary
implementation to use dlopen with RTLD_LAZY | RTLD_GLOBAL, and (c)
compiling the SML code to an object file and then linking against the
libraries.
How could this be made to work?
Sincerely, Mark.
open Foreign;
val gsl = loadLibrary "/usr/lib/x86_64-linux-gnu/libgsl.so";
val gslcblas = loadLibrary "/usr/lib/x86_64-linux-gnu/libgslcblas.so";
type gsl_matrix_t = Memory.voidStar; (* for demonstration only *)
local
      val call1 =    buildCall2((getSymbol gsl "gsl_matrix_calloc"),
(cInt,cInt), cPointer)
      val call2 =    buildCall4((getSymbol gsl "gsl_matrix_set"),
(cPointer,cInt,cInt,cDouble), cVoid)
      val call3 =    buildCall3((getSymbol gsl
"gsl_linalg_exponential_ss"), (cPointer, cPointer, cDouble), cVoid)
in
fun gsl_matrix_alloc (size1,size2) = call1(size1,size2)
fun gsl_matrix_set (ptr : gsl_matrix_t, i, j, x) = call2(ptr, i, j, x)
fun gsl_matrix (arr : real Array2.array) =
      let
      val size1 = Array2.nRows arr
      val size2 = Array2.nCols arr
      val ptr = gsl_matrix_alloc(size1, size2)
      val _ = Array2.tabulate Array2.RowMajor (size1, size2, fn
(i,j) =>
gsl_matrix_set(ptr, i, j, Array2.sub(arr, i, j)))
      in
      ptr
      end
fun gsl_mexp (arr : real Array2.array) =
      let
      val size1 = Array2.nRows arr
      val size2 = Array2.nCols arr
      val ptr = gsl_matrix arr
      val ptr2 = gsl_matrix_alloc(size1, size2)
      val _ = call3(ptr, ptr2, 0.01)
      in
      ptr2
      end
end;
fun main() =
      let
      val a = Array2.fromList [[1.0,2.0],[3.0,4.0]];
      val ptr = gsl_matrix(a);
      val m = gsl_mexp(a) (* fails *)
      in
      ()
      end;
(* symbol lookup error: /usr/lib/x86_64-linux-gnu/libgsl.so: undefined
symbol: cblas_dgemm *)
polyc -c test.sml
g++ test.o -L/usr/lib/x86_64-linux-gnu -lgsl -lgslcblas `pkg-config
--libs polyml`
./a.out
# same error
_______________________________________________
polyml mailing list
http://lists.inf.ed.ac.uk/mailman/listinfo/polyml

_______________________________________________
polyml mailing list
http://lists.inf.ed.ac.uk/mailman/listinfo/polyml

David Matthews

2018-10-17 17:19:40 UTC

Permalink

loadLibrary "" doesn't work in Windows either. It returns an error 87 -
invalid parameter.

David

Post by Phil Clayton
In fact `loadLibrary ""` doesn't work on Darwin, `loadExecutable ()` is
required.
Phil

Post by Phil Clayton
Looking at polyffi.cpp it actually looks like `loadLibrary ""` ends up
calling
dlopen ("", ...)
whereas `loadExecutable ()` ends up calling
dlopen (NULL, ...)
A couple of things I have read suggest that these often give the same
result, if not always. So for non-Windows platforms `loadLibrary ""`
and `loadExecutable ()` look similar. However, the dlopen man page
doesn't mention the empty string case, so it seems preferable to use
`loadExecutable ()` over `loadLibrary ""`.
For Windows, I don't know what LoadLibrary("") does. So again it
seems preferable to use `loadExecutable ()` over `loadLibrary ""`.
Phil

Post by Kostirya
Hello.
What is the difference between (LoadLibrary "") and (loadExecutable ()) calls?

Slight correction: using an empty string for the library has the effect of
passing NULL to dlopen which means the handle passed to dlsym is the current
program.

Post by Mark Clements
I am enjoying using polyml's Foreign structure.
However, in using the GNU Scientific Library (GSL), I have run into a
problem with linkage. The GSL library (libgsl.so) calls standard cBLAS
matrix functions with unknown symbols to allow the user to link with
different cBLAS implementations (NB: GSL provides an unoptimised cBLAS
implementation named libgslcblas.so). When I call a GSL function that
depends on the cBLAS functions, I get a symbol not defined error.
I have tried unsuccessfully (a) loading the libgslcblas library either
before or after loading libgsl, (b) changing the loadLibrary
implementation to use dlopen with RTLD_LAZY | RTLD_GLOBAL, and (c)
compiling the SML code to an object file and then linking against the
libraries.
How could this be made to work?
Sincerely, Mark.
open Foreign;
val gsl = loadLibrary "/usr/lib/x86_64-linux-gnu/libgsl.so";
val gslcblas = loadLibrary
"/usr/lib/x86_64-linux-gnu/libgslcblas.so";
type gsl_matrix_t = Memory.voidStar; (* for demonstration only *)
local
      val call1 =    buildCall2((getSymbol gsl "gsl_matrix_calloc"),
(cInt,cInt), cPointer)
      val call2 =    buildCall4((getSymbol gsl "gsl_matrix_set"),
(cPointer,cInt,cInt,cDouble), cVoid)
      val call3 =    buildCall3((getSymbol gsl
"gsl_linalg_exponential_ss"), (cPointer, cPointer, cDouble), cVoid)
in
fun gsl_matrix_alloc (size1,size2) = call1(size1,size2)
fun gsl_matrix_set (ptr : gsl_matrix_t, i, j, x) = call2(ptr, i, j, x)
fun gsl_matrix (arr : real Array2.array) =
      let
      val size1 = Array2.nRows arr
      val size2 = Array2.nCols arr
      val ptr = gsl_matrix_alloc(size1, size2)
      val _ = Array2.tabulate Array2.RowMajor (size1, size2, fn
(i,j) =>
gsl_matrix_set(ptr, i, j, Array2.sub(arr, i, j)))
      in
      ptr
      end
fun gsl_mexp (arr : real Array2.array) =
      let
      val size1 = Array2.nRows arr
      val size2 = Array2.nCols arr
      val ptr = gsl_matrix arr
      val ptr2 = gsl_matrix_alloc(size1, size2)
      val _ = call3(ptr, ptr2, 0.01)
      in
      ptr2
      end
end;
fun main() =
      let
      val a = Array2.fromList [[1.0,2.0],[3.0,4.0]];
      val ptr = gsl_matrix(a);
      val m = gsl_mexp(a) (* fails *)
      in
      ()
      end;
(* symbol lookup error: /usr/lib/x86_64-linux-gnu/libgsl.so: undefined
symbol: cblas_dgemm *)
polyc -c test.sml
g++ test.o -L/usr/lib/x86_64-linux-gnu -lgsl -lgslcblas `pkg-config
--libs polyml`
./a.out
# same error
_______________________________________________
polyml mailing list
http://lists.inf.ed.ac.uk/mailman/listinfo/polyml

_______________________________________________
polyml mailing list
http://lists.inf.ed.ac.uk/mailman/listinfo/polyml

Mark Clements

2018-09-10 16:47:59 UTC

Permalink

Excellent - thank you. I now have a script called poly_gsl:

#!/bin/bash
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libgsl.so:/usr/lib/x86_64-linux-gnu/libgslcblas.so
poly $*

-- Mark

Post by Phil Clayton
Hi Mark,

Post by Mark Clements
(* symbol lookup error: /usr/lib/x86_64-linux-gnu/libgsl.so: undefined
symbol: cblas_dgemm *)

To avoid looking for the symbol "cblas_dgemm" in libgsl (which doesn't
define it), you could let the dynamic linker look for it by using an
empty string for the library which has the effect of passing
RTLD_DEFAULT to dlsym.
val gsl = loadLibrary "";
To make sure these libraries are searched, you would need to start your
poly session with something like
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libgsl.so":"/usr/lib/x86_64-linux-gnu/libgslcblas.so
\
poly
which could be put into a wrapper script. Or you could build an
executable (which could be a poly REPL) and link to them.
Another advantage of this approach is portability. The SML code is more
portable because shared object names and their paths vary across
systems. Linking also avoids the problem of specifying the library
names because you use flags such as -lgsl and the linker works it out.
Linking can be done in an even more portable way using a utility like
pkg-config --libs gsl
(This usual requires the -devel or -dev package for the library.)
Regards,
Phil

_______________________________________________
polyml mailing list
http://lists.inf.ed.ac.uk/mailman/listinfo/polyml

David Matthews

2018-09-12 20:18:39 UTC

Permalink

I've been thinking about adding direct external references as an
alternative to dynamic loading of libraries. The mechanism is all there
now because the Poly/ML run-time library is linked to the exported code
through external references.

I've added Foreign.externalFunctionSymbol : string -> symbol
This creates an external reference which is filled in when the code is
exported. The actual value is resolved by linking with the appropriate
library. For your example that means changing it to replace references
to getSymbol to externalFunctionSymbol and removing the calls to
loadLibrary. i.e.
open Foreign;

type gsl_matrix_t = Memory.voidStar; (* for demonstration only *)
local
val call1 = buildCall2((externalFunctionSymbol "gsl_matrix_calloc"),
(cInt,cInt), cPointer)
val call2 = buildCall4((externalFunctionSymbol "gsl_matrix_set"),
(cPointer,cInt,cInt,cDouble), cVoid)
val call3 = buildCall3((externalFunctionSymbol
"gsl_linalg_exponential_ss"), (cPointer, cPointer, cDouble), cVoid)
in
fun gsl_matrix_alloc (size1,size2) = call1(size1,size2)

It seems to handle your example fine when the code is exported using
"polyc" and then linked with the libraries.

I've committed this to a separate branch "ExternalBranch" on GitHub
rather than "master". I'd like some feedback on how useful this is and
whether there are any problems before merging it.

David

Phil Clayton

2018-10-05 08:23:42 UTC

Permalink

Hi David,

Thanks - this is useful. At the very least it will point out missing
dependencies at link time rather than run time. That is, provided it is
used correctly - see the issue below.

I'm glad to see that Poly/ML does not include symbols in the object file
that are referenced from ML but not required, e.g. because they occur in
code that is never reached. That's a useful property because it allows
a library of ML bindings to support more functions than actually
provided by the installed version of a binary library and required by
the application. Hopefully it will be possible to keep that property.

I have run into a general issue. After redefining a local version of
the getSymbol function to use externalFunctionSymbol internally, I found
that programs compiled and linked fine but seg faulted before entering
the main ML function. This was due to the scope in which a symbol was
declared using externalFunctionSymbol. The attached example
call_c_test_13.tar.gz gives a simple example of the problem. The
external function symbol isn't being declared until run time so this
code is inadvertently depending on a feature of dynamic linking. I can
see that use of externalFunctionSymbol is inherently more restricted
that getSymbol: externalFunctionSymbol must be applied to a symbol name
at compile time but the symbol cannot be used until run time,
post-linking. Are there any sort of checks that could be done to warn
users if they get this wrong? I can see it's not straightforward
because run time is itself compile time for an exported function that is
subsequently linked. At the very least, could an exception be raised
rather than seg faulting if there is an attempt to use an unresolved symbol?

Finally, is there a reason you chose the name "externalFunctionSymbol"
rather than, say, "externalSymbol"? It seems to work fine for
non-functions too - see attached example call_c_test_14.tar.gz .

Phil

Post by David Matthews
I've been thinking about adding direct external references as an
alternative to dynamic loading of libraries.Â The mechanism is all there
now because the Poly/ML run-time library is linked to the exported code
through external references.
I've added Foreign.externalFunctionSymbol : string -> symbol
This creates an external reference which is filled in when the code is
exported.Â The actual value is resolved by linking with the appropriate
library.Â For your example that means changing it to replace references
to getSymbol to externalFunctionSymbol and removing the calls to
loadLibrary.Â i.e.
open Foreign;
type gsl_matrix_t = Memory.voidStar; (* for demonstration only *)
local
Â Â Â val call1 =Â Â Â buildCall2((externalFunctionSymbol
"gsl_matrix_calloc"),
(cInt,cInt), cPointer)
Â Â Â val call2 =Â Â Â buildCall4((externalFunctionSymbol "gsl_matrix_set"),
(cPointer,cInt,cInt,cDouble), cVoid)
Â Â Â val call3 =Â Â Â buildCall3((externalFunctionSymbol
"gsl_linalg_exponential_ss"), (cPointer, cPointer, cDouble), cVoid)
in
fun gsl_matrix_alloc (size1,size2) = call1(size1,size2)
It seems to handle your example fine when the code is exported using
"polyc" and then linked with the libraries.
I've committed this to a separate branch "ExternalBranch" on GitHub
rather than "master".Â I'd like some feedback on how useful this is and
whether there are any problems before merging it.
David

Post by Mark Clements
I am enjoying using polyml's Foreign structure.
However, in using the GNU Scientific Library (GSL), I have run into a
problem with linkage. The GSL library (libgsl.so) calls standard cBLAS
matrix functions with unknown symbols to allow the user to link with
different cBLAS implementations (NB: GSL provides an unoptimised cBLAS
implementation named libgslcblas.so). When I call a GSL function that
depends on the cBLAS functions, I get a symbol not defined error.
I have tried unsuccessfully (a) loading the libgslcblas library either
before or after loading libgsl, (b) changing the loadLibrary
implementation to use dlopen with RTLD_LAZY | RTLD_GLOBAL, and (c)
compiling the SML code to an object file and then linking against the
libraries.
How could this be made to work?
Sincerely, Mark.
open Foreign;
val gsl = loadLibrary "/usr/lib/x86_64-linux-gnu/libgsl.so";
val gslcblas = loadLibrary "/usr/lib/x86_64-linux-gnu/libgslcblas.so";
type gsl_matrix_t = Memory.voidStar; (* for demonstration only *)
local
Â Â Â Â val call1 =Â Â Â buildCall2((getSymbol gsl "gsl_matrix_calloc"),
(cInt,cInt), cPointer)
Â Â Â Â val call2 =Â Â Â buildCall4((getSymbol gsl "gsl_matrix_set"),
(cPointer,cInt,cInt,cDouble), cVoid)
Â Â Â Â val call3 =Â Â Â buildCall3((getSymbol gsl
"gsl_linalg_exponential_ss"), (cPointer, cPointer, cDouble), cVoid)
in
fun gsl_matrix_alloc (size1,size2) = call1(size1,size2)
fun gsl_matrix_set (ptr : gsl_matrix_t, i, j, x) = call2(ptr, i, j, x)
fun gsl_matrix (arr : real Array2.array) =
Â Â Â Â let
Â Â Â Â val size1 = Array2.nRows arr
Â Â Â Â val size2 = Array2.nCols arr
Â Â Â Â val ptr = gsl_matrix_alloc(size1, size2)
Â Â Â Â val _ = Array2.tabulate Array2.RowMajor (size1, size2, fn (i,j) =>
gsl_matrix_set(ptr, i, j, Array2.sub(arr, i, j)))
Â Â Â Â in
Â Â Â Â ptr
Â Â Â Â end
fun gsl_mexp (arr : real Array2.array) =
Â Â Â Â let
Â Â Â Â val size1 = Array2.nRows arr
Â Â Â Â val size2 = Array2.nCols arr
Â Â Â Â val ptr = gsl_matrix arr
Â Â Â Â val ptr2 = gsl_matrix_alloc(size1, size2)
Â Â Â Â val _ = call3(ptr, ptr2, 0.01)
Â Â Â Â in
Â Â Â Â ptr2
Â Â Â Â end
end;
fun main() =
Â Â Â Â let
Â Â Â Â val a = Array2.fromList [[1.0,2.0],[3.0,4.0]];
Â Â Â Â val ptr = gsl_matrix(a);
Â Â Â Â val m = gsl_mexp(a) (* fails *)
Â Â Â Â in
Â Â Â Â ()
Â Â Â Â end;
(* symbol lookup error: /usr/lib/x86_64-linux-gnu/libgsl.so: undefined
symbol: cblas_dgemm *)
polyc -c test.sml
g++ test.o -L/usr/lib/x86_64-linux-gnu -lgsl -lgslcblas `pkg-config
--libs polyml`
./a.out
# same error
_______________________________________________
polyml mailing list
http://lists.inf.ed.ac.uk/mailman/listinfo/polyml

_______________________________________________
polyml mailing list
http://lists.inf.ed.ac.uk/mailman/listinfo/polyml

David Matthews

2018-10-07 11:44:23 UTC

Permalink

Hi Phil,
Thanks for that. I've made some changes and if it all seems to work out
I'll merge it into master. I've added some comments below.

Post by Phil Clayton
I'm glad to see that Poly/ML does not include symbols in the object file
that are referenced from ML but not required, e.g. because they occur in
code that is never reached. That's a useful property because it allows
a library of ML bindings to support more functions than actually
provided by the installed version of a binary library and required by
the application. Hopefully it will be possible to keep that property.

External symbols are ML cells that are recognised specially by
PolyML.export and turned into linker symbols. Since PolyML.export only
exports reachable ML data the resulting object file will only contain
symbols that are part of the exported code. A consequence of this is
that if the object file is then linked with a static library version of
the C code the final executable will only contain the C code that is
actually required. An executable created using external symbols and a
statically linked library will generally be more compact than having a
separate dynamic library. It also avoids the need for path
specifications such as LD_LIBRARY_PATH.

Post by Phil Clayton
I have run into a general issue. After redefining a local version of
the getSymbol function to use externalFunctionSymbol internally, I found
that programs compiled and linked fine but seg faulted before entering
the main ML function. This was due to the scope in which a symbol was
declared using externalFunctionSymbol. The attached example
call_c_test_13.tar.gz gives a simple example of the problem. The
external function symbol isn't being declared until run time so this
code is inadvertently depending on a feature of dynamic linking. I can
see that use of externalFunctionSymbol is inherently more restricted
that getSymbol: externalFunctionSymbol must be applied to a symbol name
at compile time but the symbol cannot be used until run time,
post-linking. Are there any sort of checks that could be done to warn
users if they get this wrong? I can see it's not straightforward
because run time is itself compile time for an exported function that is
subsequently linked. At the very least, could an exception be raised
rather than seg faulting if there is an attempt to use an unresolved symbol?

I've been thinking about that. The symbol is created with a zero
address. The actually address is set by the linker. It now raises an
exception if it tries to use a symbol when the value is zero.

Post by Phil Clayton
Finally, is there a reason you chose the name "externalFunctionSymbol"
rather than, say, "externalSymbol"? It seems to work fine for
non-functions too - see attached example call_c_test_14.tar.gz .

Phil Clayton

2018-10-14 14:03:02 UTC

Permalink

Hi David,

Thanks for the explanation and updates. I have tested this, including
externalDataSymbol, and all appears to work on my Linux and Darwin systems.

Initially I found that I wasn't seeing the new Foreign exception for an
uninitalized external symbol, although there was no seg fault. It turns
out that was actually because the exception was occurring in a function
registered with PolyML.onEntry. It seems that PolyML.onEntry silently
suppresses any exceptions from the functions it runs. Is that intended?

I have been wondering if there is a way to have a local version of
external(Function|Data)Symbol that falls back on
getSymbol (loadExecutable ())
if the functions aren't available. Could this be achieved by using one
of two files depending on the value of PolyML.rtsVersion () ?

Thanks,
Phil

Post by David Matthews
Hi Phil,
Thanks for that. I've made some changes and if it all seems to work out
I'll merge it into master. I've added some comments below.

Post by Phil Clayton
I'm glad to see that Poly/ML does not include symbols in the object
file that are referenced from ML but not required, e.g. because they
occur in code that is never reached. That's a useful property because
it allows a library of ML bindings to support more functions than
actually provided by the installed version of a binary library and
required by the application. Hopefully it will be possible to keep
that property.

Post by Phil Clayton
I have run into a general issue. After redefining a local version of
the getSymbol function to use externalFunctionSymbol internally, I
found that programs compiled and linked fine but seg faulted before
entering the main ML function. This was due to the scope in which a
symbol was declared using externalFunctionSymbol. The attached
example call_c_test_13.tar.gz gives a simple example of the problem.
The external function symbol isn't being declared until run time so
this code is inadvertently depending on a feature of dynamic linking.
I can see that use of externalFunctionSymbol is inherently more
restricted that getSymbol: externalFunctionSymbol must be applied to a
symbol name at compile time but the symbol cannot be used until run
time, post-linking. Are there any sort of checks that could be done
to warn users if they get this wrong? I can see it's not
straightforward because run time is itself compile time for an
exported function that is subsequently linked. At the very least,
could an exception be raised rather than seg faulting if there is an
attempt to use an unresolved symbol?

I've been thinking about that. The symbol is created with a zero
address. The actually address is set by the linker. It now raises an
exception if it tries to use a symbol when the value is zero.

On the X86 and most platforms the same linker symbols are used for both
function code and data. There are a very few platforms that use
different relocations for these. See commit 76b36c9d. I've added
externalDataSymbol for data.
Regards,
David
_______________________________________________
polyml mailing list
http://lists.inf.ed.ac.uk/mailman/listinfo/polyml

David Matthews

2018-10-17 17:33:53 UTC

Permalink

Hi Phil,

Post by Phil Clayton
Hi David,
Thanks for the explanation and updates. I have tested this, including
externalDataSymbol, and all appears to work on my Linux and Darwin systems.
Initially I found that I wasn't seeing the new Foreign exception for an
uninitalized external symbol, although there was no seg fault. It turns
out that was actually because the exception was occurring in a function
registered with PolyML.onEntry. It seems that PolyML.onEntry silently
suppresses any exceptions from the functions it runs. Is that intended?

That's how the code works at the moment. I guess it could call
OS.Process.exit OS.Process.failure but there's not much else it could do
at that point.

Post by Phil Clayton
I have been wondering if there is a way to have a local version of
external(Function|Data)Symbol that falls back on
getSymbol (loadExecutable ())
if the functions aren't available. Could this be achieved by using one
of two files depending on the value of PolyML.rtsVersion () ?

Using PolyML.rtsVersion is probably the best way. I've updated the RTS
version in Git. Usually it only gets done when the new version is
released but probably it should be updated at the same time as the
compiler version: as soon as something changes in Git.

David