Kinda off-topic, but I trust most of the programmers here than the ones I work with, so, if anyone can point me in a direction, or give me step-by-step, it'd be appreciated... I know very little C - mostly learning as I go, so this may not even be possible..

I have a program (non-empeg related..) that is made from several different sources (as most projects, I'm led to believe.) What I'd like to do is instead of, in the end linking it all into one big honkin' binary file, leave some of the object files as seperate files, so the executable file is smaller, and calls functions in one of the object files. (Kinda like Windows DLL files..)
Can it be done in Linux? (I'm using RH8.0)

Many thanks!
Mike.

Yes. Under most Unices, including Linux, these are known as ``shared objects'' and have .so extensions.

When compiling the object files that will go into them, you should use (assuming that you're using gcc) the -fpic option. This tells it to produce position independent code, which means that it doesn't matter where in memory it exists, and is very important if it will be used by more than one program at once.

When creating the shared object, you'll want to have your -o option specify a file named libsomething.so and also specify the option ``-shared''. This will created a shared object that can be dynamically linked to your executable.

You can have the final program then link it in via shared object function calls, but you more likely want to tell the runtime linker to deal with it. The problem with this is that shared objects have to be findable by the runtime linker itself, which is separate from your program. Most system libraries are held in well-known places (like /usr/lib, etc.), but you probably don't want to do that with this library. The program can tell the linker other places to look, though. If you know a directory that the program will be located in, then you should add a ``-rpath <libdir>'' to your final link to indicate where the library will be found. In order to specify the library at link time, you'll have to include the option ``-lsomething'', assuming that your library is of the name libsomething.so. The library need not be in the same place at link time it will be at run time, but it does need to be able to see a copy of it, and you can specify that using a ``-L <libdir>'' option to specify a directory where it can find shared objects.

(Note that at least some of these options are at least somewhat gcc/gnu ld specific.)

So, the complete procedure for a very simple program might look like:

gcc -c -fpic test.c
gcc -shared -o libtest.so test.o
gcc -c main.c
gcc -o test main.o -L. -ltest -rpath /usr/local/test/lib
cp libtest.so /usr/local/test/lib
cp test /usr/local/test/bin
/usr/local/test/bin/test

Ah, cool, thanks for the info!
One last question:

Say I have lib1_test.c and lib2_test.c, both of which I want to convert into *.so files.
They both have a single function, say, run_test();
In lib1_test, run_test() echos "LIB1" to the screen, and in lib2_test, it echos "LIB2" to the screen. (For simplicity's sake.)

Obviously, with the same function, I can't have both in the same directory, because the program wouldn't know which run_test() I wanted. But, can I have one in place, run "test" with the one that uses "LIB1", then replace it with the one that displays "LIB2", and it will work? (i.e. does the executable look for specific filename and function in that file, or can they be interchanged?)

Sorry for the twisted question. Not exactly sure how to phrase what I'm asking.

By my understanding, you cannot have the same symbol defined twice in the same process. It would, in fact, notice this at link time (the final gcc command above, which is really handing off most of its work to ld) and not allow you to do it. However, you could have two copies of the executable, one linked to lib2_test.so and one linked to lib1_test.so. I don't know exactly what you're going for, but that sounds not like what you want to do.

What may be more appropriate for your situation is what I alluded to before, using system calls to load shared objects dynamically. I'm a novice at this sort of thing, and I'm inclined to say it's pretty specific to each individual Unix. There's probably a POSIX spec for it, but it's likely to be still fluctuating in the real world. Regardless, you'd want to use the dlopen() set of functions to deal with that instead of the -l, -L, and -rpath options to the linker.

Hmm.. maybe this will help. my real-life example.

I have a program that goes thru all my VOX files (Dialogic Audio files from a PBX system) and determines wether they're worthy enough to be archived or not, and then archive them.
The main program does all it's stuff, and then calls a 'generic' "lib_archive()" function.
Currently, depending on how I make it, it will use one of several *.o files, and link it into the executable. (One burns to CD-R, one to DVD-R, one FTPs to a remote server, and the fourth copies to an NFS share) The main program ALWAYS calls 'lib_archive()', and does not care what this function does. Once the function finally finishes, control is given back to the main program, and it finishes up.

Now, the object file can have any number of functions that are used by the main lib_archive in it.

So, can I have one of the differently named *.so files (like lib_cdr.so, lib_dvd.so, lib_ftp.so, etc etc..) in the lib directory, and will the executable try whatever file is in there until it finds a lib_archive(),
*OR*
When the program's compiled, it needs ONE *.so file, and that compiled exec is stuck with that *.so file?

(Either way, I can work with it, and it IS what I needed. Just curious as to how much flexibility there is..)

Many Thanks!
Mike

Okay, you obviously are at least in charge of final linking and the one object that contains, if nothing else, the lib_archive() function.

How do you make the determination of which archiving function to use? If it's logical, then why not just have lib_archive() figure it out and then call lib_archive_cdr() or lib_archive_dvdr() or lib_archive_ftp() or lib_archive_nfs()? That way you don't have to deal with any of this mess.

When the program's compiled, it needs ONE *.so file, and that compiled exec is stuck with that *.so file?

That one.

However:

can I have one of the differently named *.so files (like lib_cdr.so, lib_dvd.so, lib_ftp.so, etc etc..) in the lib directory, and will the executable try whatever file is in there until it finds a lib_archive()

I don't understand your question here. If your intent is to have a different one for each possibility, wouldn't each of them have a lib_archive() in it? Or were you planning to move them in and out of there as needed?

What may be more appropriate for your situation is what I alluded to before, using system calls to load shared objects dynamically. I'm a novice at this sort of thing, and I'm inclined to say it's pretty specific to each individual Unix. There's probably a POSIX spec for it, but it's likely to be still fluctuating in the real world. Regardless, you'd want to use the dlopen() set of functions to deal with that instead of the -l, -L, and -rpath options to the linker.

Ooh... I like that dlopen().
Youjust gave me the tools to fire another programmer in my company. Muahaha. (Now I just gotta put it into practice.)

Many thanks!
Me.

You just gave me the tools to fire another programmer in my company.

Well, that's not what I wanted to do.

[spooky hypnotist voice]
There is no such thing as dlopen().

There is no such thing as dlopen()!
[/spooky hypnotist voice]

Couldn't you use function pointers to good effect?

I'm not an expert on function pointers, but if his main program is just calling lib_archive(), then that function will have to exist, statically or dynamically, in the running process's symbol table. I suppose that function could then itself use function pointers, but the basic problem is knowing which actual algorithm the function should use, and if there are only four potential functions, then using function pointers seems more like an exercise in obfuscation.

Unless I'm missing a way in which they could be used. As I said, I'm not an expert.

Okay, you obviously are at least in charge of final linking and the one object that contains, if nothing else, the lib_archive() function.

Yes... I control EVERYTHING! Muahaha. *erm* Sorry.

How do you make the determination of which archiving function to use? If it's logical, then why not just have lib_archive() figure it out and then call lib_archive_cdr() or lib_archive_dvdr() or lib_archive_ftp() or lib_archive_nfs()? That way you don't have to deal with any of this mess.

Well, if I all included them into one library, then the program would be friggin' huge. (Probably a 2-3 meg file.) There's a reason for this, and unfortunately, it's something I canNOT get around.
In the end, I have four exec's. Normally, clients of mine only get ONE version. However, often they'll upgrade, or change their mind. Now instead of uploading to their site a 3 meg files (complete distribution package) for a new version, where the only thing that changes is the lib_archive() function, I'd like to simply upload a *.so file to replace the one they have. Saves time, and is much simpler.

can I have one of the differently named *.so files (like lib_cdr.so, lib_dvd.so, lib_ftp.so, etc etc..) in the lib directory, and will the executable try whatever file is in there until it finds a lib_archive()

I don't understand your question here. If your intent is to have a different one for each possibility, wouldn't each of them have a lib_archive() in it? Or were you planning to move them in and out of there as needed?

Exactly - they'd ALL have a lib_archive, and depending on their "security level" (read: how much they paid) the program would only use the one it was told to. But all the options are there, in case they want to change.

However, I've been looking at this dlopen() function, and I think it is **EXACTLY** what I've been looking for..
From the looks of it, I can tell it what file to look at, and then what function I need the address for, and go from there..

(See, my company is full of "think for the now" programmers and designers. Have of our 'C' department didn't even know what I was talking about when linking to a library. Every time a new version of a core product is available, we always have a complete new distribution package which ranges from 3-5 megs that we have to upload to clients sites via modem. So, when I started my project (completely on the side, mind you. I don't even work for the programming department) the first thing I was going to do was make it modular - so no matter what version a client has, it runs exactly the same way, and if a new option came along, I could easily make a differen source file to make a *.o file, and simply link it in. In fact, that's how I did my DVD version - CD-R was already done. The whole DVD functions took me abour 2 hours to do. They gave me, and paid me for, a week and a half, because that's how long it normally would have taken the other programmers.
Add, on top of that (and one of the reasons why the exec is so bloated in size) the fact that it uses and orriginated from Informix 4GL - which pre-compiles it down to C with their screen-handling and SQL and Form-handling functions, then links it all together. That's how I started learning C. Everything I know I've learned from taking apart created code, looking at man pages, and asking questions. At one point my C functions that I was using in the program were too fast for the Informix's built-in screen handlers, and I had to slow my C down with usleep() statements.
So, in effect, this is new territory for me, but something that's needed to be done for a while, to keep ME sane trying to upgrade all my clients that run this.)

Sorry for ramblin. :P
Thanks!
Me.

Given that information (that only one will exist at each site), then just link it to libarchive.so, provide the appropriate shared object for each thing and then have them overwrite their current one with the right one.

Simple enough.

Heh!
I got one fired already - My program replaced his (which was written entirly in Informix) mostly because.. Mine worked.
I'm not joking. 99% of the time, his would error our. The ONLY error message it EVER gave was "not enough data to burn". So, within 6 months of working at my company, my program replaced his as a core product, and he promptly left.
I'm left up to asking YOU guys for help because my own programmers don't know &*^#*@. At one point, when I started using fork(); I kept getting zombie processes, and didn't know why. (Remember, I lear as I go. Knew nothing about catching signals from child processes.) I asked three programmers in my company, and either they didn't understand what I was trying to do, or didn't know.

(I have really high hopes for people in MY department, "Technical Support", but others.. they worry me sometimes.)

Me.

I think that what the dlopen() will do, from looking at examples, I have to.. (kinda simplified..)

handle = dlopen(<filename>, <flags>); // Open whatever file is appropriate..
lib_archive_ptr = dlsym(handle, "lib_archive"); // Get the pointer address for the always-there function. Then...
(*lib_archive_ptr)(); // Execute the function at memory address I got above.


In a nutshell, that works for me. I can't do that within the Informix 4GL, of course, so I just have to create a different C function that will open the file and find the function and make sure it loads up before returning control to the Informix stuffs..


Thanks! :>
Me.

Yep. That's pretty much it.
It's still a good idea to check dlerror() still though even though that function should always be there.
If you're using C++ then remember that the compiler will do name munging so put extern "c" { } around the function prototype.
And finally, you may need to give GCC some extra arguments when linking ("-shared -Wl,-export-dynamic" from what I remember..) to tell it to export all of the symbols so the modules can access functions built into the main program.

- Trevor

Yep. That's pretty much it.
It's still a good idea to check dlerror() still though even though that function should always be there.
If you're using C++ then remember that the compiler will do name munging so put extern "c" { } around the function prototype.
And finally, you may need to give GCC some extra arguments when linking ("-shared -Wl,-export-dynamic" from what I remember..) to tell it to export all of the symbols so the modules can access functions built into the main program.

Allright, cool. Thanks. This is straight 'C' so I don't have to change much (hopin', anyways.) And yeah, I had forgotten - functions called by the 'dynamic' lib_archive() include other function in it's own code, as well as function back in the main code, so I'll have to pay attention to that..

Thanks!
Mike.

Wanted to thank everyone again, for the help. Turns out that I can't use it afterall, due to the funkiness (read: awful) way the 4GL precompiler turnes everything into C, but, I leared a heck of a lot, and that is what it's all about.

Me.

? Why's it not work if it all gets converted to C?

- Trevor

Well, mostly because it gets converted to C very, very poorly.
For example, insterad of passing arguments to functions directly, it passes the # of arguments, and then you have to use special functions to read/write to the stack. Then do the reverse upon leaving the function (write the return var to the stack, then actually return the # of values you put on the stack).
So, when I write a C function to load and then run my library, the function has to catch the # of vars I'm passing it (usually fixed), then throw it back on the stack, send the 4GL function (which, now, has been converted to C) the number of vars (again, fixed), and then put the actual value on the stack, then call the C function converted from the 4GL (which undoes what I just did which undid what the function that started it all out did).

PITA, really.

Now, when I ran it, it actually ran successfully, to a point. It called my lib_archive() function, but the lib_archive() function seemed to ignore all 'call ...' statements in it. So it just exited back with a null (which then completely effed' up my reading and modding the stak before returning to the main function, causing a segfault.)
I'm not too sure why the lib_archive() in the *so file didn't call any of it's functions.. On the other hand, I was pleasently suprised that once I found linking info online, I was able to actually get the program to run, load the library, and actually fire off the function within.. 10 minutes of starting on it.

(Oh, and also take into account that I 'stretch' the limits of the 4GL and what it 'expects' with it's stacks and all.. 1. I nosed around and changed some of the 4gl's converted functions into pure C functions (stack-less? I guess you'd call it) and use those instead. 2. I bypass some of the 4GL's SQL interface, and us the C interface, giving me more flexibility and speed. 3. I use fork()'s in my C code to run external commands, and catch the output and result. 4GL was never meant to 'fork' into other processes, and I've been able to make it work. Add all thse together, and my program ended up being too fast for it's own good, and had to sleep() it to slow it down.) The only reason I stay with the 4gl at all is that it gives me a very convienent user interface, and certain SQL functions.

When I say I do 'Mad Science Programming', I mean it. Muahaha. I seem to highly excell at taking existing code, and mutilating it in the worst way to get it to work the way I want (faster, easier, etc etc).

*erm* Sorry for the dissertation.
Me.