Thursday, April 6, 2017

Oracle TLS/SSL Encryption Tutorials

So, you want to secure communications between your application and an Oracle database with TLS/SSL...

Most TLS/SSL-capable servers are secured by generating a private key, getting a certificate, and adding a few lines to a config file to tell the server the locations of the cert and key files. Of course, if you generate the certificate yourself, then you also have to tell the server about your fake certificate authority. But, for most apps, that's it. Three parameters in a file somewhere: cert, key, and ca.

With Oracle, the idea is similar, but the details are very different.

Rather than storing certificates and keys in files and providing the paths to them in a config file, you store them in an Oracle "wallet" and provide the path to the wallet.

While Oracle wallets are more secure than flat files, the security features introduce some interesting quirks that are worth reviewing before attempting an implementation.

When you're ready to take the plunge, tutorials follow:

Saturday, December 24, 2016

Updating Your App to Support OpenSSL 1.1.0

Introduction

OpenSSL 1.1.0 introduced several API-breaking changes, and with distributions starting to use it, I found myself scrambling to get my software working. I found various docs stating that 1.1.0 changed things, but nothing that clearly stated: "if you have this now, then change it to this..." So, that's what I'll do here, and hopefully it will help you if you find your app in the same situation.

Deprecated "method" Functions

Lots of "method" functions like TLSv1_server_method() and SSLv23_client_method() and TLSv1_2_method() are deprecated in OpenSSL 1.1.0, and if you compile your software with -Werror, then that presents a problem.

All you can call now is one of:

  • TLS_method()
  • TLS_client_method()
  • TLS_server_method()

If your code allowed the user to select a SSL/TLS mode then it can't do it by calling a mode-specific function any more. It'll have to select the mode some other way that I'm not aware of. The new methods negotiate the most modern mode supported by the client and server. It's not immediately clear how to override that, or if you can. If someone knows a way, please leave a comment below.

It's easy enough to replace function calls, but if you want to support older versions, then you have to add tests for the old and new functions to your configure script and add all kinds of #ifdefs, and so forth. Actually, since various methods like these have been added over the years, you may already have such tests and #ifdefs (I did) and it won't be so bad.

Deprecated ASN1_STRING_data
unsigned char *ASN1_STRING_data(...);

has been replaced with

const unsigned char *ASN1_STRING_get0_data(...);

This is a straightforward replacement, but note the const'ness of the return value of the new function. If you're using C++ then you may have to change the type of the variable it's being assigned to (or un-const it).

Again, this is a brand new function, so, configure tests and #ifdefs.

Opaque EVP_PKEY

The EVP_PKEY struct (public key struct) is opaque now. The header files declare that an EVP_PKEY struct exists, but the definition is hidden up in the library. The error you get (with g++ at least) is fairly obtuse though, complaining about a forward declaration and an invalid use of an incomplete type. The error just means that the struct is opaque.

Fortunately there are a bunch of functions that return the various components of the struct.

I had to replace code to get the key type like:

EVP_PKEY *pubkey = X509_get_pubkey(c);
...
int t = pubkey->type;

with code like:

EVP_PKEY *pubkey = X509_get_pubkey(c);
...
int t = EVP_PKEY_base_id(pubkey);

Note that there IS an EVP_PKEY_type() function, but it does NOT return the "type" member of the struct. Ha! Not super intuitive, but the change is still straightforward.

Similarly, I had to replace code to get the key itself like:

EVP_PKEY *pubkey = X509_get_pubkey(c);
...
const unsigned char *key = pubkey->pkey.ptr;

with code like:

EVP_PKEY *pubkey = X509_get_pubkey(c);
...
const unsigned char *key = EVP_PKEY_get0(pubkey);

I'm not sure when those functions were introduced, but they appear to be available in OpenSSL 1.0.x but not in OpenSSL 0.9.x, so again, if you want to support really old versions, then you'll have to add configure tests and #ifdefs.

Opaque X509

The X509 struct is also opaque now. Like the EVP_PKEY struct, you'll have to use functions to access its members.

The one case I needed was slightly trickier though.

In the past, I was getting the signature algorithm name like:

X509 *c = SSL_get_peer_certificate(ssl);
...
char sigalg[256];
OBJ_obj2txt(pvt->_sigalg,256,c->sig_alg->algorithm,0);

But I had to change that code to:

X509 *c = SSL_get_peer_certificate(ssl);
...
char sigalg[256];
OBJ_obj2txt(pvt->_sigalg,256,OBJ_nid2obj(X509_get_signature_nid(c)),0);

Which is a little contrived. In the old code, c->sig_alg->algorithm is an "obj", and I could pass it directly into OBJ_obj2txt. In the new code, I have to get the "nid", convert it to an object, and pass that in. Ideally I'd just get the object to begin with, but there doesn't appear to be an obvious way to do that.

As a side note, the function name X509_get_signature_nid() was a little confusing at first, because it appears to return the "nid" of the signature _algorithm_ as opposed to the signature itself. But I guess the signature itself wouldn't have a "nid", so it makes sense, but it wasn't immediately intuitive to me.

Again, X509_get_signature_nid() was introduced sometime in the 1.0.x series, so you'll have to add a test for it and #ifdefs if you want to support older versions.

Conclusion

The opacity of the structs ought to help OpenSSL be more maintainable in the future. They ought to be able to change the code at will now, without breaking the API or ABI, and that's always good. I did the same kind of thing with my Rudiments library way back, and recently to SQL Relay for that exact reason. Hopefully it will work out well for both of us.

I'm not sure about the TLS_*_method() thing though. It seems like there really ought to be a way to choose a specific SSL/TLS mode. Perhaps there is though, and I just didn't see it. Somebody let me know if they figure that one out.

Saturday, January 9, 2016

Building Ruby From Source With Microsoft Visual Studio

Introduction

Ruby packages are available for Windows, and they can be used to develop Ruby applications, all day long. However, they lack the necessary files to develop Ruby modules in C/C++ and unlike PHP, where you can fairly easily shoehorn the headers into an already-installed package, it's not so easy with Ruby. In fact, to get a hold of the components to shoehorn in, you have to do an entire build from source, so you might as well just do that to begin with.

Fortunately, it's not too difficult.

Prerequisites

For this effort you will need:

  • Microsoft Visual Studio
  • The Ruby source code.
  • A few unixish commands - gunzip and tar. These can be provided by a gnuwin32 installation, a cygwin installation.
  • A text editor. Notepad or vi will do.

The Process

I last did this with Ruby 2.2.0, so I'll use it in this example.

The Ruby source code comes as a tar.gz file. There aren't any .zip files available, or if there are, I'm not sure where to find them. So you'll have to use some unixish commands to extract it.

Move the ruby sources to somewhere convenient and use the unix tools to extract it:

gunzip ruby-2.2.0.tar.gz
tar xf ruby-2.2.0

This should create a directory named ruby-2.2.0

Open the appropriate Visual Studio command prompt. If you plan on building a 32-bit version of Ruby, then open the x86 command prompt. If you plan on building a 64-bit version of Ruby, then open the x64 command prompt.

Navigate into the ruby-2.2.0 directory that was created by the extraction process above.

To get everything to compile and install correctly, I had to edit the file tool\rbinstall.rb and change line 714 to read:

rescue LoadError

This may or may not be necessary with a newer version of ruby, or the line to edit might be different.

Now, configure the build.

For a 64-bit build, run:

win32\configure.bat --prefix=C:\Ruby --target=x64-mswin64

For a 32-bit build, run:

win32\configure.bat --prefix=C:\Ruby --without-ext=fiddle 

(The --without-ext=fiddle command just disables the "fiddle" extension, which I couldn't get to build on 32-bit Windows, for some reason. Your mileage may vary.)

To actually build Ruby, run:

nmake

(This will run for a while.)

When it's done, install Ruby by running:

nmake install

Everything should get installed under C:\Ruby

To make the ruby programs generally accessible, add C:\Ruby\bin to the system PATH as follows:

  • Open the Control Panel
  • Search for Environment
  • Click the link for "Edit the system environment variables"
  • Click "Environment Variables"
  • In the System Variables pane, scroll down and click on Path
  • Click Edit...
  • Append: ;C:\Ruby\bin to the Variable Value.
  • Click OK
  • Click OK
  • Click OK

And that's it. Ruby is now built, installed and accessible.

Hopefully this helps someone out.

Good luck developing your Ruby modules on Windows.

Generating a Missing Import File From a DLL

Introduction

Occasionally I run into software packages for Windows that are missing some of the developer bits. One bit that occasionally gets left out is the import library. The headers are there, the .dll is there, but the .lib file is missing.

It's not impossible to generate a .lib from a .dll though.

Here's how.

Prerequisites

You'll need a few things to do this:

  • Microsoft Visual Studio
  • A few unixish commands - echo, cat, and cut. These can be provided by a gnuwin32 installation, a cygwin installation, or just a second linux or unix system.
  • A text editor. Notepad or vi will do.

The Process

I ran into this issue with Active Perl 5.20, so I'll use it in this example. Adjust accordingly.

First, dump the list of exports from the dll into an exports.in file. Open a Visual Studio command prompt and run some commands similar to the following:

cd C:\Perl64\bin
dumpbin /exports perl520.dll > exports

(adjusting for the path and file name of your .dll)

Now, edit the exports file.

Trim off everything prior to the first function definition. In the perl dll, that was a line like:

          1    0 000295C0 ASCII_TO_NEED

Trim off everything after the last function definition. In the perl dll, that was a line like:

       1249  4E0 001271B0 win32_write

Now, use the unixish commands to create a .def file.

(If you're doing this on a second machine, then you'll need to copy the exports file over to it and then, afterwards, copy the .def file back.)

echo EXPORTS > perl520.def
cat exports | cut -c27- >> perl520.def

Now, use Visual Studio's lib command to build the .lib file.

lib /def:perl520.def /OUT:perl520.lib /MACHINE:X64

Note: In this example, we had a 64-bit DLL. To build from a 32-bit DLL, use the /MACHINE:X86 flag instead.

Currently the .lib file is sitting in the bin directory, but .lib files generally need to go in a lib directory or some other place. So, move it now.

move perl520.lib ..\lib\CORE

You can now clean up if you like.

del exports
del perl520.def

And that's it. You can now link against the .lib file and your app will load the .dll at runtime.

Hopefully these instructions help if you ever find yourself in a similar situation.

Shoehorning Development Headers Into PHP for Windows

Introduction

In years past, I've had trouble building PHP modules on Windows because, though PHP packages are available for Windows, they lack the necessary header files to build PHP modules. One solution is to build and install PHP from source, but that takes forever, uses a ton of space, and has a long list of prerequisites. It turns out that it's actually faster and easier to shoehorn in a set of development headers into the already-installed PHP package.

Here's how.

Prerequisites

You will need the following items:

  • A Linux or Unix machine, with a working compiler toolchain. (Actually, a cygwin installation with a working compiler toolchain might suffice but I've never tried it.)
  • The PHP package for Windows.
  • The source code for the same version of PHP as the package.
  • The same version of Microsoft Visual Studio that was used to build the PHP package. The version of VS is discernible in the package filename. For example, php-7.0.2-Win32-VC14-x64.zip was built with Visual Studio 14 (aka Visual Studio 2015).
  • Several bits of software from the gnuwin32 project:

On the Windows machine, install the PHP package into C:\PHP and install Visual Studio.

Install the gnuwin32 software:

  • Create C:\gnuwin32
  • Extract each zip file by right-clicking on it and selecting Extract All.
  • Move the contents of the folders created during the extraction into C:\gnuwin32
  • Add C:\gnuwin32\bin to the PATH environment variable.
    • Open the Control Panel
    • Search for Environment
    • Click the link for "Edit the system environment variables"
    • Click "Environment Variables"
    • In the System Variables pane, scroll down and click on Path
    • Click Edit...
    • Append: ;C:\gnuwin\bin to the Variable Value.
    • Click OK
    • Click OK
    • Click OK

If you can run bison or flex from the command prompt, then it worked.

Building the Headers

Copy the source code to the Linux/Unix machine and extract it. Then build and install the headers somewhere convenient, like your home directory. In this example, we'll use /home/dmuse/php though it could be anywhere.

tar xf php-5.6.17.tar.bz2
cd php-5.6.17
./configure --prefix=/home/dmuse/php
make
make install-headers

Now, copy the entire /home/dmuse/php/include directory to the Windows machine. If the linux/unix machine is visible as a Windows share then you can just drag the directory across the network. Otherwise you can zip or tar it up, contrive to copy it over using scp, ftp, http, or something else, and then extract it by right-clicking on it and selecting Extract All.

Move the includes folder to C:\PHP\dev such that there is now a C:\PHP\dev\include folder. Verify that C:\PHP\dev\include\php exists and that C:\PHP\dev\include\include didn't get created by accident, as sometimes the top-level folder gets duplicated during extraction.

The PHP package now contains most of the headers necessary to develop PHP modules, but a few windows-specific files need to be generated and installed.

Extract the PHP source somewhere convenient on the Windows machine.

Open a Visual Studio command prompt. Make sure to open a prompt for the same architecture as the PHP binary package. For example, if the package is an x86 package, then make sure to open the x86 prompt. If the package is an x64 package, then make sure to open the x64 prompt.

Change directories to the folder that was created when you extracted the PHP source and run the following commands.

buildconf.bat
cscript /nologo configure.js

Then, copy the Windows-specific headers that were built by those commands into the PHP tree:

copy main\config.w32.h C:\PHP\dev\include\php\main
copy TSRM\tsrm_config.w32.h C:\PHP\dev\include\php\TSRM
copy Zend\zend_config.w32.h C:\PHP\dev\include\php\Zend

And that's it. You have now shoehorned development headers into the PHP package for Windows. Good luck building your PHP module!

Friday, September 25, 2015

Solaris 9 Sparc

Introduction

Qemu 2.4.0 came out on August 11th and I've been so busy that I didn't even notice. This week though, I've been sick, all week. ...and my mind just hasn't been up to its customary level of problem solving. Feeble attempts to get work done gave way to browsing the web, and eventually to sleep. During one of those periods of consciousness though, I noticed that Qemu 2.4.0 had been released. And the ChangeLog included something that piqued my interest.

In the SPARC notes:

     Fix SunOS 4.1.4 boot on sun4m with OpenBIOS

SunOS 4.1.4 is also known as Solaris 1.something-or-other. Last I checked, OpenBIOS couldn't boot anything except Solaris 7, 8 and 9, in single-user mode, and there were even some limitations to that. Older versions of Solaris were completely unsupported.

I knew that Artyom Tarasenko and friends had been poking at OpenBIOS and QEMU over the years, and bit-by-bit it could boot more and more things, but the last I'd heard, a year ago, they'd just barely gotten it to boot Solaris 9 in full single-user mode.

What's this I hear about SunOS 4.1.4? Has a bunch of progress been made since the last time I checked?

I would have to find out.

But I didn't. Not at first. Instead I slept and slept and tried to kick this cold. But later in the week, I improved somewhat, and rather than spending my conscious moments browsing the web, I nudged along some projects, including an attempt to get some version of Solaris for Sparc running.

I started with Solaris 9. I'm not sure why, really. I own copies of 1.1.1B, 2.1 and versions 7 through 11. No idea why I settled on 9. My subconscious was making most of my decisions at the time. I think it's because I'd also read that 9 sort-of-boots further and 9 was still on my mind.

At any rate, I gave 9 a try.

VM Configuration

I sort-of followed some of the directions at Artyom's Solaris/sparc under qemu how-to but with a few modifications. I had a few false starts, but it didn't take long, or much effort (thankfully) to work my way through them.

So here's what I ultimately did...

I created a 36G disk.

qemu-img create solaris9_sparc.img 36G

Why 36G in particular? Wait and see...

This is the QEMU command that I ended up using too.

qemu-system-sparc -drive file=solaris9_sparc.img,format=raw \
-cdrom solaris-9-sparc-installation.iso \
-nographic -net bridge -net nic,macaddr=aa:00:00:00:00:a0 \
-prom-env "auto-boot?=false"

Modern QEMU complains if you use a raw disk but don't tell it that you're using a raw disk, thus the -drive argument rather than just using -hda.

The -nographic option basically runs it in the terminal window rather than simulating the framebuffer, which is great because it runs much faster and the only supported framebuffer resolution is big and unwieldy.

I'm also using bridged networking. Here's a little info on that.

The weird -prom-env argument keeps it from trying to boot. There are various issues related to using the QEMU -boot option. If you tell it to boot to the CD, it does ok, but it universally attempts to boot to the wrong disk partition. It seems better to boot manually from the OpenBIOS prompt.

Also, the Solaris 9 distribution comes with a bunch of CD's, including an Installation disc, 2 Software discs, a Supplements disc, and Companion CD. I configured QEMU to boot to the installation disc here.

Pre-Installation

Running that QEMU command dropped me to an OpenBIOS prompt, where I booted to the CD, in single-user mode.

boot cdrom:d -vs

(-v means verbose, -s means single-user)

If you do this, and you look carefully, you'll notice something like this scroll up the screen:

 (sd0):
        Corrupt label; wrong magic number

As neozeed has pointed out before, Sun-provided disks were apparently pre-formatted, but if you used a non-Sun disk, you had to format it yourself, or the installer won't recognize it. My QEMU disk image was certainly a non-Sun-provided disk.

The single-user boot dropped me at a root prompt, and I ran the "format" command.

format

Which disk? Disk 0. What disk type? Other.

And then it prompted me for a dozen odd parameters. The important ones were:

  • data cylinders: 24622
  • heads: 27
  • data sectors/trac: 107
  • disk type name: qemu

I accepted defaults for the others.

BTW, the reason I used a 36G disk, in particular, is because those parameters are known to work with a 36G disk. Sun disks apparently have odd geometries, and the old geometry calculator web page is long gone.

After that, I labelled the disk, quit and rebooted.

format> label
Ready to label disk, continue? y

WARNING: /iommu@0,10000000/sbus@0,10001000/espdma@5,8400000/esp@5,8800000/sd@0,0 (sd0):
        Corrupt label; wrong magic number
format> quit
# reboot

Ok. Disk formatted. Time to get down to business for real.

Installation

At the OpenBIOS prompt, I booted to the CD normally:

boot cdrom:d -v

This time, no "Corrupt label" errors.

The installer ran, I chose to install in English. Then it wanted a location to copy the installation software to. I allowed it to repartition the disk, accepted the default 512MB swap slice size, and allowed it to start the slice at beginning of the disk. The installer then copied itself to the disk and attempted to reboot.

But, since I'd disabled auto-boot, I had to tell it where to boot:

boot disk:b -v

At this point, the installer would sometimes just run, or other times crash. If it crashed, I could restart it by running disk0_install, and then it would run fine. I never figured out exactly why it was crashing sometimes. Later I read that maybe giving QEMU more memory (using -m 256) might have fixed the problem. I never tried that though. Your mileage may vary.

When the installer came up, it was intuitive. I supplied various networking parameters, time zone parameters, and a root password.

Then, after the weird (but familiar) Web Start message...

Solaris Web Start will assist you in installing software for Solaris.

(Whatever Web Start means)

...the installer asked a few more intuitive questions and asked for Software Disc 1.

If I'd been installing on real hardware, the CD tray would have been ejected and it would have been clear that I needed to change CD's. Everything being virtual, I didn't realize that I needed to do anything, and forgot that there are like 5 CD's with software on them, so it took me a while to realize that I needed to switch into the QEMU monitor and change CD's.

(qemu) change scsi0-cd2 solaris-9-sparc-software-disc1.iso

But after I figured that out, and hit return, and went through a few more prompts, the installer continued with a crude (but familiar) progress bar.

Woohoo!

Then it wanted Software Disc 2.

(qemu) change scsi0-cd2 solaris-9-sparc-software-disc2.iso

And there was more of the same, before it finally tried to reboot again.

Once again, I had to specify the disk and partition to boot to:

boot disk:a

Though this time to a different partition, and without the verbose flag.

The installer wasn't done yet though, it wanted the Supplements CD.

(qemu) change scsi0-cd2 solaris-9-sparc-software-supplement.iso

And it installed a some stuff that I recognized like Java3D and OpenGL, but then also stuff like PC Launcher, PCfileviewer, RSC, SunForum, SunHSI PCI, and SunVTS. What are these things?

After a few more prompts, it tried to reboot again, and again I told it to boot to disk:a...

boot disk:a

But this time, everything started up for real.

1-solaris-9-sparc-firstboot

I could log in as root, and networking worked on the first try!

Post-Install Tweaks

So, I now had a working Solaris 9 Sparc installation. I hadn't seen such a thing in more than 10 years.

But the system as it was, wasn't all that useful yet. I wanted languages (well, other than java) and compilers and web servers. Companion CD to the rescue...

I swapped the CD again:

(qemu) change scsi0-cd2 solaris-9-sparc-software-companion.iso

And, after a bit of trial and error, figuring out which device was the CD, mounted it and ran the installer.

mount -F hsfs /dev/dsk/c0t2d0s0 /cdrom
cd /cdrom
./installer

But it failed miserably because it's requires a graphical environment to run. Fortunately, I had one of those, just not on that machine.

Locally, I gave the VM permission to use my display:

xhost +192.168.123.191

And then ran the installer, piping it's display here:

DISPLAY=192.168.123.7:0 ./installer

Success!

2-solaris-9-sparc-companion

It took a while to start up, and there were a bunch of intuitive prompts, and it took a really long time to install, but eventually, it did, and I had most of the utilities I'm so accustomed to having.

I did need to tweak a few things though.

First, sudo had the wrong permissions.

chmod u+s /opt/sfw/bin/sudo

Second, even though I had a 36G disk, the root division is only 2G, and it was getting full, so I moved /opt/sfw to /export/home (which had a ton of space), and symlinked it back to /opt.

cd /opt
mv sfw /export/home
ln -s /export/home/sfw sfw

I also wanted to add some paths to everyone's PATH, but vi didn't work very well. The TERM environment variable is set to "sun" by default, but even setting it to xterm, vt100 or vt220 didn't help. I'm not sure why exactly, but I was eventually able to get it to work by running a dtterm (which supports the "sun" terminal type) and editing /etc/profile from there:

DISPLAY=192.168.123.7:0 /usr/dt/bin/dtterm

And then in that terminal:

vi /etc/profile

I added the following, near the top:

PATH=$PATH:/usr/ccs/bin:/usr/ucb:/usr/openwin/bin:/usr/openwin/demo:/usr/dt/bin:/usr/sfw/bin:/opt/sfw/bin
LD_LIBRARY_PATH=/usr/sfw/lib:/opt/sfw/lib
...
export LOGNAME PATH LD_LIBRARY_PATH

I also created a user for myself:

useradd -m -d /export/home/dmuse dmuse
passwd dmuse

And I gave myself some sudo privileges, though the sudoers file was in /usr/sfw/etc/sudoers rather than /etc/sudoers.

ssh appeared to be running, but I enabled X11 forwarding in /etc/ssh/sshd_config

X11Forwarding yes

...and restarted it.

/etc/init.d/sshd stop
/etc/init.d/sshd start

I was then able to ssh into the VM, and run X apps without having to fiddle with the DISPLAY variable.

All right!

The Web

Solaris 9 comes with some version of Apache. I copied the example config file.

cd /etc/apache
cp httpd.conf-example httpd.conf

And tweaked a few things (in httpd.conf):

#ServerName 127.0.0.1
...
AddHandler cgi-script .cgi
...
Options Indexes FollowSymLinks MultiViews ExecCGI
...
DirectoryIndex index.html index.cgi

And restarted apache.

/etc/init.d/apache restart

And wrote a quick and dirty CGI (/var/apache/htdocs/env.cgi)

#!/bin/sh
echo "Content-type: text/plain"
echo
echo
env

And it worked!

3-solaris-9-sparc-web

Solaris 9 also comes with Netscape Communicator 4.78 which tries to go to www.sun.com and crashes unless you manage to stop it in time. It appeared to support png's but that's about the most modern thing it could do.

Development

Ok, so all that had been fun, but could I get any of my software to run?

Turns out yes. I'd gotten it to work on Solaris 9 x86 long ago, and wrung out all the endian, alignment, and 32/64-bit issues over the years. The companion CD provided all the dev tools that I needed, including gcc 2.95.3. Pretty old, but good enough.

I was able to build and install Rudiments and SQL Relay and access Oracle 12 via an SQL Relay server on another machine.

4-solaris-9-sparc-sqlrelay

The companion CD came with MySQL too, Relay found it, and built a connection plugin for it, but it was such an old version that it was incompatible with modern MariaDB:

mysql_real_connect failed: Client does not support authentication protocol requested by server; consider upgrading MariaDB client

Yeah, pretty old:

/opt/sfw/mysql/bin/mysql  Ver 11.15 Distrib 3.23.43, for sun-solaris2.9 (sparc)

I have various versions of Oracle for Sparc Solaris. One of them probably works on Solaris 9.

Fun for another day.

Conclusions

Having used Solaris 9 on x86 quite a bit, it was very familiar on Sparc. Backspace is delete and /home is /export/home, but I'm used to that. The most difficult thing to deal with was the inability to use vi on the console, and that wasn't too big of an issue. Other than that, it does slam the host cpu, so I can't just leave it running, but I think it does that on real hardware too, so it's not unexpected.

I've been collecting versions of Sparc Solaris for a few years now, in the hopes that someday QEMU/OpenBIOS might support it. Yeah, I know you can use the Sun ROMs but the dubious legality of those isn't my style. It's really great to find that it all works now. I'll have to try out some of the other versions soon too.

Friday, May 8, 2015

UnixWare 7.0.1

This one's been a long time coming.

Introduction

Months and months ago I acquired a scattered collection of old SCO software from a seller on eBay. It included a random hodgepodge of all kinds of stuff; a dozen manuals, dozens of CD's and floppies, packs and packs of licenses... It took a few hours just to sort out what went with what. Among the haul: a complete copy of OpenServer 5.0.0, two complete copies of OpenServer 5.0.2, and the license pack for UnixWare 7.0.1, but no UnixWare media. Seriously?! What happened to the media?

At the time I was a bit frustrated, but not terribly because I had plenty to play with. In fact, I still haven't gone through most of it. And, so I just sat on the licenses and waited.

Months later, someone was selling a copy of OpenServer 5.0.5 and the photos showed media for UnixWare 7.0.1 included in the pile. This wasn't too surprising though. SCO used to give away promotional copies of UW 7.0.0 and 7.0.1 with OSR 5.0.5. In fact, that's how I was first exposed to it, way back. They didn't give away licenses though. You could install everything, but you only had about 60 days to play with it before the demo timed out.

But, I had an actual license! And a few days later I had media to go with it!

Score.

Still, it was a few weeks later before I had enough time on my hands to start trying to get it to run. That turned into a bit of an adventure too.

Here's how it went.

UnixWare

Actually, first... What the heck is UnixWare?

I'm a little fuzzy on the history actually.

Novell and somebody else (AT&T?) got together at some point in the early 90's to put together a version of SVR4 Unix that could interoperate with Novell NetWare, for some reason. I'm not sure what the motivation was, actually. But they succeeded, released a few versions, and eventually got the attention of SCO. At the time, SCO had OpenServer, which was an SVR3 implementation. UnixWare had feature-parity with OpenServer, aside from the configuration interfaces and back-compatibility with older SCO platforms. I guess they figured it would be easier to buy an SVR4 platform and port their goodies to it than rework OpenServer.

Whatever the motivation, SCO bought UnixWare, or licensed it, or something, and made several 2.x releases. In the late 90's Sun released Solaris version 2.7 as "Solaris 7". SCO was coming out with another version of UnixWare around then, and rather than name it 3.0, they named it "UnixWare 7" as well.

SCO made a few more releases and tried to get everyone to migrate from OpenServer to UnixWare, but there weren't a lot of takers. SCO's users had legacy apps galore that still needed to run, like apps written in FoxPro for 3.2v4.2, and apps written in Microsoft Basic for Xenix, and instances of Oracle7. They could recompile their modern C/C++ apps, but FoxPro, MS Basic, and Oracle7 didn't even run on UnixWare. Those apps would have to be completely rewritten in different languages or migrated away from SCO platforms. No thanks.

SCO tried a few stunts. Unixware 7.1.2 was released as "OpenUnix 8", for example, but people stuck with OpenServer. Eventually, SCO released OpenServer 6, which piled everything that everybody liked from OpenServer (including back-compatibility) onto a UnixWare base. It looked and felt like OpenServer, ran all the legacy apps, but it was UnixWare underneath.

Technologically, they'd finally achieved what both they and their users wanted, but it might have been too late. When OSR 6 was released, SCO's courtroom shenanigans had turned the world against them. I wonder how many takers OSR6 actually has.

But I digress. My adventure begins about half of that story ago, with version 7.0.1, right about when SCO got serious about trying to get everyone to migrate off of OpenServer.

Here's how it went. (For real, this time)

VM Configuration

I'd had good success with various OpenServer's in VMware, but UnixWare is a totally different breed. Turns out it'll run too, but only if you're careful about how you do it.

In particular, UnixWare can't make heads or tails out of VMware's IDE emulation. SCSI's the way to go. And, in particular, BusLogic SCSI. Don't even fool around with LSI, it's not going to work. The hard drive has to be SCSI, and so does the CD ROM.

But, UnixWare 7.0.1 predates bootable CD's by a year or two, so you've got to come up with some boot floppies. None came with my CD, but it turns out there are floppy images on the CD. Not in an intuitive place, mind you, but they're on there. It took a little bit of digging, but I found them under INFO/IMAGES. The files are named BOOT.IMA.1 and BOOT.IMA.2. There are other disk images there too, but they're not necessary for VMware installation.

Installation basically involved adding a floppy, aiming it at BOOT.IMA.1 and powering on the VM.

Right away I was greeted with the familiar UnixWare splash screen.

unixware 7.0.1 - 1. splash

And the installation was kind-of straightforward.

I answered a few intuitive questions, swapped floppies when prompted, answered a few more intuitive questions, eventually answered some less-intuitive ones...

Do I want to enter the DCU? Maybe. What's a DCU? Ahh, "Driver Configuration Utility." I'm not sure. As it turns out, letting it auto-configure is the way to go.

The installer detects the emulated PCnet ethernet card but the driver has a lot of trouble with it. The "IBM PCI Ethernet" driver works well though.

Configuring TCP/IP is intuitive, except that you can select the ethernet frame format. The default of ETHERNET_II works well though.

Since this is UnixWare, it also supports IPX, and as it turns out, IPX can't not be installed. That's OK though, the defaults seem to work.

Unfortunately, NIS also can't not be installed. I gave my installation an NIS Type of "client" and NIS domain of "firstworks" and as I have no NIS servers anywhere, left the optional NIS servers blank.

Just about everything else was totally intuitive, and before too long, the installation proceeded unattended.

unixware 7.0.1 - 2. install

When the installation was complete, I had to disconnect the cdrom and floppy before reboot. Failing to do so resulted in several very insistent messages.

On reboot, I got a funny message about swap being smaller than memory. I'd given my VM a monstrous 256MB of RAM. In the late 90's, 32mb would have been a lot.

I also got some apparently-unimportant SCSI errors and a bunch of messages about configuring NetWare and other things. Eventually I had to configure the mouse and then it asked me to insert disc #2 to continue installing the system. I only had 1 system disc though. The other disc was the development system. I tried that disc, but apparently that's not what it was looking for. It also didn't like it when I gave it the first disc again. Oh, well. Eventually I just hit F8 to defer that step. It then prompted me for disc #3 and I just deferred it again. I might have chuckled a little too.

Fortunately it didn't ask me for disc #4, but instead started doing boot-type things and eventually presented me with a CDE login.

unixware 7.0.1 - 3. first boot

All right!

Post-Install Tweaks

Logging in got me to a familiar CDE desktop.

unixware 7.0.1 - 4. cde desktop

But, not a lot worked. I could poke around the system, but it was an island unto itself. I couldn't ping, ftp or telnet to anywhere.

This took days to figure out.

It turned out that there were two big problems. First, though above I say "the PCnet driver sucks, use the IBM PCI Ethernet driver", I hadn't done that at this point. I was still using the PCnet driver, which sucks. Tremendous packet loss.

After switching to the IBM PCI Ethernet driver, things were a lot better, but I couldn't yet tell that they were.

It eventually turned out that NIS was wreaking widespread havoc. I never figured out exactly what was going wrong, but my guess is that the /etc/hosts file is an NIS-able file, and since there were no NIS servers anywhere, all DNS lookups had to wait for an NIS timeout before proceeding past the /etc/hosts file. The effect was that if I tried to ftp or telnet to somewhere, it would hang for quite a while before actually trying the address. Telnetting or ftp'ing in from other hosts would hang too, during the reverse-lookup. If I waited, each these operations would eventually succeed, but in the beginning, I wasn't patient enough to realize this.

Once I did determine that NIS was the culprit, disabling it was still fairly tricky. I had to edit /etc/init.d/nis, set the domain to an empty string early in the file:

domain=""

And then also comment out the setting of the variable "d" because the domainname program would somehow still return my domain, even though domain has been set to "".

        #d=`domainname`

But! After that, and a reboot, everything started working really well.

I was able to download a bunch of useful software from the SCO Skunkware FTP site, but unfortunately none of it worked without first installing a libc patch. Apparently the shared version of the libc that ships with 7.0.1 is missing some symbols. Weird. The patch fixed it though.

So I installed that patch, and several bits of software, like bash, sudo, gzip, etc., all using the standard SVR4 package management tools, which are worth discussing a bit.

Package management on UnixWare, while SVR4-compliant, is (not-unexpectedly) archaic. There's no dependency-resolving "yum install xxx" or "apt-get install yyy". You just download a .pkg file, hope it doesn't have any not-too-obvious dependencies, and install it in a completely unintuitive manner.

For example, if you download a file named xxx.pkg, then you'd install it using:

pkgadd -d `pwd`/xxx.pkg

The -d meaning "the packages are contained on this device or in this directory".

What? I'm trying to install the package xxx.pkg, why do I declare it to be the directory containing packages?

Well, the pkg file is actually a cpio archive.

Aaaand?

In antiquity, packages weren't distributed individually. Dozens of expanded package trees were cpio'ed en-masse to a tape, and the tape was distributed. If you aimed pkgadd at the tape device, like pkgadd -d /dev/rmt0, or similar, it would tell you what packages are available and allow you to install one or more of them at a time. If you already knew which one(s) you wanted to install, you could supply the name(s) on the command line too.

As hard drives got bigger, someone got the idea to distribute packages as individual files, rather than collected up on tape. And, since on unix, everything is a file, pkgadd doesn't care whether the argument to -d is a file or a device, as long as it's a cpio archive.

Clever. Agreed. But completely unintuitive.

In fact, the man page for pkgin doesn't help much either. It goes on and on about how -d can refer to a directory, or a pipe, or a device (like a tape), and about the default locations, and other stuff, and unless one notices that -d can refer to a "directory, file, or named pipe", and it occurs to one that file might mean .pkg file, and that .pkg files might simulate tapes, then one might overlook that option altogether.

That's what I did, way back, when I first encountered UnixWare, and it drove me nuts. Even after I figured out the whole file=tape thing, I still couldn't get it to work because the fact that the argument to -d has to be the full pathname to the .pkg file is not mentioned ANYWHERE, AT ALL!

Somewhere I found an example though, and had to laugh, in hindsight, at how obvious it should have been.

Remote Access

It was fun using the CDE desktop and all, but not practical in the long run. Telnet worked, but it's difficult to automate.

OpenSSH packages were available, but they were apparently built for a later version of UnixWare. sshd kept complaining about libc missing the strlcmp function.

No problem though, I've built OpenSSH for a bunch of old Linux and Solaris platforms. Maybe I could get it to work on UnixWare too.

I had to install a few packages first though. OpenSSL for starters, and it required prngd and zlib. Also, I didn't have a lot of faith in the native development system, so I installed gcc-2.95.2pl1 too.

And, it turned out, building OpenSSH-3.4p1 from source was quick and easy.

./configure --prefix=/usr/local/openssh-3.4p1
make
sudo make install

Configuring it was familiar from OSR5 too.

sshd_config:

X11Forwarding yes
...
UsePrivilegeSeparation no

And I created an init script at /etc/init.d/sshd...

#!/bin/sh

case "$1" in
  start)
        /usr/local/openssh-3.4p1/sbin/sshd
        ;;
  stop)
        kill `ps -efa | grep sshd | grep -v grep | cut -c10-15`
        ;;
  *)
        echo $"Usage: $0 {start|stop}"
        exit 1
esac

exit 0

...made it executable, and set it up to run at boot:

chmod 755 /etc/init.d/sshd
cd /etc/rc2.d
ln -s ../init.d/sshd S98sshd

The last step was to uninstall OpenSSL. Oddly, for both UnixWare and OpenServer, the OpenSSL packages only contain static libraries. This creates problems for my software, and since once OpenSSL is compiled it no longer needs the OpenSSL libraries, it's safe to remove the packages.

Reboot!

unixware 7.0.1 - 5. ssh

Remote access!

But, it wasn't perfect. ksh on UnixWare is all but unusable over SSH. Neither backspace nor delete work. Bash worked though, so it was easy enough to work around.

I could ssh out too.

Looking good.

File Transfer

As mentioned earlier, ftp worked pretty well. OpenSSH came with scp and sftp. I'd installed gzip earlier. UnixWare comes with tar and cpio, though neither can gunzip on the fly. Packages for GNU tar, as well as zip, unzip and bzip were all available from skunkware.

After installing a few more packages, I was well equipped to transfer files.

The Web

No system is complete without being able to get on the web, right?

Well...

UnixWare 7.0.1 came with Netscape Navigator 3.0.4. Cutting edge for 1999, but it struggled pretty badly with today's web.

I installed wget and lynx packages from skunkware though. Good old lynx.

On the server side, things were slightly better.

UnixWare comes with the Netscape FastTrack web server. To turn it on, you have to log into a CDE desktop, run SCOAdmin and select Netscape Server Admin. This launches Netscape and aims it at FastTrack's web-based configuration interface.

After logging in as admin and using the password for the root user, you get this snazzy UI.

unixware 7.0.1 - 6. netscape fasttrack web server

The toggle switch on the left is for the web server on port 80. Switching it to the "ON" position started up the web server, and it was immediately accessible from my local browser.

unixware 7.0.1 - 7. default web page

Oh yeah!

It took a bit of digging around in the interface, but I figured out how to enable CGI's and a quick "print out the environment variables" script worked.

Skunkware packages for Apache were also available, but I didn't try them. Maybe later.

Graphics

The CDE login and desktop was enabled by default.

It can be disabled using:

scologin stop

And re-enabled using:

scologin start

When disabled, running startx from the command line, surprisingly, didn't start a CDE session though.

unixware 7.0.1 - 8. non-cde desktop

Weird. I'd apparently never done that before because I didn't recognize the desktop from OpenServer, UnixWare, or anywhere else.

It turns out that it is possible to get a CDE session with startx though, if you first create a .xinitrc file containing:

dtsession
Development

There were Skunkware packages for GNU make and CVS. What about the compiler though?

Well, I'd intalled gcc earlier to build OpenSSH, but I had 2 CD's. One with the OS, and the other with the development system. And, I had actual licenses for it in the same pack as the OS licenses.

The OpenServer development system isn't super mature, at least as of OSR 5.0.5. There's no support for 64-bit integers, and the C++ compiler is just a c-front and both quirky and not terribly capable with templates, even simple ones. I wasn't sure about the UnixWare development system though. Back in the late 90's, I never got around to playing with it.

Now would be a good time, I figured.

The CD had a README on it, so I followed its instructions to run the Development System's installer.

unixware 7.0.1 - 9. dev system

Turns out, it installed the C and C++ compilers, a ton of libraries and headers, and Java version 1.1.3. Whooo! 1.1.3. That's a blast from the past.

The UnixWare C++ compiler was a lot more capable than it's OSR cousin. In fact, calling them cousins isn't even inaccurate. It didn't appear to be a c-front, and compiled my Rudiments library with only a few tweaks.

But, when I got around to running the tests, I discovered one of its deficiencies. Like earlier versions of gcc, it didn't handle nested template invocations correctly. Darnit.

There were issues with variadic macros too, but I could just comment those out. The template thing was a show-stopper.

That's OK, there's always gcc, right?

Not exactly.

I'd installed gcc-2.95.2p1 earlier, and it was able to compile all of my software quite easily, but when I tried to run just about anything, I'd get a segfault, almost immediately.

I tracked it down to a dynamic linker error of some sort. Basically, my charstring class had an overloaded compare() function with two different signatures, and attempts to call the one with 3 arguments would crash.

I'd actually seen this same error on OpenServer 6 years ago, and eventually fixed it by compiling gcc-2.95.3, with some SCO patches from source, using the native compiler.

No such luck though. Same error.

I tried other versions of gcc-2.95.x, both packages and builds from source. Nothing worked. At best I'd get the same error. At worst, the compiler would segfault. Eventually I gave up.

And that's where it sat, for months. Sometimes you've got to get away from a problem for a while though. This afternoon I took another look at it. I noticed that there were Skunkware packages for older versions of gcc, like 2.8.0 and the egcs compilers. I couldn't remember if I'd tried them before or not.

I must not have because the compiler from the egcs-1.1.1 package ran without segfaulting, and produced working code.

The rudiments tests mostly ran fine. SQL Relay built and the command line client ran.

unixware 7.0.1 - 10. sqlrsh

SQL Relay supports lots of languages and database backends, but unfortunately not many of them are well supported on UnixWare 7.0.1.

I couldn't get it working with Perl, Python, PHP, TCL, Ruby or Java. In some cases, even the oldest language package was clearly meant for a newer version of UnixWare. In other cases, there were odd header quirks that prevented them from compiling cleanly. I'm not sure what the issue is with Java yet. By rights, it ought to work but I get weird errors at runtime.

Things weren't any better on the database front. PostgreSQL was the only database available, and the only package that would install was for version 6.5.3 - older than the oldest version I've ever tried before. UnixODBC was available, but the package was for a newer version of UnixWare.

SQL Relay builds against that old PostgreSQL, but there's some shared-memory-related issue that keeps it from running.

Ehh, who knows. There's always some quirky thing with every platform.

That's where I left it.

Quirks and Endearing Features

So, what's weird about UnixWare 7.0.1?

Not all that much, actually, especially given it's age.

The most significant thing is an issue with the filesystem. It's very easy to run out of inodes, waaaaay before you run out of space:

bash-2.05a$ df -v /
Mount Dir  Filesystem           blocks      used     avail  %used
/          /dev/root          16000740   1538304  14462436   10%
bash-2.05a$ df -i /
Mount Dir  Filesystem            iused     ifree    itotal %iused
/          /dev/root             55862      9674     65536   85%

Multiple times I had to tar and zip stuff up and delete directory trees to free up enough space to untar something else. It looks like they're using a 16-bit number for the number of inodes. I guess back in the day, when a 1.6G hard drive was gigantic, file systems would have been smaller, there would have been more of them, and that was enough. But I'd swear that one of the selling points of UnixWare was support for petabyte and exabyte sized volumes. Maybe the filesystem does support more inodes, but there's no option for it at install time, and 65536 is a small default.

Other quirks...

The ksh shell has trouble over ssh, neither backspace nor delete work.

The halt and reboot commands are in /usr/ucb, which isn't in the default PATH.

Adding directories to the PATH is remarkably difficult. /etc/profile exports PATH before setting it. Lord knows where it's getting the default PATH from. Also, CDE sessions don't appear to source /etc/profile at all. Again, it's unclear where their PATH is set.

socklen_t is defined but a lot of network functions use size_t instead.

The chmod command accepts a shorter argument list than commands like cp. I had to update several Makefiles to work around this.

If you make a change that requires a kernel relink, the relink is done during the reboot, not before. As such, there's no unix.old to boot to until after the boot fails. Actually, for that matter, it's not clear how to get to a bootloader prompt.

Ha!

Strange, old unix.

Gotta love it.