Saturday, December 24, 2016

Updating Your App to Support OpenSSL 1.1.0


OpenSSL 1.1.0 introduced several API-breaking changes, and with distributions starting to use it, I found myself scrambling to get my software working. I found various docs stating that 1.1.0 changed things, but nothing that clearly stated: "if you have this now, then change it to this..." So, that's what I'll do here, and hopefully it will help you if you find your app in the same situation.

Deprecated "method" Functions

Lots of "method" functions like TLSv1_server_method() and SSLv23_client_method() and TLSv1_2_method() are deprecated in OpenSSL 1.1.0, and if you compile your software with -Werror, then that presents a problem.

All you can call now is one of:

  • TLS_method()
  • TLS_client_method()
  • TLS_server_method()

If your code allowed the user to select a SSL/TLS mode then it can't do it by calling a mode-specific function any more. It'll have to select the mode some other way that I'm not aware of. The new methods negotiate the most modern mode supported by the client and server. It's not immediately clear how to override that, or if you can. If someone knows a way, please leave a comment below.

It's easy enough to replace function calls, but if you want to support older versions, then you have to add tests for the old and new functions to your configure script and add all kinds of #ifdefs, and so forth. Actually, since various methods like these have been added over the years, you may already have such tests and #ifdefs (I did) and it won't be so bad.

Deprecated ASN1_STRING_data
unsigned char *ASN1_STRING_data(...);

has been replaced with

const unsigned char *ASN1_STRING_get0_data(...);

This is a straightforward replacement, but note the const'ness of the return value of the new function. If you're using C++ then you may have to change the type of the variable it's being assigned to (or un-const it).

Again, this is a brand new function, so, configure tests and #ifdefs.


The EVP_PKEY struct (public key struct) is opaque now. The header files declare that an EVP_PKEY struct exists, but the definition is hidden up in the library. The error you get (with g++ at least) is fairly obtuse though, complaining about a forward declaration and an invalid use of an incomplete type. The error just means that the struct is opaque.

Fortunately there are a bunch of functions that return the various components of the struct.

I had to replace code to get the key type like:

EVP_PKEY *pubkey = X509_get_pubkey(c);
int t = pubkey->type;

with code like:

EVP_PKEY *pubkey = X509_get_pubkey(c);
int t = EVP_PKEY_base_id(pubkey);

Note that there IS an EVP_PKEY_type() function, but it does NOT return the "type" member of the struct. Ha! Not super intuitive, but the change is still straightforward.

Similarly, I had to replace code to get the key itself like:

EVP_PKEY *pubkey = X509_get_pubkey(c);
const unsigned char *key = pubkey->pkey.ptr;

with code like:

EVP_PKEY *pubkey = X509_get_pubkey(c);
const unsigned char *key = EVP_PKEY_get0(pubkey);

I'm not sure when those functions were introduced, but they appear to be available in OpenSSL 1.0.x but not in OpenSSL 0.9.x, so again, if you want to support really old versions, then you'll have to add configure tests and #ifdefs.

Opaque X509

The X509 struct is also opaque now. Like the EVP_PKEY struct, you'll have to use functions to access its members.

The one case I needed was slightly trickier though.

In the past, I was getting the signature algorithm name like:

X509 *c = SSL_get_peer_certificate(ssl);
char sigalg[256];

But I had to change that code to:

X509 *c = SSL_get_peer_certificate(ssl);
char sigalg[256];

Which is a little contrived. In the old code, c->sig_alg->algorithm is an "obj", and I could pass it directly into OBJ_obj2txt. In the new code, I have to get the "nid", convert it to an object, and pass that in. Ideally I'd just get the object to begin with, but there doesn't appear to be an obvious way to do that.

As a side note, the function name X509_get_signature_nid() was a little confusing at first, because it appears to return the "nid" of the signature _algorithm_ as opposed to the signature itself. But I guess the signature itself wouldn't have a "nid", so it makes sense, but it wasn't immediately intuitive to me.

Again, X509_get_signature_nid() was introduced sometime in the 1.0.x series, so you'll have to add a test for it and #ifdefs if you want to support older versions.


The opacity of the structs ought to help OpenSSL be more maintainable in the future. They ought to be able to change the code at will now, without breaking the API or ABI, and that's always good. I did the same kind of thing with my Rudiments library way back, and recently to SQL Relay for that exact reason. Hopefully it will work out well for both of us.

I'm not sure about the TLS_*_method() thing though. It seems like there really ought to be a way to choose a specific SSL/TLS mode. Perhaps there is though, and I just didn't see it. Somebody let me know if they figure that one out.

Saturday, January 9, 2016

Building Ruby From Source With Microsoft Visual Studio


Ruby packages are available for Windows, and they can be used to develop Ruby applications, all day long. However, they lack the necessary files to develop Ruby modules in C/C++ and unlike PHP, where you can fairly easily shoehorn the headers into an already-installed package, it's not so easy with Ruby. In fact, to get a hold of the components to shoehorn in, you have to do an entire build from source, so you might as well just do that to begin with.

Fortunately, it's not too difficult.


For this effort you will need:

  • Microsoft Visual Studio
  • The Ruby source code.
  • A few unixish commands - gunzip and tar. These can be provided by a gnuwin32 installation, a cygwin installation.
  • A text editor. Notepad or vi will do.

The Process

I last did this with Ruby 2.2.0, so I'll use it in this example.

The Ruby source code comes as a tar.gz file. There aren't any .zip files available, or if there are, I'm not sure where to find them. So you'll have to use some unixish commands to extract it.

Move the ruby sources to somewhere convenient and use the unix tools to extract it:

gunzip ruby-2.2.0.tar.gz
tar xf ruby-2.2.0

This should create a directory named ruby-2.2.0

Open the appropriate Visual Studio command prompt. If you plan on building a 32-bit version of Ruby, then open the x86 command prompt. If you plan on building a 64-bit version of Ruby, then open the x64 command prompt.

Navigate into the ruby-2.2.0 directory that was created by the extraction process above.

To get everything to compile and install correctly, I had to edit the file tool\rbinstall.rb and change line 714 to read:

rescue LoadError

This may or may not be necessary with a newer version of ruby, or the line to edit might be different.

Now, configure the build.

For a 64-bit build, run:

win32\configure.bat --prefix=C:\Ruby --target=x64-mswin64

For a 32-bit build, run:

win32\configure.bat --prefix=C:\Ruby --without-ext=fiddle 

(The --without-ext=fiddle command just disables the "fiddle" extension, which I couldn't get to build on 32-bit Windows, for some reason. Your mileage may vary.)

To actually build Ruby, run:


(This will run for a while.)

When it's done, install Ruby by running:

nmake install

Everything should get installed under C:\Ruby

To make the ruby programs generally accessible, add C:\Ruby\bin to the system PATH as follows:

  • Open the Control Panel
  • Search for Environment
  • Click the link for "Edit the system environment variables"
  • Click "Environment Variables"
  • In the System Variables pane, scroll down and click on Path
  • Click Edit...
  • Append: ;C:\Ruby\bin to the Variable Value.
  • Click OK
  • Click OK
  • Click OK

And that's it. Ruby is now built, installed and accessible.

Hopefully this helps someone out.

Good luck developing your Ruby modules on Windows.

Generating a Missing Import File From a DLL


Occasionally I run into software packages for Windows that are missing some of the developer bits. One bit that occasionally gets left out is the import library. The headers are there, the .dll is there, but the .lib file is missing.

It's not impossible to generate a .lib from a .dll though.

Here's how.


You'll need a few things to do this:

  • Microsoft Visual Studio
  • A few unixish commands - echo, cat, and cut. These can be provided by a gnuwin32 installation, a cygwin installation, or just a second linux or unix system.
  • A text editor. Notepad or vi will do.

The Process

I ran into this issue with Active Perl 5.20, so I'll use it in this example. Adjust accordingly.

First, dump the list of exports from the dll into an file. Open a Visual Studio command prompt and run some commands similar to the following:

cd C:\Perl64\bin
dumpbin /exports perl520.dll > exports

(adjusting for the path and file name of your .dll)

Now, edit the exports file.

Trim off everything prior to the first function definition. In the perl dll, that was a line like:

          1    0 000295C0 ASCII_TO_NEED

Trim off everything after the last function definition. In the perl dll, that was a line like:

       1249  4E0 001271B0 win32_write

Now, use the unixish commands to create a .def file.

(If you're doing this on a second machine, then you'll need to copy the exports file over to it and then, afterwards, copy the .def file back.)

echo EXPORTS > perl520.def
cat exports | cut -c27- >> perl520.def

Now, use Visual Studio's lib command to build the .lib file.

lib /def:perl520.def /OUT:perl520.lib /MACHINE:X64

Note: In this example, we had a 64-bit DLL. To build from a 32-bit DLL, use the /MACHINE:X86 flag instead.

Currently the .lib file is sitting in the bin directory, but .lib files generally need to go in a lib directory or some other place. So, move it now.

move perl520.lib ..\lib\CORE

You can now clean up if you like.

del exports
del perl520.def

And that's it. You can now link against the .lib file and your app will load the .dll at runtime.

Hopefully these instructions help if you ever find yourself in a similar situation.

Shoehorning Development Headers Into PHP for Windows


In years past, I've had trouble building PHP modules on Windows because, though PHP packages are available for Windows, they lack the necessary header files to build PHP modules. One solution is to build and install PHP from source, but that takes forever, uses a ton of space, and has a long list of prerequisites. It turns out that it's actually faster and easier to shoehorn in a set of development headers into the already-installed PHP package.

Here's how.


You will need the following items:

  • A Linux or Unix machine, with a working compiler toolchain. (Actually, a cygwin installation with a working compiler toolchain might suffice but I've never tried it.)
  • The PHP package for Windows.
  • The source code for the same version of PHP as the package.
  • The same version of Microsoft Visual Studio that was used to build the PHP package. The version of VS is discernible in the package filename. For example, was built with Visual Studio 14 (aka Visual Studio 2015).
  • Several bits of software from the gnuwin32 project:

On the Windows machine, install the PHP package into C:\PHP and install Visual Studio.

Install the gnuwin32 software:

  • Create C:\gnuwin32
  • Extract each zip file by right-clicking on it and selecting Extract All.
  • Move the contents of the folders created during the extraction into C:\gnuwin32
  • Add C:\gnuwin32\bin to the PATH environment variable.
    • Open the Control Panel
    • Search for Environment
    • Click the link for "Edit the system environment variables"
    • Click "Environment Variables"
    • In the System Variables pane, scroll down and click on Path
    • Click Edit...
    • Append: ;C:\gnuwin\bin to the Variable Value.
    • Click OK
    • Click OK
    • Click OK

If you can run bison or flex from the command prompt, then it worked.

Building the Headers

Copy the source code to the Linux/Unix machine and extract it. Then build and install the headers somewhere convenient, like your home directory. In this example, we'll use /home/dmuse/php though it could be anywhere.

tar xf php-5.6.17.tar.bz2
cd php-5.6.17
./configure --prefix=/home/dmuse/php
make install-headers

Now, copy the entire /home/dmuse/php/include directory to the Windows machine. If the linux/unix machine is visible as a Windows share then you can just drag the directory across the network. Otherwise you can zip or tar it up, contrive to copy it over using scp, ftp, http, or something else, and then extract it by right-clicking on it and selecting Extract All.

Move the includes folder to C:\PHP\dev such that there is now a C:\PHP\dev\include folder. Verify that C:\PHP\dev\include\php exists and that C:\PHP\dev\include\include didn't get created by accident, as sometimes the top-level folder gets duplicated during extraction.

The PHP package now contains most of the headers necessary to develop PHP modules, but a few windows-specific files need to be generated and installed.

Extract the PHP source somewhere convenient on the Windows machine.

Open a Visual Studio command prompt. Make sure to open a prompt for the same architecture as the PHP binary package. For example, if the package is an x86 package, then make sure to open the x86 prompt. If the package is an x64 package, then make sure to open the x64 prompt.

Change directories to the folder that was created when you extracted the PHP source and run the following commands.

cscript /nologo configure.js

Then, copy the Windows-specific headers that were built by those commands into the PHP tree:

copy main\config.w32.h C:\PHP\dev\include\php\main
copy TSRM\tsrm_config.w32.h C:\PHP\dev\include\php\TSRM
copy Zend\zend_config.w32.h C:\PHP\dev\include\php\Zend

And that's it. You have now shoehorned development headers into the PHP package for Windows. Good luck building your PHP module!