Code signing in a remotely working team?

[A.M. Hoornweg 144]

Hello all,

 

We're a small company, our Delphi developer team works largely remotely and I'm one of the members who actually lives in a different country from the rest of the team.

 

We must frequently release updates of our various software products. Each team member uses Finalbuilder and Signtool to automate the process of compiling, code signing and generating setups. Some of our products consist of dozens of executables and dll's so the automation of the build-and-sign process is a must-have.

 

Our Digicert EV code certificate expires in February 2025.

 

We're now faced with the problem that certificate providers seem to expect you to have the certificate on a USB device which is kinda unpractical if developers work remotely from different countries. We need common access to the certificate and we need to be able to automate the signing process.

 

I'd very much like to hear from other developer teams who are in the same boat, how they tackle this problem.

 

 

[Patrick PREMARTIN 68]

Hi

The USB token is not the only solution : some providers allow to use an online hosted token too.

If you can't find an other solution than USB, you can perhaps use Exe Bulk Signing ( https://github.com/DeveloppeurPascal/ExeBulkSigning ) in client/server mode to sign on a distant computer from yours, but exposing a signing software is safer on a local network or a VPN than over the Internet.

[msohn 28]

There's an excellent blog post by Vincent Parret:

 

https://www.finalbuilder.com/resources/blogs/code-signing-with-usb-tokens

[Anders Melander 1773]

2 hours ago, A.M. Hoornweg said:

Each team member uses Finalbuilder and Signtool to automate the process of compiling, code signing and generating setups.

There's your problem.

You should use a single central build server instead of delegating the build task to individual developers. If you don't have a central server which can function as a build server, at least designate one of the developers as the "build master".

[A.M. Hoornweg 144]

2 minutes ago, Anders Melander said:

There's your problem.

You should use a single central build server instead of delegating the build task to individual developers. If you don't have a central server which can function as a build server, at least designate one of the developers as the "build master".

We moved away from a build server several years ago;  we work on several independent projects and kept getting in each other's way.  

 

 

[Anders Melander 1773]

Hmm. Okay.

I obviously don't know how your build server setup was, or what build system you used, but it should have been possible to completely isolate the different projects.

Independent projects, with different developers, tools, etc. on the same build server is nothing out of the ordinary.

 

Anyway, if you are working on different projects, and don't want a centralized solution, then why not just use different certificates?

[A.M. Hoornweg 144]

1 minute ago, Anders Melander said:

Hmm. Okay.

I obviously don't know how your build server setup was, or what build system you used, but it should have been possible to completely isolate the different projects.

Independent projects, with different developers, tools, etc. on the same build server is nothing out of the ordinary.

 

Anyway, if you are working on different projects, and don't want a centralized solution, then why not just use different certificates?

The certificate is on the company name., having multiple ones would multiply the costs.

 

I'm just reading up on Digicert Keylocker, which appears to be a cloud-based solution.  If I understand correctly, Digicert keeps the USB device with the certificate and users can access it remotely.

 

https://signmycode.com/blog/what-is-digicert-keylocker-everything-to-know-about

https://www.digicert.com/blog/announcing-certcentrals-new-keylocker

 

 

[Anders Melander 1773]

33 minutes ago, A.M. Hoornweg said:

...and users can access it remotely.

...once they have signed in with an USB token.

I don't think you can get around the requirement for the signer to have some kind of physical identification device.

[Uwe Raabe 2055]

2 hours ago, Anders Melander said:

Independent projects, with different developers, tools, etc. on the same build server is nothing out of the ordinary.

Indeed. Actually having multiple developers building the same project can be the trickier part when it comes to shared resources.

 

Of course it may need some investment first - that is money as well as time - to get it working properly.

[Vincent Parrett 745]

We're working on a code signing server that supports tokens/pfx etc - allows you to do remote code signing very easily. All you need is network access to the server from a remote location (ideally over a vpn) and the client (a command line tool, which FinalBuilder will support). We're just tidying up loose ends (like the installer) before beta - hopefully in a few weeks. 

 

 

[Anders Melander 1773]

1 hour ago, Vincent Parrett said:

We're working on a code signing server that supports tokens/pfx etc - allows you to do remote code signing very easily. All you need is network access to the server from a remote location (ideally over a vpn) and the client (a command line tool, which FinalBuilder will support).

Let me guess: The client transmits the file to be signed to the server, and the server (which has the physical token) then signs the file and transmits it back to the client.

[Vincent Parrett 745]

1 minute ago, Anders Melander said:

Let me guess: The client transmits the file to be signed to the server, and the server (which has the physical token) then signs the file and transmits it back to the client.

No, that would be terribly wasteful - we calculate the digest on the client and send that to the server to be signed. 

[Anders Melander 1773]

Just now, Vincent Parrett said:

we calculate the digest on the client and send that to the server to be signed. 

Oh, nice. I didn't know you could do that.

 

It's a real PITA that we have to jump through all these hoops with the physical tokens and the fact that a solution like yours, which is basically a work around, is even possible proves that it was all in vain. But I guess somebody made a lot of money selling the tokens.

[Vincent Parrett 745]

Yeah for years we have been told (and I have been telling everyone) to automate everything.. then some numpty decides to throw in a manual spanner🤦‍♂️- and yes, the cost of certificates increased a lot - way more than the cost of the physical tokens. 

 

There are work arounds and we currently use one - but it has it's limitations - for example right now using the old work around we can still only code sign from one virtual machine in our build environment - so lots of file copying happens which slows down builds - with our new solution code signing can be done from any build agent machine that has the client installed. 

 

FWIW, we have only tested using Yubikey and Safenet tokens - but any token with a 64bit pkcs#11 driver dll should work. 

 

Preview of the server web interface - product name may or may not change - naming is hard!

 

 

Edit : forgot to mention, server will be for 64bit windows and linux - client 64bit windows only. 

Edited October 1 by Vincent Parrett

[corneliusdavid 213]

3 hours ago, Vincent Parrett said:

We're just tidying up loose ends (like the installer) before beta - hopefully in a few weeks. 

I bought a 3-year cert in Nov, 2022 so still have another year left on mine. But I'll definitely be looking at your offering.

 

2 hours ago, Vincent Parrett said:

we have only tested using Yubikey and Safenet tokens

I've got a couple of YubiKeys already--sounds great!

[A.M. Hoornweg 144]

6 hours ago, Vincent Parrett said:

No, that would be terribly wasteful - we calculate the digest on the client and send that to the server to be signed. 

I'm very interested!

[Vincent Parrett 745]

1 minute ago, A.M. Hoornweg said:

I'm very interested!

I'll make an announcement here when we have a beta ready.

[Kas Ob. 121]

7 hours ago, Vincent Parrett said:

product name may or may not change - naming is hard!

Name suggestion if you interested :

 

"DSign" or "DSigning" from Distributed .. ihm... i meant Delphi .

The latter yield zero search result on both DDG and Google.

[Kas Ob. 121]

@Vincent Parrett Feature suggestion even before seeing this great tool

 

As i already did 3 projects for 3 different clients, although it wasn't for USB or hardware tokens, but for usual certificate and its private key, the project was to keep the private key securely stored away from developers machines, yet monitor what executables were signed and by whom, stored IPs and devs authentication in detailed nice logs, also the administrator could allow only the digest to be shipped to the server but also allow the administrator to choose who can not sign without uploading the whole binary to be stored on the server for .. security and trust reasons.

 

My suggestion is to expand or at least consider the forced operation logging on server side, this is not something belongs to best practice and i am sure it will be requested sooner or later.

[Anders Melander 1773]

8 hours ago, Vincent Parrett said:

Preview of the server web interface - product name may or may not change - naming is hard!

Looks good - and I like the name. Sufficiently distinct to be googleable.

[Vincent Parrett 745]

1 hour ago, Kas Ob. said:

My suggestion is to expand or at least consider the forced operation logging on server side, this is not something belongs to best practice and i am sure it will be requested sooner or later.

This is on the todo list but not for the initial release - actually pretty simple to add. 

 

Oh and it does work with old school pfx certificates too.

[Vincent Parrett 745]

8 minutes ago, Anders Melander said:

Looks good - and I like the name. Sufficiently distinct to be googleable.

Thanks - yeah we spent a lot of time looking for a name that was relatable and googleable. 

[Kas Ob. 121]

16 minutes ago, Vincent Parrett said:

This is on the todo list but not for the initial release - actually pretty simple to add. 

Then i would expand on this and suggest to make like what i did, on every sing issued the result on both GUI and console client side, it does grab last few signing operation and disclose them to every one is singing, as the best security is transparency and awareness, this means any singing developer can have a glance of what have being signed last 5-10 singing procedures, with time, dates, authorization and the binary name.

 

Just food for thoughts.

 

And thank you very much !

[Anders Melander 1773]

1 hour ago, Kas Ob. said:

this means any singing developer can have a glance of what have being signed last 5-10 singing procedures, with time, dates, authorization and the binary name.

TMI; People tend to ignore walls of text.

Log operations on the server instead so they can be examined by those who care.

[Kas Ob. 121]

Just now, Anders Melander said:

TMI; People tend to ignore walls of text.

Log operations on the server instead so they can be examined by those who care.

Yes not for everyone, yet the decentralization and distribution of the signature log might help or at least increase the chance of avoiding catastrophic failure or again at least faster response, in case of compromise did happen, some might look and see logs with different colors of the names and/or the time of multi signature happened at 3AM then you go why and what Andres had signed binaries, could his station being hijacked !, lets notify him or the responsible person.

 

Anyway, food for thought, and this what i did and my clients loved it !