Virus alert. Go! Go! Go!
de

When developing web applications, viruses are not a too big issue. At least as long as no files are shared by users. Environment is completely controlled by the developers (or nearby teams), deployment is done by automatic CI/CD-pipelines provisioning virtual machines or containers. Virus scanners are simply not necessary in VMs or containers, they would only consume CPU-time with little effect. Our product kairo/m that is used for automating processes in doctors' offices on the other hand needs to be installed on desktops. And while doing this we faced new (old) problems, part of them made us shake our heads in disbelief.

Our product startet with development of a prototype. This prototyp has been implemented quick and dirty in C programming language so we could easily access Win32-API. Installation at our customer's showed no problems and all features we wanted to check so far with the prototype worked as expected.

So next step was implementing the actual product. We selected Go programming language because of it's eco-system, it's cross-compiling features and most of all a very easy deployment. Go programs are statically linked by default, so you end up with one executable that needs to be "installed". It did not take very long (again, thanks to Go) until we had our first increment we could ship to our customer. But we could not install it because of an alert from the virus scanner. Of course we stopped installation, paused development and started hunting down the would-be virus. We did not succeed though, we did not find any. Final measure was to re-install all developer machines and establish a CI/CD pipeline that would separate building artifacts (even cross-compile from linux to windows) into containers where no one could interfere and which were set up from scratch with each build. CI/CD was planned for a later sprint, but we had no other choice it seemed. After all these measures we had a fresh and cleanly built product, so heading back to the customer... and again there was a virus alert. Checking again with VirusTotal about 20 virus scanners detected a virus. But they did not agree on what kind of virus has been detected. With that in mind we were in doubt about the result. What if it's just a false alarm?

While having a deeper look at what might raise the alert, we came across the article A Big Problem in Go That No One Talks About of Marvin Wendt talking about his experiences of GO in combination with anti-virus software. Long story short:

The process of how Go compiles this binary is relatively unique, and often confuses antivirus scanners, resulting in the binary being mistakenly detected as a virus.
Marvin Wendt in A Big Problem in Go That No One Talks About
Now we're getting closer. Presumably our problem also has it's cause in the way Go programs are compiled and that there is little effort made by virus scanner developers to adapt to Go programs. Question is, what can we do about it? Re-write our code in another programming language? Get in contact with all virus scanner manufacturers and convince them that our software is not evil? Again and again for new versions?

Luckily there is code signing (which is a good idea in any case). An executable binary is signed based on a certificate. This special code signing certificate is provided by a CA (certificate authority) along with the corresponding private key of course. After that signed binary is installed onto another machine, the trustworthy CA certificate enables the machine to verify the binary's signature based on the chain of trust up to the CA. This verification will make sure that the binary really originates from the "good" manufacturer and that it has not been tampered with. This is not only done by the operating system (windows) but also by virus scanners. If both are satisfied, the binary can be executed.

Next question: where do we get a trustworthy code signing certificate from? Of course all big CAs provide such certificates, in exchange of money of course. But before spending money we wanted to know how some virus scanners will behave when encountering signed binaries.

We wrote a little Go program suitable for this test that should alert a virus scanner: 


package main

func main() {
  println("Hello Virus!")
}
Compile, install, execute... virus alert! As expected. So now for the test steps regarding code signing:
  1. create self signed CA-certificate
  2. create a code signing certificate and sign it with the previously created CA
  3. code sign the Go program
  4. install pogram in target machine
  5. execute program on target machine
Some remarkable detail: the target machine will not know about our CA certificate and hence will not be able to verify the binary's signature... And the result is something we surely did not expect: seven out of eight scanners did accept the binary, simply because it's signed! They had no means to verify the signature at all!

So what about the eightth virus scanner? Why did he raise the alert as expected while the others didn't? To find out we installed our self signed CA certificate in the target machine and repeated the test. This time, the scanner also accepted the binary, as it's supposed to work with code signing. So in the end for our use case code signing seems to be the solution.

Still I feel uneasy even with that last scanner, that supposedly acted correct. I fully agree with the concept of code signing. But if it's code signature verification uses CA certificates that come from the same infected machine, what can be assumed about the signutare? How will a scanner know that CA certificates are still trustworthy after an infection?