For years, users relied on nothing more than reputation and trust when it came to evaluating claims made by VPN providers. But more recently, VPNs have been getting independent security audits to back up their privacy and security promises with something other than marketing buzzwords.

“A security audit is one tool that demonstrates the health of a project overall,” said Harlo Holmes, Director of Digital Security at Freedom of the Press Foundation. Jon Callas, senior technology fellow for the ACLU, describes security audits as a second set of eyes. Audit firms can see whether or not a VPN is living up to its intent. But VPN audits aren’t foolproof and they’re not all the same. The question is, what does a VPN audit prove, and how can users judge the value of any given audit?

Published or Unpublished

VPNs often promote their own audits as a way of engendering (or regaining) trust from their users. NordVPN, for example, announced it was commissioning a new audit soon after news of 2018 breaches was recently made public.

Not all companies allow the public to access the results of these audits, however. Simon Migliano, Head of Research at PrivacyCo. (the parent company of Top10VPN) says that VPNs need to publish the audits online without restriction to “embrace the spirit of doing these audits” rather than going through what he sees as essentially an empty gesture. “I don’t think it adds a lot of value for users for them to put a blog post out saying, ‘Hey we’ve done an audit by a company,’ quote one or two paragraphs of supposed findings and then be like, ‘Oh, now you can trust us,'” he said.

It’s not always easy to tell which audit firms themselves are trustworthy, but many have websites that list their auditors, their credentials, and how long they’ve been in the industry, which can be a good starting point. Also, look for audits by firms that independently publish their findings. PricewaterhouseCoopers is a well-known name, and that may inspire some confidence in an audit. Cure53, another name that comes up in connection with security audits might be less well known outside the industry, but it’s more specialized in cybersecurity.

“If [an audit] was only released by the VPN’s PR team, and the independent auditor did not give them permission to use their name, and they didn’t independently publish it, that, in my mind, would raise some questions of how legitimate and how intense that audit was, or how bad were the findings were, and which ones weren’t fixed,” said Jon Camfield, Director of Global Technology Strategy at Internews.

Types of Audits

The two most common VPN audits are privacy audits (which center on verifying the organizations’ logging practices) and more comprehensive security audits (which take a broader look at the company and its security practices). The latter is the type of audit NordVPN says it’s undertaking.

“There is a deliberate muddying of the water at times from the slightly less ethical providers about what they have done when they communicate to their users,” said Migliano. “There seems to be a tendency to use the term ‘independent security audit’ as a catchall for ‘hey, now you can trust us,’ which I don’t think is a good thing.”

Although looking at whether a VPN’s logging policy matches its practices is important, Migliano said, he sees it as incomplete. Companies need to allow auditors to look at “the full spectrum of clients and applications and backend infrastructure and core services” too, he said.

What About Open Source?

Some VPNs that haven’t had their own independent third-party security audits argue that they’re unnecessary because their products use audited open-source tools and libraries. But experts say that’s not enough. “What’s not open is how exactly they tie them together,” said Camfield. “Did they configure them according to industry best practice, or even cryptographic best practice? That’s something that’s super useful for the audit to go in and look at.”

Tools built on audited and open source tools could interface with your system in an insecure way, be implemented incorrectly, have misconfigured code, or incorrectly log or store accounts. “There’s a lot more than just, ‘Oh, we use these open libraries.’ Well, did you use them correctly?” Camfield said.

The Scope of an Audit

Audits typically have a scope, which touches on what exactly is being audited, the methods used, and how comprehensive the engagement is, as well as how many people are auditing the app and how long they have. “If somebody is wanting to game things, they could certainly game it by scoping down the audit to things that they know they’re going to pass,” said Callas.

An audit may, for instance, only cover mobile apps or browser extensions rather than the full VPN you’re planning on using. Sometimes, understanding the scope might require a bit of reading between the lines. “An audit might obfuscate the fact that important things haven’t really been audited,” said Holmes. “For instance, if you have someone audit the GUI, does it actually include the underlying protocol, or the selection of protocols or which encryption protocols are offered and how configurable that is? That actually makes a huge difference.”

A glowing audit with a limited scope doesn’t tell you much about the privacy and security of a VPN at all. For example, a limited scope may only allow auditors to look at source code rather than digging through VPN systems and copies of (or even real) production servers. “Your code could be amazing, but if your backend servers weren’t included in the audit, it isn’t actually holistic or useful in any form or fashion,” said Camfield.

When evaluating audit reports and their scope, Holmes additionally looks for whether auditors implement reproducible building protocols, “so that they can, to a higher degree, prove that what you’re actually downloading and installing matches what the developers actually intended.”

Report Findings

Audit reports are technical documents that lay out what vulnerabilities have been found in the VPN that was evaluated. An audit that shows a VPN had some bugs isn’t necessarily a bad thing. In fact, it’s actually good news when auditors find problems that get corrected.

“My view on any sort of bugs and reporting is that the mark of what makes a good company is how they deal with the issues, which includes how fast they fix them, and whether or not they try to weasel out of it being a problem,” Callas said.

Audit reports should include information about fixes during a follow-up engagement when the auditors checked back in. Because security audits only reflect a moment in time, they should be something a company invests in periodically. “You’re not supposed to have an audit once and then never have one again,” said Holmes.

VPNs are updated quite frequently and security bugs are discovered all the time, so more often than not, what’s been audited is not the tool you’re using. “The perfect is the enemy of the good,” said Camfield. “In some idealized perfect world, all the code would be open, the builds would be reproducible (which means you can verify that the open source that everyone can see is exactly what generated the tool that you’re actually using on your device) and each new release would be audited independently.”

“The overhead and the restrictions that would place on tool development are not insignificant,” said Camfield. “So I really see it as, at least do something. Sure, it’s a snapshot in time, but I would much prefer to see everyone go through an annual audit or biannual audit than nothing.”

Not every potential VPN user will bother to scrutinize audits before subscribing. Sometimes you just want to unblock Netflix. But if, say, you’re looking for a VPN to keep you safe and private in an unfriendly environment, it’s probably worth your while to take this sort of thing seriously. And no matter what kind of VPN consumer you are, it can never hurt to be better informed.