Just as a note to startups here considering "penetration tests":
"Penetration test" is a term that means wildly different things depending on who you talk to.
The kind of test discussed in this post is the most common kind. People in the field call them "network penetration tests". These are the projects where someone runs nmap and Nessus and Metasploit against your network, dumps the Nessus results into a Word document, and calls it a day.
I'm not wild about these kinds of projects, and even less wild about the firms that specialize in them. They may find things on your network that you need to know. But they generally involve people just running some tools and interpreting the results, and then, if they find something blatant, spending the balance of their time using that finding to pry their way into the rest of your network.
The latter part of the project --- the part where they get to your database, dump your hashes, pivot from machine to machine, &c --- is not a great use of your security dollars. It's generally always going to be the case that if someone finds a way to run code (or SQL) on one of your servers, you're done for. The important finding is the flaw that gets attackers into your network. The findings that come after that look scary, but since there's not a whole lot you're going to be able to do to reliably lock down your internal servers, they aren't very useful to you; the next team that finds some other way onto your servers will embarrass you just as badly even after you "fix" the internal flaws from the first team.
You can get a license to run Nessus pretty cheaply. You can download nmap and Metasploit yourself. If you can build a product, you're more than qualified to run them yourself. If you don't have the bandwidth to do that, don't pay too much to have someone else do it. Also, demand that the team that does the netpen breaks out the findings that actually get them into your network, versus the less valuable findings like "older version of OpenSSL detected that we don't actually know how to exploit" or "customer records recovered after we took control of your database", and make sure the team concentrates on finding new ways into your network, rather than on extending their access into your network once they do find a way.
You'll need to ride netpen people not to waste time extending access, because the Fortune 500 companies that are the bread-and-butter clients for network penetration testers actually do want people to spend time extending access and finding "shock and awe" internal findings --- they're doing these tests for a different reason (to justify security budget), not for the reason you're doing them (to make sure it isn't easy to break into your servers).
I agree to a large extent. I've been asked for 'a non-intrusive penetration test'... Uh, what? Then you hammer it out and they're looking for what we call a 'vulnerability assessment'. The pen-test without the pen- phase.
There's a large number of people running those tools and doing exactly as you describe. I call them, 'the competition'. They bid low and ship a canned report with little to no analysis or follow-up. I highly recommend everyone scan themselves regularly. None of the tools you mentioned are too scary. I do tend to find the severity ratings to be out of whack with the real world impacts a lot of times, so if you see a bunch of red in the report don't panic. Read the text and figure out what it means to your network. The Metasploit Framework has a bit of a learning curve, but nothing too daunting and really it's not necessary for a maintenance scan. You can always invest in a licensed version if the Framework scares you.
Those tools don't provide meaningful coverage of web applications but they would give you a decent idea of the security posture of your network perimeter.
Many situations require the disinterested third party to perform the assessment or audit though. Sometimes customers/partners want to see your last report if you're dealing B2B and then there are the compliance requirements (PCI, health or personal information).
The one thing about that report that really bothered me was the remediation price tags. I know a lot of companies do similar things but we never offer remediation services since it would put us in a conflict of interest. I turn down security product installs regularly on our assessment and penetration testing customers because I don't want us in a position where we're auditing our own work. It's a point of contention sometimes but its an audit independence requirement and I won't budge on the issue.
"Penetration test" is a term that means wildly different things depending on who you talk to.
The kind of test discussed in this post is the most common kind. People in the field call them "network penetration tests". These are the projects where someone runs nmap and Nessus and Metasploit against your network, dumps the Nessus results into a Word document, and calls it a day.
I'm not wild about these kinds of projects, and even less wild about the firms that specialize in them. They may find things on your network that you need to know. But they generally involve people just running some tools and interpreting the results, and then, if they find something blatant, spending the balance of their time using that finding to pry their way into the rest of your network.
The latter part of the project --- the part where they get to your database, dump your hashes, pivot from machine to machine, &c --- is not a great use of your security dollars. It's generally always going to be the case that if someone finds a way to run code (or SQL) on one of your servers, you're done for. The important finding is the flaw that gets attackers into your network. The findings that come after that look scary, but since there's not a whole lot you're going to be able to do to reliably lock down your internal servers, they aren't very useful to you; the next team that finds some other way onto your servers will embarrass you just as badly even after you "fix" the internal flaws from the first team.
You can get a license to run Nessus pretty cheaply. You can download nmap and Metasploit yourself. If you can build a product, you're more than qualified to run them yourself. If you don't have the bandwidth to do that, don't pay too much to have someone else do it. Also, demand that the team that does the netpen breaks out the findings that actually get them into your network, versus the less valuable findings like "older version of OpenSSL detected that we don't actually know how to exploit" or "customer records recovered after we took control of your database", and make sure the team concentrates on finding new ways into your network, rather than on extending their access into your network once they do find a way.
You'll need to ride netpen people not to waste time extending access, because the Fortune 500 companies that are the bread-and-butter clients for network penetration testers actually do want people to spend time extending access and finding "shock and awe" internal findings --- they're doing these tests for a different reason (to justify security budget), not for the reason you're doing them (to make sure it isn't easy to break into your servers).