What's Hiding In The Other 97 Percent Of Your App?
I've been a developer for most of my life until more recent years, but I'll be honest; I haven't developed an actual full-fledged app in years. I do my best to keep up with how technology changes in order to be able to protect it, but WOW things change so quickly!
In the olden days, you had to write most of the app yourself. We had platforms of course, like portal frameworks that handled display and interaction from the server side, but visual frameworks and more robust server-side frameworks were limited.
Fast forward to today... almost literally your entire application can be code you borrowed from GitHub.
You can scan and test your code with static analysis tools, but when it comes to the stuff you stole-ahem... borrowed, how do you protect that?
I'll tell you, thar be nasties in there.
The good news is, you CAN solve this riddle, and you CAN protect your enterprise from open source heebie-jeebies.
First, Inventory: Find Out What's In There
For those of you who don't know me well, I got into #AppSec a few years ago after a bad run-in with the InfoSec group and their use of HP Fortify. It was NOT a pleasant experience. Since then I got involved in helping devs learn how to FIX security defects proactively, instead of just responding to a 300 page report from some stupid tool.
But even the best static-analysis tools have a weakness: They don't normally scan the open-source libraries you've included to help you leapfrog development.
This is a HUGE hole.
When I started developing the Application Security Program for Cambia, one of the areas I knew we had to solve was open-source use, a set of scanning tools referred to as Software Composition Analysis (aka SCA). I knew I wanted to support developers using open-source and I also knew whatever I implemented needed to be fast, easy, and transparent. What I didn't know was how big our exposure would be; like how much open-source were we actually using?
The answer staggered me.
Inventory Step 1: Choose a tool:
The first thing I knew I needed to find was a solution to understanding the risk. How much open-source were we using in developing our apps? Where was it being used? How was it being used? And, how would I patch it if we had an urgent flaw finding like HeartBleed or ShellShock?
I reviewed several tools in the process of selecting a winner but ultimately landed on SourceClear (SRC:CLR). They are a few year old startup as of this post, but had the best coverage from a language standpoint, incredibly fresh and accurate data, and the easiest integration of any tool around. You simply download a command line interface (CLI) binary, and off you're running.
We're now able to scan our entire code base for each app, even hundreds of megabytes in some cases, and the scans complete in just a few seconds.
Inventory Step 2: Pick your jaw off the floor
The first app I scanned was one of our largest web applications. The total code base was about 350mb which isn't that uncommon. It's a Ruby app and pretty mature code written by savvy developers.
So how much of it was 3rd party libraries/frameworks?
That means we were regularly assessing the code quality of about 3 percent of our application.
Which also means we were failing to look at literally 97% of our attack surface.
I was stunned.
What I've come to find out after using SourceClear on the rest of our apps is that this is not an uncommon reality. Software today is not just built, but it's literally one Lego from your set, and hundreds of bricks from people you've never met. Yikes.
Next, Plan Your Upgrade/Replacement Strategy
The next step in our strategy is to determine how to prioritize remediation of findings from this new discovery, and how to be proactive in the future about making good selections of open-source tools.
My recommendation here is very much crawl-walk-run.
If your developers are already used to remediating findings from your SAST, then this will just be a new evolution. At the same time, since the volume of information is likely to be huge, be sane in how you approach this.
Remediation Step 1: Prioritize
* EXPOSURE: * First we looked at how extensively each library was used. Was it used just once in some far corner of the application, or was it sprinkled all throughout our application? This reveals a dichotomoy of risk: 1) You can take a lot of risk off the table by fixing issues in one area that has a lot of repeat use. 2) Conversely, the replacement effort for a widely used package may require extensive retooling and testing. This is another great opportunity to work WITH your devs teams to prioritize vs. just demanding immediate compliance.
* AGE: * Next, you'll need to look at the results from the findings and discover any libraries or frameworks that are horribly out of date; either by actual calendar time, or possibly one or more major versions behind. In our case we agreed with dev teams that any package over two years old is a no-go for continued deployment to prod and had to be updated. Secondly we agreed that we would never be more than a single major version behind. If 3.0a is released, we're okay with 2.x but not okay with 1.x.
* KNOWN VULNS: Last, review the reports for actual known vulns and assess the risk through what you know of the exposure variable above. If it's an easily discoverable and exploitable flaw and pervasive through your app... fix it TODAY!!!* One the other hand, if it's a low-visibility and highly complex exploit found in a far corner of your app you might want to be reasonable about allowing that to wait.
Watch Out for That License!
The last major area to be aware of is the license associated with the open-source you're using.
Did You Know?: If you're using a library with a "copy-left" license, you are required not only to disclose your use of that library in your derivative work but also to make YOUR code open-source as well?
Let that sink in.
Your highly-secure, customer facing web app's innovative source code must now be made available to the world to see. And be free. And exposed to hackers.
On this one, we started with compiling a list of the licenses we actually were using and provided that to a member of our legal team. They then reviewed our license exposure, as well as other popular licenses in the wild and created a matrix of acceptable license use for our company.
If you think this is an area you can ignore because "Hey, nobody will sue us..." you should ask VMWare how that works.
Last: Be Vigilant
The best tool on your shelf is just that... a tool on your shelf. Even though you shouldn't require perfection on day one, the best thing you can give your development teams is information. Do it early in the development lifecycle, and do it often.
- Build scanning into your development pipelines - Run scans as often as they can reasonably complete between builds - SAST scans that take longer may need to simply run on a schedule - SCA scans like SourceClear that take 4 seconds to run can be implemented on each commit to your CI.
Thanks everyone for reading, and Be smart! Be Safe!
~ * Jet *