CVE‌-2019-7644: How Does this Happen?

I have enjoyed listening to the Information Security Podcast Risky Business for a long time and I had thought at some point it would be great to hear them talk about Auth0 and the security work we do. Unfortunately, the mention we received on a recent episode was not the debut I was looking for. A vulnerability we had recently disclosed was included in the news segment as the comedy bug of the week. During the “lolz” at our expense, the host Patrick asks a question that stuck with me, and it’s one I felt was worthy of some further exploration and transparency. The question: how does this happen?

Auth0 is a company that creates common vulnerabilities and exposures (CVEs) and we disclose them publicly. CVE‌-2019-7644 was no exception: we received a report of a vulnerability in one of our JWT libraries via our Responsible Disclosure Program and after the issue was patched, we published an advisory and credited the researcher. The library leaked a token signature in an error message which would allow an attacker to forge an arbitrary JWT token. Time for the mea culpa section of the blog post: this is Picard-face-palm-emoji embarrassing. I will say that to be fully exploitable, the vulnerability required our customers to expose the error messages from the library to the attacker. But, this post is not our defense of that bug — I want to share it as good learning point about managing software inventory.

How it happened

At Auth0 we run a combined Product and Application Security team. Their job is to build and break. We support developers through our systems development life cycle (SDLC): threat model and risk assess, run developers through secure code training, deployed tooling to scan code through deployed tooling during our local developement and as it makes its way through our deployment pipelines. And we check for vulnerable dependencies. The team’s job is to make sure we don’t end up shipping vulnerable code, and when it inevitably happens, their job is also to manage the remediation of discovered vulnerabilities. In this case we were pleased, our SDK team responded quickly and we had a new version within two days. But their initial reaction was telling: they had no idea this library existed.

This is where this issue gets a little more complicated. Looking at the code history, the vulnerability was introduced in 2011 (Auth0 was founded in 2013). We forked this library, published it under the Auth0 organization, and effectively forgot about it. The last changes to this code (before the recent fix) were made in 2013.

So, what about CIS control #2 inventory and control of software assets? Our Cloud-based SCM has been a blessing and a curse here. The security team has audit logs to track all changes we have built tooling to inject security into our developers’ workflows. But it has allowed sprawl, developers have been able to create repos at the speed of an idea (we will soon break the 2,000 mark) and in the early days, before tighter admin controls were introduced, it allowed the creation of public repositories without approval. When you are collecting code like this it is easy for it to become orphaned, which, especially if this code is public can quickly become toxic.

Production vs non-production code

It’s completely different for our production assets which are clearly identified and have explicit owners. Access to the code is controlled and regularly reviewed as part of our ongoing internal reviews and external audits. In addition, we have third parties pen-test and code review them at least once a year. But this case highlights gaps in non-production code and sends a signal that we should be looking harder for code that smells rotten.

As I mentioned before we have already introduced controls that would make it harder for this situation to occur again. But certainly this case has accelerated the Security teams efforts to root out orphaned, aging, and potentially toxic code. This was largely a manual effort and we now automate checks to look for indicators such as: age, last update time, and ownership (e.g. team vs individual) to identifying source to be reviewed or just archived.

While our debut on Risky Business wasn’t what I hoped for, I’m grateful for the process of answering Patrick’s question. As our CISO/VP of Operations Joan D. Pepin has said, “Vulnerabilities are a part of life in software development.” We remain committed to transparently reporting them as they occur and taking steps to remedy them as quickly as possible.