Yesterday, we were honored to participate in President Biden’s White House Cyber Security Summit where we shared recommendations to advance the administration’s cybersecurity agenda. This included our commitment to invest $10 billion over the next five years to expand zero-trust programs, help secure the software supply chain, and enhance open-source security.

At Google, we’ve long advocated for securing the software supply chain both through our internal best practices and industry efforts that enhance the integrity and security of software. That’s why we're thrilled to collaborate with the U.S. Department of Commerce’s National Institute of Standards and Technology (NIST) to support and develop a new framework that will help to improve the security and integrity of the technology supply chain.

This builds on our previous work in June of this year, where we submitted four statements in response to the National Telecommunications and Information Administration (NTIA) and NIST’s call for position papers to help guide adoption of new software supply chain security standards and guidelines that fulfill components of the Executive Order on Improving the Nation’s Cybersecurity.

The papers lay out concrete ways to increase the nation’s cybersecurity, based on Google’s experience building secure by design systems for our users and enterprise customers. Each of the suggestions are enactable solutions for software supply chain security, and were drawn from Google’s research and innovations in engineering away entire classes of vulnerabilities.

NIST and NTIA also released their guidelines in July for several of the Executive Order’s target areas (SBOM Minimum Elements, Critical Software Guidelines, Developer Verification of Software), incorporating specific recommendations from Google. Below are summaries of each of Google’s position papers, and background on our contributions and impact in each area.

High-Confidence, Scalable Secure Development

Instead of being reactive to vulnerabilities, we should eliminate them proactively with secure languages, platforms, and frameworks that stop entire classes of bugs.

Preventing problems before they leave the developer’s keyboard is safer and more cost effective than trying to fix vulnerabilities and their fallout. (Consider the enormous impact of the SolarWinds attack, which is predicted to take $100 billion to remediate.) Google promotes designs that are secure by default and impervious to simple errors that can lead to security vulnerabilities.

We want to see secure systems used as widely as possible, so we have invested in initiatives such as getting Rust into the Linux Kernel, published research papers, and shared guidance on secure frameworks.

Security Measures for Critical Software

Critical software does not exist in a vacuum; we must also harden the broader systems and run environments. Our paper outlines a list of actionable steps for critical software's configuration, the privileges with which it runs, and the network(s) to which it is connected.

Our suggestions are based on practices that have withstood the tests of time and scale, such as in our Google Cloud Products, built on one of the industry’s most trusted clouds.

Google contributes to open-source tools that help maintainers adopt these practices, such as gVisor for sandboxing, and GLOME for authentication and authorization. Additionally, to share the knowledge we have gained securing systems that serve billions of users, we released our book Building Secure and Reliable Systems, a resource for any organization that wants to design systems that are fundamentally secure, reliable, and scalable.
 
Software Source Code Testing

Continuous fuzzing is indispensable for identifying bugs and catching vulnerabilities before attackers do. We also suggest securing dependencies using automated tools such as Scorecards, Dependabot, and OSV.

Google has made huge contributions to the field of fuzzing, and has found tens of thousands of bugs with tools like libFuzzer and ClusterFuzz.

We have made continuous fuzzing available to all developers through OSS-Fuzz, and are funding integration costs and fuzzing internships. We are leading a shift in industry support: on top of bug bounties, which are rewards programs for finding bugs, we have also added patch rewards, money that can help fund maintainers remediate uncovered bugs.

Software Supply Chain Integrity

Google strongly encourages adoption of SLSA, an end-to-end framework for ensuring the integrity of software artifacts throughout the software supply chain. Four “SLSA Levels” provide incrementally adoptable guidelines that each raise the bar on security standards for open-source software.

SLSA is based on Google’s internal framework Binary Authorization for Borg (BAB) that ensures that all software packages used by the company meet high integrity standards. Given BAB’s success, we have adapted the framework to work for systems beyond Google and released it as SLSA to help protect other organizations and platforms.

We have shared many of Google’s practices for security and reliability in our Site Reliability Engineering book. Following our recent introduction of SLSA to the wider public, we are looking forward to making improvements in response to community feedback.

Minimum Requirements for SBOMs

Google submitted an additional paper in response to the NTIA’s request for comments on creating SBOMs, which will give users information about a software package’s contents. Modern development requires different approaches than classic packaged software, which means SBOMs must also deal with intermediate artifacts like containers and library dependencies.

SBOMs need a reasonable signal-to-noise ratio: if they contain too much information, they won’t be useful, so we urge the NTIA to establish both minimum and maximum requirements on granularity and depth for specific use-cases. We also recommend considerations for the creation of trustworthy SBOMs, such as using verifiable data generation methods to capture metadata, and preparing for the automation and tooling technologies that will be key for widespread SBOM adoption.

Improving Everyone’s Security

We are committed to helping advance collective cybersecurity. We also realize that too many guidelines and lists of best practices can become overwhelming, but any incremental changes in the right direction make a real difference. We encourage companies and maintainers to start evaluating today where they stand on the most important security postures, and to make improvements with the guidance of these papers in the areas of greatest risk. No single entity can fix the problems we all face in this area, but by being open about our practices and sharing our research and tools, we can all help raise the standards for our collective security.
Labels:


As an active member of the open source software (OSS) community, Google recognizes the growing threat of software supply chain attacks against OSS we use and develop. Building on our efforts to improve OSS security with an end-to-end framework (SLSA), metrics (Scorecards), and coordinated vulnerability disclosure (guide), we are excited to announce Allstar.


Allstar is a GitHub app that continuously enforces security policy settings through selectable automated enforcement actions. Allstar is already filing and closing security issues for Envoy and GoogleContainerTools, with more organizations and repositories lined up. 


See the OpenSSF announcement for more information on Allstar.




Today we are excited to announce some changes to our lineup of Titan Security Keys on the Google Store which provide a simpler experience and make choosing the right security key for you even easier. We will now offer only two types of Titan Security Keys: a USB-A and a USB-C version. Both of these keys have Near Field Communication (NFC) functionality, which allows you to use it with most mobile devices by simply tapping it on the back of your mobile device in order to sign in securely. These keys will be available for all users starting tomorrow, August 10. 


In 2018, Google introduced the Titan Security Key as a direct defense against credential phishing. Phishing occurs when an attacker tries to trick you into giving them your username and password, and it remains one of the easiest and most successful ways of breaching accounts online. Paired with our Advanced Protection Program and its industry-leading automatic protections, the Titan Security Key remains one of the best ways to keep your Google Account safe.



Introducing new Titan Security Key options


Since NFC functionality is now supported by a wide range of Android phones and iPhones, we are discontinuing the Bluetooth Titan Security Key and focusing on the easier and more widely available NFC capability. However, for existing users with our Bluetooth Titan Security Keys, these will continue to work with Bluetooth and will continue to work as an NFC key on most modern mobile devices. Applicable warranties for existing Bluetooth Titan Security Keys will continue to be honored per their terms. All Titan Security Keys are built with a hardware secure element chip that includes firmware engineered by Google to verify the key’s integrity.


If you have a computer with USB-A ports, we recommend you get the USB-A + NFC security key:

If you have a computer with USB-C ports, we recommend you get the USB-C + NFC security key:



If you have an iPad with a USB-C connector you can use the USB-C Titan Security Key. If you have an iPad with a lightning connector, it’s recommended to get a USB-A Titan Security Key with an Apple Lightning adapter:

 




To purchase a Titan Security Key, visit the Google Store. The USB-A+NFC key,which includes a USB-A to USB-C adapter, is available for $30 and the USB-C+NFC key retails for $35.

To learn more about how security keys can help protect you against phishing, visit the Titan Security Key product page.


Rather than only taking a one-bug-at-a-time perspective, preemptive actions can stop bugs from having bad effects. With Linux written in C, it will continue to have a long tail of associated problems. Linux must be designed to take proactive steps to defend itself from its own risks. Cars have seat belts not because we want to crash, but because it is guaranteed to happen sometimes.

Even though everyone wants a safe kernel running on their computer, phone, car, or interplanetary helicopter, not everyone is in a position to do something about it. Upstream kernel developers can fix bugs, but have no control over what a downstream vendor chooses to incorporate into their products. End users get to choose their products, but don't usually have control over what bugs are fixed nor what kernel is used (a problem in itself). Ultimately, vendors are responsible for keeping their product's kernels safe.

What to fix?


The statistics of tracking and fixing distinct bugs are sobering. The stable kernel releases ("bug fixes only") each contain close to 100 new fixes per week. Faced with this high rate of change, a vendor can choose to ignore all the fixes, pick out only "important" fixes, or face the daunting task of taking everything.



Fix nothing?


With the preponderance of malware, botnets, and state surveillance targeting flawed software, it's clear that ignoring all fixes is the wrong "solution." Unfortunately this is the very common stance of vendors who see their devices as just a physical product instead of a hybrid product/service that must be regularly updated.

Fix important flaws?

Between the dereliction of doing nothing and the assumed burden of fixing everything, the traditional vendor choice has been to cherry-pick only the "important" fixes. But what constitutes "important" or even relevant? Just determining whether to implement a fix takes developer time.

The prevailing wisdom has been to choose vulnerabilities to fix based on the Mitre CVE list, presuming all important flaws (and therefore fixes) would have an associated CVE. However, given the volume of flaws and their applicability to a particular system, not all security flaws have CVEs assigned, nor are they assigned in a timely manner. Evidence shows that for Linux CVEs, more than 40% had been fixed before the CVE was even assigned, with the average delay being over three months after the fix. Some fixes went years without having their security impact recognized. On top of this, product-relevant bugs may not even classify for a CVE. Finally, upstream developers aren't actually interested in CVE assignment; they spend their limited time actually fixing bugs.

A vendor relying on cherry-picking is all but guaranteed to miss important vulnerabilities that others are actively fixing, which is almost worse than doing nothing since it creates the illusion that security updates are being appropriately handled.

Fix everything!

So what is a vendor to do? The answer is simple, if painful: continuously update to the latest kernel release, either major or stable. Tracking major releases means gaining security improvements along with bug fixes, while stable releases are bug fixes only. For example, although modern Android phones ship with kernels that are based on major releases from almost two to four years earlier, Android vendors do now, thankfully, track stable kernel releases. So even though the features being added to newer major kernels will be missing, all the latest stable kernel fixes are present.

Performing continuous kernel updates (major or stable) understandably faces enormous resistance within an organization due to fear of regressions—will the update break the product? The answer is usually that a vendor doesn't know, or that the update frequency is shorter than their time needed for testing. But the problem with updating is not that the kernel might cause regressions; it's that vendors don't have sufficient test coverage and automation to know the answer. Testing must take priority over individual fixes.

Make it happen

One question remains: how to possibly support all the work continuous updates require? As it turns out, it’s a simple resource allocation problem, and is more easily accomplished than might be imagined: downstream redundancy can be moved into greater upstream collaboration.

More engineers for fixing bugs earlier

With vendors using old kernels and backporting existing fixes, their engineering resources are doing redundant work. For example, instead of 10 companies each assigning one engineer to backport the same fix independently, those developer hours could be shifted to upstream work where 10 separate bugs could be fixed for everyone in the Linux ecosystem. This would help address the growing backlog of bugs. Looking at just one source of potential kernel security flaws, the syzkaller dashboard shows the number of open bugs is currently approaching 900 and growing by about 100 a year, even with about 400 a year being fixed.

More engineers for code review

Beyond just squashing bugs after the fact, more focus on upstream code review will help stem the tide of their introduction in the first place, with benefits extending beyond just the immediate bugs caught. Capable code review bandwidth is a limited resource. Without enough people dedicated to upstream code review and subsystem maintenance tasks, the entire kernel development process bottlenecks.

Long-term Linux robustness depends on developers, but especially on effective kernel maintainers. Although there is effort in the industry to train new developers, this has been traditionally justified only by the "feature driven" jobs they can get. But focusing only on product timelines ultimately leads Linux into the Tragedy of the Commons. Expanding the number of maintainers can avoid it. Luckily the "pipeline" for new maintainers is straightforward.

Maintainers are built not only from their depth of knowledge of a subsystem's technology, but also from their experience with mentorship of other developers and code review. Training new reviewers must become the norm, motivated by making upstream review part of the job. Today's reviewers become tomorrow's maintainers. If each major kernel subsystem gained four more dedicated maintainers, we could double productivity.

More engineers for testing and infrastructure

Along with more reviewers, improving Linux's development workflow is critical to expanding everyone's ability to contribute. Linux's "email only" workflow is showing its age, but the upstream development of more automated patch tracking, continuous integration, fuzzing, coverage, and testing will make the development process significantly more efficient.

Additionally, instead of testing kernels after they're released, it's more effective to test during development. When tests are performed against unreleased kernel versions (e.g. linux-next) and reported upstream, developers get immediate feedback about bugs. Fixes can be developed before a flaw is ever actually released; it's always easier to fix a bug earlier than later.

This "upstream first" approach to product kernel development and testing is extremely efficient. Google has been successfully doing this with Chrome OS and Android for a while now, and is hardly alone in the industry. It means feature development happens against the latest kernel, and devices are similarly tested as close as possible to the latest upstream kernels, all avoiding duplicated "in-house" effort.

More engineers for security and toolchain development

Besides dealing reactively to individual bugs and existing maintenance needs, there is also the need to proactively eliminate entire classes of flaws, so developers cannot introduce these types of bugs ever again. Why fix the same kind of security vulnerability 10 times a year when we can stop it from ever appearing again?

Over the last few years, various fragile language features and kernel APIs have been eliminated or replaced (e.g. VLAs, switch fallthrough, addr_limit). However, there is still plenty more work to be done. One of the most time-consuming aspects has been the refactoring involved in making these usually invasive and context-sensitive changes across Linux's 25 million lines of code.

Beyond kernel code itself, the compiler and toolchain also need to grow more defensive features (e.g. variable zeroing, CFI, sanitizers). With the toolchain technically "outside" the kernel, its development effort is often inappropriately overlooked and underinvested. Code safety burdens need to be shifted as much as possible to the toolchain, freeing humans to work in other areas. On the most progressive front, we must make sure Linux can be written in memory-safe languages like Rust.

Don't wait another minute

If you're not using the latest kernel, you don't have the most recently added security defenses (including bug fixes). In the face of newly discovered flaws, this leaves systems less secure than they could have been. Even when mediated by careful system design, proper threat modeling, and other standard security practices, the magnitude of risk grows quickly over time, leaving vendors to do the calculus of determining how old a kernel they can tolerate exposing users to. Unless the answer is "just abandon our users," engineering resources must be focused upstream on closing the gap by continuously deploying the latest kernel release.

Based on our most conservative estimates, the Linux kernel and its toolchains are currently underinvested by at least 100 engineers, so it's up to everyone to bring their developer talent together upstream. This is the only solution that will ensure a balance of security at reasonable long-term cost.

Labels: , ,