Nicolas Lidzborski, Principal Engineer and Jaishankar Sundararaman, Sr. Director of Engineering, Google Workspace

In February, we expanded Google Workspace client-side encryption (CSE) capabilities to include Gmail and Calendar in addition to Drive, Docs, Slides, Sheets, and Meet.

CSE in Gmail was designed to provide commercial and public sector organizations an additional layer of confidentiality and data integrity protection beyond the existing encryption offered by default in Workspace. When CSE is enabled, email messages are protected using encryption keys that are fully under the customer’s control. The data is encrypted on the client device before it’s sent to Google servers that do not have access to the encryption keys, which means the data is indecipherable to us–we have no technical ability to access it. The entire process happens in the browser on the client device, without the need to install desktop applications or browser extensions, which means that users get the same intuitive productivity and collaboration experiences that they enjoy with Gmail today. Let’s take a deeper look into how it works.

How we built Client-side Encryption for Workspace

We invented and designed a new service called, Key Access Control List Service (KACLS), that is used across all essential Workspace applications. Then, we worked directly with customers and partners to make it secure, reliable, and simple to deploy. KACLS performs cryptographic operations with encryption keys after validating end-user authentication and authorization. It runs in a customer's controlled environment and provides the key management API called by the CSE-enabled Workspace clients. We have multiple partners providing software implementations of the KACLS API that can be used by our customers. 

At a high level, Workspace client code takes advantage of envelope encryption to encrypt and decrypt the user content on the client with a Data Encryption Key (DEK) and leverage the KACLS to encrypt and decrypt the DEK. In order to provide separation of duty, we use the customer's OpenID Connect (OIDC) IdP to authenticate end-users and provide a JSON Web Token assertion with a claim identifying the user (3P_JWT). For every encryption/decryption request sent to KACLS, the application (e.g. Gmail) provides a JSON Web Token assertion with a claim authorizing the current end-user operation (G_JWT). KACLS validates these authentication and authorization tokens before returning, for example, a decrypted DEK to the user’s client device.

More details on KACLS are available in Google Workspace Encryption Whitepaper and CSE reference API.

How we built CSE into Gmail

Google Workspace Engineering teams have been hard at work over multiple years to deliver to our customers the ability to have their data protected with client-side encryption. This journey required us to work closely with customers and partners to provide a capability that was secure, easy to use, intuitive and easily deployable. It was also important for CSE to work seamlessly across the Workspace products: you can create a Meet CSE scheduled meeting in Calendar CSE and follow-up with Gmail CSE emails containing links to Drive CSE files.

Client-side encryption in Gmail was built with openness and interoperability in mind. The underlying technology being used is S/MIME, an open standard for sending encrypted messages over email. S/MIME is already supported in most enterprise email clients, so users are able to communicate securely, outside of their domain, regardless of what provider the recipient is using to read their mail, without forcing the recipients to log into a proprietary portal. S/MIME uses asymmetric encryption. The public key and the email of each user are included in the user's S/MIME certificate. Similarly to TLS used for HTTPS, each certificate is digitally signed by a chain of certificate authorities up to a broadly trusted root certificate authority. The certificate acts as a virtual business card, enabling anyone getting it to encrypt emails for that user. The user's private keys are kept secure under customer control and are used by users for decryption of incoming emails and digital signature of outgoing emails.

We decided to leverage the CSE paradigm used for Drive CSE and not keep the private key on the device, to keep them as safe as possible. Instead, we extended our KACLS API to support asymmetric encryption and signature operations. This enables our customers to centrally provision and enable S/MIME, on the KACLS, for all their users without having to deploy certificates individually to each user device.

CSE in Gmail uses the end-user's client existing cryptographic functionalities (Web Crypto API for web browsers for instance) to perform local encryption operations and run client-side code to perform all S/MIME message generation.

Now let's cover the detailed user flows:

When sending an email, the Gmail client generates a MIME message, encrypts the message with a random Data Encryption Key (DEK) then uses the recipients' public keys to encrypt the DEK, calls KACLS (with the user authenticated by customer's IdP and authorized by Google) to digitally sign content and finally sends the authenticated and encrypted S/MIME message, which contains both the encrypted email and the encrypted DEK, to Google servers for delivery to the recipients. Below is an animated screenshot showing the user interface of Gmail when using CSE.

                                              

When receiving an email, Gmail will verify that the digital signature of the email is valid and matches the sender's identity, which protects the email against tampering. Gmail will trust digital identities signed by Root CA PKI as well as custom domain configurations. The Gmail client will call KACLS (with the authentication and authorization JWT) to decrypt the email encryption key, then can decrypt the email and render it to the end-user.

How we protect the application

Workspace already uses the latest cryptographic standards to encrypt all data at rest and in transit between its facilities for all services. Additionally, Gmail uses Transport Layer Security (TLS) by default for communication with other email service providers. CSE in Gmail, however, provides an additional layer of protection for sensitive content. The security of Gmail CSE is paramount to us, and we developed new additional mechanisms to ensure CSE content would be locked into a secure container. On the web, we have been leveraging iframe origin isolation, strict postMessage API, and Content Security Policy to protect the user's sensitive data. Those security controls provide multiple layers of safety to ensure that CSE content stays isolated from the rest of the application. See this simplified diagram covering the isolation protecting CSE emails during composition or display.

What’s next for Client-side encryption and why it’s important 

CSE in Gmail uses S/MIME to encrypt and digitally sign emails using public keys supplied by customers, which add an additional level of confidentiality and integrity to emails. This is done with extensive security controls to protect user data confidentiality, but also transparently integrated in Gmail UI to delight our users. However our work is not done, and we are actively partnering with Google Research to further develop client-side capabilities. You can see some of our progress in this field with our presentation at the RSA Security Conference last year where we provided insight into the challenges and the practical strategies to provide advanced capabilities, such as AI-driven phishing protection for CSE.

Julie Qiu, Go Security & Reliability, and Roger Ng, Google Open Source Security Team

“Secure your dependencies”—it’s the new supply chain mantra. With attacks targeting software supply chains sharply rising, open source developers need to monitor and judge the risks of the projects they rely on. Our previous installment of the Supply chain security for Go series shared the ecosystem tools available to Go developers to manage their dependencies and vulnerabilities. This second installment describes the ways that Go helps you trust the integrity of a Go package. 

Go has built-in protections against three major ways packages can be compromised before reaching you: 

• A new, malicious version of your dependency is published

• A package is withdrawn from the ecosystem

• A malicious file is substituted for a currently used version of your dependency

In this blog post we look at real-world scenarios of each situation and show how Go helps protect you from similar attacks.

Reproducible builds and malicious new versions

In 2018, control of the JavaScript package event-stream passed from the original maintainer to a project contributor. The new owner purposefully published version 3.3.6 with a new dependency named flatmap-stream, which was found to be maliciously executing code to steal cryptocurrency. In the two months that the compromised version was available, it had been downloaded 8 million times. This poses the question - how many users were unaware that they had adopted a new indirect dependency? 

Go ensures reproducible builds thanks to automatically fixing dependencies to a specific version (“pinning”). A newly released dependency version will not affect a Go build until the package author explicitly chooses to upgrade. This means that all updates to the dependency tree must pass code review. In a situation like the event-stream attack, developers would have the opportunity to investigate their new indirect dependency. 

Go Module Mirror and package availability

In 2016, an open-source developer pulled his projects from npm after a disagreement with npm and patent lawyers over the name of one of his open-source libraries. One of these pulled projects, left-pad, seemed to be small, but was used indirectly by some of the largest projects in the npm ecosystem. Left-pad had 2.5 million downloads in the month before it was withdrawn, and its disappearance left developers around the world scrambling to diagnose and fix broken builds. Within a few hours, npm took the unprecedented action to restore the package. The event was a wake up call to the community about what can happen when packages go missing.

Go guarantees the availability of packages.The Go Module Mirror serves packages requested by the go command, rather than going to the origin servers (such as GitHub). The first time any Go developer requests a given module, it’s fetched from upstream sources and cached within the module mirror. When a module has been made available under a standard open source license, all future requests for that module simply return the cached copy, even if the module is deleted upstream.

Go Checksum Database and package integrity

In December 2022, users who installed the package pyTorch-nightly via pip, downloaded something they didn’t expect: a package that included all the functionality of the original version but also ran a malicious binary that could gain access to environment variables, host names, and login information.  

This compromise was possible because pyTorch-nightly had a dependency named torchtriton that shipped from the pyTorch-nightly package index instead of PyPI. An attacker claimed the unused torchtriton namespace on PyPI and uploaded a malicious package. Since pip checks PyPI first when performing an install, the attacker got their package out in front of the real package—a dependency confusion attack.  

Go protects against these kinds of attacks in two ways. First, it is harder to hijack a namespace on the module mirror because publicly available projects are added to it automatically—there are no unclaimed namespaces of currently available projects. Second, package authenticity is automatically verified by Go's checksum database.  

The checksum database is a global list of the SHA-256 hashes of source code for all publicly available Go modules. When fetching a module, the go command verifies the hashes against the checksum database, ensuring that all users in the ecosystem see the same source code for a given module version. In the case of pyTorch-nightly, a checksum database would have detected that the torchtriton version on PyPI did not match the one served earlier from pyTorch-nightly.

Open source, transparent logs for verification

How do we know that the values in the Go checksum database are trustworthy? The Go checksum database is built on a Transparent Log of hashes of every Go module. The transparent log is backed by Trillian, a production-quality, open-source implementation also used for Certificate Transparency. Transparent logs are tamper-evident by design and append-only, meaning that it's impossible to delete or modify Go module hashes in the logs without the change being detected.

Secure by default

The Go team supports the checksum database and module mirror as services so that Go developers don't need to worry about disappearing or hijacked packages. The future of supply chain security is ecosystem integration, and with these services built directly into Go, you can develop with confidence, knowing your dependencies will be available and uncorrupted. 

The final part of this series will discuss the Go tools that take a “shift left” approach to security—moving security earlier in the development life cycle. For a sneak peek, check out our recent supply chain security talk from Google I/O!

Anthony Weems, Information Security Engineer

2022 was a successful year for Google's Vulnerability Reward Programs (VRPs), with over 2,900 security issues identified and fixed, and over $12 million in bounty rewards awarded to researchers. A significant amount of these vulnerability reports helped improve the security of Google Cloud products, which in turn helps improve security for our users, customers, and the Internet at large.

We first announced the Google Cloud VRP Prize in 2019 to encourage security researchers to focus on the security of Google Cloud and to incentivize sharing knowledge on Cloud vulnerability research with the world. This year, we were excited to see an increase in collaboration between researchers, which often led to more detailed and complex vulnerability reports. After careful evaluation of the submissions, today we are excited to announce the winners of the 2022 Google Cloud VRP Prize.

2022 Google Cloud VRP Prize Winners

1st Prize - $133,337: Yuval Avrahami for the report and write-up Privilege escalations in GKE Autopilot. Yuval's excellent write-up describes several attack paths that would allow an attacker with permission to create pods in an Autopilot cluster to escalate privileges and compromise the underlying node VMs. While these VMs are accessible to customers in GKE Standard, this research led to several hardening improvements in Autopilot that make it a better secure-by-default Kubernetes offering.

2nd Prize - $73,331: Sivanesh Ashok and Sreeram KL for the report and write-up SSH Key Injection on GCE. Their write-up describes the journey of discovering a vulnerability that would allow an attacker to gain access to a user's GCE VM by tricking them into clicking a link. They demonstrate the importance of persistence and turned a strange behavior in user creation into an injection of arbitrary SSH public keys.

3rd Prize -  $31,337: Sivanesh Ashok and Sreeram KL for the report and write-up Bypassing Authorization in Cloud Workstations. Their write-up describes their research process for analyzing Cloud Workstations and then a full-chain exploit to steal a user's access token by abusing the format of an OAuth state parameter.

4th Prize - $31,311: Sreeram KL and Sivanesh Ashok for the report and write-up Client-Side SSRF to Google Cloud Project Takeover. Their write-up combines a client-side SSRF, a CSRF bypass, and a clever 3xx redirect by "deactivating" a Feedburner proxy. An attacker could use this vulnerability to steal a Vertex AI user's access token by tricking them into clicking a link.

5th Prize - $17,311: Yuval Avrahami and Shaul Ben Hai for the report and write-up Kubernetes Privilege Escalation: Excessive Permissions in Popular Platforms. Their whitepaper covers privilege escalation vectors in Kubernetes and describes vulnerabilities in many Kubernetes hosting providers, including Azure's AKS, Amazon's EKS, and GKE.

6th Prize - $13,373: Obmi for the report and write-up A Few Bugs in the Google Cloud Shell. Obmi discovered vulnerabilities in the Cloud Shell file upload functionality that would have allowed an attacker to write arbitrary files to a user's Cloud Shell via cross-site request forgery.

7th Prize - $13,337: Bugra Eskici for the report and write-up Command injection in Cloud Shell. Bugra found a very curious injection point in a Cloud Shell script that led to a URL query parameter being directly injected into a Python script. This vulnerability would have given an attacker arbitrary code execution in a user's Cloud Shell if they clicked on an attacker-controlled link.

Congratulations to all the winners and happy hacking! Follow us on @GoogleVRP for future news and updates.

Posted by Anuj Goyal, Product Manager, Chrome Browser

Browser extensions, while offering valuable functionalities, can seem risky to organizations. One major concern is the potential for security vulnerabilities. Poorly designed or malicious extensions could compromise data integrity and expose sensitive information to unauthorized access. Moreover, certain extensions may introduce performance issues or conflicts with other software, leading to system instability. Therefore, many organizations find it crucial to have visibility into the usage of extensions and the ability to control them. Chrome browser offers these extension management capabilities and reporting via Chrome Browser Cloud Management. In this blog post, we will walk you through how to utilize these features to keep your data and users safe.

Visibility into Extensions being used in your environment

Having visibility into what and how extensions are being used enables IT and security teams to assess potential security implications, ensure compliance with organizational policies, and mitigate potential risks. There are three ways you can get critical information about extensions in your organization:

1. App and extension usage reporting

Organizations can gain visibility into every Chrome extension that is installed across an enterprise’s fleet in Chrome App and Extension Usage Reporting.

2. Extension Risk Assessment

CRXcavator and Spin.AI Risk Assessment are tools used to assess the risks of browser extensions and minimize the risks associated with them. We are making extension scores via these two platforms available directly in Chrome Browser Cloud Management, so security teams can have an at-a-glance view of risk scores of the extensions being used in their browser environment.

3. Extension event reporting

Extension installs events are now available to alert IT and security teams of new extension usage in their environment.

Organizations can send critical browser security events to their chosen solution providers, such as Splunk, Crowdstrike, Palo Alto Networks, and Google solutions, including Chronicle, Cloud PubSub, and Google Workspace, for further analysis. You can also view the event logs directly in Chrome Browser Cloud Management.

Extension controls for added security

By having control over extensions, organizations can maintain a secure and stable browsing environment, safeguard sensitive data, and protect their overall digital infrastructure. There are several ways IT and security teams can set and enforce controls over extensions:

1. Extension permission

Organizations can control whether users can install apps or extensions based on the information an extension can access—also known as permissions. For example, you might want to prevent users from installing any extension that wants permission to see a device location.

2. Extension workflow

Extension workflow allows end users to request extensions for review. From there, IT can either approve or deny them. This is a great approach for teams that want to control the extensions in their environment, but allow end users to have some say over the extensions they use.

We recently added a prompt for business justification for documenting why users are requesting the extension. This gives admins more information as to why the extension may or may not be helpful for their workforce, and can help speed approvals for business users.

3. Force install/block extensions

In the requests tab, you can select a requested extension, and approve that extension request and force install it, so the extension shows up automatically on all the managed browsers in that specific Organizational Unit or group.

Admins also have the ability to block extensions in their browser environment.

4. Extension version pinning

Admins have the ability to pin extensions to specific versions. As a result, the pinned extensions will not be upgraded unless the admin explicitly changes the version. This gives organizations the ability to evaluate new extension versions before they decide to allow them in their environment.

Extensions play a huge role in user productivity for many organizations, and Chrome is committed to help enterprises securely take advantage of them. If you haven’t already, you can sign up for Chrome Browser Cloud Management and start managing extensions being used in your organizations at no additional cost.

If you’re already using Chrome Browser Cloud Management, but want to see how you can get the most out of it, check out our newly released Beyond Browsing demo series where Chrome experts share demos on some of our frequently asked management questions.

Asra Ali, Razieh Behjati, Tiziano Santoro, Software Engineers

Every day, personal data, such as location information, images, or text queries are passed between your device and remote, cloud-based services. Your data is encrypted when in transit and at rest, but as potential attack vectors grow more sophisticated, data must also be protected during use by the service, especially for software systems that handle personally identifiable user data.

Toward this goal, Google’s Project Oak is a research effort that relies on the confidential computing paradigm to build an infrastructure for processing sensitive user data in a secure and privacy-preserving way: we ensure data is protected during transit, at rest, and while in use. As an assurance that the user data is in fact protected, we’ve open sourced Project Oak code, and have introduced a transparent release process to provide publicly inspectable evidence that the application was built from that source code. 

This blog post introduces Oak's transparent release process, which relies on the SLSA framework to generate cryptographic proof of the origin of Oak’s confidential computing stack, and together with Oak’s remote attestation process, allows users to cryptographically verify that their personal data was processed by a trustworthy application in a secure environment. 

Integrity and transparency with SLSA  

Project Oak recently collaborated with the SLSA community to create a new container-based builder that produces Supply-chain Levels for Software Artifacts (SLSA) Build Level 3 provenance. This new container-based builder generates non-forgeable provenance statements that capture details about build process information that allow users to perform automated, rigorous provenance verification.

With this enhanced provenance generated by the container-based builder, you can answer questions like:

• Was the artifact built with a toolchain that I know and trust?

• Was the artifact built with a command that I trust?

• Did the build command use a tool that was affected by a vulnerability?

• How can I recreate the artifact?

Project Oak is particularly interested in answering these questions about every layer of the confidential computing stack. For instance, to be sure that a released binary was built using a trusted build process (e.g., the build command did not use any potentially malicious tool), the Oak release process compares the build command against a set of allow-listed tokens. Similarly, we can verify that the builder was not tampered with. 

Transparent releases: an added layer of trust

Project Oak develops a secure runtime and a remote attestation protocol—ways to detect potential adversaries on remote servers and to protect workloads while they are running. Now, with the addition of the container-based SLSA builder, we are able to complete our transparent release process to protect against software supply chain attacks and provide an automated process for verifying the integrity and trustworthiness of a remote server, before sending sensitive information to it. 

Specifically, for each released version of the Oak secure runtime, the Oak team generates and signs an endorsement statement for the binary, using a key accessible only to the Oak team. The endorsement statement can only be generated if the provenance statement passes verification checks, ensuring that a potential malicious attacker cannot forge the statement.

When the client establishes a connection to the server, the client must verify the endorsement statement and the proof of its inclusion in a transparency log, and check that the binary identities in the attestation report and the endorsement statement are the same. This, together with signature verification for the endorsement statement, guarantees three important points of trust for the overall process: that the client is interacting with the same publicly endorsed version of the Oak secure runtime that all other clients interact with; the Oak secure runtime is open source; and that it has a publicly published non-forgeable SLSA v1.0 provenance with adherence to SLSA Build Track 3. For a more technical explanation of the process, see Project Oak’s transparent release process.

Visualization of an Oak application, with attestation verification 

Try it out

We encourage you to check out the transparent release project as a use case for SLSA. Please reach out to us via our slack channel to explore ideas related to Oak secure runtimes and remote attestation.

You don’t need to use Project Oak to take advantage of the new SLSA builder tool. If your project is open source, try one of the SLSA builders to generate non-forgeable provenance for your binaries. We encourage you to containerize your build and try the container-based SLSA 3 builder! Using a container image for your builds improves the reproducibility of your binaries. We also recommend adding the instructions for building your container image (e.g., a Dockerfile) to your GitHub repository, which improves auditability and transparency of your build process, and thus the security of your software supply chain.