Bash Uploader Security Update

The Codecov Bash Uploader provides a framework and language-agnostic method for sending your coverage reports to Codecov. The main objectives of the uploader are to detect CI specific settings in the environment, gather reports, and upload this information to Codecov. You may learn more about bash uploader here.

In the case of some third party integrations, the bash uploader is used to provide the core upload functionality needed to properly integrate with Codecov. These uploaders include:

• Codecov-action for Codecov Github Action
• Codecov-circleci-orb for the Codecov Circle Orb
• Codecov-bitrise-step for the Bitrise Step

All other supported uploaders, such as codecov-ruby, Codecov’s node uploader npm package, etc., use community contributed upload implementations that do not utilize the bash uploader.

A customer reported this to us on the morning of April 1, 2021. This customer was using the shasum that is available on our Bash Uploader to confirm the integrity of the uploader fetched from https://codecov.io/bash.

Once the customer saw a discrepancy between the shasum on Github and the shasum calculated from the downloaded Bash Uploader, they reported the issue to us, which prompted our investigation.

Based upon the forensic investigation results to date, it appears that there was periodic, unauthorized access to a Google Cloud Storage (GCS) key beginning January 31, 2021, which allowed a malicious third-party to alter a version of our bash uploader script to potentially export information subject to continuous integration (CI) to a third-party server. Codecov secured and remediated the script April 1, 2021.

We have not been able to determine conclusively who carried out the event. We are working with law enforcement and have offered our full cooperation with their investigation.

The altered version of the bash uploader script could potentially affect:

• Any credentials, tokens, or keys that our customers were passing through their CI runner that would be accessible when the bash uploader script was executed.
• Any services, datastores, and application code that could be accessed with these credentials, tokens, or keys.
• The git remote information (URL of the origin repository) of repositories using the bash uploader to upload coverage to Codecov in CI.

Yes. We have reported this matter to law enforcement and are fully cooperating with their investigation.

Since discovering the event, Codecov has been conducting an investigation with forensics experts to understand what happened and any potential impact on users. These investigations are complex, and we are working to learn what happened and what data was impacted. We took care and time to be able to obtain and deliver accurate information.

You may not have been affected. We have contacted users for whom we had email accounts and posted a notification in the app.

Out of an abundance of caution, if you used the Bash Uploaders between January 31, 2021 and April 1, 2021 and did not conduct a checksum validation of the Bash Uploader, we would suggest you re-roll all of your credentials, tokens, or keys located in the environment variables in your CI process.

You can determine the keys and tokens that are surfaced to your CI environment by running the env command in your CI pipeline. If anything returned from that command is considered private or sensitive, we strongly recommend invalidating the credential and generating a new one. Additionally, we would recommend that you audit the use of these tokens in your system.

Additionally, if customers use a locally stored version of the bash uploader, they should check that version for the following:

Curl -sm 0.5 -d “$(git remote -v)

If this appears anywhere in your locally stored bash uploader, you should immediately replace the bash files with the most recent version from https://codecov.io/bash.

Lastly, if you use a self-hosted (on-premises) version of Codecov, it is very unlikely you are impacted. To be impacted, your CI pipeline would need to be fetching the bash uploader from https://codecov.io/bash instead of from your self-hosted Codecov installation. You can verify from where you are fetching the bash uploader by looking at your CI pipeline configuration.

*the Bash Uploader is also used in related uploaders: codecov-actions uploader for Github, the codecov Circleci Orb, and the Codecov Bitrise Step (together, the “bash uploaders”)

We are still actively assessing the impact of this event on our customers. We have contacted users for whom we had email accounts and posted a notification in the app.

Out of an abundance of caution, if you used:

• Codecov-bash (bash uploader)
• Codecov-action (Github)
• Codecov-circleci-orb
• Codecov-bitrise-step

Between January 31, 2021 and April 1, 2021, and did not conduct a checksum validation, we would suggest you re-roll all of your credentials, tokens, or keys located in the environment variables in your CI process.

Yes. The altered version of the bash uploader script could potentially affect:

• Any credentials, tokens, or keys that our customers were passing through their CI runner that would be accessible when the bash uploader script was executed.
• Any services, datastores, and application code that could be accessed with these credentials, tokens, or keys.
• The git remote information (URL of the origin repository) of repositories using the bash uploader to upload coverage to Codecov in CI.

You should immediately re-roll all of your credentials, tokens, or keys located in the environment variables in your CI process.

You can determine the keys and tokens that are surfaced to your CI environment by running the env command in your CI pipeline. If anything returned from that command is considered private or sensitive, we strongly recommend invalidating the credential and generating a new one. Additionally, we would recommend that you audit the use of these tokens in your system.

Additionally, if you use a locally stored version of the bash uploader, you should check that version for the following:

Curl -sm 0.5 -d “$(git remote -v)

If this appears anywhere in your locally stored bash uploader, you should immediately replace the base files with the most recent version from https://codecov.io.bash.

Lastly, if you use a self-hosted (on-premises) version of Codecov, it is very unlikely you are impacted. To be impacted, your CI pipeline would need to be fetching the bash uploader from https://codecov.io/bash instead of from your self-hosted Codecov installation. You can verify from where you are fetching the bash uploader by looking at your CI pipeline configuration.

You can determine the keys and tokens that are surfaced to your CI environment by running the env command in your CI pipeline. If anything returned from that command is considered private or sensitive, we strongly recommend invalidating the credential and generating a new one. Additionally, we would recommend that you audit the use of these tokens in your system.

Out of an abundance of caution, look for any repository in your organization(s) that used one of Codecov's Bash Uploaders from Jan 31, 2021 to April 1, 2021.

The Bash Uploader is also used in these related uploaders: Codecov-actions uploader for Github, the Codecov CircleCl Orb, and the Codecov Bitrise Step (together, the “Bash Uploaders”)

We are making our team available to aid customers during this time. Please do not hesitate to reach out to [email protected] and/or your Codecov account manager.

Yes. Codecov takes the security of its systems and data very seriously and we have implemented numerous safeguards to protect them.

Codecov maintains a variety of information security policies, procedures, practices, and controls. We continually monitor our network and systems for unusual activity, but Codecov, like any other company, is not immune to this type of event. We have taken a number of steps to address this situation including:

• Rotating all credentials, including the key used to facilitate the event;
• Auditing where and how the key was accessible;
• Setting up monitoring and auditing tools to ensure that this cannot occur to the bash uploader again;
• Working with the hosting provider of the third-party server to ensure the malicious webserver was properly decommissioned and auditable for additional information.

We are also working to further enhance security so we can stay ahead of this type of activity, including reinforcing our security tools, policies, and procedures, as well as our security governance and staffing.

If you use a self-hosted (on-premises) version of Codecov, it is very unlikely you are impacted. To be impacted, your CI pipeline would need to be fetching the bash uploader from https://codecov.io/bash instead of your self-hosted Codecov installation. You can verify from where you are fetching the bash uploader by looking at your CI pipeline configuration.

Immediately upon becoming aware of the issue, Codecov secured and remediated the potentially affected script and began investigating the extent to which users may have been impacted.

Codecov maintains a variety of information security policies, procedures, practices, and controls. We continually monitor our network and systems for unusual activity, but Codecov, like any other company, is not immune to this type of event. We have taken a number of steps to address this situation including:

• Rotating all credentials, including the key used to facilitate the event;
• Auditing where and how the key was accessible;
• Setting up monitoring and auditing tools to ensure that this cannot occur to the bash uploader again; and
• Working with the hosting provider of the third-party server to ensure the malicious webserver was properly decommissioned and auditable for additional information.

We are also working to further enhance security so we can stay ahead of this type of activity, including reinforcing our security tools, policies, and procedures.