Google Internet Authority G3

For some time now, operating behind the scenes and going mostly unnoticed, Google has been changing the infrastructure used to provide HTTPS certificates for its sites and services.

You’ll note that I said mostly. Over the last few months, I’ve periodically encountered complaints from users who try to load a Google site and get an unexpected error page:

certerror

Now, there are a variety of different problems that can cause errors like this one– in most cases, the problem is that the user has some software (security software or malware) installed locally that is generating fake certificates that are deemed invalid for various reasons.

However, when following troubleshooting steps, we’ve determined that a small number of users encountering this NET::ERR_CERT_AUTHORITY_INVALID error page are hitting it for the correct and valid Google certificates that chain through Google’s new intermediate Google Internet Authority G3. That’s weird.

What’s going on?

The first thing to understand is that Google operates a number of different certificate trust chains, and we have multiple trust chains deployed at the same time. So a given user will likely encounter some certificate chains that go through the older Google Internet Authority G2 chain and some that go through the newer Google Internet Authority G3 chain– this isn’t something the client controls.

G2vG3

You can visit this GIA G3-specific test page to see if the G3 root is properly trusted by your system.

More surprisingly, it’s also the case that you might be getting a G3 chain for a particular Google site (e.g. https://mail.google.com) while some other user is getting a G2 chain for the same URL. You might even end up with a different chain simply based on what Google sites you’ve visited first, due to a feature called HTTP/2 connection coalescing.

In order to see the raw details of the certificate encountered on an error page, you can click the error code text on the blocking page. (If the site loaded without errors, you can view the certificate like so).

Google’s new certificate chain is certainly supposed to be trusted automatically– if your operating system (e.g. Windows 7) didn’t already have the proper certificates installed, it’s expected to automatically download the root certificate from the update servers (e.g. Microsoft WindowsUpdate) and install it so that the certificate chain is recognized as trusted. In rare instances, we’ve heard of this process not working– for instance, some network administrators have disabled root certificate updates for their enterprise’s PCs.

On modern versions of Windows, you can direct Windows to check its trusted certificate list against the WindowsUpdate servers by running the following from a command prompt:

certutil -f -verifyCTL AuthRootWU

Older versions of Windows might not support the -verifyCTL command. You might instead try downloading the R2 GlobalSign Root Certificate directly and then installing it in your Trusted Root Certification Authorities:

InstallBtnLocalMachineTrustedRootFinishyay

Overall, the number of users reporting problems here is very low, but I’m determined to help ensure that Chrome and Google sites work for everyone.

-Eric

Google Internet Authority G3

12 thoughts on “Google Internet Authority G3

  1. larrylaca says:

    Excellent! Remarks..

    (Editorial) The first thing you need to know…
    (better) A site certificate needs a valid Certificate Path or Issuer chain of site..intermediate..root certificates. (cont..) Google operates…at the same time. The new G3 is a new intermediate and also has a new root.

    So a given user…
    (end of editorial stuff)

    The RootR2.crt file should have GlobalSign in it’s name.

    Installing the new GlobalSign root fixes the Connection Not Private failure, but doesn’t cache the new G3 intermediate. This is only an efficiency hit. I’m guessing the SSL connection gets the new G3 in memory, but fails to save it to the Intermediate store, for the same reason the root is not being saved to the Trusted Root store. Inspecting the cert store with certmgr.msc never sees G3, even when the client automatically installs the root cert, unless G3 is manually installed,

    E.g. my Win10 laptop auto loads the GlobalSign-R2 root, but doesn’t save G3 to the Intermediate store. If I wasn’t debugging this I would never have noticed. GMail works just fine.

    I’ll begin redirecting trouble reports to text/plain. I was going to setup a similar condensed Chrome forum thread. This is much nicer..

    Liked by 1 person

  2. larrylaca says:

    All:

    To check if this fix applies to your problem: click the NET:ERR_CERT_AUTHORITY_INVALID tag,
    to display the diagnostic text,
    if the second PEM block begins: MIIEXDCCA0SgAwIB…
    then installing the new Google root certificate will likely fix the problem.

    Like

  3. larrylaca says:

    Eric
    I just ran a certUtil AuthRootWU test, I did:

    delete the new Google root: GlobalSignGoogleTrustSvcsRootCA-R2
    from certmgr.msc, close the certmgr.
    ran certutil -f -verifyCTL AuthRootWU (as me, with admin privs, but not runAs Admin)
    which updated many, then failed.
    It installed a GlobalSign ECC Root CA – R5, but not the new Google GlobalSign Root CA-R2.

    Accessing GMail then updated the missing Google root -R2.

    The certutil AuthRootWU update may not fix the GMail problem for users..

    Like

  4. larrylaca says:

    Eric:
    I have a user or two who is willing to post a net-export log for the GMail root cert failure. Is there a CR where we can focus the technical discussion?

    Like

  5. larrylaca says:

    Eric: The failed root cert install is affecting: IE & Chome, YouTube ads, text/plain, and a few *.google.countryCode sites.

    As this is a Windows cert management issue, who can help us track this down with Microsoft network stack experience? Are there registry/policy values we should look at?

    E.g. HKLM/Software/Policies/Microsoft/SystemCertificates/
    AuthRoot/DisableRootAutoUpdate

    Like

    1. Unfortunately, no, I don’t know anyone at Microsoft that works on this space anymore. The G3 issue likely affects many different Google sites. text/plain does not use a Google certificate and any user having problems reaching this blog likely has a different problem. It will affect any browser that uses the Windows certificate store (so, most browsers other than Firefox).

      Like

  6. nat says:

    Please help

    NET::ERR_CERT_AUTHORITY_INVALID
    Subject: *.google.co.th
    Issuer: Google Internet Authority G3
    Expires on: Jan 9, 2018
    Current date: Oct 30, 2017
    PEM encoded chain:
    —–BEGIN CERTIFICATE—–
    MIIEmjCCA4KgAwIBAgIIQQOo+4ZJLiswDQYJKoZIhvcNAQELBQAwVDELMAkGA1UE
    BhMCVVMxHjAcBgNVBAoTFUdvb2dsZSBUcnVzdCBTZXJ2aWNlczElMCMGA1UEAxMc
    R29vZ2xlIEludGVybmV0IEF1dGhvcml0eSBHMzAeFw0xNzEwMTcxMTAxMjhaFw0x
    ODAxMDkxMDAwMDBaMGgxCzAJBgNVBAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlh
    MRYwFAYDVQQHDA1Nb3VudGFpbiBWaWV3MRMwEQYDVQQKDApHb29nbGUgSW5jMRcw
    FQYDVQQDDA4qLmdvb2dsZS5jby50aDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC
    AQoCggEBALdGp9fQ2Ful6z+369LILc9qYQvwQk3efw4QLa6J5ynS8IaUJ7qyvkeP
    EGFKD/igLJEELNd2iZK0Yo4ZMGwaqRmQbL5j6QOfDFZjTxEGseHFOtNtQlEZGeHI
    A08elXaCjCKm5yRTqM4abfT2MZaesqQENS79hDdiASeO55Mynxae8l18RFs5g9pr
    gjlVXx1jpN8MPxA/nvvBNQmkETkGC8m0vlMJYEroc+CjIoUu5iQVgIKz8Vq8uOcM
    qZp8cQRUZ4Orri0YEQU5N/OfAnlKV44YP4yBnJxBbDI2F74hoghwEoyqUEWjw9OC
    POMeW/iRlfHvWlgTdUh7AN1AHB8/A2kCAwEAAaOCAVowggFWMB0GA1UdJQQWMBQG
    CCsGAQUFBwMBBggrBgEFBQcDAjAnBgNVHREEIDAegg4qLmdvb2dsZS5jby50aIIM
    Z29vZ2xlLmNvLnRoMGgGCCsGAQUFBwEBBFwwWjAtBggrBgEFBQcwAoYhaHR0cDov
    L3BraS5nb29nL2dzcjIvR1RTR0lBRzMuY3J0MCkGCCsGAQUFBzABhh1odHRwOi8v
    b2NzcC5wa2kuZ29vZy9HVFNHSUFHMzAdBgNVHQ4EFgQUan4jxk2VwITzmWtatEuh
    7Q2OprAwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBR3wrhQmmd2drEtwobQg6B+
    pn66SzAhBgNVHSAEGjAYMAwGCisGAQQB1nkCBQMwCAYGZ4EMAQICMDEGA1UdHwQq
    MCgwJqAkoCKGIGh0dHA6Ly9jcmwucGtpLmdvb2cvR1RTR0lBRzMuY3JsMA0GCSqG
    SIb3DQEBCwUAA4IBAQBBbP+rvRWMQxrFoXUmYc391tMLygLO6yfaFP9PgYIWHQCl
    CZe5pRILayQuDm8w2CLTJ0qPvPBlU3aEUE8m1YyqrLr3aw0q5h6PTWXSb6sjIluP
    DtHXd6FTdW7bQpvtSLqW6OWvuJemiwiRwLn+vyztdi6zWRTryjSDTAD8ud14TRHI
    8mF1veXi5IzF1nTcqgGLvlDWUn0iGmNKXRXL+HkklRmFTKauKpmsgh/2YJVT+aEX
    LDrEXKIjXumiUvIXNZq9/N4ZMK6Dx1wpPHB4SavVtqMV7GH1F+Uhpup+66Txzxj4
    voq/1OCvdq2Q6pa/36gZCxdMGFDgXrgTXZWYWz/r
    —–END CERTIFICATE—–

    Like

  7. larrylaca says:

    Eric: Are we looking at a short fuze? When will other (US) *.google.com servers update to the new intermediate G3 and new root GlobalSign certificates?

    Most of the Chrome forum posts are from google…CountryCode sites – several/day in the Chrome forum, similar rates in external forums (e.g. UC Browser Connection not private:
    http://forum.ucweb.com/search.php?mod=forum,
    search for ‘Connection not private’ )

    Are these rates a pre-cursor to when other servers update to G3?

    There is also a catch-22 aspect: some users cannot post their failures, because their help requests are being blocked by the same failure. Is there a fail-safe, non-HSTS URL for this situation?

    Like

      1. larrylaca says:

        OK, confirms my assumption that the GlobalSign R2-CA root cert was in the Microsoft list.

        My understanding of the root cert update process is:
        -client (user), when not in store, back tracks and receives root cert
        -polls microsoft update service to verify cert is in list
        -if OK, install in Trusted Root store (local cache)… continue

        What I meant or was leading up to is:
        is there a way to check that the affected users poll the list OK?

        I believe the windows event is: Windows Log\Application, source CAPI2,
        Event ID 7(OK), 8(fail)
        ref: https://technet.microsoft.com/en-us/library/cc749331(ws.10).aspx

        I have a broken Win7Pro user/tester who receives Win updates regularly (last 10/11),
        who fetches the GMail root R2-CA cert, but fails the install step.
        We’re trying to understand why.

        Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s