Skip to main content

How to investigate high tail latency when using Cloud Spanner

 When you use Cloud Spanner, you may encounter some high tail latency cases. Some of the causes may be on the Cloud Spanner side, but there could be some other reasons as well. In this blog post, we will  talk about how to distinguish the high latency causes and also talk about some tips to improve Cloud Spanner latency.

Check the relationship between the high latency and Cloud Spanner usage

If you can find the high latency in Cloud Spanner metrics which are available in Cloud Console or Cloud Monitoring, the latency cause is either at [3. Cloud Spanner API Front End] or [4. Cloud Spanner Database] in the diagram from the Cloud Spanner end-to-end latency guide. Further investigation at Cloud Spanner level is needed.

On the other hand, if you can't confirm the high latency in Cloud Spanner metrics, the high latency likely happened before reaching Cloud Spanner from the client.

If high latency was observed in your client metrics, I recommend you check if

  • accessing other services had high latency 
  • the client machine had any resource shortage issue
  • the high latency happened to a specific client machine

Some example causes are:

  • sudden CPU utilization spike (this itself is not the cause, but indicates other processes in the machine may have caused the latency)
  • hitting Disk I/O performance limit
  • ephemeral port exhaustion and not being able to establish a TCP connection.
  • high latency due coming from DNS queries

You can also measure the latency at [2. Google Front End] in Cloud Spanner end-to-end latency guide -- time from when GFE sends a request to when GFE gets a first response from [4. Cloud Spanner database] via [3. Cloud Spanner API Front End]. If you observe high latency in this metric, a further investigation needs to be performed on the GCP side. This can be achieved by opening a support ticket if you have a support package. (However, this should be quite rare.)

Note that this GFE metric doesn't include latency for TCP/SSL handshake. If you have no idea about the latency cause based on client, GFE, and Cloud Spanner metrics, you may need to get a packet capture and check if there is high latency in TCP/SSL handshake. (However, this should also be quite rare.)

Investigate high latency in Cloud Spanner usage

If you observe high latency in Cloud Spanner metrics, the most typical cause is the lack of Spanner nodes. Make sure that your CPU utilization is within the recommended value in Alerts for high CPU utilization. Note that low/middle priority tasks (such as generating statistics packages, compaction, schema changes) don't affect higher priority tasks when the CPU utilization is low, but low priority tasks can affect higher ones when the utilization gets close to 100%.

If your CPU utilization is high, you can narrow down affecting queries based on Investigating high CPU utilization.

If you observe high latency even though the overall CPU utilization is not high, the cause may be due to hot spots or lock wait. 

For hot spots, you can check the frequently accessed keys by Key Visualizer. In some cases, hot spots may subside due to optimizations in Cloud Spanner. However, optimizations cannot address all the cases depending on the key design or traffic pattern. Schema design best practices will be useful in such cases.

To investigate lock wait times, you can refer to Lock statistics. Note that because detailed information will become unavailable as time passes (see Data retention), it's more effective to check SPANNER_SYS.LOCK_STATS_TOP_MINUTE or SPANNER_SYS.LOCK_STATS_TOP_10MINUTE as soon as the high latency issue happens.

Also you can associate tags with your queries, read requests, and transactions. You'd be able to identify the cause of high latency more effectively by using the tagging feature and statistics tables.

Tips to avoid high latency

In most cases, you'll find the cause and measures based on the aforementioned approaches. Let me introduce some tips to avoid high latency for the use cases where you have difficulty in finding the cause based on statistics tables and Key Visualizer.

Use stale reads

Cloud Spanner guarantees strong consistency against read operations by default. However, using stale read even with short staleness (e.g. 1 sec) may improve performance dramatically. This can be effective especially when you need to read rows which are also updated frequently and don't require strong consistency with the updates.

Incorporate column data into indexes by using STORING clause

When you use FORCE_INDEX in a SELECT query, you'll get results without scanning a base table from the index if the data in SELECT columns are stored in the index itself. You can achieve this by using the STORING clause.

If you see a large time gap between latency in Scan Index and latency in its upper Distributed Union/Distributed Cross Apply, using STORING clause would provide large performance gains.

Use Partitioned DML in deleting rows

In some use cases, you may want to delete some rows periodically. Creating a row deletion policy with TTL is the convenient approach, but if you want to do it on your own, you can minimize the scope of lock ranges by using Partitioned DML because it'll be executed in parallel, hence minimizing the effect to other requests. One caveat is that the operation must be idempotent. In other words, you can't use Partitioned DML if a difference between the result of performing the operation once and the result of performing it multiple times is not acceptable.

A few second latency at p99 can happen

There are some situations where you can't suppress such latency increases. The Spanner Frontend servers ([3. Cloud Spanner API Front End] in the latency guide) are occasionally restarted due to maintenance. If your request (session) happens to be on the server which is about to restart, it takes a few seconds in session takeover to another server. The maintenance is essential to ensure the service level and the tail latency due to this event is inevitable.

That's it. I hope this article will help you find the high latency cause and measure you haven't come up with.


What's Popular ?

Use Vault for Gmail Confidential Messages and Jamboard Files

Google vault will be supporting two new formats in the future, Gmail confidential mode emails & Jamboard files stored in Google Drive. Google Vault gives you a chance to retain, hold, search, and export data to support your organization’s retention and eDiscovery needs. This dispatch includes support for new information types with the goal that you can thoroughly oversee your association's information. What happens when individuals in your association sends confidential messages? Vault can hold, retain, search, and export all confidential mode messages sent by users in your association. Messages are constantly accessible to Vault, notwithstanding when the sender sets a termination date or denies access to private messages. Here’s an example of what will see in Vault when they search for and preview this email sent by . But It’ll not work vise versa. Admins can hold, retain, search and export message headers and s

All you need to know about Cloud Storage

Cloud Storage is a global, secure, and scalable object store for immutable data such as images, text, videos, and other file formats. You can add data to it or retrieve data from it as often as your application needs. The objects stored have an ID, metadata, attributes, and the actual data. The metadata can include all sorts of things,  including the security classification of the file, the applications that can access it, and similar information. The ID, metadata, and attributes make object storage an appealing storage choice for a large variety of applications ranging from web serving to data analytics.    Storage classes You store objects in buckets that are associated with a project, which are, in turn, grouped under an organization. There are four storage classes that are based on budget, availability, and access frequency.  Standard buckets for high-performance, frequent access, and highest availability - Regional or dual-regional locations for data accessed frequently or high-th

Zoom’s Work Transformation Summit on Jan. 19: Fresh Approaches for Moving Forward

These past two years have undoubtedly reshaped work. More specifically, these past two years — shuffling between remote, in-person, and hybrid work scenarios — reshaped what employees expect out of their jobs, how they want to work, and what the office means to them.  Organizations are challenged with making big decisions to meet those expectations, and those decisions will dramatically alter how they hire, manage their facilities, buy technology, and maintain productivity. Simply adjusting policies and retooling previous work models won’t do. It takes a comprehensive reimagining. To help organizations navigate this next phase of work, Zoom is hosting our  Work Transformation Summit  on Jan. 19, a free, half-day virtual event designed to provide you and your organization with meaningful strategies, creative approaches, and innovative solutions for redefining work.  Summit attendees will have the opportunity to hear from peers and industry experts on the importance of embracing technolo

What’s new with Google Cloud

  Want to know the latest from Google Cloud? Find it here in one handy location. Check back regularly for our newest updates, announcements, resources, events, learning opportunities, and more.  Week of Mar 8-Mar 12 2021 Learn about the value of no-code hackathons —Google Cloud’s no-code application development platform, AppSheet, helps to facilitate hackathons for “non-technical” employees with no coding necessary to compete. Learn about Globe Telecom’s no-code hackathon as well as their winning AppSheet app  here . Introducing Cloud Code Secret Manager Integration — Secret Manager  provides a central place and single source of truth to manage, access, and audit secrets across Google Cloud. Integrating  Cloud Code  with Secret Manager brings the powerful capabilities of both these tools together so you can create and manage your secrets right from within your preferred IDE, whether that be VS Code, IntelliJ, or Cloud Shell Editor.  Learn more . Flexible instance configurations in Clou

New intelligent suggestions for formulas and functions in Google Sheets

  You’ll now see in-line, sequential, context-aware suggestions for formulas and functions when working with data in Google Sheets.  Formula suggestions will make it easier to write new formulas accurately and help make data analysis quicker and easier. Simply begin inserting a formula in Sheets—suggestions will be automatically displayed and as you continue to type. You can view additional incremental suggestions in the drop-down menu. We hope these formula suggestions make it easier and faster for you to work with and analyze your data. Admins: There is no admin control for this feature. End users: This feature will be available  by default and can be disabled by going to Tools > Enable formula suggestions or from the three-dot menu of the suggestion dialog box. Rapid Release and Scheduled Release domains: Gradual roll-out (up to 15 days for feature visibility) starting on August 25, 2021 Available to all Google Workspace customers, as well as G Suite Basic and Business customers

Your support partner for Google Workspace / Microsoft 365

Your Support Partner For Google Workspace / Microsoft 365     Remote work is here to stay. mentions the percentage of "remote-capable" professionals who expect to work part of their time in the office and part of their time elsewhere—even if it's just one day a week—has risen to 53%, with even more, 59%, indicating that this is their preference. Today, 42% of respondents indicate they work in a hybrid arrangement, and 53% believe they will continue to do so in the future. With this emerging yet consistent trend, it should always be backed up with responsive & effective system support for your Google Workspace - which also provides you with the top notch collaboration tools to connect, create & collaborate.  1. Support at your ease.  Our Google Workspace Service offering thrives to enable you the full potential to work from anywhere, anytime. We stand by you for any Google Wor
Every sustainable business realizes that cloud migration is essential for long-term success. However, many people are hesitant to begin the migration process because they fear data loss and are hesitant to take such an enormous leap. Nevertheless, the reality is that  data will continue to grow, necessitating additional infrastructure to handle it, both in terms of data storage and processing power for effective use of data. Organizations that are hesitant to transition to the cloud sometimes miss out on the benefits that cloud-native capabilities can deliver, such as faster machine learning timelines and the potential to leverage artificial intelligence to harvest more insights from data. If at all your organization is hesitant to move to the cloud, we got you covered! Finetech offers end to end service in all aspects of Data Migration. This would not only from on-premises to the cloud, but also : Cloud (3rd party Cloud Email providers) to Google Cloud PST (local archives) to Google C
Big announcement!  As a part of our growth, Finetech is pleased to announce the expansion of our business in Bangladesh. With 12 years of excellence in helping organizations to digitize and providing with all aspects to keep up with the relevant, and up-to-date technology required to create an efficient workplace, Finetech Pledges to do so for the years to come throughout South Asia. As an objective to bring about the best Cloud services to South Asia, we have now kick-off our venture in Bangladesh.  We have identified Bangladesh as a fast developing country, and the need for an efficient work environment has been in need ever since. We understand the urgent technology challenges organizations face with incessantly changing tortuous business requirements. We also understand that orthodox ways of seeking solutions for these problems are no longer valid in the present day. We at Finetech believe that Cloud is the next big wave in technology. Our purpose is to bring the best of Cloud solu

Set start times and import reminders in Tasks

Here comes one of the most awaited features. Tasks is one of the goals to follow what you have to do in G Suite. These new updates will help ensure the majority of your to-dos are in Tasks, and guarantee that you can monitor the due dates related with them. Moreover, importing reminders to Tasks can support your users if your association is at present changing from Inbox to Gmail. Set a date and time for your tasks and receive notifications - You’ll find a place to add date & time. Create repeating tasks - Also you can make an event recur. Import reminders into Tasks This import tool will pull your reminders (from Inbox/Gmail, Calendar, or the Assistant) into Tasks.When importing reminders into Tasks, we’ll copy over the title, date, time and recurrence of the reminder. Please note, reminders with locations associated will not be imported. Additionally, this is a one-time import and not a constant sync. - When you open Tasks on the web or your mobile app, you’ll se

Live translated captions in Google Meet are now generally available

What’s changing  In 2021, we announced a beta for live translated captions in Google Meet. We’re now making live translated captions generally available for select Google Workspace editions.  Meeting participants can translate English meetings to:  French  German  Portuguese  Spanish  Translated captions are available on Google Meet on web and mobile devices.  Who’s impacted  End users  Why it’s important  Translated captions help make Google Meet video calls more inclusive and collaborative by removing language proficiency barriers. When meeting participants consume content in their preferred language, this helps equalize information sharing, learning, and collaboration and ensures your meetings are as effective as possible for everyone. This feature can be particularly helpful for all-hands meetings or training meetings with globally distributed teams. Additionally, translated captions can be impactful in education settings, allowing educators to connect and interact with students, p