How to investigate high tail latency when using Cloud Spanner
When you use Cloud Spanner, you may encounter some high tail latency cases. Some of the causes may be on the Cloud Spanner side, but there could be some other reasons as well. In this blog post, we will talk about how to distinguish the high latency causes and also talk about some tips to improve Cloud Spanner latency.
Check the relationship between the high latency and Cloud Spanner usage
- accessing other services had high latency
- the client machine had any resource shortage issue
- the high latency happened to a specific client machine
- sudden CPU utilization spike (this itself is not the cause, but indicates other processes in the machine may have caused the latency)
- hitting Disk I/O performance limit
- ephemeral port exhaustion and not being able to establish a TCP connection.
- high latency due coming from DNS queries