July 19: Missing app logs because we forgot a `systemctl restart`
#July 19: Missing app logs because we forgot a `systemctl restart` (14:32UTC)
Some customers reported that their app logs appeared to be missing. Initially we suspected that this was related to log ingestion (we use vector on each physical host), but further investigation revealed that… we forgot to restart the query service after adding a new storage node to the logging cluster, so it only queried the previous set of nodes. Because each app’s log stream is written to one storage node, apps whose logs ended up stored on the new node would return empty or incomplete results. Restarting the log query service restored normal log visibility.