If you’ve checked our status page recently, you’ll have seen that we’ve been having issues indexing deposited metadata in our REST API. That’s caused delays and confusion for some of our members, and frustration for users of the metadata.
Apologies if you’ve noticed issues. That makes you one of those who have spotted that we’ve been migrating services to the cloud. The REST API wasn’t included (it was already running in the cloud) but it has experienced some knock-on effects.
Indexing delays and slow responses
When we receive new metadata, it is stored in our “content system” database. On a regular basis, batches of these records are sent to be indexed by the REST API. The service that creates and pushes these batches was recently migrated and it hasn’t handled the change well. We left it some time to try and settle down, but it has continued to be problematic. We’ve been able to reindex the missed records using a workaround, but that isn’t a good option longer-term. We haven’t yet been able to pin down what’s going wrong: the code for the pusher service is very old and opaque, and difficult to debug.
A consequence of the reindexing workaround is that when we repush large batches of metadata all at once, the REST API resources spend much more on writing and less on reading. This means that the API has been periodically slower than normal and some users have seen timeouts.
Rebalancing
We are looking at a few options to bring more stability back to the REST API:
- We can rebalance reading and writing. There are various parameters we can adjust to make sure that writing large amounts of data doesn’t affect metadata retrieval and we’re trying to optimise these.
- It may be quicker to rewrite the code that pushes metadata to the REST API. Even if we can debug it this time, we might be in the same situation again in a few months. We’re investigating whether we can get a rewrite done in a reasonable amount of time.
- Requests for single DOIs are getting held up by more complex requests (using filters and query parameters). To make overall performance quicker, we will put these into a separate queue and handle them quickly. That should make REST API performance more efficient all round.
- The REST API is based on elasticsearch, but we aren’t using the latest version. We will be upgrading in the coming weeks, which should improve performance.
We have a few other ideas as well. In the meantime, we ask for some patience. Keep an eye on the status page and sign up for alerts to stay informed about outages and longer-term issues.