Rate limiting
Rate limiting applies to the Shared Models service in order to ensure quality of service for all users and manage utilisation of hardware.
Status codes
The following HTTP status codes apply to inference requests to models.think.evroc.com:
429: you have made too many requests and are being rate limited.503: the service is currently experiencing high load and no server is currently available for your request.
In both cases we recommend retrying your request after the time (in seconds) set
in the Retry-After header.
The exact thresholds used for rate limiting are subject to change as necessary to maintain service availability.
Busy times
As a European inference provider, evroc Shared Models are typically busiest during local office hours, ie 09-17h European time. For non-interactive workloads, better throughput is likely to be available outside these hours.