Simulate throttling of the OpenAI API
This sample contains a preset and mocks to simulate the throttling of the OpenAI API. As more and more applications use OpenAI API to use AI, it's important that developers can verify that their apps can properly handle cases when the API is throttled. Especially when the app is using lower-tier API plans, throttling is more likely to happen so validating the app's behavior is crucial to ensure great user experience. Using this preset you can simulate throttling of the OpenAI API and see how your app will handle it. The preset includes 3 different throttling scenarios: exceeded tokens per minute, exceeded number of requests per minute and service overload, which includes dynamic simulation of handling retry-after. Proxy will simulate throttling the API using one of these modes at random.
Simulate throttling of the OpenAI API
Summary
This sample contains a preset and mocks to simulate the throttling of the OpenAI API.
As more and more applications use OpenAI API to use AI, it’s important that developers can verify that their apps can properly handle cases when the API is throttled. Especially when the app is using lower-tier API plans, throttling is more likely to happen so validating the app’s behavior is crucial to ensure great user experience.
Using this preset you can simulate throttling of the OpenAI API and see how your app will handle it.
Compatibility
Contributors
Version history
| Version | Date | Comments |
|---|---|---|
| 1.27 | October 3, 2026 | Added devproxy config get steps |
| 1.26 | September 28, 2026 | Updated to Dev Proxy v3.3.1 |
| 1.25 | September 28, 2026 | Updated error responses to match the current OpenAI API: overload is a 503, added slow_down and credit_balance_exhausted |
| 1.24 | July 1, 2026 | Updated to Dev Proxy v3.1.0 |
| 1.23 | June 17, 2026 | Updated to Dev Proxy v3.0.1 |
| 1.13 | May 30, 2026 | Updated to Dev Proxy v3.0.0 |
| 1.12 | March 26, 2026 | Updated to Dev Proxy v2.3.0 |
| 1.11 | March 11, 2026 | Updated to Dev Proxy v2.2.0 |
| 1.10 | February 4, 2026 | Updated to Dev Proxy v2.1.0 |
| 1.9 | January 18, 2026 | Moved config files to .devproxy folder |
| 1.8 | January 5, 2026 | Updated to Dev Proxy v2.0.0 |
| 1.7 | June 27, 2025 | Updated to Dev Proxy v0.29.2 |
| 1.6 | January 9, 2025 | Updated to Dev Proxy v0.23 |
| 1.5 | January 17, 2024 | Updated plugin path |
| 1.4 | January 11, 2024 | Updated to new format |
| 1.3 | January 7, 2024 | Added new throttling error |
| 1.2 | December 22, 2023 | Updated to new format |
| 1.1 | November 14, 2023 | Renamed to Dev Proxy |
| 1.0 | August 10, 2023 | Initial release |
Minimal path to awesome
- Get the sample:
-
Download just this sample:
npx gitload-cli https://github.com/pnp/proxy-samples/tree/main/samples/openai-throttlingor
-
Download as a .ZIP file and unzip it, or
-
Clone this repository
-
- Start Dev Proxy by running
devproxyin the sample’s folder
Alternatively, download the preset with Dev Proxy and start it from any folder:
devproxy config get openai-throttling
devproxy --config-file "~dataFolder/configs/openai-throttling/.devproxy/devproxyrc.json"Features
This preset simulates the throttling and capacity errors that the OpenAI API currently documents:
429exceeded tokens per minute (rate_limit_exceeded), with aRetry-Afterheader429exceeded requests per minute (rate_limit_exceeded), with aRetry-Afterheader
429request rate increased too quickly (slow_down), with aRetry-Afterheader429no prepaid credits remaining (credit_balance_exhausted). Retrying doesn’t help, so your app shouldn’t retry it.503model temporarily overloaded (server_is_overloaded), with aRetry-Afterheader
Proxy will simulate one of these errors at random. The RetryAfterPlugin checks that your app waits for the time specified in the Retry-After header before retrying.
The error responses are modeled after the OpenAI API error codes.
Help
We do not support samples, but this community is always willing to help, and we want to improve these samples. We use GitHub to track issues, which makes it easy for community members to volunteer their time and help resolve issues.
You can try looking at issues related to this sample to see if anybody else is having the same issues.
If you encounter any issues using this sample, create a new issue.
Finally, if you have an idea for improvement, make a suggestion.
Disclaimer
THIS CODE IS PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING ANY IMPLIED WARRANTIES OF FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR NON-INFRINGEMENT.