Unhandled exception: LongPollingException: HttpClientException: 429 Too Many Requests: retry after 5

[busslina]

Describe the bug

Is not usual that this bug happens, but it arrives some times, maybe once every two months. It happened to me last night, at an hour without any activity on my server.

I think this error is very clear, no further explanation needed.

Additional context

I see my version is 0.3.4 while 0.5.3 is latest. I will upgrade and test if it is resolved on last version. If anyone can confirm this point I would appreciate.

Stacktrace

Unhandled exception:
LongPollingException: HttpClientException: 429 Too Many Requests: retry after 5
#0 LongPolling._onRecursivePollingHttpError (package:teledart/src/teledart/fetch/long_polling.dart>
#1 LongPolling._recursivePolling. (package:teledart/src/teledart/fetch/long_pol>
#2 _rootRunUnary (dart:async/zone.dart:1399:47)
#3 _CustomZone.runUnary (dart:async/zone.dart:1300:19)
#4 _FutureListener.handleError (dart:async/future_impl.dart:165:22)
#5 Future._propagateToListeners.handleError (dart:async/future_impl.dart:778:47)
#6 Future._propagateToListeners (dart:async/future_impl.dart:799:13)
#7 Future._completeError (dart:async/future_impl.dart:574:5)
#8 _completeOnAsyncError (dart:async-patch/async_patch.dart:318:13)
#9 _rootRunBinary (dart:async/zone.dart:1415:47)
#10 _CustomZone.runBinary (dart:async/zone.dart:1307:19)
#11 _FutureListener.handleError (dart:async/future_impl.dart:162:22)
#12 Future._propagateToListeners.handleError (dart:async/future_impl.dart:778:47)
#13 Future._propagateToListeners (dart:async/future_impl.dart:799:13)
#14 Future._completeError (dart:async/future_impl.dart:574:5)
#15 Future.timeout. (dart:async/future_impl.dart:878:17)
#16 _rootRunBinary (dart:async/zone.dart:1415:47)
#17 _CustomZone.runBinary (dart:async/zone.dart:1307:19)
#18 _FutureListener.handleError (dart:async/future_impl.dart:162:22)
#19 Future._propagateToListeners.handleError (dart:async/future_impl.dart:778:47)
#20 Future._propagateToListeners (dart:async/future_impl.dart:799:13)
#21 Future._completeError (dart:async/future_impl.dart:574:5)
#22 Future._asyncCompleteError. (dart:async/future_impl.dart:665:7)
#23 _rootRun (dart:async/zone.dart:1391:13)
#24 _CustomZone.run (dart:async/zone.dart:1293:19)
#25 _CustomZone.bindCallback. (dart:async/zone.dart:1225:23)
#26 _microtaskLoop (dart:async/schedule_microtask.dart:40:21)
#27 _startMicrotaskLoop (dart:async/schedule_microtask.dart:49:5)
#28 _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:122:13)
#29 _Timer._runTimers (dart:isolate-patch/timer_impl.dart:405:11)
#30 _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:192:12)

[DinoLeung]

@busslina it does not look like it's a bug, the error message shows that you are using the long pull method and TeleDart is trying to retry. Please check your API key and make sure it's valid.

[busslina]

@DinoLeung Do you know how can I do to prevent my whole app crashes? Because is an unhandled exception. Maybe a configuration argument or a try catch. But it seems that if it can be resolved with try catch it has to be in your lib code (because as you can see in the StackTrace, there are no trace of my code in order to put try catch clause).

Maybe I'm missing something

[busslina]

@busslina it does not look like it's a bug, the error message shows that you are using the long pull method and TeleDart is trying to retry. Please check your API key and make sure it's valid.

My Bot Token Api is 100% valid. It has been working okay before and after that crash.

The only thing I think that I could done wrong is to have a older version of Teledart. Now I'm using the latest

[DinoLeung]

@busslina it does not look like it's a bug, the error message shows that you are using the long pull method and TeleDart is trying to retry. Please check your API key and make sure it's valid.

My Bot Token Api is 100% valid. It has been working okay before and after that crash.

The only thing I think that I could done wrong is to have a older version of Teledart. Now I'm using the latest

Sorry @busslina there's was indeed some updates to the file long_polling.dart recently, but I'm not sure it has fixed anything particularly. Please monitor the issue. The error telegram thrown was pretty clear that the server is making too many requests to the API, therefore they temporarily block the client until the cool down period has past. it might not necessarily caused by the the recursive long poll, it could be some other part of the application spamming the server and appeared to throw errors when getting updates.

[busslina]

@DinoLeung I'm sorry too, but I firmly think that is caused by your lib (maybe corrected in the newer versions) because I don't comunicate with telegram api. I only use your lib to initiate the bot, some little configurations by using setMyCommands method once. The rest is only managed by your lib to listen new messages & commands. I will attach my file to be clear.

Also, the hour at what happend the crash last night (2AM) indicates the error source: the long polling. I reviewed the server log and zero activity happend in the whole night.

The solution I think of is a) Restart my server every time crashes (via Systemd unit file) or modify your lib to catch that error (but I won't do that).

I understand that is not your fault if it works for two months in a row and suddenly it fail like this. Maybe is Telegram disturbing, I don't know, but the error is pretty clear: #0 LongPolling._onRecursivePollingHttpError (package:teledart/src/teledart/fetch/long_polling.dart> https://github.com/DinoLeung/TeleDart/pull/1 LongPolling._recursivePolling. (package:teledart/src/teledart/fetch/long_pol> and is there where patch must be done (if not yet done).

Well, if that happen again with the latest version, I will just follow the error stack trace, put in there a try/catch and send a PR.

My code:

class TelegramBot {
  // Data
  final ConfigBase config;
  late final TeleDart bot;
  final List<MessageHandler> commandMiddlewares;
  final List<BotCommand> commandList;
  final List<MessageHandler> messageHandlers;
  final bool listenToMessages;

  /// Constructor.
  TelegramBot({
    required this.config,
    required this.commandMiddlewares,
    required this.commandList,
    required this.messageHandlers,
    required this.listenToMessages,
  });

  // Functions
  //
  // - 01 - Start
  // - 02 - On message
  // - 03 - On command
  // - 04 - Send message

  //#region
  /// 01 - Start.
  Future<void> start() async {
    final username = (await Telegram(config.botToken).getMe()).username;
    bot = TeleDart(config.botToken, Event(username!));
    await _configurate();
    bot.start();

    // Listen to to messages (async)
    if (listenToMessages) {
      // (async) -- Messages
      bot.onMessage().forEach(onMessage).catchError(_onReceivingError);

      // (async) -- Commands
      bot.onCommand().forEach(onCommand).catchError(_onReceivingError);
    }
  }

  /// 02 - On message.
  Future<void> onMessage(teledart.TeleDartMessage message) async {
    // Execute handlers cascade
    for (final handler in messageHandlers) {
      // Execute handler
      if (!await handler(message)) {
        // Break cascade
        return;
      }
    }
  }

  /// 03 - On command.
  Future<void> onCommand(teledart.TeleDartMessage message) async {
    // [01]: Execute middleware handlers cascade
    for (final middleware in commandMiddlewares) {
      // Execute middleware
      if (!await middleware(message)) {
        // Break cascade
        return;
      }
    }

    // [02]: Routing to match command (if any)
    final command = _parseCommand(message.text!);
    // message.text
    for (final botCommand in commandList) {
      // Check match
      if (!botCommand.checkMatch(command)) {
        continue;
      }

      // Executing command handler
      await botCommand.handler(message);
      return;
    }
  }

  /// 04 - Send message.
  Future<teledart.Message?> sendMessage(
    int chatId,
    String message, {
    bool debug = true,
  }) async {
    try {
      return await bot.sendMessage(chatId, message);
    } catch (e) {
      _onSendingError(e, debug: debug);
      return null;
    }
  }
  //#endregion

  // Private functions
  //
  // - 01 - Configurate
  // - 02 - On receiving error
  // - 03 - On sending error
  // - 04 - Parse command

  //#region
  /// 01 - Configurate.
  Future<void> _configurate() async {
    // [01]: My commands
    if (!await bot.setMyCommands(commandList.map((e) => e.command).toList())) {
      clib.error('TelegramBot._configurate() -- Error setting my commands');
    }
  }

  /// 02 - On receiving error.
  FutureOr<dynamic> _onReceivingError(Object error) async {
    lib.debug(
        'TelegramBot._onReceivingError() -- Error type: ${error.runtimeType} -- Error: $error');
  }

  /// 03 - On sending error.
  void _onSendingError(
    Object error, {
    bool debug = true,
  }) async {
    lib.debug(
        'TelegramBot._onSendingError() -- Error type: ${error.runtimeType} -- Error: $error',
        active: debug);
  }

  /// 04 - Parse command.
  String _parseCommand(String command) {
    var ret = command.trim().split(' ').first.replaceFirst('/', '');

    // Removing bot username
    final index = ret.indexOf('@');
    if (index != -1) {
      ret = ret.substring(0, index);
    }

    return ret;
  }
  //#endregion
}

[busslina]

It happend again, now with the latest version of Teledart. I don't want to bother you but I think you are not being logically/scientifically with this issue. The error is very clear (package:teledart/src/teledart/fetch/long_polling.dart:113) and in return you are saying that the error is in my code (it would be the best case for me, because I would have fix it yet).

Same scenario that last time. 2 AM, server almost sleeping (like me)... and crash. Long polling file, line 113.

I'm sorry about my language but I don't understand your first reaction of "echar balones fuera (throw balls out)" like we say in Spain, while the error is pretty clear.

Thank you

LongPollingException: HttpClientException: 429 Too Many Requests: retry after 5
Jan 21 02:10:10 ip-xxx-xx-xx-xxx bash[54133]: #0 LongPolling._onRecursivePollingHttpError (package:teledart/src/teledart/fetch/long_polling.dart:113:7)
Jan 21 02:10:10 ip-xxx-xx-xx-xxx bash[54133]: #1 LongPolling._recursivePolling. (package:teledart/src/teledart/fetch/long_polling.d>

[busslina]

@busslina it does not look like it's a bug, the error message shows that you are using the long pull method and TeleDart is trying to retry. Please check your API key and make sure it's valid.

I'm not using the long polling method. I'm using your Library which uses the long polling method. So it is up to you (to your Library) to provide a way to catch the HTTP 429 failure status code and prevent the whole app to crash. Don't you agree?

[busslina]

I propose you a patch with this PR: #229

[HeySreelal]

Hey @busslina and @DinoLeung, I was investigating this issue. From the logs @busslina shared I'm pretty sure that the long polling with the library is actually failing. But not sure why this issue is not appearing on a regular scale.

What I came up with is a wild guess that the Telegram servers start sending 429 errors when around 5-30 requests per second occur. On faster servers where the developer's code is deployed, the _recursivePolling could easily reach this limit with the current implementation - which is basically one request right after the other is finished.

Maybe we should add a lil bit of wait time like at least a 200ms delay till the next _recursivePolling() call.

Note: The 200ms is my simple calculation as 1000 / 5.

[HeySreelal]

Oh well, I didn't see that PR. Unfortunately, I don't think a constant 5-second delay is needed whenever this 429 issue occurs. Instead, there's obviously a retry_after parameter sent by Telegram servers with the error. That'd be much better if you're actually planning to write an auto-retry setup.

[busslina]

Hey @busslina and @DinoLeung, I was investigating this issue. From the logs @busslina shared I'm pretty sure that the long polling with the library is actually failing. But not sure why this issue is not appearing on a regular scale.

What I came up with is a wild guess that the Telegram servers start sending 429 errors when around 5-30 requests per second occur. On faster servers where the developer's code is deployed, the _recursivePolling could easily reach this limit with the current implementation - which is basically one request right after the other is finished.

Maybe we should add a lil bit of wait time like at least a 200ms delay till the next _recursivePolling() call.

Note: The 200ms is my simple calculation as 1000 / 5.

I think you are 100% right.

[busslina]

Oh well, I didn't see that PR. Unfortunately, I don't think a constant 5-second delay is needed whenever this 429 issue occurs. Instead, there's obviously a retry_after parameter sent by Telegram servers with the error. That'd be much better if you're actually planning to write an auto-retry setup.

Yes, I saw that retry_after header in 429 HTTP error, but as you can see here (https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429) this header is not mandatory to be included.

I will update my PR to include the reading of this header and if it exists, await for that amount of time and if it does not exists then await 2500 ms.

Thank you

[paulobreim]

This type of error happens in any library and is due to connections with telegram itself.
I have several BOTs and they all have these issues occasionally. This already happened when I was in Ruby (I migrated everything to Dart).
What I do in all of them, to avoid problems, is to run a .sh script
#!/bin/sh
while [ 1 ]; of
gives you
./mybot
done

That way I never have problems.

[busslina]

@paulobreim Your solution is like mine: reboot after fail. But it is not an acceptable solution if a more specific solution can be done.

[busslina]

Now it comes the funny thing. Last night it crashed again and seeing the log I found it crashed at 02:10 AM, exactly the same hour the crash before (Jan. 21). So I looked for the first crash (the one that made me open this issue on Jan. 16) and surprise... same hour.

January 16:

Jan 16 02:10:13 ip-xxx-xx-xx-xxx bash[6019]: Unhandled exception:
Jan 16 02:10:13 ip-xxx-xx-xx-xxx bash[6019]: LongPollingException: HttpClientException: 429 Too Many Requests: retry after 5
Jan 16 02:10:13 ip-xxx-xx-xx-xxx bash[6019]: #0 LongPolling._onRecursivePollingHttpError (package:teledart/src/teledart/fetch/long_polling.dart:114:7)
Jan 16 02:10:13 ip-xxx-xx-xx-xxx bash[6019]: #1 LongPolling._recursivePolling. (package:teledart/src/teledart/fetch/long_polling.da>
Jan 16 02:10:13 ip-xxx-xx-xx-xxx bash[6019]: #2 _rootRunUnary (dart:async/zone.dart:1399:47)

January 21:

Jan 21 02:10:10 ip-xxx-xx-xx-xxx bash[54133]: Unhandled exception:
Jan 21 02:10:10 ip-xxx-xx-xx-xxx bash[54133]: LongPollingException: HttpClientException: 429 Too Many Requests: retry after 5
Jan 21 02:10:10 ip-xxx-xx-xx-xxx bash[54133]: #0 LongPolling._onRecursivePollingHttpError (package:teledart/src/teledart/fetch/long_polling.dart:113:7)
Jan 21 02:10:10 ip-xxx-xx-xx-xxx bash[54133]: #1 LongPolling._recursivePolling. (package:teledart/src/teledart/fetch/long_polling.d>
Jan 21 02:10:10 ip-xxx-xx-xx-xxx bash[54133]: #2 _rootRunUnary (dart:async/zone.dart:1399:47)

January 23:

Jan 23 02:10:19 ip-xxx-xx-xx-xxx bash[84580]: Unhandled exception:
Jan 23 02:10:19 ip-xxx-xx-xx-xxx bash[84580]: LongPollingException: HttpClientException: 429 Too Many Requests: retry after 5
Jan 23 02:10:19 ip-xxx-xx-xx-xxx bash[84580]: #0 LongPolling._onRecursivePollingHttpError (package:teledart/src/teledart/fetch/long_polling.dart:113:7)
Jan 23 02:10:19 ip-xxx-xx-xx-xxx bash[84580]: #1 LongPolling._recursivePolling. (package:teledart/src/teledart/fetch/long_polling.d>
Jan 23 02:10:19 ip-xxx-xx-xx-xxx bash[84580]: #2 _rootRunUnary (dart:async/zone.dart:1399:47)

I randomly restart my server at random hours, I didn't had any server activity at the time those crashes happend, ...
The first thing that I could think about it is that Telegram servers do some routinary stuff at that hour and maybe send 429, even if TeleDart is respecting the limits.

@HeySreelal proposed to me to read the retry_after header. I did that and proposed a PR (#230)

Come on @DinoLeung, mistakes are human

[paulobreim]

Some times I have HttpClientException: 429 Too Many Request.
It is occurs when some user ask something and the answers are long and divide in peaces. I think that put a delay after each answer
This is a security impose by Telegram. There are no bug in the @DinoLeung's lib.

[busslina]

Some times I have HttpClientException: 429 Too Many Request. It is occurs when some user ask something and the answers are long and divide in peaces. I think that put a delay after each answer This is a security impose by Telegram. There are no bug in the @DinoLeung's lib.

We are drowning in a glass of water. To clarify:

1. Unhandled exception occurs when an error happens in an async block of code and this error is not catched either a) on the async block of code with try/catch or b) in the caller of this async block (function) with Future.catchError method.
   (a) - https://dart.dev/guides/libraries/futures-error-handling#potential-problem-accidentally-mixing-synchronous-and-asynchronous-errors
   (b) - https://api.dart.dev/be/175791/dart-async/Future/catchError.html

2. In this particular case, unhandled exception occurs here: LongPolling._onRecursivePollingHttpError (package:teledart/src/teledart/fetch/long_polling.dart:113:7
   link.

3. I'm not interested in say this is an @DinoLeung bug/error or Telegram's . Just act logically with the facts. And the facts are pretty clear. This is an Unhandled exception because @DinoLeung code do not await here.

4. The only way I can think of to prevent an app crashes using this library when this event (bug/error) happens is to handle in a different way. @DinoLeung is throwing an error here and If he would await all async code everyone could have caught that error by surrounding Bot.start method with a try/catch. He decided an other logic, and it is okay. His error is to throw exception when receives 429 HTTP. The better way is to wait some time and re-polling again (the solution I propose here). Easy my friends. It is all about logic.

5. Lets understand the whole thing/issue deeply before talking easy. Yes I'm a little bit tired of discussing such an easy thing. I propose a solution (maybe not the better one). If ego is an obstacle to accept things it is not my problem. I prefer to everyone have a better version of this library, even if this error does not affect everyone even once, because one day it can happen (just wait to 2 AM haha).

[HeySreelal]

I totally agree with all the first 4 points. And I don't think ego is the problem here. He might already be working on many other things or even other parts of this library. I could see a lot of changes in the last few weeks. Well, we're all humans and have priorities.

Yeah, simply returning to an awaited future could solve this just as you said we could catch the error with a try-catch block.

Every library has its own problems and advantages. At this point you have choices like:

1. You can edit the library code on your local machine and run it that
2. Use another library.
   I've always been a fan of good bot libraries. My personal best choice is grammy (which is a Telegram bot framework on Node.js) If you want to stick to Dart, choose one from the pub.dev. Eg: Televerse (my own production lol, just to overcome disabilities of other packages.)

[busslina]

@HeySreelal Yes, I will use my own fork of this library locally, waiting this issue to be solved (or PR applied). I came to Dart from NodeJS and I don't want to return there. I appreciate your suggestion and I saved your lib in favourites.

If I said the ego thing is because @DinoLeung first reaction is to said it was my error, that I was doing polling requests, ... Even If you are bussy I think that it is not acceptable. But no problem, I will do it by my own.

[paulobreim]

I have 4 BOTs running.
Sometimes, errors like: LongPollingException or TimeoutException occur.
When this happens, it happens on ALL Bots at the same time, which indicates that it is something on the Telegram side and not on my application.

[busslina]

I have 4 BOTs running. Sometimes, errors like: LongPollingException or TimeoutException occur. When this happens, it happens on ALL Bots at the same time, which indicates that it is something on the Telegram side and not on my application.

Yes, I have deduced that when I realized that it was always crashing at same hour (02:10 AM CET). The point here is to not letting unexpected external behavior make your app crash. To understand that you must read and understand this Lib code. I resolved this issue here #230

@DinoLeung was causing the crash throwing that exception. With the new code (not accepted yet), instead of throwing that not manageable error, now it wait some amount of time and re-poll again.

[paulobreim]

Is this implemented in teledart: ^0.5.3 version?

I will change onde of the BOTs to see what happiness.

[busslina]

Is this implemented in teledart: ^0.5.3 version?

I will change onde of the BOTs to see what happiness.

No, it is not. It is implemented in a pending Pull Request (waiting to be accepted).

You can try this implementation using a local package (like I'm doing). You can try to download my fork repo (https://github.com/busslina/TeleDart) and use it in your app's pubspec.yaml file like in the example below:

My server didn't crash again. It was such an easy mistake. I don't understand how a good coder cannot accept it and correct it for the good of all its users.