It has recently come to my attention that donut-bot had re-processed some older posts. Thank you for bringing this to my attention.
The reason this happened is because the Reddit API returned those submissions as 'new' posts that had recently been submitted. In my years as a Reddit bot dev, I have seen this happen from time to time and the Reddit API usually corrects itself within a relatively short amount of time.
To combat this, I track the posts that have previously been processed by donut-bot so that I can avoid reprocessing them in the future (whether that is due to an API glitch or if the bot restarts). However, I only kept the previous 90 days history of post metadata in the database in an attempt to keep the database size small.
Changes Implemented:
I have implemented 2 changes which should correct this issue moving forward:
1) I will now keep post metadata (ie. the posts previously processed) in the database indefinitely. Relatively little data is stored about each post and relatively few posts are submitted each day. This should not cause database growth that we are unable to handle.
2) I added additional logic to donut-bot to determine if a post has been previously handled. Previously it only looked at the database to see if that post shows up in the history. However, there are some edge cases where a post is not written to history (such as it not meeting the 300-word limit, being rejected due to topic-limiting, and other similar edge cases). After performing the very quick lookup in the history (#1 above), if it did not find a match it will now add an additional lookup to see if donut-bot has left a top-level comment on the post previously. If it did, then we know that donut-bot has previously processed this post in the past.
Between these two changes, I feel this should prevent the issue happening again. Please let me know if this problem continues to occur or if you notice any side effects from these changes (I am effectively 'testing in production' here. This is in fact, testing the bot).
Thank you all for your continued support.
Donut-bot update to address reprocessing older/historic posts
byu/mattg1981 inethtrader
Posted by mattg1981