I'm a user and have been negatively impacted by the feed fetching optimizations - daily feeds are often a few days behind and come in bunches. Two examples:
- Penny Arcade updates its comics Monday, Wednesday, and Friday, always at 7:01AM UTC, and then news other times during the week. It's Wednesday at 4:25PM UTC - 9 hours after - and goread hasn't picked it up.
- Dinosaur Comics is updated weekdays. I'll eventually get all of them, but usually two or three at a time. For example, yesterday I marked all my feeds as read; today, I have entries from Monday and Tuesday, but not from Wednesday.
I had hoped that the move to the everyone-pays model would give you the resources (either developer or quota) to fix these issues, but they've gotten no better or maybe worse.
I haven't looked at what you're doing, but I believe Google Reader used pubsubhubbub where available to reduce/eliminate polling for many popular feeds.
I honestly didn't have a great experience with my last bug report, so I haven't tried again.
At Theneeds we use a "sliding windows" approach to deal with polling. Say you run the scraper every hour. Each feed F_i is scraped once every n_i polls. If the feed returns more than his average news, then n_i gets decreased, while if the feed returns no news, it gets increased.
Perhaps with a similar trick you can run your scraper more frequently on some feeds, still keeping the cost under control.
Thanks! I don't really know when the website has a problem, when goread has a problem, and when I have a problem, so I end up assuming that it's goread problem. I like how the desktop web version gives a "last refreshed" and "next refreshed" indication.
I'm a user and have been negatively impacted by the feed fetching optimizations - daily feeds are often a few days behind and come in bunches. Two examples:
- Penny Arcade updates its comics Monday, Wednesday, and Friday, always at 7:01AM UTC, and then news other times during the week. It's Wednesday at 4:25PM UTC - 9 hours after - and goread hasn't picked it up.
- Dinosaur Comics is updated weekdays. I'll eventually get all of them, but usually two or three at a time. For example, yesterday I marked all my feeds as read; today, I have entries from Monday and Tuesday, but not from Wednesday.
I had hoped that the move to the everyone-pays model would give you the resources (either developer or quota) to fix these issues, but they've gotten no better or maybe worse.
I haven't looked at what you're doing, but I believe Google Reader used pubsubhubbub where available to reduce/eliminate polling for many popular feeds.
I honestly didn't have a great experience with my last bug report, so I haven't tried again.