Comments on: Chunky Iterator: So You Don’t Have to Load All Your AR Objects at Once

By: Daniel Higginbotham

Daniel Higginbotham — Fri, 23 May 2008 14:15:31 +0000

Good catch Frank, thanks!

By: Frank Kuepper

Frank Kuepper — Wed, 21 May 2008 05:26:31 +0000

Shouldn’t the existing_condition in merge_conditions be enclosed in braces? If existing_condition for example was
"fresh = TRUE OR amount > 5"
the correct merge should be
"(fresh = TRUE OR amount > 5) AND #{@model_class.table_name}.id > #{id}"
whereas your code produces
"fresh = TRUE OR amount > 5 AND #{@model_class.table_name}.id > #{id}"

By: This Week in Ruby (May 20, 2008) | Zen and the Art of Programming

This Week in Ruby (May 20, 2008) | Zen and the Art of Programming — Tue, 20 May 2008 22:20:43 +0000

[…] remarkable articles from this week were: Chunky Iterator: So You Don’t Have to Load All Your AR Objects at Once, Do we really need Controller and View tests?, Guide to Unobtrusive JavaScript and SWFUpload, […]

By: Karl

Karl — Wed, 14 May 2008 00:03:50 +0000

Looks like I had to do this again today, iterate/update over a large set. So I threw together a rake task to do such, but took a different route. Should be db independent, and doesn’t depend on offsets.

I’m not saying it’s better, just a different approach. Probably slower if a large percentage of the records need updates.

http://blog.spoolz.com/2008/05/13/butt-biter-fix-state-abbreviations-large-data-set-updates/

By: links for 2008-05-13, or so says Harry Love

links for 2008-05-13, or so says Harry Love — Tue, 13 May 2008 16:31:52 +0000

[…] Flying Machine Studios » Blog Archive » Chunky Iterator: So You Don’t Have to Load All Your AR O… Iterate over a collection of AR objects in groups of n without having to load them all at once (tags: rubyonrails activerecord iterators) Tags: Found Objects […]

By: Daniel Higginbotham

Daniel Higginbotham — Tue, 13 May 2008 13:42:27 +0000

Thanks for the great comments. I didn’t realize that with limit/offset, query time can increase as offset gets larger. Thanks as well for the link to Jamis Buck’s solution - it should be easy to combine what I have here with his solution so that the “cursored” find can take the other find options.

That’s great that will_paginate has a method that will do this - before I wrote this post I was actually thinking that it would make sense to extend will_paginate to do this. Looking through the code, it looks like the ActiveRecord#paginate method uses offset, so I guess paginated_each would also take increasing amounts of time.

By: Grant Hutchins

Grant Hutchins — Tue, 13 May 2008 02:44:06 +0000

Recent versions of the will_paginate plugin include a method called paginated_each to do something similar.

http://groups.google.com/group/will_paginate/browse_thread/thread/59856834eb52b4d0

By: Evgeniy

Evgeniy — Tue, 13 May 2008 02:34:12 +0000

solution can miss some records if inserts/deletes happening at the same time (even within block to process) - imagine you iterate to update or delete, on the next iteration - offset will jump over the last end

better to stick to Jamis Buck solution
http://weblog.jamisbuck.org/2007/4/6/faking-cursors-in-activerecord

By: Peter Jones

Peter Jones — Tue, 13 May 2008 02:29:50 +0000

Looks a lot like my all_records plugin. Seems like this is a pretty universal need, and maybe should be in Rails?

By: Mark Wilden

Mark Wilden — Tue, 13 May 2008 01:44:37 +0000

Looks good. However, I’d DRY it up by using a do-while-do loop, so as to avoid duplicating the find call. Unfortunately, the only common language that supports that control structure (as far as I know) is Smalltalk, so you’d have to simulate it with code like:

do
find
if not found
break
yield
increment
loop

///ark