#29 Experimental foreach support

[jofers]

I've written preliminary support for foreach in my FreeBASIC branch. That is, it would be possible to iterate over a collection of items in a UDT by calling:
foreach i as counter_type in udt_with_iter
...
next i

It's available at
git://github.com/jofers/fbc.git

And I'm attaching a sample program

My goal was to say binary compatible with gcc's implementation of C++11's "range-based for". For my simple example, I'm able to save my UDT and iterator objects to a library, import them in a C++11 file in gcc 4.6, and iterate over that object using a range-based for.

C++11's range-based for translates the following:
for ( counter_type i : udt_with_iter )
into
for( iter_type j = udt_with_iter.begin(); j != udt_with_iter.end(); j++ )
counter_type i = *j

This has a few stumbling blocks for FB
1) There is no "++" operator in FB.
2) Having "end" for a method name conflicts with the statement

I worked around #1 by adding the "++" operator as overridable, but not usable operator. It's a temporary measure, and my thought was you could use the "++" name mangling symbol for implicit steps, or create a new "stepeach" operator. Or even add "increment x", though that seems unnecessary.

For #2, I used "ending" instead of "end". You can still use alias to stay interoperable with C++, but it's a pretty hokey solution. I'm short on ideas here.

Reasoning:
"FB has iterators" - It does, but foreach is widely implemented in other BASIC compilers
"Doesn't VB have a foreach that's a model for this?" I think all .NET languages can implement IEnumerable, but FB doesn't have interfaces yet
"It's a funky system for beginners" - True, C++'s foreach solution is pretty awkward for people not used to BASIC. But people are going to start creating libraries that use it and FB users want to import those libraries. I'm not die-hard on this style of iterator, but gcc compatibility is good.

To make it more usable for beginners, I like the idea of borrowing iterator generators from python:
iterator myiter (a as integer)
for i as integer = 1 to a
yield i
next i
end iterator
foreach x as integer in myiter(2)
print x ' prints 1 2
next x

Here, calling myiter would instead pipe the function to a fiber version of threadcall which returns an iterator object. Pretty far off, but feasible. But to get even close, you need some sort of FOREACH implemenation.

There are a few kinks:
This implements foreach for static objects. I'm guessing c++ lets you iterate over pointers (It's a brand new language feature so I dont' know), but it's a possibility for later.
C++'s range-based for is more flexible ATM, in that you can also define begin(x as udt_with_iter_type) as a global function. That could be added as well.

[jofers]

Example of using FOREACH

[dkl]

This sounds quite similar to FB's For on UDTs. It'd be good to try and reuse stuff from there if that's possible. I'm thinking for example instead of adding a ++ operator, the step() operator could be re-used, maybe in form of a special overload. But yeah, it's also a question of C++ compatibility, although for me that alone is not the most important thing.

I think a ForEach loop could be useful in FB, as in many people would use it, and it should definitly be made to support FB's strings and arrays too. Going through array elements without having to use lbound() and ubound() explicitly would be pretty nice, I imagine :)

[jofers]

Belated thanks for the feedback. We could easily use the step operator here, but we'd have to map step() with no arguments as "++" in the ABI. FB's existing iterators, much as I dislike them, could be used with foreach in a couple different ways:

Way 1:
type foo
declare constructor bar(byval as bar ptr)
declare operator for()
declare operator next() as integer
declare operator step()
end type
foreach i as foo in bar
print i.value
next i

Way 2:
type foo
' normal iterator that takes in type baz
end type
type bar
declare function iterator() as bar
declare property from as baz
declare property to as baz
end type
foreach i as foo in bar
print i.value
next i

I'm pretty swamped lately, but I started work on supporting python-esque iterator generators via fibers. I was pleased to find out VB is adding support for them in VS 2011, so I tweaked the syntax a bit from my last post to line up with their version. Unfortunately, MS decided to make their iterator generators return a generic interface (two things we don't support), so I shrugged and decided to use the "as ___" part for the yield type.

The following now works on my fork:

iterator function myiter( byval x as integer ) '' as integer <- TODO: yield type
for i as integer = 1 to 10
yield i*x
next i
end function

'' foreach i as integer in myiter( 2 )
'' print i
'' next i

dim fiber as any ptr = myiter( 2 )
dim i as integer ptr
do
'' iterate
fiberswitch fiber
i = fibergetyield( fiber )
if i = 0 then exit do

'' for loop body
print *i

loop
fiberdestroy fiber

I need to add type checking for the yield and sugar up the for loop, but I'm pretty happy how that's turning out. the iterator qualifier on "function" triggers the generation of two functions to built directly into the AST: a "funnel" function that packs the arguments into a heap and passes them to a fiber worker. And a "scatter" function which unpacks the parameters inside the fiber and calls the original function (really a sub). When a user calls the iterator, the original is swapped out for the generated funnel function. The fiber___ functions are added to the RTL and wrap to Windows fiber functions (I started but didn't complete a version using "setcontext" on Linux).

[jofers]

I got distracted for awhile, but I added support to my FBC branch for built-in iterators:

iterator function myiter( byval x as integer ) as integer
for i as integer = 1 to 10
yield i*x
next i
end function

foreach i as integer in myiter( 2 )
print i
next i

The magic bits are the fiber routines from the previous comment and a new "iterator" datatype that is basically a pointer with some extra metadata in its subtype. For instance, this also works:
dim x as integer iterator = myiter( 2 )
foreach i as integer in x
...
next i

It needs a lot of clean-up before it's production-ready, but that should be MUCH more intuitive for beginners than either C++ or FB iterators.