Understanding UniqueFilter stage from WebSpider demo

[RaelB 4]

Hello,

I am looking at the code from WebSpider demo from HL-III presentation. (The project is just called pipeline)

 

It has code like this:

procedure TfrmWebSpider.UniqueFilter(const input, output: IOmniBlockingCollection);
var
  uniqueUrls: TStringList;
  url       : string;
begin
  uniqueUrls := TStringList.Create;
  try
    uniqueUrls.Sorted := true;
    for url in input do begin
      if uniqueUrls.IndexOf(url) < 0 then begin
        uniqueUrls.Add(url);
        output.TryAdd(url);
      end
      else if FURLCount.Decrement = 0 then
        FSpider.Input.CompleteAdding;
    end;
  finally FreeAndNil(uniqueUrls); end;
end;

When I run the demo, I see that the uniqueUrls TStringList is working on all inputs received. This is what we want, but how does this work?, since it is a local variable one would expect/imagine that it goes out of scope each time UniqueFilter is "called".

 

Thanks

Rael

[Primož Gabrijelčič 223]

UniqueFilter does not go out of scope. If defined in this format (with input, output: IOmniBlockingCollection parameters), pipeline only calls it once.

 

The 'for url in input' loop is the one that processes all elements that arrive via the input pipeline.

[RaelB 4]

Thanks, that is quite amazing. You're saying the loop is actually pausing/resuming depending on queue activity.

[Primož Gabrijelčič 223]

Indeed. IOmniBlockingCollection implements an enumerator which waits for the next available value. The only way to terminate such for loop is to call input.CompleteAdding which signals the enumerator that no new values can ever be produced.