John Evans' Blog

Tag: programming

The Problem With Pipes

by on Nov.27, 2009, under Uncategorized

I’ve been playing around with Yahoo Pipes recently. This has mostly been in the realm of seeing what it can do; I haven’t found a really good use for it yet—at least not something that I am impressed by. But then, I don’t work with feeds very often, so maybe I’m not the "intended audience". There’s nothing wrong with that.

There is one interesting thing I’ve noticed, however. Pipes has a few obvious deficiencies in its language. The most obvious one is that it’s impossible (or at least very awkward, more on that later) to extract text from items. "text" and "items" are two of the Pipes data types, representing a text string and a group of feed entries, respectively.

Here’s an example to show what I mean. Let’s say you have a feed with a certain number of items, and you want to choose one specific item from it. This is not too difficult; I’ve already written a pipe to choose one item from a feed. This pipe takes two inputs: a URL and a number. It uses Fetch Site Feed to get the feed from the URL, and it uses some math and filters to get only the specified item. Simple!

The Example That Doesn’t Work

But let’s think of a slightly different example. Let’s say that you want to choose a specific item from a feed, but which item is given by a different webpage. Like you have a webpage that displays nothing but the number "3", and that means you want the third item from the feed. But you don’t know the number until run-time; it could be 3, or 4, or 1600, or anything. You want the pipe to query the webpage to find out which entry to get from the feed. This turns out to be difficult.

The problem is that while you can get the webpage data with Fetch Page—or Fetch Data, or even Fetch CSV if you want—those modules don’t return "number", they return "items". The math and filter modules need a "number" parameter to do their thing. "items" is a collection of data entries; it can’t be used as a string, even if there’s only one.

How to Solve the Problem

First, I should note that one part of this problem is not actually a problem. Pipes will convert strings into numbers. You can create a String Input containing "3", and can be hooked into a "number" input that will then be set to 3. So, the problem isn’t converting the text; the problem is getting it out of the "items" type.

I propose a new module, perhaps called String Extractor. In its simplest form, it would take in "items" and return "text" representing the default content of the first entry in the feed. If we wanted to get more fancy, we could add a "number" parameter to denote which entry to extract, and also perhaps a field to choose which element of the entry gets extracted (like Rename or Regex lets you choose).

I believe this module would be simple to implement, and while I don’t know much about the implementation, I have a good reason to believe this wouldn’t be too hard.

The Workaround

The truth is, it’s already possible to create a pipe that performs this function, using a "trick" that some Pipes developers have come up with. It actually involves creating two pipes.

1. First create a pipe that chooses a specific item from a feed. That was my first example, above.

2. Make a second pipe that fetches a number from a page. Process it until you have a feed with one item, containing the number.

3. Bring in a Loop module and place the first pipe inside it as a sub-pipe. Hook the number-feed up to the Loop module’s input. Set the sub-pipe’s number input to be "item.content".

The way this works is that "for every item in the input feed", the sub-pipe will be run on it and the content field will be used as the sub-pipe’s parameter. Of course, there’s only one item in the input feed, and it contains the specified number, so the loop is run once to choose the item from the feed.

Why a New Module?

Some might ask why we should be able to do this with Pipes at all. The new module would allow all sorts of facilities for interesting data filtering; much more like programming than simply mashing feeds together. I think this would be well worth it. And it’s obvious that the Pipes developers want this functionality, because a lot of them talk about it and know about this "trick" on the Pipes discussion forums.

So, some might also ask why a new module is needed if we can already perform this function. The answer is that the current way is completely awkward; that makes it hard for new Pipes developers to figure out. Maybe some of them have already been discouraged. Also, because this method requires a sub-pipe, it inflates the number of pipes.

And because the functionality of the module I’m proposing is already part of theLoop module, it’s certain to be easy to implement.

Leave a Comment :, , , , more...

Polymorphic function arguments

by on Nov.17, 2008, under Uncategorized

I just had this curious thought…

If you’re programming, you call functions a lot. Functions often have parameters. Sometimes they can have a lot of parameters. Sometimes it’s hard to remember which parameters are which; like, if the function is supposed to be called like “function(int amount, int x, int y)” and instead you give it the parameters “function(x, y, amt)” then something’s going to go wrong. Now, granted, in modern IDEs and editors it’s usually quite simple to look up the function definition and make sure you’re calling it correctly.

However…What if you could call the function arguments in some other order, as long as you knew what they were? Something like this: “function(x: 2, y: 3, amount: 1)”. There would be some specifier to let the compiler associate the parameters with their values.

I feel like I’ve seen some language that had this facility, but at the moment I can’t remember what it was.

4 Comments :, more...

Programming Stabilization

by on Jul.17, 2008, under Uncategorized

It seems to me that programming languages go through a sort of cycle. Someone comes up with a language or application to perform a task…it gets modified and worked on for a while…and then it stabilizes. And once that happens, people start building other languages off of it. So you have assembly language tied to a particular chip…running, say, a Linux kernel…that runs an Apache webserver…that runs Ruby on Rails (an application written in C that produces HTML as output). Each of those components has gone through years of testing and work before people figured it was mature enough to be part of this whole computing framework. At least, all of them until the final one, possibly…

Not that I’m saying people are completely finished developing Linux kernels…or even assembly language! I’m just saying they’re mostly finished. 😉

Leave a Comment : more...

Feed Me

by on Jun.02, 2008, under Uncategorized

So, recently I’ve been learning about RSS feeds. The basic idea of an RSS feed is that you have information that comes in small doses on a continuous basis. Like news updates…or blog entries.

RSS feeds are pretty easy to make. With the help of the W3C Feed Validator, I was able to code a small PHP script to create them. (If someone asks, I might be willing to share it, even though I’m sure other people have already done similar things.)

The really interesting thing about feeds is that they represent the idea of people pulling in information from different sources, sorting it, remixing it and finally re-presenting it in another form. A sort of web-bricolage.

LiveJournal itself allows you to denote feeds as part of your “friends page”. In fact, the entire “friends page” is itself an RSS aggregator, since LiveJournals may also be presented as RSS feeds. This brings me, however, to a gripe I have with LiveJournal. Why do I have to denote journals as “friends”? Why can’t I use LJ as a feed reader without all that social networking baggage? This is why I use Google Reader as my feed aggregator.

4 Comments :, , , , , more...

Beware…I live!

by on Apr.22, 2007, under Uncategorized

For those of you who might be wondering where I’ve been, I’ve been working. For my father’s company. It’s been busy.

My father, masterful ISTP artisan, has literally decades more experience programming than myself. I suspect, however, that in terms of natural intelligence we may be about equal. Today we were discussing a thorny coding problem. How would we check for out-of-sync network messages? It was the kind of puzzle you just had to beat your head against for a bit until you suddenly had an insight.

After a moment I thought to myself, well, maybe you could use the PeekMessage function to check if there were invalid messages waiting in the message queue.

And a moment later he said, “What if you peeked into the message queue…”.

Leave a Comment :, , more...


by on Dec.17, 2005, under Uncategorized

This was written to be as clear and unobfuscated as possible. (Of course, it’s much easier to read with an editor like Vim that highlights the text to show you what’s code and what’s data…)

$c = "";
$a = "\$_ = \$a;
print \"\\\$c = \\\"\$c\\\";\\n\\\$a = \\\"\$_\\\";\\n\$a\";";
$_ = $a;
print "\$c = \"$c\";\n\$a = \"$_\";\n$a";
1 Comment :, , more...


by on Dec.14, 2005, under Uncategorized

Today I finally succeeded at something I have struggled with every now and then for years.

What have I done? Well, check this out…

$a = "\$b = \$a;\$b =~ s/([\"\\\$\\\\])/\\\\\$1/g;print \"\\\$a = \\\"\$b\\\";\$a\";";$b = $a;$b =~ s/(["\$\\])/\\$1/g;print "\$a = \"$b\";$a";

4 Comments :, , more...

Looking for something?

Use the form below to search the site:

Still not finding what you're looking for? Drop a comment on a post or contact us so we can take care of it!

Visit our friends!

A few highly recommended friends...