John Evans' Blog


The Problem With Pipes

by on Nov.27, 2009, under Uncategorized

I’ve been playing around with Yahoo Pipes recently. This has mostly been in the realm of seeing what it can do; I haven’t found a really good use for it yet—at least not something that I am impressed by. But then, I don’t work with feeds very often, so maybe I’m not the "intended audience". There’s nothing wrong with that.

There is one interesting thing I’ve noticed, however. Pipes has a few obvious deficiencies in its language. The most obvious one is that it’s impossible (or at least very awkward, more on that later) to extract text from items. "text" and "items" are two of the Pipes data types, representing a text string and a group of feed entries, respectively.

Here’s an example to show what I mean. Let’s say you have a feed with a certain number of items, and you want to choose one specific item from it. This is not too difficult; I’ve already written a pipe to choose one item from a feed. This pipe takes two inputs: a URL and a number. It uses Fetch Site Feed to get the feed from the URL, and it uses some math and filters to get only the specified item. Simple!

The Example That Doesn’t Work

But let’s think of a slightly different example. Let’s say that you want to choose a specific item from a feed, but which item is given by a different webpage. Like you have a webpage that displays nothing but the number "3", and that means you want the third item from the feed. But you don’t know the number until run-time; it could be 3, or 4, or 1600, or anything. You want the pipe to query the webpage to find out which entry to get from the feed. This turns out to be difficult.

The problem is that while you can get the webpage data with Fetch Page—or Fetch Data, or even Fetch CSV if you want—those modules don’t return "number", they return "items". The math and filter modules need a "number" parameter to do their thing. "items" is a collection of data entries; it can’t be used as a string, even if there’s only one.

How to Solve the Problem

First, I should note that one part of this problem is not actually a problem. Pipes will convert strings into numbers. You can create a String Input containing "3", and can be hooked into a "number" input that will then be set to 3. So, the problem isn’t converting the text; the problem is getting it out of the "items" type.

I propose a new module, perhaps called String Extractor. In its simplest form, it would take in "items" and return "text" representing the default content of the first entry in the feed. If we wanted to get more fancy, we could add a "number" parameter to denote which entry to extract, and also perhaps a field to choose which element of the entry gets extracted (like Rename or Regex lets you choose).

I believe this module would be simple to implement, and while I don’t know much about the implementation, I have a good reason to believe this wouldn’t be too hard.

The Workaround

The truth is, it’s already possible to create a pipe that performs this function, using a "trick" that some Pipes developers have come up with. It actually involves creating two pipes.

1. First create a pipe that chooses a specific item from a feed. That was my first example, above.

2. Make a second pipe that fetches a number from a page. Process it until you have a feed with one item, containing the number.

3. Bring in a Loop module and place the first pipe inside it as a sub-pipe. Hook the number-feed up to the Loop module’s input. Set the sub-pipe’s number input to be "item.content".

The way this works is that "for every item in the input feed", the sub-pipe will be run on it and the content field will be used as the sub-pipe’s parameter. Of course, there’s only one item in the input feed, and it contains the specified number, so the loop is run once to choose the item from the feed.

Why a New Module?

Some might ask why we should be able to do this with Pipes at all. The new module would allow all sorts of facilities for interesting data filtering; much more like programming than simply mashing feeds together. I think this would be well worth it. And it’s obvious that the Pipes developers want this functionality, because a lot of them talk about it and know about this "trick" on the Pipes discussion forums.

So, some might also ask why a new module is needed if we can already perform this function. The answer is that the current way is completely awkward; that makes it hard for new Pipes developers to figure out. Maybe some of them have already been discouraged. Also, because this method requires a sub-pipe, it inflates the number of pipes.

And because the functionality of the module I’m proposing is already part of theLoop module, it’s certain to be easy to implement.

Leave a Comment :, , , , more...


by on Nov.20, 2009, under Uncategorized

I’m rereading my copy of the Dungeons & Dragons Master DM’s Book (the 1985 edition). It has a list of "artifacts" the players may come across, and near the end is a list of ideas from legendry that might be developed into other artifacts. There’s a whole list of swords: Balmung, Durandan, Excalibur, Flamberge, Gram, Joyeuse…

Wait a minute. Something about this list seems familiar…

Leave a Comment : more...

Telemarketing Fail

by on Nov.18, 2009, under Uncategorized

So I got two voicemail messages. The first says something about giving me a loan, “Don’t be alarmed, these are not new charges and you do not have to pay anything”, and it tries to entice me into calling their number. Fine, whatever, I don’t care.

The second one, however, has a woman who says “Looking over your paperwork, I think we can lower your interest rate and get you out of debt a lot sooner”.

The first amazing thing here is that I’m not in debt. They just call random numbers to see if they can get people in debt. Damn, it’s like telemarketer wardialing! There’s also an element of social engineering: “We have your paperwork. We know you’re in debt.” Of course, since I’m not, the whole thing is exposed as a lie.

The second amazing thing is that these are voicemails, and the first few seconds are cut off. This makes it obvious that these are recordings. Someone dials a number and—no, scratch that, a machine dials a number and plays a message to it. The sheer audacity of this is mind-boggling.


2 Comments : more...

by on Sep.29, 2009, under Uncategorized

WHAT THE HELL? I made this huge post and it VANISHED. I submitted the form and the text was just NOT THERE.

Note to self: Never, EVER write ANYTHING of substance in a web form. ALWAYS write it offline.

3 Comments more...

Games and politics

by on Sep.28, 2009, under Uncategorized

It turns out that one of the main guys behind Stardock watches and supports Fox Newsis supporting “fair and balanced” reporting. Well…crap. Now I’ll never be able to play those games again without being reminded of “fair and balanced” reporting. And I was looking forward to a new fantasy strategy game…

Well, there’s always Dwarf Fortress.

Leave a Comment :, , more...

News & and a quiz

by on Sep.12, 2009, under Uncategorized

So, I haven’t posted here much, but that’s all right. I have a few other places around the web where I post updates.

John Evans’ Facebook Page
John Evans’ Twitter
Chaoseed Software
Chaos Garden – Game Design Blog

In terms of actual news: Things are going quite well. Sometime in the next month or so I’ll have some news…Maybe good news, maybe disappointing, but we’ll see. Until then I won’t have much to say.

Beyond that, a friend of mine had one of those ‘quiz’ things she wanted me to do (well, I did volunteer). So, I’ll write about that…but, be warned, you may be learning more about me than you wanted to know! 😉

Reply to this post by yelling "Words!" and I will give you five words that remind me of you. Then post them in your LJ and explain what they mean to you.

Words thing

5 Comments : more...

Scams and Plans

by on Aug.04, 2009, under Uncategorized

I keep getting voicemail about “an important business matter”. I delete them. Note to scammers businesses: If you don’t identify yourself, I assume you have no legitimate reason to contact me.

In completely unrelated (?) news, I remixed an old Charles Atlas advertisement. See if you can spot the difference!

full image

7 Comments :, , more...

Long Island Driving Tips

by on Jun.30, 2009, under Uncategorized

(or, How to Drive Like a Long Island Driver)

1. If you leave any space between your car and the car in front of you, this is considered an invitation for other cars to pull into your lane in front of you. If you don’t wish to offer this option, follow as closely as you can to the car in front of you.

2. “No Turn On Red” signs were ruled unconstitutional and may be safely ignored.

3. Certain intersections are designed to provide obvious and careful ways to turn left across traffic, including clearly marked turn lanes, specific traffic light signals and signs saying “Left Turn Signal”.

…but you don’t have to use those lanes.

4. Red lights are only guidelines; if you really need to go through the intersection, you can. Even if the cars pointed at you have started to move.

5. If nobody crashed, who cares?

(this is a joke)

Leave a Comment :, more...

Looking for something?

Use the form below to search the site:

Still not finding what you're looking for? Drop a comment on a post or contact us so we can take care of it!

Visit our friends!

A few highly recommended friends...