Universal Datafeed Format. Is There One?
Submitted by Stephen Carter on Mon, 06/09/2008 - 16:54.
as a product developer, i am familiar with the approaches to parsing any form of product datafeed, but i haven't looked at the datafeeds of many merchants. i assume that some people here have.
i'm wondering what the most common format is. the reason i am wondering is that i'm trying to get an idea of what is required to write a generalized product feed parser. i'm not looking for a third-party generalized product feed parser, but if you know of one that has public documentation on what it does, that would be useful.
here is what i think about datafeeds in general, thinking about them as an affiliate:
RSS is pretty limited unless one adds custom namespaces and i assume if merchants use RSS they don't use namespaces: they stick to the basic fields: title, link, description. you cannot do much creatively with that. for example, you wouldn't have enough information to create a product comparison site.
XML is more promising because one can add lots of fields and not worry about namespaces. but i assume there is no standardization on the field elements and attribute values. that would mean one couldn't write an all-purpose parser for XML datafeeds (as in, this is the product image url, that's a hoodackey field, and so on).
CSV format is probably the worse datafeed format because you absolutely must know something beforehand about the nature of the fields you are importing. you won't find it in the file itself. so CSV will never be a candidate for universal datafeed format, even if is common.
so i'm wondering what approach most merchants take to this. if you have experience with datafeeds, please chime in with your opinion.
I've been using CSV files,
I've been using CSV files, but I agree - they are limiting.
I think XML would be the best format, however, I don't know if there are any default namespaces/schemas out there. I would love to see somebody create one though...
jeremy, you know what just
jeremy, you know what just occurred to me?
t's probably going to take a super affiliate to stand up at a conference one day and say "hey you affiliate datafeed managers, we have a problem in the datafeed area, and i think it would be great if you guys came up with a standard for us affiliates to use. here's my simple proposal, i hope you guys can improve on it..." and then the super affiliate puts up a slide with a sensible XML proposal that gets people thinking.
a proper standard would allow developers to build applications around the format, without knowing which merchant is providing the datafeed. just like it is today with RSS feeds. today anyone can provide a feed that conveys the news from their site. in the future, anyone with a standarized product datafeed will be able to publish their datafeed and have their catalog appear almost instantly on the pages of thousands of affiliate sites.
one thing is for sure. we currently live in the dark ages of affiliate marketing, and one day we'll look back and say "remember when we use to deal with CSV files? sheesh. how did we survive?"
i have been working recently with the Amazon Associates Web Service and i kind of like the way that works. the XML structure is too complex to serve as a standard, but at least you can do useful things with the data.
anyway, congratulations on bringing project Black Ink to a conclusion. you've done a remarkable job!
oops. i wasn't signed in.
just so we know who's ranting about datafeeds. it's me :)
Stephen Carter
creator of Review Foundry
Standard on data feed method and delivery
Data feed method and delivery is such a vast subject, I believe it can not be consolidated just into one standard. But I do agree we are all having a headache and waste plenty of time parsing different feed formats and structure.
Lets say we have General Products, Books, Music, Video, Services, Travel, etc... as the MAIN data feed groups. That would help narrow down the standard.
Regarding XML, I would prefer JSON. JSON stands for Javascript Object Notation. It was created for AJAX data transport. It is small, brief and concise.
I have personally worked and provided data feed / web services using XML, JSON can do the same like XML but make the programmers 's job easier by providing the data in array format already.
Good discussion keep it up...
Andrew Nurcahya
founder of DataFeedFile.com
i suspect that JSON will
i suspect that JSON will never be used as a product feed standard, simply because it contains more symbols than just the left and right angle brackets if XML. it has both arrays and hashes as part of its syntax, which does remove some ambiguities associated with XML (like whether a sequence is intended to repeat or not). so i like that, but i don't see it as a serious alternative to XML.
also, i suspect there probably wouldn't be categories like books, music, video, etc. there would only be format types (digital goods and subscriptions), shipping types (physical goods), and event types (concerts, hotel stays, etc). that way the standard could remain relatively compact and cover unanticipated product types that don't fit neatly into any proposed category hierarchy.
but you are right. coming up with a standard isn't straightforward.
Stephen Carter
creator of Review Foundry