Transforming JSON in a node stream with a map or template

I’m relatively new to Javascript and Node and I like to learn by doing, but my lack of awareness of Javascript design patterns makes me wary of trying to reinvent the wheel, I’d like to know from the community if what I want to do is already present in some form or another, I’m not looking for specific code for the example below, just a nudge in the right direction and what I should be searching for.

I basically want to create my own private IFTTT/Zapier for plugging data from one API to another.

I’m using the node module request to GET data from one API and then POST to another.

request supports streaming to do neat things like this:

request.get('http://example.com/api')
  .pipe(request.put('http://example.com/api2'));

In between those two requests, I’d like to pipe the JSON through a transform, cherry picking the key/value pairs that I need and changing the keys to what the destination API is expecting.

request.get('http://example.com/api')
  .pipe(apiToApi2Map)
  .pipe(request.put('http://example.com/api2'));

Here’s a JSON sample from the source API: http://pastebin.com/iKYTJCYk

And this is what I’d like to send forward: http://pastebin.com/133RhSJT

The transformed JSON in this case takes the keys from the value of each objects “attribute” key and the value from each objects “value” key.

So my questions:

  • Is there a framework, library or module that will make the transform step easier?

  • Is streaming the way I should be approaching this? It seems like an elegant way to do it, as I’ve created some Javascript wrapper functions with request to easily access API methods, I just need to figure out the middle step.

  • Would it be possible to create “templates” or “maps” for these transforms? Say I want to change the source or destination API, it would be nice to create a new file that maps the source to destination key/values required.

Hope the community can help and I’m open to any and all suggestions! 🙂
This is an Open Source project I’m working on, so if anyone would like to get involved, just get in touch.


Source: javascript

1 Comment

  1. Digin Dominic

    Yes you’re definitely on the right track. There are two stream libs I would point you towards, throughwhich makes it easier to define your own streams, and JSONStream which helps to convert a binary stream (like what you get from request.get) into a stream of parsed JSON documents. Here’s an example using both of those to get you started:

    var through = require('through');
    var request = require('request');
    var JSONStream = require('JSONStream');
    var _ = require('underscore');

    // Our function(doc) here will get called to handle each
    // incoming document int he attributes array of the JSON stream
    var transformer = through(function(doc) {
    var steps = _.findWhere(doc.items, {
    label: "Steps"
    });
    var activeMinutes = _.findWhere(doc.items, {
    label: "Active minutes"
    });
    var stepsGoal = _.findWhere(doc.items, {
    label: "Steps goal"
    });

    // Push the transformed document into the outgoing stream
    this.queue({
    steps: steps.value,
    activeMinutes: activeMinutes.value,
    stepsGoal: stepsGoal.value
    });
    });

    request
    .get('http://example.com/api')
    // The attributes.* here will split the JSON stream into chunks
    // where each chunk is an element of the array
    .pipe(JSONStream.parse('attributes.*'))
    .pipe(transformer)
    .pipe(request.put('http://example.com/api2'));

    Reply

Leave a Reply