CouchApp uploading attachments (part 2)


In my previous post I presented how to upload attachments to a document using HTML (there were few lines of JavaScript for retrieving the last revision of the document).

In this post I will shift to almost doing everything in JavaScript in order to have more control.

Uploading an attachment into CouchDB using JavaScript

Simplest case: validation

The very first step is introduce a validation for checking that we have chosen a file to upload. I want to get something like:

Error on CouchApp attachment upload
Error on CouchApp attachment upload

So I start with the following HTML:

<form id="upload" method="post" enctype="multipart/form-data">
    <label style="display:none; visibility: hidden;">
        Revision :
        <input id="revision" type="text" name="_rev"/>
    </label>
    <input id="attachment" type="file" name="_attachments"/>
    <br/>
    <input type="button" value="Submit"/>
</form>

Basically it’s a form with three input: one for the revision (that is hidden and filled from JavaScript); while the second is for the attachment (the file); the last one is just a button that I will use for invoking the validation procedure.

This is the piece of code for getting document revision.

// Get the document revision
var db = $.couch.db("hermesapp");
var id = "doc_with_attachment";
db.openDoc(id, {
    success:function (result) {
        // Update revision input field
        $("#revision").val(result._rev);
    },
    error:function (a, b, c) {
        alert("Error: " + c);
        console.log("Error on openDoc:", a, b, c);
    }
});

Now, the validation code that has to run when the button is clicked:

$("input[type='button']", "#upload").click(function (ev) {
    var empty = false;
    var form = $("#upload");
    $.each($("input[type='file'][name='_attachments']", form), function (idx, input) {
        if (input.value === "") {
            empty = true;
        }
    });
    if (empty) {
        alert("No attachment file defined!");
        return;
    }
    // Submit the form with the attachment.
    ...
})

I get a reference to the form and inside the form to any input which type is file and which name is _attachments and check that all of them are not an empty string (the loop is because we can send multiple attachments by repeating this input).

Submitting the validated form might be as simple as:

// Submit the form with the attachment
form.attr("action", "/hermesapp/" + $.couch.encodeDocId(id));
form.submit();

or if I want to use ajaxSubmit from jquery.form.js:

// Submit the form with the attachment
form.ajaxSubmit({
    url:"/hermesapp/" + $.couch.encodeDocId(id),
    success:function (response) {
        alert("saved");
        console.log("response", response);
    },
    error:function (a, b, c) {
        alert("error");
        console.log("errors", a, b, c);
    }
});

Avoiding the fact of showing the message received from the server:

CouchDB message on successful upload
CouchDB message on successful upload

and getting instead:

Message on successful attachment upload in CouchDB
Message on successful attachment upload in CouchDB

NOTE: this solution requires that you use jquery.forms.js that you can already find it in your CouchDB server (since futon uses it). In order to download it in your couchapp as we did with jquery.couch.js you should access http://localhost:5984/_utils/script/jquery.form.js (replacing localhost by the IP address of your CouchDB server and saving it with the other JavaScript files (see here for more details on setting up a CouchApp).

Stay tuned!

Next post convers Uploading attachments from JavaScript in CouchDB using PUT instead of POST: why and how. Subscribe to this blog to receive notice.

Advertisements

JSON floats become BigDecimal using JCoachDB…


When converting  types from one representation into another we always know that there are some issues with serialization and deserialization. Of course with JSON is not different!

Type representation representation

Assumptions:

  1. We are using Java for interfacing with CouchDB
  2. We have chosen jcouchdb.
  3. jcouchdb uses svenson for converting from Java into JSON and viceversa.
  4. We have a CouchDB BaseDocument and we want to store an Integer, a Long, a Float and a Double.

Our code is something like:

// Create a document
BaseDocument doc1 = new BaseDocument();
doc1.setId("test_0001");
doc1.setProperty("Integer", new Integer(10));
doc1.setProperty("Long", new Long(10));
doc1.setProperty("Float", new Float(10));
doc1.setProperty("Double", new Double(10));

// Persist it into the database
db.createDocument(doc1);

Lets see what we get when we read that document from the database and print its values.

// Retrieve it
BaseDocument doc2 = db.getDocument(BaseDocument.class, doc1.getId());

// Print values
System.out.println("Values...");
System.out.println("Integer      " +
        doc1.getProperty("Integer") + " / " +
        doc2.getProperty("Integer"));
System.out.println("Long         " +
        doc1.getProperty("Long") + " / " +
        doc2.getProperty("Long"));
System.out.println("Float        " +
        doc1.getProperty("Float") + " / " +
        doc2.getProperty("Float"));
System.out.println("Double       " +
        doc1.getProperty("Double") + " / " +
        doc2.getProperty("Double"));

And the output is…

Values...
Integer      10 / 10
Long         10 / 10
Float        10.0 / 10.0
Double       10.0 / 10.0

Seems fine! But… what happens if we print the types of the arguments…

// Print types
System.out.println("Types...");
System.out.println("Integer      " +
        doc1.getProperty("Integer").getClass().getSimpleName() + " / " +
        doc2.getProperty("Integer").getClass().getSimpleName());
System.out.println("Long         " +
        doc1.getProperty("Long").getClass().getSimpleName() + " / " +
        doc2.getProperty("Long").getClass().getSimpleName());
System.out.println("Float        " +
        doc1.getProperty("Float").getClass().getSimpleName() + " / " +
        doc2.getProperty("Float").getClass().getSimpleName());
System.out.println("Double       " +
        doc1.getProperty("Double").getClass().getSimpleName() + " / " +
        doc2.getProperty("Double").getClass().getSimpleName());

Now we get as output…

Types...
Integer      Integer / Long
Long         Long / Long
Float        Float / BigDecimal
Double       Double / BigDecimal

We see that types have changed integer numbers are always Long while floating point numbers are BigDecimal.

Problems with conversion and precision

The question on this is problems with precision.
The following example creates a BaseDocument with an Integer, Long, Float and Double but now instead of 10 we have a value of 123.456.

// Create a document
BaseDocument doc1 = new BaseDocument();
doc1.setId("test_0002");
doc1.setProperty("Float", new Float(123.456));
doc1.setProperty("Double", new Double(123.456));

// Persist it into the database
db.createDocument(doc1);

// Retrieve it
BaseDocument doc2 = db.getDocument(BaseDocument.class, doc1.getId());

And when we print the value using:

// Print values
System.out.println("Values...");
System.out.println("Float        " +
        doc1.getProperty("Float") + " / " +
        doc2.getProperty("Float"));
System.out.println("Double       " +
        doc1.getProperty("Double") + " / " +
        doc2.getProperty("Double"));

we get:

Values...
Float        123.456 / 123.45600000000000307
Double       123.456 / 123.45600000000000307

It is not such a big difference but if you compare numbers, of course, they will not be the same…
And the problem is with the representation of 123.456 using the different data types.

System.out.println("Float      : " + new Float(123.456));
System.out.println("Double     : " + new Double(123.456));
System.out.println("BigDecimal : " + new BigDecimal(123.456));

shows

Float      : 123.456
Double     : 123.456
BigDecimal : 123.4560000000000030695446184836328029632568359375

CouchDB poll!!!


I’d like to know how people is using CouchDB and I couldn’t find a better way than asking it!

The very first question is if you are actually using it, evaluating it, playing with it, curious about it… Might also be interesting knowing if recent movement of Damien Katz (leaving Apache CouchDB) might affect your decision.

Then if you use it from one of programming language or deploy your application inside CouchDB, or is it a HTML/JavaScript application that uses AJAX for interacting with CouchDB…

Third question is for people using it from Java and wondering about which API / library are you using…

Please, leave any comment or question that you would like to have included in this poll.

CouchDB beam uses 100% of CPU and network


While playing around with CouchDB I have observed that my system was running really slow (actually first of all I notice it heating) and the System Monitor showed that both CPU was at 100% and network usage was very high. The responsible for this was beam.

Doing some Google research I found that beam was related to CouchDB but the only thing that I was trying to do is connecting to a CouchDb server and after getting a DB Connector using a wrong user/password combination, try to create a document.

Debugging my Java code I found that the system was running some code in Apache HTTP client that for some reason did not finish (showing an error or throwing an Exception) but keep trying and trying.

Trying to find the problem I downloaded the sources of the latest version (4.2) of httpclient, httpclient-cache and httpcore for debugging them and … surprise! the problem disappear. Then I downloaded the sources for version 4.1.1 (the one that I was using before) and failed again.

Conclusion: If you have problems with beam and you are using apache-http version try upgrading it.

NOTE:

CouchDB, jcouchdb and map (in more detail)


I have already written a very first tutorial about jcouchdb and map. But for those used to write SELECT in SQL there a lot of things other than those simple queries.

Nevertheless, you should switch your mind since working with CouchDB map/reduce function will be a completely new paradigm and there are no magic recipes about how to convert SQL tables on CouchDB documents and SELECTS on map/reduce functions.

Example Database and create, update, query Views in CouchDb

First, I will refer you to the Database that I created as example here. Where I inserted some small documents with some few properties and the helper functions for creating / updating / querying views introduced here.

Limiting the number of results in jcouchdb

HelperFunctions.defineView(db, "sortedById",
                           "function(doc) { emit(doc._id, doc); }");
ViewResult result = db.queryView("views/sortedById",
                                 BaseDocument.class,
                                 new Options().limit(100),
                                 null);
// Check that we get 100 results
assertEquals(result.getRows().size(), 100);

In line 1-2 we define a map function that emits the identifier of the document  and the document itself. Then we call queryView (line 3) method asking for a document of type BaseDocument (line 4) and we introduce a new method in Options class called limit that allows you to specify the maximum number of document to retrieve.

Line 8 verifies that we actually get 100 documents.

NOTEViewResult contains a method name getTotalRows that returns the number of row that meet the condition BUT not the ones that have been retrieved.

One additional question is that map returns the results ordered by key (first argument of emit function) so, in the previous example, we got the results ordered by identifier.

Getting results ordered with jcouchdb

Consider the following code that defines a view for getting results ordered by date.

HelperFunctions.defineView(db,
                           "sortedByDate",
                           "function(doc) { emit(doc.date, doc); }");
ViewResult result = db.queryView("views/sortedByDate",
                                               BaseDocument.class,
                                               new Options().limit(100),
                                               null);
List rows = result.getRows();

// Check that we get 100 results
assertEquals(rows.size(), 100);

// Verify that they are actually sorted by date (key)
for (int i = 1; i < rows.size(); i++) {
    assertTrue(((String) rows.get(i - 1).getKey()).compareTo((String) rows.get(i).getKey()) }

As in the previous example we define a view (using our HelperFunctions) and then invoke Database queryView. We limit the number of results to 100 (line 6) and then in line 14-16 we iterate in results comparing that each date compared with previous is actually equal or greater.

NOTE: we can actually do this because we saved dates as YYYY-MM-DD.

Querying in jcouchdb for a key value that is in a set of values

Consider the same map function (sortedByDate) that in previous section and now we are going to use queryViewByKeys instead of queryView that includes an extra argument that is the list of values to match the key that we are looking for.

HelperFunctions.defineView(db, "sortedByDate", "function(doc) { emit(doc.date, doc); }");
List list = Arrays.asList("2012-01-01", "2012-01-02", "2012-01-03");

ViewResult result = db.queryViewByKeys("views/sortedByDate",
                                                     BaseDocument.class,
                                                     list,
                                                     null,
                                                     null);
List<ValueRow> rows = result.getRows();
assertTrue(rows.size() != 0);

// Check that results are actually one of the desired
for (ValueRow row : rows) {
    String date = (String) row.getValue().getProperty("date");
    assertTrue(list.contains(date));
}

In line 2 we define a list of Strings containing those dates that we want to match.

In line 4 we can see the extra argument (the list of keys to match).

Lines 13 through 16 we check that the date on the retrieved documents are actually included in the list.

NOTE: method getRow used in line 14 returns the second argument used in the emit of the map function (the document for this sortedByDate map function).

How do you see it so far?

First experiments with map-reduce


In the previous post I have shown how to create / update / query a map function for creating a view using jcouchdb Java library.

In this one I’m going to show how to use reduce functions.

Create a View in CouchDB

In that  post I created a helper function that allowed me to create / update a view but this function did not define a reduce function. So, I’m going to introduce a slight change for setting reduce function as well as map.

public class HelperFunctions {
    private static final String ViewsPath = "views";
...
    // Define view with map and reduce function
    public static void defineView(Database db,
                                  String name,
                                  String mapFn,
                                  String reduceFn) {
        DesignDocument doc = new DesignDocument(ViewsPath);

        // Check if the documents exists...
        try {
            DesignDocument old = db.getDesignDocument(doc.getId());
            doc.setRevision(old.getRevision());
            doc.setViews(old.getViews());
        } catch (NotFoundException e) {
            // Do nothing, it is enough knowing that it does not exist
        }
        View view = new View();
        view.setMap(mapFn);
        view.setReduce(reduceFn);
        doc.addView(name, view);
        db.createOrUpdateDocument(doc);
    }

    // Define view only with map function
    public static void defineView(Database db,
                                  String name,
                                  String mapFn) {
        defineView(db, name, mapFn, null);
    }
}

I created a new defineView with an extra argument (reduceFn) and moved the code to this implementation. Previous interface is kept and invokes the new with null value for reduceFn argument indicating that no reduce function.

In the previous code we add an extra instruction (line 21) for setting reduce function (view.setReduce(reduceFn)).

And that’s it! Pretty easy, isn’t it?

Using map – reduce for building a histogram

For showing how to use map and reduce, we are going to count the number of documents that we created in the previous post for each date.

The map function is:

function(doc) {
    emit(doc.date, 1);
}

and the reduce function:

function(keys, values) {
    return sum(values);
}

Basically the view emits a pair keyvalue where key is the date and the value is 1 for each date. Then reduce function uses internal sumfunction for counting the values (always 1).

But, we want the counter grouped per date (that’s why we set the key equal to document’s date). This is controlled programmatically with the third argument of queryView (Options).

        // Define map and reduce functions
        HelperFunctions.defineView(db,
                "dateHistogram",
                "function(doc) { emit(doc.date, 1); }",
                "function(keys, values) { return sum(values); }");
        // Execute query and group results by key
        ViewResult<Object> result = db.queryView("views/dateHistogram",
                Object.class,
                new Options().group(true),
                null);
        // Display results
        List<ValueRow<Object>> rows = result.getRows();
        for (ValueRow<Object> row : rows) {
            System.out.printf("%s - %s\n", row.getKey(), row.getValue());
        }

In previous examples of queryViews I didn’t set any option, here I set group to true for grouping results by key, this is simply done with new Options().group(true).

NOTE: queryViews by default executes reduce function, if defined, but you can disable it invoking the method reduce(false) from Options.
The output is:

2012-01-01 - 351
2012-01-02 - 328
2012-01-03 - 339
2012-01-04 - 347
2012-01-05 - 359
2012-01-06 - 311
2012-01-07 - 327
2012-01-08 - 326
2012-01-09 - 356
2012-01-10 - 361
2012-01-11 - 337
2012-01-12 - 320
2012-01-13 - 309
2012-01-14 - 336
2012-01-15 - 334
2012-01-16 - 302
2012-01-17 - 325
2012-01-18 - 297
2012-01-19 - 323
2012-01-20 - 356
2012-01-21 - 347
2012-01-22 - 304
2012-01-23 - 312
2012-01-24 - 332
2012-01-25 - 347
2012-01-26 - 329
2012-01-27 - 332
2012-01-28 - 384
2012-01-29 - 333
2012-01-30 - 336

If we would like to count the number of values that are even and odd, it’s enough defining a map function as:

function(doc) {
    emit(doc.even, 1);
}

The output is:

no - 5000
yes - 5000

First experiments with map function


While in Relational Databases you write select statements for choosing a subset of your original database, in CouchDB  you write map-reduce functions that lets you extract part of the data to work with.

The reason for calling the views as map is because extracted data are pairs {key, value}. Where both key and value can be any JSON data structure. Views are sorted by keys.

Some characteristics of map functions are:

  • Filter documents from your database to let you work with some of them.
  • Extract part of the original document.
  • Sort (filtered) documents.
  • Do calculations on the data of the document (using reduce).
In addition, CouchDB builds indexes on the extracted data making its usage very efficient.

Write documents

Lets start writing a bunch of documents in our Database that then we will use for playing with map and reduce.

        Random random = new Random(System.currentTimeMillis());
        int totalSize = 10000;
        int bulkSize = 1000;

        // Create totalSize documents
        int records = 0;
        while (records < totalSize) {
            // Create a list of documents that are going to be created using 'bulk'
            List docList = new ArrayList();
            Date firstDate = new GregorianCalendar(2012, GregorianCalendar.JANUARY, 1).getTime();
            Calendar calendar = GregorianCalendar.getInstance();
            SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd");

            for (int i = 0; i < bulkSize; i++) {
                int value = (records + i);
                calendar.setTime(firstDate);
                calendar.add(Calendar.DATE, random.nextInt(30));

                BaseDocument doc = new BaseDocument();
                doc.setProperty("name", "name." + value);
                doc.setProperty("value", value);
                doc.setProperty("even", (value % 2) == 0 ? "yes" : "no");
                doc.setProperty("date", sdf.format(calendar.getTime()));
                docList.add(doc);
            }
            List infoList = db.bulkCreateDocuments(docList);
            assertEquals(infoList.size(), docList.size());
            for (DocumentInfo info : infoList) {
                assertNotNull(info.getId());
                assertNotNull(info.getRevision());
            }
            records += bulkSize;
        }

Each document has the following fields:

  • name: The string name plus a dot and a sequence number (example: name.1234).
  • value: The sequence number used in name.
  • even: yes or no depending if the number is even or odd.
  • date: January 1st, 2012 plus a random number of day between 0 and 30 and formatted as yyyy-MM-dd (year with 4 digits, month as 2 digits and day as 2 digits). The reason for choosing this format year-month-day is because then they are sorted and will allow us to choose dates in a range.

In order to check that we actually created bulkSize documents on each iteration, we iterate on the list of DocumentInfo returned by bulkCreateDocuments and use assert to check that they actually have an identifier.

This is a pretty small document but will allow me to show some basics on map-reduce.

Define a view

Views are defined using JavaScript functions where you evaluate if one document has to be emitted (be part of the final view).

In the following example we choose those documents where the value of even field is yes and value is less than 10. The view contains as key the name and value is the identifier of the document.

function(doc) {
    if (doc.even == 'yes' && doc.value < 10) {
        emit (doc.name, doc._id);
    }
}

The resulting displayed using CouchDB Futon is:

emit function has two arguments that are going to be the key and the value of the resulting map.

Keys might be null (useful if you need all results in no special order) or complex structures if you need more than one item (used in views where you want to choose by more than one key).

Create a View in CouchDb

Views are Design Documents and we I have defined a helper function for defining and updating a view.

public class HelperFunctions {
    private static final String ViewsPath = "views";

    public static void defineView(Database db,
                                  String name,
                                  String mapFn) {
        DesignDocument doc = new DesignDocument(ViewsPath);

        // Check if the documents exists...
        try {
            DesignDocument old = db.getDesignDocument(doc.getId());
            doc.setRevision(old.getRevision());
            doc.setViews(old.getViews());
        } catch (NotFoundException e) {
            // Do nothing, it is enough knowing that it does not exist
        }
        View view = new View();
        view.setMap(mapFn);
        doc.addView(name, view);
        db.createOrUpdateDocument(doc);
    }
}

See this for more details on this function.

Basic query view in CouchDB

Pretty simple…

        ViewResult result = db.queryView(map, BaseDocument.class, null, null);

We just need two arguments, the name of the map function and the class of the output documents (here we used BaseDocument). The result is a ViewResult<BaseDocument> than includes the list of BaseDocuments emitted by the map function.

        // Define a view called lessThan10
        // that emits as key a field named value and as value the id of the document
        HelperFunctions.defineView(db, "lessThan10",
                "function(doc) { if (doc.value < 10) emit(doc.value, doc._id); }");
        // Query view with no options and not keys
        ViewResult result = db.queryView("views/lessThan10", BaseDocument.class, null, null);

        // Check that we get 10 results as expected (see DB creation above)
        assertEquals(result.getRows().size(), 10);

        // Display results
        List<ValueRow> rows = result.getRows();
        for (ValueRow row : rows) {
            System.out.printf("%3d - %s\n", ((Long) row.getKey()).intValue(), row.getValue());
        }

This code displays:

  0 - b4956242da33280c47337b61254c7d68
  1 - b4956242da33280c47337b61254c839a
  2 - b4956242da33280c47337b61254c8c88
  3 - b4956242da33280c47337b61254c8eed
  4 - b4956242da33280c47337b61254c9e1b
  5 - b4956242da33280c47337b61254ca6c5
  6 - b4956242da33280c47337b61254cb10e
  7 - b4956242da33280c47337b61254cb14e
  8 - b4956242da33280c47337b61254cc069
  9 - b4956242da33280c47337b61254cce23

NOTE: We have being able to cast the key of the document to a Long since we know from our map function that that’s the case.