Back to blog home page

Importing Large Datasets Into Backand with Bulk Processing

Posted by on Apr 06, 2017

While our SDK includes several methods to create and modify single records, many applications won’t be able to get by with just single-record edits. In this post, we’ll take a look at Backand’s Bulk API. Using our bulk functionality, you can perform up to 1,000 operations with a single API request.

Instead of calling the server once for each object update, you can batch your operations and call the SDK’s “bulk.general()” method, performing only a single API call. The array of commands passed in will be processed, and the results included with the server response. This allows your code to be more efficient, while reducing the cost of operation of your application by reducing the sheer number of API calls made. In this post, we’ll look at how to authenticate your requests, how to send a request to the server, and what operations are available to perform in bulk.

Structure of a Bulk Request Entry

A bulk request entry is an array of command hashes to be executed against your app’s REST API. Each entry has a set of keys that correspond to the portions of the request that will be executed to perform the specified update. Here is the general format of a request, and a description of each element:

FIELDVALUEREQUIRED?DESCRIPTION
methodPOST/ PUT/ DELETE
yesThe HTTP verb to be used. POST creates an object, PUT updates an object, and DELETE removes an object
urlurl for the objectyesThis is the URL used to access the object being modified. It is constructed in the SDK using the Defaults and Constants properties, along with the object name:
Backand.defaults.apiUrl + Backand.constants.URLS.objects + "/" + "YOUR_OBJECT_NAME"
dataJSON hashyesThis only applies to POST and PUT actions. It represents the data to be created (for POST requests), or the updates to existing data (for PUT requests). Provided as a series of key-value pairs
parametersJSON stringnoAdditional parameters, and values for those parameters, to send with the request
headersJSON stringnoThis is used to set custom HTTP headers. You can use it to dynamically set the Authorization header for each REST call. If you do not need to set a separate header for each request, you can use a general-purpose token such as your Anonymous Access Token, or simply use the same header for each portion of the request

All requests are performed as HTTP POST requests. These requests are sent to the bulk endpoint, which is available at https://api.backand.com/1/bulk. Simply build the authentication header for the request by setting the AppName and Authorization HTTP headers per the next section, set your command details to the request’s data member, and await the response from Backand’s servers.

Bulk Operations with the SDK

In addition to sending requests to the bulk endpoint using AJAX or cURL, you can also send bulk upload requests using the Backand SDK. Using the ‘bulk’ property of the endpoint, you can easily send your compiled request data to Backand using a single function call:

This call handles setting up the headers, making the request, then providing the response via a promise. Simply register your promise resolver using .then, and your error handler using .catch, and you are ready to start using our bulk endpoint to make modifications to your application!

Constructing Bulk Requests

Performing each of the standard non-retrieval CRUD operations in bulk is as simple as specifying the appropriate actions to take with each request. We start by constructing the URL we need in order to manipulate each object. This is done with the Backand SDK using the following code:

Simply replace “YOUR_OBJECT_NAME” with the name of the object you wish to manipulate.

Once you have the object URL, you simply need to provide the appropriate command. To create a new record, for example, use the following hash as a starting point:

In this hash, we specify the HTTP verb to be used – POST, in this case, which will create a new record. We set the “url” parameter to the calculated URL from above, and then provide the new record’s data as a JSON hash in the “data” parameter. Simply provide the list of key and value pairs that represent your object’s data, and you’re set.

Once you have a single request hash, you can add others by simply appending them to the request. For example, if you wanted to update two records in the same request (using the PUT method), you would send the following:

You can also mix and match operations, varying the objects updated, the authorization headers used, even the type of request for each record to be updated. There are no restrictions on types of operations that can be performed in a single bulk request, so feel free to mix and match to suit your application’s needs. For example, the following request will create two new records (one news article, one author), update an author record, then delete a news record:

Authenticating the Request

Each request hash can authenticate as a different user within your application. Any valid token to connect to your application can be provided along with each bulk item being manipulated. You can also dynamically specify the app name for each bulk item. To do so, simply use the “headers” element of the bulk operations structure as follows:

Simply supply the following elements in the header:

  • YOUR_ACCESS_TOKEN – an access token, obtained via a call to

  • YOUR_APP_NAME – the app name to operate on

Refer to our documentation on user security for more information on how to construct the authentication headers.

Conclusion

The Bulk Operations endpoint can be used to greatly simplify working with multiple objects simultaneously. It allows you to easily perform tasks such as adding multiple records at once, deleting multiple rows from a table, updating several records across multiple tables, or any combination of the above that makes sense for your app’s use case. You can even perform the action within the context of different users by simply providing the appropriate authorization headers for each operation to be performed. With the bulk operations endpoint, you can streamline your app’s behavior to handle complex updates and modifications with ease. Simply consult our documentation to get started!

Don’t have a Backand account? You can get one free.