Fetching Data

Introduction

The initial release of the Kiva API is concerned with helping you find and present public data from Kiva. We want to help you find things like loans, lenders, and partners and help you represent those in your applications and products. We think there are a lot of really useful things that can be done just by helping the Kiva community find information in different ways or experience it in a new context. Moreover, understanding our core set of data is the most important building block to doing more complex operations in future versions of the API.

In talking about data, we'll start by defining some common terms used by the API and how that relates to the language on the Kiva website. Then, we'll explain some of the data types in more detail and write about some practical approaches on putting the different types of data calls together.

(PS, if you're just interested in working with RSS, you might want to skip ahead to Feeds).

Objects in the Kiva World

Loans, lenders, entrepreneurs, businesses… all of these terms can get a bit overwhelming without a proper introduction. When we created the Kiva API we wanted to choose terminology that was objective and easy to understand from a developer (and sometimes, finance) perspective. This means that the language on the Kiva website may not always match up with the terminology in the API – the Kiva website is a dynamic, evolving user experience but the API should be a consistent tool on which your own dynamic applications can depend.

Loan

A loan is the most important data object at Kiva. Most other objects are in some way related to a loan. It makes sense; facilitating loans that change lives is at the heart of Kiva's mission. You've probably seen loans referred to as “businesses” or “entrepreneurs” in other places because of the tight relationship between all these concepts. However, in the API, if you want to show a listing of any of these things in your application you should think about fetching and rendering loan objects. For example, from an API perspective, the Lend tab at Kiva.org shows a listing of loans, and the front page of Kiva features a entrepreneur by fetching a loan that is currently fundraising.

Borrower

A borrower is someone who has requested a loan. We often refer to these individuals as “entrepreneurs” at Kiva because we believe strongly in the entrepreneurial spirit of our borrowers working to make a difference in their lives. A loan may have more than one borrower, and in this case the loan is considered a “group loan.”

Lender

A lender is a user registered on the Kiva website for the purposes of lending money and participating in the community. Most lenders have public profiles, or Lender Pages, on Kiva where they can share a bit about what they do and why they lend. Not all lenders have made loans, and not every lender has opted to display public information. Lenders without public information are referred to as "anonymous" lenders.

Partner

A partner, or Kiva Field Partner, is a microfinance institution with whom we work to find and fund loans. Every loan at Kiva is offered by a partner to a borrower, and the partner works with Kiva to get funding for that loan from lenders. The association of a loan to a partner is very important since the risk associated with a loan correlates closely to the reputation of a partner. This is why every partner has a risk rating.

Journals and Journal Entries

Every loan has a journal where Kiva or a partner can provide updates about a loan. Each of these updates is called a “journal entry.” Some journal entries have photos or videos associated with them and provide personal updates about a borrower. Others are more informational or automated updates and are classified as “bulk” journal entries. Lenders can post comments on journal entries.

Team

Lenders can coordinate their activity on Kiva through Lending Teams, found on the Kiva community tab. Teams are usually formed around a common interest (sometimes related to lending, sometimes not) or may represent real-world groups outside of Kiva such as a church or a classroom. Currently, most team activity is centered around making loans as every lender can credit a loan to one team every time a loan is made. As such, teams are associated with both teams and lenders. Teams also have profile information, similar to lenders, where they post a team photo and why the team has come together to loan.

Simple Data Types

In addition to our specialized data objects, there are some very simple data types that are helpful to understand when working with the API.

IDs

Most data objects have unique IDs so you can track them and refer to them later. For example, every loan has a loan ID. This is an obvious yet powerful part of the API. When the field is simply called "id" we're usually referring to an integer used to refer to an object like a loan or an image. Other types of IDs are unique strings that you can also use to refer to a data object; examples of these include a lender_id (for referencing a lender) and youtube_id (which references a video on YouTube).

Dates

All dates in the Kiva API are represented as timestamps according to ISO 8601. Here's an example of what a date looks like in this format:

2009-01-09T09:50:08Z

The dates are usually in UTC as represented by the 'Z' time zone designator at the end of the string. Most programming languages have open source libraries or built-in support for handling the ISO 8601 format.

Images and Other Media

Things like images are too complex to represent in a simple data response. Instead, media is stored separately and we give you information on how to access it or integrate it into your application. Images, for example, are represented by an ID and a template URL. We talk more about how to access images in a following section.

Country and Language Codes

Countries and languages are represented by codes in the API rather than using their full English name. This is true for API responses as well as parameters. Countries are coded according to ISO 3166-1-alpha-2. Languages may be specified as two or three characters according to the IANA Language Subtag Registry and RFC 4646.

Location

We think location is a real cool property of the objects our data represents. Many of our data responses have location information built into them, including geospatial information. A location object typically has conventional information like the name of a town or country related to its parent (i.e., a Loan). In most cases, we also provide a geo element which is based on the GeoRSS Model for geospatial information.

As you can see, our geo element has 3 components; they are defined as such:

level

Reflects the level of accuracy available for the supplied geometry. Popular values might be town, country, or exact. Use this information to help you decide how to represent the data, or if you should do your own geocoding for the object.

type

The type of geometry defined by the coordinate pairs provided. This can be any of the shapes supported by the GeoRSS Model — point, line, box, or polygon.

pairs

The coordinate pairs for the geometry. This value is formatted according to the GeoRSS-Simple specification for serializing coordinates. For example, in the sample above a point is represented by latitude, followed by longitude, and separated by a space.

Whearabouts

In comparison to location, whereabouts are some sort of vague notion we have about the location or origin of the related data. Currently, lender data has a “whereabouts” field to convey the information a lender has supplied about where he is from. Since we can't vouch for the accuracy or meaning of this data, and we don't have a good way of resolving it to some place on the globe, we just call it “whereabouts.” We figured you'd get the point.

Loans

When you fetch loans, they are returned in one of two ways – a loan listing or a loan detail view. Loan listings are returned from API calls where the results can be numerous and span many pages of data. If you fetch the loans associated with a lender, you'll get loan listings. Each listing has enough information for showing a summary of the key details on the loan, such as the name, the photo, the loan amount, and the planned use of the loan. The simplest way to get a listing of loans is call the loans/newest method without any parameters:

This returns listings for up to 20 of the newest loans at Kiva that are currently raising funds. Here's what one loan listing looks like in JSON:

Here we can see that Le Thi Tan of Quang Xuong, Viet Nam is requesting an individual loan to buy more ducks and chickens for a poultry business. We can also show that $125 of the $550 needed has already been funded. The loan description is available in two languages, Vietnamese and English, and the loan was posted on January 9, 2008 at 1:50am PST.

A couple of details are not obvious from this view, namely, the image and the details on the field partner. Requesting Kiva images is a relatively simple process that we'll cover in a later section. Requesting partner information is also pretty simple and is something we expect most applications to do infrequently but regularly.

Field Partner Information

Though there are tens of thousands of loans on Kiva, there are currently less than hundred active partners. For this reason, we've designed the API so you can fetch the list of partners and cache it for reuse over lots of different calls. This way, we don't have to send you redundant information about partners every time you call the API. For now, the list of partners is available in a single page of data at:

Each partner is indexed with a partner ID which is easy to match up with the partner_id field from a loan listing. For instance, the loan from Viet Nam in the earlier example has the partner ID 67 which matches up with the partner, TYM Fund. The partner started posted loans at Kiva in June 2007 and, at the time of writing, has a 4 star rating. When rendering the loan it would make sense to show some of this information about the partner as well.

Flipping things on their head, you can also start with a partner ID (presumably fetched from the partner API method) and then fetch the list of all loans offered by the partner. For this we use the loans/search method.

This call returns a list of all loans from one of our most active partners, Microfinanzas PRISMA. This call begins to show capability of one of the most useful and powerful methods in the API, loan search. For instance, let's say we wanted to get a listing of all the loans from partners in Uganda which are currently fundraising. To do this, we would again use the partners method in conjunction with the loans/search call. First, we look through the data returned by the partner call to find all partners operating in Uganda. Then we plug those in as an array of IDs along with a value for the status parameter to the loan search:

Loan Status

The status of a loan is an especially important concept to understand when working with loans. Here is the list of loan statuses and what each means:

fundraising

The loan has not yet been funded. This typically represents the kind of loans we advertise on the front page of Kiva.org and on the Lend tab. Lenders can only lend to loans that are fundraising. You can find the amount funded so far by checking funded_amount.

funded

This loan request has been completely funded and is not available for new loans by lenders. The loan may be waiting for disbursal to the borrower(s), assuming the loan was posted to Kiva before the field partner disbursed the funds. The field funded_date shows the exact time at which the loan was fully funded on Kiva.

in_repayment

The loan has been disbursed to the borrowers and they are in the process of using the funds and making payments on the loan to the field partner. Loans in this state may see journal updates and lenders to this loan will get repayments when the borrower's payments are reconciled with Kiva.

paid

The loan has been paid back in full by the borrower. The payments have been distributed back to the lenders and the loan is closed to most new activity on Kiva.

defaulted

Occassionaly, a borrower or a field partner may fail to make payments on a loan, either to the field partner or to Kiva, respectively. Usually when this happens, a loan simply becomes delinquent and remains in the in_repayment status. When a loan remains delinquent 6 months after the end of the loan payment schedule, the loan becomes defaulted. Usually defaulted loans will never be paid back and are a financial loss to the lenders to that loan. Most loans only default in part, but it is possible for the entire amount of the loan to not be repaid.

refunded

It's rare, but occasionally Kiva needs to refund the funded portion of the loan to lenders after the loan has been partially funded, fully funded, or even during repayment. There are many reasons why a loan could be refunded, but usually it is because there is an error with the loan posting or the loan itself has been found to violate Kiva's policy for loans. Currently, you cannot search for loans based on this status.

You may have noticed that the terms used in the Kiva API for loan status do not always match up with the terminology used on the site. We feel the terms used in the API are clear and acceptable for communicating to lenders, though you are free to choose the terminology we currently use on the website as well.

Loan Detail

As we've seen so far, there are lots of ways to fetch listings of loans. Loan details, on the other hand, are only available by explicit request once you have the ID for a loan but they include all the specifics about a loan that you want to represent in your application. This includes the loan description, the loan schedule (in both US dollars and the local currency), any payments made to the loan so far, the list of borrowers on the loan, and the number and type of journal entries the loan has.

The best way to get familiar with this view is to make a request in JSON or in XML and take a look at the data yourself. However, we do want to go over some of the detailed terminology associated with a loan. It can get a bit complex even if you happen to have a background in finance.

disbursal_amount

The amount of money distributed to the borrower(s) in the local currency. Comparing this amount to the loan amount shows the currency conversion rate locked in for the loan when it was posted.

disbursal_currency

The ISO 4217 code for the currency used to distribute the loan the the borrower. This is usually the local currency for the borrower's country.

disbursal_date

The date at which the funds from the loan were given to the borrowers. Note that it is possible for the money to be disbursed to borrowers before the loan is posted on Kiva.

loss_liability

A set of values which describes who is liable for loss on a loan, and how. For nonpayment, the party liable can either be the lender or partner. For currency_exchange the liability can be shared, fully resting on the partner, or none if the currency is locked to the US dollar. If currency exchange loss is a shared liability, the currency_exchange_coverage_rate will also be listed. (For more information on these topics, search for "default protection" or "currency exchange" at our Help Center.)

currency_exchange_loss_amount

If a currency exchange loss is shared between a partner and the lender, then this value represents the amount in USD lost by the lender due to fluctuations in the value of the local currency against the US dollar. This will result in the paid amount of loan being less than the full loan amount even when the status of the loan is listed as paid.

local_payments

A list of the payments the borrower will make to the field partner, also commonly called the “loan schedule” from the perspective of the borrower. Each payment has an amount (in local currency) and a due date (the approximate day and time the payment is due to the field partner).

scheduled_payments

A list of the payments (in US dollars) due to lenders and the dates when lenders can expect a repayment if the loan is not delinquent. This is considered the loan schedule from the lenders' point of view. Since both Kiva and the field partner work to facilitate the loan between the lender and the borrowers there is more than one way to represent the loan schedule.

payments

The list of payments that have been made by the borrower and that have been reconciled with Kiva's accounting system. A payment usually has an amount (in USD), a local amount (in the disbursal currency), a processed date (the approximate date at which the borrower paid the partner), and a settlement date (the date at which funds for payment from the partner were reconciled by Kiva). Payments might also have comments.

journal_totals

The values here show the number of total journal entries for the loan as well as the number which are automated or “bulk” entries. Bulk entries are typically much less interesting than other journal entries. Checking these counts can help you determine if it is worthwhile for your application to fetch the journals for a loan.

Also, there are a couple items which only apply to loans in fundraising status:

funded_amount

This is the amount of the loan which has been purchased by Kiva lenders. When this amount becomes equal to the loan_amount the loan moves to a funded state (soon followed by in_repayment).

basket_amount

This is the amount of the loan which lenders have saved in shopping baskets, but has not been confirmed as purchased. It is possible that in the near future a portion or all of this amount could become available to other lenders, but until that time this amount of the loan is reserved. In the case where funded_amount + basket_amount >= loan_amount you may consider the loan as unavailable through it is still in the state of raising funds. (NOTE: The Kiva website currently renders the "amount raised" for a loan as funded_amount + basket_amount and filters out loans which have no amount currently available for purchase. Choose the way you best see fit to handle this in your own application.)

Linking Loans back to Kiva.org

Since the Kiva API currently does not let you facilitate loan transactions, it is likely that you will want to link a loan from your application to a page at Kiva.org where a user can lend to an entrepreneur. For now, we suggest linking loans to the entrepreneur's page using this URL pattern:

where <id> is the ID of the loan you want to link.

Lenders

Like loans, lenders have two forms of representation, that suitable for listing and a more detailed view. Lender listings are fairly simple and will include a name, a lender ID, and possibly locational information. Here's an example of the lender listing for one of Kiva's first engineers, Jeremy:

This is enough information to show a picture of Jeremy with his name as well as where he is from.

As an example of lender listings, you can request a list of lenders associated with a particular loan:

Since there may be many lenders to a loan it is possible you have to span a few pages of data to fetch all a loan's lenders. However, by giving you up to 50 lenders per page of data we hope this case is rare.

Lender IDs and Lender Pages

You may have noticed that lenders are not referenced by numerical IDs but by alphanumeric identifiers. By default, lender IDs are automatically issued when a lender signs up for Kiva and creates a public profile. However, a lender can change his ID (as well as most other details as well) so you should never expect that this identifier will always stay the same for a lender, much less, valid as an identifier on Kiva. However, it is true that most lenders rarely change their lender ID since changing this also modifies the URL for their lender page.

If you want to provide a link in your application to a lender's Lender Page on the Kiva website, all you need is the lender_id and this URL template:

So, as an example, Jeremy's lender page is available at:

http://www.kiva.org/lender/jeremy

What if a lender doesn't know his Lender ID? We've added a handy page to the Kiva website to help lenders figure this out. Direct users this URL and they will be shown their lender ID after successful login to Kiva:

Lender Detail

The best way to get familar with the lender detail view is to make a request like this one and inspect the results:

Loans for a Lender

It's probably pretty obvious by now but fetching the list of loans is not much different that any other loan-listing API method. To see the 20 most recent loans Matt, Kiva's CEO, has made, you would use this call:

As you would expect, the format of the results looks like that of loans/newest or loans/search.

Media

Images and other media at Kiva are composed of an immense amount of data as compared with other types of data in the Kiva API. Since it's not reasonable to embed this data our API responses, and since most people want to embed images in applications and web pages anyway, we've come up with a simple template system for referencing images.

Images

An image in the Kiva API is represented by an id and a template_id. For example, here's a sample image reference for a borrower picture:

The specific template you can use for an image can depend on the photo. This photo uses the image template with ID 1 which looks like this:

The ID part of the template makes sense, but what about the size? For any image at Kiva, you can choose from any of the following values for size:

• w80h80
• w200h200
• w325h250
• w450h360
• s300
• fullsize

For each of the values that specify both height and width, you will get an image whose maximum dimension is scaled to the value supplied. For an image that is 800 pixels wide and 600 pixels high, using w80h80 will provide an image sized to 80 pixels wide and 60 pixels high. fullsizereturns the largest version of the image available. For the "s" (ex: "s300") pattern, you will get a squared image (in this example: 300px x 300px). Squared images are handy for various layout types, such as Kiva's home page. A word of caution: squared images can sometimes crop people out of the photo; for wide photos, the cropping routine cuts off the left and right sides evenly. This can sometimes crop individuals out if they are on the sides which is especially common in group photos. If the image is tall, the cropping routine cuts the bottom of the photo off since most images of borrowers have their heads in the top part of the photos. Squared images are good for certain layout types, but when showing a "zoomed in" version, it's best to use the other sizing options to prevent cropping people out.

Knowing the template pattern, we can put the values together and create an image URL from our template:

Image URL Templates

A good practice would be to cache the output from this call in your application and only refresh it when you see a template ID that isn't in your cached version. (We promise we'll never change the value of a template for given ID.) You could take things a step further and include a cached version as a part of your application's data resources. Since we currently only use one template ID for all the images at Kiva, it's possible that your app would never refresh the template list for images! However, a robust application would be prepared for the introduction of a new template at any time.

Video

Kiva supports video only in loan profiles and journal entries at the moment. You won't find a video for every loan or journal entry, but when you do it will look like this:

Videos don't have templates. Instead we refer you to the YouTube API for information on how to embed, present, or otherwise handle video in your application. Currently all video at Kiva is hosted by YouTube.

A thumbnail image for every video is available. Currently the thumbnail image is the same as the primary image for the loan or journal entry.

More

Lending Actions

Lending actions are cool bits of data that we've offered on the front page of Kiva for a while. On the website we've called this “recent activity” and you might have seen us call these “transactions” as well. Basically, a lending action is the singular action of a lender making a loan to an entrepreneur on Kiva. It's comprised of an ID, a loan listing, a lender listing, and a date at which the lending action was completed.

Currently, only the 50 most recent lending actions are available via the API through this call:

A lender might make any loan amount or even a few loans to the same entrepreneur but it will show up as one lending action. Also, lending actions made more than 48 hours ago or those by anonymous lenders are not shown.

Journal Entries and Comments

A loan can have any number of journal entries. These are updates about the status of the loan, the borrower, or sometimes, the field partner. Also, visitors to the Kiva site (not necessarily lenders) can make comments to a journal entry. Journal entries are typically accessed by passing in the ID for a specific loan:

The methods for fetching journal entries take an optional parameter, include_bulk. By default this is set to 0 since we assume that in mose cases you're not interested in boring automated notifications (often these are triggered by payments made to a loan – information you can fetch from a loan's detail). However, to get bulk entries as well as the meatier entries, just set this parameter to 1.

Each journal entry listing will have a comment_count field showing the number of comments posted to the entry. Since most entries don't have comments, you can use this to only fetch comments for entries with comments or those that appear to have especially lively discussion around them. The call to fetch comments for a journal entry only requires a journal entry ID (which is guaranteed to be unique for every journal, regardless of loan). Such a call looks like this:

Teams

A lending team is comprised of a number of lenders, and at least one lender (the team captain). Lenders come together in a team to pool their lending toward a common cause or interest, and as such, teams are also related to loans. Teams are identified by a numerical ID, but can also have an customizable alias, called a shortname, which may also be used for identification. Typically you get information about a team using the numeric team ID:

You can get the list of lenders in a team by passing in the same ID to this call:

and the loans atrributed to the team are fetched in similar way:

If a user happens to know the shortname of their team, it's also possible to do a lookup of team information using that alias instead of the team ID:

Team API methods are a work in progress - we should have more soon that allow reverse lookup (such as finding out which teams a lender belongs to...)

Data Snapshots

Important Note: Our database has ourgrown our snapshot generation script. Snapshots are still available, but extremely out-of-date. We'll update this page once they're working again.

If you want to work with a lot of Kiva's data, making hundreds or thousands of requests to the Kiva API can be overbearing — both for you and your network. As such, we make much of our data available through snapshots which are compressed into a simple singular download. The data is archived nightly so it is most useful for apps that don't require live data, such as data analyses and visuals. However, some applications might find these handy as a way to seed local data sources, supplementing the snapshots with calls to the Kiva API for the most recent data.

A data snapshot is composed of multiple files, delivered in a single compressed ZIP archive. Data snapshots are available in XML and JSON, just like responses in the Kiva API. For the most part, the format of the documents in the snapshots are the same as an API response, with a few exceptions.

Downloading Snapshots

The latest data snapshots are available in the format of your choice at the following URLs:

JSON: http://s3.kiva.org/snapshots/kiva_ds_json.zip

XML: http://s3.kiva.org/snapshots/kiva_ds_xml.zip

In the future, we'll have RSS feeds and API methods available for confirming the date and location of all snapshots published.

Archive Structure

When you decompress and extract a data snapshot you'll have a collection of documents with the following structure:

At the root level of the snapshot is a collection of directories, one each for the type of data archived. For now, we simply capture detailed data about a singular object such as a loan or lender. In the future, there might be data collections for relational data, such as lenders_to_loan. Each of these data collections is a series of numbered files in the chosen format, JSON or XML. Together all of these files make up the complete data collection as if we captured the data in a single file (e.g., lenders.json). The complete data set is broken up into multiple files so it is easy to process the data with conventional parsers.

The number of each file in the collection corresponds to the page of data in the series, much like pages of data accessed through the Kiva API. So, lenders/2.json would be similar to http://api.kivaws.org/v1/lenders/search.json?page=2. The only difference is that the page size used in data snapshots is much larger than that in the Kiva API.

File Format

As mentioned, the format of data in each snapshot document is similar to that of a response in the Kiva API. The main difference is the way the data segment is wrapped and how meta-data is expressed for each segment of the snapshot. Here's an example of the beginning of a sample document, lenders/2.json:

The structure begins with a header element which contains meta-data for this snapshot segment. We see the total number of elements in the data collection, 444938, as well as the date of the snapshot (which should be identical for all documents in the snapshot). We read from the header that size of each data page is 1000 elements (in this case, lender elements) so we know there are 445 total pages (ceiling of total/page_size). The page number of the document is also reinforced in the header.

Afterwards, follows an ordered array of lender elements, sorted from oldest to newest. Each of these elements is identical to a detailed lender record in the Kiva API.

The XML version of snapshots is a direct translation of the JSON format shown above. The root node used for XML documents is <snapshot>.