Exporting raw data via the API
Exports raw data to downloadable files. Data is split into roughly 256 MB files, and is not necessarily ordered. Each file contains 1 line per session, with each session JSON-encoded. Export files are automatically deleted 24 hours after export occurs. Data becomes available to export every 2 to 4 hours, and only for complete sessions. You cannot export data that has become available more than 60 days ago. You may only export data 24 times per day. Exports with invalid arguments do not count towards this limit.
To construct an api call to export raw data, you will need, at least, the following:
- APP_ID_KEY - Can be found in the keys & settings of your app.
- EXPORT_KEY - Can be found in the keys & settings of your app.
- startDate - this is the date that you would want to export data from.
And your API call will be constructed to look like the following:
https://api.leanplum.com/api?appId=APP_ID_KEY&clientKey=EXPORT_KEY&action=exportData&apiVersion=1.0.6&startDate=YYYYMMDD
The following arguments are optional:
| parameter | type | description |
| endDate | String | Last date in range to include in PDT/PST (format: YYYYmmdd). Defaults to startDate if not provided. Example: 20141118 |
| startTime | Number | First time (when data became available) to include (seconds since midnight UTC on January 1, 1970). If not provided, accepts all times before endTime, or all times if endTime is also not provided. The main use is to set this to the last time you exported data to only get the new data since your last export. |
| endTime | Number | Last time (when data became available) to include (seconds since midnight UTC on January 1, 1970). If not provided, accepts all times after startTime, or all times if startTime is also not provided. |
| callbackUrl | String | URL to POST a response to when the export completes. The response is the response format of getExportResults. |
| exportFormat | String | The format to export data. Can be either json or csv. Default: json. |
| s3BucketName | String | The name of an AWS S3 bucket to copy exported files to. |
| s3AccessId | String | The AWS Access ID used to authenticate to S3. Required if s3BucketName is set. |
| s3AccessKey | String | The AWS Secret Access Key used to authenticate to S3. Required if s3BucketName is set. |
| s3ObjectPrefix | String | An optional prefix of files to write to S3. Example: dirname/ to write files to a directory within the S3 bucket. |
| compressData | Boolean | An option to compress the data. Only works when uploading to S3. If set to true, the files will be compressed using gzip before being uploaded. |
Response
| parameter | type | description |
| jobId | String | The job ID of the pending export job, if data matching supplied arguments is available. |
To then get the files with the jobId, the call would be constructed in the following manner:
http://api.leanplum.com/api?appId=APP_ID_KEY&clientKey=EXPORT_KEY&apiVersion=1.0.6&action=getExportResults&jobId=JOBID_HERE
The getExportResults call should be repeated periodically (e.g. every minute) until the state is either FAILED or FINISHED. If the job is failed it should be retried. If the job is finished, files can be consumed from the provided URI-s.
The resulting file will give you data about a user's session such as their device, the events that they've triggered, userAttributes and etc. The file will either be in JSON or CSV format, depending on how you construct the call.
To automate this process with a Python Script, see more here.
Historical export
Historical export is, for example, an export of all sessions data for a specific historical period. This is usually an ad-hoc export. Data can be exported for up to 60 in the past.
Parameters startDate and endDate are used to specify the historical period. All currently available sessions that started between 00:00:00 PST on startDate and 23:59:59 PST on endDate will be exported.
Example request:
https://api.leanplum.com/api?action=exportData&appId=APP_ID_KEY&clientKey=EXPORT_KEY&startDate=20191101&endDate=20191131
Streaming export
Export sessions data continuously as it become available in Leanplum.
We strongly recommend using auto-export for this use case instead of relying on the exportData API.
Auto-export can be configured following this article
To implement this use case with the Leanplum API, one needs to maintain a timestamp for the last successful export. The exportData requests should be made periodically (e.g. every 2 hours or daily). Any subsequent exportData request should only be made if the last export request status is FINISHED or FAILED, otherwise some data may be exported twice.
The exportData parameters should be configured as follows:
| Parameter | Value | Example value (assuming now = 2019-05-08 10:00:00 UTC) | Notes |
|---|---|---|---|
| startDate | now -8 days in PST | 20190501 | To capture offline sessions |
| endDate | now +2 days in PST | 20190510 | |
| startTime | the endTime of the last successful export job | 1557302400 | This value should be maintained (remembered) and updated |
| endTime | now (epoch seconds) | 1557309600 | To capture scheduled sends |
Example request:
https://api.leanplum.com/api?action=exportData&appId=APP_ID_KEY&clientKey=EXPORT_KEY&startDate=20190501&endDate=20190531&startTime=1557302400&endTime=1557309600
Wait for job completion
| Parameter | Description |
|---|---|
| status | a job transitions through PENDING, RUNNING states and after that it is either FINISHED or FAILED |
| numSessions | number of exported sessions |
| numBytes | size of exported data |
| files | list of URI-s of exported files |
Example request:
https://api.leanplum.com/api?action=getExportResults&appId=app_key&clientKey=exp_key&jobId=export_4842631999258624_4250561969883062942
Repeat this call periodically (e.g. every minute) until the state is either FAILED or FINISHED. Retry the job if its status is FAILED. If the job status is FINISHED, you can consume the files from the provided URI-s.
Updated almost 3 years ago