User Profile Bulk Import¶

The User Profile Bulk Import is capable of importing large amounts of CSV formatted user profiles exported by your database into Repro.

Uploaded CVS data will be processed in request order. After the data has beenimported you will be notified regarding the user profiles in details via an email.Processing time of imports depend on the uploaded file size and number of user profiles.

Get Started¶

CSV files¶

The User Profile Bulk Import uses a CSV file to update the user profile.

File format¶

user_id,birthday,name,point
123456,2000-01-01T00:00:00+09:00,foo,123
234567,2001-02-01T00:00:00+09:00,bar,234

Header and Column¶

The header value of the first row will be registered as the User Profile key.

user_id: Specify the User ID that are set with the User ID API from your app. This field is required.
REPRO_ERRORS column is used to show error messages which will be covered later in this document. User Profile with the name REPRO_ERRORS will not be created and the value will be ignored.
The header value will be regarded as the key and values in the following rows will be the value of the incoming User Profile with the rest of the columns.
Empty string cannot be used for the header as the key, and the maximum number of characters is 255.
Characters will be treated as encoded with UTF-8.
Empty characters before or after keys and values will not be automatically truncated and be used as is.
The value of an unspecified key is not updated.
If the value is empty, the User Profile of the specified User ID will not be updated.

Type¶

While Repro's User Profile have four different data types, CSV doesn't allow having data types, so the following rule is used when determining the data type of each column.

For User Profile key names already registered on Repro, the original data type will be respected.
For new User Profile Keys, the data type will be determined from the values after reading certain number of columns.
1. The key will be considered as datetime if the given values follow the iso8601 format.
2. The key will be considered as decimal if the given values only contain number with floating point.
3. The key will be considered as int if the given values only contain numbers.
4. Everything else will be considered as string.
If there's any row that does not follow the inferred type rule, there will be an CSV file created which contains error information as an additional column, and an email with a link to download the error file will be sent.

Upload via dashboard¶

You can upload a CSV file containing user profiles from the dashboard.

Warning

You can upload CSV files up to 150MB via the dashboard.

Upload a CSV file containing user profiles from the dashboard¶

Access the USER PROFILE > DATA IMPORT page.
Click the "Upload CSV File" button and select the CSV file to upload.

S3 upload¶

You can register user profiles by uploading CSV files to the AWS S3 bucket provided by Repro.

Note

Before using this feature, you need to obtain authentication credentials for uploading objects to the AWS S3 bucket provided by Repro. For more details, please refer to the section 'Create an access key for uploading to the AWS S3 bucket'.

You can find the S3 bucket and path information in USER PROFILE > DATA IMPORT .

Warning

CSV file names must not contain forward slashes (/) or whitespace characters.
The uploaded CSV file must be 1GB or smaller in size.

For example, when using the AWS CLI, you can upload a CSV file containing user profiles to S3 with the following command.

$ export AWS_ACCESS_KEY_ID=AKIAXXXXXXX
$ export AWS_SECRET_ACCESS_KEY=XXXXXX
$ aws s3 cp user_profiles.csv s3://repro-data-store/XXXXXXX/user_profile_bulk_import/user_profiles.csv

The S3 path to upload data varies depending on the project you are using.Please use the path specified in the "Amazon S3 Path" section at the bottom of the USER PROFILE > DATA IMPORT page.

Create an access key for uploading to the AWS S3 bucket¶

Switch Credentials Tab at SETTINGS > PRODUCT SETTINGS.
Click the "ADD AMAZON S3 ACCESS KEY" button.

When you click the button, a popup will appear. Please store the displayed 'Access Key ID' and 'Secret Access Key' securely. (The Secret Access Key cannot be displayed again after the pop-up window is closed.)

User Profile Bulk Import API v3¶

Differences from User Profile API¶

While User Profile API updates a single user's User Profile with one request, the User Profile Bulk Import API updates multiple users' User Profile in bulk with the given CSV file from the request.User Profile Bulk Import API is suitable when you wish to update many users' data at a time while not having the need the update to be instantaneous, typically when updating pre-existing user's data.

Note

Standard User Profile cannot be updated via the User Profile Bulk Import API.

Get the Repro API token¶

To use the User Profile Bulk Import API, you need to set the Repro API token

You can find the Repro API Token at SETTINGS > PRODUCT SETTINGS > Credentials

Warning

"REGISTER API TOKEN" makes the current API token unavailable. Once you do this operation, You never use the key again and revert to the previous state.

Access limit per unit time¶

Access is limited per time for User Profile Bulk Import API.

・The limit is set to 10 accesses for every 10 minutes per API token.
When the request is completed, there will be a notification at the X-RateLimit-Limit header showing the maximum number of requests per time.
When the request count exceeds the maximum access number, the following response HTTP Status 429 (Too Many Requests) will appear.

For more information on response headers, please see Response.

Warning

The maximum access number may fluctuate without warning.
The access limit setting value is per API token per function.

For example, the push API and the user profile API use the same API token, but they are different functions, so even if they are requested at the same time, each is counted as one request.

The User Profile Bulk Import API v3 requires you to send two requests to different endpoints to update the user profile.

Execute POST /v3/user_profiles/bulk_import and obtain pre-signed URL which is required to direct upload to S3.
Execute PUT <pre-signed URL> and upload a file to S3.

Note

The pre-signed URL is valid for 5 minutes. Therefore, please initiate a second request within 5 minutes of receiving the first response.

Sample code¶

A sample code is shown below.

#!/bin/bash

set -e

API_TOKEN="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

URI="https://api.reproio.com/v3/user_profiles/bulk_import"

file=$(realpath -e $1)
checksum=$(openssl dgst -md5 -binary ${file} | base64)
content_size=$(stat -c %s ${file})

body=$(cat <<EOF
{
  "checksum": "${checksum}",
  "byte_size": "${content_size}"
}
EOF
)

# First request
response=$(curl -X POST \
                -H "X-Repro-Token: ${API_TOKEN}" \
                -H "Content-Type: application/json" \
                --data "${body}" \
                ${URI})

# Get the value needed for the second request from the response
url=$(echo ${response} | jq -r .direct_upload.url)
content_type=$(echo ${response} | jq -r '.direct_upload.headers["Content-Type"]')
content_md5=$(echo ${response} | jq -r '.direct_upload.headers["Content-MD5"]')

echo $response

# Second request
curl -X PUT \
    -H "Content-Type: ${content_type}" \
    -H "Content-MD5: ${content_md5}" \
    --upload-file ${file} \
    "${url}"

echo done

First request¶

Endpoint¶

HTTP Method	Endpoint
`POST`	`https://api.reproio.com/v2/user_profiles/bulk_import`

Request header¶

Header	Description	Optional / Required
Content-Type	Set application/json	Required
X-Repro-Token	Set the token retrieved with Get the API endpoint and Repro API token	Required

Request parameter¶

Field	Description	Optional / Required
checksum	The base64 encoded md5 checksum of the file used for import. This can be calculated with `openssl dgst -md5 -binary hoge.csv \| base64`.	Required
content_type	Specify either `text/csv` or `application/gzip` . If you omit it, it becomes `text/csv` .	Optional
byte_size	File size of the file used for importing	Required

Payload Example¶

To upload a CSV file

{
  "checksum": "J1h240z2CdsRjz2Et5mnkA==",
  "byte_size": "123456"
}

To upload a gzip compressed file

{
  "checksum": "J1h240z2CdsRjz2Et5mnkA==",
  "content_type": "application/gzip",
  "byte_size": "12345"
}

Response header¶

The response header contains the following values.

Status	Header	Description
Success	`X-RateLimit-Limit`	Access limit per unit time
Success	`X-RateLimit-Remaining`	The number of requests remaining in the current rate limit window.
Success	`X-RateLimit-Reset`	The time at which the current rate limit window resets in UTC epoch seconds. (Unix UTC timestamp)
Request limited	`Retry-After`	The time how long the user agent should wait before making a follow-up request.

The response of succeeded¶

The response body contains belows as a response in JSON format.

Field	Description
response.direct_upload	An object containing the information needed for the second request
response.direct_upload.url	Endpoint URL for the second request
response.direct_upload.headers	Header information which is required to the second request.

Response body example of succeeded¶

{
  "direct_upload": {
    "url": "https://repro-direct-upload-production.s3.ap-northeast-1.amazonaws.com/6pWMZB1797H4bNVqTwFDcPzX?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAITZGHBHURRYGMYHA%2F20200618%2Fap-northeast-1%2Fs3%2Faws4_request&X-Amz-Date=20200618T011630Z&X-Amz-Expires=300&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost&X-Amz-Signature=2902f97f060894f900141b4dde2ca2bfae9de2bd8ff3c6e3f098eab6faf674d6",
    "headers": {
      "Content-Type": "text/csv",
      "Content-MD5": "QYLqokRq51xGBqakDLsBvA=="
    }
  }
}

Error response¶

The list below contains all status codes and error messages of the responses.

400 Bad Request¶

You have an error in your request. Please check the error message.

The checksum contains an invalid string.

Response body

{
  "error": {
    "messages": [
      "Validation failed: Checksum is invalid"
    ]
  }
}

The byte_size is not specified.

Response body

{
  "error": {
    "messages": [
      "param is missing or the value is empty: byte_size"
    ]
  }
}

401 Unauthorized¶

You don't have Repro API Token set.

Response body

{
  "status": "unauthorized",
  "error": {
    "messages": [
      "Please include your Repro API Token as \"X-Repro-Token\" HTTP header."
    ]
  }
}

403 Forbidden¶

The specified Repro API Token is invalid.

Response body

{
  "status": "forbidden",
  "error": {
    "messages": [
      "Please check your Repro API Token as \"X-Repro-Token\" HTTP header."
    ]
  }
}

404 Not Found¶

The endpoint does not exist.

Response body

{
  "status": "not_found",
  "error": {
    "messages": [
      "Not found."
    ]
  }
}

429 Too Many Requests¶

Too many requests. Please wait the seconds indicated in the Retry-After header and make a follow-up request.

Response body

{
  "status": "too_many_requests",
  "error": {
    "messages": [
      "Too many requests hit the API too quickly."
    ]
  }
}

Second request¶

Endpoint¶

HTTP Method	Endpoint
`PUT`	The value of `response.direct_upload.url` included in the response when the first request was succeeded.

Request header¶

Please specify all of the value of response.direct_upload.headers included in the response as a header if the first request was succeeded.

Request parameter¶

Please set up a CSV file to update your user profile. The maximum file size is as follows.

csv: 1 GB
gzip: 100 MB

Please refer to CSV files and check the CSV format.

The response of succeeded¶

If the request is successful, a status code of 200 is returned.

Error response¶

Please refer to this document for the details on error responses.

Import notification¶

Usage

You will be notified via an email with details of the User Profile update once the processing completes. There will be a link to download the error file if there was anything wrong when performing the update.

Error CSV file¶

Usage

CSV files will be validated once the processing finishes. Rows that contain invalid data will have an extra column and error information as REPRO_ERRORS which can be downloaded from Import notification link. If the file size used in the request is too large or the file is empty, the REPRO_ERRORS column may not be present in the CSV containing the error information. If the REPRO_ERRORS column is present, values in the it will all be ignored, so you can modify the CSV with error information and re-send as another API request.

Sample error CSV file

REPRO_ERRORS,user_id,birthday,name,point
birthday should be iso8601 format,123456,2000-01-01 00:00:00.000,foo,123
point should be integer,234567,2001-02-01T00:00:00+09:00,bar,234.0

Error details¶

messages	Description
too many values	Too many columns of values for header
user_id is unregistered user	We could not identify a user with the given user_id.
[column name] should be UTF-8	There is a UTF-8 incompatible data.
[column name] should be integer	There is a non-integer value.
[column name] should be decimal	There is a non-decimal value.
[column name] should be iso8601 format	There is a value that does not follow the iso8601 format.

Note

When there was no error the download link will not be inserted.

(Deprecated)User Profile Bulk Import API v2¶

Warning

This API v2 is scheduled to be deprecated on August 31, 2021. If you wish to use the API, please use User Profile Bulk Import API v3.

API format¶

Sample request

curl -X POST \
--verbose \
-H 'Content-Type: text/csv' \
-H 'X-Repro-Token:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' \
--data-binary '@sample.csv' \
'https://api.reproio.com/v2/user_profiles/bulk_import'

Request¶

Request header

The response header contains the following values.

Header	Description	Required
Content-Type	Setting `Content-Type: text/csv` is required when performing a POST request with CSV files, whereas `Content-Type: application/gzip` is required with gzipeed CSV files.	Required
X-Repro-Token	Set the token retrieved with Get the Repro API token	Required

Request body

Set the CSV or the gzipped CSV to HTTP body. For more details with CSV format, see CSV files

Response¶

Response header

The response header contains the following values.

Status	Header	Description
Success	`X-RateLimit-Limit`	Access limit per unit time
Success	`X-RateLimit-Remaining`	The number of requests remaining in the current rate limit window.
Success	`X-RateLimit-Reset`	The time at which the current rate limit window resets in UTC epoch seconds. (Unix UTC timestamp)
Request limited	`Retry-After`	The time how long the user agent should wait before making a follow-up request.

Response body

The list below contains ((all)) status codes and error messages of User Profile Bulk Import API responses.

202 Accepted¶

The request was successful. The result of type guesses of the column of CSV posted will be returned as the response body.

Response body

{
  "user_profile": {
    "birthday": {
      "type": "datetime"
    },
    "name": {
      "type": "string"
    },
    "point": {
      "type": "int"
    }
  }
}

400 Bad Request¶

Response body

{
  "error": {
    "code": 2001,
    "messages": [
      "user_id column is required"
    ]
  }
}

CSV format is invalid. Please refer to CSV files and check the CSV format.

messages	Description
empty file	The file is empty. Please check the file you tried to use.
header size over 102400 bytes	Header size of the CSV file is over 100KB. Please make it below 100KB.
newline character not found	A new line is missing.
empty columns	The column is empty.
user_id column is required	The header of the CSV is missing user_id column.
[key name] should be UTF-8	The CSV file contains UTF-8 incompatible character data. Please encode the CSV file in UTF-8.
duplicate columns [key names]	Can not accept duplicate columns.

Response body

{
  "error": {
    "messages": [
      "not in gzip format"
    ]
  }
}

The requested file is not in gzip format.

Response body

{
  "error": {
    "messages": [
      "file maybe corrupt"
    ]
  }
}

The gzip filed that has been POSTed may be corrupt.

Response body

{
  "error": {
    "messages": [
      "This app has exceeded the limit of enabled user profile(x)"
    ]
  }
}

Number of the User Profiles exceeds the maximum

401 Unauthorized¶

You don't have Repro API Token set.

Response body

{
  "status": "unauthorized",
  "error": {
    "messages": [
      "Please include your Repro API Token as \"X-Repro-Token\" HTTP header."
    ]
  }
}

403 Forbidden¶

The specified Repro API Token is invalid.

Response body

{
  "status": "forbidden",
  "error": {
    "messages": [
      "Please check your Repro API Token as \"X-Repro-Token\" HTTP header."
    ]
  }
}

404 Not Found¶

The endpoint does not exist.

Response body

{
  "status": "not_found",
  "error": {
    "messages": [
      "Not found."
    ]
  }
}

413 Payload Too Large¶

The requested CSV file is too large.

Response body

{
  "error": {
    "messages": [
      "client intended to send too large body"
    ]
  }
}

415 Unsupported Media Type¶

The CSV file contains unacceptable data format. Please check if the Content-Type header as well as the request endpoint.

Response body

{
  "error": "415 Unsupported Media Type"
}

429 Too Many Requests¶

Too many requests. Please wait the seconds indicated in the Retry-After header and make a follow-up request.

Response body

{
  "status": "too_many_requests",
  "error": {
    "messages": [
      "Too many requests hit the API too quickly."
    ]
  }
}

Import notification¶

please see here.