User Profile Bulk Import API¶
The User Profile Bulk Import API is capable of importing large amounts of CSV formatted user profiles exported by your database into Repro.
Uploaded CVS data will be processed in request order. After the data has beenimported you will be notified regarding the user profiles in details via an email.Processing time of imports depend on the uploaded file size and number of user profiles.
Get Started¶
Differences from User Profile API¶
While User Profile API updates a single user's User Profile with one request, the User Profile Bulk Import API updates multiple users' User Profile in bulk with the given CSV file from the request.User Profile Bulk Import API is suitable when you wish to update many users' data at a time while not having the need the update to be instantaneous, typically when updating pre-existing user's data.
Note
Standard User Profile cannot be updated via the User Profile Bulk Import API.
Get the Repro API token¶
To use the User Profile Bulk Import API, you need to set the Repro API token
You can find the
Repro API Token
at SETTINGS > PRODUCT SETTINGS > Credentials
Warning
"REGISTER API TOKEN" makes the current API token unavailable. Once you do this operation, You never use the key again and revert to the previous state.
Access limit per unit time¶
Access is limited per time for User Profile Bulk Import API.
・The limit is set to 10 accesses for every 10 minutes per API token.
When the request is completed, there will be a notification at the
X-RateLimit-Limit
header showing the maximum number of requests per time.When the request count exceeds the maximum access number, the following response
HTTP Status 429 (Too Many Requests)
will appear.
For more information on response headers, please see Response.
Warning
The maximum access number may fluctuate without warning.
- The access limit setting value is per API token per function.For example, the push API and the user profile API use the same API token, but they are different functions, so even if they are requested at the same time, each is counted as one request.
CSV files¶
The User Profile Bulk Import API uses a CSV file to update the user profile.
File format¶
user_id,birthday,name,point
123456,2000-01-01T00:00:00+09:00,foo,123
234567,2001-02-01T00:00:00+09:00,bar,234
Header and Column¶
The header value of the first row will be registered as the User Profile key.
user_id
: Specify the User ID that are set with the User ID API from your app. This field is required.REPRO_ERRORS
column is used to show error messages which will be covered later in this document. User Profile with the nameREPRO_ERRORS
will not be created and the value will be ignored.The header value will be regarded as the key and values in the following rows will be the value of the incoming User Profile with the rest of the columns.
Empty string cannot be used for the header as the key, and the maximum number of characters is 255.
Characters will be treated as encoded with UTF-8.
Empty characters before or after keys and values will not be automatically truncated and be used as is.
The value of an unspecified key is not updated.
If the value is empty, the User Profile of the specified User ID will not be updated.
Type¶
While Repro's User Profile have four different data types, CSV doesn't allow having data types, so the following rule is used when determining the data type of each column.
For User Profile key names already registered on Repro, the original data type will be respected.
For new User Profile Keys, the data type will be determined from the values after reading certain number of columns.
The key will be considered as datetime if the given values follow the iso8601 format.
The key will be considered as decimal if the given values only contain number with floating point.
The key will be considered as int if the given values only contain numbers.
Everything else will be considered as string.
If there's any row that does not follow the inferred type rule, there will be an CSV file created which contains error information as an additional column, and an email with a link to download the error file will be sent.
User Profile Bulk Import API v3¶
The User Profile Bulk Import API v3 requires you to send two requests to different endpoints to update the user profile.
Execute
POST /v3/user_profiles/bulk_import
and obtain pre-signed URL which is required to direct upload to S3.Execute
PUT <pre-signed URL>
and upload a file to S3.
Note
The pre-signed URL is valid for 5 minutes. Therefore, please initiate a second request within 5 minutes of receiving the first response.
Sample code¶
A sample code is shown below.
#!/bin/bash
set -e
API_TOKEN="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
URI="https://api.reproio.com/v3/user_profiles/bulk_import"
file=$(realpath -e $1)
checksum=$(openssl dgst -md5 -binary ${file} | base64)
content_size=$(stat -c %s ${file})
body=$(cat <<EOF
{
"checksum": "${checksum}",
"byte_size": "${content_size}"
}
EOF
)
# First request
response=$(curl -X POST \
-H "X-Repro-Token: ${API_TOKEN}" \
-H "Content-Type: application/json" \
--data "${body}" \
${URI})
# Get the value needed for the second request from the response
url=$(echo ${response} | jq -r .direct_upload.url)
content_type=$(echo ${response} | jq -r '.direct_upload.headers["Content-Type"]')
content_md5=$(echo ${response} | jq -r '.direct_upload.headers["Content-MD5"]')
echo $response
# Second request
curl -X PUT \
-H "Content-Type: ${content_type}" \
-H "Content-MD5: ${content_md5}" \
--upload-file ${file} \
"${url}"
echo done
First request¶
Endpoint¶
HTTP Method
|
Endpoint |
---|---|
POST |
|
Request header¶
Header |
Description |
Optional / Required |
---|---|---|
Content-Type | Set application/json |
Required |
X-Repro-Token | Set the token retrieved with Get the API endpoint and Repro API token |
Required |
Request parameter¶
Field |
Description |
Optional / Required |
---|---|---|
checksum | The base64 encoded md5 checksum of the file used for import. This can be calculated with |
Required |
content_type | Specify either |
Optional |
byte_size | File size of the file used for importing |
Required |
Payload Example¶
To upload a CSV file
{
"checksum": "J1h240z2CdsRjz2Et5mnkA==",
"byte_size": "123456"
}
To upload a gzip compressed file
{
"checksum": "J1h240z2CdsRjz2Et5mnkA==",
"content_type": "application/gzip",
"byte_size": "12345"
}
Response header¶
The response header contains the following values.
Status |
Header |
Description |
---|---|---|
Success |
X-RateLimit-Limit |
Access limit per unit time |
Success |
X-RateLimit-Remaining |
The number of requests remaining in the current rate limit window. |
Success |
X-RateLimit-Reset |
The time at which the current rate limit window resets in UTC epoch seconds. (Unix UTC timestamp) |
Request limited |
Retry-After |
The time how long the user agent should wait before making a follow-up request. |
The response of succeeded¶
The response body contains belows as a response in JSON format.
Field |
Description |
---|---|
response.direct_upload | An object containing the information needed for the second request |
response.direct_upload.url | Endpoint URL for the second request |
response.direct_upload.headers | Header information which is required to the second request. |
Response body example of succeeded¶
{
"direct_upload": {
"url": "https://repro-direct-upload-production.s3.ap-northeast-1.amazonaws.com/6pWMZB1797H4bNVqTwFDcPzX?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAITZGHBHURRYGMYHA%2F20200618%2Fap-northeast-1%2Fs3%2Faws4_request&X-Amz-Date=20200618T011630Z&X-Amz-Expires=300&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost&X-Amz-Signature=2902f97f060894f900141b4dde2ca2bfae9de2bd8ff3c6e3f098eab6faf674d6",
"headers": {
"Content-Type": "text/csv",
"Content-MD5": "QYLqokRq51xGBqakDLsBvA=="
}
}
}
Error response¶
The list below contains all status codes and error messages of the responses.
400 Bad Request¶
You have an error in your request. Please check the error message.
The checksum contains an invalid string.
Response body
{
"error": {
"messages": [
"Validation failed: Checksum is invalid"
]
}
}
The byte_size is not specified.
Response body
{
"error": {
"messages": [
"param is missing or the value is empty: byte_size"
]
}
}
401 Unauthorized¶
You don't have Repro API Token set.
Response body
{
"status": "unauthorized",
"error": {
"messages": [
"Please include your Repro API Token as \"X-Repro-Token\" HTTP header."
]
}
}
403 Forbidden¶
The specified Repro API Token is invalid.
Response body
{
"status": "forbidden",
"error": {
"messages": [
"Please check your Repro API Token as \"X-Repro-Token\" HTTP header."
]
}
}
404 Not Found¶
The endpoint does not exist.
Response body
{
"status": "not_found",
"error": {
"messages": [
"Not found."
]
}
}
429 Too Many Requests¶
Too many requests. Please wait the seconds indicated in the Retry-After
header and make a follow-up request.
Response body
{
"status": "too_many_requests",
"error": {
"messages": [
"Too many requests hit the API too quickly."
]
}
}
Second request¶
Endpoint¶
HTTP Method |
Endpoint |
---|---|
PUT |
The value of |
Request header¶
Please specify all of the value of response.direct_upload.headers
included in the response as a header if the first request was succeeded.
Request parameter¶
Please set up a CSV file to update your user profile. The maximum file size is as follows.
- csv: 1 GB
- gzip: 100 MB
Please refer to CSV files and check the CSV format.
The response of succeeded¶
If the request is successful, a status code of 200
is returned.
Error response¶
Please refer to this document for the details on error responses.
Import notification¶
Usage
You will be notified via an email with details of the User Profile update once the processing completes. There will be a link to download the error file if there was anything wrong when performing the update.
Error CSV file¶
Usage
CSV files will be validated once the processing finishes. Rows that contain invalid data will have an extra column and error information as REPRO_ERRORS which can be downloaded from Import notification link. If the file size used in the request is too large or the file is empty, the REPRO_ERRORS column may not be present in the CSV containing the error information. If the REPRO_ERRORS column is present, values in the it will all be ignored, so you can modify the CSV with error information and re-send as another API request.
Sample error CSV file
REPRO_ERRORS,user_id,birthday,name,point
birthday should be iso8601 format,123456,2000-01-01 00:00:00.000,foo,123
point should be integer,234567,2001-02-01T00:00:00+09:00,bar,234.0
Error details¶
messages | Description |
---|---|
too many values | Too many columns of values for header |
user_id is unregistered user | We could not identify a user with the given user_id. |
[column name] should be UTF-8 | There is a UTF-8 incompatible data. |
[column name] should be integer | There is a non-integer value. |
[column name] should be decimal | There is a non-decimal value. |
[column name] should be iso8601 format | There is a value that does not follow the iso8601 format. |
Note
When there was no error the download link will not be inserted.
(Deprecated)User Profile Bulk Import API v2¶
Warning
This API v2 is scheduled to be deprecated on August 31, 2021. If you wish to use the API, please use User Profile Bulk Import API v3.
API format¶
Sample request
curl -X POST \
--verbose \
-H 'Content-Type: text/csv' \
-H 'X-Repro-Token:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' \
--data-binary '@sample.csv' \
'https://api.reproio.com/v2/user_profiles/bulk_import'
Request¶
Request header
The response header contains the following values.
Header |
Description |
Required |
---|---|---|
Content-Type | Setting |
Required |
X-Repro-Token | Set the token retrieved with Get the Repro API token |
Required |
Request body
Set the CSV or the gzipped CSV to HTTP body. For more details with CSV format, see CSV files
Response¶
Response header
The response header contains the following values.
Status |
Header |
Description |
---|---|---|
Success |
X-RateLimit-Limit |
Access limit per unit time |
Success |
X-RateLimit-Remaining |
The number of requests remaining in the current rate limit window. |
Success |
X-RateLimit-Reset |
The time at which the current rate limit window resets in UTC epoch seconds. (Unix UTC timestamp) |
Request limited |
Retry-After |
The time how long the user agent should wait before making a follow-up request. |
Response body
The list below contains ((all)) status codes and error messages of User Profile Bulk Import API responses.
202 Accepted¶
The request was successful. The result of type guesses of the column of CSV posted will be returned as the response body.
Response body
{
"user_profile": {
"birthday": {
"type": "datetime"
},
"name": {
"type": "string"
},
"point": {
"type": "int"
}
}
}
400 Bad Request¶
Response body
{
"error": {
"code": 2001,
"messages": [
"user_id column is required"
]
}
}
CSV format is invalid. Please refer to CSV files and check the CSV format.
messages | Description |
---|---|
empty file | The file is empty. Please check the file you tried to use. |
header size over 102400 bytes | Header size of the CSV file is over 100KB. Please make it below 100KB. |
newline character not found | A new line is missing. |
empty columns | The column is empty. |
user_id column is required | The header of the CSV is missing user_id column. |
[key name] should be UTF-8 | The CSV file contains UTF-8 incompatible character data. Please encode the CSV file in UTF-8. |
duplicate columns [key names] | Can not accept duplicate columns. |
Response body
{
"error": {
"messages": [
"not in gzip format"
]
}
}
The requested file is not in gzip format.
Response body
{
"error": {
"messages": [
"file maybe corrupt"
]
}
}
The gzip filed that has been POSTed may be corrupt.
Response body
{
"error": {
"messages": [
"This app has exceeded the limit of enabled user profile(x)"
]
}
}
Number of the User Profiles exceeds the maximum
401 Unauthorized¶
You don't have Repro API Token set.
Response body
{
"status": "unauthorized",
"error": {
"messages": [
"Please include your Repro API Token as \"X-Repro-Token\" HTTP header."
]
}
}
403 Forbidden¶
The specified Repro API Token is invalid.
Response body
{
"status": "forbidden",
"error": {
"messages": [
"Please check your Repro API Token as \"X-Repro-Token\" HTTP header."
]
}
}
404 Not Found¶
The endpoint does not exist.
Response body
{
"status": "not_found",
"error": {
"messages": [
"Not found."
]
}
}
413 Payload Too Large¶
The requested CSV file is too large.
Response body
{
"error": {
"messages": [
"client intended to send too large body"
]
}
}
415 Unsupported Media Type¶
The CSV file contains unacceptable data format. Please check if the Content-Type
header as well as the request endpoint.
Response body
{
"error": "415 Unsupported Media Type"
}
429 Too Many Requests¶
Too many requests. Please wait the seconds indicated in the Retry-After
header and make a follow-up request.
Response body
{
"status": "too_many_requests",
"error": {
"messages": [
"Too many requests hit the API too quickly."
]
}
}