Java SDK for downloading and filtering item feed files
- Summary
- Setup
- Downloading feed files
- Unzipping feed files
- Filtering feed files
- Schemas
- Logging
- Usage
- Performance
- Important notes
- License
The item feed files provide a rich set of data regarding active ebay listings. The feed files for any supported marketplace can be downloaded through the feed API.
However, since the volume of data is large, the download operation is performed in chunks, which may be cumbersome.
The feed SDK abstracts all the complexities and provides a very simple interface to -
The SDK can be added as a maven dependency or the entire repository can be cloned/forked and changes can be made.
You are most welcome to collaborate and enhance the existing code base.
<!-- https://mvnrepository.com/artifact/com.ebay.api/feed-sdk -->
<dependency>
<groupId>com.ebay.api</groupId>
<artifactId>feed-sdk</artifactId>
<version>1.0.1.beta-RELEASE</version>
</dependency>
For setting up the project in your local environment
- Clone the repository
- Run mvn clean install. This should generate an uber jar with the following naming convention
feed-sdk-{version}-uber.jar
Since the feed API supports feed files which may be as big as 5 GB, there is a capability which was built into the system to download the file in chunks of 100 MB.
The SDK abstracts the complexity involved in calculating the request header 'range' based on the response header 'content-range'.
To download a feed file which is -
- bootstrap : (feed_scope = ALL_ACTIVE)
- L1 category 1 : (category_id = 1)
- marketplace US : (X-EBAY-C-MARKETPLACE-ID: EBAY_US)
FeedRequest.FeedRequestBuilder builder = new FeedRequestBuilder();
builder.feedScope("ALL_ACTIVE")
.categoryId("1")
.siteId("EBAY_US")
.token(<TOKEN>)
.type(FeedTypeEnum.ITEM);
Feed feed = new FeedImpl();
GetFeedResponse response = feed.get(builder.build(), null);
The GetFeedResponse.filePath denotes the location where the file was downloaded.
The default download location is the current working directory. The download location can be changed by specifying the optional 'downloadDirectory' parameter in the feed method. For example, to download to the location /tmp/feed -
feed.get(builder.build(), "/tmp/feed");
Since the feed file is gzipped, it needs to be unzipped before post processing tasks can be run on it.
To unzip a downloaded feed file -
Feed feed = new FeedImpl();
String unzippedFilePath = feed.unzip(filePath)
The SDK provides the capability to filter the feed files based on :-
- List of leaf category ids
- List of seller usernames
- List of item locations
- Price range
- Level two Categories (the token scope should include api_scope and buy.item.feed scope)
- Level three Categories (the token scope should include api_scope and buy.item.feed scope)
- Item IDs
- EPIDs
- GTINs
On successful completion of a filter operation, a new filtered file is created.
To filter a feed file on leaf category ids would require these 3 lines of code -
FeedFilterRequest filterRequest = new FeedFilterRequest();
filterRequest.setLeafCategoryIds(<Set of leaf category ids>);
Response response = feed.filter(<path to unzipped feed file>, filterRequest);
The Response.filePath denotes the location of the filtered file.
The SDK provides the freedom to combine the filter criteria.
To filter on leaf category ids and seller user names for listings in the price range of 1 to 100
FeedFilterRequest filterRequest = new FeedFilterRequest();
filterRequest.setLeafCategoryIds(<Set of leaf category IDs>);
filterRequest.setSellerNames(<Set of seller user names>);
filterRequest.setPriceLowerLimit(1);
filterRequest.setPriceUpperLimit(100);
Response response = feed.filter(<path to unzipped feed file>, filterRequest);
filterRequest.setLevelTwoCategories(<Set of level two category ids>);
Response response = feed.filter(<path to unzipped feed file>, filterRequest);
filterRequest.setLevelThreeCategories(<Set of level three category ids>);
Response response = feed.filter(<path to unzipped feed file>, filterRequest);
This section provides more detail on what information is contained within the instances that are returned from the SDK method calls.
The ApiResponse instance is returned from the Feed.get() method.
private Integer statusCode;
private String message;
private String filePath;
List<ErrorData> errors;
| Field name | Description |
|---|---|
| statusCode | Integer: 0 indicates a successful response. Any non zero value indicates an error |
| message | String: Detailed information on the status |
| filePath | String: Absolute path of the location of the resulting file |
| errors | List: Detailed error information |
This Response instance is returned from Feed.unzip and Feed.filter methods.
private Integer statusCode;
private String message;
private String filePath;
| Field name | Description |
|---|---|
| statusCode | Integer: 0 indicates a successful response. Any non zero value indicates an error |
| message | String: Detailed information on the status |
| filePath | String: Absolute path of the location of the resulting file |
Logging is configured using slf4j and rolling log files are created in the current directory.
Ensure that appropriate permissions are present to write to the directory
- The current log file name is : feed-sdk-log.log
- Rolling Log files are created per day with the pattern : feed-sdk-log.{yyyy-MM-dd}.log
- Log files are retained for a period of 7 days
- Total log file size is capped at 100 MB
- Individual log file size is capped at 3 MB, at which new file will be created
The log file location can be changed overriding the LOG_HOME variable, and passing as VM argument. For example, to change the log file location to /var/log
-DLOG_HOME="/var/log"
The following sections describe the different ways in which the SDK can be used
All the capabilities of the SDK can be invoked using the command line.
To see the available options and filters , use '-help'
java -jar feed-sdk-1.0.0.beta-RELEASE-uber.jar -help
usage: Feed SDK CLI
-help display help
-dt <arg> the date when feed file was generated
-c1 <arg> the l1 category id of the feed file
-scope <arg> the feed scope. Available scopes are
ALL_ACTIVE or NEWLY_LISTED
-mkt <arg> the marketplace id for which feed is
being request. For example - EBAY_US
-token <arg> the oauth token for the consumer. Omit
the word 'Bearer'
-c2f <arg> list of l2 categories which are used to
filter the feed
-c3f <arg> list of l3 categories which are used to
filter the feed
-lf <arg> list of leaf categories which are used
to filter the feed
-sellerf <arg> list of seller names which are used to
filter the feed
-locf <arg> list of item locations which are used
to filter the feed
-pricelf <arg> lower limit of the price range for
items in the feed
-priceuf <arg> upper limit of the price range for
items in the feed
-epidf <arg> list of epids which are used to filter
the feed
-iepidf <arg> list of inferred epids which are used
to filter the feed
-gtinf <arg> list of gtins which are used to filter
the feed
-itemf <arg> list of item ids which are used to
filter the feed
-dl,--downloadlocation <arg> override for changing the directory
where files are downloaded
-cl,--credentiallocation <arg> directory where the credentials file is
located
-authscopes,--oauthscopes <arg> list of scopes
-env <arg> Environment Type. Supported Environments are SANDBOX and PRODUCTION
For example, to use the command line options to download, unzip and filter feed files
Using token
java -jar feed-sdk-1.0.0.beta-RELEASE-uber.jar -dt 20180701 -c1 1281 -scope ALL_ACTIVE -lf 46310 177789 -mkt EBAY_US -c3f 177792 116381 -pricelf 2 -priceuf 100 -locf US CN -token xxx
Using credentials file
java -jar feed-sdk-1.0.0.beta-RELEASE-uber.jar -dt 20180801 -c1 11700 -scope ALL_ACTIVE -mkt EBAY_US -pricelf 2 -priceuf 100 -locf US CN -cl <absolute path of credentials file>
All the capabilities of the SDK can be leveraged via a config file. The feed file download and filter parameters can be specified in the config file for multiple files, and SDK will process them sequentially.
The structure of the config file
{
"requests": [{
"feedRequest": {
"categoryId": "1",
"marketplaceId": "EBAY_US",
"date": "20180708",
"feedScope": "ALL_ACTIVE",
"type": "ITEM"
},
"filterRequest": {
"leafCategoryIds": ["11675", "3226"],
"itemLocationCountries": ["US", "CN"],
"priceLowerLimit": 10.0,
"priceUpperLimit": 100.0
},
"levelThreeCategories": ["1313", "13583", "39"]
}]
}
An example of using the SDK through a config file is located at
Config file based approach example - ConfigFileBasedExample.java
Samples showing the usage of available operations and filters.
All the examples are located here
- Filter by item location
- Filter by leaf categories
- Filter by price
- Filter by seller user names
- Combining multiple filters
- Filter on level three category
- Filter on EPID
- Filter on GTIN
- Filter on item IDs
- Filter by 'env'
| Category | Type | Size gz | Size unzipped | Records | Applied Filters | Filter Time | Total Time |
|---|---|---|---|---|---|---|---|
| 1 | BOOTSTRAP | 570 MB | 2.3 GB | 2.6 Million | l3 categories, sellers | ~ 20 seconds | ~ 2 minutes |
| 11450 | BOOTSTRAP | 3.87 GB | 65 GB | 47 Million | categories | ~ 2 mins | ~ 25 minutes |
| 293 | BOOTSTRAP | 530 MB | 3 GB | 3 Million | sellers, item locations, price | ~ 30 seconds | ~ 1.5 minutes |
| 1281 | BOOTSTRAP | 121 MB | 1.1 GB | 900,000 | leaf categories, l3 categories, item locations, price | ~ 10 seconds | ~ 30 seconds |
| 11232 | BOOTSTRAP | 128 MB | 670 MB | 600,000 | leaf categories, item locations, price | ~ 10 seconds | ~ 30 seconds |
| 220 | BOOTSTRAP | 700 MB | 3.5 GB | 5.5 Million | leaf categories, item locations, price | ~ 30 seconds | ~ 2 minutes |
- Ensure there is enough storage for unzipped files
- Ensure that the log and file storage directories have appropriate write permissions
- In case of failure in downloading due to network issues, the process needs to start again. There is no capability at the moment, to resume.
Copyright 2018 eBay Inc. Developer: Shankar Ganesh
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.