Access REDS in Amazon S3
Amazon Simple Storage Service (S3) is a cloud storage service The Trade Desk uses to store and share REDS data securely for reporting and analytics. With S3, you can integrate REDS feeds into your own data pipelines. Here's what you need to know about S3 storage and data processing:
- Some partners process more than 2000 log files per hour.
- Daily log size totals vary based on volume and range from 250MB to over 10GB per day. These will fluctuate based on the level of bidding activity.
- There may be duplicate information within stream files because we receive raw, unprocessed, and unattributed data from bidders.
- Log files exported close to hour changes may be located in the previous or next hour's folder.
- It may take 60-90 minutes after the hour ends for all files from that hour to arrive in your logs folder.
TIP: We recommend waiting for delivery to complete before processing that hour or implementing a rolling 3-hour window for processing.
Configuration options
Provide your Technical Account Manager with the following information for each feed:
| Question | Information To Provide |
|---|---|
| What partner or advertiser data should be included in the feed? | Choose partner, individual advertiser, or a group of advertisers and provide the corresponding Partner Advertiser IDs. |
| What types of feeds you'd like to be enabled for the selected partner and advertisers? | You can choose any of the following feed types:
|
| Would you like your files batched into larger files, instead of having to process smaller files? | Yes/No |
| What data processing option you'd like for handling GDPR consent, if your customer data may fall under GDPR regulations? | You have the following data processing options:
IMPORTANT: All configurations are subject to business approval. |
| Would you like to include the GDPR Consent feed? | The GDPR consent stream is optional for the Exclude Row and Scrub Personal Data processing preferences, but it is required for all streams with the Pass-Through processing preference. |
How to access REDS log files in Amazon S3
All log files are stored in The Trade Desk Amazon S3 bucket in partner-specific folders. To access your log files in your assigned partner folder, you must create an AWS access ID and secret key by following the Amazon S3 Access guide.
Log file names
Here's what you need to know about log file names in our Amazon S3 bucket.
- Each log file will have a file extension of
.log.gz. All stream files are gzip-compressed to reduce the overall delivery size. - All log files are tab-delimited.
- Each log file name adheres to the following convention, with parameters defined in the table that follows.
[FeedType]_[PartnerId]_ver_[FeedVersion]_[StartDateTime]_[EndDateTime]_[ProcessedDateTime]_[FeedId]_ [UniqueHashedValue].log.gz
| Field | Description | Sample Value Used In Examples |
|---|---|---|
[FeedType] |
Each log file is a record of one event type and will be labeled with one of these four values: clicks, conversions, impressions, videoevents. | impressions |
[PartnerId] |
Our unique, system-wide identifier that we set up for you as a partner. | 123xyz |
[AdvertiserId] |
Our unique, system-wide identifier that we generate for each advertiser. | 234abc |
[GroupId] |
A customizable group label for multi-advertiser feeds. | dealerships |
[Batched] |
An indicator that smaller files are combined into larger files and delivered on an hourly cadence, rather than as soon as they're ready. This is reflected in the folder name. | /batched |
[SchemaVersion] |
The schema version, which includes the following:
|
3-0-0 |
[FeedVersion] |
The version number for this particular feed. | 1 |
[StartDateTime] |
The earliest timestamp found within the file, formatted as yyyy-MM-ddTHHmmss. |
2024-05-01T080100 |
[EndDateTime] |
The latest timestamp found within the file, formatted as yyyy-MM-ddTHHmmss. |
2024-05-01T085900 |
[ProcessedDateTime] |
The timestamp created when the file is written, formatted as yyyy-MM-ddTHHmmss. This may be used to help differentiate multiple files that have the same start date and end date. |
2024-05-01T090100 |
[FeedId] |
A unique identifier we generate for each instance of a feed. | 123 |
[UniqueHashedValue] |
An MD5 hash of the contents of the log file appended to the filename to ensure no two files are identically named. | 82ce3baa0ca8afbe57069b785a7ea2af |
Here's a log file name example:
impressions_123xyz_ver_1_2024-05-01T080100_2024-05-01T085900_2024-05-01T090100_123_82ce3baa0ca8afbe57069b785a7ea2af.log.gz
The following examples show complete S3 paths for both delivery modes at the partner-level default location:
Unbatched:
s3://thetradedesk-useast-partner-datafeed/123xyz/reds/impressions/sv=3-0-0-1/date=2024-05-01/hour=8/impressions_123xyz_ver_1_2024-05-01T080100_2024-05-01T085900_2024-05-01T090100_123_82ce3baa0ca8afbe57069b785a7ea2af.log.gz
Batched:
s3://thetradedesk-useast-partner-datafeed/123xyz/reds/impressions/sv=3-0-0-1/batched/date=2024-05-01/hour=8/impressions_123xyz_ver_1_2024-05-01T080100_2024-05-01T085900_2024-05-01T090100_123_82ce3baa0ca8afbe57069b785a7ea2af.log.gz
Amazon S3 log folder default locations
The following sections provide the URIs of the log folder default locations.
Partner-level feeds
The logs folder base URI on Amazon is as follows, where PartnerId is the partner identifier we've assigned and provided to you:
The following examples show complete S3 paths for both delivery modes at the partner-level default location:
Unbatched:
s3://thetradedesk-useast-partner-datafeed/[PartnerId]/reds/[FeedType]/sv=[SchemaVersion]-[FeedVersion]
Batched:
s3://thetradedesk-useast-partner-datafeed/[PartnerId]/reds/[FeedType]/sv=[SchemaVersion]-[FeedVersion]/batched
The following table provides partner-level feed examples.
| Type | URI |
|---|---|
| Unbatched | s3://thetradedesk-useast-partner-datafeed/123xyz/reds/impressions/sv=3-0-0-1 |
| Batched | s3://thetradedesk-useast-partner-datafeed/123xyz/reds/impressions/sv=3-0-0-1/batched |
Individual advertiser-level feeds
The logs folder base URI on Amazon is as follows, where PartnerId is the partner identifier and AdvertiserId is the advertiser identifier we've assigned and provided to you:
The following examples show complete S3 paths for both delivery modes at the partner-level default location:
Unbatched:
s3://thetradedesk-useast-partner-datafeed/[PartnerId]/advertisers/[AdvertiserId]/reds/[FeedType]/sv=[SchemaVersion]-[FeedVersion]
Batched:
s3://thetradedesk-useast-partner-datafeed/[PartnerId]/advertisers/[AdvertiserId]/reds/[FeedType]/sv=[SchemaVersion]-[FeedVersion]/batched
| Type | URI |
|---|---|
| Unbatched | s3://thetradedesk-useast-partner-datafeed/123xyz/advertisers/234abc/reds/impressions/sv=3-0-0-1 |
| Batched | s3://thetradedesk-useast-partner-datafeed/123xyz/advertisers/234abc/reds/impressions/sv=3-0-0-1/batched |
Grouped advertiser-level feeds
The logs folder base URI on Amazon is as follows, where PartnerId is the partner identifier and GroupId is a grouping folder we set up for you. Depending on how you would like the advertisers groups, some examples could be banks or dealerships.
Unbatched:
s3://thetradedesk-useast-partner-datafeed/[PartnerId]/advertisers/mutliadvertiser_[Feedid]/reds/[FeedType]/sv=[SchemaVersion]-[FeedVersion]
Batched:
s3://thetradedesk-useast-partner-datafeed/[PartnerId]/advertisers/mutliadvertiser_[Feedid]/reds/[FeedType]/sv=[SchemaVersion]-[FeedVersion]/batched
| Type | URI |
|---|---|
| Unbatched | s3://thetradedesk-useast-partner-datafeed/123xyz/advertisers/multiadvertiser_999/reds/impressions/sv=3-0-0-1 |
| Batched | s3://thetradedesk-useast-partner-datafeed/123xyz/advertisers/multiadvertiser_999/reds/impressions/sv=3-0-0-1/batched |
Date and hour sub-folders
- Within the logs folder, files are grouped into sub-folders by date and then by the hour.
- The hour format for the folder structure is H. for hours that are less than 10, and the notation will be 8 instead of 08.
- The timezone for all date and time data is UTC, unless otherwise specified.
.../date=[yyyy-MM-dd]/hour=[H]
Example:
.../date=2024-05-01/hour=8
Transitioning between feed versions
Here's what you need to know about feed versions:
- When we make a requested configuration change to your feed, it is updated with a new feed version. Configuration changes include modifications to included and excluded columns, feed location, and privacy preferences.
- The new feed version starts receiving files from the time we made the change.
- The old feed and the new feed will run in parallel until you confirm you have migrated to the new feed.
TIP: We recommend monitoring your new feed for a fully completed hour. Once there is a full hour of data in the new feed, we recommend configuring the old feed to collect one more hour (plus recommended delay) and then cutting over to the new feed for the following hour.
FAQs
The following are some of the commonly asked questions about accessing REDS in Amazon S3.
Can I create more than one set of access keys?
Yes, however, you can have only two sets of keys (access key and secret key) at a time.