Access curated REDS in Amazon S3

Amazon Simple Storage Service (S3) is a cloud storage service The Trade Desk uses to store and share REDS data securely for reporting and analytics. With S3, you can integrate REDS feeds into your own data pipelines. Here's what you need to know about S3 storage and data processing:

Some partners process more than 2000 log files per hour.
Daily log size totals vary based on volume and range from 250MB to over 10GB per day. These will fluctuate based on the level of bidding activity.
There may be duplicate information within stream files because we receive raw, unprocessed, and unattributed data from bidders.
Log files exported close to hour changes may be located in the previous or next hour's folder.
It may take 60-90 minutes after the hour ends for all files from that hour to arrive in your logs folder.

TIP: We recommend waiting for delivery to complete before processing that hour or implementing a rolling 3-hour window for processing.

How to access REDS log files in Amazon S3

All log files are stored in The Trade Desk Amazon S3 bucket in partner-specific folders. To access your log files in their assigned partner folder, you must create an AWS access ID and secret key by following the Amazon S3 Access guide.

Log file names

Here's what you need to know about log file names in our Amazon S3 bucket.

Each log file will have a file extension of .log.gz. All stream files are gzip-compressed to reduce the overall delivery size.
All log files are tab-delimited.
Each log file name adheres to the following convention, with parameters defined in the table that follows.

[FeedType]_[PartnerId]_ver_[FeedVersion]_[StartDateTime]_[EndDateTime]_[ProcessedDateTime]_[FeedId]_ [UniqueHashedValue].log.gz

Field	Description	Sample Value Used In Examples
`[FeedType]`	Each log file is a record of one event type and will be labeled with one of these four values: clicks, conversions, impressions, videoevents.	`impressions`
`[PartnerId]`	Our unique, system-wide identifier that we set up for you as a partner.	`123xyz`
`[AdvertiserId]`	Our unique, system-wide identifier that we generate for each advertiser.	`234abc`
`[FeedVersion]`	The version number for this particular feed.	`1`
`[StartDateTime]`	The earliest timestamp found within the file, formatted as `yyyy-MM-ddTHHmmss`.	`2024-05-01T080100`
`[EndDateTime]`	The latest timestamp found within the file, formatted as `yyyy-MM-ddTHHmmss`.	`2024-05-01T085900`
`[ProcessedDateTime]`	The timestamp created when the file is written, formatted as `yyyy-MM-ddTHHmmss`. This may be used to help differentiate multiple files that have the same start date and end date.	`2024-05-01T090100`
`[FeedId]`	A unique identifier we generate for each instance of a feed.	`123`
`[UniqueHashedValue]`	An MD5 hash of the contents of the log file appended to the filename to ensure no two files are identically named.	`82ce3baa0ca8afbe57069b785a7ea2af`

Here's a log file name example:

impressions_123xyz_ver_1_2024-05-01T080100_2024-05-01T085900_2024-05-01T090100_123_82ce3baa0ca8afbe57069b785a7ea2af.log.gz

Here's a complete folder and file name example using the partner-level default location:

s3://thetradedesk-useast-partner-datafeed/123xyz/reds/impressions/1/date=2024-05-01/hour=8/impressions_123xyz_ver_1_2024-05-01T080100_2024-05-01T085900_2024-05-01T090100_123_82ce3baa0ca8afbe57069b785a7ea2af.log.gz

FAQs

The following are some of the commonly asked questions about accessing REDS in Amazon S3.

Can I create more than one set of access keys?

Yes, however, you can have only two sets of keys (access key and secret key) at a time.