Dimensional bidding custom algorithms

Dimensional bidding is a feature that allows you to apply one bid factor to a combination of target vector values. For example, you could apply a bid factor of 3.14 to the combination of mobile (device type) and Singapore (geo). If you targeted each of those vectors separately with a different bid factor, your two bid factors would multiply together. Multiplying multiple bid factors together to target precise, high-value market niches can result in unnecessarily high bids, which may result in the need for a max bid to act as a cap on your bid possibilities. To solve for this, you can extend your current ad group strategy with dimensional bidding to look for small pockets of high value while avoiding an exponential rise in bid value.

The following diagram illustrates the workflow process.

Data source configuration

Dimensional bidding is key performance indicator (KPI)-agnostic, so it doesn't matter whether you choose CTR, CPA, or something else as your goal. Dimensional bidding is represented as a list of item combinations, and each bid list is accessible at a selected level in your platform architecture. The code samples below assume an ad group level bid list.

This guide uses data from the REDS service as its input source, but you can use any data available to you to tailor your bidding strategy. The KPI you choose will determine the type of REDS events that you need to ingest.

	CPA	CTR	CPC	CPCV	TVCR	Viewability
Impressions	*	*	*	*	*	*
Conversions	*
Clicks	*	*	*
Video Events				*	*	*

Considerations

To create a bid list with more than two dimensions, like the examples in this article, please contact your Account Manager.

Maintaining multiple bid lists opens up the possibility of finding different kinds of small, high-value pockets of inventory. However, each bid list must contain a distinct combination type (e.g., site and geo, device type and ad environment, etc.).
Consider using the bid list at a single ad group level first, and, if the results are successful, copy the bid list to use in multiple ad groups under the same campaign or advertiser.
With dimensional bidding, you trade bid scale for specificity. To maintain scale, bear in mind that not all targeting needs to be made of bid combinations. By forcing your ad group's strategy into a bid list, you may risk losing the capability to bid competitively on a wider swath of the QPS (queries per second) available to you. Setting bid factors for vector combinations naturally limits the scope of what the ad group can bid on, as it requires the exact combination to be present in order to bid, rather than being able to bid on either of the vectors independently of each other.
You may want to incorporate statistical tests to show whether or not using dimensional bidding had a meaningful effect on your ad group's KPI. Assuming independence between one ad group without dimensional bidding and a test ad group with dimensional bidding, you could, as one example, implement an unpaired two-sample "T" test.

Identifying and ingesting data

Once you've selected your data source and selected the appropriate ad groups, you will be able to begin ingesting and parsing data.

Data retrieval

Prior Knowledge Needed and Call-Outs

Before reading further, please familiarize yourself with the background information and best practices on our REDS page. This data analysis for dimensional bidding is performed with the logs provided by this service.
Download the Python code sample that walks you through retrieval of your log-level data from Amazon S3.

Below, you will find a Python code snippet that connects to the Amazon s3 API via a Boto package (2.x version) to retrieve two types of data events: impressions and conversions.

Each new data logs' rows are appended to a single CSV file, which occurs inside the function ingestGzipLogs.

The final result includes two master CSV data files:

One file for impressions
One file for clicks

You can reconfigure this code snippet to allow for more frequent retrievals of the data, as opposed to a one-day batch retrieval, depending on your individual needs.

3.1.1 ingest logs - Python code sample

def ingestGzipLogs(dateFilter):
    for key in mybucket.list(FILTER_PATH + str(dateFilter)):
        logName = key.name
        if "impressions" in logName:
            key = ReadOnce(key)
            gz_file = gzip.GzipFile(fileobj=key, mode='rb')
            for line in gz_file:
                newLine = line.decode('utf-8').split('\t')
                with open(impCSV, "a") as output:
                    writer = csv.writer(output)
                    writer.writerow(newLine)
        elif "clicks" in logName:
            key = ReadOnce(key)
            gz_file = gzip.GzipFile(fileobj=key, mode='rb')
            for line in gz_file:
                newLine = line.decode('utf-8').split('\t')
                with open(clicksCSV, "a") as output2:
                    writer = csv.writer(output2)
                    writer.writerow(newLine)
        else:
            continue

You will use the impression and click data to calculate CTR distributions at the ad group level for a particular dimensional bidding combination.

Analysis and data transformation

In this section, we'll be using the R language (with its quick prototyping advantages) to look at the main themes of performing data analysis and transformation.

Download r sample code to guide you through the data-transformation operations.

Reading CSV files

To start, you can use the fread function from the data.table library package (as a faster alternative to read.table or read.csv) to quickly read your two CSV files from the initial retrieval. You may use functions such as dim(impData) or summary(impData) to get a general understanding of the dimensions of the data set, basic statistics along each column, and whether there are missing values.

impData <- fread("./DA_impressions_2018-02-16.csv", header = FALSE)
colnames(impData) <- c("logentryTime", "impID", "partnerID", "advertiserID", "campaignID", "adgroupID", "privateContractID", "audienceID", "creativeID", "adformat", "frequency", "ssp", "publisherID", "dealID", "site", "referrerCatList", "foldPosition", "userHourOfWeek",
 "userAgent", "IPAddress", "TDID", "country", "region", "metro", "city", "deviceType", "OSFamily", "OS", "browser", "recency", "languageCode", "mediaCost", "feeFeatureCost", "dataUsageCost", "TTDCost", "partnerCost",
 "advertiserCost", "lat", "long", "deviceID", "zipcode", "processedTime", "deviceMake", "deviceModel", "renderingContext", "carrierID", "tempCelsius", "tempBucketStartCelsius", "tempBucketEndCelsius","placementID")
 #provide header for impressions data
clicksData <- fread("./CLICKS_DATA.csv", header = FALSE)
colnames(clicksData) <- c("logentryTime", "clickID", "IPAddress", "referrerURL", "redirectURL", "campaignID", "channelID", "advertiserID", "displayImpressionID", "keyword", "keywordID", "matchType", "distributionNetwork", "TDID", "rawURL", "processedTime", "deviceID")
#provide header for click data

The sample code above assumes that you want to produce a range of dimensional bidding combinations for every ad group under the partner. Of course, you do have the option to filter down to the appropriate subset of data.

The first major code snippet showcases the importance of choosing the most appropriate dimensional-bidding combination for your ad group. In this example, you may believe that the combination of SSP, publisher ID, site, and impression placement ID could be important. You can also experiment with different kinds of combinations in parallel to see which perform best.

subImpData <- impData2 %>% select(impID, advertiserID, campaignID, adgroupID, ssp, publisherID, site, placementID)
subImpData$SPSPID = paste(subImpData$ssp, subImpData$publisherID, subImpData$site, subImpData$placementID, sep="_")

Following your selection of specific combinations, you can then determine how to handle the calculation of the click-through rates (CTRs) for each combination (in this case, the combination of SSP, publisher, site, and impression placement ID). The CTR ratio requires the numerator count, or the number of times a combination has appeared for a specific campaign and ad group.

clickCountData <- joinedData %>% add_count(campaignID.x, adgroupID, SPSPID)
allCountData <- subImpData %>% add_count(campaignID, adgroupID)
colnames(clickCountData)[colnames(clickCountData)=="n"] <- "numeratorCTR"
colnames(allCountData)[colnames(allCountData)=="n"] <- "denominatorCTR"

Then, you can merge these datasets together to perform the CTR calculation.

joinedData2 <- merge(allCountData, clickCountData, by.x = 'impID', by.y = 'impID', all.x = TRUE)

Finally, using the "dplyr" package under R, you can string together a sequence of data transformations to finish the calculation.

transformedData <- joinedData2 %>% select(campaignID, adgroupID.x, SPSPID.x, numeratorCTR, denominatorCTR) %>% group_by(campaignID, adgroupID.x) %>% mutate(ctr = numeratorCTR/denominatorCTR)

If you wanted to create distributions of CTR values per campaign and ad group, you could calculate the mean CTR and standard deviation of that distribution.

transformedData2 <- transformedData %>% group_by(campaignID, adgroupID.x) %>% mutate(ctrThreshold = mean(ctr) + sd(ctr))

With the above calculations finished, you could create a mathematical threshold that equates to keeping any combination that is one standard deviation above the mean for relatively high performance. In other words, you are employing a rule-based model to affect what the ad group optimizations should look like. There is also an additional step in which a "sdFactor" is calculated that effectively becomes the bid factor for a specific combination. This represents how far the calculated CTR is above the determined threshold (in a ratio term). Please note that the 0.1 constant dampens the effect of the sdFactor because of high variability based on one day's worth of won impressions.

transformedData3 <- transformedData2 %>% group_by(campaignID, adgroupID.x) %>% mutate(sdFactor = 0.1*(ctr/ctrThreshold))
transformedData3 <- unique(transformedData3, by = c("campaignID", "adgroupID.x", "SPSPID.x"))
finalData <- transformedData3 %>% filter(ctr >= ctrThreshold & ctr > 0)

Activate

Now it is time to activate the outcomes, which are the sdFactors from the previous step, via the Dimensional Bidding API.

Format data into a JSON bid list

You can start by creating a subset on one ad group in the final dataset in R.

finalData[finalData$adgroupID.x =='ADGROUP_ID',][,3:8]

This results in the following:

The goal is to take the sdFactors and feed them into a dimensional-bidding bid list JSON structure, shown in the following request.

POST /v3/bidlist request

In this example, we are creating the BidList at the AdGroup level.

{
  "Name": "Bidlist 1",
  "BidListAdjustmentType": "Optimized",
  "ResolutionType": "ApplyMultiplyAdjustment",
  "BidLines": [
    {
      "BidAdjustment": 1.02,
      "DomainFragment": "com.strawdogstudios.simonscatcrunchtime",
      "SupplyVendorId": 99,
      "PublisherId": "FYBER-139222",
      "ImpressionPlacementValue": "530c4d2894e444a1976f41d527c03a07"
    },
    {
      "BidAdjustment": 1.02,
      "DomainFragment": "com.pandora.android",
      "SupplyVendorId": 50,
      "PublisherId": "71711271",
      "ImpressionPlacementValue": "cKClcPNTu1Te"
    },
    {
      "BidAdjustment": 1.02,
      "DomainFragment": "com.merriamwebster",
      "SupplyVendorId": 118,
      "PublisherId": "11653",
      "ImpressionPlacementValue": "177976"
    },
    {
      "BidAdjustment": 2.03,
      "DomainFragment": "com.cardgame.solitaire.flat",
      "SupplyVendorId": 1,
      "PublisherId": "f77344fc701f4b1f981238390e0b8ffb",
      "ImpressionPlacementValue": "f37ac1804f8040a4b49295e5d13167ff"
    }
  ],
  "BidListOwner": "AdGroup",
  "BidListOwnerId": "sberg7l"
}

POST /v3/bidlist response

{
    "IsGlobal": false,
    "BidListDimensions": [
        "HasDomainFragmentId",
        "HasSupplyVendorId",
        "HasImpressionPlacementId",
        "HasPublisherId"
    ],
    "BidListId": "2131712",
    "Name": "Bidlist 1",
    "BidListSource": "User",
    "BidListAdjustmentType": "Optimized",
    "ResolutionType": "ApplyMultiplyAdjustment",
    "BidLines": [
        {
            "BidLineId": "1216340",
            "BidAdjustment": 2.030000,
            "DomainFragment": "com.cardgame.solitaire.flat",
            "SupplyVendorId": 1,
            "PublisherId": "f77344fc701f4b1f981238390e0b8ffb",
            "ImpressionPlacementValue": "f37ac1804f8040a4b49295e5d13167ff"
        },
        {
            "BidLineId": "1216341",
            "BidAdjustment": 1.020000,
            "DomainFragment": "com.merriamwebster",
            "SupplyVendorId": 118,
            "PublisherId": "11653",
            "ImpressionPlacementValue": "177976"
        },
        {
            "BidLineId": "1216342",
            "BidAdjustment": 1.020000,
            "DomainFragment": "com.pandora.android",
            "SupplyVendorId": 50,
            "PublisherId": "71711271",
            "ImpressionPlacementValue": "cKClcPNTu1Te"
        },
        {
            "BidLineId": "1216343",
            "BidAdjustment": 1.020000,
            "DomainFragment": "com.strawdogstudios.simonscatcrunchtime",
            "SupplyVendorId": 99,
            "PublisherId": "FYBER-139222",
            "ImpressionPlacementValue": "530c4d2894e444a1976f41d527c03a07"
        }
    ],
    "BidListOwner": "AdGroup",
    "BidListOwnerId": "sberg7l",
    "IsAvailableForLibraryUse": false
}

Associate the new bid list to an ad group

The last step is to associate the BidListId with the AdGroup AssociatedBidLists so that the AdGroup uses the correct bid lists during live bidding.

Adding a bid list to an existing ad group

When using PUT /v3/adgroup to associate a new bid list to an existing ad group, it's important to check if there are any other bid lists already associated to that ad group. PUT /v3/adgroup replaces the contents of the AssociatedBidLists object with what is sent in the request.

When adding a BidList to an AdGroup that was previously created, use GET /v3/adgroup/{adGroupId} to gather the current state of all editable properties in the AdGroup before using PUT /v3/adgroup to send through the same information, appending the existing BidList to AssociatedBidLists.

IsDefaultForDimension for each AssociatedBidList should be sent with each PUT request. If the value is currently true and is not passed with the PUT request, it will revert back to the default of false and may cause unexpected behavior.

GET /v3/adgroup/{adGroupId} curl Request

curl --location --request GET 'https://api.thetradedesk.com/v3/adgroup/sberg7l' \
--header 'Content-Type: application/json' \
--header 'TTD-Auth: yourauthtoken' \
--data-raw ''

GET /v3/adgroup/{adGroupId} Response

{
    "CampaignId": "t0ncimu",
    "AdGroupId": "sberg7l",
    "AdGroupName": "2020-03-09 Ad Group Test",
    "Description": null,
    "IsEnabled": false,
    "IsHighFillRate": false,
    "AdGroupCategory":{
        "CategoryId": 8311
    },
    "RTBAttributes": {}, /*This object was collapsed to save page space.*/ 
    "Availability": "Available",
    "CreatedAtUTC": "2020-03-11T00:39:53.227",
    "LastUpdatedAtUTC": "2020-03-11T00:39:53.227",
    "AssociatedBidLists": [
        {
            "BidListId": "2131711",
            "IsEnabled": true,
            "IsDefaultForDimension": true,
            "BidListSource": "User",
            "BidListAdjustmentType": "Optimized",
            "BidListDimensions": [
                "HasGeoSegmentId"
            ]
        }
    ],
    "AvailableRecommendations": [],
    "CustomLabels": [],
    "AreFutureKoaFeaturesEnabled": false,
    "PredictiveClearingEnabled": false,
    "CurrentAndFutureAdditionalFeeCards": []
}

PUT /v3/adgroup Request

{
    "AdGroupId": "sberg7l",
	"AssociatedBidLists": [
        {
            "BidListId": "2131711",
            "IsEnabled": true,
            "IsDefaultForDimension": true,
            "BidListAdjustmentType": "Optimized",
        },
        {
            "BidListId": "2131712",
            "IsEnabled": true,
            "IsDefaultForDimension": true,
            "BidListAdjustmentType": "Optimized",
        }
    ]
}

PUT /v3/adgroup Response

{
    "CampaignId": "t0ncimu",
    "AdGroupId": "sberg7l",
    "AdGroupName": "2020-03-09 Ad Group Test",
    "Description": null,
    "IsEnabled": false,
    "IsHighFillRate": false,
    "AdGroupCategory":{
        "CategoryId": 8311
    },
    "RTBAttributes": {}, /*This object was collapsed to save page space.*/ 
    "Availability": "Available",
    "CreatedAtUTC": "2020-03-11T00:39:53.227",
    "LastUpdatedAtUTC": "2020-03-11T00:39:53.227",
    "AssociatedBidLists": [
        {
            "BidListId": "2131711",
            "IsEnabled": true,
            "IsDefaultForDimension": true,
            "BidListSource": "User",
            "BidListAdjustmentType": "Optimized",
            "BidListDimensions": [
                "HasGeoSegmentId"
            ]
        },
        {
            "BidListId": "2131712",
            "IsEnabled": true,
            "IsDefaultForDimension": true,
            "BidListSource": "User",
            "BidListAdjustmentType": "Optimized",
            "BidListDimensions": [
                "HasDomainFragmentId",
                "HasSupplyVendorId",
                "HasImpressionPlacementId",
                "HasPublisherId"
            ]
        }
    ],
    "AvailableRecommendations": [],
    "NewBidLists": [],
    "CustomLabels": [],
    "AreFutureKoaFeaturesEnabled": false,
    "PredictiveClearingEnabled": false,
    "CurrentAndFutureAdditionalFeeCards": []
}