How can I store or freeze content from HTTPGet()

Question

I use a lot of HTTPGets to pull in content for our emails. The problem I have is that if the email is viewed as a webpage, the HTTPGet runs again and new content is displayed. If the website I am retrieving has updated since send time the email and the online view look different and this will not work for us.

The method I currently use in AMPScript to retrieve and store the content is like this:

%%[
  set @content = HTTPGet(@url)
  TreatAsContent(@content)
]%%

I am looking for a way to store or freeze this HTML so that the request is only made at send time and the view online displays what was sent. At first I thought to maybe store it in a Data Extension and run logic off of __SendContext, but the content exceeds the character limit on DE text fields so it cannot be stored. Does anyone have any other ways I can achieve this?

Answer 1

I would suggest storing the HTTGET() results to a data extension row. To get around the 4000 character limitation, you could break up the stream into chunks.

%%[

var @em, @sid, @jid, @sk, @url, @extContent, @i, @chunkNum, @chunkStart, @chunkEnd, @l, @de, @chunk
set @jid = AttributeValue("jobid")
set @sid = AttributeValue("subscriberid")
set @sk = AttributeValue("_subscriberkey")
set @em = AttributeValue("emailaddr")
set @de = "chunkTest"
set @url = "http://www.degdigital.com"
set @extContent = httpGet(@url)

/* the > and < encoding will need to be undone when after retrieval */
set @extContent = replace(replace(replace(replace(@extContent,char(10),""),char(13),""),">","&gt;"),"<","&lt;")
set @l = length(@extContent)

outputline(concat("<br>url: ",@url))
outputline(concat("<br>len: ",@l,"<br>"))

set @chunkNum = 0
set @chunkSize = 3000

for @i = 1 to @l do

  if mod(@i,@chunkSize) == 0 OR @i == @l then

    if mod(@i,@chunkSize) == 0 then

        set @chunkStart = subtract(@i,subtract(@chunkSize,1))
        set @chunkEnd = @i
        set @chunk = substring(@extContent,@chunkStart,@chunkSize)

    elseif @i == @l then

        set @chunkStart = add(multiply(@chunkSize,@chunkNum),1)
        set @chunkEnd = @l
        set @chunk = substring(@extContent, @chunkStart, @chunkSize)

    endif

    set @chunkNum = add(@chunkNum,1)

    UpsertDE(@de, 4, "jobid", @jid, "subscriberId",@sid, "chunkNum",@chunkNum, "created", now(), "subscriberKey", @sk, "emailAddress", @em, "chunk", @chunk)

    outputline(concat("<br>chunk #",@chunkNum, ": ",@chunkStart, " - ", @chunkEnd))

  endif

next @i

]%%

Data Extension: ChunkTest

JobID, Number, PK
SubscriberID, Number, PK
ChunkNum, Number, PK
Created, Date, Current Date
SubscriberKey, Text (254)
EmailAddress, EmailAddress
Chunk, Text (max)

Sample Input / Output:

url: http://www.degdigital.com 
len: 29752

chunk #1: 1 - 3000 
chunk #2: 3001 - 6000 
chunk #3: 6001 - 9000 
chunk #4: 9001 - 12000 
chunk #5: 12001 - 15000 
chunk #6: 15001 - 18000 
chunk #7: 18001 - 21000 
chunk #8: 21001 - 24000 
chunk #9: 24001 - 27000 
chunk #10: 27001 – 29752

In a landing page, or the VAWP version of the email, you could perform an AMPScript lookup on ChunkTest for a specific JobID and SubscriberID. Then iterate through the result-set and concatenate all chunks into a single variable and undo the encoding for display.