Contacts pagination error/limit?

[klouf]

In retrieving the set of (all) contacts for the account I am working with, I have run into a problem:

The account has 486 total contacts (192 active), however after paging through the returned results, I am only getting 300 results (6 pages) before the next link cycles back to page 5 and therefore becomes a loop (which was easy enough to detect and move past).

What's more is that of the 300 contacts returned, nearly half are duplicates or triplets, and in the end I am only getting 152 unique contacts.

Is this a bug or a limitation or both or neither?

For what it's worth, the project I am working on is in C# (I have also reproduced the same results in a web browser), but the problem is not on my side, rather it is server (API)-related.

Thanks in advance for any help/insight - it is greatly appreciated. I can certainly provide more information if needed.

Thanks,
Ken Louf

EDIT: I just confirmed this on a second account -- the results were again limited to 300 (6 pages); this account has 10,010 contacts (2483 active), and there were 50 duplicates this time, leaving 250 unique records accessible.

So it seems to me that there is a limitation as to the max results with the ContactsCollection, but I still don't understand the duplicates.

Confirmation of the limitation would be great and help with the duplicates too.

Thanks again in advance,
Ken Louf

[DaveBerard]

Hi Ken,

I'm trying to replicate this on our end, but I think it will be easier if you can post the code you're using to run the looped call backs in here so I can use the exact same calls you are using. If you can remove any personal information from your code and post here, it will be very helpful.

[klouf]


// using the Syndication stuff provided in .NET 3.5
using System.ServiceModel.Syndication;

private String baseUri = "http://api.constantcontact.com/ws/customers/" + + "/";
private List allContacts = new List<SyndicationItem>();
private List allContactsDocs = new List<XmlDocument>();

private void refreshContacts()
{
allContacts.Clear();
SyndicationFeed contactsFeed = null;
try
{
contactsFeed = getFeed(baseUri + "contacts");
}
catch (WebException we)
{
throw we;
}

foreach (SyndicationItem si in contactsFeed.Items)
{
allContacts.Add(si);

allContactsDocs.Add(getXmlFeed(si.Id));
}

bool hasMore = false;
SyndicationLink nextLink = null;
foreach (SyndicationLink sl in contactsFeed.Links)
{
if (sl.RelationshipType == "next")
{
hasMore = true;
nextLink = sl;
break;
}
}
List visited = new List();
while (hasMore)
{
SyndicationFeed nextFeed = null;
try
{
nextFeed = getFeed("http://api.constantcontact.com" + nextLink.Uri.ToString());
}
catch (WebException we)
{
throw we;
}

foreach (SyndicationItem si in nextFeed.Items)
{
allContacts.Add(si);

allContactsDocs.Add(getXmlFeed(si.Id));
}

visited.Add(nextLink.Uri.ToString());
hasMore = false;
foreach (SyndicationLink sl in nextFeed.Links)
{
if (sl.RelationshipType == "next")
{
if (!visited.Contains(sl.Uri.ToString()))
{
hasMore = true;
nextLink = sl;
break;
}
}
}
}

}

private SyndicationFeed getFeed(String uriPath)
{

HttpWebRequest hwr = WebRequest.Create(uriPath) as HttpWebRequest;
CredentialCache myCache = new CredentialCache();
myCache.Add(new Uri(baseUri), "Digest", new NetworkCredential(API_KEY + "%" + this.Uid, this.Pw));
hwr.Credentials = myCache;
HttpWebResponse resp = null;
try
{
resp = hwr.GetResponse() as HttpWebResponse;
}
catch (WebException we)
{
throw we;
}


SyndicationFeed sf = SyndicationFeed.Load(System.Xml.XmlReader.Create(resp.GetResponseStream()));
resp.Close();
return sf;
}

private XmlDocument getXmlFeed(String uriPath)
{

HttpWebRequest hwr = WebRequest.Create(uriPath) as HttpWebRequest;
CredentialCache myCache = new CredentialCache();
myCache.Add(new Uri(baseUri), "Digest", new NetworkCredential(API_KEY + "%" + this.Uid, this.Pw));
hwr.Credentials = myCache;
HttpWebResponse resp = null;
try
{
resp = hwr.GetResponse() as HttpWebResponse;
}
catch (WebException we)
{
throw we;
}

XmlDocument doc = new XmlDocument();
doc.Load(System.Xml.XmlReader.Create(resp.GetResponseStream()));
resp.Close();
return doc;
}


I think that's pretty much all the relevant code, although I certainly may have missed something, so let me know if there is anything else you need or if you have any questions. Like I said before, I'm pretty sure that it's not a code issue as I was able to get the same results plugging in the feed URLs into Firefox, grabbing the next urls from the page source, but feel free to show me I am wrong :)

Thanks,
Ken

[DaveBerard]

Hi Ken,

I'm able to come to the same problems you are on multiple accounts as well. I'm bringing this problem to our engineering group. I'll post here once we have a solution.

I the mean time, you can use the Activities Collection to accomplish a similar task using queries to the different lists (All, Active, Removed, Do Not Mail or any of the lists you have created). This will return a file that you will need to parse instead (.csv format generally) but will work correctly.

[klouf]

Dave,

Thanks for looking into this. I did know about the Activities collection, although I had already written the code to use the Contacts Collection, so I figured I would inquire about the issue first, before spending more time writing more code :) This isn't a super-critical project, so I'll probably wait it out for an answer, but yeah, in the end the result would be the same with either method.

Again, if you need anything else from me, just let me know and I'd be glad to help.

Thanks again,
Ken

[BradS57]

Hi klouf,

Did you ever find a solution for this issue, I'm currently dealing with the same problem.

[PaulW79]

5 years later and this bug still exists. Can I assume it won't be fixed in API v1?

[askrenes]

I wouldn't hold my breath. I can't even get them to fix API v2. 

[DaveBerard]

Thanks for pointing this out.  The issue reported in 2008 was resolved.  However, we have confirmed that is definitely an issue with pagination affecting some of our API customers (regardless of the version of the API) which results in pagnination looping.  We have engineers looking it now, no immediate update as to cause or expected time to fix.  We are treating this as a critical blocking defect with the API and have multiple people looking at getting a solution hot fixed to production as soon as we can.

[PaulW79]

In that case, sorry for the saucy post and thanks for the update :-)

[DaveBerard]

Not a problem, I understand how frustrating this can be.  Expectation is that pagination works and when it stops and goes into a loop like this, would be very hard to detect that and could cause integrations to go crazy. 

We do have good news.  On further investigation from the team, our v2 API is paginating correctly (pagination next links for v2 look surprisingly similar due to hashing, was a user error thinking they were the same when off by one character).  We are looking into the way the v1 API hashes pagination values to see why we're getting stuck in a loop now.  Hopefully will have another update soon.