How to avoid tons of API calls - by design or my mistake?

[kftb]

Hi!

I want to use discogs_client to assess the value of vinyls I have found in a box. The idea is to put all titles in a spreadsheet and get the info back from the API.

In my limited experience with APIs, my mental model would suggest that I query an endpoint, get all the data back and then pick out the elements I need.

With my below code, I would expect it to make one or a max of three API calls (one for search, one for releases, one for marketplace), instead I notice that it makes a new call for every single price item, which immediately bumps me into the API limit if done for a few of them.

d = discogs_client.Client("basement_vinyls/0.1", user_token=USER_TOKEN)

titles = ['Brett Dancer Urban Visionary Series - Pt. II']
    for count, item in enumerate(titles):
       
        results = d.search(item, type='release')
        artist = results[0].fetch('artists')

        titles[count]['record_id'] = results[0].id
        titles[count]['matched_title'] = results[0].title
        titles[count]['matched_artist'] = artist[0]['name']

        titles[count]['price_poor'] = results[0].price_suggestions.poor.value
        titles[count]['price_good'] = results[0].price_suggestions.good.value
        titles[count]['price_mint'] = results[0].price_suggestions.mint.value

These are my logs

INFO:root:Begin processing search term #1: Brett Dancer Urban Visionary Series - Pt. II 

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.discogs.com:443
DEBUG:urllib3.connectionpool:https://api.discogs.com:443 "GET /database/search?type=release&q=Brett%20Dancer%20Urban%20Visionary%20Series%20-%20Pt.%20II&page=1&per_page=50&token=[myAuthToken] HTTP/1.1" 200 None

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.discogs.com:443
DEBUG:urllib3.connectionpool:https://api.discogs.com:443 "GET /releases/37105?token=[myAuthToken] HTTP/1.1" 200 None

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.discogs.com:443
DEBUG:urllib3.connectionpool:https://api.discogs.com:443 "GET /marketplace/price_suggestions/37105?token=[myAuthToken] HTTP/1.1" 200 None

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.discogs.com:443
DEBUG:urllib3.connectionpool:https://api.discogs.com:443 "GET /marketplace/price_suggestions/37105?token=[myAuthToken] HTTP/1.1" 200 None

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.discogs.com:443
DEBUG:urllib3.connectionpool:https://api.discogs.com:443 "GET /marketplace/price_suggestions/37105?token=[myAuthToken] HTTP/1.1" 200 None

Is this the expected behavior or am I doing something wrong? Any help would be incredibly useful!

Thank you!

[JOJ0]

Hi, thanks for reaching out.

The api limit should be handled by discogs_client automagically. Which version are you using?

[kftb]

Thank you for checking in! I am less concerned about running into the API limit, but more about the general amount of API calls that are generated.

Is it common that if I request a price suggestions in three qualities that that would result in three separate API calls, or am I doing something wrong with my code above? (and I strongly believe its the latter)

[JOJ0]

@kftb, I think it works as designed. The library doesnt have a caching mechsnism. Not sure it that's all technically correct, other maintainers might correct me if I'm wrong.

But I'm still curious about what version you are using. Since version 2.3.10 we have a backoff mechanism that we thought works rock solid. If it doesnt in your usecase, that's the thing we should really be concerned about IMHO :-)

[AnssiAhola]

Is it common that if I request a price suggestions in three qualities that that would result in three separate API calls, or am I doing something wrong with my code above? (and I strongly believe its the latter)

As @JOJ0 said theres no caching so when you call the price_suggestions on the result 3 times it makes 3 seperate API calls.
So maybe do

prices_suggestions = results[0].price_suggestions
price_suggestions.mint.value
...

[JOJ0]

Thanks a lot @AnssiAhola, I justed tested your suggestion with requests debugging enabled. It seems on first access of suggestionsit fetches but then uses stuff from the variable:

>>> suggestions = results[0].price_suggestions
>>> print(suggestions.poor.value)
send: b'GET /marketplace/price_suggestions/37105?token=secret HTTP/1.1\r\nHost: api.discogs.com\r\nUser-Agent: Testclient\r\nAccept-Encoding: gzip\r\nAccept: */*\r\nConnection: keep-alive\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: Tue, 08 Feb 2022 07:19:42 GMT
header: Content-Type: application/json
header: Transfer-Encoding: chunked
header: Connection: keep-alive
header: Vary: Accept-Encoding
header: X-Discogs-Ratelimit: 60
header: X-Discogs-Ratelimit-Remaining: 59
header: X-Discogs-Ratelimit-Used: 1
header: Access-Control-Allow-Headers: Content-Type, authorization, User-Agent, Private-Auth-Secret, Discogs-UID
header: Access-Control-Allow-Methods: HEAD,OPTIONS,GET,OPTIONS
header: Access-Control-Expose-Headers: Location
header: Content-Language: en
header: X-Discogs-Media-Type: discogs.v2
header: Access-Control-Allow-Origin: *
header: X-Frame-Options: SAMEORIGIN
header: X-XSS-Protection: 1; mode=block
header: X-Content-Type-Options: nosniff
header: Cache-Control: public, must-revalidate
header: Content-Encoding: gzip
header: Strict-Transport-Security: max-age=15724800
header: CF-Cache-Status: DYNAMIC
header: Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
header: Set-Cookie: __cf_bm=T5bxHV2FHQyrLTlW8s_uReTC7QvKE1g3IXriotD.cg0-1644304782-0-AR5HuevtLQVgKkHIoI2C2R6KLRLnDq4/ioktR9n1VCeAwnkWqF5smY/kd/HnI5OeluVKWmmEa1iUT0WbEufOkj4=; path=/; expires=Tue, 08-Feb-22 07:49:42 GMT; domain=.discogs.com; HttpOnly; Secure; SameSite=None
header: Server: cloudflare
header: CF-RAY: 6da317592d9a7819-VIE
0.46
>>>
>>> print(suggestions.poor.value)
0.46
>>> print(suggestions.poor.value)
0.46
>>> print(suggestions.good.value)
1.38
>>> print(suggestions.mint.value)
8.739999999999998
>>>

[JOJ0]

@kftb does that work out for you? Do you need anything else or should we mark the last post as "answered"?
HTH

[kftb]

This worked, thank you so much!