Version 0.5.1

• The default rate limit for WaybackClient.search has been reduced to 24 requests/minute in order to match the actual hard limits now set on Wayback Machine servers. You can still customize rate limits by using a custom WaybackSession. (#208)

• The timemap_calls_per_second parameter for WaybackSession has been deprecated and will be removed in v0.6.0. The API it handled rate limiting for is now the same service and shares the same rate limiting as the one handled by search_calls_per_second. You should now only use search_calls_per_second. (#208)

• We’ve simplified the docs for the limit parameter of WaybackClient.search, since some of the Wayback Machine bugs it described workarounds for have been fixed. 🎉 (#209)

Version 0.5.0

The main focus of this release is continued refinement of rate limiting. You can now share or _not_share rate limits across sessions in a clearer way. Server-side rate limiting errors (if you set the rate limit too high in this package) are also now raised directly to your code instead of causing many-minute-long pauses.

There are number of other useful improvements here, too! Read on for details.

Breaking Changes

• Wayback now requires Python 3.8 or newer. Going forward, we intend to drop support for older Python releases once they hit end-of-life (i.e. they no longer receive any security updates or support from the Core Python team). These updates will always be noted as breaking changes.

• Wayback will no longer automatically pause and retry when the server returns rate-limit errors. Instead, it will raise a wayback.exceptions.WaybackRetryError exception (which includes how long you should probably pause for in the time attribute).

  The previous behavior helped solve some issues with the way rate limits were implemented on the client side that have since been fixed. It was also designed around users who had custom limits on Internet Archive servers, which is an unusual situation. The new approach is safer and can help prevent your IP address from getting blocked. If you need to retry after rate limit errors, make sure your code handles the exception and pauses all requests on all client instances for an appropriate amount of time.

Features

• Rate limiting has been significantly updated under the hood. Separate sessions or clients with customized limits now operate completely independently (their limits previously overlapped in a hard-to-predict way).

  If you need to make multiple wayback.WaybackSession instances that share limits, you can create an instance of wayback.RateLimit and pass it to each session via the *_calls_per_second arguments instead of passing an integer or float. For example:

  # Each of these sessions will be limited to 1 search request every 2
  # seconds. During a period where only one of these sessions is in use, it
  # will continue to operate at 1 request every 2 seconds:
  session1 = WaybackSession(search_calls_per_second=0.5)
  session2 = WaybackSession(search_calls_per_second=0.5)
  
  # These sessions will be limited to 1 search request each second as a
  # combined total (what you usually want). During a period where only one of
  # these sessions is in use, it will make 1 request per second, but if both
  # are actively in use, each will make 1 request every 2 seconds:
  rate = RateLimit(per_second=1)
  session1 = WaybackSession(search_calls_per_second=rate)
  session2 = WaybackSession(search_calls_per_second=rate)

  Sessions using the default limits share them via this same mechanism.

• Several attributes of wayback.CdxRecord have been renamed to match their names in Wayback’s actual CDX API. The old names still work for now, but using them will log a deprecation warning. We plan to remove them entirely in a future release. (#187)

  The renamed attributes are:

  • key (now urlkey).
  • url (now original).
  • mime_type (now mimetype).
  • status_code (now statuscode).

• You can now use a datetime.timedelta instance for the target_window argument to wayback.WaybackClient.get_memento() (an integer indicating a number of seconds still works, too). (#193)

  For example, to get the content of epa.gov from anytime within 5 days of January 1, 2008:

  client.get_memento(
    'https://epa.gov/',
    datetime(2008, 1, 1),
    exact=False,
    target_window=timedelta(days=5)
  )
  # <wayback.Memento url="http://www.epa.gov/" timestamp="2008-01-03T00:07:13+00:00">

Fixes & Maintenance

• The default rate limits have been further tweaked since v0.4.4 based on closer collaboration with Internet Archive staff. Rate limits are also now more accurately applied to each individual request (they were previously applied more roughly, without respect to retries and redirects).

• Updated license identifier to follow PEP 639 style. This should make sure the license is surfaced better and more clearly on PyPI. (#186)

• wayback.Memento now has a nicer, more informative repr when you are using a notebook or Python REPL. (#192)

• Tests where rate limits don’t matter now run faster. (#194)

• Source code is now autoformatted, type-checked, and more comprehensively covered by lint and style rules. (#185)

New Contributors

A huge thanks to three new contributors in this release!

• Derzan Chiang
• Beckett Frey
• @Adeelp1

v0.4.5

In v0.4.4, we broke archived mementos of rate limit errors — they started raising exceptions instead of returning the actual memento. We now correctly return mementos of rate limit errors while still raising exceptions for actual live rate limit errors from the Wayback Machine itself. (#158)

Full Changelog: v0.4.4...v0.4.5

Version 0.4.4

This release makes some small fixes to rate limits and retries in order to better match with the current behavior of Wayback Machine servers:

• Updates the WaybackClient.search rate limit to 1 call per second (it was previously 1.5 per second). (#140)

• Delays retries for 60 seconds when receiving rate limit errors from the server. (#142)

• Adds more logging around requests and rate limiting. This should make it easier to debug future rate limit issues. (#139)

• Fixes calculation of the time attribute on wayback.exceptions.WaybackRetryError. It turns out it was only accounting for the time spent waiting between retries and skipping the time waiting for the server to respond! (#142)

• Fixes some spots where we leaked HTTP connections during retries or during exception handling. (#142)

The next minor release (v0.5) will almost certainly include some bigger changes to how rate limits and retries are handled.

Version 0.4.3

This is mainly a compatibility release: it adds support for urllib3 v2.x and the next upcoming major release of Python, v3.12.0. It also adds support for multiple filters in searches. There are no breaking changes.

Features

You can now apply multiple filters to a search by using a list or tuple for the filter_field parameter of WaybackClient.search. Previously, you could only supply a string with a single filter. (#119)

For example, to search for all captures at nasa.gov with a 404 status code and “feature” somewhere in the URL:

client.search('nasa.gov/',
              match_type='prefix',
              filter_field=['statuscode:404',
                            'urlkey:.*feature.*'])

Fixes & Maintenance

• Add support for Python 3.12.0. (#123)
• Add support for urllib3 v2.x (urllib3 v1.20+ also still works). (#116)

Version 0.4.3a1

This is a test release for properly supporting the upcoming release of Python 3.12.0. Please file an issue if you encounter issues using on Python 3.12.0rc3 or later. (#123)

Version 0.4.2

Wayback is not compatible with urllib3 v2, and this release updates the package's requirements to make sure Pip and other package managers install compatible versions of Wayback and urllib3. There are no other fixes or new features.

Version 0.4.1

Features

wayback.Memento now has a links property with information about other URLs that are related to the memento, such as the previous or next mementos in time. It’s a dict where the keys identify the relationship (e.g. 'prev memento') and the values are dicts with additional information about the link. (#57)

For example::

{
    'original': {
        'url': 'https://www.fws.gov/birds/',
        'rel': 'original'
    },
    'first memento': {
        'url': 'https://web.archive.org/web/20050323155300id_/http://www.fws.gov:80/birds',
        'rel': 'first memento',
        'datetime': 'Wed, 23 Mar 2005 15:53:00 GMT'
    },
    'prev memento': {
        'url': 'https://web.archive.org/web/20210125125216id_/https://www.fws.gov/birds/',
        'rel': 'prev memento',
        'datetime': 'Mon, 25 Jan 2021 12:52:16 GMT'
    },
    'next memento': {
        'url': 'https://web.archive.org/web/20210321180831id_/https://www.fws.gov/birds',
        'rel': 'next memento',
        'datetime': 'Sun, 21 Mar 2021 18:08:31 GMT'
    },
    'last memento': {
        'url': 'https://web.archive.org/web/20221006031005id_/https://fws.gov/birds',
        'rel': 'last memento',
        'datetime': 'Thu, 06 Oct 2022 03:10:05 GMT'
    }
}

One use for these is to iterate through additional mementos. For example, to get the previous memento::

client.get_memento(memento.links['prev memento']['url'])

Fixes & Maintenance

• Fix an issue where the Memento.url attribute might be slightly off from the exact URL that was captured (it could have a different protocol, different upper/lower-casing, etc.). (#99)

• Fix an error when getting a memento for a redirect in view mode. If you called wayback.WaybackClient.get_memento with a URL that turned out to be a redirect at the given time and set the mode option to wayback.Mode.view, you’d get an exception saying “Memento at {url} could not be played.” Now this works just fine. (#109)

Version 0.4.0

Breaking Changes

This release includes a significant overhaul of parameters for WaybackClient.search.

• Removed parameters that did nothing, could break search, or that were for internal use only: gzip, showResumeKey, resumeKey, page, pageSize, previous_result.

• Removed support for extra, arbitrary keyword parameters that could be added to each request to the search API.

• All parameters now use snake_case. (Previously, parameters that were passed unchanged to the HTTP API used camelCase, while others used snake_case.) The old, non-snake-case names are deprecated, but still work. They’ll be completely removed in v0.5.0.

  • matchType → match_type
  • fastLatest → fast_latest
  • resolveRevisits → resolve_revisits

• The limit parameter now has a default value. There are very few cases where you should not set a limit (not doing so will typically break pagination), and there is now a default value to help prevent mistakes. We’ve also added documentation to explain how and when to adjust this value, since it is pretty complex. (#65)

• Expanded the method documentation to explain things in more depth and link to more external references.

While we were at it, we also renamed the datetime parameter of WaybackClient.get_memento to timestamp for consistency with the CdxRecord and Memento classes. The old name still works for now, but it will be fully removed in v0.5.0.

Features

• Memento.headers is now case-insensitive. The keys of the headers dict are returned with their original case when iterating, but lookups are performed case-insensitively. For example:

  list(memento.headers) == ['Content-Type', 'Date']
  memento.headers['Content-Type'] == memento.headers['content-type']

  (#98)

• There are now built-in, adjustable rate limits for calls to both search() and get_memento(). The default values should keep you from getting temporarily blocked by the Wayback Machine servers, but you can also adjust them when instantiating WaybackSession:

  # Limit get_memento() calls to 2 per second (or one every 0.5 seconds):
  client = WaybackClient(WaybackSession(memento_calls_per_second=2))
  
  # These now take a minimum of 0.5 seconds, even if the Wayback Machine
  # responds instantly (there's no delay on the first call):
  client.get_memento('http://www.noaa.gov/', timestamp='20180816111911')
  client.get_memento('http://www.noaa.gov/', timestamp='20180829092926')

  A huge thanks to @LionSzl for implementing this. (#12)

Fixes & Maintenance

• All API requests to archive.org now use HTTPS instead of HTTP. Thanks to @sundhaug92 for calling this out. (#81)

• Headers from the original archived response are again included in Memento.headers. As part of this, the headers attribute is now case-insensitive (see new features above), since the Internet Archive servers now return headers with different cases depending on how the request was made. (#98)

Version 0.3.3

This release extends the timestamp parsing fix from version 0.3.2 to handle a similar problem, but with the month portion of timestamps in addition to the day. It also implements a small performance improvement in timestamp parsing. Thanks to @edsu for discovering this issue and addressing this. (#88)

Full Changelog: v0.3.2...v0.3.3