Skip to content

Add request timeouts - #296

Open
csherry04 wants to merge 1 commit into
opencap-org:devfrom
csherry04:fix/request-timeout
Open

Add request timeouts#296
csherry04 wants to merge 1 commit into
opencap-org:devfrom
csherry04:fix/request-timeout

Conversation

@csherry04

Copy link
Copy Markdown

Added timeouts to retry helper as per #277

10 seconds for most calls and 300 sec on response for s3 uploads conservatively, unsure of best practice here

revised tests within the test file to align with new timeout

@AlbertoCasasOrtiz
AlbertoCasasOrtiz self-requested a review July 30, 2026 21:58
@AlbertoCasasOrtiz

Copy link
Copy Markdown
Member

I run the tests and got the following response:

(opencap-core) PS C:\Workspaces\opencap\opencap-core> pytest .\tests\
=============================================================================== test session starts ===============================================================================
platform win32 -- Python 3.9.23, pytest-8.4.2, pluggy-1.6.0
rootdir: C:\Workspaces\opencap\opencap-core
collected 44 items                                                                                                                                                                 

tests\test_api.py ...F                                                                                                                                                       [  9%]
tests\test_main.py ......                                                                                                                                                    [ 22%]
tests\test_sync.py ..................................                                                                                                                        [100%]

==================================================================================== FAILURES =====================================================================================
___________________________________________________________________________ test_success_after_retries ____________________________________________________________________________

mock_get_conn = <MagicMock name='_get_conn' id='2223048902736'>

    @patch("urllib3.connectionpool.HTTPConnectionPool._get_conn")
    def test_success_after_retries(mock_get_conn):
        def make_response(status):
            response = Mock(status=status, headers={})
            response.stream.return_value = []
            response._original_response = None
            return response
    
        mock_get_conn.return_value.getresponse.side_effect = [
            make_response(500),
            make_response(502),
            make_response(200),
            make_response(429),
        ]
    
>       response = makeRequestWithRetry('GET',
                                        'https://test.com',
                                        retries=5,
                                        backoff_factor=0.1)

tests\test_api.py:95: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
utils.py:2122: in makeRequestWithRetry
    response = session.request(method,
C:\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\requests\sessions.py:589: in request
    resp = self.send(prep, **send_kwargs)
C:\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\requests\sessions.py:703: in send
    r = adapter.send(request, **kwargs)
C:\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\requests\adapters.py:644: in send
    resp = conn.urlopen(
C:\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\urllib3\connectionpool.py:736: in urlopen
    response = self.ResponseCls.from_httplib(
C:\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\urllib3\response.py:649: in from_httplib
    headers = HTTPHeaderDict(headers.items())
C:\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\urllib3\_collections.py:148: in __init__
    self.extend(headers)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = HTTPHeaderDict({}), args = (<Mock name='mock.msg.items()' id='2223049371120'>,), kwargs = {}, other = <Mock name='mock.msg.items()' id='2223049371120'>

    def extend(self, *args, **kwargs):
        """Generic import function for any type of header-like object.
        Adapted version of MutableMapping.update in order to insert items
        with self.add instead of self.__setitem__
        """
        if len(args) > 1:
            raise TypeError(
                "extend() takes at most 1 positional "
                "arguments ({0} given)".format(len(args))
            )
        other = args[0] if len(args) >= 1 else ()
    
        if isinstance(other, HTTPHeaderDict):
            for key, val in other.iteritems():
                self.add(key, val)
        elif isinstance(other, Mapping):
            for key in other:
                self.add(key, other[key])
        elif hasattr(other, "keys"):
>           for key in other.keys():
E           TypeError: 'Mock' object is not iterable

C:\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\urllib3\_collections.py:250: TypeError
-------------------------------------------------------------------------------- Captured log call --------------------------------------------------------------------------------
DEBUG    urllib3.connectionpool:connectionpool.py:475 https://test.com:443 "GET / <MagicMock name='_get_conn()._http_vsn_str' id='2223049337680'>" 500 <Mock name='mock.length' id='2223049353728'>
WARNING  urllib3.connectionpool:connectionpool.py:490 Failed to parse headers (url=https://test.com:443/): expected httplib.Message, got <class 'unittest.mock.Mock'>.
Traceback (most recent call last):
  File "C:\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\urllib3\connectionpool.py", line 488, in _make_request
    assert_header_parsing(httplib_response.msg)
  File "C:\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\urllib3\util\response.py", line 56, in assert_header_parsing
    raise TypeError("expected httplib.Message, got {0}.".format(type(headers)))
TypeError: expected httplib.Message, got <class 'unittest.mock.Mock'>.
WARNING  urllib3.connectionpool:connectionpool.py:323 Connection pool is full, discarding connection: test.com. Connection pool size: 10
================================================================================ warnings summary =================================================================================
..\..\..\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\matplotlib\_fontconfig_pattern.py:64
  C:\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\matplotlib\_fontconfig_pattern.py:64: PyparsingDeprecationWarning: 'oneOf' deprecated - use 'one_of'
    prop = Group((name + Suppress("=") + comma_separated(value)) | oneOf(_CONSTANTS))

..\..\..\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\matplotlib\_fontconfig_pattern.py:85
..\..\..\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\matplotlib\_fontconfig_pattern.py:85
..\..\..\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\matplotlib\_fontconfig_pattern.py:85
..\..\..\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\matplotlib\_fontconfig_pattern.py:85
..\..\..\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\matplotlib\_fontconfig_pattern.py:85
..\..\..\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\matplotlib\_fontconfig_pattern.py:85
  C:\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\matplotlib\_fontconfig_pattern.py:85: PyparsingDeprecationWarning: 'parseString' deprecated - use 'parse_string'
    parse = parser.parseString(pattern)

..\..\..\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\matplotlib\_fontconfig_pattern.py:89
..\..\..\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\matplotlib\_fontconfig_pattern.py:89
..\..\..\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\matplotlib\_fontconfig_pattern.py:89
..\..\..\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\matplotlib\_fontconfig_pattern.py:89
..\..\..\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\matplotlib\_fontconfig_pattern.py:89
..\..\..\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\matplotlib\_fontconfig_pattern.py:89
  C:\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\matplotlib\_fontconfig_pattern.py:89: PyparsingDeprecationWarning: 'resetCache' deprecated - use 'reset_cache'
    parser.resetCache()

..\..\..\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\matplotlib\_mathtext.py:45
  C:\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\matplotlib\_mathtext.py:45: PyparsingDeprecationWarning: 'enablePackrat' deprecated - use 'enable_packrat'
    ParserElement.enablePackrat()

..\..\..\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\tensorflow\python\framework\dtypes.py:205
  C:\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\tensorflow\python\framework\dtypes.py:205: DeprecationWarning: `np.bool8` is a deprecated alias for `np.bool_`.  (Deprecated NumPy 1.24)
    np.bool8: (False, True),

..\..\..\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:326
  C:\Users\NMBL\anaconda3\envs\opencap-core\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:326: DeprecationWarning: `np.bool8` is a deprecated alias for `np.bool_`.  (Deprecated NumPy 1.24)
    np.bool8: (False, True),

tests/test_main.py: 216 warnings
tests/test_sync.py: 216 warnings
  C:\Workspaces\opencap\opencap-core\utilsSync.py:619: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
    lNan = np.intersect1d(lOccluded,np.arange(np.argwhere(confidence_out[mkr,:]>0)[1],np.argwhere(confidence_out[mkr,:]>0)[-1])) # don't add nans to 0 pad

tests/test_main.py: 216 warnings
tests/test_sync.py: 216 warnings
  C:\Workspaces\opencap\opencap-core\utilsSync.py:628: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
    rNan = np.intersect1d(rOccluded,np.arange(np.argwhere(confidence_out[mkr,:]>0)[1],np.argwhere(confidence_out[mkr,:]>0)[-1])) # don't add nans to 0 pad

tests/test_main.py: 12 warnings
tests/test_sync.py: 12 warnings
  C:\Workspaces\opencap\opencap-core\utilsSync.py:464: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
    iStart = int(np.argwhere(timeVecs[iCam]==tStart))

tests/test_main.py: 12 warnings
tests/test_sync.py: 12 warnings
  C:\Workspaces\opencap\opencap-core\utilsSync.py:465: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
    iEnd = int(np.argwhere(timeVecs[iCam]==tEnd))

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================================================= short test summary info =============================================================================
FAILED tests/test_api.py::test_success_after_retries - TypeError: 'Mock' object is not iterable
============================================================= 1 failed, 43 passed, 928 warnings in 201.40s (0:03:21) =============================================================

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants