Http

flowtask.interfaces.http

HTTPService

HTTPService(*args, **kwargs)

Bases: CredentialsInterface, PandasDataframe

HTTPService.

Overview

    Interface for making connections to HTTP services.

api_get async

api_get(url, cookies=None, params=None, headers=None, use_proxy=None, free_proxy=False, use_http2=True)

Make an asynchronous HTTP GET request.

Parameters:

Name	Type	Description	Default
url	str	The URL to send the GET request to.	required
cookies	Cookies	Cookies to include in the request.	None
params	dict	Dictionary of query parameters to include in the URL.	None

Returns:

Name	Type	Description
dict	Dict[str, Any]	The JSON response from the API if the request is successful.
	Dict[str, Any]	Returns an empty dictionary if the request fails.

async_request async

async_request(url, method='GET', data=None, use_json=False, use_proxy=False, accept=None)

Asynchronously sends an HTTP request using aiohttp.

:param url: The URL to send the request to. :param method: The HTTP method to use (e.g., 'GET', 'POST'). :param data: The data to send in the request body. :param use_json: Whether to send the data as JSON. :param use_proxy: force proxy usage. :return: A tuple containing the result and any error information.

build_url

build_url(url, queryparams='', args=None)

Constructs a full URL with optional query parameters and arguments.

:param url: The base URL to be formatted. :param queryparams: Additional query parameters to be appended to the URL. :param args: Arguments to format within the URL. :return: The fully constructed URL.

evaluate_error async

evaluate_error(response, message)

evaluate_response.

Check Response status and available payloads. Args: response (type): description url (str): description

Returns:

Name	Type	Description
tuple	tuple	description

get_proxies async

get_proxies(session_time=0.4, free_proxy=False)

Asynchronously retrieves a list of free proxies. TODO: SELECT or rotate the free/paid proxies.

process_request async

process_request(future, url)

Processes the result of an asynchronous HTTP request.

:param future: The future object representing the asynchronous operation. :param url: The URL that was requested. :return: A tuple containing the result and any error information.

process_response async

process_response(response, url)

Processes the response from an HTTP request.

:param response: The response object from aiohttp. :param url: The URL that was requested. :return: A tuple containing the processed result and any error information.

refresh_proxies async

refresh_proxies()

Asynchronously refreshes the list of proxies if proxy usage is enabled.

request async

request(url, method='GET', data=None, use_proxy=False, accept=None)

Sends an HTTP request using the requests library.

:param url: The URL to send the request to. :param method: The HTTP method to use (e.g., 'GET', 'POST'). :param data: The data to send in the request body. :return: A tuple containing the result and any error information.

session async

session(url, method='get', data=None, cookies=None, headers=None, use_json=False, follow_redirects=False, use_proxy=False, accept=None, return_response=False)

Asynchronously sends an HTTP request using HTTPx.

:param url: The URL to send the request to. :param method: The HTTP method to use (e.g., 'GET', 'POST'). :param data: The data to send in the request body. :param use_json: Whether to send the data as JSON. :param cookies: A dictionary of cookies to send with the request. :param headers: A dictionary of headers to send with the request. :return: A tuple containing the result and any error information.

bad_gateway_exception

bad_gateway_exception(exc)

Check if the exception is a 502 Bad Gateway error.