utils package¶

Submodules¶

utils.cache module¶

Utils for caching functions.

utils.cache.cache(cache_name)¶

Wrap class Cache as decorator.

Args:: cache_name (str): cache name

from utils.cache import cache

@cache('test')
def long_operation():
    import time
    time.sleep(100)
    return 1

>>> long_operation() # takes >100s to run
>>> long_operation() # runs almost instantly

utils.db module¶

Database utils.

utils.db.get_id_key(entity_name)¶

Get entity id key.

Follows the simple convention that id_key is the entity_name followed by ‘_id’.

Args:: entity_name (str): entity_name
Return:: id_key (str)

>>> from utils.db import get_id_key
>>> entity_name = 'province'
>>> print(get_id_key(province))
'province_id'

utils.ds module¶

Utils related to lists and dicts.

utils.ds.dict_list_to_index(dict_list, key)¶

Given a list of dicts, returns an index mapping each dict item to the dict.

Args:: dict_list (list of dicts): list of dicts key (str): key
Return:: index (dict)

>>> from utils.ds import dict_list_to_index
>>> dict_list = [
    {'name': 'Alice', 'age': 20},
    {'name': 'Bob', 'age': 25},
    {'name': 'Charlie', 'age': 55},
]
>>> print(dict_list_to_index(dict_list, 'name'))
{
    'Alice': {'name': 'Alice', 'age': 20},
    'Bob': {'name': 'Bob', 'age': 25},
    'Charlie': {'name': 'Charlie', 'age': 55},
}

utils.ds.flatten(list_of_list)¶

Flatten list of lists.

Args:: list_of_list(list): list of lists
Return:: flattened list

>>> print(flatten([[1, 2], [3, 4, 5], [6], []]))
[1, 2, 3, 4, 5, 6]

utils.ds.unique(lst)¶

Get unique values from list.

Args:: lst (list): List
Return:: list of unique values

>>> from utils.ds import unique
>>> lst = [1, 1, 1, 2]
>>> print(unique(lst))
[1, 2]

utils.dt module¶

Utils related to simple data types

utils.dt.parse_float(float_str, default=None)¶

Parse float.

Args:: float_str (str): float as string default (, optional): optional value to return of parsing fails
Return:: float value

>>> from utils.dt import parse_float
>>> parse_float('1.23')
1.23
>>> parse_float('1.23abc')
None
>>> parse_float('1.23abc', 'abc')
'abc'

utils.flask module¶

Flask (https://en.wikipedia.org/wiki/Flask_(web_framework)) utils.

class utils.flask.FlaskClient(server_name, host, port)¶

Bases: object

Implements FlaskClient.

Args:: server_name(str): Name of flask app host (str): host port (int): port

>>> from utils.flask import FlaskClient
>>> client = FlaskClient('geo', '127.0.0.1', 5000)

get_url(cmd, param_list)¶

Get URL.

Args:: cmd(str): command param_list (list): params
Assumes URL building convention:: http://<host>:<port>/<server_name>_server/<cmd>/<params>
Returns:: URLs

run(cmd, param_list)¶

Run client request.

Args:: cmd(str): command param_list (list): params
Returns:: Flask server response

>>> from utils.flask import FlaskClient
>>> client = FlaskClient('gig', '127.0.0.1', 5000)
>>> client.run('get_entity', 'LK-11')

utils.geo module¶

Geo-Spatial Utils.

utils.geo.deg_to_rad(deg)¶

Convert degrees to radians.

Args:: deg (float): Angle in degrees
Return:: Angle in radians

>>> from utils import geo
>>> print(geo.deg_to_rad(180))
3.141592653589793

utils.geo.get_distance(latlng1, latlng2)¶

Get distance between two points.

Args:: latlng1 ([lat, lng]): First point latlng2 ([lat, lng]): Second point
Returns:: Distance in km
Note:: Assumes EARTH_RADIUS = 6373.0 km

>>> from utils import geo
>>> print(geo.get_distance(geo.LAT_LNG_COLOMBO, geo.LAT_LNG_KANDY))
94.36504869698388

utils.geo.parse_latlng(latlng_str)¶

Parse latlng string.

Args:: latlng_str(str): String containing lat, lng
Return:: [lat, lng] float pair

>>> from utils import geo
>>> print(geo.parse_latlng('5N,70E'))
(5.0, 70.0)
>>> print(geo.parse_latlng('5°N,70°E'))
(5.0, 70.0)
>>> print(geo.parse_latlng('5,70'))
(5.0, 70.0)

utils.jsonx module¶

JSON utils.

>>> from utils import jsonx
>>> data = {'name': 'Alice', 'age': 20}
>>> file_name = '/tmp/data.json'
>>> jsonx.write(file_name, data)
>>> data2 = jsonx.read(file_name)
>>> data == data2
True

utils.jsonx.read(file_name)¶

Read JSON from file.

Args:: file_name (str): file name
Returns:: Parsed JSON data

utils.jsonx.write(file_name, data)¶

Write data as JSON to file.

Args:: file_name (str): file name data: data as serializable object

utils.mr module¶

Map-Reduce utils.

utils.mr.map_parallel(func_map, params_list, max_threads=4)¶

Run list(map(…)) in parallel.

Args:: func_map (function): Mapper function params_list (list): Params to be mapped max_threads (int, optional): Maximum parallel threads.

DEFAULT_MAX_THREADS = 4

>>> from utils import mr
>>> print(mr.map_parallel(lambda x: x ** 2, [1, 2, 3, 4]))
[1, 4, 9, 16]

utils.sysx module¶

System utils.

utils.sysx.log_metrics()¶

Log system metrics.

>>> from utils import sysx
>>> print(sysx.log_metrics())
{"ut": 1620724794.43984, "pid": 15129,
    "cpu_percent": 16.3, "vm_percent": 65.7}

Note:

Needs psutil

pip install psutil

utils.timex module¶

Time utils.

class utils.timex.StopWatch¶

Bases: object

Implements StopWatch.

>>> import time
>>> from utils import timex
>>> sw = timex.StopWatch()
>>> time.sleep(3)
>>> print(sw.stop('Test'))
3930.06706237793

reset()¶: Reset StopWatch.

stop(label='')¶

Stop StopWatch, print and return elapsed time.

Args:: label(str, optional): Optional label to prefix the output
Return:: Elapsed time in miliseconds (float)

utils.tsv module¶

Utils for reading and writing Tab-Seperated Variable (TSV) files.

The default tab delimiter can be overridden,: by passing a delimiter parameter to read and write.

>>> import utils.tsv
>>> data = [{'name': 'Alice', 'age': '20'}]
>>> file_name = '/tmp/data.tsv'
>>> utils.tsv.write(file_name, data)
>>> data2 = utils.tsv.read(file_name)
>>> data == data2
True

utils.tsv.read(file_name, delimiter='\t')¶

Read from xsv file, and output dict list.

Args:: file_name (str): File name delimiter (str, optional): delimiter
Returns:: TSV file contents as dict list

utils.tsv.write(file_name, dict_list, delimiter='\t')¶

Write dict_list to file.

Args:: file_name (str): File name dict_list (dict list): TSV content delimiter (str, optional): delimiter

utils.www module¶

Utils for reading remote files.

utils.www.read(url)¶

Read url.

Args:: url (str): URL
Return:: Contents at URL

utils.www.read_json(url)¶

Read JSON content from url.

Args:: url (str): URL
Return:: JSON parsed data at URL

utils.www.read_tsv(url)¶

Read TSV content from url.

Args:: url (str): URL
Return:: TSV parsed data at URL

Module contents¶

Implements simple extensions to the core python libraries.