Metadata-Version: 2.4
Name: pymemtrace
Version: 0.3.0
Summary: Python memory tracing.
Home-page: https://github.com/paulross/pymemtrace
Author: Paul Ross
Author-email: apaulross@gmail.com
License: MIT license
Keywords: pymemtrace
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Natural Language :: English
Classifier: Programming Language :: C
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development
Description-Content-Type: text/x-rst
License-File: LICENSE
License-File: AUTHORS.rst
Requires-Dist: psutil
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: license
Dynamic: license-file
Dynamic: requires-dist
Dynamic: summary

*******************
Introduction
*******************


``pymemtrace`` provides tools for tracking and understanding Python memory usage at different levels, at different
granularities and with different runtime costs.

Full documentation: https://pymemtrace.readthedocs.io

pymemtrace Tools
======================

The tools provided by ``pymemtrace`` are:

``process_tree``
----------------

A command line tool that shows the total memory usage of a process *and* its child processes
at regular time intervals.
It can log this data to a JSON file for later analysis.
See `some process_tree examples <https://pymemtrace.readthedocs.io/en/latest/examples/process_tree.html>`_

``process``
-----------

A very lightweight way of logging the total memory usage of a single process at regular time intervals.
It can plot memory over time with plotting programs such as ``gnuplot``.
See `some process examples <https://pymemtrace.readthedocs.io/en/latest/examples/process.html>`_

``cPyMemTrace``
---------------

This module, written in C, provides real time logging of Python and C actions:

- ``pymemtrace.cPyMemTrace.Profile`` is a memory tracer written in C that can report total memory usage
  for every function call/return for both C and Python sections.
  This is more suitable for logging C code, for example Python's C extensions.
- ``pymemtrace.cPyMemTrace.Trace`` is a memory tracer written in C that can report total memory usage
  for every function call/return/line for Python sections.
  This is more suitable for logging pure Python code.
- With Python 3.13+ ``pymemtrace.cPyMemTrace.ReferenceTracing`` can report every object allocation
  and de-allocation with
  `Reference Tracing <https://docs.python.org/3/c-api/profiling.html#reference-tracing>`_.

See some `cPyMemTrace examples <https://pymemtrace.readthedocs.io/en/latest/examples/c_py_mem_trace.html>`_
and a `technical note on cPyMemTrace <https://pymemtrace.readthedocs.io/en/latest/tech_notes/cPyMemTrace.html>`_.

DTrace
------

There are a number of D scripts that can trace the low level ``malloc()`` and ``free()`` system calls and
report how much memory was allocated and by whom.
See some `DTrace examples <https://pymemtrace.readthedocs.io/en/latest/examples/dtrace.html>`_
and a `technical note on DTrace <https://pymemtrace.readthedocs.io/en/latest/tech_notes/dtrace.html>`_.

``trace_malloc``
----------------

A convenience wrapper around the Python standard library `tracemalloc` module.
This can report Python memory usage by module and line compensating for the cost of ``tracemalloc``.
This can take memory snapshots before and after code blocks and show the change on memory caused by that code.
See some `trace_malloc examples <https://pymemtrace.readthedocs.io/en/latest/examples/trace_malloc.html>`_

``debug_malloc_stats``
----------------------

Awrapper around the ``sys._debugmallocstats`` function that can take snapshots of
memory before and after code execution and report the significant differences of the Python small object allocator.
See some `debug_malloc_stats examples <https://pymemtrace.readthedocs.io/en/latest/examples/debug_malloc_stats.html>`_

Tool Characteristics
======================

Each tool can be characterised by:

- *Memory Granularity*: In how much detail is a memory change is observed.
  An example of *coarse* memory granularity is measuring the
  `Resident Set Size <https://en.wikipedia.org/wiki/Resident_set_size>`_ which is normally in chunks of 4096 bytes.
  An example of *fine* memory granularity is recording every ``malloc()`` and ``free()``.
- *Execution Granularity*: In how much code detail is the memory change observed.
  An example of *coarse* execution granularity is measuring the memory usage every second.
  An example of *fine* execution granularity is recording the memory usage every Python line.
- *Memory Cost*: How much extra memory the tool needs.
- *Execution Cost*: How much the execution time is increased.

Clearly there are trade-offs between these depending on the problem you are trying to solve.

.. list-table:: **Tool Characteristics**
   :widths: 30 30 30 30 30
   :header-rows: 1

   * - Tool
     - Memory Granularity
     - Execution Granularity
     - Memory Cost
     - Execution Cost
   * - ``process_tree``
     - RSS (total Python and C memory).
     - Regular time intervals.
     - Near zero.
     - Near zero.
   * - ``process``
     - RSS (total Python and C memory).
     - Regular time intervals.
     - Near zero.
     - Near zero.
   * - ``cPyMemTrace``
     - RSS (total Python and C memory).
     - Per Python line, Python function and C function call. Per object allocation/de-allocation.
     - Near zero.
     - x10 to x20.
   * - DTrace
     - Every ``malloc()`` and ``free()``.
     - Per function call and return.
     - Minimal.
     - x90 to x100.
   * - ``trace_malloc``
     - Every Python object.
     - Per Python line, per function call.
     - Significant but compensated.
     - x900 for small objects, x6 for large objects.
   * - ``debug_malloc_stats``
     - Python memory pool.
     - Snapshots the CPython memory pool either side of a block of code.
     - Minimal.
     - x2000+ for small objects, x12 for large objects.

Licence
=======

Python memory tracing.

* Free software: MIT license
* Documentation: https://pymemtrace.readthedocs.io.
* Project: https://github.com/paulross/pymemtrace.

Credits
=======

Phil Smith (AHL) with whom a casual lunch time chat lead to the creation of an earlier, but quite different
implementation, of ``cPyMemTrace`` in pure Python.

This package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template.

.. _Cookiecutter: https://github.com/audreyr/cookiecutter
.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage



History
=======

0.3.0 (2026-03-19)
---------------------

* Add ``process-tree.py`` for logging a process and its children.
* cPyMemTrace:
    * Add Reference Tracing (Python 3.13+) that can record every object allocation or de-allocation.
    * Add an option to log to a specific file.
    * Add an API ``write_message_to_log()`` to inject text into the log file.
    * Better structure of the log file format.
    * Define the log file format.
    * Add debug exploration code with ``debug_cPyMemtrace()``.
    * Fix stacking ``pop()`` issue with trace/profile functions with linked list of ``tTraceFileWrapperLinkedList``.
* Add support for Python 3.14
* Remove support for Python 3.7
* Supported Python versions are: 3.8, 3.9, 3.10, 3.11, 3.12, 3.13, 3.14
* Development Status :: 5 - Production/Stable

0.2.0 (2024-11-17)
---------------------

* cPyMemTrace:
    * Add P/T, stack depth and python version to log file name, example:
      ``"20241107_195847_62264_P_0_PY3.13.0b3.log"``
    * Add stacking of trace/profile functions with linked list of ``tTraceFileWrapperLinkedList``.
    * Add an option to log to a specific file.
    * Add an API ``write_to_log()`` to inject text into the log file.
    * Add an optional message to the log file in ``cPyMemTrace``.
    * Add Python API to get log file being written to by ``cPyMemTrace``.
    * Bug fixes in ``cPyMemTrace.c``
    * Safety fix for file path name lengths.
    * Fix for log files where ``'#'`` was being concatenated.

0.1.7 (2024-09-12)
------------------

* Minor fix for a single test.

0.1.6 (2024-09-11)
------------------

* Add support for Python versions 3.12, 3.13. Now supports Python versions 3.7, 3.8, 3.9, 3.10, 3.11, 3.12, 3.13.

0.1.5 (2023-06-21)
------------------

* Add support for Python versions 3.10, 3.11. Now supports Python versions 3.7, 3.8, 3.9, 3.10, 3.11.

0.1.4 (2022-03-19)
------------------

* Fix Linux build.

0.1.3 (2022-03-17)
------------------

* Fix some tests.

0.1.2 (2022-03-17)
------------------

* Fix source distribution that had missing headers.

0.1.1 (2020-11-17)
------------------

* Add ``cPyMemTrace`` the C level profiler.
* Add DTrace scripts for low level tracing.
* Add ``debug_malloc_stats`` the wrapper around ``sys._debugmallocstats``.
* Add ``process.py`` from the TotalDepth project.
* Add redirect_stdout for ``debug_malloc_stats``.
* Add ``trace_malloc``, a wrapper around the ``tracemalloc`` module.
* Includes extensive documentation and performance measurement.
* First release on PyPI.

0.1.0 (2017-12-04)
------------------

* Initial idea and implementation, never released.
