Python equivalent of ignoreboth:erasedups

Question

I'm running iPython (Jupyter) through Anaconda, on a Mac Sierra, through iTerm, with $SHELL=bash - if I've missed any helpful set up details, just let me know.

I love the $HISTCONTROL aspect of bash, mentioned here. To sum that answer up: when traversing history (aka hitting the up arrow), it's helpful to remove duplicate entries so you don't scroll past the same command multiple times, and this is accomplished with $HISTCONTROL=ignoreboth:erasedups.

Is there any equivalent for this inside the Python interpreter (or iPython, specifically)? I have readline installed and feel like that's a good place to start, but nothing jumped out as obviously solving the problem, and I would've thought this was built in somewhere.

Answer 1

Through some deep-diving into IPython, sifting through poorly-explained and/or deprecated documentation, I've pieced together a solution that seems to work fine, though I'm sure it's not optimal for a number of reasons, namely:

• it runs a GROUP BY query on the history database every time I run a line in IPython
• it doesn't take care to clean up/coordinate the database tables - I only modify history, but ignore output_history and sessions tables

I put the following in a file (I named it dedupe_history.py, but name is irrelevant) inside $HOME/.ipython/profile_default/startup:

import IPython
import IPython.core.history as H
## spews a UserWarning about locate_profile() ... seems safe to ignore
HISTORY = H.HistoryAccessor()


def dedupe_history():
    query = ("DELETE FROM history WHERE rowid NOT IN "
        "(SELECT MAX(rowid) FROM history GROUP BY source)")
    db = HISTORY.db
    db.execute(query)
    db.commit()


def set_pre_run_cell_event():
    IPython.get_ipython().events.register("pre_run_cell", dedupe_history)

## dedupe history at start of new session - maybe that's sufficient, YMMV
dedupe_history()
## run dedupe history every time you run a command
set_pre_run_cell_event()