ENH: re-implement DataFrame.lookup.

[erfannariman]

DataFrame.lookup was deprecated in #35224 in 1.2. After some feedback (#39171 ) I opened this ticket to discuss re-implentation of lookup in a performant way. As mentioned in the discussion on 35244: "but it would have to be performant and not be yet another indexing api".

This ticket can be a starting point for proposed methods, although the old implementation was actually quite performant look at given tests in the discussion of 35244:

pandas/pandas/core/frame.py

Lines 3848 to 3861 in b5958ee

	if not self._is_mixed_type or n > thresh:
	values = self.values
	ridx = self.index.get_indexer(row_labels)
	cidx = self.columns.get_indexer(col_labels)
	if (ridx == -1).any():
	raise KeyError("One or more row labels was not found")
	if (cidx == -1).any():
	raise KeyError("One or more column labels was not found")
	flat_index = ridx * len(self.columns) + cidx
	result = values.flat[flat_index]
	else:
	result = np.empty(n, dtype="O")
	for i, (r, c) in enumerate(zip(row_labels, col_labels)):
	result[i] = self._get_value(r, c)

[challisd]

I think there should definitely be a lookup function. Since the old one seems to work well, is un-deprecating it an option? I find the proposed alternative using melt to be unreadable, and based on the (sadly heated) discussion here the old lookup function is faster than the melt alternative suggested. Pandas is a module used by many thousands of programmers and scientists who often have only a vague (or no) idea what the melt function does. The ability to run a quick series of lookups using lists of row and column coordinates is a fairly ordinary task, but if you don't provide this lookup function most users will likely fall back on using a slow for loop; and if that's too slow for them, decide to forget it and just use NumPy where you can do the_data[row_index_list, column_index_list]
Can we please keep this function?

[berkgercek]

Going to throw my voice in here and say that this is a pretty important feature for dataframes that allows for numpy-like behavior with labeled complex indexes and columns.

My personal use case for Pandas is often reliant on using it to keep labels and data together, and working with a method like lookup is a part of how I use it. It also fits in the scope of the package description provided in the documentation:

Intelligent label-based slicing, fancy indexing, and subsetting of large data sets

It's not something I use often, but the proposed solution linked in the deprecation notice is very inelegant.

One use case I have today is to do something similar to the following (obviously with meaningful data), which I have done with the above solution:

df1 = pd.DataFrame(np.random.normal(size=[1000, 3], columns=['A', 'B', 'C'])
df2 = pd.DataFrame(np.random.normal(size=[1000, 3], columns=['A', 'B', 'C'])
maxidx = df.idxmax(axis=1)
# The solution suggested by the deprecation notice is below.
idx, cols = pd.factorize(maxidx)
df2_lookup = pd.Series(df2.reindex(cols, axis=1).to_numpy()[np.arange(len(df2)), idx], index=df2.index)

This is not a readable solution for me and when others need to maintain this code it will be very much not obvious at a glance what I'm doing here.

If there is another solution that would work equally well but be more intuitive I am happy to use that instead, but I see no alternative to the .lookup method for this use case.

[rhshadrach]

+1 on re-implementing unless there is a more understandable alternative; I too find it hard to discern what the current alternative is doing.

[challisd]

Any word on if or when this feature will be added back in, or has anyone figured out a viable alternative?

[erfannariman]

Just to check, is there an agreement that this will be added back in if there's a viable PR before someone (or myself) starts to work on it. @jorisvandenbossche @mroeschke @rhshadrach