feat: enhance pool storage with token addresses and fees #528

fengtality · 2025-10-14T21:41:39Z

Change 1: Reformat Pools

Sumamry

Enhanced pool storage format to include authoritative on-chain data (token addresses and fees) from pool-info endpoints. This ensures stored pool data always matches actual on-chain state and eliminates runtime lookups.

Changes

Core Pool Storage Refactoring

Pool Interface: Added baseTokenAddress, quoteTokenAddress, and feePct fields to Pool type
POST /pools Route: Refactored to automatically fetch pool-info and extract token addresses/fees
Validation: Enhanced pool validation to require new fields and validate fee percentages
Helper Functions: Created pool-info-helpers.ts with connector-agnostic pool info fetching

Pool Template Migration

Created comprehensive migration script (migrate-pool-templates.ts)
Raydium: Migrated 14/40 pools (26 failed - stale/deprecated pools)
Meteora: Migrated 3/3 pools (100% success)
Uniswap: Migrated 33/43 pools (10 failed - stale/deprecated pools)
All migrated pools now include authoritative token addresses and fees

Pancakeswap Pool Addition

Added 4 new Pancakeswap pools with enhanced format:
- ASTER/USDT (CLMM V3) - 0.25% fee
- USDT/WBNB (CLMM V3) - 0.01% fee
- CAKE/USDT (CLMM V3) - 0.25% fee
- WBNB/BUSD (AMM V2) - 0.25% fee
Created add-pancakeswap-pools.ts script for automated pool addition

BSC Token Addition

Added 3 missing tokens to bsc.json:
- ASTER (Aster - 0x000Ae314E2A2172a039B26378814C252734f556A)
- Cake (PancakeSwap Token - 0x0E09FaBB73Bd3Ade0a17ECC321fD13a19e81cE82)
- BUSD (BUSD Token - 0xe9e7CEA3DedcA5984780Bafc599bD69ADd087D56)
Created add-bsc-tokens.ts script for automated token metadata fetching

Documentation

CLAUDE.md: Added comprehensive Pool Storage Format section
README.md: Added Pool Management Routes and migration documentation
Planning Docs: Created POOL_STORAGE_REFACTOR_PLAN.md and POOL_MIGRATION_OUTSTANDING.md

New Pool Format

{
  "type": "amm",
  "network": "mainnet-beta",
  "baseSymbol": "SOL",
  "quoteSymbol": "USDC",
  "baseTokenAddress": "So11111111111111111111111111111111111111112",
  "quoteTokenAddress": "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v",
  "feePct": 0.25,
  "address": "58oQChx4yWmvKdwLLZzBi4ChoCc2fqCUWBkwMihLYQo2"
}

Change 2:Improve Gas Pricing for Ethereum Networks

Summary

Adds support for configurable EIP-1559 gas parameters (maxFeePerGas and maxPriorityFeePerGas) for Ethereum and EVM-compatible networks. Users can now control transaction costs by setting custom gas values or dynamically using network-fetched prices.
Added Etherscan API support for more accurate gas prices for certain networks

Problem

Previously, Gateway always fetched gas prices from the network at transaction time, ignoring any configured gas price parameters for
EIP-1559 networks. This made it impossible to:

Set predictable, low-cost gas prices for development/testing
Control maximum transaction costs in production
Override high network fees during congestion

Solution

1. Configuration Layer

Added maxFeePerGas and maxPriorityFeePerGas config parameters to all EIP-1559 network templates
Updated JSON schema and TypeScript interfaces to support new parameters
Both parameters are optional - if not set, Gateway fetches from network

Example configuration (conf/chains/ethereum/base.yml):

  chainID: 8453
  nodeURL: https://mainnet.base.org
  nativeCurrencySymbol: ETH
  minGasPrice: 0.1
  # EIP-1559 gas parameters (in GWEI)
  # If not set, will fetch from network
  maxFeePerGas: 0.1
  maxPriorityFeePerGas: 0.01

2. Gas Estimation (estimateGasPrice)

EIP-1559 networks: Always fetches and logs network values, then uses configured values if set or falls back to network
Legacy networks: Fetches network gas price, logs it, and applies minGasPrice floor
Caches gas data with isEIP1559 flag for 10 seconds

Example logs showing transparency:
Network EIP-1559 fees: baseFee=0.0012 GWEI, maxFee=1.5043 GWEI, priority=0.0001 GWEI
Using configured EIP-1559 fees: maxFee=0.1 GWEI, priority=0.01 GWEI
Estimated: 0.1 GWEI for network base

3. Transaction Preparation (prepareGasOptions)

EIP-1559 networks: Returns Type 2 transaction with maxFeePerGas and maxPriorityFeePerGas
Legacy networks: Returns Type 0 transaction with single gasPrice
No network fetching at transaction time - uses cached values from estimateGasPrice()

4. API Response Enhancement

Updated /chains/ethereum/estimate-gas endpoint to return:
{
"feePerComputeUnit": 0.1,
"denomination": "gwei",
"computeUnits": 300000,
"feeAsset": "ETH",
"fee": 0.00003,
"timestamp": 1760553074872,
"gasType": "eip1559",
"maxFeePerGas": 0.1,
"maxPriorityFeePerGas": 0.01
}

Network Support

EIP-1559 Networks (Type 2 transactions):

Ethereum Mainnet (chainID: 1)
Base (chainID: 8453)
Polygon (chainID: 137)
Arbitrum (chainID: 42161)
Optimism (chainID: 10)

Legacy Networks (Type 0 transactions):

BSC (chainID: 56)
Avalanche (chainID: 43114)
Celo (chainID: 42220)
Sepolia (chainID: 11155111)

Documentation

Added comprehensive GAS_CONFIGURATION_GUIDE.md covering:

EIP-1559 vs Legacy gas models explained
Parameter definitions and use cases
Configuration strategies (dynamic, fixed, hybrid)
Network-specific examples
Troubleshooting guide
Best practices

This commit refactors pool storage to include authoritative on-chain data: - Add baseTokenAddress, quoteTokenAddress, and feePct to pool objects - POST /pools now fetches pool-info to get token addresses and fees - Update validation to require new fields - Create migration script for existing pool templates - Migrate Raydium (14 pools), Meteora (3 pools), and Uniswap (33 pools) - Add Pancakeswap pools (4 pools) with new format - Add missing BSC tokens (ASTER, CAKE, BUSD) - Update documentation (CLAUDE.md, README.md) The new pool format ensures stored data matches actual on-chain state, eliminating runtime lookups and preventing token ordering mismatches. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Add support for configurable EIP-1559 gas parameters (maxFeePerGas and maxPriorityFeePerGas) for Ethereum and EVM-compatible networks. This allows users to control transaction costs by setting custom gas values or using network-fetched prices. Changes: - Add maxFeePerGas and maxPriorityFeePerGas config parameters - Separate EIP-1559 (Type 2) and legacy (Type 0) transaction handling - Enhance gas estimation with network fee logging and comparison - Update estimate-gas API to return gasType and EIP-1559 details - Add comprehensive gas configuration guide Networks affected: - EIP-1559: mainnet, base, polygon, arbitrum, optimism - Legacy: bsc, avalanche, celo, sepolia 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Updated test/pools/pools.routes.test.ts to work with the enhanced Pool interface that now includes baseTokenAddress, quoteTokenAddress, and feePct fields. Changes: - Added required fields (baseTokenAddress, quoteTokenAddress, feePct) to all mock Pool objects - Updated POST /pools test expectation to use expect.objectContaining() since the route now fetches and adds pool info automatically - Removed outdated planning documents (POOL_MIGRATION_OUTSTANDING.md, POOL_STORAGE_REFACTOR_PLAN.md) All 13 tests now passing. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

- Replace per-network scanAPIKey with single etherscanAPIKey in ethereum.yml - Migrate from V1 to V2 API endpoint (https://api.etherscan.io/v2/api) - Add automatic chain support detection (Ethereum, Polygon, BSC verified) - Implement chainId-based service initialization instead of network names - Add comprehensive test suite with 17 tests covering API calls and error handling - Update gas configuration guide with Etherscan V2 integration details - Gracefully fall back to RPC for unsupported chains (Base, Arbitrum, Optimism) This change provides more accurate EIP-1559 gas estimates using Etherscan's gastracker module across all supported chains with a single API key. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

src/chains/ethereum/ethereum.ts

- Replace per-network scanAPIKey references with unified etherscanAPIKey - Clarify that etherscanAPIKey is set once in ethereum.yml, not per network - Document which chains support Etherscan gastracker (Ethereum, Polygon, BSC) - Add explicit notes about unsupported chains (Base, Arbitrum, Optimism) - Update all examples and troubleshooting sections - Fix minGasPrice default description to reference network templates - Update API key setup guide with V2 approach - Add Etherscan V2 API migration guide to references 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

- Update Base/Optimism/Arbitrum examples to use 0.01 GWEI (from 0.1) - Update BSC example to use 3.0 GWEI for legacy transactions - Update all log examples to reflect actual configured values - Clarify network-specific examples in configuration strategies - Update API response examples with correct gas values - Add notes explaining minGasPrice floor behavior for each network type 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

- Change Polygon minGasPrice from 30 to 10 GWEI - Update maxFeePerGas and maxPriorityFeePerGas to 10 GWEI - Update nodeURL to rpc.ankr.com/polygon - Update nativeCurrencySymbol from MATIC to POL 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Remove legacy pool lookup code from CLMM open-position routes to align with the existing schema requirement that poolAddress is mandatory. Changes: - Remove unused baseTokenSymbol and quoteTokenSymbol parameters from Raydium CLMM openPosition function - Remove dead pool lookup code (12 lines) from Raydium implementation - Remove misleading baseToken/quoteToken examples from Uniswap schema - Update Uniswap poolAddress example to show real address This is a non-breaking change since the schema already requires poolAddress. All existing tests pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

…itions Update minGasPrice values to reflect current network requirements: - BSC: 0.1 → 3 GWEI - Polygon: 0.1 → 10 GWEI (align with conf template) These values match the actual network conditions and align with the values already documented in the configuration templates. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Move getPool querystring schema to src/pools/schemas.ts and use Type.String with enum property instead of inline definition to enable proper dropdown selection in Swagger UI. Changes: - Add GetPoolRequestSchema to src/pools/schemas.ts - Use enum: ['amm', 'clmm'] for type field to create dropdown - Import and use schema in getPool route This allows users to select pool type from a dropdown in Swagger UI instead of having the value hardcoded. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Update GetPoolRequestSchema to default to 'mainnet-beta' instead of 'mainnet' to align with Solana connector defaults (raydium, meteora). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

- Explicitly override network default to 'mainnet-beta' in querystring - Change tradingPair examples to show SOL-USDC first instead of ETH-USDC - Ensure Swagger UI displays correct defaults for Solana connectors 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

nikspz · 2025-10-16T21:11:32Z

commit 4374adb34541f4198ab52e52c296b9ff3fddedd6 + commit 75beff7

cloned and installed PR7816 and PR528
set up gateway generate certs and pnpm start —passphrase —dev
review gateway ONLINE
- ethereum
  - gateway connect ethereum: ok
  - gateway balance: ok
  - gateway config ethereum update
    - updated defaultNetwork to base
  - gateway pool uniswap/amm WETH-USDC: showed pool ✅https://app.uniswap.org/explore/tokens/base/0x88a43bbdf9d098eec7bceda4e2494615dfd9bb9c
    - === Pool Information === Connector: uniswap/amm Trading Pair: WETH-USDC Pool Type: amm Network: base Base Token: WETH Quote Token: USDC Base Token Address: 0x4200000000000000000000000000000000000006 Quote Token Address: 0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913 Fee: 0.3% Pool Address: 0x88A43bbDF9D098eEC7bCEda4e2494615dfD9bB9C
  - gateway pool uniswap/clmm WETH-USDC: showed pool ✅https://app.uniswap.org/explore/pools/base/0xd0b53d9277642d899df5c87a3966a349a798f224
    - === Pool Information === Connector: uniswap/clmm Trading Pair: WETH-USDC Pool Type: clmm Network: base Base Token: WETH Quote Token: USDC Base Token Address: 0x4200000000000000000000000000000000000006 Quote Token Address: 0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913 Fee: 0.05% Pool Address: 0xd0b53d9277642d899df5c87a3966a349a798f224
  - gateway lp uniswap/clmm add-liquidity: added successfully ✅(https://basescan.org/tx/0x8e08c53a017e2eb0da50d90584c189c548ade216a35b82a6898131fe6775648a)
    - Using cached EIP-1559 pricing: maxFee=0.01 GWEI, priority=0.01 GWEI ✅
  - gateway lp uniswap/clmm remove-liquidity ❌
    - WETH-USDC
    - Selected position (correct 4061553) matched with uniswap position number ✅
    - Current Price wrong ❌
    - Status: In Range: wrong ❌
    - Holdings: wrong ❌ not matched with current position
  - forced 100 % of gateway lp uniswap/clmm remove-liquidity
    - got error AttributeError: 'NoneType' object has no attribute 'split' ❌
    - position not removed ❌

logs_hummingbot.log

            2025-10-17 04:00:29,614 - 29890 - hummingbot.client.hummingbot_application - INFO - Found pool for WETH-USDC: 0xd0b53d9277642d899df5c87a3966a349a798f224 | Base: WETH (0x4200000000000000000000000000000000000006) | Quote: USDC (0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913) | Type: CLMM | Fee: 0.05%
            2025-10-17 04:00:53,553 - 29890 - hummingbot.core.utils.async_utils - ERROR - Unhandled error in background task: 'NoneType' object has no attribute 'split'
            Traceback (most recent call last):
              File "/home/etozhe/7816pool/hummingbot/core/utils/async_utils.py", line 9, in safe_wrapper
                return await c
                       ^^^^^^^
              File "/home/etozhe/7816pool/hummingbot/connector/gateway/gateway_lp.py", line 400, in _clmm_close_position
                self.start_tracking_order(order_id=order_id,
              File "/home/etozhe/7816pool/hummingbot/connector/gateway/gateway_base.py", line 486, in start_tracking_order
                GatewayInFlightOrder(
              File "/home/etozhe/7816pool/hummingbot/connector/gateway/gateway_in_flight_order.py", line 47, in __init__
                self._fee_asset = trading_pair.split("-")[0]  # defaults to base asset
                                  ^^^^^^^^^^^^^^^^^^
            AttributeError: 'NoneType' object has no attribute 'split'

…ate pairs Update POST /pools route to include all pool schema fields and allow multiple pools with the same trading pair but different addresses. Changes: - Add baseTokenAddress, quoteTokenAddress, feePct as optional fields in PoolAddRequestSchema (fetched from pool-info if not provided) - Update PoolAddRequest interface to include new optional fields - Remove duplicate token pair check from PoolService.addPool - Only prevent duplicate pool addresses, not duplicate pairs - Update addPool route to use provided fields or fetch from pool-info - Set mainnet-beta as default network in PoolAddRequestSchema - Include fee percentage in success messages This allows users to add multiple pools for the same token pair with different fee tiers (e.g., JUP-USDC with 0.01%, 0.25%, 1% fees). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

…T /pools Change baseTokenAddress and quoteTokenAddress from optional to required fields in the add pool endpoint to ensure data consistency. Changes: - Remove Type.Optional wrapper from baseTokenAddress and quoteTokenAddress in PoolAddRequestSchema - Update PoolAddRequest interface to make these fields required (not optional) - Simplify addPool route logic since token addresses are now always provided - Only fetch pool-info if feePct is missing This ensures all pools have validated token addresses and prevents inconsistencies in pool data. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Add metadata-based duplicate detection to prevent multiple pools with the same characteristics (type, network, feePct, token addresses) but different addresses. Changes: - Add getPoolByMetadata method to PoolService to find pools by metadata instead of just address - Update addPool route to check for existing pools with matching metadata - If pool with same metadata exists but different address: remove old pool and add new one (replacement) - If pool with same metadata and address exists: update it - If no matching metadata: add as new pool This ensures the pool database doesn't contain duplicate entries with identical trading characteristics but different addresses. Example: Prevents having two JUP-USDC CLMM 0.01% pools on mainnet-beta with different addresses. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Change pool duplicate detection to match only on type, network, and token addresses (baseTokenAddress, quoteTokenAddress), ignoring feePct. When a new pool is added with the same token pair but different fee tier, it replaces the existing pool. Changes: - Remove feePct parameter from getPoolByMetadata method - Match pools based only on type, network, and token pair - When same token pair with different fee tier is added: replace old pool - When same token pair with same address is added: update fee tier - Updated success messages to show old and new fee tiers on replacement This ensures only one pool per token pair (per type/network) exists, preventing duplicate entries with different fee tiers. Example: Adding JUP-USDC 0.25% will replace existing JUP-USDC 0.01% 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Update pool routes tests to match new required fields (baseTokenAddress, quoteTokenAddress) and new pool duplicate detection logic. Changes: - Add getPoolByMetadata and updatePool to mock PoolService - Update POST /pools test payloads to include required baseTokenAddress and quoteTokenAddress fields - Add feePct to test payloads - Update "duplicate pool" test to "update existing pool" test since duplicate addresses now trigger updates instead of errors - Add proper mocks for getPoolByMetadata and getPoolByAddress All 13 pool routes tests now pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

The third parameter to sendAndConfirmTransaction was being interpreted as priority fee per compute unit (lamports/CU) instead of total compute units, resulting in an excessive 7.838 SOL transaction fee (400000 lamports/CU × 19595 CU) that caused transactions to fail. Now uses auto-estimated priority fee like other Meteora operations. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

fengtality changed the base branch from main to development October 14, 2025 21:42

fengtality mentioned this pull request Oct 14, 2025

feat: align client pool commands with Gateway PR #528 changes hummingbot/hummingbot#7816

Open

rapcmia requested a review from nikspz October 15, 2025 13:42

rapcmia added this to Pull Request Board Oct 15, 2025

rapcmia moved this to Backlog in Pull Request Board Oct 15, 2025

fengtality and others added 3 commits October 15, 2025 11:58

github-advanced-security bot found potential problems Oct 15, 2025

View reviewed changes

src/chains/ethereum/ethereum.ts Dismissed Show dismissed Hide dismissed

fengtality and others added 8 commits October 15, 2025 13:10

fengtality and others added 5 commits October 16, 2025 14:29

rapcmia moved this from Backlog to Under Review in Pull Request Board Oct 17, 2025

rapcmia mentioned this pull request Oct 17, 2025

Raydium CLMM - Closing position endpoint fails due to transaction failed to confirm #525

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

feat: enhance pool storage with token addresses and fees #528

feat: enhance pool storage with token addresses and fees #528

Uh oh!

fengtality commented Oct 14, 2025 •

edited

Loading

Uh oh!

Uh oh!

nikspz commented Oct 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

feat: enhance pool storage with token addresses and fees #528

Are you sure you want to change the base?

feat: enhance pool storage with token addresses and fees #528

Uh oh!

Conversation

fengtality commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change 1: Reformat Pools

Sumamry

Changes

Core Pool Storage Refactoring

Pool Template Migration

Pancakeswap Pool Addition

BSC Token Addition

Documentation

New Pool Format

Change 2:Improve Gas Pricing for Ethereum Networks

Summary

Problem

Solution

1. Configuration Layer

2. Gas Estimation (estimateGasPrice)

3. Transaction Preparation (prepareGasOptions)

4. API Response Enhancement

Network Support

EIP-1559 Networks (Type 2 transactions):

Legacy Networks (Type 0 transactions):

Documentation

Uh oh!

Uh oh!

nikspz commented Oct 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

fengtality commented Oct 14, 2025 •

edited

Loading