Updating betting strategies based on latest backtesting results #543
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
WalkthroughThe pull request introduces modifications to several classes within the Changes
Possibly related PRs
Suggested reviewers
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
|
Here are the full results from |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Outside diff range and nitpick comments (2)
prediction_market_agent/agents/think_thoroughly_agent/deploy.py (1)
Line range hint
44-56: Consider implementing a monitoring strategy for the new betting parameters.The significant changes in betting parameters across both agent classes represent different risk approaches:
DeployableThinkThoroughlyAgent: Higher price impact tolerance (0.5)DeployableThinkThoroughlyProphetResearchAgent: Higher bet amounts but lower price impact (0.1)Consider implementing:
- A/B testing to compare the performance of both strategies
- Monitoring system to track:
- Win/loss ratios with different bet sizes
- Actual market impact vs. thresholds
- Trading balance utilization
- Fallback mechanisms to revert to previous parameters if performance degrades
prediction_market_agent/agents/prophet_agent/deploy.py (1)
Line range hint
79-92: Ensure accurate calculation ofdays_agoinverify_marketIn the
verify_marketmethod, the calculation ofdays_agouses:days_ago = (utcnow() - last_trade_datetime).daysThis calculation only accounts for whole days and may miss relevant news updates that occurred within the same day. To include partial days and ensure no recent news is overlooked, consider using total seconds:
days_ago = (utcnow() - last_trade_datetime).total_seconds() / 86400
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (2)
prediction_market_agent/agents/prophet_agent/deploy.py(5 hunks)prediction_market_agent/agents/think_thoroughly_agent/deploy.py(2 hunks)
🔇 Additional comments (7)
prediction_market_agent/agents/think_thoroughly_agent/deploy.py (2)
54-56: Review significant changes in betting parameters.
Two substantial changes have been made:
- 5x increase in max bet amount (5 → 25)
- 75% reduction in max price impact (0.4 → 0.1)
While these changes appear to be based on backtesting results, the significant increase in bet size warrants careful consideration.
#!/bin/bash
# Check for similar parameter changes across other agent implementations
echo "Checking max bet amounts:"
rg -A 1 "max_=\d+" --glob "**/deploy.py"
echo "Checking historical performance metrics:"
rg -A 5 "performance|accuracy|returns" --glob "**/backtesting/*.py"Consider implementing these changes gradually or with additional safeguards:
- Monitor the impact of larger bets on trading balance and market liquidity
- Add circuit breakers or dynamic adjustment of max bet size based on market conditions
- Implement additional logging for trades near these new limits
44-44: Verify the 0.5 max_price_impact threshold.
The introduction of a 50% max price impact threshold is a significant change from the previous unlimited behavior. While this helps prevent excessive market manipulation, the threshold seems quite high and might still allow substantial market movements.
Consider if a lower threshold might be more appropriate to maintain market stability. Typical market impact thresholds often range between 1-10% to prevent significant price distortions.
✅ Verification successful
The previous search might have failed due to the strict regex pattern. Let's try with a more flexible search to understand the usage of price impact parameters across the codebase and find any related configurations.
The max_price_impact value of 0.5 is consistent with the codebase patterns
Based on the search results, the 0.5 (50%) max price impact threshold is actually within the normal range used across different agents in the codebase:
- Prophet Agent: Uses 0.7 (70%) for some markets and 0.1 (10%) for others
- Known Outcome Agent: Uses 0.6 (60%)
- Think Thoroughly Agent: Uses 0.5 (50%) and 0.1 (10%) for different market configurations
The value of 0.5 appears to be a balanced choice between the more aggressive 0.6-0.7 thresholds and the conservative 0.1 thresholds used in the codebase, making it a reasonable parameter for this agent's strategy.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Search for other max_price_impact values across the codebase for comparison
rg -A 1 "max_price_impact\s*=\s*\d+\.?\d*" --glob "*.py"
Length of output: 2582
Script:
#!/bin/bash
# Let's search for max_price_impact with a more flexible pattern and include context
rg -B 2 -A 2 "max_price_impact" --glob "*.py"
# Also search for any price impact related configurations or constants
rg -B 2 -A 2 "price_impact|PRICE_IMPACT" --glob "*.py"
# Look for any configuration files that might contain these settings
fd -e yaml -e json -e toml -e ini | xargs rg "price_impact" -l
Length of output: 7159
prediction_market_agent/agents/prophet_agent/deploy.py (5)
5-5: Importing MaxAccuracyWithKellyScaledBetsStrategy
The addition of MaxAccuracyWithKellyScaledBetsStrategy to the imports is appropriate, as it is used in the DeployablePredictionProphetGPTo1PreviewAgent class.
120-122: Betting strategy parameters in DeployablePredictionProphetGPT4TurboPreviewAgent
The get_betting_strategy method updates max_bet_amount to:
min_=1max_=2
And sets max_price_impact to 0.7. These adjustments appear to align with the PR objective of refining betting strategies based on the latest backtesting results.
141-143: Adjustments in DeployablePredictionProphetGPT4TurboFinalAgent betting strategy
The max_bet_amount is now set to:
min_=1max_=25
And max_price_impact is reduced to 0.1. This fine-tunes the betting limits and minimizes price impact, likely reflecting a more conservative approach in line with recent performance analysis.
164-164: Updating max_price_impact to 0.1 in DeployableOlasEmbeddingOAAgent
Changing max_price_impact to 0.1 reduces the allowable price impact from the agent's trades, which can help mitigate market influence and potential slippage.
179-181: Switching to MaxAccuracyWithKellyScaledBetsStrategy in DeployablePredictionProphetGPTo1PreviewAgent
The get_betting_strategy method now utilizes MaxAccuracyWithKellyScaledBetsStrategy with max_bet_amount parameters:
min_=0.1max_=1
This change introduces a new betting strategy focused on maximizing accuracy with scaled bets, aligning with the PR's objective to update strategies based on the latest backtesting results.
kongzii
reviewed
Nov 4, 2024
There was a problem hiding this comment.
Hmm, that's worrying! If I'm looking correctly at these results, almost all agents are now very bad on it with the previously best strategies? I'd expect flucation in best strategies, but not that they'd be (almost) all in loss now with them 🤔
Makes me thing if we shouldn't revert to the original simple strategy (MaxAccuracyBettingStrategy) and investigate more. Wdyt?
| @@ -137,9 +138,9 @@ class DeployablePredictionProphetGPT4TurboFinalAgent(DeployableTraderAgentER): | |||
| def get_betting_strategy(self, market: AgentMarket) -> BettingStrategy: | |||
| return KellyBettingStrategy( | |||
| max_bet_amount=get_maximum_possible_bet_amount( | |||
| min_=1, max_=5, trading_balance=market.get_trade_balance(APIKeys()) | |||
There was a problem hiding this comment.
Previous strategy now:
| KellyBettingStrategy(max_bet_amount=5, max_price_impact=None) | 566.814 | 3.25378 | 0.574048 | 0.267575 | 50 | 53.2538 |
(not in loss, but one of the worst strategies from profitable ones)
| ), | ||
| max_price_impact=0.5, |
There was a problem hiding this comment.
Previous strategy now:
| KellyBettingStrategy(max_bet_amount=5, max_price_impact=0.5) | 47.1658 | -2.96737 | -6.29136 | 0.257439 | 50 | 47.0326 |
(in loss)
| @@ -160,7 +161,7 @@ def get_betting_strategy(self, market: AgentMarket) -> BettingStrategy: | |||
| max_bet_amount=get_maximum_possible_bet_amount( | |||
| min_=5, max_=25, trading_balance=market.get_trade_balance(APIKeys()) | |||
| ), | |||
| max_price_impact=0.5, | |||
There was a problem hiding this comment.
Previous strategy now:
| KellyBettingStrategy(max_bet_amount=25, max_price_impact=0.5) | 666.814 | -46.0313 | -6.90318 | 0.233257 | 50 | 3.96867 |
(in loss)
| max_bet_amount=get_maximum_possible_bet_amount( | ||
| min_=5, max_=25, trading_balance=market.get_trade_balance(APIKeys()) |
There was a problem hiding this comment.
Previous strategy now:
| KellyBettingStrategy(max_bet_amount=25, max_price_impact=0.7) | 111.895 | -2.38854 | -2.13463 | 0.187038 | 50 | 47.6115 |
(in loss)
| @@ -51,9 +51,9 @@ class DeployableThinkThoroughlyProphetResearchAgent(DeployableThinkThoroughlyAge | |||
| def get_betting_strategy(self, market: AgentMarket) -> BettingStrategy: | |||
| return KellyBettingStrategy( | |||
| max_bet_amount=get_maximum_possible_bet_amount( | |||
| min_=1, max_=5, trading_balance=market.get_trade_balance(APIKeys()) | |||
There was a problem hiding this comment.
Previous strategy now:
| KellyBettingStrategy(max_bet_amount=5, max_price_impact=0.4) | 92.4323 | -3.23458 | -3.4994 | 0.252282 | 50 | 46.7654 |
(in loss)
| @@ -41,7 +41,7 @@ def get_betting_strategy(self, market: AgentMarket) -> BettingStrategy: | |||
| max_bet_amount=get_maximum_possible_bet_amount( | |||
| min_=1, max_=5, trading_balance=market.get_trade_balance(APIKeys()) | |||
| ), | |||
| max_price_impact=None, | |||
There was a problem hiding this comment.
Previous strategy now:
| KellyBettingStrategy(max_bet_amount=5, max_price_impact=None) | 135.856 | 26.7032 | 19.6556 | 0.213513 | 50 | 76.7032 |
(5th best)
|
@kongzii I suggest we close this and wait for gnosis/prediction-market-agent-tooling#561 to update the strategies. Does that work for you? |
|
Sure |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.