I have a dataset similar to investment recommendation platforms where users submit BUY/SELL suggestions for different stocks. I want to build a system that evaluates how accurate these predictions turn out to be.
My goal is to:
Rate each recommendation on a scale from 1-5 based on actual stock performance
Track which users make better predictions in different market conditions
Use machine learning to predict future recommendation accuracy
I think some users perform better during bull markets while others excel in bear markets. Some might be better at identifying short opportunities versus long positions.
My plan is to create evaluation screens where I can manually rate recommendations based on actual stock performance over specific time periods (like 90 days). This would create training data for the ML model.
The main challenge: I want to build a machine learning system that can identify patterns and predict the likelihood that new recommendations will be accurate. The system should consider both individual user history and market conditions.
My background: I’m completely new to AI and machine learning. I’m looking for guidance on which algorithms to explore, similar projects I could study, or educational resources to get started.
I’d like to implement this using F# and hopefully create something impressive for my portfolio. Any suggestions on where to begin would be greatly appreciated!
In my experience building prediction systems for financial data, feature engineering is crucial before diving into algorithms. Your current features seem basic; consider incorporating technical indicators such as RSI, moving averages, and volume patterns relative to the timing of recommendations.
Be cautious with a uniform 90-day evaluation; momentum-driven trades can resolve in weeks while value-oriented picks may take months. I learned this through instances where my model unfairly penalized effective long-term calls.
For F#, utilize Deedle for data manipulation in conjunction with ML.NET, as functional programming suits backtesting time series well.
Implementing walk-forward validation from the start is vital, as financial data has temporal dependencies that traditional cross-validation overlooks. Remember, your model should predict future performance strictly based on historical data, not assumptions.
Finally, rather than searching for a one-size-fits-all algorithm, try a simple ensemble model that combines multiple weak learners. I achieved improved results by merging user accuracy metrics with market regime detection using Hidden Markov Models, outperforming more intricate deep learning approaches.
f# isn’t great for beginners doing ml projects. most resources and libraries are built for python, so you’ll hit walls constantly. I’d start with python/pandas to get your prototype working, then consider f# later if you want. good call on tracking performance across different market conditions though - just heads up, you’ll need way more data than you expect to get anything statistically meaningful.
Built something similar a few years back for tracking analyst recommendations at my company. The tricky part isn’t the ML algorithm - it’s defining what “accurate” actually means.
Don’t manually rate 1-5. Use actual stock performance metrics instead. Calculate percentage returns over your time windows (30, 60, 90 days) and let the math do the rating. Way more objective.
You’re missing some crucial features. Add recommendation timing (market open vs close), volatility index when rec was made, and stock price momentum. These matter more than you’d think.
Start simple with logistic regression or random forest before jumping into complex stuff. I used scikit-learn initially (Python though, not F#) and got decent results. Random forest worked best because it handled the mixed categorical and numerical features well.
One gotcha: your data will be heavily imbalanced. Most recommendations probably perform “okay” while very few are spectacular wins or total disasters. Use stratified sampling when training.
Check out ML.NET for F#. Microsoft’s framework plays nice with F# and has decent documentation. The classification tutorials there match what you’re trying to do.
My biggest mistake was not accounting for market-wide movements. A “good” buy recommendation during a market crash looks different than during a bull run. Normalize your performance metrics against benchmark indices.
Start with a simple binary classifier (good vs bad recommendation) before attempting the 1-5 scale. Much easier to validate and debug.