AI Driven Stock Analysis and Prediction

Abstract

The comprehensive stock market analysis application revolutionizes stock prediction by utilizing a diverse range of input sources and advanced modules. Historical stock pricing data, financial statements, economic indicators, financial news, social media tweets, and company annual reports are intelligently gathered to build a robust foundation for accurate predictions. While the subsequent modules — SWOT assessment of audit reports, auditor’s report health check, financial ratios-based stock forecasting, and auditor report summarizer — play pivotal roles in providing deeper insights and enhancing analysis capabilities. Leveraging techniques such as text classification, natural language processing and regression models, these modules offer invaluable functionalities for evaluating internal and external factors, extracting vital information, predicting stock prices, and generating concise summaries. The AI stock market analysis application empowers users with informed decision-making tools, unlocking new dimensions in stock market analysis and prediction.

Introduction

The stock market is a dynamic and complex environment where investors and traders strive to make informed decisions to maximize their returns. Accurate predictions of stock prices play a crucial role in these decision-making processes. However, existing stock market analysis tools often fall short in providing comprehensive insights and fail to incorporate real-time economic data, leading to limited accuracy in predictions.

To address these limitations, there is a need for a comprehensive stock market analysis application that leverages a wide range of data sources and employs advanced techniques to forecast stock prices accurately in real-time. This blog presents a proposed solution that aims to fulfill this need and revolutionize the way stock market analysis is conducted.

The main objective of the proposed application is to provide users with a powerful and reliable tool that incorporates historical stock prices, financial statements, economic indicators, news data, and sentiment analysis over news and Twitter data. By integrating these diverse sources of information, the application aims to capture a holistic view of the market and enable users to make more informed investment decisions.

One of the key features of the application is the integration of real-time economic data. This ensures that the predictions and forecasts generated by the application are based on the most up-to-date information available, allowing users to respond quickly to market changes. By incorporating real-time economic data, the application offers comprehensive insights that are crucial for accurate stock price predictions.

Furthermore, the application utilizes machine learning algorithms, such as LSTM (Long Short-Term Memory) and Random Forest models, to analyze and interpret the complex relationships between various data points. Independent component analysis is applied to extract meaningful features and reduce dimensionality, while sentiment analysis on Twitter and news data enhances prediction accuracy. These advanced techniques enable the application to generate more reliable and accurate predictions of stock prices for various timeframes, including the next day, next 7 days, and next 15 days.

Input and Input Sources

The comprehensive stock market analysis application utilizes a diverse range of input sources to gather crucial information for accurate predictions.

1. Historical stock pricing data, including date, open, close, high, low, adjusted price, volume, dividend, and splits, is sourced from Yahoo Finance.

2. Financial statements such as cash flow, balance sheet, and income statement are obtained from AlphaVantage and EOD websites.

3. Economic indicators such as CPI, manufacturing index, inflation, housing price, job index, and labour price are collected from FRED — Federal Reserve Economic Data.

4. To capture the latest market sentiment, financial news of companies and Twitter tweets are included as inputs.

5. Company annual reports in the form of 10K forms are sourced from the United States Securities and Exchange Commission.

By incorporating these diverse and reliable input sources, the comprehensive stock market analysis application ensures a comprehensive and robust foundation for accurate and insightful predictions.

Modules

1. AI-Driven Stock Analysis

Upon selecting a stock in the user interface (UI), the comprehensive stock market analysis workflow initiates real-time data fetching for inputs such as Historical Stock Price, Financial Statements, News, Tweets, and Audit Reports. Simultaneously, economic indicators’ values are retrieved from S3, where they are stored, to be used during predictions. To provide deeper insights, sentiment analysis is performed on News and Tweets data, generating sentiment scores specific to the selected stock. Subsequently, all these distinct inputs are merged and employed for model training and predictions.

Input Data Columns for Stock Prediction model include the following:

The AI Stock Analysis model, a key component of the workflow, leverages two powerful machine learning algorithms LSTM (Long Short-Term Memory) and Random Forest models. Also, sentiment scores derived from sentiment analysis serve as valuable input features for the LSTM and Random Forest models, to increase prediction accuracy. As a result, the AI model proficiently forecasts the closing stock price for the next day, next 7 days, and next 15 days.

2. Audit Report SWOT Assessment

The SWOT analysis is a valuable tool that allows organizations to assess their internal strengths and weaknesses, while also considering the external opportunities and threats they may encounter. To conduct this analysis effectively, a combination of Text Classification Model and Natural Language Processing (NLP) techniques is employed. For identifying keywords, a dataset is obtained from the Screener website, specifically focusing on the financial results and price chart. The audit report pertaining to the stock is then extracted and processed, while eliminating any commonly occurring words with little semantic meaning, known as stop-words. To streamline the analysis process, certain limitations are implemented. Moreover, specific types of keywords are assigned to indicate strengths, weaknesses, opportunities, and threats, enabling the identification of pertinent information within the extracted audit report.

3. Auditor Report — Health Check

Auditor Report — Health Check is a very useful tool to analyze the Auditor’s Report for the selected stock and generate answers to predefined questions, several steps are taken. Firstly, the text of the report is loaded using a document loader. Next, a VectorstoreIndexCreator is utilized to create an index, enabling efficient access to the most relevant sections of the report based on the given question. The index object’s query method is then employed to pass the question to the Language Model (LLM) and retrieve the most pertinent lines from the report. Finally, the report is processed through the question answering chain using the load_qa_chain method, which retrieves the answer to the predefined question from the report’s content.

4. Financial Ratios-Based Stock Forecasting

The financial ratios-based stock forecasting approach combines the power of financial ratios and historical stock price data to train regression models that can accurately predict the stock price for the upcoming day. To gain valuable insights into the company’s financial health and performance, the audit report is carefully examined to extract relevant financial ratios. To ensure the forecasting model incorporates the latest market information, real-time stock price data is obtained from Yahoo Finance. The collected data is then utilized to train multiple regression models, including Linear Regression, Ridge Regression, Random Forest Regression, Decision Tree Regression, ElasticNet, Lasso, and Support Vector Machine (SVM). These models effectively capture the intricate relationship between the financial ratios and historical stock prices. By leveraging the input features, namely the financial ratios, these regression models generate predictions for the stock price on the following day.

5. Auditor Report Summarizer

The Auditor Report Summarizer is designed to process the company’s annual 10-K report as input, extracting the relevant text and generating concise summaries for the pages. The code is powered by the TextRank algorithm, a widely used unsupervised algorithm in web page ranking. By assessing the connectivity between sentences, scores are assigned, prioritizing those with higher scores as more significant, and subsequently incorporating them into the summary. This comprehensive overview provides an outline of the technical aspects involved in the Auditor Report Summarizer’s implementation and operation.

Summary

The AI stock market analysis application incorporates various input sources, including historical stock pricing data, financial statements, economic indicators, financial news, social media tweets, and company annual reports. These inputs serve as the foundation for accurate predictions and insightful stock market analysis. The application consists of five key modules, each contributing distinct and valuable functionalities. The AI-driven stock analysis module utilizes machine learning algorithms, LSTM and Random Forest, to forecast stock prices, while also incorporating sentiment analysis to enhance prediction accuracy. The SWOT assessment of audit reports employs text classification and NLP techniques to identify strengths, weaknesses, opportunities, and threats within the reports. The auditor’s report health check module utilizes a language model to extract relevant information and provide answers to predefined questions. The financial ratios-based stock forecasting module leverages regression models to predict stock prices based on extracted financial ratios. Lastly, the auditor report summarizer generates concise summaries of company annual reports. These modules collectively offer a broad and advanced approach to stock market analysis, empowering users with valuable insights for informed decision-making.