Introducing AI & Machine Learning
We are proud to share we have been diligently working on technology enhancements to our consultancy platform. Our latest strategic partnership with a prestigious university in UK has resulted in the implementation of a 2-year project, which ultimately introduces Artificial Intelligence and Machine Learning to our commodity analysis. You can find more information about the project below.
October 2022 predicted prices by our current model against actual prices
A note from the M.D
Looking back to when our data on cotton began in the 1800s, seasons and weather were believed to be a result of a higher power. Jump forward to the 21st century, modern science and information have spread across the globe and with it attitudes have evolved. What was once considered a mystical phenomena is now commonly put down to physics and mathematics. Patterns appear all around us and as computing power has increased so too has our ability to identify and model these patterns. From social media algorithms that seem to be ‘listening to you’ to stock market trading algorithms that analyse trends on a global scale. It is only a matter of time before all phenomena can be reduced down to inputs and near-certain responses.
The commodity market and stock market are the largest data processors on the planet. They take information from a plethora of macro and micro factors from global events to the emotions of traders and reduce this information into a single metric: price. The price of a commodity or stock follow logic and patterns. The science of price has been studied for millennia and while some have come close to predicting it the vast majority haven’t come close to understanding the whole picture.
So What has Changed?
Firstly, very few of the people that studied price so intently have been price agnostic. Take cotton for example, from farmers to producers both have a direction they hope price will go. This has been one of the reasons EAP have been so successful in predicting the market in recent times. They don’t hold a position in the markets they advise on, so they read the patterns without a fixed idea of direction. This has led EAP to developing their own models and rules and respecting said rules and models as any price agnostic market analyst would do. Nevertheless, EAP are not the first market analyst of their kind, nor are they the first to develop their own models. It is the introduction of Artificial Intelligence that is taking EAP to the next level.
Artificial Intelligence is a buzzword used by scientists and journalists alike, but in simple terms what does it mean and how does it relate to EAP? It is a way of testing theories and the theory in question is the price prediction of cotton and other soft commodities and grains. The accuracy and usefulness of any AI test is determined by the quality of two factors: information and the model used to analyse it. EAP have over 100 years worth of quality data on commodity markets and their tried and tested analysis models have been proven right time and time again. The thousands of data points EAP have collected on price, supply and demand and macro events are too many for one analyst or team of analysts to memorise and consult every time the markets move, but we have developed a computer programme to process and test historical market patterns in a matter of seconds to deliver a prediction, rather than the days or weeks it would take a person to process. This model is being improved every day. It is proving EAP’s market theories right and wrong using this historical pattern testing system and refining the accuracy of the predictions it delivers.
Currently the model is producing predictions with an 80% accuracy rate, this has been and will continue to be improved as the data processing model is refined by the experienced analyst team at EAP. This is not possible without AI expertise and it is why EAP are partnering with Liverpool University who bring over 30 years of Artificial Intelligence expertise and have expressed a great interest in the model and its impressive accuracy at this early stage. EAP is proving markets are not random and modelling the patterns that they follow. We are calculating future prices, rather than predicting them and we hope to become leaders in price prediction science over the next decade.
Project Summary
Earlam and Partners (EAP) provide commodity consultancy in the context of cotton markets. EAP has a mathematical model for analysing price movements, which holds market information, to which weights are assigned manually to make predictions, delivered in weekly market reports.
To deliver EAP’s services at scale, automation and enhanced accuracy are required. Using Lightweight Explainable AI the patterns underpinning the weighting of information can be identified. Patterns that can then be used to build a Commodity Price Prediction (CPP) system that is both Lightweight (runs on a standard desktop machine) and explainable (tells users how predictions were derived). Delivery of live predictions using the proposed CPP system will help clients across the supply chain to become more resilient and better manage their operations in the face of increasing market volatility.
The vision is to expand beyond Cotton to other commodities (Soybeans, Corn, Wheat, Coffee, Sugar and Cocoa) where farmers and manufacturers are increasingly impacted by volatility from climate change, war and other factors. After 10 years of developing and refining an analysis model for cotton, EAP need to bring in academic AI expertise, to expand across more markets and help more people than they can alone, hence the proposed KTP.
Strategic Aim
EAP have successfully provided commodity consultancy services for over 13 years. EAP has proven processes for collecting data and delivering insights into 5 areas: Cotton market fundamentals (1), hedge fund positions in Cotton (2), reading of Cotton chart patterns (3), Cotton seasonal statistical analysis (4) and overall macro analysis (5). Today the organisation has a trained team of consultants whose job is to provide support to existing clientele to ensure well-informed operational and financial decisions. EAP manage their decisions by weighting the information (“1 to 5” aforementioned) according to experience and the exact goings on in the international market place and is led by Jo Earlam.
The vision of this partnership is to bring the expertise in Artificial Intelligence modelling from the University of Liverpool and use it to test and improve the EAP mathematical model, which holds micro and macro information, both current and historical. More effective commodity price predictions, for a wider range of commodities, can be made by harnessing the tools and techniques of deep learning. Having a software that mirrors EAP analyst decision making will help deliver more consistent insights and teach analysts in the company to understand the markets more algorithmically and therefore better. Through the development of Artificial Intelligence market prediction model, price prediction and insights will become increasingly automated, accurate and timely and reliance on individual expertise will wane.
EAP has ambitions of delivering its services through software rather than as a traditional consultancy that interacts with clients solely via phone calls and emails. Recently, EAP have built a web platform to improve the scalability of their services. However analysis is limited by the capacity of their in-house analysts. EAP does not have inhouse machine learning and artificial intelligence expertise and therefore wish to partner with a prestigious British University. By the end of this project, EAP plans to have a system that delivers automated market predictions that can be accessed around the clock. The predictions will be supported by, but not reliant upon, direct consultations and insights from EAP analysts.
Biweekly reporting is the current means through which EAP clients receive insights, these reports are shared with clients across every part of the cotton supply chain. This partnership will ultimately make EAP’s prediction model more accurate and available on-demand, which is disruptive and like no other consultancy offering in the space. This partnership has the potential to transform EAP into innovation leaders in cotton and agricultural commodity consultancy.
Missing Knowledge
EAP are interested in providing their services using an online environment, the Commodity Price Prediction (CPP) system, that features an AI prediction model that is both Lightweight and Explainable. The environment will be hosted on a web-based platform (hence the model needs to be “lightweight”) underpinned by AI mechanisms that will predict commodity price indexes in an explainable manner. The vision is to provide support for growers, especially cotton growers across the world. Not just large growers but also small growers farming only a few acres of land.
EAP see an AI solution as central to the delivery of the proposed platform. Realisation of the platform will require a range of AI technologies that EAP does not currently have, including: (i) data cleaning (Project Stage 1), (ii) prediction (Project Stage 1), (iii) Explainable AI and Lightweight AI (Project Stage 2), (iv) data visualisation (Project Stage 4) and (iv) the maintenance of prediction of models (Project Stage 5). All areas of AI where the Knowledge Base partner has expertise. AI is one of the Department of Computer Science at Liverpool’s two research pillars with some twenty academics, including the KB Supervisors and Academic Lead. Explainable AI, or EXAI as it is frequently referred to, Lightweight AI and the maintenance of prediction models are all current areas of computer science research of interest to the Knowledge Base partner. ExAI is motivated by the global need to gain the trust of end users of AI systems so that the potential of AI can be fully realised. One way of achieving this is by providing explanations of how AI systems make decisions. EAP believe that explainability is an essential feature for their envisioned CPP system; EAP’s customers will want to understand how the predictions made by the CPP system are arrived at. Lightweight AI is concerned with building deep learning prediction models that require only a limited amount of processing power. The motivation is to allow access to deep learning using devices with only limited power and, from an environmental perspective, to reduce the amount of power that deep learning systems use. The fundamental idea is to build deep learning models and then to compress them so that the underlying network reduces in size. EAP believe Lightweight AI will be essential given that they wish to deliver predictions using an online platform. The maintenance of prediction models has become of increasing concern motivated by the idea that we do not wish to relearn deep learning models whenever new data comes available; instead we would wish to only update the relevant parts of the model.