MLOps Lifecycle - Revision history

Admino739mjm7: /* Tools and Solutions */

2023-11-20T11:07:18Z

Tools and Solutions

← Older revision		Revision as of 11:07, 20 November 2023
Line 14:		Line 14:

	== Tools and Solutions==		== Tools and Solutions==
			The following section enlists indicative tools that are suitable for each phase of the AI/MLOps lifecycle. The list is non-exhaustive.
	=== Phase I: Design===		=== Phase I: Design===

Admino739mjm7: /* Relevant Categories */

2023-11-16T14:36:59Z

Relevant Categories

← Older revision		Revision as of 14:36, 16 November 2023
Line 121:		Line 121:

	== Relevant Categories==		== Relevant Categories==
	[[Category:Theoretical Foundations]]		[[Category:Theoretical Foundations]][[Category:MLOps / AIOps]]

Admino739mjm7 at 15:18, 10 November 2023

2023-11-10T15:18:49Z

← Older revision		Revision as of 15:18, 10 November 2023
Line 22:		Line 22:


	{\| class="wikitable" style="margin:auto; width:100%"		{\| class="wikitable" style="margin:auto; width:100%; line-height: 1.5em"
	\|-		\|-
	! MLOps Step!! Tool Name!! Tool Description!! Link		! MLOps Step!! Tool Name!! Tool Description!! Link
Line 57:		Line 57:


	{\| class="wikitable" style="margin:auto; width:100%"		{\| class="wikitable" style="margin:auto; width:100%; line-height: 1.5em"
	\|-		\|-
	! MLOps Step!! Tool Name!! Tool Description!! Link		! MLOps Step!! Tool Name!! Tool Description!! Link
Line 90:		Line 90:


	{\| class="wikitable" style="margin:auto; width:100%"		{\| class="wikitable" style="margin:auto; width:100%; line-height: 1.5em"
	\|-		\|-
	! MLOps Step!! Tool Name!! Tool Description!! Link		! MLOps Step!! Tool Name!! Tool Description!! Link

Admino739mjm7 at 15:17, 10 November 2023

2023-11-10T15:17:46Z

← Older revision		Revision as of 15:17, 10 November 2023
Line 2:		Line 2:

	== MLOps Lifecycle Basics ==		== MLOps Lifecycle Basics ==
	<p>		<p style="line-height: 1.5em">
	The emergence of Artificial Intelligence (AI) and Machine Learning (ML) has revolutionised multiple industries, ranging from healthcare and finance to manufacturing and transportation. With the fast growth of data availability and the need for real-time decision-making, Cloud-Edge AI ML Operations (AI MLOps) has become a powerful approach, being able to combine cloud computing, edge devices, and advanced ML algorithms.		The emergence of Artificial Intelligence (AI) and Machine Learning (ML) has revolutionised multiple industries, ranging from healthcare and finance to manufacturing and transportation. With the fast growth of data availability and the need for real-time decision-making, Cloud-Edge AI ML Operations (AI MLOps) has become a powerful approach, being able to combine cloud computing, edge devices, and advanced ML algorithms.
	</p>		</p>
Line 9:		Line 9:
	</p>		</p>

	<p>		<p style="line-height: 1.5em">
	The lifecycle of MLOps encompasses the end-to-end management and optimisation of ML models and workflows, integrating DevOps and data science practices. The lifecycle always begins with the problem definition and data collection, where the business goals and relevant data sources are identified. This is followed by data preprocessing, including cleaning, transformation, and feature engineering, to ensure the data is ready for modelling. The next phase involves model development, exploring and evaluating various algorithms and techniques. Once a suitable model is selected, it undergoes training and validation using historical data. After model training, the focus shifts to deployment and monitoring. The model is deployed to production environments where it interacts with real-time data, and monitoring tools are put in place to track its performance, detect anomalies, and ensure reliability.		The lifecycle of MLOps encompasses the end-to-end management and optimisation of ML models and workflows, integrating DevOps and data science practices. The lifecycle always begins with the problem definition and data collection, where the business goals and relevant data sources are identified. This is followed by data preprocessing, including cleaning, transformation, and feature engineering, to ensure the data is ready for modelling. The next phase involves model development, exploring and evaluating various algorithms and techniques. Once a suitable model is selected, it undergoes training and validation using historical data. After model training, the focus shifts to deployment and monitoring. The model is deployed to production environments where it interacts with real-time data, and monitoring tools are put in place to track its performance, detect anomalies, and ensure reliability.
	</p>		</p>

Admino739mjm7: /* Phase III: Operations */

2023-11-10T10:13:04Z

Phase III: Operations

← Older revision		Revision as of 10:13, 10 November 2023
Line 83:		Line 83:

	=== Phase III: Operations===		=== Phase III: Operations===

			The Operations Phase is the final phase of the MLOps Lifecycle and is comprised of 3 sub-phases:
			*'''ML Model Deployment''': To integrate a trained machine learning model into a production environment so that it can be used to make predictions or decisions based on new data
			*'''CI/CD Pipelines''': The purpose of continuous integration and continuous deployment (CI/CD) pipelines is to automate the machine learning model development and deployment process
			*'''Monitoring and Triggering''': To continuously monitor the performance of machine learning models deployed in production environments and trigger actions when necessary


	{\| class="wikitable" style="margin:auto; width:100%"		{\| class="wikitable" style="margin:auto; width:100%"
	\|-		\|-
	! MLOps Step!! Tool Name!! Tool Description!! Link		! MLOps Step!! Tool Name!! Tool Description!! Link
	\|-		\|-
	\| ~~Example~~\|\| ~~Example~~\|\| ~~Example~~\|\| ~~Example~~		\| ML Model Deployment\|\| Kubeflow\|\| Open-source platform for deploying and managing machine learning workflows on Kubernetes\|\| https://www.kubeflow.org
	\|-		\|-
	\| ~~Example~~\|\| ~~Example~~\|\| ~~Example~~\|\| ~~Example~~		\| ML Model Deployment\|\| Docker\|\| Platform for building, packaging, and deploying applications in containers\|\| https://www.docker.com/
			\|-
			\| ML Model Deployment\|\| Microsoft Azure Machine Learning\|\| Cloud-based platform for building, training, and deploying machine learning models\|\| https://azure.microsoft.com/en-us/products/machine-learning/
			\|-
			\| CI/CD Pipelines\|\| Jenkins\|\| Open-source automation server that provides a wide range of features\|\| https://www.jenkins.io/
			\|-
			\| CI/CD Pipelines\|\| GitLab CI/CD\|\| Platform for continuous integration and continuous deployment that provides features for building, testing, and deploying machine learning models\|\| https://docs.gitlab.com/ee/ci/
			\|-
			\| CI/CD Pipelines\|\| CircleCI\|\| Cloud-based continuous integration and continuous deployment platform\|\| https://circleci.com
			\|-
			\| Monitoring and Triggering\|\| Prometheus\|\| Open-source monitoring system that provides features for monitoring and alerting on various aspects of the machine learning pipeline\|\| https://prometheus.io/
			\|-
			\| Monitoring and Triggering\|\| Grafana\|\| Open-source platform for data visualisation and monitoring that can be used to create dashboards and alerts for monitoring machine learning models\|\| https://grafana.com/
			\|-
			\| Monitoring and Triggering\|\| Kibana\|\| Open-source platform for data visualisation and analysis that can be used to monitor and analyse the performance of machine learning models\|\| https://www.elastic.co/es/kibana/
			\|-
			\| Monitoring and Triggering\|\| Industreweb\|\| Industreweb Collect Engine allows for detection of events and the triggering of mitigating actions. Industreweb Display dashboards allow for screens to be created to reflect ML status\|\| https://www.industreweb.co.uk/
	\|}		\|}

Admino739mjm7: /* Phase II: Model Development */

2023-11-10T10:02:57Z

Phase II: Model Development

← Older revision		Revision as of 10:02, 10 November 2023
Line 50:		Line 50:

	=== Phase II: Model Development===		=== Phase II: Model Development===

			The Model Development Phase has the following 3 sub-phases:
			*'''Data Engineering''': To design, build, and maintain the infrastructure required to support the collection, storage, processing, and analysis
			*''' ML Model Engineering''': To design and develop machine learning models that can perform complex tasks with high accuracy and reliability
			*'''Model Testing and Validation''': To evaluate the performance and accuracy of a machine learning model and to ensure that it can generalise well to new data


	{\| class="wikitable" style="margin:auto; width:100%"		{\| class="wikitable" style="margin:auto; width:100%"
	\|-		\|-

Admino739mjm7: /* Phase I: Design */

2023-11-10T10:00:54Z

Phase I: Design

← Older revision		Revision as of 10:00, 10 November 2023
Line 15:		Line 15:
	== Tools and Solutions==		== Tools and Solutions==
	=== Phase I: Design===		=== Phase I: Design===

	The Design Phase encompasses 3 sub-phases:		The Design Phase encompasses 3 sub-phases:
	*Requirements Engineering: To identify, analyse, document, and manage the needs and expectations of stakeholders for a software development project		*'''Requirements Engineering''': To identify, analyse, document, and manage the needs and expectations of stakeholders for a software development project
	*ML Use Cases Prioritisation: The process of identifying and prioritising potential use cases for applying machine learning algorithms in a particular domain or industry		*'''ML Use Cases Prioritisation''': The process of identifying and prioritising potential use cases for applying machine learning algorithms in a particular domain or industry
	*Data Availability Check: To ensure that the necessary data is available and accessible for use in a machine learning project		*'''Data Availability Check''': To ensure that the necessary data is available and accessible for use in a machine learning project


	{\| class="wikitable" style="margin:auto; width:100%"		{\| class="wikitable" style="margin:auto; width:100%"
	\|-		\|-

Admino739mjm7: /* Phase I: Design */

2023-11-10T10:00:07Z

Phase I: Design

← Older revision		Revision as of 10:00, 10 November 2023
Line 15:		Line 15:
	== Tools and Solutions==		== Tools and Solutions==
	=== Phase I: Design===		=== Phase I: Design===
			The Design Phase encompasses 3 sub-phases:
			*Requirements Engineering: To identify, analyse, document, and manage the needs and expectations of stakeholders for a software development project
			*ML Use Cases Prioritisation: The process of identifying and prioritising potential use cases for applying machine learning algorithms in a particular domain or industry
			*Data Availability Check: To ensure that the necessary data is available and accessible for use in a machine learning project
	{\| class="wikitable" style="margin:auto; width:100%"		{\| class="wikitable" style="margin:auto; width:100%"
	\|-		\|-
Line 27:		Line 31:
	\| Requirements Engineering\|\| Diagrams.net\|\| Good general purpose Technical Diagram tool to help standardise, between partners, the format of architectural, process diagrams and layouts\|\| https://app.diagrams.net/		\| Requirements Engineering\|\| Diagrams.net\|\| Good general purpose Technical Diagram tool to help standardise, between partners, the format of architectural, process diagrams and layouts\|\| https://app.diagrams.net/
	\|-		\|-
	\| ML Use ~~cases~~ Prioritisation\|\| DataRobot\|\| Automated machine learning platform that includes features for use case prioritisation. It uses a combination of artificial intelligence and human expertise to identify and prioritise potential use cases\|\| https://www.datarobot.com/		\| ML Use Cases Prioritisation\|\| DataRobot\|\| Automated machine learning platform that includes features for use case prioritisation. It uses a combination of artificial intelligence and human expertise to identify and prioritise potential use cases\|\| https://www.datarobot.com/
	\|-		\|-
	\| ML Use ~~cases~~ Prioritisation\|\| Microsoft Azure ML Studio\|\| It includes tools for identifying and prioritising use cases\|\| https://azure.microsoft.com/en-us/products/machine-learning/		\| ML Use Cases Prioritisation\|\| Microsoft Azure ML Studio\|\| It includes tools for identifying and prioritising use cases\|\| https://azure.microsoft.com/en-us/products/machine-learning/
	\|-		\|-
	\| ML Use ~~cases~~ Prioritisation\|\| Rapid Miner\|\| It includes tools for evaluating and selecting the most promising use cases for implementation\|\| https://rapidminer.com/		\| ML Use Cases Prioritisation\|\| Rapid Miner\|\| It includes tools for evaluating and selecting the most promising use cases for implementation\|\| https://rapidminer.com/
	\|-		\|-
	\| Data Availability Check\|\| SageMaker Data Wrangler\|\| A tool that provides a user-friendly interface for data cleaning and preparation\|\| https://aws.amazon.com/sagemaker/data-wrangler		\| Data Availability Check\|\| SageMaker Data Wrangler\|\| A tool that provides a user-friendly interface for data cleaning and preparation\|\| https://aws.amazon.com/sagemaker/data-wrangler

Admino739mjm7: /* Phase II: Model Development */

2023-11-10T09:56:33Z

Phase II: Model Development

@@ Line 47: / Line 47: @@
 ! MLOps Step!! Tool Name!! Tool Description!! Link
 |-
-| Example|| Example|| Example|| Example
+| Data Engineering|| Apache Spark|| Open-source data processing engine that provides features for batch processing, streaming, and machine learning|| https://spark.apache.org/
 |-
-| Example|| Example|| Example|| Example
+| Data Engineering|| Apache Kafka|| Distributed streaming platform that provides features for data processing and messaging|| https://kafka.apache.org/
 |}

Admino739mjm7: /* Phase I: Design */

2023-11-10T09:45:26Z

Phase I: Design

← Older revision		Revision as of 09:45, 10 November 2023
Line 31:		Line 31:
	\| ML Use cases Prioritisation\|\| Microsoft Azure ML Studio\|\| It includes tools for identifying and prioritising use cases\|\| https://azure.microsoft.com/en-us/products/machine-learning/		\| ML Use cases Prioritisation\|\| Microsoft Azure ML Studio\|\| It includes tools for identifying and prioritising use cases\|\| https://azure.microsoft.com/en-us/products/machine-learning/
	\|-		\|-
	\| ~~example~~\|\| ~~example~~\|\| ~~example~~\|\| ~~example~~		\| ML Use cases Prioritisation\|\| Rapid Miner\|\| It includes tools for evaluating and selecting the most promising use cases for implementation\|\| https://rapidminer.com/
	\|-		\|-
	\| ~~example~~\|\| ~~example~~\|\| ~~example~~\|\| ~~example~~		\| Data Availability Check\|\| SageMaker Data Wrangler\|\| A tool that provides a user-friendly interface for data cleaning and preparation\|\| https://aws.amazon.com/sagemaker/data-wrangler
	\|-		\|-
	\| ~~example~~\|\| ~~example~~\|\| ~~example~~\|\| ~~example~~		\| Data Availability Check\|\| Trifacta\|\| Data preparation platform that provides features for data profiling, cleaning, and enrichment\|\| https://www.trifacta.com
	\|-		\|-
	\| ~~example~~\|\| ~~example~~\|\| ~~example~~\|\| ~~example~~		\| Data Availability Check\|\| Azure Data Factory\|\| Data integration platform that provides features for data preparation, transformation, and validation\|\| https://azure.microsoft.com/en-us/products/data-factory
	\|-		\|-
	\| ~~example~~\|\| ~~example~~\|\| ~~example~~\|\| ~~example~~		\| Data Availability Check\|\| Industreweb\|\| Large library of Protocol shopfloor edge connectivity, can be used to verify if a suitable protocol connector is available, and if not then add a new type to support the pilot requirements\|\| https://www.industreweb.co.uk/
	\|-
	~~\| example\|\| example\|\| example\|\| example~~
	\|-
	~~\| example\|\| example\|\| example\|\| example~~
	\|}		\|}