--- title: "Deploying a prediction service with Plumber" Author: Hong Ooi output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Plumber model deployment} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{utf8} --- This document shows how you can deploy a fitted model as a web service using ACR, ACI and AKS. The framework used is [Plumber](https://www.rplumber.io), a package to expose your R code as a service via a REST API. ## Fit the model We'll fit a simple model for illustrative purposes, using the Boston housing dataset which ships with R (in the MASS package). To make the deployment process more interesting, the model we fit will be a random forest, using the randomForest package. ```r data(Boston, package="MASS") library(randomForest) # train a model for median house price as a function of the other variables bos_rf <- randomForest(medv ~ ., data=Boston, ntree=100) # save the model saveRDS(bos_rf, "bos_rf.rds") ``` ## Scoring script for plumber Now that we have the model, we also need a script to obtain predicted values from it given a set of inputs: ```r # save as bos_rf_score.R bos_rf <- readRDS("bos_rf.rds") library(randomForest) #* @param df data frame of variables #* @post /score function(req, df) { df <- as.data.frame(df) predict(bos_rf, df) } ``` This is fairly straightforward, but the comments may require some explanation. They are plumber annotations that tell it to call the function if the server receives a HTTP POST request with the path `/score`, and query parameter `df`. The value of the `df` parameter is then converted to a data frame, and passed to the randomForest `predict` method. ## Create a Dockerfile Let's package up the model and the scoring script into a Docker image. A Dockerfile to do this would look like the following. This uses the base image supplied by Plumber (`trestletech/plumber`), installs randomForest, and then adds the model and the above scoring script. Finally, it runs the code that will start the server and listen on port 8000. ```dockerfile # example Dockerfile to expose a plumber service FROM trestletech/plumber # install the randomForest package RUN R -e 'install.packages(c("randomForest"))' # copy model and scoring script RUN mkdir /data COPY bos_rf.rds /data COPY bos_rf_score.R /data WORKDIR /data # plumb and run server EXPOSE 8000 ENTRYPOINT ["R", "-e", \ "pr <- plumber::plumb('/data/bos_rf_score.R'); pr$run(host='0.0.0.0', port=8000)"] ``` ## Build and upload the image The code to store our image on Azure Container Registry is as follows. If you are running this code, you should substitute the values of `tenant`, `app` and/or `secret` from your Azure service principal. Similarly, if you are using the public Azure cloud, note that all ACR instances share a common DNS namespace, as do all ACI and AKS instances. For more information on how to create a service principal, see the [AzureRMR readme](https://github.com/cloudyr/AzureRMR). ```r library(AzureContainers) # create a resource group for our deployments deployresgrp <- AzureRMR::get_azure_login()$ get_subscription("sub_id")$ create_resource_group("deployresgrp", location="australiaeast") # create container registry deployreg_svc <- deployresgrp$create_acr("deployreg") # build image 'bos_rf' call_docker("build -t bos_rf .") # upload the image to Azure deployreg <- deployreg_svc$get_docker_registry(as_admin=TRUE) deployreg$push("bos_rf") ``` If you run this code, you should see a lot of output indicating that R is downloading, compiling and installing randomForest, and finally that the image is being pushed to Azure. (You will see this output even if your machine already has the randomForest package installed. This is because the package is being installed to the R session _inside the container_, which is distinct from the one running the code shown here.) All Docker calls in AzureContainers, like the one to build the image, return the actual docker commandline as the `cmdline` attribute of the (invisible) returned value. In this case, the commandline is `docker build -t bos_rf .` Similarly, the `push()` method actually involves two Docker calls, one to retag the image, and the second to do the actual pushing; the returned value in this case will be a 2-component list with the command lines being `docker tag bos_rf deployreg.azurecr.io/bos_rf` and `docker push deployreg.azurecr.io/bos_rf`. ## Deploy to an Azure Container Instance The simplest way to deploy a service is via a Container Instance. The following code creates a single running container which contains our model, listening on port 8000. ```r # create an instance with 2 cores and 8GB memory, and deploy our image deployaci <- deployresgrp$create_aci("deployaci", image="deployreg.azurecr.io/bos_rf", registry_creds=deployreg, cores=2, memory=8, ports=aci_ports(8000)) ``` Once the instance is running, let's call the prediction API with some sample data. By default, AzureContainers will assign the container a domain name with prefix taken from the instance name. The port is 8000 as specified in the Dockerfile, and the URI path is `/score` indicating we want to call the scoring function defined earlier. The data to be scored---the first 10 rows of the Boston dataset---is passed in the _body_ of the request as a named list, encoded as JSON. A feature of Plumber is that, when the body of the request is in this format, it will extract the elements of the list and pass them to the scoring function as named arguments. This makes it easy to pass around relatively large amounts of data, eg if the data is wide, or for scoring multiple rows at a time. For more information on how to create and interact with Plumber APIs, consult the [Plumber documentation](https://www.rplumber.io/docs/). ```r response <- httr::POST("http://deployaci.australiaeast.azurecontainer.io:8000/score", body=list(df=MASS::Boston[1:10,]), encode="json") httr::content(response, simplifyVector=TRUE) #> [1] 25.9269 22.0636 34.1876 33.7737 34.8081 27.6394 21.8007 22.3577 16.7812 18.9785 ``` ## Deploy to a Kubernetes cluster Deploying a service to a container instance is simple, but lacks many features that are important in a production setting. A better alternative for production purposes is to deploy to a Kubernetes cluster. Such a cluster can be created using Azure Kubernetes Service (AKS). ```r # create a Kubernetes cluster with 2 nodes, running Linux (the default) deployclus_svc <- deployresgrp$create_aks("deployclus", agent_pools=agent_pool("pool1", 2)) ``` Unlike an ACI resource, creating a Kubernetes cluster can take several minutes. By default, the `create_aks()` method will wait until the cluster provisioning is complete before it returns. Having created the cluster, we can deploy our model and create a service. We'll use a YAML configuration file to specify the details for the deployment and service API. The image to be deployed is the same as before. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: bos-rf spec: selector: matchLabels: app: bos-rf replicas: 1 template: metadata: labels: app: bos-rf spec: containers: - name: bos-rf image: deployreg.azurecr.io/bos_rf ports: - containerPort: 8000 resources: requests: cpu: 250m limits: cpu: 500m imagePullSecrets: - name: deployreg.azurecr.io --- apiVersion: v1 kind: Service metadata: name: bos-rf-svc spec: selector: app: bos-rf type: LoadBalancer ports: - protocol: TCP port: 8000 ``` The following code will obtain the cluster endpoint from the AKS resource and then deploy the image and service to the cluster. The configuration details for the `deployclus` cluster are stored in a file located in the R temporary directory; all of the cluster's methods will use this file. Unless told otherwise, AzureContainers does not touch your default Kubernetes configuration (`~/kube/config`). ```r # grant the cluster pull access to the registry deployreg_svc$add_role_assignment(deployclus_svc, "Acrpull") # get the cluster endpoint deployclus <- deployclus_svc$get_cluster() # create and start the service deployclus$create("bos_rf.yaml") ``` To check on the progress of the deployment, run the `get()` methods specifying the type and name of the resource to get information on. As with Docker, these correspond to calls to the `kubectl` commandline tool, and again, the actual commandline is stored as the `cmdline` attribute of the returned value. ```r deployclus$get("deployment bos-rf") #> Kubernetes operation: get deployment bos-rf --kubeconfig=".../kubeconfigxxxx" #> NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE #> bos-rf 1 1 1 1 5m svc <- read.table(text=deployclus$get("service bos-rf-svc")$stdout, header=TRUE) #> Kubernetes operation: get service bos-rf-svc --kubeconfig=".../kubeconfigxxxx" #> NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE #> bos-rf-svc LoadBalancer 10.0.8.189 52.187.249.58 8000:32276/TCP 5m ``` Once the service is up and running, as indicated by the presence of an external IP in the service details, let's test it with a HTTP request. The response should be the same as it was with the container instance. Notice how we extract the IP address from the service details above. ```r response <- httr::POST(paste0("http://", svc$EXTERNAL.IP[1], ":8000/score"), body=list(df=MASS::Boston[1:10, ]), encode="json") httr::content(response, simplifyVector=TRUE) #> [1] 25.9269 22.0636 34.1876 33.7737 34.8081 27.6394 21.8007 22.3577 16.7812 18.9785 ``` Finally, once we are done, we can tear down the service and deployment. Depending on the version of Kubernetes the cluster is running, deleting the service may take a few minutes. ```r deployclus$delete("service", "bos-rf-svc") deployclus$delete("deployment", "bos-rf") ``` And if required, we can also delete all the resources created here, by simply deleting the resource group (AzureContainers will prompt you for confirmation): ```r deployresgrp$delete() ``` ### Security note One important thing to note about the above example is that it is **insecure**. The Plumber service is exposed over HTTP, and there is no authentication layer: anyone on the Internet can contact the service and interact with it. Therefore, it's highly recommended that you should provide at least some level of authentication, as well as restricting the service to HTTPS only (this will require deploying an ingress controller to the Kubernetes cluster). You can also create the AKS resource as a private cluster; however, be aware that if you do this, you can only interact with the cluster endpoint from a host which is on the cluster's own subnet. ## See also Plumber is a relatively simple framework for creating and deploying services. As an alternative, the [RestRserve](https://restrserve.org) package is a more comprehensive framework, built on top of functionality provided by Rserve. It includes features such as automatic parallelisation, support for HTTPS, and support for basic and bearer authentication schemes. See the vignette "Deploying an ACI service with HTTPS and authentication" for more information.