Dutch Analytics - Xenia documentation

Pipeline deployment quickstart

This tutorial describes the steps to create and deploy a Pipeline in UbiOps. With pipelines, a user can set up a data flow through a sequence of Models and Connectors from a single entry point. Read the Pipelines page for more information.

In this tutorial, a pipeline will be created for weather temperature prediction. The first model in the pipeline performs an API request to retrieve publicly available KNMI data. The second model predicts the temperature of the next hour, using a moving average. Finally, there is the possibility to create a table in your database (for example, a PostgreSQL or MySQL database) and to include a connector to this database in the pipeline.

In this tutorial you will:

  • Create two models
  • Use environment variables
  • Create a pipeline
  • Add objects to the pipeline
  • Perform a pipeline request
  • Create a connector for a database
  • Perform a pipeline request and see the results in the database

Create models

Create two models by using the following specifications. If you are not familiar with creating a model, please read the Model deployment tutorial first.

Model 1:

  • Name: knmi-data-collector
  • Description: This model collects the temperature per hour from the publicly available KNMI API.
  • Input fields (structured): end_date (string)
  • Output fields (structured): data (array of strings)
  • Version name: v1
  • Language: Python 3.7
  • Memory allocation: 512
  • Zip-file: download zip

Model 2:

  • Name: temperature-predictor
  • Description: This model predicts the temperature of the next hour.
  • Input fields (structured): data (array of strings)
  • Output fields (structured): forecast (double precision)
  • Version name: moving-average
  • Language: Python 3.7
  • Memory allocation: 512
  • Zip-file: download zip

Create model environment variables

The next step is to specify the Environment Variables. The following environment variables are used in the models:

Model 1:

  • API_URL: the API URL
  • N_DAYS (optional, default 3): the number of days in the future for which predictions are computed

Model 2:

  • WINDOW_SIZE (optional, default 3): the number of hours that are used by the moving average.

There are three levels at which an environment variable can be created: project, model, and model version level. The inheritance follows the order project --> model --> model_version, which means that a project environment variable can be overwritten by a model environment variable, etc. In this tutorial, API_URL will be created at model level, and N_DAYS and WINDOW_SIZE at model version level.

Please, follow the next steps to create the environment variables.

API_URL

  1. Go to UbiOps and login.
  2. In the navigation bar go to Models -> Overview.
  3. Go to the model details page of model 1: knmi-data-collector.
  4. Click on the Advanced tab-button at the top.
  5. The table contains all environment variables available at model level. Click on the Create variable button to create a new environment variable on model level. A new row will appear in the table.
  6. Set the Name to API_URL.
  7. Set the Value to http://projects.knmi.nl/klimatologie/uurgegevens/getdata_uur.cgi.
  8. Set the Secret to No (default). Note that you are able to set Secret to true, which enables you to hide the value of the environment variable. This is, for example, very useful for API keys and passwords.
  9. Click on the create button .

N_DAYS

  1. Click on the General tab-button at the top to return to the version overview of the model knmi-data-collector.
  2. Select Model version v1 by clicking on v1.
  3. Click on the Advanced tab-button at the top.
  4. Now the table contains all environment variables available at model version level, which also shows the inherited environment variables of project and model level. Click on the Create variable button to create a new environment variable on model version level. A new row will appear in the table.
  5. Set the Name to N_DAYS.
  6. Set the Value to 1.
  7. Set the Secret to No (default).
  8. Click on the create button .

WINDOW_SIZE

  1. In the navigation bar go to Models -> Overview.
  2. Go to the model details page of model 2: temperature-predictor.
  3. Click on the Advanced tab-button at the top.
  4. Now the table contains all environment variables available at model version level. Click on the + symbol to create a new environment variable on model version level. A new row will appear in the table.
  5. Set the Name to WINDOW_SIZE.
  6. Set the Value to 2.
  7. Set the Secret to No (default).
  8. Click on the create button .

The specified environment variables are now available in the code by importing the environ variable from the os library (from os import environ). If you followed the steps above, environ['API_URL'] and environ['N_DAYS'] can be used in the knmi-data-collector model package and environ['WINDOW_SIZE'] in the temperature-predictor model package. See the ZIP files for these two models provided above.

Create pipeline

Now that the models are created and the environment variables assigned, the pipeline can be created:

  1. Go to UbiOps and login.
  2. In the navigation bar go to Pipelines -> Create pipeline.
  3. Set the Name to weather-forecast.
  4. Set the Input type to structured.
  5. Set the Input fields to:
  • Add an input field current_date with data-type string .
  1. Click on the Create button.

Add and connect pipeline objects

To add the two models to the created pipeline:

  1. Go to UbiOps and login.
  2. In the navigation bar go to Pipelines -> Overview.
  3. Click on the name of the pipeline weather-forecast.

Object 1

  1. Click on the Add new object button.
  2. Set the Reference type to Model.
  3. Set the Reference name to knmi-data-collector.
  4. Set the Reference version to v1.
  5. Set the Name to data-collector.
  6. Click on the Create button.

Object 2

  1. Click on the Add new object button.
  2. Set the Reference type to Model.
  3. Set the Reference name to temperature-predictor.
  4. Set the Reference version to moving-average.
  5. Set the Name to moving-average.
  6. Click on the Create button.

Connection 1

  1. Click on the Connect two objects button, or use drag and drop to connect start with the input of data-collector.
  2. Set the Source object to pipeline_start.
    If you use drag and drop, this is done automatically.
  3. Set the Destination object to data-collector.
    If you use drag and drop, this is done automatically.
  4. Set the Mapping.
    • Select current_date to be connected to end_date.
  5. Click on the Create button.

Connection 2

  1. Click on the Connect two objects button, or use drag and drop to connect the output of data-collector with the input of moving-average.
  2. Set the Source object to data-collector.
    If you use drag and drop, this is done automatically.
  3. Set the Destination object to moving-average.
    If you use drag and drop, this is done automatically.
  4. Set the Mapping.
    • Select data to be connected to data.
  5. Click on the Create button.

Pipeline request

Now that the pipeline has been deployed in UbiOps, it is ready to receive data. You can test the pipeline using either the UbiOps user interface or the UbiOps API.

Pipeline request using the user interface

You can test your newly created pipeline by going to its page.

  1. Click on Pipelines in the sidebar on the left and then click on your pipeline name.

  2. Now click CREATE DIRECT REQUEST. Fill in the parameters:
    - The current_date parameter. You could set the current date by using the year-month-day format (for example, "2020-01-01"). If you leave the current date empty, the model will automatically select today as current date:
    Click Create to create the request.

  3. When the model has finished processing, the results are displayed for each object in the pipeline.

Pipeline request using the API

🚧

API Token

For this part of the tutorial you will need an API Token with the correct permissions.

  1. Visit Create pipeline request.
    - Before making your first call click lock icon on the upper right corner, and fill in your UbiOps API Token in the format:
    Token 1ig03sggdqvqtvl2n7xc4q7npn7hqqe97od1qiyg
    - Click on the Try it button in order to make a manual API request.
  2. In the API explorer set all parameters for the manual pipeline insert:
    • The name of your project.
    • The name of your pipeline. Should be weather-forecast if you followed our example above.
    • The data should contain the current_date parameter. You could set the current date by using the year-month-day format (for example "2020-01-01"). If you set the current date to an empty string (""), the model will automatically select today as current date:
{
  "current_date": ""
}
  1. Execute the API request by pressing Execute button, located at the bottom of the form.

  2. When the API call is successful, you will receive a server response code 200 with a dictionary containing:

  • pipeline: The name of the pipeline.
  • model_requests: A list of requests, with each:
    • pipeline_object: The name of the pipeline object.
    • success: The state of the prediction request. Should be "true", which indicates that the model request finished.
    • request_data: The request input of the model.
    • result: The request output of the model.
    • error_message: The error that the model would generate if it were to fail. It should be "null" for this example.

(Optional) Create a table in your database

The next optional steps will describe the creation of a table in your own database that will be used to collect the results of the previous described weather forecast.

PostgreSQL
Create a table in your PostgreSQL database containing a unique identifier and a temperature_forecast column. We use in the example code below schema="weather_forecast" and table="pipeline_results". You can change this if you wish.

CREATE TABLE weather_forecast.pipeline_results
(
    id bigserial NOT NULL,
    temperature_forecast double precision NOT NULL,
    PRIMARY KEY (id)
)
WITH (
    OIDS = FALSE
);

MySQL
Create a table in your MySQL database containing a unique identifier and a temperature column. We use in the example code below table="weather_forecast". You can change this if you wish.

CREATE TABLE `weather_forecast` 
( 
  `id` SERIAL NOT NULL, 
  `temperature_forecast` DOUBLE NOT NULL 
);

You could also create a table or stream in, for example, Google Cloud BigQuery or Amazon Kinesis.

Giving UbiOps access
To make UbiOps able to access your created database, you need to whitelist our IP address:

📘

IP address of connectors

UbiOps connectors will query through a proxy with the following IPv4 address: 34.90.157.84. Make sure your firewall allows access to your storage from this IP address.

(Optional) Create a connector

Now that your table is created in your database and UbiOps has access to it, you can create a Connector .

  1. Go to UbiOps and login.
  2. In the navigation bar go to Connectors -> Create connector.
  3. Set the Name to weather-forecast-storage.
  4. Set the Connector type to the database type that matches the one you used in the previous step.
  5. Select New credentials.
  6. Set the Name to, for example, my-credentials.
  7. Set the Credentials to correspond with your database.
  8. Set the Configurations to the specific table you created for the pipeline.
  9. Specify the Input fields.
  • Set the Name to temperature_forecast.
  • Set the Type to Double precision.
  1. Click on the Create button.

After the creation, the credentials will be automatically verified. Only if the verification succeeded, you can continue with the next step.

(Optional) Add a connector to pipeline

The above described pipeline contains two objects: one that performs an API call to collect the KNMI data, and one that estimates the temperature of the next hour by calculating the moving average. This pipeline can be extended by the previously created connector. To do this, please follow the next steps.

Connector object

  1. Go to UbiOps and login.
  2. In the navigation bar go to Pipelines -> Overview.
  3. Click on the pipeline name or the details icon of the created pipeline. Should be weather-forecast if you followed our example above.
  4. Click on the Add new object to pipeline button.
  5. Set the Reference type to Connector.
  6. Set the Reference name to weather-forecast-storage.
  7. Set the Name to store-forecast.
  8. Click on the Create button.

Connector connection

  1. Click on the Connect two objects button, or use drag and drop to connect the output of moving-average with the input of store-forecast.
  2. Set the Source object to moving-average.
    If you use drag and drop, this is done automatically.
  3. Set the Destination object to store-forecast.
    If you use drag and drop, this is done automatically.
  4. Set the Mapping.
    • Select forecast to be connected to temperature_forecast.
  5. Click on the Create button.

(Optional) Create pipeline request and see results in database.

To create a pipeline request, please follow the same steps described in Section 5.

When the request is finished, the results will be available in your database.

Updated 4 days ago

Pipeline deployment quickstart


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.