Overview

The TWAICE Pull Stack retrieves data from your databases and APIs on a scheduled, batched basis. It is designed for periodic retrieval of data.

Key Features

Flexible connectivity. The Pull Stack can connect to data sources hosted on-site, on-premise, or in another cloud environment.

Batch retrieval. Pull intervals are configured based on your infrastructure to retrieve data efficiently without overloading your network or data source. The Pull Stack is not suitable for stream data processing typical in bus systems.

Alerting and backfill. The system alerts on data retrieval failures. If a connection is interrupted, missed data is automatically backfilled once the connection is restored.

Security. Static IP addresses are provided for firewall whitelisting.

Site-to-Site VPN Support. A Site-to-Site VPN can be configured separately, at additional cost, to ensure secure and encrypted data transmission to TWAICE.

System Requirements

Python Library: A Python library (version 3.12 or higher) is required for TWAICE to connect and retrieve data from your data source.
Data Source Accessibility: The Pull Stack must be able to access your data source in batches.

Implementation Overview

Pull Stack setup and testing takes approximately 2 weeks once the necessary data connection details, credentials, and parameters are provided to TWAICE.

Set up the Pull Stack: TWAICE generates and deploys a pull client to interface with your database or API. You must have a Python library (version 3.12 or higher) that is installed and configured.
Configure Site-to-Site VPN (if applicable): If the database is on-premises, TWAICE sets up a Site-to-Site VPN to securely connect the Pull Stack to the local database.
1. Note: Site2Site VPN comes with additional costs and requires additional scoping.
Deployment: TWAICE provisions the Pull Stack in an isolated VPC within the AWS Cloud.
Configure Security Settings (optional): TWAICE applies any necessary security configurations, including IP whitelisting and other access controls.
Schedule Batch Data Retrieval: TWAICE creates the batch processing schedule to periodically pull data from the your data source.

Configuration Requirements

1. Credentials of the Datastore

A) REST Endpoint

URL

If your data source is a REST endpoint, please provide:

Field	Description
URL	The endpoint URL where the data can be accessed.

Authentication:

Please provide the credentials for whichever method your endpoint uses.

Field	Description
API key	If your endpoint uses API key-based authentication
Token	If token-based authentication is used
Username / Password	If basic authentication is used

B) Datastore

If your data source is a traditional datastore (e.g., SQL database, NoSQL database), please provide:

Field	Description
Connection string	The connection string or URL to access the datastore
Username	Username with the necessary permissions to access the datastore
Password	Password for the provided username

2. Data Format Requirements

To ensure compatibility with our Pull Stack, the data returned from your datastore or REST endpoint should be structured with the following fields:

Field	Description
Sensor tag	A unique identifier for each sensor
Timestamp	The timestamp of the recorded data
Value	The value recorded by the sensor

Example Data Format

[
  {
    "sensor_tag": "temperature_sensor_01",
    "relative_time": 1622547800,
    "sensor_value": 22.5
  },
  {
    "sensor_tag": "humidity_sensor_02",
    "relative_time": 1622547860,
    "sensor_value": 55.2
  }
]

3. Additional Configuration Parameters

TWAICE has a required questionnaire that must be answered prior to any configuration or setup efforts. These parameters ensure that the pull client setup is aligned with your system capabilities and infrastructure.

Responses should be provided to your Customer Success contact or to support@twaice.com.

The Pull Stack Questionnaire can be found here.

Need Help?

Contact your TWAICE representative or email support@twaice.com.

Data Connection: Pull Stack Configuration