Joule
  • Welcome to Joule's Docs
  • Why Joule?
    • Joule capabilities
  • What is Joule?
    • Key features
    • The tech stack
  • Use case enablement
    • Use case building framework
  • Concepts
    • Core concepts
    • Low code development
    • Unified execution engine
    • Batch and stream processing
    • Continuous metrics
    • Key Joule data types
      • StreamEvent object
      • Contextual data
      • GeoNode
  • Tutorials
    • Getting started
    • Build your first use case
    • Stream sliding window quote analytics
    • Advanced tutorials
      • Custom missing value processor
      • Stateless Bollinger band analytics
      • IoT device control
  • FAQ
  • Glossary
  • Components
    • Pipelines
      • Use case anatomy
      • Data priming
        • Types of import
      • Processing unit
      • Group by
      • Emit computed events
      • Telemetry auditing
    • Processors
      • Common attributes
      • Filters
        • By type
        • By expression
        • Send on delta
        • Remove attributes
        • Drop all events
      • Enrichment
        • Key concepts
          • Anatomy of enrichment DSL
          • Banking example
        • Metrics
        • Dynamic contextual data
          • Caching architecture
        • Static contextual data
      • Transformation
        • Field Tokeniser
        • Obfuscation
          • Encryption
          • Masking
          • Bucketing
          • Redaction
      • Triggers
        • Change Data Capture
        • Business rules
      • Stream join
        • Inner stream joins
        • Outer stream joins
        • Join attributes & policy
      • Event tap
        • Anatomy of a Tap
        • SQL Queries
    • Analytics
      • Analytic tools
        • User defined analytics
          • Streaming analytics example
          • User defined analytics
          • User defined scripts
          • User defined functions
            • Average function library
        • Window analytics
          • Tumbling window
          • Sliding window
          • Aggregate functions
        • Analytic functions
          • Stateful
            • Exponential moving average
            • Rolling Sum
          • Stateless
            • Normalisation
              • Absolute max
              • Min max
              • Standardisation
              • Mean
              • Log
              • Z-Score
            • Scaling
              • Unit scale
              • Robust Scale
            • Statistics
              • Statistic summaries
              • Weighted moving average
              • Simple moving average
              • Count
            • General
              • Euclidean
        • Advanced analytics
          • Geospatial
            • Entity geo tracker
            • Geofence occupancy trigger
            • Geo search
            • IP address resolver
            • Reverse geocoding
            • Spatial Index
          • HyperLogLog
          • Distinct counter
      • ML inferencing
        • Feature engineering
          • Scripting
          • Scaling
          • Transform
        • Online predictive analytics
        • Model audit
        • Model management
      • Metrics engine
        • Create metrics
        • Apply metrics
        • Manage metrics
        • Priming metrics
    • Contextual data
      • Architecture
      • Configuration
      • MinIO S3
      • Apache Geode
    • Connectors
      • Sources
        • Kafka
          • Ingestion
        • RabbitMQ
          • Further RabbitMQ configurations
        • MQTT
          • Topic wildcards
          • Session management
          • Last Will and Testament
        • Rest endpoints
        • MinIO S3
        • File watcher
      • Sinks
        • Kafka
        • RabbitMQ
          • Further configurations
        • MQTT
          • Persistent messaging
          • Last Will and Testament
        • SQL databases
        • InfluxDB
        • MongoDB
        • Geode
        • WebSocket endpoint
        • MinIO S3
        • File transport
        • Slack
        • Email
      • Serialisers
        • Serialisation
          • Custom transform example
          • Formatters
        • Deserialisers
          • Custom parsing example
    • Observability
      • Enabling JMX for Joule
      • Meters
      • Metrics API
  • DEVELOPER GUIDES
    • Setting up developer environment
      • Environment setup
      • Build and deploy
      • Install Joule
        • Install Docker demo environment
        • Install with Docker
        • Install from source
        • Install Joule examples
    • Joulectl CLI
    • API Endpoints
      • Mangement API
        • Use case
        • Pipelines
        • Data connectors
        • Contextual data
      • Data access API
        • Query
        • Upload
        • WebSocket
      • SQL support
    • Builder SDK
      • Connector API
        • Sources
          • StreamEventParser API
        • Sinks
          • CustomTransformer API
      • Processor API
      • Analytics API
        • Create custom metrics
        • Define analytics
        • Windows API
        • SQL queries
      • Transformation API
        • Obfuscation API
        • FieldTokenizer API
      • File processing
      • Data types
        • StreamEvent
        • ReferenceDataObject
        • GeoNode
    • System configuration
      • System properties
  • Deployment strategies
    • Deployment Overview
    • Single Node
    • Cluster
    • GuardianDB
    • Packaging
      • Containers
      • Bare metal
  • Product updates
    • Public Roadmap
    • Release Notes
      • v1.2.0 Join Streams with stateful analytics
      • v1.1.0 Streaming analytics enhancements
      • v1.0.4 Predictive stream processing
      • v1.0.3 Contextual SQL based metrics
    • Change history
Powered by GitBook
On this page
  • Overview
  • Example & DSL attributes
  • Response
  • Attributes schema
  • Spatial Entities
  • Parser example
  • Attributes schema

Was this helpful?

  1. Components
  2. Analytics
  3. Analytic tools
  4. Advanced analytics
  5. Geospatial

Geo search

Real-time view on what entities are nearby with respect to geofence radius

Overview

Often when performing real-time geospatial analytics the objective is to gain a view of what entities are nearby such as friends, businesses, local interests, cell towers etc,. This processor provides the ability to perform such a function using a configurable search area and entities that may be of concern.

A geo search is performed against a spatial index using the passed event attributes which will return 0 or more nearby entities.

Example & DSL attributes

geo search:
    name: nearbyCellTowers
    search area: 8

    spatial index:
        top left coordinates: [ 0,0 ]
        width: 512
        area: 262144
        max levels: 6
        preferred max elements: 601

    spatial entities:
        parser: com.fractalworks.streams.processors.geospatial.domain.telco.CellTowerArrowParser
        source: data/mini_cells.csv

Response

geoSearchResult:
    - CellTower{radioType=UMTS, mcc=262, net=2, area=801, cell=71555, unit=-1, lon=13.359251, lat=52.52103, range=383, samples=5, changeable=1, created=1286420769, updated=1289843835, averageSignal=-92, coordinates=Tuple{x=216.0, y=132.0}, key=null}
    - CellTower{radioType=UMTS, mcc=262, net=2, area=15, cell=6652412, unit=-1, lon=13.413759, lat=52.520469, range=0, samples=1, changeable=1, created=1288894949, updated=1288894949, averageSignal=-98, coordinates=Tuple{x=216.0, y=132.0}, key=null}
    - .....

Attributes schema

Attribute
Description
Data Type
Required

name

Descriptive name of the processor function

String

Default: Random UUID

search area

Area to search

Integer Default: 8

response field

Custom field name to place result of search lookup

String Default: geoSearchResult

latitude field

Custom field name to gain latitude dimension

String Default: latitude

longitude field

Custom field name to gain longitude dimension

String Default: longitude

convertWGS84ToTileCoords

Flag to convert WGS84 coordinates to tile coordinates

Boolean Default: true

spatial index

An area is set out as a search tree. The default area is a flattened world divided out in to rectangles.

spatial entities

Spatial entities to return on a successful search

Spatial Entities

These are entities that have the required location attributes that enable the search algorithm to be executed against. Since entities are specific to the use case and business domain and/or industry the ability to leverage these assets within Joule a parser API is provided. The parser translates existing entity structures, via a file, in to a support Joule data structure.

Parser example

/*
 * Copyright 2020-present FractalWorks Ltd.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 *  you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *  http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package com.fractalworks.streams.processors.geospatial.domain.telco;

import com.fractalworks.streams.core.exceptions.TranslationException;
import com.fractalworks.streams.sdk.referencedata.serialization.ReferenceDataArrowParser;
import org.apache.arrow.vector.FieldVector;
import org.apache.arrow.vector.VectorSchemaRoot;

/**
 * CellTower Arrow row parser.
 * <p>
 * Format
 * - radio,mcc,net,area,cell,unit,lon,lat,range,samples,changeable,created,updated,averageSignal
 * <p>
 * Created by Lyndon Adams
 */
public class CellTowerArrowParser implements ReferenceDataArrowParser<CellTower> {

    /**
     * Parse provided vectorSchemaRoot to CellTower.
     *
     * @param row
     * @return CellTower
     */
    @Override
    public CellTower translate(VectorSchemaRoot row) throws TranslationException {
        CellTower cellTower = new CellTower();
        for(FieldVector v : row.getFieldVectors()){
            var str = v.getObject(0);
            switch (v.getName().toLowerCase() ){
                case "radio" -> cellTower.setRadioType(RadioType.getEnum(str.toString().intern()));
                case "mcc" -> cellTower.setMcc( ((Long)str).intValue());
                case "net" -> cellTower.setNet(((Long)str).intValue());
                case "area" -> cellTower.setArea(((Long)str).intValue());
                case "cell" -> cellTower.setCell(((Long)str).intValue());
                case "unit" -> cellTower.setUnit( (str!=null) ? ((Long)str).intValue() : -1);
                case "lon" -> cellTower.setLon((Double)str);
                case "lat" -> cellTower.setLat((Double)str);
                case "range" -> cellTower.setRange(((Long)str).intValue());
                case "samples" -> cellTower.setSamples(((Long)str).intValue());
                case "changeable" -> cellTower.setChangeable(((Long)str).intValue());
                case "created" -> cellTower.setCreated((Long) str);
                case "updated" -> cellTower.setUpdated((Long) str);
                case "averagesignal" -> cellTower.setAverageSignal( (str!=null) ?((Long)str).intValue() : -1);
                default -> throw new TranslationException(String.format("Unknown field '%s'", v.getName()));
            }
        }
        return cellTower;
    }
}

Attributes schema

Attribute
Description
Data Type
Required

parser

String

source

Source of the file

String

PreviousGeofence occupancy triggerNextIP address resolver

Last updated 6 months ago

Was this helpful?

See

See section

This example parses a cell tower file using the Apache Arrow library. CellTower extends the class to simplify the implementation.

Custom parser that converts to a custom type that extends GeoNode, see for further details on implementing a GeoNode entity

GeoNode
spatial index documentation
documentation
Spatial Entities